Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

schnafte · 2024-03-11 08:20:31

Hi all,

my colleagues and me are working on time-synchronized network measurements (in time-sensitive networking/TSN). We have a testbed with cheap commodity hardware, that we use to send and transmit data, usually via a TSN-capable Ethernet switch (dedicated hardware).

In the past, we have employed Ubuntu for software compatibility reasons (with a kernel with realtime patches [1], created by us and the official Ubuntu kernel via Ubuntu Pro).
Now, we have set up a second, hardware equivalent system with Arch Linux. Fortunately, the rt-patched kernel (`linux-rt-lts`) is available from the repository without any circumstances (thanks!).

The strange is thing and the reason for this post is, that we run `cyclictest` [2,3] on both machines and the performance differs a lot! Basically, the Arch Linux installation behaves as if there is not real-time patch enabled.

We have run the `cyclictest` for a while but it is basically immediately, even without stress on the CPU, noticeable.

We've used multiple kernels: 5.11 and 5.15 on Ubuntu, and 6.1 and 6.6 on Arch. We have also tried building the kernel by ourselves.

We also want to conduct measurements with the exact same hardware (not 2x equal hardware) using both distros and both w/o rt and w/ rt kernel to be 100% sure. At the moment, there are only snapshots and short tests, but in general it is like this: the Ubuntu rt kernel has latencies in `cyclictest` with less than 15us as maximum (similar to the Wiki), whereas Arch linux-rt-lts has maximum values in the range of hundreds of microseconds.

Before doing further tests, I just wanted to ask, if there is something we forgot to enable/set?

We need very low jitter for our research, as we use the machine as very simple cyclic sender (talker). With Ubuntu and a Python 3.12 script, we can send data in a 1ms cycle with usually less than 10us of deviation.

Thanks for your help. I try to conduct further measurement asap to proof the above mentioned.

[1] https://wiki.linuxfoundation.org/realtime/start
[2] https://wiki.linuxfoundation.org/realti … test/start
[3] https://wiki.archlinux.org/title/Realti … cyclictest

schnafte · 2024-03-22 12:45:11

Hi, here are some measurements now. It is less critical though as I thought. However, I think it also depends on the hardware. But you can still see, that the latency is higher for the arch system.
Kernel used was 6.6 for Arch and 5.19 for Ubuntu.

Ubuntu 22.04 LTS

policy: fifo: loadavg: 5.07 5.03 4.59 6/153 3207 
T: 0 ( 3198) P:80 1:200 C:9165511 Min: 2 Act: 2 Avg: 2 Max: 79 
T: 1 ( 3199) P:80 1:200 1:9165503 Min: 2 Act: 2 Avg: 2 Max: 54 
T: 2 ( 3200) P:80 1:200 1:9165460 Min: 2 Act: 3 Avg: 2 Max: 138 
T: 3 ( 3201) P:80 1:200 C:9165480 min: 2 Act: 2 Avg: 2 Max: 108 
real 30m33.174s 
user 0m3.376s 
sys 0m12.077s

Ubuntu 22.04 RT-LTS

policy: fifo: loadavg: 6.75 6.90 6.03 5/195 1074 
T: 0 ( 1042) P:80 1:200 C:9095321 Min: 3 Act: 3 Avg: 3 Max: 19
T: 1 ( 1043) P:80 1:200 C:9095298 Min: 3 Act: 3 Avg: 3 Max: 20
T: 2 ( 1044) P:80 1:200 C:9095255 Min: 2 Act: 3 Avg: 3 Max: 42
T: 3 ( 1045) P:80 1:200 1:9095212 Min: 2 Act: 4 Avg: 3 Max: 22
real 30m19.152s 
user 0m1.160s 
sys 0m20.680s 

Archlinux LTS

policy: fifo: loadavg: 4.07 4.06 3.57 5/136 776 
T: 0 ( 729) P:80 1:200 C:9095184 Min: 2 Act: 2 Avg: 2 Max: 154 
T: 1 ( 730) P:80 1:200 6:9095167 Min: 2 Act: 3 Avg: 2 Max: 155
T: 2 ( 731) P:80 1:200 6:9095167 Min: 2 Act: 2 Avg: 2 Max: 156
T: 3 ( 732) P:80 1:200 C:9095151 Min: 2 Act: 2 Avg: 2 Max: 154
real 30m19.127s 
user 0m1.182s 
sys 0m13.068s 

Archlinux RT-LTS

policy: fifo: loadavg: 7.16 7.31 6.60 5/197 87456 
T: 0 (18858) P:80 1:200 C:8999792 Min: 2 Act: 5 Avg: 5 Max: 49
T: 1 (18859) P:80 1:200 C:8999755 Min: 2 Rct: 5 Avg: 5 Max: 46
T: 2 (18860) P:80 1:200 C:8999727 Min: 2 Act: 5 Avg: 5 Max: 54 
T: 3 (18861) P:80 1:200 C:8999702 Min: 2 Act: 5 Avg: 5 Max: 54
real 30m0.069s 
user 0m3.242s 
sys 0m38.393s

Lone_Wolf · 2024-03-23 09:37:24

Since kernel timers are involved, have you verified whether ubuntu and arch kernels use the same value for CONFIG_HZ ?

schnafte · 2024-03-25 07:43:27

Hi, I haven't considered this, yet. Thanks for making me aware of this. If I understand correctly, this is the value of the timer frequency, so if they would not match it could be a possible reason. In general, a higher frequency could help to increase the accuracy of the obtained values.

I checked on the Ubuntu file system:

grep "CONFIG_HZ" /boot/config-5.15.0-1057-realtime
# CONFIG_HZ_PERIODIC is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

And if I am not wrong, the setting for the Arch Linux kernel should be defined here:
https://gitlab.archlinux.org/archlinux/ … Kconfig.hz
which means, it is by default also set to 250Hz.

Last edited by schnafte (2024-03-25 07:43:53)

jl2 · 2024-03-25 09:30:50

I don't really know, but these are the default values if you do 'make defconfig'. It would make sense that these are upstream defaults that Arch doesn't bother changing, but rather changes it in .config.
(as said, don't know)

Instead, compare 'zcat /proc/config.gz | grep CONFIG_HZ'

schnafte · 2024-03-25 14:58:32

Alright, I checked with this command, and there it is 1000Hz. Can someone explain to me, what this actually means? A link to a good source is probably enough.

jl2 · 2024-03-25 19:42:47

Alright, I checked with this command, and there it is 1000Hz. Can someone explain to me, what this actually means? A link to a good source is probably enough.

The Hz defines how often the kernel pauses all computing stuff and does scheduling and timekeeping tasks (from here)

If you're really motivated you can compile your own kernel with ubuntu's config ( 'zcat /proc/config.gz' while in ubuntu), and measure if you feel any difference.

schnafte · 2024-03-26 06:53:56

This might be an option, we have compiled the kernel under Ubuntu before we were aware to a the PREEMPT_RT patches via Ubuntu Pro.

But do you think this can cause the differences in measurements? For the ones above, we have reused the same hardware and compiled cyclictest from git. So the only difference is the Linux distribution itself. As scheduling and related stuff happens in the kernel space, I would expect fairly similar results. Can the newer Arch Linux kernel be a reason or are there more parameters related to the measurement deviations that can have an impact?

jl2 · 2024-03-26 07:54:41

This might be an option, we have compiled the kernel under Ubuntu before we were aware to a the PREEMPT_RT patches via Ubuntu Pro.

That's not the point. I was asking if you could compile a rt kernel with the ubuntu kernel's config under Arch.
This will give you ubuntu's kernel under arch linux so we can test that one.

Last edited by jl2 (2024-03-26 07:54:58)

schnafte · 2024-03-26 09:14:33

Ah, alright, I misunderstood you. I have compiled an Arch Linux kernel as described here: https://wiki.archlinux.org/title/Kernel … ild_system
Could you please briefly outline, how I can use the Ubuntu configuration? Would I still use the Arch system or would it be "simply" cloning the vanilla kernel via git (from here: https://github.com/torvalds/linux) and then use default Linux tools to configure the kernel?

jl2 · 2024-03-26 11:20:16

use this command for the rt-kernel:

pkgctl repo clone --protocol=https linux-rt-lts

then 'zcat /proc/config.gz > config' while in ubuntu, somehow get that file over to arch:

/where_ever_you_ran_the_pkgctl_command/linux-rt-lts/-+
               +--config <=replace this file with the config
               \__PKGBUILD

And then run 'makepkg -sifc --skipinteg' in that directory. that should install it's makedeps, then compile and install it and clean up after.

schnafte · 2024-03-26 11:35:08

Thanks a lot, I will get back to you, when I have new results.

Arch Linux

#1 2024-03-11 08:20:31

Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#2 2024-03-22 12:45:11

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#3 2024-03-23 09:37:24

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#4 2024-03-25 07:43:27

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#5 2024-03-25 09:30:50

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#6 2024-03-25 14:58:32

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#7 2024-03-25 19:42:47

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#8 2024-03-26 06:53:56

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#9 2024-03-26 07:54:41

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#10 2024-03-26 09:14:33

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#11 2024-03-26 11:20:16

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

#12 2024-03-26 11:35:08

Re: Performance of Arch linux-rt-lts kernel: higher latency than Ubuntu?

Board footer