You are not logged in.
- Topics: Active | Unanswered
#51 2021-07-07 06:37:00
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
You could try bfq on the main kernel for a cross test and then see https://github.com/zen-kernel/zen-kernel/issues
Offline
#52 2021-07-07 10:05:06
- sabroad
- Member
- Registered: 2015-05-24
- Posts: 242
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Curious indeed (unfortunately no HDDs avail here).
The Vanilla kernel is performing a lot better than Zen wrt to refaults:
workingset_nodes 463785 419052 -9.6%
workingset_refault_anon 64467 37 -99.9% # swap thrashing
workingset_refault_file 177280 22388 -87.4% # page-cache thrashing
workingset_activate_anon 0 0 +0 # is THP refaulting pages into inactive?
workingset_activate_file 0 2144 +2144 # is readahead refaulting pages into inactive?
workingset_restore_anon 33 0 -33
workingset_restore_file 454 276 -39.2%
workingset_nodereclaim 308254 442951 +43.7%
pswpin 2852 0 -2852
pswpout 5082 0 -5082Digging deeper:
MemFree: 147920 156908 kB +6.1%
MemAvailable: 15714816 13988416 kB -11.0%
[...]
Active(anon): 65912 516 kB -99.2%
Inactive(anon): 29452 1635776 kB +5,454.0% # much less memory pressure
Active(file): 67204 432884 kB +544.1% # much better identifying active page cache
Inactive(file): 15267532 13408828 kB -12.2%
[...]
Dirty: 6491020 2516284 kB -61.2% # much less dirty pages accumulated
[...]
AnonHugePages: 45056 0 kB -45056Something is off about the dirty pages accumulated: https://www.kernel.org/doc/Documentation/sysctl/vm.txt
dirty_ratio
Contains, as a percentage of total available memory that contains free pages
and reclaimable pages, the number of pages at which a process which is
generating disk writes will itself start writing out dirty data.The total available memory is not equal to total system memory.
It's as if Zen is ignoring vm.dirty_ratio entirely and allowing the copy process to grow it to 40.9%:
~ > sudo sysctl -a | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200
--
saint_abroad
Offline
#53 2021-07-07 12:11:32
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
@seth running BFQ on vanilla kernel:
Activating bfq as scheduler on Vanilla kernel:
# echo bfq > /sys/block/sdb/queue/scheduler Checking if it's active:
~ > cat /sys/block/sdb/queue/scheduler
mq-deadline kyber [bfq] noneMeminfo during the 100 1GB files copy test:
~ > cat /proc/meminfo
MemTotal: 16341556 kB
MemFree: 160836 kB
MemAvailable: 13714688 kB
Buffers: 209524 kB
Cached: 13224116 kB
SwapCached: 12 kB
Active: 1089080 kB
Inactive: 14125280 kB
Active(anon): 3040 kB
Inactive(anon): 1816100 kB
Active(file): 1086040 kB
Inactive(file): 12309180 kB
Unevictable: 11260 kB
Mlocked: 11260 kB
SwapTotal: 10485756 kB
SwapFree: 10485488 kB
Dirty: 2408348 kB
Writeback: 15544 kB
AnonPages: 1791992 kB
Mapped: 606940 kB
Shmem: 34236 kB
KReclaimable: 496244 kB
Slab: 590700 kB
SReclaimable: 496244 kB
SUnreclaim: 94456 kB
KernelStack: 12592 kB
PageTables: 27960 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18656532 kB
Committed_AS: 6279948 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 34712 kB
VmallocChunk: 0 kB
Percpu: 3008 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1299280 kB
DirectMap2M: 10186752 kB
DirectMap1G: 5242880 kBVmstat during the test:
~ > vmstat 5 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 4 268 153064 209292 13735728 0 0 417 12348 937 5508 18 10 49 23 0
0 3 268 152712 209348 13728972 0 0 10 107187 1654 8706 3 7 33 57 0
0 4 268 154180 209372 13728636 0 0 39 121697 1172 7302 3 6 37 54 0Iostat during the test:
~ > iostat -xk 5 3
Linux 5.12.14-arch1-1 (archlinux) 07/07/2021 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
17.41 0.10 10.44 23.35 0.00 48.70
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 0.13 3.31 0.00 0.00 0.63 26.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sdb 17.37 598.21 12.41 41.68 15.85 34.44 24.89 18369.82 10.45 29.57 201.88 737.97 0.00 0.00 0.00 0.00 0.00 0.00 1.96 51.59 5.40 27.30
avg-cpu: %user %nice %system %iowait %steal %idle
2.44 0.00 6.96 52.98 0.00 37.62
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 5.20 20.80 0.00 0.00 203.15 4.00 120.00 125575.20 9.20 7.12 249.31 1046.46 0.00 0.00 0.00 0.00 0.00 0.00 0.80 287.50 31.20 99.66
avg-cpu: %user %nice %system %iowait %steal %idle
3.00 0.21 6.20 68.68 0.00 21.91
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 12.40 83.20 0.00 0.00 101.95 6.71 107.40 97447.20 26.40 19.73 219.28 907.33 0.00 0.00 0.00 0.00 0.00 0.00 4.40 120.18 25.34 97.86Output of egrep 'workingset|pswp' /proc/vmstat after the test:
~ > egrep 'workingset|pswp' /proc/vmstat
workingset_nodes 385065
workingset_refault_anon 4
workingset_refault_file 18200
workingset_activate_anon 1
workingset_activate_file 623
workingset_restore_anon 0
workingset_restore_file 131
workingset_nodereclaim 394399
pswpin 0
pswpout 0Other notes:
- Responsibility doesn't suffer much
- Already open apps have next to no lags/freezes
- However, opening new apps takes longer
I checked the zen kernel and came across this issue, although it's from 2015:
https://github.com/zen-kernel/zen-kernel/issues/104
The above issue says to check the configuration of liquorix and zen, and I was able to find both but I couldn't compare them as the files were too long. I tried using diff to compare them but the output is still long, enough to exceed my terminal scrollback limit.
Liquorix configuration: https://github.com/damentz/liquorix-pac … ig-arch-64
Zen configuration: https://github.com/archlinux/svntogit-p … unk/config
@sabroad Is there any way to get Zen to obey the vm.dirty_ratio? I've also posted the results of bfq on vanilla linux, so the problem doesn't seem to lie in the I/O scheduler.
Offline
#54 2021-07-08 08:23:24
- sabroad
- Member
- Registered: 2015-05-24
- Posts: 242
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
@sabroad Is there any way to get Zen to obey the vm.dirty_ratio?
The value vm.dirty_ratio is actually tracked by nr_pages (but reported in bytes).
Can you post following stats during test?
egrep 'nr_.*(pages|dirty)' /proc/vmstatLast edited by sabroad (2021-07-08 10:39:33)
--
saint_abroad
Offline
#55 2021-07-08 11:56:27
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Here's the output of that command, taken during the copy test, on the Zen kernel:
~ > egrep 'nr_.*(pages|dirty)' /proc/vmstat
nr_free_pages 37671
nr_zspages 0
nr_anon_pages 71049
nr_file_pages 3743980
nr_dirty 1526795
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 31
nr_page_table_pages 5341
nr_dirty_threshold 1865901
nr_dirty_background_threshold 746178EDIT:
The output of the same command for the Vanilla kernel:
~ > egrep 'nr_.*(pages|dirty)' /proc/vmstat
nr_free_pages 39102
nr_zspages 0
nr_anon_pages 333621
nr_file_pages 3561143
nr_dirty 629868
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 0
nr_page_table_pages 5842
nr_dirty_threshold 710244
nr_dirty_background_threshold 354688I also found this virtual memory parameter: vm.min_free_kbytes. I checked its value using sysctl on the vanilla kernel, and it seems too low to me, which is probably causing apps to load slower when memory is almost fully occupied by cache:
~ > sysctl vm.min_free_kbytes
vm.min_free_kbytes = 67584Last edited by HotDogEnemy (2021-07-08 19:56:56)
Offline
#56 2021-07-09 08:25:57
- sabroad
- Member
- Registered: 2015-05-24
- Posts: 242
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
How sure are you about this for your system on the Zen kernel?
~ > sudo sysctl -a | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200
Because the Zen dirty defaults are
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 20
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 50
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200Which sure matches up with the thresholds on your system:
Zen:
~ > egrep 'nr_.*(pages|dirty)' /proc/vmstat
nr_free_pages 37671
nr_zspages 0
nr_anon_pages 71049
nr_file_pages 3743980
nr_dirty 1526795
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 31
nr_page_table_pages 5341
nr_dirty_threshold 1865901 # ~50% of 3743980
nr_dirty_background_threshold 746178 # ~20% of 3743980Vanilla:
~ > egrep 'nr_.*(pages|dirty)' /proc/vmstat
nr_free_pages 39102
nr_zspages 0
nr_anon_pages 333621
nr_file_pages 3561143
nr_dirty 629868
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 0
nr_page_table_pages 5842
nr_dirty_threshold 710244 # ~20% of 3561143
nr_dirty_background_threshold 354688 # ~10% of 3561143Try explicitly setting values in /etc/sysctl.d/40-dirty.conf to see if this solves/improves the situation for you.
Last edited by sabroad (2021-07-09 09:07:11)
--
saint_abroad
Offline
#57 2021-07-09 18:04:23
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
I booted into zen, and checked the values again, and you're right, they line up perfectly with the zen defaults:
~ > sudo sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 20
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 50
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200My boot manager is rEFInd and it seems to switch kernels based on which ones got updated recently, that's why the values might've been incorrect. I apologize for not checking for the kernel version at first because I thought the problem wasn't related to the kernel.
I used the forum post you sent to make a 40-dirty.conf with the following contents:
~ > cat /etc/sysctl.d/40-dirty.conf
vm.dirty_background_bytes = 300000000
vm.dirty_bytes = 500000000After rebooting, these were the values for dirty on Zen:
~ > sudo sysctl -a | grep dirty
vm.dirty_background_bytes = 300000000
vm.dirty_background_ratio = 0
vm.dirty_bytes = 500000000
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 0
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200However it didn't help the zen kernel much, it still swapped and caused lags. Here's the meminfo:
~ > cat /proc/meminfo
MemTotal: 16339592 kB
MemFree: 156604 kB
MemAvailable: 14940096 kB
Buffers: 26120 kB
Cached: 14722796 kB
SwapCached: 98092 kB
Active: 399632 kB
Inactive: 14817184 kB
Active(anon): 254684 kB
Inactive(anon): 217036 kB
Active(file): 144948 kB
Inactive(file): 14600148 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485756 kB
SwapFree: 9455116 kB
Dirty: 385508 kB
Writeback: 16540 kB
AnonPages: 422220 kB
Mapped: 164728 kB
Shmem: 3772 kB
KReclaimable: 375876 kB
Slab: 484952 kB
SReclaimable: 375876 kB
SUnreclaim: 109076 kB
KernelStack: 12864 kB
PageTables: 22376 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18655552 kB
Committed_AS: 5442408 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 36080 kB
VmallocChunk: 0 kB
Percpu: 2848 kB
HardwareCorrupted: 0 kB
AnonHugePages: 26624 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 338768 kB
DirectMap2M: 15341568 kB
DirectMap1G: 1048576 kBAnd the nr values using egrep:
~ > egrep 'nr_.*(pages|dirty)' /proc/vmstat
nr_free_pages 37882
nr_zspages 0
nr_anon_pages 137184
nr_file_pages 3692359
nr_dirty 105784
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 13
nr_page_table_pages 5672
nr_dirty_threshold 122071
nr_dirty_background_threshold 73243Same story on the vanilla kernel, it doesn't help much - while copying the ram gets full of cache and opening new apps takes about 5 times longer, but already open windows have minimal lags and freezes.
Meminfo during copy on vanilla kernel:
MemTotal: 16341556 kB
MemFree: 162448 kB
MemAvailable: 13607000 kB
Buffers: 173980 kB
Cached: 13194484 kB
SwapCached: 280 kB
Active: 794156 kB
Inactive: 14585084 kB
Active(anon): 3464 kB
Inactive(anon): 2079052 kB
Active(file): 790692 kB
Inactive(file): 12506032 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485756 kB
SwapFree: 10484720 kB
Dirty: 449820 kB
Writeback: 7564 kB
AnonPages: 2010516 kB
Mapped: 534596 kB
Shmem: 71748 kB
KReclaimable: 485440 kB
Slab: 580132 kB
SReclaimable: 485440 kB
SUnreclaim: 94692 kB
KernelStack: 14112 kB
PageTables: 28272 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18656532 kB
Committed_AS: 7355708 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 36360 kB
VmallocChunk: 0 kB
Percpu: 2896 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 457552 kB
DirectMap2M: 11028480 kB
DirectMap1G: 5242880 kBnr values using egrep:
nr_free_pages 38765
nr_zspages 0
nr_anon_pages 500712
nr_file_pages 3344319
nr_dirty 109776
nr_shmem_hugepages 0
nr_file_hugepages 0
nr_anon_transparent_hugepages 0
nr_page_table_pages 7070
nr_dirty_threshold 122071
nr_dirty_background_threshold 73243Offline
#58 2021-07-09 21:35:45
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Same story on the vanilla kernel, it doesn't help much - while copying the ram gets full of cache and opening new apps takes about 5 times longer, but already open windows have minimal lags and freezes.
That's borderline expectable because of the load on the disk from the copy process, loading executable, libraries, icons, configs … all takes much longer.
BFQ is supposed to prefer interactive™ workloads over backgroundstuff™, though and ionice should™ work (it won't on mq-deadline)
Offline
#59 2021-07-10 09:59:31
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
So I tested this using BFQ with both the zen and vanilla kernels with ionice disabled and enabled, on the default dirty settings for each kernel. For testing the amount of time to open apps, I tried opening GIMP while the copying process was going on.
Average time to start gimp on Zen when idle: 13s
Average time to start gimp on Vanilla when idle:
On mq-deadline: ~15s
On bfq: ~ 14s
>Zen kernel + BFQ
Ram gets used super quickly, and once swapping begins everything begins to stutter. Apps get pushed out of RAM and into swap, htop shows that the RAM in use decreases from 2.15GB to 1.21GB without having closed any applications.
Time to start GIMP: 5 min (23 times longer than default)
> Zen kernel + bfq + ionice
Still laggy, but the cursor doesn't lag as much as long as you keep actively using the system. Swap still gets used, but that's expected by now on Zen. Somehow if I don't actively use the system, GIMP takes 6 minutes to start up.
Time to start GIMP: 1min 45s (8 times longer)
>Vanilla kernel + bfq
Set the scheduler to bfq, and ran the test. Already open apps have minimal lag, however opening new apps still takes long. All in all, more responsive than zen during copying.
Time to start GIMP: 2min 49s
> Vanilla kernel + bfq + ionice
Interactivity remains the same as with no ionice, but opening gimp takes a little less time. However over multiple tests the time varied wildly, from 1 min to 3 min.
Time to start GIMP: 1min 25s
Last edited by HotDogEnemy (2021-07-10 10:06:39)
Offline
#60 2021-07-10 14:15:53
- sabroad
- Member
- Registered: 2015-05-24
- Posts: 242
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
>Zen kernel + BFQ
Ram gets used super quickly, and once swapping begins everything begins to stutter. Apps get pushed out of RAM and into swap, htop shows that the RAM in use decreases from 2.15GB to 1.21GB without having closed any applications.
Time to start GIMP: 5 min (23 times longer than default)
Now try with Zen defaults and nocache with sensible bwlimit (eg. 50000 KBPS):
vm.dirty_background_ratio = 20
vm.dirty_ratio = 50The only way (TM) to avoid this [1] is to throttle the copy to match the [...] HDD bandwidth:
nocache rsync --bwlimit=KBPS src dst
Theory:
1. nocache won't grow/trash the file cache;
2. bwlimit enables dirty flushes to stay in the background;
3. Not all swap usage is bad (if pages aren't swapped in);
Can you post following stats during test?
egrep 'workingset|pswp' /proc/vmstatLast edited by sabroad (2021-07-10 14:27:23)
--
saint_abroad
Offline
#61 2021-07-10 20:59:59
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Zen dirty settings:
~ > sudo sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 20
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 50
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200The command for the test:
~ > nocache rsync --bwlimit=50000 iotest/* iotest2/This time for the test, I only let rsync copy 38GB of files because I didn't want it to take too long.
Cachestats of test files directory, before copy:
~ > cachestats iotest/*
pages in cache: 0/262144 (0.0%) [filesize=1048576.0K, pagesize=4K]Cachestats of test files directory after copy:
~ > cachestats iotest/*
pages in cache: 0/262144 (0.0%) [filesize=1048576.0K, pagesize=4K]/proc/meminfo before copy:
MemTotal: 16339592 kB
MemFree: 12929216 kB
MemAvailable: 14002852 kB
Buffers: 154492 kB
Cached: 1175576 kB
SwapCached: 0 kB
Active: 2350584 kB
Inactive: 676944 kB
Active(anon): 1734576 kB
Inactive(anon): 15952 kB
Active(file): 616008 kB
Inactive(file): 660992 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485756 kB
SwapFree: 10485756 kB
Dirty: 5972 kB
Writeback: 0 kB
AnonPages: 1648116 kB
Mapped: 570152 kB
Shmem: 53060 kB
KReclaimable: 99248 kB
Slab: 167932 kB
SReclaimable: 99248 kB
SUnreclaim: 68684 kB
KernelStack: 14624 kB
PageTables: 26312 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18655552 kB
Committed_AS: 6231712 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 37808 kB
VmallocChunk: 0 kB
Percpu: 2880 kB
HardwareCorrupted: 0 kB
AnonHugePages: 239616 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 316240 kB
DirectMap2M: 6975488 kB
DirectMap1G: 9437184 kB/proc/meminfo after copy:
MemTotal: 16339592 kB
MemFree: 1082408 kB
MemAvailable: 14250664 kB
Buffers: 18872 kB
Cached: 13054292 kB
SwapCached: 178272 kB
Active: 691848 kB
Inactive: 13196604 kB
Active(anon): 540664 kB
Inactive(anon): 321952 kB
Active(file): 151184 kB
Inactive(file): 12874652 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485756 kB
SwapFree: 8881852 kB
Dirty: 108 kB
Writeback: 0 kB
AnonPages: 726500 kB
Mapped: 230480 kB
Shmem: 47328 kB
KReclaimable: 479872 kB
Slab: 621460 kB
SReclaimable: 479872 kB
SUnreclaim: 141588 kB
KernelStack: 13056 kB
PageTables: 30152 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18655552 kB
Committed_AS: 6993756 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 36280 kB
VmallocChunk: 0 kB
Percpu: 2992 kB
HardwareCorrupted: 0 kB
AnonHugePages: 92160 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 381776 kB
DirectMap2M: 14249984 kB
DirectMap1G: 2097152 kBegrep during copy (NOTE: the values below were zero before swap came into use):
~ > egrep 'workingset|pswp' /proc/vmstat
workingset_nodes 100632
workingset_refault_anon 55587
workingset_refault_file 17394
workingset_activate_anon 0
workingset_activate_file 6975
workingset_restore_anon 3
workingset_restore_file 2139
workingset_nodereclaim 3014
pswpin 10641
pswpout 21061While running the test, the ram got filled with cache slower, most likely due to nocache and bwlimit. It was all smooth until swap started being used- as more time went on, the laggier it became and the more interactivity suffered. It slowly became similar to the experience when using cp without nocache on Zen, like I've described before.
And finally, time to open GIMP: ~30s (might be wrong, the system began lagging a bit more just after testing this, so it could've taken more time.)
On the contrary, using nocache with cp prevents swapping and buildup of cache, though apps still take a bit of time to load (GIMP took 45 seconds), but this is expected with high I/O. With nocache, apart from the no caching of copied files, zen behaves just like the vanilla kernel when using cp.
This line from the very bottom of the nocache github page might explain the rsync behaviour:
Note however, that rsync uses sockets, so if you try a nocache rsync, only the local process will be intercepted.
Last edited by HotDogEnemy (2021-07-10 21:26:45)
Offline
#62 2021-07-10 21:51:01
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Notice that unless you dropped the caches before running the tests, the gimp loading time can be impacted by that (depending on how much of its file requirements are still available in the cache)
It seems that despite more active anon pages and more inactive file pages the zen kernel swaps more aggressively - what if you set vm.swappiness to "1"?
Offline
#63 2021-07-10 22:17:33
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Yep, I took the cache into account when testing gimp startup time, I used the below command to clear cache before starting each test:
# echo 3 > /proc/sys/vm/drop_cachesAlso, while testing with vm.swappiness=1, should I use rsync or cp, and should I use nocache?
Should I change anything else? This article mentions changing vm.vfs_cache_pressure to 50 : https://rudd-o.com/linux-and-free-softw … o-fix-that
It's late here so I'll post a reply with the results as soon as I'm free.
Offline
#64 2021-07-10 22:25:57
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
nocache maybe but essentially should™ not matter - nor should™ vm.vfs_cache_pressure in this context (which is about the directory tree, ie. what files are where)
We just want to know whether we can prevent the Zen kernel from swapping by telling it to preferably drop caches.
Offline
#65 2021-07-11 08:47:14
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Zen sysctl vm settings:
~ > sysctl -a | grep vm
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.compaction_proactiveness = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 20
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 50
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.extfrag_threshold = 500
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256 256 32 0 0
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
sysctl: permission denied on key 'vm.stat_refresh'
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.numa_stat = 1
vm.numa_zonelist_order = Node
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.page_lock_unfairness = 5
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 1 # Changed value for the purpose of test, will revert to 60 after reboot
vm.unprivileged_userfaultfd = 0
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0/proc/meminfo before the test:
MemTotal: 16339592 kB
MemFree: 11974432 kB
MemAvailable: 13207296 kB
Buffers: 173432 kB
Cached: 1319988 kB
SwapCached: 0 kB
Active: 2588164 kB
Inactive: 792172 kB
Active(anon): 1924636 kB
Inactive(anon): 16096 kB
Active(file): 663528 kB
Inactive(file): 776076 kB
Unevictable: 9864 kB
Mlocked: 9864 kB
SwapTotal: 10485756 kB
SwapFree: 10485756 kB
Dirty: 1840 kB
Writeback: 40 kB
AnonPages: 1829240 kB
Mapped: 583404 kB
Shmem: 49648 kB
KReclaimable: 92548 kB
Slab: 162512 kB
SReclaimable: 92548 kB
SUnreclaim: 69964 kB
KernelStack: 13280 kB
PageTables: 25568 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18655552 kB
Committed_AS: 6504624 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 36672 kB
VmallocChunk: 0 kB
Percpu: 3008 kB
HardwareCorrupted: 0 kB
AnonHugePages: 342016 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 897872 kB
DirectMap2M: 8491008 kB
DirectMap1G: 7340032 kB/proc/meminfo during the test:
MemTotal: 16339592 kB
MemFree: 159348 kB
MemAvailable: 14714588 kB
Buffers: 12700 kB
Cached: 14481692 kB
SwapCached: 78848 kB
Active: 310560 kB
Inactive: 14523480 kB
Active(anon): 211600 kB
Inactive(anon): 142484 kB
Active(file): 98960 kB
Inactive(file): 14380996 kB
Unevictable: 9864 kB
Mlocked: 9864 kB
SwapTotal: 10485756 kB
SwapFree: 8836316 kB
Dirty: 6255768 kB
Writeback: 8036 kB
AnonPages: 325284 kB
Mapped: 124528 kB
Shmem: 9804 kB
KReclaimable: 412764 kB
Slab: 541212 kB
SReclaimable: 412764 kB
SUnreclaim: 128448 kB
KernelStack: 13520 kB
PageTables: 25904 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18655552 kB
Committed_AS: 6496036 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 36656 kB
VmallocChunk: 0 kB
Percpu: 3008 kB
HardwareCorrupted: 0 kB
AnonHugePages: 38912 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 944976 kB
DirectMap2M: 11589632 kB
DirectMap1G: 4194304 kBegrep of workingset during the test:
~ > egrep 'workingset|pswp' /proc/vmstat
workingset_nodes 347070
workingset_refault_anon 136321
workingset_refault_file 76130
workingset_activate_anon 0
workingset_activate_file 3312
workingset_restore_anon 12
workingset_restore_file 1063
workingset_nodereclaim 7489
pswpin 19382
pswpout 27890Changing swappiness to 1 doesn't help interactivity, and zen still prefers swapping over dropping caches it seems. Even audio seems to be stuttering while the copy test is going on, something I didn't test before. Zen vm.dirty values are at default.
GIMP opening time: 1min 50s
Offline
#66 2021-07-11 14:02:34
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
It looks like the zen kernel prefers swap thrashing over dropping inactive file caches - regardless even of the configured swappiness…
=> https://github.com/zen-kernel/zen-kernel/issues
Offline
#67 2021-07-11 21:16:38
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
I assumed you were telling me to check the issues section of the zen github page, so I went and had a look, and these two issues were the most similar:
1. Kernel is lagging when copying large files : (https://github.com/zen-kernel/zen-kernel/issues/26)
In this one the poster says that they tracked down the issue being in setting dirty_background_ratio and dirty_ratio too high. Proposed solution was to set dirty_background_bytes and dirty_bytes to something low such as 128MB and 256MB respectively. I'll try it out with with the dirty_ratio values of the vanilla kernel and the values I just mentioned, and post the results when its done.
2. Zen kernel swapping as hell : (https://github.com/zen-kernel/issues/104)
Here one of the developers says to try and return to the default dirty values, but another developer says that it will only delay the behaviour that is described. He then goes on to say to change the kernel configuration and use Liquorix configuration as a reference. The only differences i know of is that Liquorix uses MuQSS by default while zen doesn't, plus I don't know how to change the configuration of a kernel.
Oh also, I've been thinking of updating the very first page of this thread with what we know so far so that more people can help out easier than being intimidated by the three pages worth of posts. What do you think?
Offline
#68 2021-07-11 21:23:35
- seth
- Member
- Registered: 2012-09-03
- Posts: 51,684
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
No, I'm suggesting to file an issue there. This looks like a zen-only bug.
What do you think?
Very good idea and also include "zen" in the subject.
Offline
#69 2021-07-12 05:03:53
- HotDogEnemy
- Member
- Registered: 2020-11-24
- Posts: 72
Re: [SOLVED] I/O operations cause lags/stutters on zen kernel
Alright, done, let me know if I missed anything in the edited comment(#1). I've also opened an issue on the linux-zen github.
Offline