[SOLVED] zram swapping activates too late compared to other distros

arijo1 · 2024-03-05 05:04:36

I've tested bunch of distros (Lubuntu,Fedora,Pop!_os) and they all seem to do much better with zram than Arch, the problem is that I can't convince Arch to work like this.
On Arch swapping to zram begins too late and the system either becomes completely unresponsive before it starts swapping or the CPU usage spikes and it starts dumping to zram in the last moment.
Zswap is disabled.

Some examples
On Arch (bare metal) with 16GB RAM swapping begins when 14GB/15.6GB (reported by htop) is being used, the system just don't have chance to use zram and it becomes unresponsive most of the times with half or less of the swap being used.
Pop_os, Fedora and Lubuntu (all tested in VMs) start swapping when system reaches 50-65% memory usage which leads to zram actually being used and no crazy cpu usage spikes. Only pop_os seems to be using something different than the defaults. I also tested Arch in VM and it seems to have the same behaviour as when on bare metal.

I've tried the following on Arch but changing settings has no effect

/etc/systemd/zram-generator.conf

[zram0]
zram-size = ram / 1.5
compression-algorithm = zstd
swap-priority = 180
fs-type = swap

/etc/sysctl.d/99-vm-zram-parameters.conf

vm.vfs_cache_pressure=500
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0

Last edited by arijo1 (2024-03-06 12:01:02)

solskogen · 2024-03-05 08:16:42

Does it help if you set vm.swappiness to 100 (which is max)?

arijo1 · 2024-03-05 08:59:19

solskogen wrote:

Does it help if you set vm.swappiness to 100 (which is max)?

Archwiki says that this is only for older kenels, new ones should have max of 200.
But I just tested with X=10,20,100 for swap-priority = X and vm.swappiness = X , didn't notice any difference.

Meanwhile Lubuntu seems to be using the defaults

/usr/bin/init-zram-swapping

  echo $mem > /sys/block/zram0/disksize
  mkswap /dev/zram0
  swapon -p 5 /dev/zram0

It also uses the same values as default Arch for
vm.swappiness
vm.watermark_boost_factor
vm.watermark_scale_factor
vm.page-cluster

seth · 2024-03-05 09:06:06

Edit: F5, @solskogen
No it's not and it also doesn't control "please use swap way before it's actually required"
https://chrisdown.name/2018/01/02/in-de … setting-be

start swapping when system reaches 50-65% memory usage which leads to zram actually being used and no crazy cpu usage

I kinda doubt that theory.
Try "transparent_hugepage=never" or "transparent_hugepage=madvise" (just a hunch on what's actually going on)

Last edited by seth (2024-03-05 09:06:51)

arijo1 · 2024-03-05 09:37:33

Tested and there seems to be no difference

I kinda doubt that theory.

It's easy to test with just live iso of Lubuntu since it comes with zram enabled. On VM with 2gb ram it starts swapping when ~1.1GB is full, Arch on same VM starts swapping ~1.45GB but begins more aggressively (or at least it seems so). I tested VMs with more RAM and things go similarly.

Last edited by arijo1 (2024-03-05 09:37:55)

seth · 2024-03-05 13:57:58

"RAM full" isn't really a thing. What's your actual metric?
/proc/meminfo? free? Some GUI gauge?

How do you fill the RAM?
Open a browser?

cat <(head -c 1G /dev/zero) <(sleep 60) | tail

Are there other swap devices?

Reg. the CPU, do you use the same algorithm on all systems?
Are the other conditions equal

zramctl --output-all

You could try eg. the zen-kernel, but I don't think that the kernel on any OS will start to random swap out w/o any need just because it feels like it.

arijo1 · 2024-03-05 20:54:44

I have wasted your time for which I'm sorry(Pop_OS is still rather interesting since it swaps so early), I just tried few times

 cat <(head -c 8G /dev/zero) <(sleep 60) | tail

in my Arch VM with 4GB ram and it went surprisingly well (despite arch swapping much later than the other distros) until both the ram and zram were full and system froze but swapping didn't cause any freezing (unlike on my bare metal install which always freezes before swapping).
Turns out that my HDD is trash and the way 'stress' fills the RAM makes it struggle but I didn't notice since it's really quiet, other distros did better since I was running them from live iso.
I'm just gonna reinstall the bare metal Arch and mark as solved if this fixes it, could be something I messed years ago.

Previously i was using htop to check while opening multiple tabs in browser or using 'stress'

Now I tested with

 cat <(head -c 1000M /dev/zero) <(sleep 60) | tail

by increasing with 100MB while watching what 'free - h' reports

On VM with 4GB RAM, this time I installed them
Arch starts swapping when 'free' reports 3.6Gi or 3.7Gi used memory out of 3.8Gi
Lubuntu starts around 3.4Gi, lzo compression
Pop_OS starts around 2.6Gi, zstd

seth · 2024-03-05 21:24:56

Pop!_OS might use a highly customized kernel because of the System76 ties.
One thing you may want to try, in case other tests should be inconclusive, is to read urandom instead of zero (the latter obviously being insanely compressible)

You btw. didn't waste anything here, you're generating information.

arijo1 · 2024-03-06 11:58:09

After reinstall I got it to work properly but it wasn't only the reinstall that helped and you might be right about the custom kernel on Pop_OS since I can't replicate their behavior even after copying their config for zram.

Here is important thing from Pop_OS configuration

# Ensure at least 1% of total memory is free to avoid system freeze.
MINIMUM=$(awk '/MemTotal/ {printf "%.0f", $2 * 0.01}' /proc/meminfo)
CURRENT=$(sysctl vm.min_free_kbytes | awk '{print $3}')
if ((MINIMUM > CURRENT)); then
    sysctl -w "vm.min_free_kbytes=${MINIMUM}"
fi

Indeed setting "vm.min_free_kbytes" to something like 100MB avoids the freeze which can take few seconds before swapping begins or quite longer because system runs out of memory if there is program that fills the RAM too fast.
I think this should be included in the wiki since the difference it makes is significant.

"vm.swappiness=180" also makes a difference but it's kind of irrelevant compared to the difference "vm.min_free_kbytes = 100000" made for me.

Also, this for some reason does nothing and doesn't fill the RAM

  cat <(head -c 1000M /dev/urandom) <(sleep 60) | tail

I tested by opening 10-15GB files with nano, zram definitely is doing its thing because without it my system freezes after some time but with it the file eventually opens. System may become bit slow since nano is reading from the HDD but if I kill it things eventually normalize.
Also tested with C++ program that fills vector with random values since I wanted to see how things work without HDD being read constantly, this way I didn't even notice slowness when zram was filling.
The test seemed to work properly since the compression ratio I got from zramctl was quite poor, >1 but <2

In the end what made things even better was oom killer since the default one is terrible, just default earlyoom in my case works nice. Now the system won't even freeze, it just kills the process that takes too much RAM after both RAM and zram swap are full.

seth · 2024-03-06 15:17:46

https://wiki.archlinux.org/title/Improv … M_handling

But iirc it's RAM dependent anyway.

cat /proc/meminfo
sysctl vm.min_free_kbytes # for the default value you get

dev/urandom seems to cause random (:)) escape sequences here that prevent the redirection, I get some output on the terminal on direct "head -c100M /dev/urandom | tail" invocation.

arijo1 · 2024-03-06 19:09:31

Default "vm.min_free_kbytes" is 67584 on both my 16GB RAM system and 4GB VM.
Seems OK but I guess that if RAM is filled too fast (which Is happening here) the system tends to get too low on memory and freeze for moment before starting to swap.
Changing the value to 100000 made huge difference, with 67584 the system freezes every few seconds but with value being 100000 I don't even notice when swap starts being used.
If zram gets too full and compression ratio is bad (happens with a lot of youtube tabs opened) things tend to get slow but I don't expect zram to be replacement for buying more RAM.

As for /dev/urandom I would never have guessed and I'm glad I didn't spend unreasonable amount of time trying to get it to work.

I'm quite happy how things are working now. Already enabled zram and earlyoom on my Arch custom ISO that I use on old laptop with soldered 4GB RAM, it works great.

Arch Linux

#1 2024-03-05 05:04:36

[SOLVED] zram swapping activates too late compared to other distros

#2 2024-03-05 08:16:42

Re: [SOLVED] zram swapping activates too late compared to other distros

#3 2024-03-05 08:59:19

Re: [SOLVED] zram swapping activates too late compared to other distros

#4 2024-03-05 09:06:06

Re: [SOLVED] zram swapping activates too late compared to other distros

#5 2024-03-05 09:37:33

Re: [SOLVED] zram swapping activates too late compared to other distros

#6 2024-03-05 13:57:58

Re: [SOLVED] zram swapping activates too late compared to other distros

#7 2024-03-05 20:54:44

Re: [SOLVED] zram swapping activates too late compared to other distros

#8 2024-03-05 21:24:56

Re: [SOLVED] zram swapping activates too late compared to other distros

#9 2024-03-06 11:58:09

Re: [SOLVED] zram swapping activates too late compared to other distros

#10 2024-03-06 15:17:46

Re: [SOLVED] zram swapping activates too late compared to other distros

#11 2024-03-06 19:09:31

Re: [SOLVED] zram swapping activates too late compared to other distros

Board footer