32 GB RAM + 7000MB/s NVME: Proper swap setup?

Utini · 2023-03-03 16:14:01

I am about to get a Thinkpad T14s AMD Gen 3 with 32GB RAM (6400) and a 2TB Spidigm P44 Pro NVME.

I usually dont use all of my RAM and since the NVME is also very speedy, I wonder what the proper setup would be?

4GB Swap File
zswap enabled
zram disabled

Or

swap/zswap/zram all together disabled

Or

?

Thanks in advance!

d.ALT · 2023-03-03 17:08:55

I'd go with just zram only

Utini · 2023-03-03 18:11:36

And what is your reason for that?

Brocellous · 2023-03-03 20:37:32

Since it is a laptop you might want to allocate enough swap space to reliably hibernate. The kernel can compress memory when it writes it out for hibernation, but 4GB probably isn't enough if you actually use a sizable portion of that 32GB.

Trilby · 2023-03-04 00:03:56

I'd not use swap at all - but use cases differ. With a modern SSD boot time is effectively as fast as resuming from hibernation (or faster), and given I don't need to preserve any "open" programs / documents, I'd never hibernate (just sleep/suspend or poweroff). So swap for hibernation is irrelevant to me.

Also, I've never get anywhere close to using all available RAM such that space would be needed to swap out pages - in any instance that my RAM comes anywhere close to being full, I'd assume there was a problem, and I'd prefer to let the kernel OOM killer start doing it's thing over allowing the machine to grind itself into a slow and often uninterruptable death of non-stop swapping.

But these are my preferences based on my use and goals. There is no "Proper" swap setup as everyone's usage differs.

Utini · 2023-03-04 07:51:49

Trilby wrote:

I'd not use swap at all - but use cases differ. With a modern SSD boot time is effectively as fast as resuming from hibernation (or faster), and given I don't need to preserve any "open" programs / documents, I'd never hibernate (just sleep/suspend or poweroff). So swap for hibernation is irrelevant to me.
Also, I've never get anywhere close to using all available RAM such that space would be needed to swap out pages - in any instance that my RAM comes anywhere close to being full, I'd assume there was a problem, and I'd prefer to let the kernel OOM killer start doing it's thing over allowing the machine to grind itself into a slow and often uninterruptable death of non-stop swapping.
But these are my preferences based on my use and goals. There is no "Proper" swap setup as everyone's usage differs.

This is also what I was thinking / what my user case would be. No hibernation, never really filling the 32GB RAM.

But there are other opinions about the requirements if swap. The article is from 2018 though

https://chrisdown.name/2018/01/02/in-de … -swap.html

Last edited by Utini (2023-03-04 07:52:19)

jl2 · 2023-03-04 08:35:08

Utini wrote:

Trilby wrote:
I'd not use swap at all - but use cases differ. With a modern SSD boot time is effectively as fast as resuming from hibernation (or faster), and given I don't need to preserve any "open" programs / documents, I'd never hibernate (just sleep/suspend or poweroff). So swap for hibernation is irrelevant to me.
Also, I've never get anywhere close to using all available RAM such that space would be needed to swap out pages - in any instance that my RAM comes anywhere close to being full, I'd assume there was a problem, and I'd prefer to let the kernel OOM killer start doing it's thing over allowing the machine to grind itself into a slow and often uninterruptable death of non-stop swapping.
But these are my preferences based on my use and goals. There is no "Proper" swap setup as everyone's usage differs.
This is also what I was thinking / what my user case would be. No hibernation, never really filling the 32GB RAM.
But there are other opinions about the requirements if swap. The article is from 2018 though
https://chrisdown.name/2018/01/02/in-de … -swap.html

I only use swap for hibernation. Otherwise I don't recommend having one (especially on 32 g ram). If you fill it up, the only thing that happened to me without oom killer was a drop to ~1 fps.

Last edited by jl2 (2023-03-04 08:37:34)

whompus · 2023-03-04 08:56:35

I have 32GB of Ram and have never had swap set up, as per what Trilby mentioned my use case does not use suspend or hibernate. The most Ram I have seen used ever was 50% when I left quite a big program compiling in the background.

Not sure if it makes any difference but I have set vm.swappiness=1 in sysctl.conf.

Last edited by whompus (2023-03-04 09:02:46)

ugjka · 2023-03-04 10:25:37

[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]

1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option

Last edited by ugjka (2023-03-04 11:09:09)

kokoko3k · 2023-03-04 12:05:23

I always use a little swap partition so that the kernel can use zswap, which i prefer over zram.

Trilby · 2023-03-04 13:25:08

The defense of swap article reads a bit odd to me. There is no need to defend swap. If it is useful for you, use it. But if it is not useful for you, no amount of "defense" of it can be relevant. That reads to me like an analysis of the effects of interior house paint pigments on house fires: "you should paint all your walls white because in the event of a housefire, heavily pigmented paints can burn a little faster". Whether that's true or false I'm not entire sure, but I am sure that by that time I really don't care - so I chose paint colors for their asthetic appeal when the house is not on fire.

ratcheer · 2023-03-04 13:31:32

I have 32 GB RAM, and I have seen swap used when I was compiling some huge application, such as chromium. This is not to recommend that you setup a swap facility, just saying that it CAN happen.

I personally do have swap, because what can it hurt when it is (almost) never used?

3beb6e7c46a615a · 2023-03-04 13:53:24

Systemd-oomd recommends to have swap enabled. I use it, so I always enable swap on zram with zram-generator, even with 32G memory.

Last edited by 3beb6e7c46a615a (2023-03-04 13:53:41)

Brocellous · 2023-03-04 18:46:01

Trilby wrote:

Whether that's true or false I'm not entire sure, but I am sure that by that time I really don't care - so I chose paint colors for their asthetic appeal when the house is not on fire.

That is a terrible analogy? The whole point of the article is to combat the misconception that swap is only useful in "emergency" situations or when you need strictly more physical memory than is otherwise available. In this case the house isn't on fire, and the paint is actively preventing that disaster before it happens. The goal is to use the available memory for something useful, and its entirely possible that a larger pagecache has more utility than pinning some rarely used chunk of anonymous memory. It could even allow operation without thrashing where a swapless configuration could not with the same memory usage. It's debatable what application this has to desktop linux specifically, as that isn't the author's focus, but even that depends on your usage.

If you're worried that enabling swap opens the possibility of thrashing, the direction of most distros that handle this kinda stuff for you is toward keeping swap (for exactly the reasons given in the article) and using other tools to handle "fires" (read: aggressive thrashing), like [1] [2], which as configurable tools are almost certainly more prepared to handle disaster than the kernel. You can benefit from swap and still not endure thrashing in the worst case.

Totally orthogonal to all that, at least in my opinion, suspend-then-hibernate is a really useful feature for laptops so it's worth having a sizable non-volatile swap to support it. Zswap is enabled by default in our kernel and is worth keeping. The size of the swap partition doesn't strongly influence the frequency of reclaim and anything useful for hibernate is probably larger than you might ever need for typical not-on-fire usage so that's the only thing I'd bother to consider. A default image size of 2/5 physical memory suggests a ~12G swap file/partition should let you hibernate 99% of the time as long as you're not trying to hibernate in the middle of a blender render or something.

[1] https://docs.kernel.org/admin-guide/mm/ … prevention
[2] https://man.archlinux.org/man/systemd-oomd.service.8.en

topcat01 · 2023-03-04 20:38:40

ugjka wrote:

[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]
1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option

That is a nice compression ratio! I assume you disabled the default zswap configuration?

agapito · 2023-03-04 21:51:16

ugjka wrote:

[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]
1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option

Free extra memory but slower...

kokoko3k wrote:

I always use a little swap partition so that the kernel can use zswap, which i prefer over zram.

What is the reason?

I've been using 4 Zram devices (2 LZ4 & 2 ZSTD) without a Swap partition for the last few years, but a few days ago I was thinking about whether it's better to use a Swap partition with Zswap + ZSTD.

kokoko3k · 2023-03-05 09:15:41

agapito wrote:

ugjka wrote:
[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]
1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option
Free extra memory but slower...
kokoko3k wrote:
I always use a little swap partition so that the kernel can use zswap, which i prefer over zram.
What is the reason?
I've been using 4 Zram devices (2 LZ4 & 2 ZSTD) without a Swap partition for the last few years, but a few days ago I was thinking about whether it's better to use a Swap partition with Zswap + ZSTD.

See: https://stackoverflow.com/questions/184 … -and-zswap

Utini · 2023-03-05 10:31:47

kokoko3k wrote:

agapito wrote:
ugjka wrote:
[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]
1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option
Free extra memory but slower...
kokoko3k wrote:
I always use a little swap partition so that the kernel can use zswap, which i prefer over zram.
What is the reason?
I've been using 4 Zram devices (2 LZ4 & 2 ZSTD) without a Swap partition for the last few years, but a few days ago I was thinking about whether it's better to use a Swap partition with Zswap + ZSTD.
See: https://stackoverflow.com/questions/184 … -and-zswap

I wonder if the performance benefit of zram over zswap is nearly non existend with the new PCI4 NVMEs reaching 7000MB/s read & write?

Especially since that zswap shouldn't even really be used much in the first place on a system with 32GB RAM.

Last edited by Utini (2023-03-05 10:31:58)

Utini · 2023-03-06 14:08:33

It seems like zram is still a million times faster than zswap (even on modern PCI 4 NVME with 7000MB/S read/write).

From all the benchmarks, kernel documentation and real world reports that I have seen, it also seems that in most cases the system will benefit from swap, even when not all of RAM is being used.
Especially with zram as "swap in RAM".

The most up-to-date and best performing configuration seems to be something like this:

    # Make sure no swap partition or file is enabled
    swapoff -a
    rm -f /swapfile
    sudo systemctl --type swap -> mask the .swap unit
    /etc/dstab -> remove relevant swap entries

    # Disable zswap by setting the kernel parameter
    zswap.enabled=0
    
    # Set vm. parameters
    nano /etc/sysctl.d/99-vm-zram-parameters.conf:
    vm.swappiness = 180
    vm.watermark_boost_factor=0
    vm.watermark_scale_factor = 125
    vm.page-cluster=0
    
    # Create zram configuration file in use with zram-generator
    nano /etc/systemd/zram-generator.conf:
    [zram0]
    zram-size = ram / 2 #this means 16GB out of 32GB total available RAM
    compression-algorithm = zstd
    swap-priority = 1000
    fs-type = swap
    
    # Enable zram with zram-genator
    sudo systemctl enable systemd-zram-setup@zram0.service
    
    # Reboot system / check swap status
    
    ------------------
    
    # Alternative / manual way instead of zram-generator, will be reverted after reboot:
    modprobe zram
    zramctl --size 16G --algorithm zstd /dev/zram0
    mkswap --label zram0 /dev/zram0
    swapon --priority 1000 /dev/zram0

If I understand correctly then this:
1. Allows zram to use 50% of total available RAM
2. In case of 32GB total ram, this means 16GB zram. Which at a compression rate of 1:3 is 48GB zram + 16GB RAM.
3. Even if something goes into zram, it is not yet compressed and will just be handles as normal RAM.
4. Only when something is unused it will be compressed within zram
5. Or when we run out of ram and/or zram, the content of zram will also start to become compressed
6. The system still "sees" a proper swap file/device (which should be the compressed zram if I understand correctly)

Basically we have absolutely no drawback in terms of performance as we still have 100% of our RAM at full performance.
Only when we run out of RAM or when unused stuff is in RAM, it will be compressed in ZRAM.
And then we basically get 2-4 times the amount of total RAM available due to the compression ration of zram.
The performance impact of the compression is very low compared to the performance gains we get from increasing our ram basically 2-4 times.

Please correct and explain if something is wrong here :S

Last edited by Utini (2023-03-06 14:14:31)

kokoko3k · 2023-03-06 19:39:57

Where did you read that zram is million times faster than zswap?

Utini · 2023-03-06 20:08:11

kokoko3k wrote:

Where did you read that zram is million times faster than zswap?

DDR4 RAM: 20GB/s
DDR5 RAM: 50GB/s
Latest / fastest consumer PCIe 4 NVME: 7GB/s

And this doesn't even take into account the access times / latency difference.
RAM can access stuff within the order of a few dozen nanoseconds in best case, but still be within the nanoseconds scale at worst case.
Flash memory is still in microseconds scale at best case.

A normal hard disk has a response time of about 16ms, a good SSD will respond in 0.05ms, RAM will respond in 50ns. (notice ms vs. ns).
To put that in perspective, 0.05ms is equal to 50,000ns.
This means that RAM can serve up data in memory 1000 times faster than a NVMe drive even though the file size they can carry is about the same.

Last edited by Utini (2023-03-06 20:10:17)

topcat01 · 2023-03-06 20:10:14

Note though that zswap also compresses to RAM as much as possible (loosely speaking), so it might not use the disk at all.

Utini · 2023-03-06 20:37:59

Ye but the compression ratio is not as good and not as fast as zram.

Additionally, with the high compression ratio of zram, a swapfile on disc most likely never going to be needed (in terms of available space).

I also believe that several other "big distros" have moved to zram as default (fedora,gentoo,debian).

frostschutz · 2023-03-06 21:09:41

Utini wrote:

If I understand correctly then this:
1. Allows zram to use 50% of total available RAM
2. In case of 32GB total ram, this means 16GB zram. Which at a compression rate of 1:3 is 48GB zram + 16GB RAM.
3. Even if something goes into zram, it is not yet compressed and will just be handles as normal RAM.
4. Only when something is unused it will be compressed within zram
5. Or when we run out of ram and/or zram, the content of zram will also start to become compressed
6. The system still "sees" a proper swap file/device (which should be the compressed zram if I understand correctly)

There is no 48GB+16GB.

zram is a block device. You can use if for anything you like, not just swap. Kernel docs suggest also using it for tempdirs and caches.

You set up zram with a certain size (16G). That size is fixed. You mkswap, swapon, then Swap is 16G, that size is fixed. Regardless whether it can be compressed or not, you can put 16G on it, not more, not less. That size does not change. If you want it a different size, you have to set it up with that size in the first place. You can set up another zram device and add as a 2nd swap device, or swapoff, zramctl --reset and start from scratch.

The dynamic part is on the other side: It stores a fixed 16G amount of data, but how much RAM it needs to represent those 16G, that depends on compression ratio and overhead. If you fill it up with incompressible data (like random data), zram will use 16G of RAM to store 16G of data. Plus some overhead that zram generally has (like a few megabytes per gigabyte of data). If the data is very compressible, then that 16G of data will use a lot less than 16G of RAM. Only then does swapping into compressed ram make any sense. If it can't be compressed, you gain nothing by encapsulating it.

Utini · 2023-03-07 04:50:41

frostschutz wrote:

There is no 48GB+16GB.
zram is a block device. You can use if for anything you like, not just swap. Kernel docs suggest also using it for tempdirs and caches.
You set up zram with a certain size (16G). That size is fixed. You mkswap, swapon, then Swap is 16G, that size is fixed. Regardless whether it can be compressed or not, you can put 16G on it, not more, not less. That size does not change. If you want it a different size, you have to set it up with that size in the first place. You can set up another zram device and add as a 2nd swap device, or swapoff, zramctl --reset and start from scratch.
The dynamic part is on the other side: It stores a fixed 16G amount of data, but how much RAM it needs to represent those 16G, that depends on compression ratio and overhead. If you fill it up with incompressible data (like random data), zram will use 16G of RAM to store 16G of data. Plus some overhead that zram generally has (like a few megabytes per gigabyte of data). If the data is very compressible, then that 16G of data will use a lot less than 16G of RAM. Only then does swapping into compressed ram make any sense. If it can't be compressed, you gain nothing by encapsulating it.

Thank you for your explanation but I think there is a yes and no to your abstract.
It is also now the first time that I understand, that zram is not automatically swap!

zram as swap, being in RAM, costs nearly no performance when something is moved in it?
Actually it can greatly improve performance before you run out of RAM and especially when you run out of RAM.
See:
https://linuxreviews.org/Zram
https://github.com/pop-os/default-settings/pull/163
https://www.reddit.com/r/pop_os/comment … or_pop_os/

Theoretically one could even use all of his RAM has zram swap.
Since everything gets moved in there and nearly no performance loss, nothing bad happens.
We can only gain something by getting the contents compressed and allowing to store even more in zram.
There for theoretically increasing the total RAM of 32GB to e.g. 96GB (1:3 compression ration, even 1:4 is realistic from what I read).

The author from the pop_os github pull seriously recommend that all ram = zram swap.
I would rather go with 50% of my RAM or atleast 25%.
Since I don't really see how moving anything from RAM to zram-swap can hurt performance wise, I currently prefer at least 50% of ram as zram-swap.

Only unused files will be moved from RAM into zram-swap.
I also had the impression that compression in zram-swap only starts to happen when the available zram-space gets lower.

Arch Linux

#1 2023-03-03 16:14:01

32 GB RAM + 7000MB/s NVME: Proper swap setup?

#2 2023-03-03 17:08:55

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#3 2023-03-03 18:11:36

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#4 2023-03-03 20:37:32

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#5 2023-03-04 00:03:56

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#6 2023-03-04 07:51:49

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#7 2023-03-04 08:35:08

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#8 2023-03-04 08:56:35

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#9 2023-03-04 10:25:37

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#10 2023-03-04 12:05:23

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#11 2023-03-04 13:25:08

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#12 2023-03-04 13:31:32

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#13 2023-03-04 13:53:24

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#14 2023-03-04 18:46:01

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#15 2023-03-04 20:38:40

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#16 2023-03-04 21:51:16

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#17 2023-03-05 09:15:41

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#18 2023-03-05 10:31:47

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#19 2023-03-06 14:08:33

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#20 2023-03-06 19:39:57

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#21 2023-03-06 20:08:11

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#22 2023-03-06 20:10:14

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#23 2023-03-06 20:37:59

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#24 2023-03-06 21:09:41

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#25 2023-03-07 04:50:41

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

Board footer