32 GB RAM + 7000MB/s NVME: Proper swap setup?

kokoko3k · 2023-03-07 05:13:21

Utini wrote:

Ye but the compression ratio is not as good and not as fast as zram

Where did you read about this?

Utini · 2023-03-07 06:02:12

kokoko3k wrote:

Utini wrote:
Ye but the compression ratio is not as good and not as fast as zram
Where did you read about this?

https://linuxreviews.org/Zram#Performance_Implications
https://www.computerbase.de/forum/threa … k.2107871/

And I am sure I read some more zram / zswap / no-swap benchmarks but I didn't save all the sources

Last edited by Utini (2023-03-07 06:03:45)

Brocellous · 2023-03-07 07:03:14

Utini wrote:

https://linuxreviews.org/Zram#Performance_Implications

These benchmarks need larger zpools. Setting the max_pool_percent so low will just force it to swap to disk, and the numbers just seem to show they benefit from the largest size each time, since they'll get much greater compression ratios with zsmalloc over zbud/z3fold ignoring any other tradeoff. Their own numbers above show a 5:1 ratio with zsmalloc+zstd on zram while z3fold artificially caps at ~2.7:1.

BTW Linux 6.2 has writeback for zsmalloc so it may be the best choice for zswap now.

Utini · 2023-03-07 09:18:44

I guess the best option is to run my own test / benchmark once my new laptop arrives and is configured.

Smth like:
Task with 25% RAM usage
Task with 50% RAM usage
Task with 75% RAM usage
Task with 100% RAM usage
Task with 125% RAM usage

And run the same test/benchmark with zram, zswap and no swap.

Any suggestion on how to perform such a task?
E.g. compiling packages with different sizes or so?

Utini · 2023-03-07 18:04:36

This is also crazy. The following video shows a setup with zram being 2.5 times (250%) the size of the actual RAM.
I believe this is working because the compression ratio it 1:3.

So this is essentially creating 40GB of RAM out of 16GB of physical RAM.

Link: https://www.youtube.com/watch?v=nMYTBsjeoTc

agapito · 2023-03-07 20:23:50

kokoko3k wrote:

agapito wrote:
ugjka wrote:
[ugjka@ugjka ~]$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7,8G  3,1G 497,9M 546,6M       4 [SWAP]
1:6 compression ratio, basically free extra ram, just makes sure to mount swap with the discard option
Free extra memory but slower...
kokoko3k wrote:
I always use a little swap partition so that the kernel can use zswap, which i prefer over zram.
What is the reason?
I've been using 4 Zram devices (2 LZ4 & 2 ZSTD) without a Swap partition for the last few years, but a few days ago I was thinking about whether it's better to use a Swap partition with Zswap + ZSTD.
See: https://stackoverflow.com/questions/184 … -and-zswap

Oh, I already knew the differences between zram and zswap, I just wanted to know if you had a personal reason for choosing zswap over zram.

Utini wrote:

This is also crazy. The following video shows a setup with zram being 2.5 times (250%) the size of the actual RAM.
I believe this is working because the compression ratio it 1:3.
So this is essentially creating 40GB of RAM out of 16GB of physical RAM.
Link: https://www.youtube.com/watch?v=nMYTBsjeoTc

And now your RAM is 3 times slower than before and you have multiplied the RAM latency by 30 times. You don't have 40 GB of RAM, try to open a 34 GB Virtual Machine and you understand it. As i told you previously Zram usage has a cost. Zram is a good method to avoid or try to delay an OOM as much as possible and simply that. The more a Zram device fills up, the worse your RAM bandwidth gets, until it reaches a point where the computer is unusable. 75% of the total RAM should be the maximum allowed.

Utini · 2023-03-07 21:06:32

agapito wrote:

Utini wrote:
This is also crazy. The following video shows a setup with zram being 2.5 times (250%) the size of the actual RAM.
I believe this is working because the compression ratio it 1:3.
So this is essentially creating 40GB of RAM out of 16GB of physical RAM.
Link: https://www.youtube.com/watch?v=nMYTBsjeoTc
And now your RAM is 3 times slower than before and you have multiplied the RAM latency by 30 times. You don't have 40 GB of RAM, try to open a 34 GB Virtual Machine and you understand it. As i told you previously Zram usage has a cost. Zram is a good method to avoid or try to delay an OOM as much as possible and simply that. The more a Zram device fills up, the worse your RAM bandwidth gets, until it reaches a point where the computer is unusable. 75% of the total RAM should be the maximum allowed.

Ye well my intention never were to create such a huge zram swap block.
I would stick with 50% of my RAM (so 16 GB out of 32 GB total RAM).
It was just interesting to see the effect in the youtube video.

Actually ZRAM also grows dynamically.
So by setting the zram swap block to 16GB, we do not instantly limit our total memory to minus 16GB.
We allocate a zram swap block which is allowed to have a maximum size of 16GB but at the start it is going to be 0GB.
Only when the system decides that content in the RAM is good to be swapped, the content will be compressed and stored in the zram swap block.
Only in that scenario we are going to trade RAM for zram swap (with a ratio e.g. 1:3 or whatever the compression ratio is).
E.g. free our RAM by 3GB, by compressing and moving the 3GB from the RAM into zram swap block where it allocates 1GB.
In total we saved 2GB of RAM that way.

I think zram is the smartest solution, given all the replies and worries in this topic, since all the scenarios described here will be handled best with zram.

Lets take a 32GB RAM system as an example:

Scenario #1:

We have 32GB RAM which is way more then enough for our usage.
The RAM will never be fully filled and usually only is filled to 25%.
So one choice would be to disable swap entirely.
With ZRAM, swap will basically be disabled until it is actually needed (for whatever reason that might be).
So there is no benefit in not enabling zram. Only the disadvantage of eventually still ever running out of RAM.
According to some benchmarks / tests and even the kernel documentation, swapping occurs way before we run out of RAM.
And that kind of swapping actually increases performance.

Scenario #2

We have 32GB RAM and use quite a good amount of it.
Swapping could still be disabled but we get to the limits and it will hurt performance.
ZRAM will swap out what is not needed and therefor increase the amount of RAM left.
The data in the remaining RAM will be able to work with more performance since it has more RAM left than without ZRAM

Scenario #3

We have 32GB RAM and need more than that (e.g. compiling or whatever).
ZRAM will try to move whatever it can into ZRAM and compress it.
It will do its best to swap out and in 99% of cases the newly creates RAM space (due to compressing and swapping out as much as possible) will save the day

Scenario #4

We have 32GB RAM and need way more than that and also way more than what ZRAM is capable.
Now we would need another way to further swap out. E.g. from ZRAM to disk.
Or we let the OOM killer kick in.
What ever the outcome is, it is not worse than not having zram at all.
The benefit is that we tried to postpone the event of crippling our system for as long as possible thanks to zram.
This scenario would not ever happen for most users.

Last edited by Utini (2023-03-07 21:11:24)

Utini · 2023-03-09 09:20:16

I tried this method on another device (16GB total RAM, 8 GB ZRAM).

I can verify that the ZRAM initially uses no RAM at all and instead dynamically grows in RAM usage.
There for the system still has 16GB of total RAM with ZRAM enabled initially.
As RAM gets swapped into ZRAM, the ZRAM size increases while the RAM size decreases.
This happens in a compression ratio of 1:3 or 1:4, depending on what data is being swapped out.

This could end up with the following stats on a 32GB system with 16GB ZRAM that is completely "filled":
16GB RAM + 64GB ZRAM
Or if during normal usage:
32GB RAM + 0GB ZRAM
Without any ZRAM configuration:
32GB RAM

Easy way to test this and fill up the RAM with 10GB:

dd if=/dev/zero of=/dev/null iflag=fullblock bs=10G

Once the RAM usage is gone, ZRAM will shrink again.
There for the system is never uselessly "pressured" into swapping out.
It only does so when required.

Also it seems like swapping really is not only just being used when we run out of RAM but also for other reasons.
And also then (without running out of RAM) swapping causes a performance improvement.
One reason probably is that the system will have again more free RAM available once it is swapped out.
Therefor it can manage resources differently.

Last edited by Utini (2023-03-09 09:21:47)

Arch Linux

#26 2023-03-07 05:13:21

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#27 2023-03-07 06:02:12

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#28 2023-03-07 07:03:14

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#29 2023-03-07 09:18:44

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#30 2023-03-07 18:04:36

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#31 2023-03-07 20:23:50

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#32 2023-03-07 21:06:32

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

#33 2023-03-09 09:20:16

Re: 32 GB RAM + 7000MB/s NVME: Proper swap setup?

Board footer