Myths about swappiness and performance

PBS · 2021-11-12 07:00:27

Reducing the swappiness setting is widely recommended across the Internet as a catch-all means to improve system performance. Many users set it to low values like 5 and 10, as I did myself for many years, in the hope that it will make their system faster.

This simply isn't true. In my experience on a laptop with 4GB of RAM, setting swappiness to 100 results in noticeably increased system performance, at least under my typical workload.

Let's take a look at why this might be the case. Suppose that a user has a very large and memory intensive program (let's call it Android Studio) running in the background, but they're not actually using it right now. Instead, they have switched to another desktop, and are doing some light web browsing. At swappiness=0, they would find that their browser is extremely sluggish, and moreover, remains sluggish no matter how long they continue using it. They are baffled by this. "After all," they say, "the browser only requires a small amount of data to be in RAM for it to run fast - just the browser executable, its runtime data, and a few databases. Sure it was slow when I first first opened it, because that data was on disk. But now that data is in RAM, and it should stay there because I'm still actively using it? Right?". Well, no. What the user has overlooked is that at swappiness=0, the system prioritises keeping Android Studio's runtime data in memory over the browser's databases and other working files. So their browser databases are put out to disk, loaded into RAM each time the browser reads from them, which is all the time, then immediately evicted from RAM again. This results in disk thrashing, and system performance grinds to a halt.

The moral of the story is that if you have a lot of background processes, it doesn't make sense to set swappiness=0, because that's saying you would rather keep them in memory over what you're currently working on. By contrast, swappiness=100 sets an equal balance between runtime data and disk-backed files, meaning the data necessary to keep your browser working fast will stay in RAM, like it should. In general, the default of 60 should also work well.

For some reason, there is no mention of these facts on the Arch Wiki. Are they correct?

Last edited by PBS (2021-11-12 21:01:10)

seth · 2021-11-12 08:18:28

https://bbs.archlinux.org/viewtopic.php?id=231265

And yes: suggestions to use low vm.swappiness are usually retarded bullshit by people who genuinly believe that otherwise the kernel will simply prefer swap over ram…
https://wiki.archlinux.org/title/Swap#Performance is kinda misleading w/ an unreferenced anecdotal "low value on sufficient memory is known to improve responsiveness on many systems" ("many people say…")
https://rudd-o.com/linux-and-free-softw … o-fix-that presents an artificial test that turns processes inactive and charges the file cache and lo and behold, the file cache gets the lion share of the RAM because it needs it… *headshakingemoji*

ngoonee · 2021-11-12 08:46:28

This is interesting, though I have no direct experience as I've never bothered with swappiness (current machine has 32 GB of RAM and it just has not been a problem for me so far). I do think the wiki page needs updating though.

PBS · 2021-11-12 09:10:50

@seth I already know about those two pages. The last one is linked from the Arch Wiki, despite containing several misconceptions. It does however raise one good point, which is that it would be nice to have a utility for downloading a big file without displacing anything out of cache.

Last edited by PBS (2021-11-12 09:14:17)

seth · 2021-11-12 09:46:04

https://lonesysadmin.net/2013/12/22/bet … rty_ratio/
The rudd-o article actually also touches on that.

It's possible to avoid the file cache, https://unix.stackexchange.com/question … ific-files and dd actually has a flag for that.
https://unix.stackexchange.com/question … stem-cache links an effort to wrap that using LD_PRELOAD

I haven't (yet) read https://lwn.net/Articles/789623/ but it certainkly looks related

sabroad · 2021-11-12 11:00:12

PBS wrote:

Reducing the swappiness setting is widely recommended [...]
on the Arch Wiki. Are they correct?

Perhaps 10 years ago...

A lot has changed since then including (but not-limited to) zswap by default and better cost-estimation by the LRU:
https://lwn.net/Articles/821105/
https://lwn.net/Articles/851184/ (mm)

PBS wrote:

In general, the default of 60 should also work well.

Indeed.

PBS · 2021-11-12 11:21:37

@seth: In a nutshell you're saying the answer is O_DIRECT. This isn't quite what I asked for in #4. O_DIRECT avoids the page cache entirely, comes with restrictions, and breaks on btrfs. I want something that simply says, "drop these writes from cache at the earliest convenience". This doesn't seem to be possible without kernel support.

EDIT: I'm an idiot and didn't read the whole of your answer. It also recommends POSIX_FADV_DONTNEED.

Last edited by PBS (2021-11-12 13:20:41)

kokoko3k · 2021-11-12 12:04:58

PBS:
Check this: https://github.com/Feh/nocache
There's a related package in the AUR.

PBS · 2021-11-12 13:12:49

@kokoko3k Fantastic! nocache is the utility I was looking for, and POSIX_FADV_DONTNEED is the kernel support. There's even a stackexchange answer which checks that it really does what it's supposed to:

https://unix.stackexchange.com/question … l-in-linux

Ironically, the answer then goes on to recommend O_DIRECT as a better approach, which I doubt. Of course, I would have to investigate further before I'm sure.

@sabroad I'm not even sure it was even right 10 years ago, because that's the age of the laptop I tested.

The feedback and insights in this thread have been very useful. I think I'll put the swap page on my list of planned edits.

Last edited by PBS (2021-11-12 13:13:42)

sabroad · 2021-11-12 21:07:23

PBS wrote:

@sabroad I'm not even sure it was even right 10 years ago, because that's the age of the laptop I tested.

10 years ago it was mostly kernel 2.6, and that most certainly exhibited problems with swappiness.

These days we have

sabroad wrote:

better cost-estimation by the LRU

ugjka · 2021-11-12 21:28:02

The default 60 is from the spinning rust days, nowadays with SSDs and NVMEs swappiness of 100 is a safe bet. But then with 16Gigs of ram or more * who cares*

Last edited by ugjka (2021-11-12 21:38:52)

PBS · 2021-11-13 04:07:49

@sabroad:
Thanks, I think I misread your post about what exactly had changed in 10 years.

Perhaps just as importantly, the other major thing that has changed in the last 10 years is how memory-hungry apps have become, as programmers get lazier and lazier. (ELECTRON.) This would mean that low swappiness, appropriate when memory utilisation is low, becomes less applicable.

@ugika:
Hmm, I wasn't aware of that. Could you explain a bit more? It seems to go against the following reasoning. On an SSD you want to minimise writes, so it's preferable to evict file-backed pages instead of swapping, the latter always which requires a write. Hence the balance would shift in favour of lower swappiness, though not all the way down to 5 or 10.

This also raises an interesting question: can you tune the kernel to prefer evicting file-backed pages that DON'T need a write in favour of those that do? This may also be a better idea on SSDs. However, I've not been able to find such a kernel parameter so far.

Last edited by PBS (2021-11-13 07:03:33)

ugjka · 2021-11-13 15:12:04

Knock yourself out

https://chrisdown.name/2018/01/02/in-de … -swap.html

On SSDs, swapping out anonymous pages and reclaiming file pages are essentially equivalent in terms of performance/latency. On older spinning disks, swap reads are slower due to random reads, so a lower vm.swappiness setting makes sense there (read on for more about vm.swappiness).

seth · 2021-11-13 16:02:44

Nachts ist kälter als draußen.

1. Writing NAND is typically slower than reading NAND
2. Writing NAND is destructive, reading NAND is not
1+2: You want to minimize writing on NAND if you can alternatively read it

3. SSDs are still magnitudes slower than RAM
1+2+3: You want to avoid using the disk (HDD/SSD, read/write) as much as possible

4. More aggressive swappiness values in either direction (and general circumstances) set you up for thrashing under OOM (drop file A to read file B, drop file B to read file A, drop file A to … - likewise swapping in and out semi-active anon pages in the other direction)

Whether a HDD reads swap slower than files depends on the fragmentation and what files in which order need to be read (so on average likely yes. But how much?)
Also whether efforts to protect (modern) SSDs from writing are reasonable is questionable, https://techmonitor.ai/techonology/data … eally-last
(It depends on how often and much you need to swap - it's certainly possible to get that to 40GB / day - EVERYDAY! - but maybe consider getting more RAM in that case)

Cookie cutter solutions:
1. Have more RAM
2. Compress the RAM (equivalent to #1) - esp. on CPU overcapacity

ugjka · 2021-11-13 16:12:30

The premise here is that Swap works and is accessed in the same fashion as RAM. NO!

seth · 2021-11-13 16:28:57

What? Where? How? And why? And who?
What?

The premise is that needless disk IO slows everything down and that one can cause needless disk IO w/ aggressive swappiness values.
I'm not sure how you read that as "swap is basically slow RAM" (though, well… that's essentially what thrashing describes, where the RAM is turned into an uncached buffer for the filesystem or the swap)

ugjka · 2021-11-13 17:42:39

Ya'll probably right, but on my old driver i've been in situation where i have 2 to 3 gigs swapped out and things still were smooth, maybe 100ms lag here and there but yeah I'm divided about this swap is bad thing

seth · 2021-11-13 18:28:51

There may be some misunderstandings.

1. I'm NOT saying tha swap is "bad"
2. The only cookie cutter solution that's uaranteed for everyone to work is to have more RAM
3. Beyond that, it depends on access patterns
4. You want to avoid using the disk - wheter for swap or to re-read files - and choose the swappiness accordingly and wrt to your access patterns
5. You *absolutely* want to avoid thrashing
6. The more extreme swap strategies are more prone to cause thrashing
7. The extremes a 0 and 200 - 100 is actually the even balance
8. SSDs lower the costs of invoking the disk, but don't remove them, so tbh, "hdd or ssd" will typically have no impact on the swapping strategy - SSDs just make disk access less annoying (part of that assumtion is that you're not constantly swapping and swapping your SSD to death or that you don't caer and just get a new SSD every ywear, but again: in this case: get ,ore RAM!)

ugjka · 2021-11-13 18:52:57

My old daily driver was capped at 4Gigs of RAM by mobo so I had to rely on swap a lot because no way to upgrade. It had a 60GB ssd, cheap one from Kingston. My swap usage was mostly 1 to 2G daily. Thrashing happened when I pushed the memory beyond what can be swapped but i learned to live with it and later I learned to manage the offending processes with cgroups.

The SSD is still alive today after almost of a decade of swapping and thrashing. Now it is a secondary machine for spotify and IRC.

http://ix.io/3ETS

Last edited by ugjka (2021-11-13 19:05:57)

PBS · 2021-11-14 05:16:30

ugjka wrote:

Knock yourself out
https://chrisdown.name/2018/01/02/in-de … -swap.html

In Defence of Swap is an excellent write-up. I read it a few months ago but didn't understand it properly until now. However, it focuses purely on performance and ignores longevity of the drive. That's fine for a server, but I'm happy to trade a little performance for increased longevity, because performance is already sky-high for my laptop use case. Hence why I said,

pbs wrote:

On an SSD you want to minimise writes

Prompted by the article's plug of cgroups, I wonder if some of the things I've been asking how to do can be accomplished in a more modern way using cgroups?

Last edited by PBS (2021-11-14 08:49:49)

sabroad · 2021-11-14 09:22:22

Nuance has brought us full circle.

PBS wrote:

I'm happy to trade a little performance for increased longevity, because performance is already sky-high

pbs wrote:

On an SSD you want to minimise writes

pbs wrote:

setting swappiness to 100 results in noticeably increased system performance
[...]
In general, the default of 60 should also work well.

Zod · 2021-11-14 13:05:33

Dumb question time...

Wiki wrote:

The swappiness sysctl parameter represents the kernel's preference (or avoidance) of swap space. Swappiness can have a value between 0 and 200 (max 100 if Linux < 5.8), the default value is 60.

Why would I set swappiness over 100?

seth · 2021-11-14 13:45:25

"file_prio=200-vm.swappiness" - I am not sure whether the < 5.8 claim is accurate (maybe some bug) but it would likely just have been clamped in the sysctl then.

https://bbs.archlinux.org/viewtopic.php?id=231265

PBS · 2021-11-14 14:14:56

sabroad wrote:

Nuance has brought us full circle.

To be clear, there are actually two different laptops I'm talking about here:

The one in #1. It has a HDD, and I only care about performance.
The one I've been talking about since. It has an SSD, is already performant enough, and I also care about longevity.

I was hoping nobody would notice...

Last edited by PBS (2021-11-14 14:23:25)

seth · 2021-11-14 14:16:43

So 'twas evolution that brought us around

Arch Linux

#1 2021-11-12 07:00:27

Myths about swappiness and performance

#2 2021-11-12 08:18:28

Re: Myths about swappiness and performance

#3 2021-11-12 08:46:28

Re: Myths about swappiness and performance

#4 2021-11-12 09:10:50

Re: Myths about swappiness and performance

#5 2021-11-12 09:46:04

Re: Myths about swappiness and performance

#6 2021-11-12 11:00:12

Re: Myths about swappiness and performance

#7 2021-11-12 11:21:37

Re: Myths about swappiness and performance

#8 2021-11-12 12:04:58

Re: Myths about swappiness and performance

#9 2021-11-12 13:12:49

Re: Myths about swappiness and performance

#10 2021-11-12 21:07:23

Re: Myths about swappiness and performance

#11 2021-11-12 21:28:02

Re: Myths about swappiness and performance

#12 2021-11-13 04:07:49

Re: Myths about swappiness and performance

#13 2021-11-13 15:12:04

Re: Myths about swappiness and performance

#14 2021-11-13 16:02:44

Re: Myths about swappiness and performance

#15 2021-11-13 16:12:30

Re: Myths about swappiness and performance

#16 2021-11-13 16:28:57

Re: Myths about swappiness and performance

#17 2021-11-13 17:42:39

Re: Myths about swappiness and performance

#18 2021-11-13 18:28:51

Re: Myths about swappiness and performance

#19 2021-11-13 18:52:57

Re: Myths about swappiness and performance

#20 2021-11-14 05:16:30

Re: Myths about swappiness and performance

#21 2021-11-14 09:22:22

Re: Myths about swappiness and performance

#22 2021-11-14 13:05:33

Re: Myths about swappiness and performance

#23 2021-11-14 13:45:25

Re: Myths about swappiness and performance

#24 2021-11-14 14:14:56

Re: Myths about swappiness and performance

#25 2021-11-14 14:16:43

Re: Myths about swappiness and performance

Board footer