Swap usage slowly creeps up -- how to avoid using swap at all costs

cptpcrd · 2020-11-30 03:50:34

Note: I'm posting this in "Kernel & Hardware" because none of the categories seem quite appropriate, and this was the closest match I found. If you think it belongs somewhere else then there's a good chance you're right.

Personal background: I've been using Arch on a daily basis for just over 2 years now. Usually I'm able to resolve problems like this by poking around on Stack Exchange, the Arch WIki, and other sources, but this time I haven't been quite so lucky.

Since around the time of the kernel 5.0 upgrade (yes, I know, it's been quite a while), I've been noticing the following chain of events occurring frequently: a small amount of memory is moved to swap (8K or so) during a RAM usage spike, then that amount slowly increases over time, and before I know it I'm 200M, or 500M, or 900M into swap. (I have RAM/swap usage indicators in my status bar, so I pay a lot of attention to this.) Recently, I've walked away from my machine a few times and returned to find my swap usage at 1.9G (I have 8G of RAM and an 8G swap partition). It's gotten so bad that I wrote a bash script to run basically 'swapoff -a && swapon -a', because I was sick of retyping it.

Now, I know swap usage isn't necessarily bad, but this swap partition is on an SSD, and I want to avoid unnecessary wear and tear (yes, yes, however small it might be). Really, I only have a swap partition to avoid encounters with the OOM killer in case I completely run out of RAM.

Currently, my best guess regarding the cause is that the combination of used memory + buffers + cache fills the RAM (my RAM usage is normally quite high), and the kernel solves this by moving things into swap. I don't want it to do that. I want to tell the kernel "only use swap space as an absolute last resort, right before you engage the OOM killer; you can throw out all the caches first if that will help." (Not sure if there's a way to do that, just want to make it clear what I'm trying to do here.) I'm willing to take minor performance penalties (because I know the kernel is probably smarter at managing memory for maximum efficiency) to accomplish this.

And I've tried. To start, I set 'sysctl vm/swappiness=0' (yes, in /etc/sysctl.d, so it's preserved across reboots). It made no difference whatsoever. I searched around to try to find any other sysctls I could tune, but the few things I found went beyond my comfort zone (my knowledge of how the kernel handles memory is limited). I tried making a few changes based on things I saw on Stack Exchange, but I didn't notice any difference and I reverted them quickly. ('sysctl vm/vfs_cache_pressure' shows 100, in case that helps -- I saw it mentioned in some of the results.) The problem has persisted; if anything, it seems to have gotten worse following recent kernel releases.

I know this post is long, but I've been getting rather frustrated by this. Any suggestions would be greatly appreciated. :-)

kokoko3k · 2020-11-30 07:07:23

Take a look to zswap.
https://wiki.archlinux.org/index.php/Zswap

Also, check what is in the swap area:
http://northernmost.org/blog/find-out-w … your-swap/

Last edited by kokoko3k (2020-11-30 07:10:32)

seth · 2020-11-30 10:16:02

swappiness=0 afair gets special treatment because it's stupid. Don't do that.
Try "10" for a very low value.

But really look at the swap users.
The decribed pattern sounds very much like a bug - assuming it's not the swap implementation, one (?) of your (long running?) processes piles up anon pages that it never™ requires ie.: it leaks.

cptpcrd · 2020-11-30 14:15:06

kokoko3k wrote:

Take a look to zswap.
https://wiki.archlinux.org/index.php/Zswap

Looking at that page, it seems that zswap should be enabled by default already (?). Is there anything further I can do to tune it?

seth wrote:

swappiness=0 afair gets special treatment because it's stupid. Don't do that.
Try "10" for a very low value.

Good to know. Applied.

kokoko3k wrote:

Also, check what is in the swap area:
http://northernmost.org/blog/find-out-w … your-swap/

seth wrote:

But really look at the swap users.
The decribed pattern sounds very much like a bug - assuming it's not the swap implementation, one (?) of your (long running?) processes piles up anon pages that it never™ requires ie.: it leaks.

It happened again about an hour ago (I went about 2M into swap), and I was able to get some more concrete information.

So I looked at the processes that were using swap space (for some reason I never thought of that before). It was sd-pam, gnome-keyring-daemon, and another background daemon (which I realized I don't need anymore and stopped). Their RAM usage was very low (something like 0.1% to 1% of my RAM, when I had other programs that were using on the order of 4%!), which seems to suggest that maybe they just got moved in because they're usually fairly inactive.

I also managed to grab this screenshot from htop as things were getting moved into swap, which further supports that theory (green=used, blue=buffers, yellow=cached):

This is consistent with what I've seen before -- my status bar shows MemAvailable / MemTotal * 100%, and usually it's at least 20% or 30% when things start getting moved into swap. AFAICT I could definitely use more RAM without going into swap by throwing out caches, and that's what I want the kernel to do.

CarbonChauvinist · 2020-11-30 14:23:07

I really suggest you read a little more background about swap, this is a fantastic blog post on the matter that can be clarifying.

seth · 2020-11-30 14:24:48

You can explicity drop caches, but what the kernel does is to move "old" anonymous pages into swap (because nobody apparently ever uses them) instead of sacrificing more recently used (and likely to be re-used) file caches (because the most likely consequence is that it just has to re-read them)

The graphic is odd - it says it only uses ~½ of your RAM but the colored bar suggests it's actually all?
Please post the output of "free" and even better: "cat /proc/meminfo" instead of relying on fancy system monitors (shell or not - these things just exist to be able to brag on unixporn ;-)

another background daemon

In particular?

One thing I could imagine to mess a lot with this are the file indexers, because they run across your disk, open every file and look into it …
They also run very cyclic.

Ropid · 2020-11-30 15:48:22

cptpcrd wrote:

[...]
I also managed to grab this screenshot from htop as things were getting moved into swap, which further supports that theory (green=used, blue=buffers, yellow=cached): https://i.ibb.co/cXsHSGw/htop-screenshot.png
[...]

Did you already use "vm.swappiness = 10" setting when you took that screenshot?

What your screenshot shows happened to me here only with the default "swappiness = 60". The "60" value made my system start using swap when around half of RAM was in use (plus cache filling up rest of RAM). After changing over to values like "1" or "10", the usage of swap stopped and cache started shrinking.

cptpcrd · 2020-11-30 18:17:20

CarbonChauvinist wrote:

I really suggest you read a little more background about swap, this is a fantastic blog post on the matter that can be clarifying.

So I looked through that article.

I understand that swap can be helpful, but it just doesn't seem normal for a desktop/laptop system (this is a laptop, in case you were wondering, and no I don't care about hibernation) to be going 1G or 2G into swap on a regular basis when it has plenty of available RAM. I reboot daily, so this creep occurs on the order of minutes or hours, not days. I've gotten to the point of trying to effectively disable swap entirely because I'm just frustrated with this continuing to happen (maybe I gave into the XY problem a bit with my original post).

If that really is normal then I can learn to deal with it, but I just get the feeling that something's wrong here.

cptpcrd · 2020-11-30 18:18:12

seth wrote:

You can explicity drop caches, but what the kernel does is to move "old" anonymous pages into swap (because nobody apparently ever uses them) instead of sacrificing more recently used (and likely to be re-used) file caches (because the most likely consequence is that it just has to re-read them)
The graphic is odd - it says it only uses ~½ of your RAM but the colored bar suggests it's actually all?
Please post the output of "free" and even better: "cat /proc/meminfo" instead of relying on fancy system monitors (shell or not - these things just exist to be able to brag on unixporn ;-)

So I ran 'swapoff -a && swapon -a' (I did not echo 1 >/proc/sys/vm/drop_caches), pushed RAM usage just a little (but still around 30% MemAvailable) to reproduce it, and then got this a few seconds after it started moving things in:

$ free
              total        used        free      shared  buff/cache   available
Mem:        8026600     3850128      575356     1320760     3601116     2454476
Swap:       8388604        3328     8385276

$ free -h  # You didn't ask, but I always find it easier to read at a glance
              total        used        free      shared  buff/cache   available
Mem:          7.7Gi       3.7Gi       561Mi       1.3Gi       3.4Gi       2.3Gi
Swap:         8.0Gi       3.0Mi       8.0Gi

$ cat /proc/meminfo
MemTotal:        8026600 kB
MemFree:          575120 kB
MemAvailable:    2454240 kB
Buffers:          459180 kB
Cached:          2996884 kB
SwapCached:           52 kB
Active:          2017672 kB
Inactive:        4696912 kB
Active(anon):     212468 kB
Inactive(anon):  4318432 kB
Active(file):    1805204 kB
Inactive(file):   378480 kB
Unevictable:      229888 kB
Mlocked:            2872 kB
SwapTotal:       8388604 kB
SwapFree:        8385276 kB
Dirty:              1748 kB
Writeback:             0 kB
AnonPages:       3488148 kB
Mapped:          1599724 kB
Shmem:           1320760 kB
KReclaimable:     145052 kB
Slab:             284616 kB
SReclaimable:     145052 kB
SUnreclaim:       139564 kB
KernelStack:       25088 kB
PageTables:        60176 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12401904 kB
Committed_AS:   18438700 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       49488 kB
VmallocChunk:          0 kB
Percpu:             6624 kB
HardwareCorrupted:     0 kB
AnonHugePages:     88064 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      559016 kB
DirectMap2M:     7708672 kB
DirectMap1G:           0 kB

I'll admit that I'm not exactly familiar with most of the /proc/meminfo metrics.

seth wrote:

another background daemon
In particular?
One thing I could imagine to mess a lot with this are the file indexers, because they run across your disk, open every file and look into it …
They also run very cyclic.

It was a VPN daemon, and I wasn't connected. I restarted it and strace'd it to see what it does when inactive; it looked like it enters a 1-second poll loop waiting for commands (yeah, not terribly well designed...). Nothing that should be remotely I/O intensive.

cptpcrd · 2020-11-30 18:18:44

Ropid wrote:

cptpcrd wrote:
[...]
I also managed to grab this screenshot from htop as things were getting moved into swap, which further supports that theory (green=used, blue=buffers, yellow=cached): https://i.ibb.co/cXsHSGw/htop-screenshot.png
[...]
Did you already use "vm.swappiness = 10" setting when you took that screenshot?
What your screenshot shows happened to me here only with the default "swappiness = 60". The "60" value made my system start using swap when around half of RAM was in use (plus cache filling up rest of RAM). After changing over to values like "1" or "10", the usage of swap stopped and cache started shrinking.

Yes, this was with vm.swappiness=10. Same with my above info dumps as requested by @seth.

Making that change has seemed to help slightly, but to really know for sure I'd need to run it for another few weeks to see how frequently it happens and how much gets pushed in, for comparison with previous occasions.

seth · 2020-11-30 20:13:50

There're only 560MB free in that meminfo, so we're certainly in the "do some" territory.
~92% of the inactive pages are anonymous, only 8% is made up by the file cache - so this fits your swappines value.
Likewise 95.3% of the anonymous pages (stuff that need swapping) are inactive, 17.3% of the cache is.

What would concern me more, but isn't reflected in the stats, is the situation where "Swap usage slowly creeps up" (we're also only at 3MB swap and the kernel will not re-transfer the swap just because there's some free RAM. This just creates more disk I/O with the option for even more disk I/O if the same pages have to be swapped out soon)

cptpcrd · 2020-12-01 23:44:18

seth wrote:

There're only 560MB free in that meminfo, so we're certainly in the "do some" territory.
~92% of the inactive pages are anonymous, only 8% is made up by the file cache - so this fits your swappines value.
Likewise 95.3% of the anonymous pages (stuff that need swapping) are inactive, 17.3% of the cache is.

So are you saying that below about 500-600MB free (edit: and I mean free, not available), the kernel will just step in and start doing something? Because if so, maybe I just need to look for services to disable -- I probably push that threshold on a regular basis.

seth wrote:

What would concern me more, but isn't reflected in the stats, is the situation where "Swap usage slowly creeps up" (we're also only at 3MB swap and the kernel will not re-transfer the swap just because there's some free RAM. This just creates more disk I/O with the option for even more disk I/O if the same pages have to be swapped out soon)

The "creep" mostly happens during somewhat elevated RAM usage (though I don't think to the point that I actually run out of RAM). For example, like this (swap usage doubled shortly after I took this):

$ free
              total        used        free      shared  buff/cache   available
Mem:        8026588     4547172      221276     1679392     3258140     1525796
Swap:       8388604      134912     8253692

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.7Gi       4.3Gi       218Mi       1.6Gi       3.1Gi       1.5Gi
Swap:         8.0Gi       131Mi       7.9Gi

$ cat /proc/meminfo
MemTotal:        8026588 kB
MemFree:          223136 kB
MemAvailable:    1527872 kB
Buffers:          259812 kB
Cached:          2890444 kB
SwapCached:         2988 kB
Active:          3270696 kB
Inactive:        3795416 kB
Active(anon):    2196056 kB
Inactive(anon):  3392084 kB
Active(file):    1074640 kB
Inactive(file):   403332 kB
Unevictable:      207508 kB
Mlocked:            2880 kB
SwapTotal:       8388604 kB
SwapFree:        8253692 kB
Dirty:              1564 kB
Writeback:             0 kB
AnonPages:       4121712 kB
Mapped:          1237036 kB
Shmem:           1677568 kB
KReclaimable:     104464 kB
Slab:             255916 kB
SReclaimable:     104464 kB
SUnreclaim:       151452 kB
KernelStack:       25280 kB
PageTables:        66228 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12401896 kB
Committed_AS:   20732380 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       50004 kB
VmallocChunk:          0 kB
Percpu:             6112 kB
HardwareCorrupted:     0 kB
AnonHugePages:     22528 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1114024 kB
DirectMap2M:     7153664 kB
DirectMap1G:           0 kB

Last edited by cptpcrd (2020-12-01 23:45:00)

seth · 2020-12-02 08:16:28

Still similar situation (and you're not using much more SWAP at this time) - the file cache only makes up ~10% of the inactive memory.
Still nothing to see, you've just more active anonymous pages (~39%) ie. the system is doing something ;-)

You want some™ spare free memory because when you open a new tab in chromium and that tries to allocate 500MB, that's not the moment when you want to start shuffle stuff around.
Active memory is to be avoided because it's currently™ in use, so swapping or dropping it will likely just lead to more I/O (thrashing - can happen when you run really low on RAM, sucks), so you're looking at the inactive memory for "do some" territory and "do what" is defined by the swappiness.

If you constantly keep hitting your RAM boundaries, get more RAM or do less, resp. look out for leaking patterns (processes that increase their memory amount while they probably shouldn't)

Arch Linux

#1 2020-11-30 03:50:34

Swap usage slowly creeps up -- how to avoid using swap at all costs

#2 2020-11-30 07:07:23

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#3 2020-11-30 10:16:02

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#4 2020-11-30 14:15:06

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#5 2020-11-30 14:23:07

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#6 2020-11-30 14:24:48

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#7 2020-11-30 15:48:22

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#8 2020-11-30 18:17:20

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#9 2020-11-30 18:18:12

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#10 2020-11-30 18:18:44

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#11 2020-11-30 20:13:50

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#12 2020-12-01 23:44:18

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

#13 2020-12-02 08:16:28

Re: Swap usage slowly creeps up -- how to avoid using swap at all costs

Board footer