Memory is rapidly swapped out with plenty free

anthonybilinski · 2021-03-22 03:15:11

I have 16GB of physical RAM, with a 16GB swap volume. When using a very memory hungry program, I initially see my RAM usage climb above 15GB, file system cache drop to a few hundred MB, and swap climb to about 2GB. After using the program for a couple minutes, I see RAM usage drop steadily down to 8GB usage, file system cache still a couple hundred MB, and over 7GB of RAM completely free, with swap grown to 12GB. At that point the program becomes very unresponsive, hitting swap non-stop, but free memory stays around 7-8GB.

I'm watching memory usage constantly from the time I open the program, and when it hits the point where it seemingly starts trying to empty half of RAM, memory usage decreases by about 50MB/s.

I tried changing swappiness to 0, and the issue was unchanged.

Is this likely to be a kernel bug, or is there some way this memory hungry program could trigger this behaviour normally? I don't generally trust it to behave reasonably.

/proc/meminfo while exhibiting this behaviour:

MemTotal:       16354048 kB
MemFree:         7155168 kB
MemAvailable:    7064996 kB
Buffers:              12 kB
Cached:           115884 kB
SwapCached:       158564 kB
Active:          1916016 kB
Inactive:        1781228 kB
Active(anon):    1838776 kB
Inactive(anon):  1764904 kB
Active(file):      77240 kB
Inactive(file):    16324 kB
Unevictable:          96 kB
Mlocked:              96 kB
SwapTotal:      16777212 kB
SwapFree:        4130208 kB
Dirty:                 0 kB
Writeback:            96 kB
AnonPages:       3448652 kB
Mapped:           114280 kB
Shmem:             22184 kB
KReclaimable:      60136 kB
Slab:             606172 kB
SReclaimable:      60136 kB
SUnreclaim:       546036 kB
KernelStack:       21600 kB
PageTables:        80528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    24954236 kB
Committed_AS:   26157952 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      261100 kB
VmallocChunk:          0 kB
Percpu:             2208 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    15181636 kB
DirectMap2M:     1558528 kB

Last edited by anthonybilinski (2021-03-22 03:34:58)

seth · 2021-03-22 06:54:12

I tried changing swappiness to 0

Please don't do that.

This sounds like what you'd get if verymemoryhungryprogram™ runs in a memory limited cgroup

cat /proc/$(pidof verymemoryhungryprogram)/cgroup

You're looking for the memory subgroup, eg. user.slice/user-1000.slice
Then check the restrictions for that group

cat /sys/fs/cgroup/memory/user.slice/user-1000.slice/memory.limit_in_bytes

It's however also possible that the system has to cycle some super huge pages and there's only "free" memory because the only time you get to look at it is inbetween - you're using 12GB swap…

xerxes_ · 2021-03-22 17:40:28

You can try zswap which should decrease I/O to your hard drive (SSD or HDD?) and increase responsiveness of your system.

Last edited by xerxes_ (2021-03-22 17:44:12)

ugjka · 2021-03-22 18:41:07

If there is not some funky configuration going then the kernel probably has a good reason to do what it does

anthonybilinski · 2021-03-22 18:49:53

Thanks for the replies. Swapiness changed back to default (60).

I checked the memory cgroup, and have:

4:memory:/user.slice/user-1000.slice/user@1000.service

which is limited to

/sys/fs/cgroup/memory/user.slice/user-1000.slice/memory.limit_in_bytes
9223372036854771712

8 exabytes, so it doesn't seem to be that.

For a little more context, this is just a steam game, not running under proton on a pretty regular desktop install.

Regarding cycling huge pages, from /proc/meminfo earlier, it looks like I have 0 huge pages, and that each page would only be 2MiB even if I had some? The only other option I can think of is that the program is allocating 8GB, writing to it to get it comited, then freeing it, before every tick of my memory monitor.. but that seems a bit ridiculous even for this.

Regarding zswap, I'm more concerned about only half my RAM being used, rather than the performance of the swap that is being used. If I'm still having performance issues with full RAM usage I will look into that more.

anthonybilinski · 2021-03-22 19:06:21

ugjka wrote:

If there is not some funky configuration going then the kernel probably has a good reason to do what it does

My config is pretty standard and this is a new install. I'm using the regular arch kernel, and haven't touched much system config at all. The one weird thing about my system is that it's using ZFS, which handles filesystem cache itself, but it is heavily limited too to an ARC (in-memory read cache) of 300MB (and would show as either kernel memory or filesystem cache anyway).

I'd generally think the kernel has a good reason too, but I'm getting really bad performance due to constant disk IO, all while half my RAM is completely free, not being used by buffers or caches, so I'm a bit unconvinced in this case.

seth · 2021-03-22 22:26:04

If you run steamgame™ in a "clean" environment (nothing else, also no desktop environment and drop all caches etc.), does the used swap still exceed the free RAM?
In that case we'd know that the system is swapping because of steamgame™ and probably causing some thrashing there.
Can you lower the RAM demands of steamgame™ and get the total demand (RAM+swap < 16GB)? Does it still cause crazy swapping?

What might also happen is that steamgame™ reads a huge file into RAM, therefore needs to swap, then requires some swapped memory and drops the file.
However, the inactive anon pages occupy almost have your RAM, so that's not really likely either…

ugjka · 2021-03-23 02:49:34

What kind of graphics card do you have?

sabroad · 2021-03-23 10:45:13

https://serverfault.com/a/154000 wrote:

The fact that the pages are copied back into RAM, not moved, is important and can lead to confusion if you are not aware of it. The page will not be deallocated from swap unless it is no longer needed at all (i.e. the page is deallocated completely), is changed in RAM (so the copy in swap is no longer correct), or swap is running low (and the on-disk blocks are needed to swap some other pages out). This way if the page needs to be swapped out again in future no disk write is needed as the kernel knows there is already a good copy on disk - this can greatly reduce "thrashing" when available RAM becomes critically low but swap space is not also congested.

If pages swapped are not written to (eg. texture assets), they'll remain allocated in swap. This is to avoid the worse condition where the working set (larger than memory) is constantly paging in and out (aka thrashing).

To see swap activity, try

vmstat 1

ugjka · 2021-03-23 11:31:25

I'm wondering about 8gigs being free in constant fashion, is this integrated graphics with gpu memory shared on ram

anthonybilinski · 2021-03-24 09:48:01

If you run steamgame™ in a "clean" environment (nothing else, also no desktop environment and drop all caches etc.), does the used swap still exceed the free RAM?
In that case we'd know that the system is swapping because of steamgame™ and probably causing some thrashing there.
Can you lower the RAM demands of steamgame™ and get the total demand (RAM+swap < 16GB)? Does it still cause crazy swapping?

I haven't tested in a clean environment yet, but I've done a test that I think shows a similar thing. I was able to lower the game's memory requirements, and disabled swap completely. Running the game, it had 8.3G resident, 13.3G virtual and was running smoothly, filesystem cache was about 800MB,

I turned swap back on, and the game's memory was gradually swapped out. After about 10 minutes, it was at 600MB resident, 13.3G virtual. Resident was climbing down to 500MB, then the game would freeze for a few seconds and resident would go up to about 700MB, and that would repeat endlessly. Swap usage was 9.5G aft that point, and total system memory usage was 5.2G with a 500MB filesystem cache.

I wrote a program that allocated 8.5G in 10MB chunks, and writes to all allocated memory to avoid over commit, then never touched the memory again. After over 10 minutes (using the computer in the background), the process had 8505M resident and 8508 virt.

So it seems like whatever's happening is specific to the game process, where its in-use memory is being swapped aggressively out but a completely idle program's memory isn't being.

What kind of graphics card do you have?
I'm wondering about 8gigs being free in constant fashion, is this integrated graphics with gpu memory shared on ram

r9 390, which has 8GB VRAM. When this is happening, about 6GB of VRAM is in use. I tried lowering texture qualities to minimum and didn't see a change in behaviour.

To see swap activity, try

In the "steady state" case where resident memory is no longer decreasing and the game is running very poorly:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 8575488 8281512     12 123352 1384    0  5798     0 7067 23029 30 12 57  2  0
 3  0 8562432 8201256     12 121924 22608 10740 56444 10740 15181 59507 25 21 44 11  0
 0  0 8562432 8201924     12 123028   96    0   172     0 6557 20002 30 12 58  0  0
 1  0 8562432 8233264     12 122968  304    0  1280  1468 6594 21134 29 12 54  4  0
 2  0 8562176 8255548     12 122956  100    0   251 12113 7002 25094 28 17 55  1  0
 5  0 8561920 8255284     12 122956  112    0   220     0 6467 19626 30 13 58  0  0
 3  0 8561664 8254092     12 123344  104    0   751     0 6393 19348 29 12 59  0  0
 0  1 8552960 8246340     12 124160 6144    0 14066     0 6819 22390 13  9 63 15  0
 0  2 8550144 8228240     12 124276 1176    0 15687  1908 5372 11632  5  6 68 21  0
 0  1 8546048 8122404     12 125120 2568    0 52106  1003 5598 13723  4  5 70 21  0
 1  0 8547768 8102456     12 123744 4164 10832 37269 10832 8111 18691 13  8 61 19  0
 1  2 8545792 8149684     12 123028 9224 16680 30737 16680 10541 35282 16 13 59 12  0
 2  1 8570368 8157364     12 116192 5924 36352 59867 36352 29992 37822 34 22 29 15  0
 2  1 8567296 8120768     12 117040 1768    0 52265     0 7279 23733 36 13 34 18  0
 0  2 8563968 8270596     12 117068 2288    0 16485 65058 11139 49778 26 26 24 24  0
 1  0 8575744 8230364     12 112192  676 12760 23107 12784 9635 23660 34 13 36 17  0
 1  0 8593408 8134340     12 109892  616 18592 37608 18592 10036 24076 34 15 46  5  0
 4  0 8593152 8219584     12 111576  176    0   516     0 6409 19819 35 13 53  0  0
 2  0 8593152 8219020     12 111732   88    0   147     0 6167 18772 35 12 53  0  0
 3  0 8593152 8280088     12 111796   88    0   329 32144 8547 37590 35 19 40  6  0
 4  0 8591872 8274092     12 112204  852    0  4918     0 6771 21804 36 13 49  2  0
 0  1 8604416 8126240     12 110320  592 13688 42832 13688 48666 23724 36 16 43  5  0
 0  1 8621056 8157188     12 110244  548 17240 54493 17240 7413 25386 32 14 45  9  0
 0  0 8616960 8236124     12 111280 3224    0 13133     0 7336 25064 32 10 55  3  0
 3  0 8616192 8290856     12 111276  844    0  2396 34454 8527 35761 30 17 46  7  0
 2  0 8616192 8284112     12 111300   76    0  2693     0 6252 18983 30 10 57  3  0
 0  0 8615936 8286164     12 111324  112    0  1203     0 6248 18889 31  9 59  1  0

Is it possible for the game process to be asking or forcing itself to be swapped out by mistake? I can't see how this is rational behaviour if it's totally up to Linux, but it also seems to be different for this one process than others.

seth · 2021-03-24 16:16:21

My gut feeling is that the process maps & reads a file, the file cache forces the kernel to swap out some anon pages, then the anon pages are required, read from swap, the file dropped from the cache to allow from that, then the file is re-read.
The problem w/ that theory are the inactive pages in your meminfo, though I guess one could clumsily cycle through the memory so that the most inactive pages are the ones required right after reading the file.
Have you googled for known memory management issues with steamgame™ ? Does it come with/read a humongous file (megatexture)?

anthonybilinski · 2021-03-24 21:36:15

Regarding the large file read hypothesis, my vmstat output doesn't show much "bi" under "io", which should be 1K block reads according to the manpage. During that session, it was only reading in at most 54M in a second with 8G free and it was still swapping out more. I checked the install directory and there are no files larger than about 500MB (though the total install is still about 5GB). Searching online I see lots of people complaining about high memory usage, but not specifically this over-swapping, but almost all users seem to be on Windows.

I don't remember seeing this before switching to Arch recently. I'll see if I can repro on another distro with an older kernel, and maybe try to find a disk to re-install and test without ZFS so that I don't have a tainted kernel.

anthonybilinski · 2021-03-25 01:49:52

Tested on Ubuntu 20.10 with kernel 5.8.0-44, still on ZFS, and I didn't have the same issue, which is 100% repro on Arch. Running the game for about 30 mins, the process had 7.7GB res, 13.0G virt. The system had 2GB fs cache, 150MB swap usage, which all seems sane, and the game was running smoothly.

I feel like this points more towards some kernel regression, but it still seems weird that it effects this game and not every other process for everyone else. Since the game appears to actually matter, for the record it's "ARK: Survival Evolved".

I'll try bisecting old kernels on Arch to see if that changes anything.

Last edited by anthonybilinski (2021-03-25 07:48:12)

Arch Linux

#1 2021-03-22 03:15:11

Memory is rapidly swapped out with plenty free

#2 2021-03-22 06:54:12

Re: Memory is rapidly swapped out with plenty free

#3 2021-03-22 17:40:28

Re: Memory is rapidly swapped out with plenty free

#4 2021-03-22 18:41:07

Re: Memory is rapidly swapped out with plenty free

#5 2021-03-22 18:49:53

Re: Memory is rapidly swapped out with plenty free

#6 2021-03-22 19:06:21

Re: Memory is rapidly swapped out with plenty free

#7 2021-03-22 22:26:04

Re: Memory is rapidly swapped out with plenty free

#8 2021-03-23 02:49:34

Re: Memory is rapidly swapped out with plenty free

#9 2021-03-23 10:45:13

Re: Memory is rapidly swapped out with plenty free

#10 2021-03-23 11:31:25

Re: Memory is rapidly swapped out with plenty free

#11 2021-03-24 09:48:01

Re: Memory is rapidly swapped out with plenty free

#12 2021-03-24 16:16:21

Re: Memory is rapidly swapped out with plenty free

#13 2021-03-24 21:36:15

Re: Memory is rapidly swapped out with plenty free

#14 2021-03-25 01:49:52

Re: Memory is rapidly swapped out with plenty free

Board footer