RAM Being consumed by apparently nothing

KirottuM · 2022-02-27 17:44:35

Over the last few weeks I have noticed a pattern on both my desktop and my laptop:
After a long time of being on the RAM usage goes way over what it should be, even reporting over 10 gigabytes used with a singular web browser open. Closing all active applications and even my graphical session does not eliminate the RAM usage, and using utilities like "smem" which adds up RAM usage of my programs the RAM used by them is fine. But the system RAM usage is way over what it should be, and consuming a lot of SWAP.

A few possible scenarios that I believe are false:
- The RAM is being used for buffers and cache:
If it was buffers or cache, it wouldn't or at least shouldn't use SWAP and would not be counted towards the "used" amount.
- There is a memory leak in one of my applications:
It would show up in system monitors and such, and would be solved by closing applications.

And there are probably more.

After compiling a kernel with kmemleak support and running with it, it reported no memory leaks so I am at a complete loss here. Only possible theory I have is a memory leak or something like that in the kernel as it would not be picked up by my system monitoring utilities.

Any help would be appreciated.

seth · 2022-02-27 18:03:32

"RAM usage" as reported by what? Fancy but useless desktop widget? "free"?

I believe are false

Don't believe, check:

cat /proc/meminfo

Also check the dmesg, some RAM might be used by GART/GTT for an IGP.

KirottuM · 2022-02-27 18:56:53

It is also reported by the utility "free", desktop widgets have some other weirdness but the values are more or less in line with what free reports.

For the dmesg part GART seems to be a thing, a message

 [timestamp] [drm] PCIE GART of 1024M enabled.

would appear multiple times at least on my laptop. Both systems also use Ryzen CPUs, but only the laptop has an integrated GPU. The desktop uses a dedicated Radeon GPU.

seth · 2022-02-27 21:15:42

And what does "free" exactly report?
Or, *much* bettter

seth wrote:

cat /proc/meminfo

KirottuM · 2022-02-28 05:22:15

Here is /proc/meminfo:

MemTotal:       11661120 kB
MemFree:         4492528 kB
MemAvailable:    4827168 kB
Buffers:            7168 kB
Cached:           492884 kB
SwapCached:        53476 kB
Active:          1933584 kB
Inactive:        1633784 kB
Active(anon):    1515468 kB
Inactive(anon):  1572420 kB
Active(file):     418116 kB
Inactive(file):    61364 kB
Unevictable:        1608 kB
Mlocked:            1608 kB
SwapTotal:      10485756 kB
SwapFree:        9583944 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       3007104 kB
Mapped:           375728 kB
Shmem:             20572 kB
KReclaimable:     173560 kB
Slab:             428120 kB
SReclaimable:     173560 kB
SUnreclaim:       254560 kB
KernelStack:       19264 kB
PageTables:        46832 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16316316 kB
Committed_AS:   10692612 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       52660 kB
VmallocChunk:          0 kB
Percpu:            12544 kB
HardwareCorrupted:     0 kB
AnonHugePages:    530432 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:     75776 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     6448844 kB
DirectMap2M:     5468160 kB
DirectMap1G:     1048576 kB

And here is free -h:

               total        used        free      shared  buff/cache   available
Mem:            11Gi       6.2Gi       4.3Gi        19Mi       658Mi       4.6Gi
Swap:            9Gi       879Mi       9.1Gi

Also smem -tk as good measure:

  PID User     Command                         Swap      USS      PSS      RSS
 1235 kirottu  /bin/sh /usr/bin/startx /us   640.0K     8.0K    12.0K   504.0K
 1301 kirottu  xinit /usr/bin/startplasma-   240.0K     8.0K    12.0K   548.0K
 1552 kirottu  /usr/lib/bluetooth/obexd      904.0K    12.0K    22.0K     1.0M
 1473 kirottu  /usr/bin/touchegg               2.4M    32.0K    71.0K     1.7M
 1315 kirottu  /usr/bin/startplasma-x11        4.7M    20.0K    73.0K     2.4M
13964 kirottu  /usr/lib/xdg-permission-sto   564.0K    52.0K    74.0K     1.3M
 1374 kirottu  /usr/lib/dconf-service        612.0K   152.0K   188.0K     1.5M
 6817 kirottu  /usr/lib/at-spi-bus-launche   696.0K   400.0K   430.0K     1.8M
 1471 kirottu  /usr/bin/gmenudbusmenuproxy     4.5M   364.0K   432.0K     3.0M
13952 kirottu  /usr/lib/xdg-document-porta     2.3M   424.0K   447.0K     1.7M
 3349 kirottu  /usr/lib/kf5/kioslave5 /usr     1.9M   284.0K   449.0K     2.8M
 1456 kirottu  /usr/bin/xembedsniproxy         4.5M   388.0K   457.0K     3.0M
 1551 kirottu  /usr/lib/kf5/kscreen_backen     2.5M   424.0K   487.0K     2.9M
 1344 kirottu  /usr/bin/plasma_session         4.1M   428.0K   489.0K     2.9M
14046 kirottu  /usr/lib/xdg-desktop-portal     5.4M   432.0K   505.0K     3.4M
 1458 kirottu  /usr/bin/kaccess                5.5M   544.0K   638.0K     3.6M
 1234 kirottu  /usr/bin/kwalletd5 --pam-lo     5.3M   644.0K   725.0K     3.7M
 1322 kirottu  /usr/bin/dbus-daemon --sess   668.0K   980.0K     1.1M     2.3M
13930 kirottu  /usr/lib/xdg-desktop-portal     3.7M     1.1M     1.3M     3.1M
 1470 kirottu  /usr/lib/DiscoverNotifier      24.1M     1.3M     1.6M     6.0M
 1378 kirottu  /usr/bin/kglobalaccel5          5.9M     1.4M     1.6M     5.6M
 1699 kirottu  /usr/lib/baloorunner            4.7M     1.3M     1.6M     6.1M
 1222 kirottu  /usr/lib/systemd/systemd --     1.9M     1.2M     2.0M     4.4M
76384 kirottu  /usr/bin/fish                 796.0K     1.7M     2.6M     7.3M
20508 kirottu  /usr/bin/fish                   1.3M     1.9M     2.7M     5.6M
 1364 kirottu  /usr/lib/kactivitymanagerd      5.5M     2.1M     2.7M     8.3M
 1428 kirottu  /usr/bin/ksmserver              4.7M     2.1M     2.8M     9.0M
 1631 kirottu  /usr/lib/kdeconnectd            7.0M     2.4M     2.8M     7.5M
 1606 kirottu  /usr/bin/wireplumber            3.8M     3.3M     3.5M     5.8M
77753 kirottu  /usr/bin/fish                      0     2.7M     3.7M     8.9M
77923 kirottu  ssh Kirottu@lish-frankfurt.        0     3.3M     4.0M     7.8M
 7432 kirottu  /usr/lib/librewolf/librewol    12.6M     2.2M     4.5M    21.7M
 1535 kirottu  /usr/bin/ksystemstats           2.7M     4.5M     4.9M     8.8M
 1605 kirottu  /usr/bin/pipewire               4.3M     7.0M     7.4M     8.8M
 1360 kirottu  /usr/bin/kded5                 14.5M     9.2M    10.8M    19.6M
20490 kirottu  /usr/bin/alacritty             49.7M    10.9M    12.1M    17.9M
 6839 kirottu  /usr/lib/librewolf/librewol    10.9M    14.6M    18.6M    69.6M
78494 kirottu  /usr/lib/librewolf/librewol     1.6M    11.9M    19.1M    81.8M
78457 kirottu  /usr/lib/librewolf/librewol     1.6M    12.0M    19.2M    82.0M
78514 kirottu  /usr/lib/librewolf/librewol     1.6M    12.1M    19.4M    82.5M
 1607 kirottu  /usr/bin/pipewire-pulse        16.7M    19.5M    19.9M    21.6M
 1302 kirottu  /usr/lib/Xorg :0 -nolisten     52.1M    24.9M    26.6M    31.5M
77247 kirottu  /usr/lib/librewolf/librewol     1.6M    24.4M    29.2M    94.7M
78606 kirottu  python /usr/bin/smem -tk           0    29.2M    29.3M    31.8M
76367 kirottu  /usr/bin/alacritty              7.9M    35.0M    42.5M   106.5M
 1474 kirottu  /usr/bin/nextcloud --backgr    59.1M    41.5M    44.2M    59.1M
77735 kirottu  /usr/bin/alacritty                 0    43.2M    51.4M   117.3M
75885 kirottu  /usr/lib/librewolf/librewol    22.6M    44.0M    53.9M   174.9M
76126 kirottu  /usr/bin/corectrl               9.7M    52.0M    66.2M   149.4M
77671 kirottu  /usr/lib/librewolf/librewol     1.7M    58.8M    75.3M   219.9M
76440 kirottu  /usr/lib/librewolf/librewol     3.2M    76.6M    86.6M   206.5M
77118 kirottu  /usr/lib/librewolf/librewol     1.7M    81.1M    91.8M   216.3M
77567 kirottu  /usr/lib/librewolf/librewol     1.7M    87.0M   103.4M   245.6M
76670 kirottu  /usr/lib/librewolf/librewol     1.7M    92.8M   103.8M   231.3M
74449 kirottu  /usr/bin/krunner               12.9M    88.6M   104.0M   193.3M
 7085 kirottu  /usr/lib/librewolf/librewol    21.6M   100.5M   106.6M   170.9M
 1363 kirottu  /usr/bin/kwin_x11              45.4M   108.0M   113.8M   144.7M
 1460 kirottu  /usr/bin/plasmashell           67.6M   119.9M   129.8M   181.6M
76567 kirottu  /usr/lib/librewolf/librewol     1.7M   148.7M   159.9M   286.4M
 6702 kirottu  /usr/lib/librewolf/librewol   115.8M   343.4M   361.5M   521.5M
 7125 kirottu  /usr/lib/librewolf/librewol    80.9M   376.9M   384.2M   451.6M
76501 kirottu  /usr/lib/librewolf/librewol     2.0M   822.2M   834.2M   963.0M
-------------------------------------------------------------------------------
   62 1                                      737.8M     2.9G     3.1G     5.2G

seth · 2022-02-28 07:22:52

smem accounts for 5.2GB, the brunt of that going to librewolf
free shows 4.3GB free what matches the meminfo data
What can get you worried is that MemFree+Active+Inactive only accounts for 7.7GB, so we're looking for ~3GB that are not claimed by process memory.
This sounds eerily like https://bbs.archlinux.org/viewtopic.php?id=271247 so see whether you can reclaim that RAM from the GPU
https://bbs.archlinux.org/viewtopic.php … 9#p2004169 (best exit the graphical session, I assume librewolf is shuffling a lot of RAM into GTT)

KirottuM · 2022-02-28 13:45:29

Alright, by running that stress-ng command repeatedly I was in fact able to regain the memory, but it is just a workaround and really not ideal. For my laptop I kinda do understand why that memory would be allocated but for my desktop that runs with no iGPU it is very weird.

seth · 2022-02-28 13:55:28

It's not a workaround either, but evidencing that there's no problem. The RAM will be reclaimed from the GTT when required.
As to whether and why this is a thing on either machine, please post a complete dmesg fro both of them.

KirottuM · 2022-02-28 14:56:40

The problem especially on my desktop is the fact that after a while the amount of RAM consumed starts pushing things to my swap, and especially when playing memory intensive games it starts to really hurt performance. And when that happens everything starts getting really sluggish as you may expect. I do not currently have access to my desktop but will have in a few days, so will reply with similar dmesg logs when I do.

seth · 2022-02-28 16:38:56

The data you provided is from the notebook exclusively then?
Desktop and notebook might be different problems, so please get smem and meminfo from there as well.
If the GTT (librewolf?) using process is still active, the memory can possibly not be reclaimed.

You can

modinfo amdgpu | grep -iE '(gart|gtt)'
modinfo radeon | grep -iE '(gart|gtt)'

(depending on the used module) but this might not result an improvement of the situation and you want to check the default values as a baseline in the dmesg first.

KirottuM · 2022-02-28 16:42:41

Symptoms are same on both systems, unaccounted memory that is but I have not collected more in depth info on the desktop. I also do remember this happening with other browsers as well, it would be a shame if I needed to close my browser every time I want to do something else on my system.

seth · 2022-02-28 16:46:30

Similar symptoms don't require similar causes - pressure in your chest can be an imminent heart attack… or bad food.
You'd first also want to try and see whether closing the browser will allow you to reclaim the RAM and whether it's really impossible to reclaim it on your current session.
Because "I spend RAM on GTT" and "Game runs OOM" are likewise not necessarily linked.

KirottuM · 2022-02-28 17:37:44

On my laptop at least closing the browser does not reclaim the memory. Only when occupying way more ram does it get freed. And yes it may as well be a whole another thing on my desktop.

seth · 2022-02-28 17:40:57

*Just* exiting the process won't do anything. The RAM isn't claimed by the process but the kernel module - so this is absolutely expected.
What you want to figure is whether you can reclaim the memory w/ head/tail/stress-ng despite the browser process is still running (as whether or not its the relevant process can't be told, though there might be some AMD specific tool like nvidia-smi to reveal such)

KirottuM · 2022-03-01 19:36:25

Hmm yes I have verified that librewolf does in fact eat up GTT, which is fine I guess but the GTT does not get reclaimed after I close the browser. And due to my system having SWAP that claimed RAM will just stay claimed, and performance is going to be degraded due to everything running off the SWAP.

seth · 2022-03-01 20:32:39

but the GTT does not get reclaimed after I close the browser

Once again: that is perfectly normal.

And due to my system having SWAP that claimed RAM will just stay claimed

That would be a bug.
Do you really have to allocate memory excessing your combined free RAM and swap in order to reclaim memory from the GTT?
Or did you just assume this to be a condition?

Do not make guesses based on the behavior of some game - there're far too many variables in this.

* Have the browser allocate some GTT
* use head/tail/stress-ng to allocate a bit more RAM than you've currently free/available (but not so much that you'll expectably have to resort to swap
=> do you get more free RAM afterwards (ie. can you reclaim some from the GTT)? Or do you instead start swapping?
* close the browser
* use head/tail/stress-ng to allocate a bit more RAM than you've currently free/available (but not so much that you'll expectably have to resort to swap
=> do you get more free RAM afterwards (ie. can you now reclaim some from the GTT)? Or do you instead start swapping?
* make sure there's no lingering librewolf process
* use head/tail/stress-ng to allocate a bit more RAM than you've currently free/available (but not so much that you'll expectably have to resort to swap
=> do you get more free RAM afterwards (ie. can you now reclaim some from the GTT)? Or do you instead start swapping?

KirottuM · 2022-03-02 16:00:07

Alright I have gotten to my desktop, and this is peculiar. It seems to allocate 8 gigabytes of my RAM for GTT right at boot too looking at the timestamps

[    0.184750] Linux agpgart interface v0.103
[    6.541731] amdgpu 0000:08:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[    6.541792] [drm] amdgpu: 8192M of GTT memory ready.
[    6.541797] [drm] GART: num cpu pages 65536, num gpu pages 65536
[    6.544332] [drm] PCIE GART of 256M enabled (table at 0x000000F4007E9000).
[    6.982059] kfd kfd: amdgpu: Allocated 3969056 bytes on gart

which does seem to go in line with the RAM usage statistics, ~5 GB used by actual programs and ~13 GB used in total (5 GB SWAP and 8 GB RAM). Now this is even more bizarre than on my laptop, as it does not get progressively allocated when needed but rather allocated right at boot and never actually being used properly (radeontop reports ~600 MB used).

seth · 2022-03-02 18:27:53

You can probalby configure that in the BIOS (next to the kernel module parameter) but the more interesting question is whether you can reclaim it on demand.
Boot the multi-user.target (2nd link below) and check the memory situation there. If you're indeed missing ~8GB, try to allocate them (not so much that you would have to tap into SWAP but maybe so that you would have to tap into SWAP if the 8GB are not claimable)

iirc the GTT claims a percentage of the available RAM, you probably have at least 32GB on that system?

KirottuM · 2022-03-03 16:31:51

This is making even less sense now... that 8GB seems to be occupied only at some points and others not? But it is always displayed in radeontop and never even nearly used. And on my desktop none of those "GART enabled" messages show up so I have absolutely no clue what is occupying that memory but it still is occupied. No matter the kernel either it seems, although I have only used recent kernels. Also the RAM usage right after boot is always normal, this problem only appears after a while of usage.

seth · 2022-03-03 20:26:24

KirottuM wrote:

Also the RAM usage right after boot is always normal, this problem only appears after a while of usage.

Because something requires amdgpu to claim RAM for GTT - the only remaining and so far unanswered question errr … remains:

Can you reclaim it and if not, what process is in the way.

(And to re-iterate: test. Don't infer assumptions from some game behavior. There're too many variables since eg. the game itself might allocate over GTT)

KirottuM · 2022-03-05 10:14:33

Alright did those tests,

With the browser open there was no change in ram usage after running the command.
With the browser closed there was also no change (Also verified the lack of lingering processess)
After exiting my graphical session completely the values seemed to be more in line with what they should be, but still not how much should be consumed. Running the command also did not change anything in that scenario.

seth · 2022-03-05 15:41:07

That would suggest that it's the compositor rather than anything else that allocates the GTT.

After exiting my graphical session completely

Does that mean isolating the multi-user.target or "logout to my bloated DM"?

I'd try the behavior w/ xinit/startx and openbox…

KirottuM · 2022-03-05 19:51:19

I have no display manager set up, I login directly to a tty which runs startx automatically if it is tty1 so that should not be the case.

seth · 2022-03-05 20:34:44

What kind of session?
And how much RAM do you lose for when you check it before starting the GUI session and try to reclaim it afterwards?
(Compare at least the "free" outputs)

KirottuM · 2022-03-06 17:19:35

I use KDE plasma as the DE, if that is what you mean by session. Also right after boot and logging in to a different tty free reports roughly 300 megabytes of RAM used, and after running free again later on with the graphical session closed down roughly 1.2 gigabytes of RAM used was reported. And yes that was after trying to reclaim the RAM.

Last edited by KirottuM (2022-03-06 17:21:08)

Arch Linux

#1 2022-02-27 17:44:35

RAM Being consumed by apparently nothing

#2 2022-02-27 18:03:32

Re: RAM Being consumed by apparently nothing

#3 2022-02-27 18:56:53

Re: RAM Being consumed by apparently nothing

#4 2022-02-27 21:15:42

Re: RAM Being consumed by apparently nothing

#5 2022-02-28 05:22:15

Re: RAM Being consumed by apparently nothing

#6 2022-02-28 07:22:52

Re: RAM Being consumed by apparently nothing

#7 2022-02-28 13:45:29

Re: RAM Being consumed by apparently nothing

#8 2022-02-28 13:55:28

Re: RAM Being consumed by apparently nothing

#9 2022-02-28 14:56:40

Re: RAM Being consumed by apparently nothing

#10 2022-02-28 16:38:56

Re: RAM Being consumed by apparently nothing

#11 2022-02-28 16:42:41

Re: RAM Being consumed by apparently nothing

#12 2022-02-28 16:46:30

Re: RAM Being consumed by apparently nothing

#13 2022-02-28 17:37:44

Re: RAM Being consumed by apparently nothing

#14 2022-02-28 17:40:57

Re: RAM Being consumed by apparently nothing

#15 2022-03-01 19:36:25

Re: RAM Being consumed by apparently nothing

#16 2022-03-01 20:32:39

Re: RAM Being consumed by apparently nothing

#17 2022-03-02 16:00:07

Re: RAM Being consumed by apparently nothing

#18 2022-03-02 18:27:53

Re: RAM Being consumed by apparently nothing

#19 2022-03-03 16:31:51

Re: RAM Being consumed by apparently nothing

#20 2022-03-03 20:26:24

Re: RAM Being consumed by apparently nothing

#21 2022-03-05 10:14:33

Re: RAM Being consumed by apparently nothing

#22 2022-03-05 15:41:07

Re: RAM Being consumed by apparently nothing

#23 2022-03-05 19:51:19

Re: RAM Being consumed by apparently nothing

#24 2022-03-05 20:34:44

Re: RAM Being consumed by apparently nothing

#25 2022-03-06 17:19:35

Re: RAM Being consumed by apparently nothing

Board footer