[Solved, Not Kernel issue] Half of my RAM disappeared into the ether

A_Arch_Linux_User999 · 2025-08-23 17:25:09

Hello, I am on Linux 6.16.2-arch1-1, and right now, most of my memory is occupied for no reason. Specifically, out of 64 GB, only 13 GiB (49.2 GiB used) are available (as according to `free -h`). Looking at htop (as root), less than 25% of my total system memory is being actively used by running processes (therefore, we should be seeing around 16 GiB of usage, but we see 3x that).

To give further context: my host runs a KDE desktop environment where I run virtual machines. I don't run much software on my host and have killed most virtual machines (with the remaining machines not having enough memory allocated to ever come close to this total). This means that there isn't much that can be hogging all these resources. I don't know how to view kernel memory usage but I suspect that's where most of the usage is.

How do I check kernel memory usage and how can I check which module is using all this memory (perhaps a bug with a driver?)

Last edited by A_Arch_Linux_User999 (2025-08-27 13:00:50)

A_Arch_Linux_User999 · 2025-08-23 17:26:37

No, this isn't just RAM being used as host cache (especially considering that my entire host OS could fit into less than 16 GiB because, as I said, my host is pretty minimal and more-so works as a hypervisor with a desktop than anything).

A_Arch_Linux_User999 · 2025-08-23 17:35:42

Resolved with `echo 3 > /proc/sys/vm/drop_caches`
Which doesn't make sense, I've encountered performance issues with running out of memory. Shouldn't this be done automatically?

loqs · 2025-08-23 17:46:56

Why would the kernel drop caches when nothing requires the memory?

A_Arch_Linux_User999 · 2025-08-23 18:01:34

loqs wrote:

Why would the kernel drop caches when nothing requires the memory?

Because my machine keeps using up all my memory and when most virtual machines, by default, include a memory balloon driver to allocate memory as needed, I find myself randomly getting absolute terrible performance which I suspect is from the kernel constantly having to free caches every time a VM needs memory.

loqs · 2025-08-23 18:30:52

Is the kernel not working as expected in dropping caches when there is memory pressure from the VM? What do you want the kernel to do differently? Please also change your thread title to reflect that you have not found a kernel leak.

A_Arch_Linux_User999 · 2025-08-26 12:34:07

loqs wrote:

Is the kernel not working as expected in dropping caches when there is memory pressure from the VM? What do you want the kernel to do differently? Please also change your thread title to reflect that you have not found a kernel leak.

I have disabled all memory ballooning and yet still the kernel is eating all my memory for no reason.
It fails to drop caches in time and despite that my total memory usage does not exceed 34 GB (out of 64 GB of memory) which is statically allocated to a virtual machine, my memory usage reaches maximum and slows down my entire PC to an unusable point where I must kill a virtual machine ASAP lest my entire machine crash.

There is NO reason for the kernel to be allocating this much memory when my host is doing literally nothing.
I'm at a point where I'm going to have to put `echo 3 > /proc/sys/vm/drop_caches` into a cron job running every 15 minutes if I want my computer to be usable after I bought nearly double the RAM my workload actually needed.

What I want the kernel to do is to not cache nonsense and then only drop the absolute minimum based on requests.

Last edited by A_Arch_Linux_User999 (2025-08-26 12:35:02)

topcat01 · 2025-08-26 18:54:40

Not freeing cached memory in time is not a thing. I operate multiple 64 GB machines (one of them being my laptop also running corporate/M$ junk like Teams meetings at the same time) for weeks with < 10 GB "Available" and have never seen a slowdown. I believe the OP's observations, but I think something else is wrong. I have big things in tmp, run huge builds (20 c++ processes each taking up more than 1 GB), and have (qemu) vms.

Edit: I don't have ZFS, but its ARC is known to be a little slow to free memory, since it operates separately from the Linux VM system. However, since ZFS wasn't mentioned I'm assuming that isn't the issue.

Last edited by topcat01 (2025-08-26 18:56:02)

xerxes_ · 2025-08-26 19:23:28

You didn't write what software do you use to run virtual machines and how? Do you use Wayland or X server?

When you login to KDE but don't start any VMs or programs, do you still have high memory usage and free memory shrinking?

What if you try to run other DE to see if you would have memory problems or not?

seth · 2025-08-26 19:24:59

Instead of narrating stuff, please illustrate the situation w/

cat /proc/meminfo

Also, did you fool around w/ vm.swappiness or vm.dirty* ?
https://lonesysadmin.net/2013/12/22/bet … rty_ratio/

Edit: now we're piling on

Last edited by seth (2025-08-26 19:25:31)

A_Arch_Linux_User999 · 2025-08-26 21:03:42

topcat01 wrote:

Not freeing cached memory in time is not a thing. I operate multiple 64 GB machines (one of them being my laptop also running corporate/M$ junk like Teams meetings at the same time) for weeks with < 10 GB "Available" and have never seen a slowdown. I believe the OP's observations, but I think something else is wrong. I have big things in tmp, run huge builds (20 c++ processes each taking up more than 1 GB), and have (qemu) vms.

Well clearly it *is* because it's happening to me. Also I use ext4 on all drives (with the exception of vfat on my efi part)

xerxes_ wrote:

You didn't write what software do you use to run virtual machines and how? Do you use Wayland or X server?
When you login to KDE but don't start any VMs or programs, do you still have high memory usage and free memory shrinking?
What if you try to run other DE to see if you would have memory problems or not?

KDE Wayland. No X11 software AFAIK.
I use QEMU/KVM via virt-manager. I also use looking glass for GPU passthrough virtual machines.
I haven't tried other DEs nor have I waited while doing nothing for cache to max out my memory.

I've just now tried setting the dirty/background dirty ratio to 10 and 5 respectively. But I shouldn't need to do this. The kernel should NEVER be using this much memory.
I'd really prefer to tell the Linux kernel to never use more than 4 GiB of memory.

loqs · 2025-08-26 21:21:56

seth wrote:

Instead of narrating stuff, please illustrate the situation w/
cat /proc/meminfo

Please provide the requested output. Is zswap in use? Is a swapfile in use if so what storage is it on? How long does echo 3 > /proc/sys/vm/drop_caches` take to return with and without sync being run just before when the issue occurs? Please also read the drop_caches entry of https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Edit:
Does the system have any custom sysctl entries or other entries designed to improve performance?

Last edited by loqs (2025-08-26 21:25:25)

peelz · 2025-08-27 03:51:14

It's a mesa bug that affects Looking Glass: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13780

You can temporarily work around the issue by closing the LG client and running `echo 3 > /proc/sys/vm/drop_caches` to free some memory before hitting OOM.
(or downgrade to mesa 25.1.7)

A_Arch_Linux_User999 · 2025-08-27 12:59:06

loqs wrote:

seth wrote:
Instead of narrating stuff, please illustrate the situation w/
cat /proc/meminfo
Please provide the requested output. Is zswap in use? Is a swapfile in use if so what storage is it on? How long does echo 3 > /proc/sys/vm/drop_caches` take to return with and without sync being run just before when the issue occurs? Please also read the drop_caches entry of https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Edit:
Does the system have any custom sysctl entries or other entries designed to improve performance?

No idea what zswap is, so probably not.
My swapfile in on root partition on an NVME SSD
The drop_caches takes less than a second (very quick)
No sysctl changes

But it looks like the issue is linked by the other user. Most likely a looking-glass memory leak.

At this point in time all my RAM was nearly gone and virtual machines should have been using no more than 26 GiB
```
MemTotal: 65060288 kB
MemFree: 2662412 kB
MemAvailable: 2591396 kB
Buffers: 34696 kB
Cached: 668120 kB
SwapCached: 212576 kB
Active: 24553364 kB
Inactive: 3526696 kB
Active(anon): 24361548 kB
Inactive(anon): 3323836 kB
Active(file): 191816 kB
Inactive(file): 202860 kB
Unevictable: 12612 kB
Mlocked: 1704 kB
SwapTotal: 33554428 kB
SwapFree: 31535316 kB
Zswap: 367368 kB
Zswapped: 1634260 kB
Dirty: 2032 kB
Writeback: 0 kB
AnonPages: 27387368 kB
Mapped: 528128 kB
Shmem: 308572 kB
KReclaimable: 257572 kB
Slab: 775528 kB
SReclaimable: 257572 kB
SUnreclaim: 517956 kB
KernelStack: 25600 kB
PageTables: 95512 kB
SecPageTables: 21656 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 66084572 kB
Committed_AS: 40669912 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 297696 kB
VmallocChunk: 0 kB
Percpu: 33536 kB
HardwareCorrupted: 0 kB
AnonHugePages: 26798080 kB
ShmemHugePages: 8192 kB
ShmemPmdMapped: 0 kB
FileHugePages: 40960 kB
FilePmdMapped: 40960 kB
CmaTotal: 0 kB
CmaFree: 0 kB
Unaccepted: 0 kB
Balloon: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 33589876 kB
DirectMap2M: 28542976 kB
DirectMap1G: 4194304 kB
```

Last edited by A_Arch_Linux_User999 (2025-08-27 13:03:28)

A_Arch_Linux_User999 · 2025-08-27 13:01:59

I've edited the title to reflect that this is a non-kernel issue. Sure do wish that tools to view memory made it more clear where memory leaks come from.

seth · 2025-08-27 13:16:38

https://wiki.archlinux.org/title/Zswap

Zswap:            367368 kB
Zswapped:        1634260 kB

Please use [code][/code] tags, the BBS predates markdown by approximately your age. Edit your post in this regard.

The meminfo shows ~26.4GB in anonymous pages, 2.5GB free and only 385M(!)B in file caches (half active, half inactive) and ~760MB in slabs - there're >> 30GB completely unaccounted for.
1. this is *after* dropping the file caches?
2. mesa bug aside: is this an AMD GPU? => https://bbs.archlinux.org/viewtopic.php … 9#p2243899

Arch Linux

#1 2025-08-23 17:25:09

[Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#2 2025-08-23 17:26:37

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#3 2025-08-23 17:35:42

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#4 2025-08-23 17:46:56

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#5 2025-08-23 18:01:34

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#6 2025-08-23 18:30:52

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#7 2025-08-26 12:34:07

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#8 2025-08-26 18:54:40

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#9 2025-08-26 19:23:28

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#10 2025-08-26 19:24:59

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#11 2025-08-26 21:03:42

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#12 2025-08-26 21:21:56

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#13 2025-08-27 03:51:14

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#14 2025-08-27 12:59:06

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#15 2025-08-27 13:01:59

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

#16 2025-08-27 13:16:38

Re: [Solved, Not Kernel issue] Half of my RAM disappeared into the ether

Board footer