[SOLVED] Significant memory leak after upgrade to kernel 6.0

odror · 2022-11-21 18:13:29

Hi

I have kernel 6.0 with zfs and nvidia, xone dkms modules

I am experiencing a sever memory leak at least 20GB/day (my total RAM is 80GB). I have to reboot my computer everyday to reset it.

I have AMD hardware (5950x) and Nvidia driver. I downgraded the nvidia driver from 520 to 5.15.76. It did not resolve the issue. I tried to switch from gnome to plasma and it did not resolve the issue.

Any ideas where it is coming from and how to find the source of the issue.

There is another issue that I am having with the new kernel, which is possibly originating from the same bug. I lost the boot messages during boot. I get a blank screen with a streak of line. And then gnome shows up. I lost all the vt terminals (tty2-5). I get the same distorted blank screen. As if there is an issue with the text mode. I have the same issue with the nvidia driver 520 and 515. I did not have this issue with kernel 5.19.

I would have liked to revert my kernel to 5.19. This is not so simple with the zfs kernel.

Any help will be appreciated.

Last edited by odror (2022-11-22 22:25:36)

ewaller · 2022-11-21 18:41:30

Just so we are all on the same page, how did you determine there exists a memory leak?

odror · 2022-11-21 18:49:03

After booting the system in the morning with gnome only runnning I am using only 6-7GB.

During f the day am using Libvirt with a couple of guest and multiple browser. Then closing the VMs and restarting gnome should show only 6-7GB used, but its shows close to 30GB of memory used. Also if I will not be rebooting the machine in a couple of days, the VMs will crush on boot because of insufficient memory.

ewaller · 2022-11-21 19:00:00

I agree that the problem with the VMs boot is indicative.

There have been many threads in the forums about memory and often assertion of a memory leak is based upon a misunderstanding of what the tools are reporting (I am not asserting that is the case here). Are you using free? or perhaps a tool built into a desktop environment? I am trying to explore whether we are looking at memory that is truly allocated and forgotten, or whether we are dealing with things that are merely caches and buffers that could be released when needed.

... but then I go back to the VMs failing.

odror · 2022-11-21 19:09:22

I am check the memory usage with 'free -m, It is not cached memory, because it cannot be reclaimed by any process. In particular failure of the VM to boot.

for example now I am using only 1 VM (it has allocated 26GB of RAM) I have gnome running and a few browsers.

free -m
shows only 14.5GB free. It should be at least 40GB free

total used free shared buff/cache available
Mem: 80358 63856 14580 550 1920 15200
Swap: 0 0 0

ewaller · 2022-11-21 19:17:39

I'd say I agree.

topcat01 · 2022-11-21 20:36:45

I assume all is fine with the lts kernel?

odror · 2022-11-21 20:39:36

I have zfs on root. Switching kernel is not simple, but there were no issues with kernel 5.19

topcat01 · 2022-11-21 20:50:25

Ok. What was the size of the ZFS ARC at the time the free output was generated?

Last edited by topcat01 (2022-11-21 20:50:40)

xerxes_ · 2022-11-21 21:27:58

You could post output of 'cat /proc/meminfo'. You can also clear cache by command 'sync; echo 1 > /proc/sys/vm/drop_caches' to avoid reboot and see if it free your RAM to some similar amount what you have after you boot system or not (it might show that you have some amount of RAM that can't be freed - real memory leak).

Last edited by xerxes_ (2022-11-21 21:28:43)

odror · 2022-11-21 21:55:05

No VM is running. gnome and a few browsers are running. I should be using less than 10 GB

Free -m:

               total        used        free      shared  buff/cache   available
Mem:           80358       37498       40577         849        2282       41252
Swap:              0           0           0

cat /proc/spl/kstat/zfs/arcstats | grep arc :

arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    21038844080
arc_meta_limit                  4    31598172672
arc_dnode_limit                 4    3159817267
arc_meta_max                    4    22621319584
arc_meta_min                    4    16777216
arc_need_free                   4    0
arc_sys_free                    4    2911842176
arc_raw_size                    4    0

cat/proc/meminfo:

MemFree:        41728120 kB
MemAvailable:   42420136 kB
Buffers:            5080 kB
Cached:          2025996 kB
SwapCached:            0 kB
Active:           702748 kB
Inactive:        8690688 kB
Active(anon):      49696 kB
Inactive(anon):  8011616 kB
Active(file):     653052 kB
Inactive(file):   679072 kB
Unevictable:          48 kB
Mlocked:              48 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               188 kB
Writeback:            92 kB
AnonPages:       7168376 kB
Mapped:          1402380 kB
Shmem:            698952 kB
KReclaimable:     135516 kB
Slab:            5426168 kB
SReclaimable:     135516 kB
SUnreclaim:      5290652 kB
KernelStack:       44912 kB
PageTables:        98868 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    41143452 kB
Committed_AS:   26195368 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     2467476 kB
VmallocChunk:          0 kB
Percpu:            46976 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1968128 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:    133120 kB
FilePmdMapped:    112640 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    30702048 kB
DirectMap2M:    25843712 kB
DirectMap1G:    28311552 kB

topcat01 · 2022-11-21 23:45:04

I'm also interested in:

grep -e '^size' /proc/spl/kstat/zfs/arcstats

odror · 2022-11-21 23:57:30

xerxes_ wrote:

You could post output of 'cat /proc/meminfo'. You can also clear cache by command 'sync; echo 1 > /proc/sys/vm/drop_caches' to avoid reboot and see if it free your RAM to some similar amount what you have after you boot system or not (it might show that you have some amount of RAM that can't be freed - real memory leak).

How will this make a difference if I get the leak without the VM. I noticed it with the VM sooner because they use big chunk of my memory.

Also "/proc/sys/vm/drop_caches" is not available when no VM is running.

odror · 2022-11-22 02:57:48

topcat01 wrote:

I'm also interested in:
grep -e '^size' /proc/spl/kstat/zfs/arcstats

size 4 28400998240

seth · 2022-11-22 09:00:44

/proc/sys/vm/drop_caches has nothing to to w/ you running some virtualbox windows. "vm" means "virtual memory" here.

https://bbs.archlinux.org/viewtopic.php?id=281011

Relogin + Drop caches --> DROPS THE MEMORY.

See whether you can repeat that pattern.

odror · 2022-11-22 16:04:28

xerxes_ wrote:

You could post output of 'cat /proc/meminfo'. You can also clear cache by command 'sync; echo 1 > /proc/sys/vm/drop_caches' to avoid reboot and see if it free your RAM to some similar amount what you have after you boot system or not (it might show that you have some amount of RAM that can't be freed - real memory leak).

I tried it. It did not increase the free memory.

topcat01 · 2022-11-22 20:35:43

odror wrote:

topcat01 wrote:
I'm also interested in:
grep -e '^size' /proc/spl/kstat/zfs/arcstats
size 4 28,400,998,240

That indicates ZFS ARC using about 28 GB RAM which shows up as "used memory". This cannot be freed normally unless ZFS frees it.

odror · 2022-11-22 20:50:24

topcat01 wrote:

odror wrote:
topcat01 wrote:
I'm also interested in:
grep -e '^size' /proc/spl/kstat/zfs/arcstats
size 4 28,400,998,240
That indicates ZFS ARC using about 28 GB RAM which shows up as "used memory". This cannot be freed normally unless ZFS frees it.

So ZFS is holding the missing memory?
How can I free it without rebooting?

topcat01 · 2022-11-22 20:53:18

In order to further confirm this, we can correlate this number with memory usage on subsequent reboots. I think it should free memory under pressure. For example we can test with a simple program which allocates and touches a large amount of memory, but does not fill the RAM to the point of crashing the system (like in C) and then see how the ARC reacts to that.

A simple loop which allocates and writes say 500 MB per iteration and waits for Enter key, for example.

Last edited by topcat01 (2022-11-22 20:55:04)

odror · 2022-11-22 21:18:51

I created such program. It did free the ARC memory and increased the Free memory.

Is it possible that the VM demand for immediate large amount of memory (over 20GB) did not keep up with freeing the ARC memory and the VM crashed.
also is there any advantage to set limit to the ARC memory, may be 10GB (out of 80GB total)

topcat01 · 2022-11-22 21:56:12

Nice, this tracks with my understanding of the situation. I'm not an expert on ZFS, but generally (for a file server) one would not limit the memory (to maximise caching). However, in your specific scenario maybe there is an acceptable middle ground. So, basically limit the ARC memory via some trial and error to find a good number to set the limit to.

Arch Linux

#1 2022-11-21 18:13:29

[SOLVED] Significant memory leak after upgrade to kernel 6.0

#2 2022-11-21 18:41:30

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#3 2022-11-21 18:49:03

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#4 2022-11-21 19:00:00

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#5 2022-11-21 19:09:22

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#6 2022-11-21 19:17:39

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#7 2022-11-21 20:36:45

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#8 2022-11-21 20:39:36

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#9 2022-11-21 20:50:25

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#10 2022-11-21 21:27:58

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#11 2022-11-21 21:55:05

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#12 2022-11-21 23:45:04

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#13 2022-11-21 23:57:30

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#14 2022-11-22 02:57:48

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#15 2022-11-22 09:00:44

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#16 2022-11-22 16:04:28

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#17 2022-11-22 20:35:43

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#18 2022-11-22 20:50:24

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#19 2022-11-22 20:53:18

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#20 2022-11-22 21:18:51

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

#21 2022-11-22 21:56:12

Re: [SOLVED] Significant memory leak after upgrade to kernel 6.0

Board footer