You are not logged in.

#1 2021-02-22 15:18:03

Wild Penguin
Member
Registered: 2015-03-19
Posts: 197

Kernel memory leak since recently (5.10.16-zen)

Hi,

Since recently I've repeatedly had memory leak which I suspect is in the Kernel space, since I can not see any processes using the RAM. Earlyoom starts to kill my actual software, but still 20GiB+ of RAM (on a system of 32GiB total RAM) keeps being occupied by something (Kernel?) and is not available for processes (<10GiB is not enough to run my software). I'm using amdgpu, in case that makes a difference.

After stopping X.org (sddm), so that the system has no services up which should be using a lot of RAM, around 20GB+ is still in use (I'm expecting ~1GiB or less). I can not see any proccesses using a lot of RAM in (h)top! (only the available amount has diminished as reported by 'top', 'free' etc...).

Has anyone else noticed a similar problem recently?

Any other ideas save compiling kernel with CONFIG_DEBUG_KMEMLEAK do find out, what is actually eating the RAM?

Or, hot to conclusively confirm if this is a kernel, kernel module or - after all - some process eating RAM? (I've used top/htop sorted by RAM usage).

Some info:

Some inxi info:

System:    Host: ArkkiVille Kernel: 5.10.16-zen1-1-zen x86_64 bits: 64 compiler: N/A Desktop: KDE Plasma 5.21.0 
           Distro: Arch Linux 
Machine:   Type: Desktop System: ASUS product: All Series v: N/A serial: <filter> 
           Mobo: ASUSTeK model: MAXIMUS VII GENE v: Rev 1.xx serial: <filter> UEFI: American Megatrends v: 3503 
           date: 04/18/2018 
Battery:   ID-1: hidpp_battery_0 charge: N/A condition: N/A model: Logitech G703 Wired/Wireless Gaming Mouse 
           status: Discharging 
CPU:       Topology: Quad Core model: Intel Core i7-4790K bits: 64 type: MT MCP arch: Haswell rev: 3 L2 cache: 8192 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 63994 
           Speed: 4000 MHz min/max: 800/4800 MHz Core speeds (MHz): 1: 4000 2: 4000 3: 4000 4: 4000 5: 4000 6: 4001 7: 4000 
           8: 4000 
Graphics:  Device-1: Advanced Micro Devices [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] driver: amdgpu v: kernel 
           bus ID: 03:00.0 
           Display: x11 server: X.Org 1.20.10 driver: amdgpu resolution: 3440x1440~100Hz 
           OpenGL: renderer: Radeon RX Vega (VEGA10 DRM 3.40.0 5.10.16-zen1-1-zen LLVM 11.1.0) v: 4.6 Mesa 20.3.4 
           direct render: Yes 
Audio:     Device-1: Intel 9 Series Family HD Audio vendor: ASUSTeK driver: snd_hda_intel v: kernel bus ID: 00:1b.0 
           Device-2: Advanced Micro Devices [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] driver: snd_hda_intel v: kernel 
           bus ID: 03:00.1 
           Device-3: Digital Devices Octopus DVB Adapter driver: ddbridge v: 0.9.33-integrated bus ID: 04:00.0 
           Device-4: Micronas BLUE USB Audio 2.0 type: USB driver: snd-usb-audio bus ID: 5-2.4:5 
           Device-5: Creative Sound BlasterX G6 type: USB driver: hid-generic,snd-usb-audio,usbhid bus ID: 5-2.3:4 
           Sound Server: ALSA v: k5.10.16-zen1-1-zen 
Network:   Device-1: Intel Ethernet I218-V vendor: ASUSTeK driver: e1000e v: kernel port: f040 bus ID: 00:19.0 
           IF: eno1 state: up speed: 1000 Mbps duplex: full mac: <filter> 

/proc/meminfo during a leak:

MemTotal:       32820080 kB
MemFree:         6456052 kB
MemAvailable:    9874904 kB
Buffers:          197772 kB
Cached:          3749128 kB
SwapCached:            0 kB
Active:          1087708 kB
Inactive:        5288832 kB
Active(anon):      25724 kB
Inactive(anon):  2742628 kB
Active(file):    1061984 kB
Inactive(file):  2546204 kB
Unevictable:        1760 kB
Mlocked:            1760 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       2388068 kB
Mapped:           887796 kB
Shmem:            337236 kB
KReclaimable:     274584 kB
Slab:            1978800 kB
SReclaimable:     274584 kB
SUnreclaim:      1704216 kB
KernelStack:       20016 kB
PageTables:        40252 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16410040 kB
Committed_AS:   10225220 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       78300 kB
VmallocChunk:          0 kB
Percpu:             6528 kB
HardwareCorrupted:     0 kB
AnonHugePages:    440320 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:     14336 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    10412600 kB
DirectMap2M:    23080960 kB
DirectMap1G:     1048576 kB

/proc/slabinfo during a leak:

slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
fat_inode_cache      357    441    752   21    4 : tunables    0    0    0 : slabdata     21     21      0
fat_cache            714    714     40  102    1 : tunables    0    0    0 : slabdata      7      7      0
kvm_async_pf           0      0    136   30    1 : tunables    0    0    0 : slabdata      0      0      0
kvm_vcpu               0      0  11328    1    4 : tunables    0    0    0 : slabdata      0      0      0
kvm_mmu_page_header      0      0    168   24    1 : tunables    0    0    0 : slabdata      0      0      0
x86_emulator           0      0   2672   12    8 : tunables    0    0    0 : slabdata      0      0      0
x86_fpu                0      0   4160    7    8 : tunables    0    0    0 : slabdata      0      0      0
nfs4_layout_stateid      0      0    296   27    2 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_delegations      0      0    248   33    2 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_files            0      0    288   28    2 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_lockowners       0      0    392   20    2 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_openowners       0      0    432   37    4 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_clients          0      0   1296   25    8 : tunables    0    0    0 : slabdata      0      0      0
rpc_inode_cache       46     46    704   23    4 : tunables    0    0    0 : slabdata      2      2      0
ovl_inode              0      0    696   23    4 : tunables    0    0    0 : slabdata      0      0      0
fuse_request           0      0    152   26    1 : tunables    0    0    0 : slabdata      0      0      0
fuse_inode             0      0    896   36    8 : tunables    0    0    0 : slabdata      0      0      0
ext4_groupinfo_4k 141596 141596    144   28    1 : tunables    0    0    0 : slabdata   5057   5057      0
ext4_fc_dentry_update      0      0     80   51    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache   93288  93288   1192   27    8 : tunables    0    0    0 : slabdata   3456   3456      0
ext4_allocation_context    256    256    128   32    1 : tunables    0    0    0 : slabdata      8      8      0
ext4_io_end         5056   5056     64   64    1 : tunables    0    0    0 : slabdata     79     79      0
ext4_extent_status  74766  74766     40  102    1 : tunables    0    0    0 : slabdata    733    733      0
jbd2_journal_handle   5110   5110     56   73    1 : tunables    0    0    0 : slabdata     70     70      0
jbd2_journal_head   1939   2074    120   34    1 : tunables    0    0    0 : slabdata     61     61      0
jbd2_revoke_table_s   1024   1024     16  256    1 : tunables    0    0    0 : slabdata      4      4      0
jbd2_revoke_record_s   1024   1024     32  128    1 : tunables    0    0    0 : slabdata      8      8      0
bio-1                294    378    384   21    2 : tunables    0    0    0 : slabdata     18     18      0
dm_bufio_buffer-72    576    576    224   36    2 : tunables    0    0    0 : slabdata     16     16      0
dm_bio_prison_cell   1806   1806     96   42    1 : tunables    0    0    0 : slabdata     43     43      0
kcopyd_job           108    108   3312    9    8 : tunables    0    0    0 : slabdata     12     12      0
dm_uevent              0      0   2888   11    8 : tunables    0    0    0 : slabdata      0      0      0
fsverity_info          0      0    256   32    2 : tunables    0    0    0 : slabdata      0      0      0
fscrypt_info           0      0    136   30    1 : tunables    0    0    0 : slabdata      0      0      0
MPTCPv6                0      0   2048   16    8 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    184   22    1 : tunables    0    0    0 : slabdata      0      0      0
PINGv6                 0      0   1216   26    8 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                663    676   1216   26    8 : tunables    0    0    0 : slabdata     26     26      0
UDPv6                240    240   1344   24    8 : tunables    0    0    0 : slabdata     10     10      0
tw_sock_TCPv6         66     66    248   33    2 : tunables    0    0    0 : slabdata      2      2      0
request_sock_TCPv6      0      0    304   26    2 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                195    195   2496   13    8 : tunables    0    0    0 : slabdata     15     15      0
scsi_sense_cache     640    640    128   32    1 : tunables    0    0    0 : slabdata     20     20      0
bfq_io_cq            875    875    160   25    1 : tunables    0    0    0 : slabdata     35     35      0
mqueue_inode_cache    272    272    960   34    8 : tunables    0    0    0 : slabdata      8      8      0
userfaultfd_ctx_cache      0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dnotify_struct         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
dio                    0      0    640   25    4 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace        224    224    144   28    1 : tunables    0    0    0 : slabdata      8      8      0
ip4-frags             20     20    200   20    1 : tunables    0    0    0 : slabdata      1      1      0
MPTCP                  0      0   1856   17    8 : tunables    0    0    0 : slabdata      0      0      0
request_sock_subflow      0      0    376   21    2 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    768   21    4 : tunables    0    0    0 : slabdata      0      0      0
PING                1257   1408   1024   32    8 : tunables    0    0    0 : slabdata     44     44      0
RAW                  896    896   1024   32    8 : tunables    0    0    0 : slabdata     28     28      0
tw_sock_TCP          297    297    248   33    2 : tunables    0    0    0 : slabdata      9      9      0
request_sock_TCP     208    208    304   26    2 : tunables    0    0    0 : slabdata      8      8      0
TCP                  135    182   2368   13    8 : tunables    0    0    0 : slabdata     14     14      0
hugetlbfs_inode_cache     50     50    640   25    4 : tunables    0    0    0 : slabdata      2      2      0
dquot                256    256    256   32    2 : tunables    0    0    0 : slabdata      8      8      0
eventpoll_pwq       1792   1792     72   56    1 : tunables    0    0    0 : slabdata     32     32      0
dax_cache            126    126    768   21    4 : tunables    0    0    0 : slabdata      6      6      0
bio_crypt_ctx       9180   9180     40  102    1 : tunables    0    0    0 : slabdata     90     90      0
request_queue         96     96   2032   16    8 : tunables    0    0    0 : slabdata      6      6      0
biovec-max           202    232   4096    8    8 : tunables    0    0    0 : slabdata     29     29      0
biovec-128           240    272   2048   16    8 : tunables    0    0    0 : slabdata     17     17      0
biovec-64            384    384   1024   32    8 : tunables    0    0    0 : slabdata     12     12      0
khugepaged_mm_slot    288    288    112   36    1 : tunables    0    0    0 : slabdata      8      8      0
user_namespace       240    240    536   30    4 : tunables    0    0    0 : slabdata      8      8      0
dmaengine-unmap-256     15     15   2112   15    8 : tunables    0    0    0 : slabdata      1      1      0
dmaengine-unmap-128     30     30   1088   30    8 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache    3432   3432    832   39    8 : tunables    0    0    0 : slabdata     88     88      0
skbuff_ext_cache    2065   2247    192   21    1 : tunables    0    0    0 : slabdata    107    107      0
skbuff_fclone_cache   1236   1504    512   32    4 : tunables    0    0    0 : slabdata     47     47      0
skbuff_head_cache   2368   2368    256   32    2 : tunables    0    0    0 : slabdata     74     74      0
file_lock_cache      296    296    216   37    2 : tunables    0    0    0 : slabdata      8      8      0
file_lock_ctx      12987  13651     56   73    1 : tunables    0    0    0 : slabdata    187    187      0
fsnotify_mark_connector  15616  15616     32  128    1 : tunables    0    0    0 : slabdata    122    122      0
net_namespace         78     78   5056    6    8 : tunables    0    0    0 : slabdata     13     13      0
x86_lbr                0      0    800   20    4 : tunables    0    0    0 : slabdata      0      0      0
task_delay_info     5154   5202     80   51    1 : tunables    0    0    0 : slabdata    102    102      0
taskstats            184    184    352   23    2 : tunables    0    0    0 : slabdata      8      8      0
proc_dir_entry      2735   2814    192   21    1 : tunables    0    0    0 : slabdata    134    134      0
pde_opener           816    816     40  102    1 : tunables    0    0    0 : slabdata      8      8      0
proc_inode_cache   15429  15939    688   23    4 : tunables    0    0    0 : slabdata    693    693      0
seq_file             272    272    120   34    1 : tunables    0    0    0 : slabdata      8      8      0
bdev_cache           312    312    832   39    8 : tunables    0    0    0 : slabdata      8      8      0
shmem_inode_cache   3067   3696    728   22    4 : tunables    0    0    0 : slabdata    168    168      0
kernfs_node_cache  49445  50528    128   32    1 : tunables    0    0    0 : slabdata   1579   1579      0
mnt_cache           2030   2100    320   25    2 : tunables    0    0    0 : slabdata     84     84      0
filp               19181  20768    256   32    2 : tunables    0    0    0 : slabdata    649    649      0
inode_cache        22035  22230    616   26    4 : tunables    0    0    0 : slabdata    855    855      0
dentry            166057 167013    192   21    1 : tunables    0    0    0 : slabdata   7953   7953      0
names_cache           64     64   4096    8    8 : tunables    0    0    0 : slabdata      8      8      0
buffer_head       260247 260247    104   39    1 : tunables    0    0    0 : slabdata   6673   6673      0
uts_namespace        148    148    440   37    4 : tunables    0    0    0 : slabdata      4      4      0
vm_area_struct     80323  81620    200   20    1 : tunables    0    0    0 : slabdata   4081   4081      0
mm_struct            390    390   1088   30    8 : tunables    0    0    0 : slabdata     13     13      0
files_cache          368    368    704   23    4 : tunables    0    0    0 : slabdata     16     16      0
signal_cache         896    896   1152   28    8 : tunables    0    0    0 : slabdata     32     32      0
sighand_cache        541    555   2112   15    8 : tunables    0    0    0 : slabdata     37     37      0
task_struct         1276   1368   7872    4    8 : tunables    0    0    0 : slabdata    342    342      0
cred_jar            1979   2205    192   21    1 : tunables    0    0    0 : slabdata    105    105      0
anon_vma_chain     35568  39040     64   64    1 : tunables    0    0    0 : slabdata    610    610      0
anon_vma           20573  22632     88   46    1 : tunables    0    0    0 : slabdata    492    492      0
pid                 3712   3712    128   32    1 : tunables    0    0    0 : slabdata    116    116      0
Acpi-Operand        4872   4872     72   56    1 : tunables    0    0    0 : slabdata     87     87      0
Acpi-ParseExt        312    312    104   39    1 : tunables    0    0    0 : slabdata      8      8      0
Acpi-State           408    408     80   51    1 : tunables    0    0    0 : slabdata      8      8      0
numa_policy       136895 137360     24  170    1 : tunables    0    0    0 : slabdata    808    808      0
trace_event_file    3956   3956     88   46    1 : tunables    0    0    0 : slabdata     86     86      0
ftrace_event_field   8415   8415     48   85    1 : tunables    0    0    0 : slabdata     99     99      0
pool_workqueue       749   1152    256   32    2 : tunables    0    0    0 : slabdata     36     36      0
radix_tree_node    82516  82516    584   28    4 : tunables    0    0    0 : slabdata   2947   2947      0
task_group           224    224    576   28    4 : tunables    0    0    0 : slabdata      8      8      0
vmap_area          18937  19072     64   64    1 : tunables    0    0    0 : slabdata    298    298      0
dma-kmalloc-8k         0      0   8192    4    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0   4096    8    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   2048   16    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   1024   32    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0    512   32    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    256   32    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     16  256    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0      8  512    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0   8192    4    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0   4096    8    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   2048   16    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   1024   32    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512     2199   3744    512   32    4 : tunables    0    0    0 : slabdata    117    117      0
kmalloc-rcl-256      608    608    256   32    2 : tunables    0    0    0 : slabdata     19     19      0
kmalloc-rcl-192      671   3171    192   21    1 : tunables    0    0    0 : slabdata    151    151      0
kmalloc-rcl-128     3552   3680    128   32    1 : tunables    0    0    0 : slabdata    115    115      0
kmalloc-rcl-96      5334   5334     96   42    1 : tunables    0    0    0 : slabdata    127    127      0
kmalloc-rcl-64     34752  34752     64   64    1 : tunables    0    0    0 : slabdata    543    543      0
kmalloc-rcl-32         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     16  256    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0      8  512    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k           194    200   8192    4    8 : tunables    0    0    0 : slabdata     50     50      0
kmalloc-4k        199245 199392   4096    8    8 : tunables    0    0    0 : slabdata  24924  24924      0
kmalloc-2k        257858 257984   2048   16    8 : tunables    0    0    0 : slabdata  16124  16124      0
kmalloc-1k        209829 213056   1024   32    8 : tunables    0    0    0 : slabdata   6658   6658      0
kmalloc-512        41273  62880    512   32    4 : tunables    0    0    0 : slabdata   1965   1965      0
kmalloc-256        20173  47200    256   32    2 : tunables    0    0    0 : slabdata   1475   1475      0
kmalloc-192        37779  56070    192   21    1 : tunables    0    0    0 : slabdata   2670   2670      0
kmalloc-128        47482  49760    128   32    1 : tunables    0    0    0 : slabdata   1555   1555      0
kmalloc-96        108187 110082     96   42    1 : tunables    0    0    0 : slabdata   2621   2621      0
kmalloc-64        350083 356672     64   64    1 : tunables    0    0    0 : slabdata   5573   5573      0
kmalloc-32         82316  89600     32  128    1 : tunables    0    0    0 : slabdata    700    700      0
kmalloc-16         50432  50432     16  256    1 : tunables    0    0    0 : slabdata    197    197      0
kmalloc-8          12288  12288      8  512    1 : tunables    0    0    0 : slabdata     24     24      0
kmem_cache_node      512    512     64   64    1 : tunables    0    0    0 : slabdata      8      8      0
kmem_cache           288    288    256   32    2 : tunables    0    0    0 : slabdata      9      9      0

Last edited by Wild Penguin (2021-02-22 15:27:19)

Offline

#2 2021-02-22 17:06:45

xerxes_
Member
Registered: 2018-04-29
Posts: 343

Re: Kernel memory leak since recently (5.10.16-zen)

Don't worry, these are just buffers. Just use more your system RAM and when you will be low on free memory kernel will use that buffers and other memory it finds to your newer load.

BTW. do you use some kind of swap, zswap, zram, etc.? Even if you have plenty of RAM, you may need some kind of swap or zram if your normal memory load is high and reaching limit.
See this:
https://chrisdown.name/2018/01/02/in-de … -swap.html

Offline

#3 2021-02-22 18:00:50

Wild Penguin
Member
Registered: 2015-03-19
Posts: 197

Re: Kernel memory leak since recently (5.10.16-zen)

Hi xerxes_,

Thanks for your reply, but this is really a real case of RAM getting eaten up.

For comparison, here is a normal situation where no (at least a large / easily detectable amount of) leak has occurred, after an uptime of little shy of 3 hours:

MemTotal:       32820080 kB
MemFree:        13566588 kB
MemAvailable:   26347332 kB
Buffers:          156964 kB
Cached:         13977724 kB

For comparison see the same figures in the leak situation in my earlier post. I didn't mention, but there I had an uptime of nearly two days. Buffers and cache take up only around 4GiB of RAM.

Any (more relevant) input is welcome!

Last edited by Wild Penguin (2021-02-22 18:01:21)

Offline

Board footer

Powered by FluxBB