You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 2020-06-09 04:36:19
- wholesome1
- Member
- Registered: 2020-06-09
- Posts: 3
Kernel memory leak?
Hi Arch Linux Forum,
I have a VPS with 2GB of memory. I don't use it a lot, leaving it mostly idle except the IPv6 BGP session (over 6in4) I have with he.net and NetAssist. I don't load the routing table; I just route everything to he.net.
However, I noticed that the memory usage increases slowly over time. I need to hard-reset the VPS every few weeks to restore it. This behaviour started roughly since 2020 started but I haven't made any significant changes to the configuration
Thanks to this beautiful graph from hetrixtools hopefully it explains better:
Memory usage is quite high:
[me@lax2 ~]$ free -h
total used free shared buff/cache available
Mem: 2.1Gi 1.6Gi 95Mi 1.0Mi 366Mi 289Mi
Swap: 303Mi 20Mi 283MiYou can see that there's actually no application actively consuming memory:
[me@lax2 ~]$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.3 109744 8264 ? Ss May13 5:58 /sbin/init
root 2 0.0 0.0 0 0 ? S May13 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? I< May13 0:00 [rcu_gp]
root 4 0.0 0.0 0 0 ? I< May13 0:00 [rcu_par_gp]
root 6 0.0 0.0 0 0 ? I< May13 0:00 [kworker/0:0H-kblockd]
root 8 0.0 0.0 0 0 ? I< May13 0:00 [mm_percpu_wq]
root 9 0.0 0.0 0 0 ? S May13 1:39 [ksoftirqd/0]
root 10 0.0 0.0 0 0 ? S May13 0:00 [rcuc/0]
root 11 0.0 0.0 0 0 ? I May13 11:11 [rcu_preempt]
root 12 0.0 0.0 0 0 ? S May13 0:00 [rcub/0]
root 13 0.0 0.0 0 0 ? S May13 0:04 [migration/0]
root 14 0.0 0.0 0 0 ? S May13 0:00 [idle_inject/0]
root 16 0.0 0.0 0 0 ? S May13 0:00 [cpuhp/0]
root 17 0.0 0.0 0 0 ? S May13 0:00 [kdevtmpfs]
root 18 0.0 0.0 0 0 ? I< May13 0:00 [netns]
root 19 0.0 0.0 0 0 ? S May13 0:00 [rcu_tasks_kthre]
root 20 0.0 0.0 0 0 ? S May13 0:03 [kauditd]
root 21 0.0 0.0 0 0 ? S May13 0:00 [khungtaskd]
root 22 0.0 0.0 0 0 ? S May13 0:00 [oom_reaper]
root 23 0.0 0.0 0 0 ? I< May13 0:00 [writeback]
root 24 0.0 0.0 0 0 ? S May13 0:01 [kcompactd0]
root 25 0.0 0.0 0 0 ? SN May13 0:00 [ksmd]
root 26 0.0 0.0 0 0 ? SN May13 0:00 [khugepaged]
root 114 0.0 0.0 0 0 ? I< May13 0:00 [kintegrityd]
root 115 0.0 0.0 0 0 ? I< May13 0:00 [kblockd]
root 116 0.0 0.0 0 0 ? I< May13 0:00 [blkcg_punt_bio]
root 117 0.0 0.0 0 0 ? I< May13 0:00 [ata_sff]
root 118 0.0 0.0 0 0 ? I< May13 0:00 [edac-poller]
root 119 0.0 0.0 0 0 ? I< May13 0:00 [devfreq_wq]
root 120 0.0 0.0 0 0 ? S May13 0:00 [watchdogd]
root 121 0.0 0.0 0 0 ? S May13 4:59 [kswapd0]
root 124 0.0 0.0 0 0 ? I< May13 0:00 [kthrotld]
root 125 0.0 0.0 0 0 ? I< May13 0:00 [acpi_thermal_pm]
root 126 0.0 0.0 0 0 ? I< May13 0:00 [nvme-wq]
root 127 0.0 0.0 0 0 ? I< May13 0:00 [nvme-reset-wq]
root 128 0.0 0.0 0 0 ? I< May13 0:00 [nvme-delete-wq]
root 129 0.0 0.0 0 0 ? I< May13 0:00 [ipv6_addrconf]
root 140 0.0 0.0 0 0 ? I< May13 0:00 [kstrp]
root 146 0.0 0.0 0 0 ? I< May13 0:00 [zswap-shrink]
root 147 0.0 0.0 0 0 ? I< May13 0:00 [kworker/u3:0]
root 158 0.0 0.0 0 0 ? I< May13 0:00 [charger_manager]
root 188 0.0 0.0 0 0 ? S May13 0:00 [scsi_eh_0]
root 189 0.0 0.0 0 0 ? I< May13 0:00 [scsi_tmf_0]
root 190 0.0 0.0 0 0 ? S May13 0:00 [scsi_eh_1]
root 191 0.0 0.0 0 0 ? I< May13 0:00 [scsi_tmf_1]
root 195 0.0 0.0 0 0 ? I< May13 0:15 [kworker/0:1H-kblockd]
root 206 0.0 0.0 0 0 ? S May13 0:10 [jbd2/vda2-8]
root 207 0.0 0.0 0 0 ? I< May13 0:00 [ext4-rsv-conver]
root 234 0.0 2.2 172832 47940 ? Ss May13 3:28 /usr/lib/systemd/systemd-journald
root 242 0.0 0.0 78076 688 ? Ss May13 0:00 /usr/bin/lvmetad -f
root 247 0.0 0.2 30940 4732 ? Ss May13 0:02 /usr/lib/systemd/systemd-udevd
systemd+ 250 0.0 0.2 26260 5208 ? Ss May13 0:05 /usr/lib/systemd/systemd-networkd
systemd+ 276 0.0 0.1 91784 4276 ? Ssl May13 0:03 /usr/lib/systemd/systemd-timesyncd
root 289 0.0 0.0 6588 1968 ? Ss May13 0:07 /usr/bin/crond -n
dbus 290 0.0 0.1 6780 2772 ? Ss May13 3:35 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 291 0.0 0.2 17504 5660 ? Ss May13 1:47 /usr/lib/systemd/systemd-logind
v2ray 294 0.1 0.6 1168500 13268 ? Ssl May13 14:52 /usr/bin/v2ray -config /etc/v2ray/0.json
bird 295 0.0 0.0 7156 1268 ? Ss May13 2:00 /usr/bin/bird -s /run/bird/bird.ctl
root 296 0.0 0.0 5424 1304 tty1 Ss+ May13 0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
http 315 0.0 0.5 1255500 12164 ? Ssl May13 0:54 /usr/bin/caddy -log stdout -agree -conf /etc/caddy/caddy.conf -root=/usr/share/caddy
root 3555765 0.0 0.0 0 0 ? I 21:27 0:00 [kworker/0:0-rcu_gp]
root 3566553 0.0 0.0 0 0 ? I 21:33 0:00 [kworker/u2:1-ext4-rsv-conversion]
root 3567234 0.0 0.0 0 0 ? I 21:33 0:00 [kworker/0:1-events]
root 3569037 0.0 0.3 10704 7448 ? Ss 21:34 0:00 sshd: me [priv]
me 3569138 0.0 0.4 18972 9776 ? Ss 21:34 0:00 /usr/lib/systemd/systemd --user
me 3569140 0.0 0.0 113348 2116 ? S 21:34 0:00 (sd-pam)
me 3569145 0.0 0.1 10704 4156 ? S 21:34 0:00 sshd: me@pts/0
me 3569146 0.0 0.1 7488 4000 pts/0 Ss 21:34 0:00 -bash
root 3577610 0.0 0.0 0 0 ? I 21:38 0:00 [kworker/u2:0-flush-254:0]
root 3584705 0.1 0.0 0 0 ? I 21:42 0:00 [kworker/0:2-events]
root 3588276 0.0 0.0 0 0 ? I 21:44 0:00 [kworker/u2:2-events_unbound]
root 3591768 0.0 0.1 9636 3600 ? S 21:46 0:00 /usr/bin/CROND -n
hetrixt+ 3591769 0.0 0.1 7144 2860 ? Ss 21:46 0:00 /bin/sh -c bash /etc/hetrixtools/hetrixtools_agent.sh >> /etc/hetrixtools/hetrixtools_cron.log 2>&1
hetrixt+ 3591770 0.3 0.1 7144 3392 ? S 21:46 0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593070 0.0 0.0 7144 1816 ? S 21:46 0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593071 0.0 0.0 7972 1184 ? S 21:46 0:00 vmstat 3 2
hetrixt+ 3593072 0.0 0.0 5340 580 ? S 21:46 0:00 tail -1
me 3593073 0.0 0.1 9500 3596 pts/0 R+ 21:46 0:00 ps aux[me@lax2 ~]$ cat /proc/meminfo
MemTotal: 2163084 kB
MemFree: 111300 kB
MemAvailable: 302496 kB
Buffers: 5500 kB
Cached: 70908 kB
SwapCached: 1008 kB
Active: 56160 kB
Inactive: 54484 kB
Active(anon): 10912 kB
Inactive(anon): 22148 kB
Active(file): 45248 kB
Inactive(file): 32336 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 311292 kB
SwapFree: 289952 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 33544 kB
Mapped: 46160 kB
Shmem: 1624 kB
KReclaimable: 284608 kB
Slab: 1881652 kB
SReclaimable: 284608 kB
SUnreclaim: 1597044 kB
KernelStack: 1520 kB
PageTables: 1616 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1392832 kB
Committed_AS: 476668 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10888 kB
VmallocChunk: 0 kB
Percpu: 37680 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1665012 kB
DirectMap2M: 563200 kB$slabtop
Active / Total Objects (% used) : 14075841 / 14088832 (99.9%)
Active / Total Slabs (% used) : 332194 / 332194 (100.0%)
Active / Total Caches (% used) : 94 / 129 (72.9%)
Active / Total Size (% used) : 1835557.66K / 1838777.06K (99.8%)
Minimum / Average / Maximum Object : 0.01K / 0.13K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4723200 4723200 100% 0.01K 9225 512 36900K kmalloc-8
2095170 2094988 99% 0.19K 99770 21 399080K cred_jar
1584128 1584128 100% 0.03K 12376 128 49504K kmalloc-32
1498624 1498324 99% 0.06K 23416 64 93664K anon_vma_chain
749248 749248 100% 0.09K 16288 46 65152K anon_vma
577376 577376 100% 0.12K 18043 32 72172K kernfs_node_cache
402400 402050 99% 0.25K 25150 16 100600K filp
296960 296960 100% 0.12K 9280 32 37120K pid
251538 251538 100% 0.09K 5989 42 23956K kmalloc-96
214336 214336 100% 0.06K 3349 64 13396K kmem_cache_node
214290 214290 100% 0.44K 23810 9 95240K kmem_cache
214284 211399 98% 0.19K 10204 21 40816K dentry
181760 181760 100% 0.02K 710 256 2840K kmalloc-16
177308 177308 100% 0.81K 9332 19 149312K sock_inode_cache
148304 148256 99% 0.25K 9269 16 37076K skbuff_head_cache
138630 138630 100% 1.06K 9242 15 147872K mm_struct
135808 135808 100% 0.06K 2122 64 8488K kmalloc-64
131911 131331 99% 0.60K 10147 13 81176K inode_cache
74552 74552 100% 1.00K 9319 8 74552K PING
71076 71050 99% 1.00K 8885 8 71080K kmalloc-1k
46968 46968 100% 4.00K 5871 8 187872K kmalloc-4k
32384 32384 100% 0.50K 4048 8 16192K kmalloc-512
22197 22197 100% 0.19K 1057 21 4228K kmalloc-192
19312 19312 100% 0.25K 1207 16 4828K kmalloc-256
15639 8516 54% 0.10K 401 39 1604K buffer_head
15246 15105 99% 0.57K 1089 14 8712K radix_tree_node
9856 9856 100% 0.12K 308 32 1232K kmalloc-128
9768 9752 99% 2.00K 1221 8 19536K kmalloc-2k
9435 9435 100% 0.08K 185 51 740K Acpi-State
4080 4080 100% 0.05K 48 85 192K ftrace_event_field
2651 2174 82% 1.07K 95 29 3040K ext4_inode_cache
2142 1836 85% 0.04K 21 102 84K ext4_extent_status
1725 1556 90% 0.67K 75 23 1200K proc_inode_cache
1568 1568 100% 0.07K 28 56 112K eventpoll_pwq
1426 1426 100% 0.09K 31 46 124K trace_event_file
1280 1280 100% 0.06K 20 64 80K vmap_area
1120 1120 100% 0.07K 20 56 80K Acpi-Operand
1116 1116 100% 0.11K 31 36 124K khugepaged_mm_slot
1034 1016 98% 0.71K 94 11 752K shmem_inode_cache
969 969 100% 0.08K 19 51 76K task_delay_info
816 816 100% 0.04K 8 102 32K pde_opener
714 546 76% 0.09K 17 42 68K kmalloc-rcl-96
660 660 100% 0.31K 55 12 220K mnt_cache
640 588 91% 0.06K 10 64 40K kmalloc-rcl-64
510 510 100% 0.12K 15 34 60K jbd2_journal_head
504 504 100% 0.19K 24 21 96K skbuff_ext_cache[me@lax2 ~]$ sudo smem -tw
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 2018832 357292 1661540
userspace memory 59244 33928 25316
free memory 85008 85008 0
----------------------------------------------------------
2163084 476228 1686856 [me@lax2 ~]$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
dev 268304 343 267961 1% /dev
run 270385 438 269947 1% /run
/dev/vda2 636480 462678 173802 73% /
tmpfs 270385 1 270384 1% /dev/shm
tmpfs 270385 18 270367 1% /sys/fs/cgroup
tmpfs 270385 17 270368 1% /tmp
tmpfs 270385 22 270363 1% /run/user/1000[me@lax2 ~]$ sudo cat /proc/slabinfo && cat /proc/meminfo && free -h
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_4k 84 84 144 28 1 : tunables 0 0 0 : slabdata 3 3 0
ext4_inode_cache 4919 11687 1096 29 8 : tunables 0 0 0 : slabdata 403 403 0
ext4_system_zone 102 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0
ext4_io_end 64 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0
ext4_extent_status 3278 3876 40 102 1 : tunables 0 0 0 : slabdata 38 38 0
jbd2_journal_handle 438 438 56 73 1 : tunables 0 0 0 : slabdata 6 6 0
jbd2_journal_head 748 918 120 34 1 : tunables 0 0 0 : slabdata 27 27 0
jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 0 0 : slabdata 1 1 0
jbd2_revoke_record_s 128 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0
scsi_sense_cache 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0
fsverity_info 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0
fscrypt_info 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0
MPTCPv6 0 0 1664 19 8 : tunables 0 0 0 : slabdata 0 0 0
ip6-frags 0 0 184 22 1 : tunables 0 0 0 : slabdata 0 0 0
PINGv6 0 0 1216 13 4 : tunables 0 0 0 : slabdata 0 0 0
RAWv6 13 13 1216 13 4 : tunables 0 0 0 : slabdata 1 1 0
UDPv6 240 240 1344 12 4 : tunables 0 0 0 : slabdata 20 20 0
tw_sock_TCPv6 32 32 248 16 1 : tunables 0 0 0 : slabdata 2 2 0
request_sock_TCPv6 13 13 304 13 1 : tunables 0 0 0 : slabdata 1 1 0
TCPv6 247 247 2432 13 8 : tunables 0 0 0 : slabdata 19 19 0
bfq_io_cq 0 0 160 25 1 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 8 8 960 8 2 : tunables 0 0 0 : slabdata 1 1 0
dnotify_struct 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
pid_namespace 0 0 208 19 1 : tunables 0 0 0 : slabdata 0 0 0
ip4-frags 0 0 200 20 1 : tunables 0 0 0 : slabdata 0 0 0
MPTCP 0 0 1472 11 4 : tunables 0 0 0 : slabdata 0 0 0
request_sock_subflow 0 0 344 23 2 : tunables 0 0 0 : slabdata 0 0 0
xfrm_state 0 0 768 10 2 : tunables 0 0 0 : slabdata 0 0 0
PING 70000 70000 1024 8 2 : tunables 0 0 0 : slabdata 8750 8750 0
RAW 8 8 1024 8 2 : tunables 0 0 0 : slabdata 1 1 0
UDP 98 98 1152 14 4 : tunables 0 0 0 : slabdata 7 7 0
tw_sock_TCP 16 16 248 16 1 : tunables 0 0 0 : slabdata 1 1 0
request_sock_TCP 39 39 304 13 1 : tunables 0 0 0 : slabdata 3 3 0
TCP 42 42 2304 14 8 : tunables 0 0 0 : slabdata 3 3 0
hugetlbfs_inode_cache 12 12 640 12 2 : tunables 0 0 0 : slabdata 1 1 0
dquot 48 48 256 16 1 : tunables 0 0 0 : slabdata 3 3 0
eventpoll_pwq 1288 1288 72 56 1 : tunables 0 0 0 : slabdata 23 23 0
dax_cache 10 10 768 10 2 : tunables 0 0 0 : slabdata 1 1 0
request_queue 15 15 2104 15 8 : tunables 0 0 0 : slabdata 1 1 0
biovec-max 78 112 4096 8 8 : tunables 0 0 0 : slabdata 14 14 0
biovec-128 8 8 2048 8 4 : tunables 0 0 0 : slabdata 1 1 0
biovec-64 8 8 1024 8 2 : tunables 0 0 0 : slabdata 1 1 0
khugepaged_mm_slot 1044 1044 112 36 1 : tunables 0 0 0 : slabdata 29 29 0
user_namespace 0 0 536 15 2 : tunables 0 0 0 : slabdata 0 0 0
dmaengine-unmap-256 15 15 2112 15 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-128 15 15 1088 15 4 : tunables 0 0 0 : slabdata 1 1 0
sock_inode_cache 166478 166478 832 19 4 : tunables 0 0 0 : slabdata 8762 8762 0
skbuff_ext_cache 420 420 192 21 1 : tunables 0 0 0 : slabdata 20 20 0
skbuff_fclone_cache 48 48 512 8 1 : tunables 0 0 0 : slabdata 6 6 0
skbuff_head_cache 139488 139488 256 16 1 : tunables 0 0 0 : slabdata 8718 8718 0
file_lock_cache 36 36 216 18 1 : tunables 0 0 0 : slabdata 2 2 0
file_lock_ctx 146 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0
fsnotify_mark_connector 128 128 32 128 1 : tunables 0 0 0 : slabdata 1 1 0
net_namespace 0 0 4928 6 8 : tunables 0 0 0 : slabdata 0 0 0
task_delay_info 765 765 80 51 1 : tunables 0 0 0 : slabdata 15 15 0
taskstats 11 11 352 11 1 : tunables 0 0 0 : slabdata 1 1 0
proc_dir_entry 357 357 192 21 1 : tunables 0 0 0 : slabdata 17 17 0
pde_opener 612 612 40 102 1 : tunables 0 0 0 : slabdata 6 6 0
proc_inode_cache 1610 1702 688 23 4 : tunables 0 0 0 : slabdata 74 74 0
bdev_cache 19 19 832 19 4 : tunables 0 0 0 : slabdata 1 1 0
shmem_inode_cache 1029 1034 728 11 2 : tunables 0 0 0 : slabdata 94 94 0
kernfs_node_cache 541632 541632 128 32 1 : tunables 0 0 0 : slabdata 16926 16926 0
mnt_cache 708 708 320 12 1 : tunables 0 0 0 : slabdata 59 59 0
filp 378420 378656 256 16 1 : tunables 0 0 0 : slabdata 23666 23666 0
inode_cache 122986 123487 616 13 2 : tunables 0 0 0 : slabdata 9499 9499 0
dentry 198930 205359 192 21 1 : tunables 0 0 0 : slabdata 9779 9779 0
names_cache 16 16 4096 8 8 : tunables 0 0 0 : slabdata 2 2 0
buffer_head 15458 20865 104 39 1 : tunables 0 0 0 : slabdata 535 535 0
uts_namespace 9 9 440 9 1 : tunables 0 0 0 : slabdata 1 1 0
mm_struct 129135 129135 1088 15 4 : tunables 0 0 0 : slabdata 8609 8609 0
files_cache 121 121 704 11 2 : tunables 0 0 0 : slabdata 11 11 0
signal_cache 255 255 1088 15 4 : tunables 0 0 0 : slabdata 17 17 0
sighand_cache 195 195 2112 15 8 : tunables 0 0 0 : slabdata 13 13 0
task_struct 138 164 7424 4 8 : tunables 0 0 0 : slabdata 41 41 0
cred_jar 2030693 2030931 192 21 1 : tunables 0 0 0 : slabdata 96711 96711 0
anon_vma_chain 1407381 1407872 64 64 1 : tunables 0 0 0 : slabdata 21998 21998 0
anon_vma 696072 696072 88 46 1 : tunables 0 0 0 : slabdata 15132 15132 0
pid 276544 276544 128 32 1 : tunables 0 0 0 : slabdata 8642 8642 0
Acpi-Operand 1120 1120 72 56 1 : tunables 0 0 0 : slabdata 20 20 0
Acpi-ParseExt 39 39 104 39 1 : tunables 0 0 0 : slabdata 1 1 0
Acpi-State 8823 8823 80 51 1 : tunables 0 0 0 : slabdata 173 173 0
numa_policy 170 170 24 170 1 : tunables 0 0 0 : slabdata 1 1 0
trace_event_file 1426 1426 88 46 1 : tunables 0 0 0 : slabdata 31 31 0
ftrace_event_field 4080 4080 48 85 1 : tunables 0 0 0 : slabdata 48 48 0
pool_workqueue 128 128 256 16 1 : tunables 0 0 0 : slabdata 8 8 0
radix_tree_node 15189 15624 584 14 2 : tunables 0 0 0 : slabdata 1116 1116 0
task_group 36 36 640 12 2 : tunables 0 0 0 : slabdata 3 3 0
vmap_area 1216 1216 64 64 1 : tunables 0 0 0 : slabdata 19 19 0
dma-kmalloc-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-2k 0 0 2048 8 4 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-1k 0 0 1024 8 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-512 8 8 512 8 1 : tunables 0 0 0 : slabdata 1 1 0
dma-kmalloc-256 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-192 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-2k 0 0 2048 8 4 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-1k 0 0 1024 8 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-512 0 0 512 8 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-256 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-192 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-128 96 96 128 32 1 : tunables 0 0 0 : slabdata 3 3 0
kmalloc-rcl-96 1988 2184 96 42 1 : tunables 0 0 0 : slabdata 52 52 0
kmalloc-rcl-64 1145 2048 64 64 1 : tunables 0 0 0 : slabdata 32 32 0
kmalloc-rcl-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-8k 35 56 8192 4 8 : tunables 0 0 0 : slabdata 14 14 0
kmalloc-4k 43968 43968 4096 8 8 : tunables 0 0 0 : slabdata 5496 5496 0
kmalloc-2k 9165 9176 2048 8 4 : tunables 0 0 0 : slabdata 1147 1147 0
kmalloc-1k 64640 64640 1024 8 2 : tunables 0 0 0 : slabdata 8080 8080 0
kmalloc-512 32608 32608 512 8 1 : tunables 0 0 0 : slabdata 4076 4076 0
kmalloc-256 18192 18192 256 16 1 : tunables 0 0 0 : slabdata 1137 1137 0
kmalloc-192 20643 20643 192 21 1 : tunables 0 0 0 : slabdata 983 983 0
kmalloc-128 9280 9280 128 32 1 : tunables 0 0 0 : slabdata 290 290 0
kmalloc-96 236292 236292 96 42 1 : tunables 0 0 0 : slabdata 5626 5626 0
kmalloc-64 127570 127872 64 64 1 : tunables 0 0 0 : slabdata 1998 1998 0
kmalloc-32 1585408 1585408 32 128 1 : tunables 0 0 0 : slabdata 12386 12386 0
kmalloc-16 170752 170752 16 256 1 : tunables 0 0 0 : slabdata 667 667 0
kmalloc-8 4296704 4296704 8 512 1 : tunables 0 0 0 : slabdata 8392 8392 0
kmem_cache_node 201344 201344 64 64 1 : tunables 0 0 0 : slabdata 3146 3146 0
kmem_cache 201330 201330 448 9 1 : tunables 0 0 0 : slabdata 22370 22370 0
MemTotal: 2163104 kB
MemFree: 133316 kB
MemAvailable: 346344 kB
Buffers: 17868 kB
Cached: 99716 kB
SwapCached: 264 kB
Active: 84384 kB
Inactive: 90608 kB
Active(anon): 18548 kB
Inactive(anon): 36044 kB
Active(file): 65836 kB
Inactive(file): 54564 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 311292 kB
SwapFree: 310768 kB
Dirty: 816 kB
Writeback: 0 kB
AnonPages: 57180 kB
Mapped: 69860 kB
Shmem: 1796 kB
KReclaimable: 281144 kB
Slab: 1796448 kB
SReclaimable: 281144 kB
SUnreclaim: 1515304 kB
KernelStack: 1568 kB
PageTables: 1912 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1392844 kB
Committed_AS: 462684 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10832 kB
VmallocChunk: 0 kB
Percpu: 35412 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1638388 kB
DirectMap2M: 589824 kB
total used free shared buff/cache available
Mem: 2.1Gi 1.6Gi 130Mi 1.0Mi 389Mi 338Mi
Swap: 303Mi 0.0Ki 303MiOffline
#2 2020-06-09 07:11:34
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 77,707
Re: Kernel memory leak?
SUnreclaim: 1597044 kBThat's ~1.5 GB, so "yes"…
High filp smells fileystem related, but it seems there's only one ext4?
Do you have many deleted files ("sudo lsof | grep deleted")?
Offline
#3 2020-06-09 07:23:55
- wholesome1
- Member
- Registered: 2020-06-09
- Posts: 3
Re: Kernel memory leak?
I am not sure 
running outdated binaries doesn't count, does it?
[me@lax2 ~]$ sudo lsof | grep deleted
systemd-j 231 root txt REG 254,2 149352 408867 /usr/lib/systemd/systemd-journald (deleted)
lvmetad 238 root txt REG 254,2 85496 412290 /usr/bin/lvmetad (deleted)
systemd-u 243 root txt REG 254,2 411504 408904 /usr/lib/systemd/systemd-udevd (deleted)
systemd-n 247 systemd-network txt REG 254,2 915376 408878 /usr/lib/systemd/systemd-networkd (deleted)
systemd-t 273 systemd-timesync txt REG 254,2 51032 408903 /usr/lib/systemd/systemd-timesyncd (deleted)
systemd-t 273 284 sd-resolv systemd-timesync txt REG 254,2 51032 408903 /usr/lib/systemd/systemd-timesyncd (deleted)
systemd-l 289 root txt REG 254,2 260000 408871 /usr/lib/systemd/systemd-logind (deleted)
v2ray 291 v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 295 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 296 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 297 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 310 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 311 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
v2ray 291 41631 v2ray v2ray txt REG 254,2 20852896 587435 /usr/lib/v2ray/v2ray (deleted)
systemd 2275523 isaac txt REG 254,2 1570680 408820 /usr/lib/systemd/systemd (deleted)
(sd-pam) 2275524 isaac txt REG 254,2 1570680 408820 /usr/lib/systemd/systemd (deleted)
gpg-agent 2280407 root txt REG 254,2 414336 418475 /usr/bin/gpg-agent (deleted)
[me@lax2 ~]$ I tried to kill the biggest offender (v2ray above) but it is not particularly helping.
Before killing:
[me@lax2 ~]$ free -h && cat /proc/meminfo | grep SU
total used free shared buff/cache available
Mem: 2.1Gi 1.2Gi 261Mi 1.0Mi 578Mi 656Mi
Swap: 303Mi 1.0Mi 302Mi
SUnreclaim: 1200392 kBAfter killing:
[me@lax2 ~]$ free -h && cat /proc/meminfo | grep SU
total used free shared buff/cache available
Mem: 2.1Gi 1.2Gi 287Mi 1.0Mi 572Mi 673Mi
Swap: 303Mi 1.0Mi 302Mi
SUnreclaim: 1199216 kBlsof now looks like this:
[isaac@lax2 ~]$ sudo lsof | grep deleted
systemd-j 231 root txt REG 254,2 149352 408867 /usr/lib/systemd/systemd-journald (deleted)
lvmetad 238 root txt REG 254,2 85496 412290 /usr/bin/lvmetad (deleted)
systemd-u 243 root txt REG 254,2 411504 408904 /usr/lib/systemd/systemd-udevd (deleted)
systemd-n 247 systemd-network txt REG 254,2 915376 408878 /usr/lib/systemd/systemd-networkd (deleted)
systemd-t 273 systemd-timesync txt REG 254,2 51032 408903 /usr/lib/systemd/systemd-timesyncd (deleted)
systemd-t 273 284 sd-resolv systemd-timesync txt REG 254,2 51032 408903 /usr/lib/systemd/systemd-timesyncd (deleted)
systemd-l 289 root txt REG 254,2 260000 408871 /usr/lib/systemd/systemd-logind (deleted)
systemd 2275523 isaac txt REG 254,2 1570680 408820 /usr/lib/systemd/systemd (deleted)
(sd-pam) 2275524 isaac txt REG 254,2 1570680 408820 /usr/lib/systemd/systemd (deleted)
gpg-agent 2280407 root txt REG 254,2 414336 418475 /usr/bin/gpg-agent (deleted)Offline
#4 2020-06-09 14:25:44
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 77,707
Re: Kernel memory leak?
No, we were looking for MAAAAAAAAAAAAAAAAANY files…
Do you need LVM? Otherwise disable the lvmetad, https://wiki.archlinux.org/index.php/LV … ed_lvmetad (though "use_lvmetad = 0")
Otherwise v2ray seems to be the long running service and is more on the "exoctic" side - can you run the system for a while w/o it (at all) and see whether the leak still happens (since the pages are lost in the kernel, killing userspace processes will not release them)
Offline
#5 2020-10-05 09:14:22
- wholesome1
- Member
- Registered: 2020-06-09
- Posts: 3
Re: Kernel memory leak?
Fixed after disabling socket-activated OpenSSHD.
https://wiki.archlinux.org/index.php/Op … management
https://bugs.archlinux.org/task/62248
Last edited by wholesome1 (2020-10-05 09:14:33)
Offline
Pages: 1