You are not logged in.
Pages: 1
Hello.
I recently changed my work computer with another machine with 32GB of RAM, because I really needed to run more demanding processes.
When the computer boots up, everything is OK, and I have 30GB of free memory. However, after running a process (a memory and CPU intensive task, no significant IO, networking of graphical resources), it seems that not all memory is released back to the system.
That means if I try to run it again overnight, I will get back the following day it it was killed. Just the other day I had only 3GB of free memory running only a terminal!
I am looking for help understanding where that memory is being held up.
$ free
total used free shared buff/cache available
Mem: 32792156 19647600 10510292 382252 2634264 12306648
Swap: 34601960 18042072 16559888
From the above I see that the memory is tied up at some place. It is not appear to be in cache or buffers. The memory allocated for tmpfs is a bit too much (I will lookup how to reduce it), but it held only about 51MB on /tmp. I believe that memory is not captive and seeing as df says the total size of tmpfs filesystems exceed the total RAM installed reinforces this belief (but tell me if I'm wrong).
$ cat /proc/meminfo
MemTotal: 32792156 kB
MemFree: 10470024 kB
MemAvailable: 12266412 kB
Buffers: 28 kB
Cached: 2458488 kB
SwapCached: 17632860 kB
Active: 2795056 kB
Inactive: 18797024 kB
Active(anon): 1594616 kB
Inactive(anon): 17928160 kB
Active(file): 1200440 kB
Inactive(file): 868864 kB
Unevictable: 267940 kB
Mlocked: 48 kB
SwapTotal: 34601960 kB
SwapFree: 16559896 kB
Dirty: 1020 kB
Writeback: 0 kB
AnonPages: 1769300 kB
Mapped: 377568 kB
Shmem: 390156 kB
KReclaimable: 183716 kB
Slab: 263576 kB
SReclaimable: 183716 kB
SUnreclaim: 79860 kB
KernelStack: 7568 kB
PageTables: 16748 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 50998036 kB
Committed_AS: 5565988 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 4352 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 404636 kB
DirectMap2M: 29857792 kB
DirectMap1G: 3145728 kB
/proc/meminfo shows some large numbers I do not know. Active(anon) could very well be the memory allocated to processes, but perhaps something else? In fact, the total memory used up by existing programs is not that high, as can be seen below.
$ ps -e -orss=,args= | sort -nr | head
1238144 palemoon
226172 /usr/bin/emacs --resume-layouts
95584 /usr/lib/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
63000 /usr/bin/python /usr/bin/mpsyt
45240 /usr/bin/python /usr/bin/mpsyt
38288 urxvtd -q -o -f
28212 pamac-tray
26832 nm-applet
26404 /usr/lib/systemd/systemd-journald
14072 /usr/lib/udisks2/udisksd
And finally, here is the output from vmstat.
$ vmstat -w
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 18036224 10758424 28 2585984 34 217 50 252 78 44 44 3 53 0 0
It seems the swapping is tying up some memory, but I thought it is used up only as a convenience and released once the resource is required. That is not what I observe.
I am not using zram or any memory tricks. It seems it could help me in here, but I'm seeking assistance in finding out what is happening with this system. The previous system had only half of the memory but never acted up in this way.
Does anyone have a suggestion? Thank you for your time.
Edit: What tag should I use to make the name of programs and files stand out?
Last edited by Gede (2019-11-15 22:27:47)
Offline
#!/bin/bash
# Display the top applications of memory usage
# http://www.cyberciti.biz/faq/linux-check-memory-usage/#comment-51021
while read command percent rss; do
if [[ "${command}" != "COMMAND" ]]; then
rss="$(bc <<< "scale=2;${rss}/1024")"
fi
printf " %-26s%-8s%s\n" "${command}" "${percent}" "${rss} MB" \
| sed 's/COMMAND/PROGRAM/' | sed 's/RSS MB/#MEM/'
done < <(ps -A --sort -rss -o comm,pmem,rss | head -n 20)
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
Thank you for your interest, graysky. However, there is no process eating up that memory. Nothing even close.
ps, top, htop show no process taking more than 3% of the memory "gone missing", and the number of processes is small.
Offline
32792156 - (18797024+2795056+10470024) = 730052 (total - (inactive + active + free) = 730MB, that's reasonable w/o further inspection)
10GB are free.
You've 2.7GB in active pages, but 17.9GB in inactive pages, mostly anon ie. memory that some process(es) has(have) claimed but not returned and also not touched in a long while. The moment you kill that process(es), the memory should be returned to the system.
I'd suggest to post a complete process list for an open blame game, but the more "exotic" the process, the more likely it's the cause.
Just the other day I had only 3GB of free memory running only a terminal!
That's however not the situation indicated above. DId you check that there were no other stale processes?
Offline
Here is an update then, from this mornig:
$ free
total used free shared buff/cache available
Mem: 32023 29464 1086 245 1472 1867
Swap: 33790 28785 5005
So less than 2GB free.
$ cat /proc/meminfo
MemTotal: 32792156 kB
MemFree: 1113788 kB
MemAvailable: 1915928 kB
Buffers: 88 kB
Cached: 1329876 kB
SwapCached: 29369672 kB
Active: 1081316 kB
Inactive: 30005660 kB
Active(anon): 560604 kB
Inactive(anon): 29451212 kB
Active(file): 520712 kB
Inactive(file): 554448 kB
Unevictable: 148364 kB
Mlocked: 48 kB
SwapTotal: 34601960 kB
SwapFree: 5125736 kB
Dirty: 304 kB
Writeback: 0 kB
AnonPages: 536416 kB
Mapped: 213044 kB
Shmem: 254808 kB
KReclaimable: 183508 kB
Slab: 253736 kB
SReclaimable: 183508 kB
SUnreclaim: 70228 kB
KernelStack: 5632 kB
PageTables: 7048 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 50998036 kB
Committed_AS: 2455872 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 4192 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 359580 kB
DirectMap2M: 25708544 kB
DirectMap1G: 7340032 kB
Inactive(anon) and SwapCached are once more high.
$ ps -e -orss,size,vsz,args | sort -nr
600092 1151852 2555668 palemoon
69388 63856 585996 /usr/lib/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
26276 30372 333880 clipit
24500 18908 47188 urxvtd -q -o -f
20180 19240 81292 /usr/lib/systemd/systemd-journald
14760 27624 266708 pamac-tray
14572 45592 404292 nm-applet
14464 44344 403920 /usr/lib/udisks2/udisksd
10756 26636 208564 volumeicon
9244 69748 1933832 /usr/lib/polkit-1/polkitd --no-debug
8772 35380 324844 /usr/bin/dunst
7828 52720 484980 /usr/bin/NetworkManager --no-daemon
6240 1468 38096 i3
6008 26588 207428 xfce4-power-manager
5832 61224 598716 conky -c /usr/share/conky/conky_maia
5548 25588 186828 /sbin/init
5148 2920 33112 i3bar --bar_id=bar-0 --socket=/run/user/1000/i3/ipc-socket.1137
4992 836 13264 /usr/bin/wpa_supplicant -u
4676 14124 128536 compton -b
4244 34660 254740 /usr/lib/upowerd
3744 1876 8504 bash
3592 1064 31488 /usr/lib/systemd/systemd --user
3140 26384 204492 /usr/lib/polkit-gnome/polkit-gnome-authentication-agent-1
3100 1148 12768 ps -e -orss,size,vsz,args
2908 732 25736 /usr/lib/systemd/systemd-logind
2876 2876 82456 conky -c /usr/share/conky/conky1.10_shortcuts_maia
2752 1988 39924 /usr/lib/systemd/systemd-udevd
2688 1508 11408 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
2628 25772 237868 /usr/lib/gvfsd
2492 34464 316328 /usr/bin/ModemManager
2468 17580 163732 /usr/lib/at-spi2-registryd --use-gnome-session
1956 748 10648 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1828 780 21944 i3status
1552 16884 24932 sort -nr
1408 664 19592 /usr/lib/gconfd-2
1172 736 10760 avahi-daemon: running [L-AG2-SD-GL.local]
1160 1260 8436 /usr/bin/crond -n
820 696 12276 /usr/lib/bluetooth/bluetoothd
720 504 10404 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
20 34284 310816 /usr/bin/lightdm
4 9336 73524 (sd-pam)
4 480 10504 avahi-daemon: chroot helper
4 372 3964 xautolock -time 10 -locker blurlock
4 25676 230900 /usr/lib/xfce4/xfconf/xfconfd
4 17828 199268 lightdm --session-child 12 19
4 17208 156936 /usr/lib/dconf-service
RSS SIZE VSZ COMMAND
0 8688 78064 /usr/bin/lvmetad -f
0 6016 8120 /usr/sbin/haveged -w 1024 -v 1 --Foreground
0 44528 380932 /usr/lib/gvfsd-fuse /run/user/1000/gvfs -f
0 34024 306124 /usr/lib/at-spi-bus-launcher
0 25748 238340 /usr/lib/accounts-daemon
0 0 0 [writeback]
0 0 0 [watchdogd]
0 0 0 [usb-storage]
0 0 0 [tpm_dev_wq]
0 0 0 [scsi_tmf_3]
0 0 0 [scsi_tmf_2]
0 0 0 [scsi_tmf_1]
0 0 0 [scsi_tmf_0]
0 0 0 [scsi_eh_3]
0 0 0 [scsi_eh_2]
0 0 0 [scsi_eh_1]
0 0 0 [scsi_eh_0]
0 0 0 [rcu_tasks_kthre]
0 0 0 [rcu_preempt]
0 0 0 [rcu_par_gp]
0 0 0 [rcu_gp]
0 0 0 [rcuc/7]
0 0 0 [rcuc/6]
0 0 0 [rcuc/5]
0 0 0 [rcuc/4]
0 0 0 [rcuc/3]
0 0 0 [rcuc/2]
0 0 0 [rcuc/1]
0 0 0 [rcuc/0]
0 0 0 [rcub/0]
0 0 0 [oom_reaper]
0 0 0 [nvme-wq]
0 0 0 [nvme-reset-wq]
0 0 0 [nvme-delete-wq]
0 0 0 [netns]
0 0 0 [mm_percpu_wq]
0 0 0 [migration/7]
0 0 0 [migration/6]
0 0 0 [migration/5]
0 0 0 [migration/4]
0 0 0 [migration/3]
0 0 0 [migration/2]
0 0 0 [migration/1]
0 0 0 [migration/0]
0 0 0 [kworker/u17:6-kcryptd/254:0]
0 0 0 [kworker/u17:4-kcryptd/254:0]
0 0 0 [kworker/u17:3-kcryptd/254:0]
0 0 0 [kworker/u17:2-kcryptd/254:0]
0 0 0 [kworker/u17:1-kcryptd/254:0]
0 0 0 [kworker/u17:0-kcryptd/254:0]
0 0 0 [kworker/u16:9-btrfs-endio-write]
0 0 0 [kworker/u16:8-btrfs-endio-write]
0 0 0 [kworker/u16:7-btrfs-endio-write]
0 0 0 [kworker/u16:5-events_unbound]
0 0 0 [kworker/u16:4-btrfs-endio-write]
0 0 0 [kworker/u16:3-btrfs-endio-write]
0 0 0 [kworker/u16:1]
0 0 0 [kworker/u16:0-events_unbound]
0 0 0 [kworker/7:1H-events_highpri]
0 0 0 [kworker/7:1-events]
0 0 0 [kworker/7:0H-kblockd]
0 0 0 [kworker/7:0-events]
0 0 0 [kworker/6:2]
0 0 0 [kworker/6:1H-events_highpri]
0 0 0 [kworker/6:0H-kblockd]
0 0 0 [kworker/6:0-events]
0 0 0 [kworker/5:2H]
0 0 0 [kworker/5:1H-events_highpri]
0 0 0 [kworker/5:1-cgroup_destroy]
0 0 0 [kworker/5:0-mm_percpu_wq]
0 0 0 [kworker/4:2-cgroup_destroy]
0 0 0 [kworker/4:1H-events_highpri]
0 0 0 [kworker/4:1-events_long]
0 0 0 [kworker/4:0H-kblockd]
0 0 0 [kworker/3:2-events]
0 0 0 [kworker/3:1H-events_highpri]
0 0 0 [kworker/3:1-events]
0 0 0 [kworker/3:0H-kblockd]
0 0 0 [kworker/2:2H-kblockd]
0 0 0 [kworker/2:2]
0 0 0 [kworker/2:1H-events_highpri]
0 0 0 [kworker/2:0-mm_percpu_wq]
0 0 0 [kworker/1:2-mm_percpu_wq]
0 0 0 [kworker/1:1-mm_percpu_wq]
0 0 0 [kworker/1:1H-kblockd]
0 0 0 [kworker/1:0-kec_query]
0 0 0 [kworker/1:0H-kblockd]
0 0 0 [kworker/0:2-events]
0 0 0 [kworker/0:1H-events_highpri]
0 0 0 [kworker/0:0H-kblockd]
0 0 0 [kworker/0:0-cgroup_destroy]
0 0 0 [kthrotld]
0 0 0 [kthreadd]
0 0 0 [kswapd0]
0 0 0 [kstrp]
0 0 0 [ksoftirqd/7]
0 0 0 [ksoftirqd/6]
0 0 0 [ksoftirqd/5]
0 0 0 [ksoftirqd/4]
0 0 0 [ksoftirqd/3]
0 0 0 [ksoftirqd/2]
0 0 0 [ksoftirqd/1]
0 0 0 [ksoftirqd/0]
0 0 0 [ksmd]
0 0 0 [kmemstick]
0 0 0 [kintegrityd]
0 0 0 [khungtaskd]
0 0 0 [khugepaged]
0 0 0 [kdmflush]
0 0 0 [kdmflush]
0 0 0 [kdevtmpfs]
0 0 0 [kcryptd_io/254:]
0 0 0 [kcryptd_io/254:]
0 0 0 [kcryptd/254:1]
0 0 0 [kcryptd/254:0]
0 0 0 [kcompactd0]
0 0 0 [kblockd]
0 0 0 [kauditd]
0 0 0 [jbd2/sdb1-8]
0 0 0 [irq/67-smo8800]
0 0 0 [irq/51-DELL0816]
0 0 0 [irq/129-iwlwifi]
0 0 0 [irq/127-mei_me]
0 0 0 [irq/123-aerdrv]
0 0 0 [irq/122-aerdrv]
0 0 0 [ipv6_addrconf]
0 0 0 [idle_inject/7]
0 0 0 [idle_inject/6]
0 0 0 [idle_inject/5]
0 0 0 [idle_inject/4]
0 0 0 [idle_inject/3]
0 0 0 [idle_inject/2]
0 0 0 [idle_inject/1]
0 0 0 [idle_inject/0]
0 0 0 [ext4-rsv-conver]
0 0 0 [edac-poller]
0 0 0 [dmcrypt_write/2]
0 0 0 [dmcrypt_write/2]
0 0 0 [devfreq_wq]
0 0 0 [crypto]
0 0 0 [cpuhp/7]
0 0 0 [cpuhp/6]
0 0 0 [cpuhp/5]
0 0 0 [cpuhp/4]
0 0 0 [cpuhp/3]
0 0 0 [cpuhp/2]
0 0 0 [cpuhp/1]
0 0 0 [cpuhp/0]
0 0 0 [charger_manager]
0 0 0 [cfg80211]
0 0 0 [btrfs-worker-hi]
0 0 0 [btrfs-worker]
0 0 0 [btrfs-transacti]
0 0 0 [btrfs-submit]
0 0 0 [btrfs-rmw]
0 0 0 [btrfs-readahead]
0 0 0 [btrfs-qgroup-re]
0 0 0 [btrfs-freespace]
0 0 0 [btrfs-flush_del]
0 0 0 [btrfs-fixup]
0 0 0 [btrfs-extent-re]
0 0 0 [btrfs-endio-wri]
0 0 0 [btrfs-endio-rep]
0 0 0 [btrfs-endio-rai]
0 0 0 [btrfs-endio-met]
0 0 0 [btrfs-endio-met]
0 0 0 [btrfs-endio]
0 0 0 [btrfs-delayed-m]
0 0 0 [btrfs-delalloc]
0 0 0 [btrfs-cleaner]
0 0 0 [btrfs-cache]
0 0 0 [ata_sff]
0 0 0 [acpi_thermal_pm]
The sum of each of these columns is about 932k, 2068k and 11692k. Still far from the value of used RAM value reported by free.
(In ps, vsz is reported in KB, according to the man page. I think it is likely the other values are too. Thus running processes account only for less than 12GB of memory in a worse-case scenario.)
$ vmstat -w
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 29476224 1018408 88 1543032 0 76 4 80 86 1 12 0 88 0 0
So, it seems the problem is related to swap, somehow?
Update
# swapoff -a
Got me back my memory.
Last edited by Gede (2019-11-15 12:24:30)
Offline
"swpd" is the amount of virtual memory, not "swap". When you run into this, start shooting processes one-by-one (starting w/ the least important ones, eg. palemoon or so, not systemd-logind or stuff like that)
Something™ is likely requesting HUUUUUUUUUGE pages and unable to return them to the system, so while the actual memory usage is low, large amounts of pot. memory are occupied.
I guess the browser is a good contender, but tehcnically it could be everything.
Edit: is your swap partition inside the LVM?
Last edited by seth (2019-11-15 12:33:29)
Offline
My swap partition is not inside the LVM. I'm not even using LVM. I have a primary partition for the swap and another for the root FS.
Supposedly, they are both encrypted but, while I can see that the filesystem is mounted through /dev/mapper/luks-..., I do not know how to inspect the swap and see if things are fine.
However, I created a swapfile and the system has been running just fine this way.
$ free
total used free shared buff/cache available
Mem: 32023 4135 27034 210 853 27251
Swap: 33791 6501 27290
This makes me want to reevaluate the swap partition to see if everything is fine. I think I will look into it and open a new question if needed. Thank you.
Offline
Please don't use the [CLOSED] tag in your thread title, it's reserved for threads that have been locked from further posts by the mods.
Thanks.
Offline
Please don't use the [CLOSED] tag in your thread title, it's reserved for threads that have been locked from further posts by the mods.
It the problem is not solved but I changed the title to avoid confusion in the forum. If I could use some other label that is better fitting, please let me know. Thanks.
Offline
Pages: 1