You are not logged in.

#1 2019-11-14 20:21:51

Gede
Member
Registered: 2009-09-03
Posts: 28

[solved] What is using all the memory?

Hello.
I recently changed my work computer with another machine with 32GB of RAM, because I really needed to run more demanding processes.
When the computer boots up, everything is OK, and I have 30GB of free memory. However, after running a process (a memory and CPU intensive task, no significant IO, networking of graphical resources), it seems that not all memory is released back to the system.
That means if I try to run it again overnight, I will get back the following day it it was killed. Just the other day I had only 3GB of free memory running only a terminal!

I am looking for help understanding where that memory is being held up.

$ free

              total        used        free      shared  buff/cache   available
Mem:       32792156    19647600    10510292      382252     2634264    12306648
Swap:      34601960    18042072    16559888

From the above I see that the memory is tied up at some place. It is not appear to be in cache or buffers. The memory allocated for tmpfs is a bit too much (I will lookup how to reduce it), but it held only about 51MB on /tmp. I believe that memory is not captive and seeing as df says the total size of tmpfs filesystems exceed the total RAM installed reinforces this belief (but tell me if I'm wrong).

$ cat /proc/meminfo

MemTotal:       32792156 kB
MemFree:        10470024 kB
MemAvailable:   12266412 kB
Buffers:              28 kB
Cached:          2458488 kB
SwapCached:     17632860 kB
Active:          2795056 kB
Inactive:       18797024 kB
Active(anon):    1594616 kB
Inactive(anon): 17928160 kB
Active(file):    1200440 kB
Inactive(file):   868864 kB
Unevictable:      267940 kB
Mlocked:              48 kB
SwapTotal:      34601960 kB
SwapFree:       16559896 kB
Dirty:              1020 kB
Writeback:             0 kB
AnonPages:       1769300 kB
Mapped:           377568 kB
Shmem:            390156 kB
KReclaimable:     183716 kB
Slab:             263576 kB
SReclaimable:     183716 kB
SUnreclaim:        79860 kB
KernelStack:        7568 kB
PageTables:        16748 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    50998036 kB
Committed_AS:    5565988 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             4352 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      404636 kB
DirectMap2M:    29857792 kB
DirectMap1G:     3145728 kB

/proc/meminfo shows some large numbers I do not know. Active(anon) could very well be the memory allocated to processes, but perhaps something else? In fact, the total memory used up by existing programs is not that high, as can be seen below.

$ ps -e -orss=,args= | sort -nr | head

1238144 palemoon
226172 /usr/bin/emacs --resume-layouts
95584 /usr/lib/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
63000 /usr/bin/python /usr/bin/mpsyt
45240 /usr/bin/python /usr/bin/mpsyt
38288 urxvtd -q -o -f
28212 pamac-tray
26832 nm-applet
26404 /usr/lib/systemd/systemd-journald
14072 /usr/lib/udisks2/udisksd

And finally, here is the output from vmstat.

$ vmstat -w

procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 0  0     18036224     10758424           28      2585984   34  217    50   252   78   44  44   3  53   0   0

It seems the swapping is tying up some memory, but I thought it is used up only as a convenience and released once the resource is required. That is not what I observe.
I am not using zram or any memory tricks. It seems it could help me in here, but I'm seeking assistance in finding out what is happening with this system. The previous system had only half of the memory but never acted up in this way.

Does anyone have a suggestion? Thank you for your time.
Edit: What tag should I use to make the name of programs and files stand out?

Last edited by Gede (2019-11-15 22:27:47)

Offline

#2 2019-11-14 20:49:08

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: [solved] What is using all the memory?

#!/bin/bash
# Display the top applications of memory usage
# http://www.cyberciti.biz/faq/linux-check-memory-usage/#comment-51021
 
while read command percent rss; do 
  if [[ "${command}" != "COMMAND" ]]; then 
    rss="$(bc <<< "scale=2;${rss}/1024")" 
  fi
  printf " %-26s%-8s%s\n" "${command}" "${percent}" "${rss} MB" \
  | sed 's/COMMAND/PROGRAM/' | sed 's/RSS MB/#MEM/'
done < <(ps -A --sort -rss -o comm,pmem,rss | head -n 20)

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#3 2019-11-14 21:36:53

Gede
Member
Registered: 2009-09-03
Posts: 28

Re: [solved] What is using all the memory?

Thank you for your interest, graysky. However, there is no process eating up that memory. Nothing even close.
ps, top, htop show no process taking more than 3% of the memory "gone missing", and the number of processes is small.

Offline

#4 2019-11-14 22:51:58

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: [solved] What is using all the memory?

32792156 - (18797024+2795056+10470024) = 730052 (total - (inactive + active + free) = 730MB, that's reasonable w/o further inspection)

10GB are free.
You've 2.7GB in active pages, but 17.9GB in inactive pages, mostly anon ie. memory that some process(es) has(have) claimed but not returned and also not touched in a long while. The moment you kill that process(es), the memory should be returned to the system.
I'd suggest to post a complete process list for an open blame game, but the more "exotic" the process, the more likely it's the cause.

Just the other day I had only 3GB of free memory running only a terminal!

That's however not the situation indicated above. DId you check that there were no other stale processes?

Offline

#5 2019-11-15 11:14:10

Gede
Member
Registered: 2009-09-03
Posts: 28

Re: [solved] What is using all the memory?

Here is an update then, from this mornig:

$ free

              total        used        free      shared  buff/cache   available
Mem:          32023       29464        1086         245        1472        1867
Swap:         33790       28785        5005

So less than 2GB free.

$ cat /proc/meminfo

MemTotal:       32792156 kB
MemFree:         1113788 kB
MemAvailable:    1915928 kB
Buffers:              88 kB
Cached:          1329876 kB
SwapCached:     29369672 kB
Active:          1081316 kB
Inactive:       30005660 kB
Active(anon):     560604 kB
Inactive(anon): 29451212 kB
Active(file):     520712 kB
Inactive(file):   554448 kB
Unevictable:      148364 kB
Mlocked:              48 kB
SwapTotal:      34601960 kB
SwapFree:        5125736 kB
Dirty:               304 kB
Writeback:             0 kB
AnonPages:        536416 kB
Mapped:           213044 kB
Shmem:            254808 kB
KReclaimable:     183508 kB
Slab:             253736 kB
SReclaimable:     183508 kB
SUnreclaim:        70228 kB
KernelStack:        5632 kB
PageTables:         7048 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    50998036 kB
Committed_AS:    2455872 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             4192 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      359580 kB
DirectMap2M:    25708544 kB
DirectMap1G:     7340032 kB

Inactive(anon) and SwapCached are once more high.

$ ps -e -orss,size,vsz,args | sort -nr
600092 1151852 2555668 palemoon
69388 63856 585996 /usr/lib/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
26276 30372 333880 clipit
24500 18908  47188 urxvtd -q -o -f
20180 19240  81292 /usr/lib/systemd/systemd-journald
14760 27624 266708 pamac-tray
14572 45592 404292 nm-applet
14464 44344 403920 /usr/lib/udisks2/udisksd
10756 26636 208564 volumeicon
 9244 69748 1933832 /usr/lib/polkit-1/polkitd --no-debug
 8772 35380 324844 /usr/bin/dunst
 7828 52720 484980 /usr/bin/NetworkManager --no-daemon
 6240  1468  38096 i3
 6008 26588 207428 xfce4-power-manager
 5832 61224 598716 conky -c /usr/share/conky/conky_maia
 5548 25588 186828 /sbin/init
 5148  2920  33112 i3bar --bar_id=bar-0 --socket=/run/user/1000/i3/ipc-socket.1137
 4992   836  13264 /usr/bin/wpa_supplicant -u
 4676 14124 128536 compton -b
 4244 34660 254740 /usr/lib/upowerd
 3744  1876   8504 bash
 3592  1064  31488 /usr/lib/systemd/systemd --user
 3140 26384 204492 /usr/lib/polkit-gnome/polkit-gnome-authentication-agent-1
 3100  1148  12768 ps -e -orss,size,vsz,args
 2908   732  25736 /usr/lib/systemd/systemd-logind
 2876  2876  82456 conky -c /usr/share/conky/conky1.10_shortcuts_maia
 2752  1988  39924 /usr/lib/systemd/systemd-udevd
 2688  1508  11408 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
 2628 25772 237868 /usr/lib/gvfsd
 2492 34464 316328 /usr/bin/ModemManager
 2468 17580 163732 /usr/lib/at-spi2-registryd --use-gnome-session
 1956   748  10648 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
 1828   780  21944 i3status
 1552 16884  24932 sort -nr
 1408   664  19592 /usr/lib/gconfd-2
 1172   736  10760 avahi-daemon: running [L-AG2-SD-GL.local]
 1160  1260   8436 /usr/bin/crond -n
  820   696  12276 /usr/lib/bluetooth/bluetoothd
  720   504  10404 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
   20 34284 310816 /usr/bin/lightdm
    4  9336  73524 (sd-pam)
    4   480  10504 avahi-daemon: chroot helper
    4   372   3964 xautolock -time 10 -locker blurlock
    4 25676 230900 /usr/lib/xfce4/xfconf/xfconfd
    4 17828 199268 lightdm --session-child 12 19
    4 17208 156936 /usr/lib/dconf-service
  RSS  SIZE    VSZ COMMAND
    0  8688  78064 /usr/bin/lvmetad -f
    0  6016   8120 /usr/sbin/haveged -w 1024 -v 1 --Foreground
    0 44528 380932 /usr/lib/gvfsd-fuse /run/user/1000/gvfs -f
    0 34024 306124 /usr/lib/at-spi-bus-launcher
    0 25748 238340 /usr/lib/accounts-daemon
    0     0      0 [writeback]
    0     0      0 [watchdogd]
    0     0      0 [usb-storage]
    0     0      0 [tpm_dev_wq]
    0     0      0 [scsi_tmf_3]
    0     0      0 [scsi_tmf_2]
    0     0      0 [scsi_tmf_1]
    0     0      0 [scsi_tmf_0]
    0     0      0 [scsi_eh_3]
    0     0      0 [scsi_eh_2]
    0     0      0 [scsi_eh_1]
    0     0      0 [scsi_eh_0]
    0     0      0 [rcu_tasks_kthre]
    0     0      0 [rcu_preempt]
    0     0      0 [rcu_par_gp]
    0     0      0 [rcu_gp]
    0     0      0 [rcuc/7]
    0     0      0 [rcuc/6]
    0     0      0 [rcuc/5]
    0     0      0 [rcuc/4]
    0     0      0 [rcuc/3]
    0     0      0 [rcuc/2]
    0     0      0 [rcuc/1]
    0     0      0 [rcuc/0]
    0     0      0 [rcub/0]
    0     0      0 [oom_reaper]
    0     0      0 [nvme-wq]
    0     0      0 [nvme-reset-wq]
    0     0      0 [nvme-delete-wq]
    0     0      0 [netns]
    0     0      0 [mm_percpu_wq]
    0     0      0 [migration/7]
    0     0      0 [migration/6]
    0     0      0 [migration/5]
    0     0      0 [migration/4]
    0     0      0 [migration/3]
    0     0      0 [migration/2]
    0     0      0 [migration/1]
    0     0      0 [migration/0]
    0     0      0 [kworker/u17:6-kcryptd/254:0]
    0     0      0 [kworker/u17:4-kcryptd/254:0]
    0     0      0 [kworker/u17:3-kcryptd/254:0]
    0     0      0 [kworker/u17:2-kcryptd/254:0]
    0     0      0 [kworker/u17:1-kcryptd/254:0]
    0     0      0 [kworker/u17:0-kcryptd/254:0]
    0     0      0 [kworker/u16:9-btrfs-endio-write]
    0     0      0 [kworker/u16:8-btrfs-endio-write]
    0     0      0 [kworker/u16:7-btrfs-endio-write]
    0     0      0 [kworker/u16:5-events_unbound]
    0     0      0 [kworker/u16:4-btrfs-endio-write]
    0     0      0 [kworker/u16:3-btrfs-endio-write]
    0     0      0 [kworker/u16:1]
    0     0      0 [kworker/u16:0-events_unbound]
    0     0      0 [kworker/7:1H-events_highpri]
    0     0      0 [kworker/7:1-events]
    0     0      0 [kworker/7:0H-kblockd]
    0     0      0 [kworker/7:0-events]
    0     0      0 [kworker/6:2]
    0     0      0 [kworker/6:1H-events_highpri]
    0     0      0 [kworker/6:0H-kblockd]
    0     0      0 [kworker/6:0-events]
    0     0      0 [kworker/5:2H]
    0     0      0 [kworker/5:1H-events_highpri]
    0     0      0 [kworker/5:1-cgroup_destroy]
    0     0      0 [kworker/5:0-mm_percpu_wq]
    0     0      0 [kworker/4:2-cgroup_destroy]
    0     0      0 [kworker/4:1H-events_highpri]
    0     0      0 [kworker/4:1-events_long]
    0     0      0 [kworker/4:0H-kblockd]
    0     0      0 [kworker/3:2-events]
    0     0      0 [kworker/3:1H-events_highpri]
    0     0      0 [kworker/3:1-events]
    0     0      0 [kworker/3:0H-kblockd]
    0     0      0 [kworker/2:2H-kblockd]
    0     0      0 [kworker/2:2]
    0     0      0 [kworker/2:1H-events_highpri]
    0     0      0 [kworker/2:0-mm_percpu_wq]
    0     0      0 [kworker/1:2-mm_percpu_wq]
    0     0      0 [kworker/1:1-mm_percpu_wq]
    0     0      0 [kworker/1:1H-kblockd]
    0     0      0 [kworker/1:0-kec_query]
    0     0      0 [kworker/1:0H-kblockd]
    0     0      0 [kworker/0:2-events]
    0     0      0 [kworker/0:1H-events_highpri]
    0     0      0 [kworker/0:0H-kblockd]
    0     0      0 [kworker/0:0-cgroup_destroy]
    0     0      0 [kthrotld]
    0     0      0 [kthreadd]
    0     0      0 [kswapd0]
    0     0      0 [kstrp]
    0     0      0 [ksoftirqd/7]
    0     0      0 [ksoftirqd/6]
    0     0      0 [ksoftirqd/5]
    0     0      0 [ksoftirqd/4]
    0     0      0 [ksoftirqd/3]
    0     0      0 [ksoftirqd/2]
    0     0      0 [ksoftirqd/1]
    0     0      0 [ksoftirqd/0]
    0     0      0 [ksmd]
    0     0      0 [kmemstick]
    0     0      0 [kintegrityd]
    0     0      0 [khungtaskd]
    0     0      0 [khugepaged]
    0     0      0 [kdmflush]
    0     0      0 [kdmflush]
    0     0      0 [kdevtmpfs]
    0     0      0 [kcryptd_io/254:]
    0     0      0 [kcryptd_io/254:]
    0     0      0 [kcryptd/254:1]
    0     0      0 [kcryptd/254:0]
    0     0      0 [kcompactd0]
    0     0      0 [kblockd]
    0     0      0 [kauditd]
    0     0      0 [jbd2/sdb1-8]
    0     0      0 [irq/67-smo8800]
    0     0      0 [irq/51-DELL0816]
    0     0      0 [irq/129-iwlwifi]
    0     0      0 [irq/127-mei_me]
    0     0      0 [irq/123-aerdrv]
    0     0      0 [irq/122-aerdrv]
    0     0      0 [ipv6_addrconf]
    0     0      0 [idle_inject/7]
    0     0      0 [idle_inject/6]
    0     0      0 [idle_inject/5]
    0     0      0 [idle_inject/4]
    0     0      0 [idle_inject/3]
    0     0      0 [idle_inject/2]
    0     0      0 [idle_inject/1]
    0     0      0 [idle_inject/0]
    0     0      0 [ext4-rsv-conver]
    0     0      0 [edac-poller]
    0     0      0 [dmcrypt_write/2]
    0     0      0 [dmcrypt_write/2]
    0     0      0 [devfreq_wq]
    0     0      0 [crypto]
    0     0      0 [cpuhp/7]
    0     0      0 [cpuhp/6]
    0     0      0 [cpuhp/5]
    0     0      0 [cpuhp/4]
    0     0      0 [cpuhp/3]
    0     0      0 [cpuhp/2]
    0     0      0 [cpuhp/1]
    0     0      0 [cpuhp/0]
    0     0      0 [charger_manager]
    0     0      0 [cfg80211]
    0     0      0 [btrfs-worker-hi]
    0     0      0 [btrfs-worker]
    0     0      0 [btrfs-transacti]
    0     0      0 [btrfs-submit]
    0     0      0 [btrfs-rmw]
    0     0      0 [btrfs-readahead]
    0     0      0 [btrfs-qgroup-re]
    0     0      0 [btrfs-freespace]
    0     0      0 [btrfs-flush_del]
    0     0      0 [btrfs-fixup]
    0     0      0 [btrfs-extent-re]
    0     0      0 [btrfs-endio-wri]
    0     0      0 [btrfs-endio-rep]
    0     0      0 [btrfs-endio-rai]
    0     0      0 [btrfs-endio-met]
    0     0      0 [btrfs-endio-met]
    0     0      0 [btrfs-endio]
    0     0      0 [btrfs-delayed-m]
    0     0      0 [btrfs-delalloc]
    0     0      0 [btrfs-cleaner]
    0     0      0 [btrfs-cache]
    0     0      0 [ata_sff]
    0     0      0 [acpi_thermal_pm]

The sum of each of these columns is about 932k, 2068k and 11692k. Still far from the value of used RAM value reported by free.
(In ps, vsz is reported in KB, according to the man page. I think it is likely the other values are too. Thus running processes account only for less than 12GB of memory in a worse-case scenario.)

$ vmstat -w

procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 0  0     29476224      1018408           88      1543032    0   76     4    80   86    1  12   0  88   0   0

So, it seems the problem is related to swap, somehow?

Update

# swapoff -a

Got me back my memory.

Last edited by Gede (2019-11-15 12:24:30)

Offline

#6 2019-11-15 12:32:20

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: [solved] What is using all the memory?

"swpd" is the amount of virtual memory, not "swap". When you run into this, start shooting processes one-by-one (starting w/ the least important ones, eg. palemoon or so, not systemd-logind or stuff like that)
Something™ is likely requesting HUUUUUUUUUGE pages and unable to return them to the system, so while the actual memory usage is low, large amounts of pot. memory are occupied.
I guess the browser is a good contender, but tehcnically it could be everything.

Edit: is your swap partition inside the LVM?

Last edited by seth (2019-11-15 12:33:29)

Offline

#7 2019-11-15 17:58:03

Gede
Member
Registered: 2009-09-03
Posts: 28

Re: [solved] What is using all the memory?

My swap partition is not inside the LVM. I'm not even using LVM. I have a primary partition for the swap and another for the root FS.
Supposedly, they are both encrypted but, while I can see that the filesystem is mounted through /dev/mapper/luks-..., I do not know how to inspect the swap and see if things are fine.

However, I created a swapfile and the system has been running just fine this way.

$ free

              total        used        free      shared  buff/cache   available
Mem:          32023        4135       27034         210         853       27251
Swap:         33791        6501       27290

This makes me want to reevaluate the swap partition to see if everything is fine. I think I will look into it and open a new question if needed. Thank you.

Offline

#8 2019-11-15 18:12:59

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [solved] What is using all the memory?

Please don't use the [CLOSED] tag in your thread title, it's reserved for threads that have been locked from further posts by the mods.

Thanks.


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#9 2019-11-15 22:32:01

Gede
Member
Registered: 2009-09-03
Posts: 28

Re: [solved] What is using all the memory?

Slithery wrote:

Please don't use the [CLOSED] tag in your thread title, it's reserved for threads that have been locked from further posts by the mods.

It the problem is not solved but I changed the title to avoid confusion in the forum. If I could use some other label that is better fitting, please let me know. Thanks.

Offline

Board footer

Powered by FluxBB