You are not logged in.

#1 2016-11-26 18:37:46

silver_remal
Member
Registered: 2016-11-26
Posts: 6

[RESOLVED]swap not completely used after the ram ran out

(tl;dr version: a computer running Arch Linux, Chrome and VMWare with 12GB RAM and 9GB swap will only use 4GB of the swap after all RAM are used, and seems not swapping pages out)

Hi everyone,

Having used linux (in my daily life) not more than three months, and met a strange problem.

My notepad have installed 12GB RAM, and have a 9GB swap on SSD. VMWare and Chrome are frequently used during worktime, and at least 2 vms will be opened. I found when after all my ram and 4gb of my swap are used, the system will respond very slow, and nearly do nothing according to my instruction.

I've checked RAM and SSD, and can make sure they are all in good state. Today the situation happened again and I found those log in `dmesg` by accident (see a detailed log at https://paste.ubuntu.com/23538896/, around the last few lines):

[112711.237268] alsa-sink-CX207 invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[112711.237269] alsa-sink-CX207 cpuset=/ mems_allowed=0
[112711.237274] CPU: 1 PID: 1405 Comm: alsa-sink-CX207 Tainted: G        W  O    4.8.10-1-ARCH #1
[112711.237275] Hardware name: LENOVO 20C6A0B6CD/20C6A0B6CD, BIOS J9ET9AWW (2.20 ) 07/29/2015
[112711.237278]  0000000000000286 000000003139694b ffff8802fd647880 ffffffff812fdee0
[112711.237281]  ffff8802fd647a40 ffff8802833a0000 ffff8802fd6478f0 ffffffff81206032
[112711.237283]  ffff8802fd6478a0 000000003139694b 00000000ffffffff 0000000000000000
[112711.237286] Call Trace:
[112711.237292]  [<ffffffff812fdee0>] dump_stack+0x63/0x83
[112711.237295]  [<ffffffff81206032>] dump_header+0x5c/0x1ea
[112711.237298]  [<ffffffff8117ca35>] oom_kill_process+0x265/0x410
[112711.237300]  [<ffffffff810864c7>] ? has_capability_noaudit+0x17/0x20
[112711.237302]  [<ffffffff8117cfb0>] out_of_memory+0x380/0x420
...
[112711.237357]  [<ffffffff815f9398>] page_fault+0x28/0x30
[112711.237358] Mem-Info:
[112711.237362] active_anon:2590659 inactive_anon:273122 isolated_anon:1415
                 active_file:6183 inactive_file:6089 isolated_file:192
                 unevictable:8 dirty:0 writeback:17 unstable:0
                 slab_reclaimable:23268 slab_unreclaimable:36236
                 mapped:1567129 shmem:1836071 pagetables:39784 bounce:0
                 free:25899 free_pcp:13 free_cma:0
[112711.237366] Node 0 active_anon:10362636kB inactive_anon:1092488kB active_file:24732kB inactive_file:24356kB unevictable:32kB isolated(anon):5660kB isolated(file):768kB mapped:6268516kB dirty:0kB writeback:68kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 235520kB anon_thp: 7344284kB writeback_tmp:0kB unstable:0kB pages_scanned:193897 all_unreclaimable? no
[112711.237367] Node 0 DMA free:296kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15852kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:15556kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[112711.237371] lowmem_reserve[]: 0 2893 11889 11889 11889
[112711.237374] Node 0 DMA32 free:52332kB min:16420kB low:20524kB high:24628kB active_anon:2683148kB inactive_anon:177748kB active_file:3276kB inactive_file:328kB unevictable:0kB writepending:0kB present:3049324kB managed:2973572kB mlocked:0kB slab_reclaimable:13928kB slab_unreclaimable:21128kB kernel_stack:2640kB pagetables:11396kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[112711.237378] lowmem_reserve[]: 0 0 8996 8996 8996
[112711.237381] Node 0 Normal free:50968kB min:51072kB low:63840kB high:76608kB active_anon:7679488kB inactive_anon:914964kB active_file:21456kB inactive_file:23976kB unevictable:32kB writepending:68kB present:9410560kB managed:9212060kB mlocked:32kB slab_reclaimable:79144kB slab_unreclaimable:108260kB kernel_stack:22960kB pagetables:147740kB bounce:0kB free_pcp:52kB local_pcp:0kB free_cma:0kB
[112711.237384] lowmem_reserve[]: 0 0 0 0 0
[112711.237387] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 0*128kB 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 296kB
[112711.237396] Node 0 DMA32: 1311*4kB (UM) 1202*8kB (UM) 768*16kB (UME) 381*32kB (UME) 127*64kB (UME) 34*128kB (UME) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52332kB
[112711.237405] Node 0 Normal: 12609*4kB (UMEH) 24*8kB (UMH) 1*16kB (H) 0*32kB 1*64kB (H) 3*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 51604kB
[112711.237415] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[112711.237416] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[112711.237417] 1852155 total pagecache pages
[112711.237418] 3471 pages in swap cache
[112711.237419] Swap cache stats: add 18930265, delete 18926794, find 7537789/9492209
[112711.237419] Free swap  = 5393080kB
[112711.237420] Total swap = 10485756kB
[112711.237421] 3118966 pages RAM
[112711.237421] 0 pages HighMem/MovableOnly
[112711.237422] 68595 pages reserved
[112711.237422] 0 pages hwpoisoned
[112711.237423] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name

After more try (actually, just open more vms, consume more RAM and wait until system is not responding), I think the system is stuck for having no memory available. That would explain why the "oom-killer"(don't know much about it, but think it is used to recycle some unused pages) is triggered, and the problem will usually happen when more vms are opened. The swap seems living in a 32bit world.

Some repeat experiment shows this problem happeneds when the swap is used about 40%(which means still 5gb swap is available):

$ free -h 
              total        used        free      shared  buff/cache   available
Mem:            11G        5.3G        1.4G        757M        5.0G        5.3G
Swap:            9G        3.7G        5.3G

And `lsblk` shows the swap is much larger than 4G:

$ lsblk   
NAME                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                        8:0    0 223.6G  0 disk 
├─sda4                     8:4    0    10G  0 part [SWAP]
├─sda2                     8:2    0  39.5G  0 part /
├─sda5                     8:5    0  49.8G  0 part /home
├─sda3                     8:3    0    16G  0 part /var
├─sda1                     8:1    0   512M  0 part /boot
└─sda6                     8:6    0 107.8G  0 part /dat

The swapiness is set to 60:

$ sudo sysctl vm           
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256   256     32      1
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

My kernel is:

$ uname -a
Linux remal_work_laptop 4.8.10-1-ARCH #1 SMP PREEMPT Mon Nov 21 11:55:43 CET 2016 x86_64 GNU/Linux

I've also checked my VMWare settings. The 2 vms both have 4GB of ram, using 2 processors. The "Reserved memory" in global settings are set to 8192 MB and I've allowed most virtual machine memory to be swapped.

(And if you need more thing about my configuration, just reply and I'll add it up)

Tried to google some relative problems, but most of them are using up all of their swap, instead of ram.

Since adding more RAM may not solve this problem, my question is, with the RAM is fully used, and swap still have a lot of free space, why kernel don't swap some pages out? Anything wrong with my configurements?

Thanks.

Last edited by silver_remal (2016-12-05 18:32:27)

Offline

#2 2016-11-26 19:20:25

bioshacker001
Member
Registered: 2016-11-26
Posts: 6

Re: [RESOLVED]swap not completely used after the ram ran out

I can think that a big part of this problem is that your virtual machine's claimed memory will almost NEVER be swapped, due to the nature of virtual machines (this would cause issues if an application in the virtual machine needs access to the RAM, and the virtualized kernel doesn't know about the swapping). I might be wrong on this, however, as I'm not an expert on virtual machines.
On another note:
You have a pretty heavy amount of system usage going on there. It requires some processing power to swap pages, each time, and either way, What processor do you have?

Last edited by bioshacker001 (2016-11-26 19:24:13)

Offline

#3 2016-11-26 20:24:00

silver_remal
Member
Registered: 2016-11-26
Posts: 6

Re: [RESOLVED]swap not completely used after the ram ran out

bioshacker001 wrote:

I can think that a big part of this problem is that your virtual machine's claimed memory will almost NEVER be swapped, due to the nature of virtual machines (this would cause issues if an application in the virtual machine needs access to the RAM, and the virtualized kernel doesn't know about the swapping). I might be wrong on this, however, as I'm not an expert on virtual machines.
On another note:
You have a pretty heavy amount of system usage going on there. It requires some processing power to swap pages, each time, and either way, What processor do you have?

Thanks for the reply.

Though it is possible, I think vmware is not likely to make such a mistake...I'll try with chrome, virtual box and some other software later, try to open as much as possible, and see if other swap space can be used. Considering my settings for vmware, it may not useful but I'll try to make sure.

CPU usage is not mentioned here because the problem happened too quick, so I have no chance to open a terminal to see cpu usage. My CPU is:

$ lscpu | grep name
Model name:            Intel(R) Core(TM) i7-4712MQ CPU @ 2.30GHz

Offline

#4 2016-11-26 20:42:31

bioshacker001
Member
Registered: 2016-11-26
Posts: 6

Re: [RESOLVED]swap not completely used after the ram ran out

silver_remal wrote:
bioshacker001 wrote:

I can think that a big part of this problem is that your virtual machine's claimed memory will almost NEVER be swapped, due to the nature of virtual machines (this would cause issues if an application in the virtual machine needs access to the RAM, and the virtualized kernel doesn't know about the swapping). I might be wrong on this, however, as I'm not an expert on virtual machines.
On another note:
You have a pretty heavy amount of system usage going on there. It requires some processing power to swap pages, each time, and either way, What processor do you have?

Thanks for the reply.

Though it is possible, I think vmware is not likely to make such a mistake...I'll try with chrome, virtual box and some other software later, try to open as much as possible, and see if other swap space can be used. Considering my settings for vmware, it may not useful but I'll try to make sure.

CPU usage is not mentioned here because the problem happened too quick, so I have no chance to open a terminal to see cpu usage. My CPU is:

$ lscpu | grep name
Model name:            Intel(R) Core(TM) i7-4712MQ CPU @ 2.30GHz

Alright, so your processor isn't likely to be a huge bottleneck here. I mean, it's not exactly the best to be using for multiple-system virtualization (laptop processor, which means poorer cooling and more throttling inherently by being in a laptop), but it would work.
A few questions:

Do you have any drives besides the SSD that are in use when you're using your VMs?
If you do, do you know if you have more than one SATA controller in your laptop? If you don't know, you probably don't.
Is anything disk-intensive going on when the issues start occuring?

[EDIT: Cleaned up the questions, so that it wasn't just bombarding you with incomprehensible inquiries]

Last edited by bioshacker001 (2016-11-26 20:49:51)

Offline

#5 2016-11-27 17:12:43

silver_remal
Member
Registered: 2016-11-26
Posts: 6

Re: [RESOLVED]swap not completely used after the ram ran out

bioshacker001 wrote:

Alright, so your processor isn't likely to be a huge bottleneck here. I mean, it's not exactly the best to be using for multiple-system virtualization (laptop processor, which means poorer cooling and more throttling inherently by being in a laptop), but it would work.
A few questions:

Do you have any drives besides the SSD that are in use when you're using your VMs?
If you do, do you know if you have more than one SATA controller in your laptop? If you don't know, you probably don't.
Is anything disk-intensive going on when the issues start occuring?

[EDIT: Cleaned up the questions, so that it wasn't just bombarding you with incomprehensible inquiries]

One of the vm is running on a USB 3.0 hard-drive, another is on the SSD. Only one SATA controller in my laptop.

Do you think the disk is a bottleneck which caused this problem? Umm, I don't know. Considering that the problem always happen when swap is used about 4GB, maybe it is not related to the disk. However I'll try to check it, by using some program which would take up lots of memory but do less disk access, and try to monitor the disk usage when the problem happened.

I've also tried to use an external fan for my laptop, but nothing changed except for the tempature...

[UPDATE] at Sun Nov 27 17:33:53 UTC 2016 :
I tried to use

sudo memtester 10G

while running chromium, and from

top

I can see both RAM and swap almost fully used. Seems a problem with VMWare. However, a futher analysis needs to be done, to make sure if it is a problem of VMWare itself, or it is a problem of kernel, which means, if VMWare DO requested the pages should not be swapped, or the kernel did the wrong thing. Besides, I don't know much about VT-x or something else, so could it be a problem with some "virtualization module"(maybe work for VT-x or nested pages or something like that, if exists) in the kernel?

Last edited by silver_remal (2016-11-27 17:38:08)

Offline

#6 2016-12-05 18:32:05

silver_remal
Member
Registered: 2016-11-26
Posts: 6

Re: [RESOLVED]swap not completely used after the ram ran out

So, finally, the problem seems resolved.

I think it's just my misunderstanding. I tried to connect from my raspberry pi when the computer seems stuck and the shell from SSH seems okay.

A possible solution is that the VMWare may have some "aggressive" memory usage strategy, which means, if it is trying to open a VM with 4GB memory, it will first try to allocate 4GB out from current ram, which would, certainly, make system trying to move some pages to swap. Though it's SSD, the procedure still seems slow(and maybe some bug still exists here, about disk IO or memory paging algorithm, which means I have no idea to research them at now). And, if pages being moved are just occupied by KDE or something related to my desktop environment, it would seems like the kernel is "stuck". However, if you try to wait for about 3 minutes while always trying to use Ctrl+Alt+F2 to go to another TTY, and then change back into the desktop environment, everything seems fun. Swap usage is normal now.

Thanks for everyone who had reviewed this thread. It might be a good idea to have a SSH open on your machine if you met some desktop related problems.

The problem will be marked as resolved later.

Last edited by silver_remal (2016-12-05 18:33:04)

Offline

Board footer

Powered by FluxBB