Arch Linux

regder

Member

Registered: 2013-06-16

Posts: 42

kswapd0 using 100% cpu on one core for no obvious reason

Running Arch on a Chromebox that functions as a NAS/Kodi box/server

Been happening for a while now, can't figure it out. Goes away for a bit if I reboot, but sure enough if I check it a few hours later it's back to the same.

top - 14:20:51 up 3 days, 13:46,  1 user,  load average: 1.52, 2.05, 2.41
Tasks: 116 total,   2 running, 112 sleeping,   2 stopped,   0 zombie
%Cpu0  :   0.0/99.3   99[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu1  :  13.1/38.1   51[|||||||||||||||||||||||||||||||||||||||||||||||||||                                                 ]
GiB Mem : 63.9/1.836    [                                                                                                    ]
GiB Swap: 16.2/2.000    [                                                                                                    ]

  PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND                                                                                                                
   32 root      20   0    0.0m   0.0m  99.3  0.0   1596:48 R kswapd0                                                                                                                
  508 kodi      20   0 2580.6m 436.2m  15.1 23.2 248:36.08 S kodi.bin                                                                    

free -m
              total        used        free      shared  buff/cache   available
Mem:           1880        1089         172          65         618         684
Swap:          2047         332        1715

cat /proc/meminfo
MemTotal:        1925396 kB
MemFree:          168544 kB
MemAvailable:     696880 kB
Buffers:          101768 kB
Cached:           473224 kB
SwapCached:        67160 kB
Active:          1234152 kB
Inactive:         386108 kB
Active(anon):     885032 kB
Inactive(anon):   227512 kB
Active(file):     349120 kB
Inactive(file):   158596 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        1756868 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:        986380 kB
Mapped:           129856 kB
Shmem:             67404 kB
Slab:              62244 kB
SReclaimable:      40816 kB
SUnreclaim:        21428 kB
KernelStack:        4272 kB
PageTables:        15240 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3059844 kB
Committed_AS:    2206856 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      368492 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:    712704 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      113692 kB
DirectMap2M:     1923072 kB
DirectMap1G:           0 kB

Fackamato

Member

Registered: 2006-03-31

Posts: 579

Re: kswapd0 using 100% cpu on one core for no obvious reason

Maybe a start: http://www.cyberciti.biz/faq/linux-whic … sing-swap/

regder

Member

Registered: 2013-06-16

Posts: 42

Re: kswapd0 using 100% cpu on one core for no obvious reason

So appears to be something wrong with memory management or something along those lines. The second I run out of ram kswapd0 goes to 100%, regardless of whether I have swap turned on or not.

If I clear system cache (google gave me this command, sync; echo 1 > /proc/sys/vm/drop_caches) the issue goes away until the next time it runs out of ram

Fackamato

Member

Registered: 2006-03-31

Posts: 579

Re: kswapd0 using 100% cpu on one core for no obvious reason

So appears to be something wrong with memory management or something along those lines. The second I run out of ram kswapd0 goes to 100%, regardless of whether I have swap turned on or not.

If I clear system cache (google gave me this command, sync; echo 1 > /proc/sys/vm/drop_caches) the issue goes away until the next time it runs out of ram

Stop running out of RAM.

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: kswapd0 using 100% cpu on one core for no obvious reason

This kodi.bin seems to have allocated more memory than you have physical RAM. If CPU usage goes down after stopping this, you simply need to get more RAM or accept swapping. Depending on how this process uses its memory, decreasing vm.swappiness may reduce swapping by some amount, at the cost of having less space for file caches.

regder

Member

Registered: 2013-06-16

Posts: 42

Re: kswapd0 using 100% cpu on one core for no obvious reason

Stop running out of RAM.

Not the solution, issue happens with swap on

This kodi.bin seems to have allocated more memory than you have physical RAM. If CPU usage goes down after stopping this, you simply need to get more RAM or accept swapping. Depending on how this process uses its memory, decreasing vm.swappiness may reduce swapping by some amount, at the cost of having less space for file caches.

It is possible that Kodi is the issue, but doubtful. The problem remains after I kill Kodi, only fix is to reboot or clear cache. Not sure if there is any way to simulate using a lot of ram to see if this problem is application independent.

This is a Chromebox so no way to add ram, I have no issue with swapping, but it is not using swap correctly. Once again, the second I run out of ram and it starts using swap (even if there is no swap enabled) the cpu % shoots to 100.

regder

Member

Registered: 2013-06-16

Posts: 42

Re: kswapd0 using 100% cpu on one core for no obvious reason

In case anyone is running into this issue, it appears to be a problem in the Kernel. Rolled back the Kernel a couple versions (I think to 4.0.1) without any other changes and my system is working normally again.

evenreven

Member

Registered: 2015-12-13

Posts: 1

Re: kswapd0 using 100% cpu on one core for no obvious reason

I have a Chromebox for Kodi/NAS/server purposes as well, only I'm using Ubuntu Server 15.10 (with kernel 4.3.2 installed from .deb files), and I have the exact same issue. A search led me here. Do you think it's related to this bug: https://bugs.launchpad.net/ubuntu/+sour … ug/1518457 ?

There's some testing going on over in that thread now, but it's Ubuntu-specific.

lkollenberger

Member

Registered: 2015-11-25

Posts: 6

Re: kswapd0 using 100% cpu on one core for no obvious reason

I have the exact same issue on my netbook, an Asus X201E (Celeron 847, 2GB RAM, 240GB Kingston SSD).
I remember it happening on older kernels, then it went away for a while (wasn't present on 4.2.5 for sure) and it came back now on 4.3.3.
I have a regular cron job (every 15 minutes) that runs

echo 3 | tee /proc/sys/vm/drop_caches

and it helps most times (even prevents this bug from occurring) but eventually it ends up making no effect, requiring a reboot to fix.
It seems to trigger with or without swap enabled, and with different vm.swappiness values (I tried 10 and the default, 60). Swap management looks completely normal otherwise.
I should note this is a recent Arch installation, I reinstalled about 10 days ago when I got my SSD, but as I said, used to happen in my previous installation on a regular HDD. Both times I've been using UEFI, if that matters.

On another note, my other laptop (Toshiba L355, Core 2 Duo T5800, 4GB RAM, 120GB Kingston SSD) is unaffected, swap works fine, same kernel, most of the same packages/DE installed. Windows 7 in a VM works perfectly too. This machine boots using BIOS though.

regder

Member

Registered: 2013-06-16

Posts: 42

Re: kswapd0 using 100% cpu on one core for no obvious reason

Can confirm problem is still there for me on 4.3.3 Was there on 4.2.5 as well though

george55

Member

Registered: 2011-08-10

Posts: 68

Re: kswapd0 using 100% cpu on one core for no obvious reason

Exact same issue here. Also using an Asus X201E. This problem started to occur after I updated my system yesterday. Updated kernel from 4.2.5-1 -> 4.3.3-2. Solved it temporarily by running the same command (echo 3 | tee /proc/sys/vm/drop_caches), but not sure how to fix this more permanently.

lucke

Member

From: Poland

Registered: 2004-11-30

Posts: 4,018

Re: kswapd0 using 100% cpu on one core for no obvious reason

I'd see if it happens on linux-ck.

george55

Member

Registered: 2011-08-10

Posts: 68

Re: kswapd0 using 100% cpu on one core for no obvious reason

Did someone figure out how to resolve this? By running "echo 3 | tee /proc/sys/vm/drop_caches" I am only able to suppress it for so long, and have to reboot after a while. Very frustrating issue I must say...

lkollenberger

Member

Registered: 2015-11-25

Posts: 6

Re: kswapd0 using 100% cpu on one core for no obvious reason

Did someone figure out how to resolve this? By running "echo 3 | tee /proc/sys/vm/drop_caches" I am only able to suppress it for so long, and have to reboot after a while. Very frustrating issue I must say...

I've managed to get around this by doing

sudo swapoff -a
echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo swapon -a
echo 3 | sudo tee /proc/sys/vm/drop_caches

which seems to quiet kswapd0 down for a while, but it comes back quickly afterwards.

I'd see if it happens on linux-ck.

I've installed linux-ck-sandybridge last night and so far so good... I'm having 22+ hours of uptime without kswapd raging. Could it be an issue with the Arch kernel?

lkollenberger

Member

Registered: 2015-11-25

Posts: 6

Re: kswapd0 using 100% cpu on one core for no obvious reason

Okay, I rebooted after a day and a half of uptime without issues (with linux-ck), and the issue came back after a while.
Rebooted again, now it's even worse: kswapd went crazy right after logging into GNOME.

george55

Member

Registered: 2011-08-10

Posts: 68

Re: kswapd0 using 100% cpu on one core for no obvious reason

Here things are similarly erratic. At times I can work for hours without a problem; at other times CPU spikes 5 times within the hour (in which case I reboot and it goes away again for a while...). There is a lot of discussion in the Ubuntu bug report (https://bugs.launchpad.net/ubuntu/+sour … ug/1518457) but no solution for Ubuntu yet.

lkollenberger

Member

Registered: 2015-11-25

Posts: 6

Re: kswapd0 using 100% cpu on one core for no obvious reason

Tried linux-lts, and seems fine.
I compiled the kernel from Linus' git, and it's still affected. Kswapd didn't flinch while compiling.
I'm gonna try doing a bisect, starting with 4.1 good, and 4.3 bad.

In the Ubuntu forums there's talk about udev being involved, if it is it surely depends on kernel version too, not just systemd.

veyen

Member

Registered: 2013-11-20

Posts: 5

Re: kswapd0 using 100% cpu on one core for no obvious reason

Add one more confirmation for ASUS 1015E (Intel Celeron 847). Problem appeared in kernel v4.3.3-2.

Looks like we are building consensus that this problem may affect ASUS notebooks in particular.

I will be downgrading the kernel to LTS later today.

post-factum

Member

From: /cz

Registered: 2008-09-12

Posts: 152

Website

Re: kswapd0 using 100% cpu on one core for no obvious reason

http://lkml.iu.edu//hypermail/linux/ker … 03462.html

Try what Kirill suggested here:

http://lkml.iu.edu//hypermail/linux/ker … 03564.html

---

uname == latest pf-kernel

lkollenberger

Member

Registered: 2015-11-25

Posts: 6

Re: kswapd0 using 100% cpu on one core for no obvious reason

http://lkml.iu.edu//hypermail/linux/ker … 03462.html

Try what Kirill suggested here:

http://lkml.iu.edu//hypermail/linux/ker … 03564.html

I just compiled the latest sources from git with that line added, and everything seems ok so far.

Add one more confirmation for ASUS 1015E (Intel Celeron 847). Problem appeared in kernel v4.3.3-2.

Looks like we are building consensus that this problem may affect ASUS notebooks in particular.

I will be downgrading the kernel to LTS later today.

Weird, I don't think it's limited to Asus hardware (or the Celeron 847), it's affecting multiple machines, even under Xen. The only correlation I could find is Intel CPUs newer than Sandy Bridge, still that's a very broad definition.

foosa

Member

Registered: 2016-02-02

Posts: 1

Re: kswapd0 using 100% cpu on one core for no obvious reason

I am having the same problem on my ASUS 1015E.  Is there another resolution besides a kernel downgrade?

Fackamato

Member

Registered: 2006-03-31

Posts: 579

Re: kswapd0 using 100% cpu on one core for no obvious reason

Y'all guys with kernel issues need to go upstream, not hang around in a distro forum.

veyen

Member

Registered: 2013-11-20

Posts: 5

Re: kswapd0 using 100% cpu on one core for no obvious reason

Upstream bug report here:

https://bugzilla.kernel.org/show_bug.cgi?id=110501

I wouldn't expect a fix too soon to be honest, but a fix will come eventually. Try the work around suggested by compiling your own kernel. It seems to help. Personally, lts kernel is much more convenient and works perfectly fine for me.

mattlyons0

Member

Registered: 2015-01-28

Posts: 2

Re: kswapd0 using 100% cpu on one core for no obvious reason

This has been occurring for me for over a month now and I don't even have a swap partition. Seems to happen when less than 2.5gb is available for cache. Instead of overwriting the cache ram with data it is invoking the OOM Process killer, kswapd0 is using an entire core, my ssd is being used a lot (hdd activity light is solid) and I usually can't even TTY. I have 12gb of ram on a Asus zenbook 303ln. Will try the LTS kernel soon.

veyen

Member

Registered: 2013-11-20

Posts: 5

Re: kswapd0 using 100% cpu on one core for no obvious reason

A workaround has been suggested by Tim Edwards in the upstream bug report:

Tim Edwards 2016-02-21 07:27:10 UTC

I've found a workaround that works well for me so far: create a file /etc/sysctl.d/60-workaround-kswapd-allcpu.conf with the following contents and reboot:
vm.min_free_kbytes=67584

The idea behind this workaround is a post by Kirill A. Shutemov on LKML (http://lkml.iu.edu//hypermail/linux/ker … 03564.html) and this Gallium OS bug report: https://github.com/GalliumOS/galliumos-distro/issues/52

Would be interesting to know if this helps others