System freezes due to sudden disk IO

nullundnix · 2016-01-10 22:38:00

Hi, I currently often experience the following scenario:
During working on my laptop, the UI suddenly freezes and the LED indicating disk activity turns on and stays on.
The system is not completely frozen, but everything updates really slowly, maybe every few seconds. Since this happens about once or twice a day, I currently leave iotop running, and I see that almost ALL processes suddenly start to read huge ammounts of data from disk, even simple processes like xfce4-terminal, i3bar and nm-applet. Below you can see a screenshot taken shortly after the disk io ceased.

The screenshot shows the output of "iotop -oPa" on the left and the output of htop on the right. As iotop indicates, many processes had read multiple gigabytes from disk which caused the laptop to lag/freeze for 20 minutes. Also the load still appears very high. I use a fast SSD but still the disk IO caused the system to freeze almost completely.

I don't know what causes this behavior and am grateful for any tips which could help to fix this issue. I suspect it has something to do with me not having a swap file/partition and the system running out of memory, but I'm not sure. htop did not show a full memory in the screenshot, but maybe some processes were already killed due to insufficient memory at that time.
I don't think this should be a normal behavior if running out of memory, so I suspect I misconfigured something.
I use Linux version 4.3.2-1-ck on x86_64 Intel i5-3317U, my memory is 8GB, of which 4 GB are mounted as a tmpfs. Here are the relevant lines from /etc/fstab:

tmpfs		/tmp	tmpfs	nodev,nosuid,size=4G	0	0
tmpfs		/var/log	tmpfs	nodev,nosuid,size=50M	0	0

df -h outputs:

dev                                                3,8G       0  3,8G    0% /dev
run                                                3,8G    1,1M  3,8G    1% /run
/dev/sda6                                           40G     30G  7,9G   79% /
tmpfs                                              3,8G    150M  3,7G    4% /dev/shm
tmpfs                                              3,8G       0  3,8G    0% /sys/fs/cgroup
tmpfs                                              4,0G    707M  3,4G   18% /tmp
tmpfs                                               50M     36K   50M    1% /var/log
/dev/sda7                                           77G     70G  2,8G   97% /home
/home/jan/.Private                                  77G     70G  2,8G   97% /home/jan
tmpfs                                              770M     60K  770M    1% /run/user/1000

If you need more Information about my setup, just ask.
I really appreciate any help you can provide.

Last edited by nullundnix (2016-01-10 22:38:52)

lucke · 2016-01-10 23:24:22

Sounds like you run out of memory and the oom killer doesn't trigger (quickly enough). You could see if enabling Magic SysRq and invoking the oom killer manually via Alt+SysRq+f helps.

nullundnix · 2016-01-12 13:07:24

lucke wrote:

Sounds like you run out of memory and the oom killer doesn't trigger (quickly enough). You could see if enabling Magic SysRq and invoking the oom killer manually via Alt+SysRq+f helps.

Thanks for this tip, I will try that, I wasn't aware of Magic SysRq, I think this might help to get the laptop to respond again. Still I would like to know why this behavior occurs in the first place.

ewaller · 2016-01-12 15:29:52

Sounds like you are thrashing

When this occurs, what is the output of free ?
What is the output of cat /proc/swaps ?

nullundnix · 2016-01-12 16:24:06

ewaller wrote:

Sounds like you are thrashing
When this occurs, what is the output of free ?
What is the output of cat /proc/swaps ?

I just provoked this scenario just by opening some more tabs in chromium. The output of your commands was:

% free
              total        used        free      shared  buff/cache   available
Mem:     7877000     5859524       52040     1812308     1965436       95768
Swap:             0           0           0
% cat /proc/swaps
Filename				Type		Size	Used	Priority

As I mentioned earlier, I don't currently use a swap file/partition, so /proc/swaps does not show any swaps. Would using swap help in this situation?

ewaller · 2016-01-12 16:32:11

nullundnix wrote:

As I mentioned earlier, I don't currently use a swap file/partition, so /proc/swaps does not show any swaps. Would using swap help in this situation?

Missed that More coffee for ewaller.

Yes, I think it would be help to have a swap partition in this case. I don't like running while depending on swap, and my system almost never uses it unless I am doing something extreme -- but it does provide a way to degrade a bit more gracefully.

And yes, your output shows you were low on memory

lucke · 2016-01-12 16:41:34

You have 93 MB of memory available. Either an oom_killer kills something or programs fight for the available memory.

Adding swap would surely help (you could also use zswap or zram). Quite a lot of memory pages may be left untouched at all and reside in swap instead of occupying RAM.

nullundnix · 2016-01-12 16:48:18

Okay, I just created 2GB swap file and activated it, not it starts swapping when memory runs out and it at least does not cause the system to freeze.
I guess I will also shrink my /tmp tmpfs in order to not run out of memory all that often. Thanks for your help, I think I now understand what the problem was

lucke · 2016-01-12 16:58:56

Contents of tmpfs can also be swapped out. And tmpfs size only determines how much memory (RAM + swap) can be taken at maximum.

For reference, the output of "free -m" on my desktop right now, with zswap and with everything working fluidly:

              total        used        free      shared  buff/cache   available
Mem:           2019        1382         154          88         482         508
Swap:          3071         864        2207

As one can see, stuff in swap is almost half my RAM.

Last edited by lucke (2016-01-12 17:05:32)

Arch Linux

#1 2016-01-10 22:38:00

System freezes due to sudden disk IO

#2 2016-01-10 23:24:22

Re: System freezes due to sudden disk IO

#3 2016-01-12 13:07:24

Re: System freezes due to sudden disk IO

#4 2016-01-12 15:29:52

Re: System freezes due to sudden disk IO

#5 2016-01-12 16:24:06

Re: System freezes due to sudden disk IO

#6 2016-01-12 16:32:11

Re: System freezes due to sudden disk IO

#7 2016-01-12 16:41:34

Re: System freezes due to sudden disk IO

#8 2016-01-12 16:48:18

Re: System freezes due to sudden disk IO

#9 2016-01-12 16:58:56

Re: System freezes due to sudden disk IO

Board footer