My computer freezes when it strarts swapping

Batou · 2017-09-08 18:44:07

Hi all,

I have an issue and it's troubling me deeply. I've had few instances in the past when my computer froze when some memory intensive task started. I've had Firefox end up using something like 15GB of RAM due to some memory leak. Whenever that happened, I've had to use Ctrl+Alt+F1 to get the shell so I can kill the process manually. I've also had Xorg go haywire and end up consuming lots of RAM and end up maxing out one core of the CPU.

I have 16GB of RAM, 16GB partition on an SSD, and a 4 core CPU. Yet when some process uses more than available RAM and hits the swap and maxes out only one core, everything freezes until the process stops or I manually kill it. Not even the mouse cursor can move.

Some stats:

 $ swapon --show
NAME      TYPE       SIZE USED PRIO
/dev/dm-3 partition 15.8G     -1

$ cat /proc/sys/vm/swappiness
1

$ cat /etc/sysctl.d/99-sysctl.conf
vm.swappiness=1
vm.vfs_cache_pressure=25

To see how your system handles this, make sure you have Imagemagick installed and run this command that creates a large blank image:

$ convert -size 100000x100000 xc:white white.gif

It completely freezes my machine until the process dies.

Is there anything I can do to make these runaway or memory intensive processes behave better? I'm afraid of losing data if I'm unable to recover in some of these instances...

Thanks!

olive · 2017-09-08 19:08:12

I think ImageMagick work with 16bits per pixels by default. A rapid computation shows that (before compression) your image would be about 19T (19 terabytes) large. By default, ImageMagick creates a temporary file in /tmp which is mounted in RAM. Even with your 16GB RAM + 16GB swap, you are far from that. You can redirect the temporary file to another folder by setting the TMPDIR environment variable. You can redirect it to a device with more than 19T of available space if you have one.

For me, it does not freeze the computer, but there is no way it can successfully create a so large file, so it will die. Note that hard disk, even SSD are much slower than RAM. You can indeed render a system unresponsive by constantly soliciting the hard disk (I think there is a ionice but don't know much about it).

Last edited by olive (2017-09-08 19:08:41)

loqs · 2017-09-08 19:52:51

Batou wrote:

Is there anything I can do to make these runaway or memory intensive processes behave better? I'm afraid of losing data if I'm unable to recover in some of these instances...

You could limit the memory made available to a particular process/user but when that memory limit is hit and an allocation fails it will depend on how the process handles it if data loss occurs.
Alternatively you can allow the allocations to occur until memory is exhausted then the OOM will kill a process and again data loss may occur.

Batou · 2017-09-09 02:44:36

olive wrote:

I think ImageMagick work with 16bits per pixels by default. A rapid computation shows that (before compression) your image would be about 19T (19 terabytes) large. By default, ImageMagick creates a temporary file in /tmp which is mounted in RAM. Even with your 16GB RAM + 16GB swap, you are far from that. You can redirect the temporary file to another folder by setting the TMPDIR environment variable. You can redirect it to a device with more than 19T of available space if you have one.
For me, it does not freeze the computer, but there is no way it can successfully create a so large file, so it will die. Note that hard disk, even SSD are much slower than RAM. You can indeed render a system unresponsive by constantly soliciting the hard disk (I think there is a ionice but don't know much about it).

Thank you so much olive! Can you tell my why your computer doesn't freeze? What kind of performance tweaks have you applied?

loqs wrote:

Batou wrote:
Is there anything I can do to make these runaway or memory intensive processes behave better? I'm afraid of losing data if I'm unable to recover in some of these instances...
You could limit the memory made available to a particular process/user but when that memory limit is hit and an allocation fails it will depend on how the process handles it if data loss occurs.
Alternatively you can allow the allocations to occur until memory is exhausted then the OOM will kill a process and again data loss may occur.

Thanks so much loqs! Your replies have helped me a lot in the past and I truly appreciate it when you take time and comment.

After much searching, I've seen a bunch of references to ulimit. Is that the way to go to limit memory usage? I saw one StackOverflow answer with something like:

ulimit -Sv 10000000

which is suppose to limit each process to 10GB or RAM. Not sure where to place that in though...

Also, any other performance tips?

Last edited by Batou (2017-09-09 02:55:08)

circleface · 2017-09-09 02:56:18

https://wiki.archlinux.org/index.php/Im … erformance has some tips on improving performance.

olive · 2017-09-09 09:48:48

Batou wrote:

Thank you so much olive! Can you tell my why your computer doesn't freeze? What kind of performance tweaks have you applied?

To be honest, I don't know exactly. I use an SSD exclusively, which is much faster than a hard disk. The freeze is usually due to the kernel waiting an I/O to complete; if there are I/O all the time , you experience a freeze sooner with a traditional hard disk than an SSD, because the SSD is able to answer almost instantly. Do you use a traditional hard disk too or have you exclusively an SSD? Note that I see a slowdown, but it is definitively not a "freeze". Note that my machine is not as powerful as yours, only 4Gb memory (+ 4Gb swap).

Last edited by olive (2017-09-09 09:49:34)

loqs · 2017-09-09 11:16:10

Batou wrote:

After much searching, I've seen a bunch of references to ulimit. Is that the way to go to limit memory usage? I saw one StackOverflow answer with something like:
ulimit -Sv 10000000
which is suppose to limit each process to 10GB or RAM. Not sure where to place that in though...

If you login through a display manager that starts a desktop environment then I am not sure where to place it so that all processes in that session are affected.
You might also look at /etc/security/limits.conf and man 5 systemd-system.conf in reference to DefaultLimitAS for /etc/systemd/user.conf.
I am also not sure if an activity such as imagemagick writing to a file on /tmp which is backed by tmpfs would be covered by memory related restrictions.

seth · 2017-09-09 16:38:46

cat /sys/block/sda/queue/scheduler # use your swap disk here
man ionice

Why is your swap partition in a device mapper? LVM or encrypted?

HiImTye · 2017-09-11 16:44:44

I tried your imagemagick command, but my system never froze, and conky reported that I never went above 8½ GiB of RAM, nor did I ever touch swap. it just ran for a bit, and then complained that it ran out of storage on /tmp

[ tye@t: ~ ]$ convert -size 100000x100000 xc:white white.gif                                                                                                  
convert: unable to write pixel cache `/tmp/magick-27855DhSjCZs5hIzM': No space left on device @ error/cache.c/WritePixelCachePixels/5958.

my swappiness is set to the default 60, and I haven't made any performance tweaks, aside from running the -zen kernel

Arch Linux

#1 2017-09-08 18:44:07

My computer freezes when it strarts swapping

#2 2017-09-08 19:08:12

Re: My computer freezes when it strarts swapping

#3 2017-09-08 19:52:51

Re: My computer freezes when it strarts swapping

#4 2017-09-09 02:44:36

Re: My computer freezes when it strarts swapping

#5 2017-09-09 02:56:18

Re: My computer freezes when it strarts swapping

#6 2017-09-09 09:48:48

Re: My computer freezes when it strarts swapping

#7 2017-09-09 11:16:10

Re: My computer freezes when it strarts swapping

#8 2017-09-09 16:38:46

Re: My computer freezes when it strarts swapping

#9 2017-09-11 16:44:44

Re: My computer freezes when it strarts swapping

Board footer