You are not logged in.

#1 2017-12-29 17:56:28

bachtiar
Member
Registered: 2005-02-08
Posts: 64

Freeze on out of memory

I'm trying to understand what exactly is happening when my machine goes out of memory.

I have 32GB RAM, I am running kernel 4.14.8-1 and a few QEMU/KVM guests which consume approx. 20% of memory during normal operation (along with other programs). I have a 32GB encrypted swap partition on SSD. When I run a large 7zip compression (7za a -md=30), it starts slowly eating memory away, until all memory is exhaused. At that exact moment, the machine appears to freeze completely - keyboard, mouse and graphics stop responding, and any TCP connections to it hang in SYN phase.

My expectation was that the kernel would first use the swap, and if even that wouldn't be enough, it would then use a sophisticated mechanism (oom-killer) to select and kill a process when the memory is exhausted. This would mean that in the worst case, 7za would be killed and the machine would be usable again. However, that's obviously not the case as I can see moments before freezing that swap usage is still negligible. I was hoping to learn something from kernel messages, but it seems that get lost when the machine freezes too. Once that happens, the only way out is to power cycle.

Can somebody please explain to me this behaviour? Surely, I must be doing something wrong and it must possible to configure something to avoid "hard" freezes...

Thanks!

Offline

#2 2017-12-29 18:01:29

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: Freeze on out of memory

swap enabled?
tried unencrypted swap device?
did you mess with swappiness et al defaults?

Online

#3 2017-12-29 20:34:06

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Freeze on out of memory

Could be thrashing (whatever gets swapped out is immediately demanded to be put back in ram because the process needs it)

http://blog.scoutapp.com/articles/2015/ … -you-worry

Last edited by ugjka (2017-12-29 20:35:51)


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Offline

#4 2017-12-30 05:59:07

bachtiar
Member
Registered: 2005-02-08
Posts: 64

Re: Freeze on out of memory

Yes, swap is enabled.

Freezes/hangs happen with default sysctl settings, including vm.swappiness. Furthermore, as far as swappiness documentation goes, it should be safe even if set to 0 ("avoid swapping unless absolutely neccessary"). It would just postpone swapping until memory is full, and should still not killl any processes (or, even less, hang) until swap is exhausted.

I have not yet tried an unencrypted swap, as this is really a no-go for my case.

I don't think there is any thrashing at all - there is no disk activity, the swap usage (at the point of hang) is almost zero. As far as I can see it, even if there was thrashing, it would only slow down the system, not freeze it completely.

Additionally, even if there was no swap, I don't believe kernel is supposed to hang on out-of-memory - it should just use oom-killer and kill a process!

Or am I missing something?

Offline

#5 2017-12-30 08:20:44

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Freeze on out of memory

Swap is not a macigal thing that will swap out all your ram when you reach oom condition, not everything can be swapped out. The kernel will unliky swap out something that is currently executing.

Also setting swappiness to 0 does not do what you think it does https://unix.stackexchange.com/question … by-default

The vm.swappiness option is a modifier that changes the balance between swapping out file cache pages in favour of anonymous pages. The file cache is given an arbitrary priority value of 200 from which vm.swappiness modifier is deducted (file_prio=200-vm.swappiness). Anonymous pages, by default, start out with 60 (anon_prio=vm.swappiness). This means that, by default, the priority weights stand moderately in favour of anonymous pages (anon_prio=60, file_prio=200-60=140). The behaviour is defined in mm/vmscan.c in the kernel source tree.

Given a vm.swappiness of 100, the priorities would be equal (file_prio=200-100=100, anon_prio=100). This would make sense for an I/O heavy system if it is not wanted that pages from the file cache being evicted in favour of anonymous pages.

Conversely setting the vm.swappiness to 0 will prevent the kernel from evicting anonymous pages in favour of pages from the file cache. This might be useful if programs do most of their caching themselves, which might be the case with some databases. In desktop systems this might improve interactivity, but the downside is that I/O performance will likely take a hit.

Basically when you set swappiness to 0 and fill up your ram with programs all the file caches get pushed out and the kernel now needs to read everything from the filesystem directly which is slow



I roll with swappiness set to 100, here's my thread
https://bbs.archlinux.org/viewtopic.php?id=231265


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Offline

#6 2017-12-30 09:36:44

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: Freeze on out of memory

ugjka wrote:

Basically when you set swappiness to 0 and fill up your ram with programs all the file caches get pushed out and the kernel now needs to read everything from the filesystem directly which is slow

Worse: it's probably the #1 cause for thrashing (because it's prone to cause that and because of the misconception about its meaning)

Please ensure to try an unencrypted swap to isolate the problem and from the "no-go for my case" comment, lmg: you're also encrypting your RAM? (How *precisely* and yes: this implies to test the behavior w/ unencrypted RAM as well ;-)

Online

#7 2017-12-31 17:00:16

bachtiar
Member
Registered: 2005-02-08
Posts: 64

Re: Freeze on out of memory

I did some differential tests to narrow down the problem. I've found that:

  • Encrypted swap on LVM volume => machine freezes.

  • Encrypted swap on partition => everything OK (swap gets used as expected, machine does not freeze).

So it would appear that the problem is somehow related to LVM. I have used the same physical partition in both cases, so it's not disk-related either. During tests, i've left vm.swappiness to 60 (default).

Just as a side note - during one particular test, I've noticed that in htop, one "notch" appeared in swap bar just before machine froze. So the kernel actually started to use swap, but it only lasted for about 3 seconds.

Does anybody have any idea how to explain this? The problem should be easily reproducible.

Last edited by bachtiar (2017-12-31 17:01:13)

Offline

#8 2017-12-31 18:45:01

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: Freeze on out of memory

https://bbs.archlinux.org/viewtopic.php?id=231083
Last post suggests LVM as well.

As for why this would happen - no idea at hand (except that it's a bug) - I'd have to research that as well (and it's not easily reproducible if you don't have an LVM setup present ;-)

Online

#9 2018-02-18 18:23:13

catalin.hritcu
Member
From: Paris, France
Registered: 2014-04-27
Posts: 26
Website

Re: Freeze on out of memory

Does anyone have any news about an official fix for this problem? In particular, any fix that doesn't involve re-partitioning?
I'm also hitting it with a swap file on an encrypted LVM volume.

Last edited by catalin.hritcu (2018-02-18 18:59:40)

Offline

#10 2018-02-18 19:07:08

catalin.hritcu
Member
From: Paris, France
Registered: 2014-04-27
Posts: 26
Website

Re: Freeze on out of memory

One thing I noticed is that this problem goes away if I turn off swapping completely. Processes run out of memory when my RAM is full, but I have 16GB, so this is not frequent and still much better than my whole system dying.

Offline

#11 2018-02-19 00:13:57

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: Freeze on out of memory

catalin.hritcu wrote:

Does anyone have any news about an official fix for this problem? In particular, any fix that doesn't involve re-partitioning?

It would probably need to be reported upstream and the cause located before there can be an official fix.

Offline

#12 2018-02-19 08:39:33

seth
Member
Registered: 2012-09-03
Posts: 49,963

Re: Freeze on out of memory

This does not seem to be a recent regression, see eg, https://unix.stackexchange.com/question … heavy-load

Increasing vm.min_free_kbytes *might* prevent this.

Online

#13 2018-02-27 16:11:42

lopardo
Member
Registered: 2018-02-27
Posts: 1

Re: Freeze on out of memory

I've also hit this problem on 3 different computers (2 running Arch, 1 running Manjaro). All using LVM on LUKS as per the wiki (Manjaro used this setup too).

They all also have 8 GB of RAM and a large enough swap partition for hibernating.

Has anyone found any effective workarounds?

Offline

#14 2018-03-16 02:53:14

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 14
Website

Re: Freeze on out of memory

Same problem for quite a long time, this is getting very annoying as I work with ram intensive software.

Offline

#15 2021-06-22 20:06:00

bachtiar
Member
Registered: 2005-02-08
Posts: 64

Re: Freeze on out of memory

Does anybody know the status of this problem with more recent kernels (i.e. 5.4 or newer)?

Has it been fixed, or at least reported somewhere?

(I apologize for necrobumping but I believe it makes sense here because of the people involved.)

Offline

#16 2021-06-22 20:10:27

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 14
Website

Re: Freeze on out of memory

Haven't encountered it in a while

Offline

#17 2021-06-22 20:11:43

Zom
Member
From: Sweden
Registered: 2007-10-27
Posts: 430

Re: Freeze on out of memory

bachtiar wrote:

Does anybody know the status of this problem with more recent kernels (i.e. 5.4 or newer)?

Has it been fixed, or at least reported somewhere?

(I apologize for necrobumping but I believe it makes sense here because of the people involved.)

I only skimmed the thread because of the age, but I think since systemd 248 there's an oom service, systemd-oomd, available that'll help you avoid hard locks at least.

Offline

Board footer

Powered by FluxBB