Out of RAM & no swap: system freezes rather than invoking OOM killer.

kindaro · 2018-01-23 22:35:45

When I run memory-intensive tasks, my system freezes, requiring hard reboot. If a process allocates memory slowly, I may notice, terminate it and breathe out, or be away for a minute and return to a frozen system. If a process allocates memory quickly, the system will freeze completely in a few seconds.

When I say freezes I mean it freezes completely. With a usual thrashing case, when a system starts swapping due to a runaway process, you may move your mouse, however slowly, and type things in a terminal, one word per minute, eventually terminating the offending process. In my case, I was not ever able to elicit a single screen redraw, not in X or Wayland, nor even in a virtual terminal.

One symptom is an unusually active hard drive. The drive activity (noticeable via slight but audible scratches as it moves its heads) does not stop until I force a hard poweroff. A hypothetic explanation I can offer for this fact is that, due to memory shortage, the system drops the cache for shared libraries and even for the program text of running (crawling?..) processes, thus requiring reading the same program text anew every time there is a context switch. My knowledge is insufficient to judge the sanity of this guess.

What offends me most about this issue is that OOM killer is never invoked.

A particularity of my system is that, against recommendations, I usually have no swap enabled. If I keep some swap around, the issue is not present (or, perhaps, just not so easily achieved). But I am reluctant to enabling swap, and I would rather have OOM deal with a runaway process that ate the memory.

I have two Arch Linux laptops: an HP ProBook and a Lenovo. I experience this issue with them both.

There is some prior art on this topic, specifically these two posts on Stack Exchange, but no resolution was reached.

Last edited by kindaro (2018-01-23 22:40:56)

kindaro · 2018-01-23 22:47:21

As this resource explains:

Without swap: The OOM killer is triggered more quickly as anonymous pages are locked into memory and cannot be reclaimed. We’re more likely to thrash on memory, but the time between thrashing and OOMing is reduced.

This is in harmony with my expectations, as well as with my experience of memory issues on Unix-like systems other than Arch.

Steef435 · 2018-01-24 11:50:31

I think hwertz's answer in your second link is in the right direction.

Even if you were to trigger the OOM killer, it's not like she will kill the process you want her to kill. Just did a little test on my system with a process rapidly requesting memory. The first time, my dwm session was killed (and thus also the faulty child process). The second time, my browser was killed.

kindaro · 2018-01-24 12:45:29

Either the OOM killer terminates some processes on my system, or I will have to terminate all of them. The math is in favor of the OOM killer.

The question is, how to make it work.

Did you have swap on at the time of your testing? Can you try the same test, but with swap off?

seth · 2018-01-24 12:46:59

Thrashing. The first link has the "correct" approach (vm.overcommit_memory=2) but rules it out high-handedly.

vm.min_free_kbytes is usually not the correct approach to it. Raising the value will simply cause the same problem (which you correctly identified as juggling libraries) faster.
There's but one exception: if you're using a raid or lvm. In that case the kernel can freeze because it wants to free memory by syncing to disk but can't because it doesn't have the memory left to do it.

Windows btw. handles this "smarter" by using - wait for it - a swapfile m(
And so should you - reyling on the OOM killer is, w/o further measures to control it, russian roulette.

Trilby · 2018-01-24 12:54:06

I don't use swap. I like russian roulette. I just stay outside the maximum effective range of the revolver in play! (I never use up my ram).

But swap is the right solution here. You can use a swap file can configure swapiness related settings so it is effectively never used except in these dire situations where the only alternative was thrashing/freezing. Certainly using a little swap in these circumstances is a better alternative.

Steef435 · 2018-01-24 17:05:30

kindaro wrote:

Either the OOM killer terminates some processes on my system, or I will have to terminate all of them. The math is in favor of the OOM killer.
The question is, how to make it work.
Did you have swap on at the time of your testing? Can you try the same test, but with swap off?

I'd agree with you, but I ended the second test early after it killed my browser. Then I wanted to start a terminal and nothing popped up because my terminal client server was apparently killed. Then three hours later I noticed the clock in my dwm panel had stopped ticking. One can only begin to imagine how many more background processes were killed without me directly noticing. It can take a very long time before the "misbehaving" process gets killed.

I don't use swap, but like Trilby, I stay far inside the boundaries. Which is pretty easy if you have 16GB RAM like I do. Wouldn't let my parents' computer run without swap though (4GB memory, fat DE).

It's unrealistic to ask for a system that continuously monitors memory usage (which is different from allocation on Linux) by software and consistently kills a misbehaving program before slowdowns and/or hangs occur. That would require quite a bit of processing. (and maybe a crystal ball)

redfish · 2020-09-01 03:27:53

EarlyOOM tool from Fedora works very well for this nasty problem:
https://fedoraproject.org/wiki/Changes/EnableEarlyoom

2ManyDogs · 2020-09-01 03:42:00

Closing this old topic.

Arch Linux

#1 2018-01-23 22:35:45

Out of RAM & no swap: system freezes rather than invoking OOM killer.

#2 2018-01-23 22:47:21

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#3 2018-01-24 11:50:31

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#4 2018-01-24 12:45:29

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#5 2018-01-24 12:46:59

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#6 2018-01-24 12:54:06

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#7 2018-01-24 17:05:30

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#8 2020-09-01 03:27:53

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

#9 2020-09-01 03:42:00

Re: Out of RAM & no swap: system freezes rather than invoking OOM killer.

Board footer