Erratic failure of desktop environment due to low memory

clupus · 2021-07-05 16:15:43

Hello,

I have few weeks ago found that my system gets unstable when running multiple programs. It manifested itself that after a bigger update (and a power outage) firefox was no longer able to start. After a certain time (between 1 second and a few minutes), the browser was killed. Some messages indicated a memory problem (see below).

The problem was not really deterministic, sometimes only the browser was killed, sometimes other processes as well (contact, plasmashell, node.js apps). A few times my complete plasma environment was killed and I had to religion. This is always nasty as the risk of lost data is high.

Before that update, I had occasionally a stall if I took too much memory into the browser. This however lasted over months (only suspending for the night but not shutting down the programs). So, it was dramatically better than after the update.

The update was around 5GB in total, so I cannot pinpoint the exact program that caused the issue as half the system was upgraded during it.

I could not really find a culprit for the issue. I have the intention it is memory-related. The behavior is similar to the OOM attacking some user-started programs. I eventually found some messages in the dmesg output but could not backtrace if these are just the recent killing or not (the machine was suspended for some time so the timestamps are no longer valid).
Normally, I see a stalling effect when the memory gets full and the system gets slower and slower until it only crawls. This is not happening here. The problem here is like a thunderbolt. I am working on something and, peng, no more browser where I was just working. I also started to keep a console with htop open on the second monitor to have an eye on potential problems. Nothing was visible there as well.

When running firefox from the command line as well, I get few messages. Here are some messages I got recently (various orders and numbers):

Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.

###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost


###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

out of memory: 0x0000000000001000 bytes requested
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.

[Parent 1248841, IPC I/O Parent] WARNING: Failed to launch tab subprocess: file /build/firefox/src/firefox-89.0.2/ipc/glue/GeckoChildProcessHost.cpp:761
[Parent 1248841, IPC I/O Parent] WARNING: Failed to launch tab subprocess: file /build/firefox/src/firefox-89.0.2/ipc/glue/GeckoChildProcessHost.cpp:761
[Parent 1248841, IPC I/O Parent] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /build/firefox/src/firefox-89.0.2/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc:19
[Parent 1248841, IPC I/O Parent] WARNING: FileDescriptorSet destroyed with unconsumed descriptors: file /build/firefox/src/firefox-89.0.2/ipc/chromium/src/chrome/common/file_descriptor_set_posix.cc:19

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

[Parent 1249481, IPC I/O Parent] WARNING: Failed to launch tab subprocess: file /build/firefox/src/firefox-89.0.2/ipc/glue/GeckoChildProcessHost.cpp:761
[Parent 1249481, Main Thread] WARNING: fallocate failed to set shm size: No space left on device: file /build/firefox/src/firefox-89.0.2/ipc/chromium/src/base/shared_memory_posix.cc:388
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.

(PS: I had to translate the string No space left on device as it was localized, so it might not be a 1:1 exact match)

Here you see, why I thought about a memory problem. I read the no space left on device message but I do not get on which device the memory missing. Neither the home nor temp folder is full. There are gigabytes of free space available on both. The mention of shm in the error message lets me think of RAM/swap shortage. But there are still a few hundred MB or even a few GB of RAM free and around 20GB of swap is unused. Also, it is strange that the system does not start swapping (getting slower) before OOM gets triggered. So I suspect it is not pure RAM missing. Is there another boundary for shared memory that might trigger this issue?

I thought the problem was the vast number of open tabs in firefox but I closed most of them and no enhancement. I even created a new profile that crashed with 3 or 4 tabs open researching for possible reasons for my problem.
I even tried to play with zram devices to boost the memory usage.

When I stop larger programs like akonadi, MS Teams, or biscuit, the problem seems to be reduced or not present at all. It is however not an option to shut down all other programs just to start my main browser. It is my work machine and I need to have mails and teams open for communication.

Can anyone give me a hint at what I could do in order to get to the bottom of this problem? At the moment I am a bit lost and I fear I do not know how to tackle this any further.

Thank you
Christian

twelveeighty · 2021-07-05 21:10:49

How much total RAM do you have on that box? The "fallocate" error looks like a legit out-of-memory error, which would also explain why closing memory intensive apps such as MS Teams would help.

However, swap should be tapped into in these types of scenarios. What are you using to check "free" and "swap" RAM?

Here's something to try: start without any apps open, as bare as you can get. Open a terminal and 'watch free -h' - this should refresh with memory usage every 2s. If you see the numbers change significantly over a couple of minutes' time, don't launch anything else, start looking into your systemd services, dbus launches, anything your DE runs, etc, because the problem would be there. If things remain still, open Firefox with just one tab. Keep monitoring and slowly adding tabs. If you have a crazy amount of tabs open, but memory usage is fine, it's not Firefox. Quit Firefox and see if your memory rebounds back (expect buff/cache to remain where it is).
If Firefox doesn't appear to be the culprit, start adding one app at a time (MS Teams, for example) and keep monitoring. This should narrow down the issue.

Lone_Wolf · 2021-07-06 11:34:20

Are you running stuff on /tmp , /run/user or any other tmpfs ?

df output helps to check that.

kokoko3k · 2021-07-06 12:48:15

Any artificially limited cgroup or running into a container?
Before starting the browser from a shell, from the same shell, issue and post the output of:

grep . /sys/fs/cgroup/$(cut  -d '/' -f 2- /proc/$$/cgroup)/memory.*max

clupus · 2021-07-06 15:32:56

twelveeighty wrote:

How much total RAM do you have on that box?

The box has 16GB of RAM (plus the 24GB of Swap attached).

twelveeighty wrote:

The "fallocate" error looks like a legit out-of-memory error, which would also explain why closing memory intensive apps such as MS Teams would help.
However, swap should be tapped into in these types of scenarios. What are you using to check "free" and "swap" RAM?

That was my impression as well. I use either free -m or htop to check memory consumption.

Currently (running Firefox, Teams, akonadi and kontact) I have an output of free -m of

               total        used        free      shared  buff/cache   available
Mem:           15768        8090        2196        1721        5481        5568
Swap:          23071         482       22589

You see quite some available RAM. During the last few minutes, no big change in this value was visible but nevertheless, I had a crash of plasmashell 5 min before.

twelveeighty wrote:

Here's something to try: start without any apps open, as bare as you can get. Open a terminal and 'watch free -h' - this should refresh with memory usage every 2s. If you see the numbers change significantly over a couple of minutes' time, don't launch anything else, start looking into your systemd services, dbus launches, anything your DE runs, etc, because the problem would be there. If things remain still, open Firefox with just one tab. Keep monitoring and slowly adding tabs. If you have a crazy amount of tabs open, but memory usage is fine, it's not Firefox. Quit Firefox and see if your memory rebounds back (expect buff/cache to remain where it is).
If Firefox doesn't appear to be the culprit, start adding one app at a time (MS Teams, for example) and keep monitoring. This should narrow down the issue.

Unfortunately, I do not have a clean test case. Sometimes it crashed almost immediately. Sometimes, it took more than a quarter of an hour. I will try to have a close look. However, I had not yet found a way to trigger it in a reproducible manner.

Lone_Wolf wrote:

Are you running stuff on /tmp , /run/user or any other tmpfs ?
df output helps to check that.

Yes, I am running on /tmp from time to time.

$ LANG=C df -h
df: /tmp/.mount_BiscuiAt3ZhF: Transport endpoint is not connected
df: /tmp/.mount_Biscuir5jXwF: Transport endpoint is not connected
Filesystem                   Size  Used Avail Use% Mounted on
dev                          7.7G     0  7.7G   0% /dev
run                          7.7G  2.6M  7.7G   1% /run
/dev/mapper/root              63G   52G  9.2G  85% /
tmpfs                         12G  152M   12G   2% /dev/shm
/dev/nvme0n1p2               340M   89M  230M  28% /boot
/dev/nvme0n1p1               256M   29M  228M  12% /boot/EFI
tmpfs                         24G   98M   24G   1% /tmp
/dev/mapper/var               58G   51G  5.3G  91% /var
/dev/mapper/system-aurbuild   20G  2.0G   17G  11% /var/lib/aurbuild
/dev/mapper/virtual          192G  120G   64G  66% /var/lib/virtual
/dev/mapper/docker            59G  6.3G   50G  12% /var/lib/docker
/dev/mapper/home             109G   94G   11G  91% /home
/dev/mapper/private          168G  136G   25G  85% /home/private
overlay                       59G  6.3G   50G  12% /var/lib/docker/overlay2/864ff03de1f3f3a4a91ed88c42ecdb9178ad140dfa7cb847486875b7e03264f4/merged
overlay                       59G  6.3G   50G  12% /var/lib/docker/overlay2/5edc8913e12a06826ec03471c4487e0924717463f2eb90702474655f8f9087ce/merged
overlay                       59G  6.3G   50G  12% /var/lib/docker/overlay2/5b26f1ff008716a1a94e4700c9388e0f0eae2da9fb796d3bf8af8d618c1793f1/merged
tmpfs                        1.6G  204K  1.6G   1% /run/user/1000

I have the feeling, that I had to increase /dev/shm a bit to allow firefox to start up.

kokoko3k wrote:

Any artificially limited cgroup or running into a container?
Before starting the browser from a shell, from the same shell, issue and post the output of:

Not that I am aware of. I considered restricting the memory footprint of Firefox and other apps like teams to avoid excessive memory usage I saw earlier. I did not succeed in getting it running. As I only tested on firefox, if something went wrong there only firefox should be affected. Especially not plasmashell should be killed as it happened a few minutes ago.

$ grep . /sys/fs/cgroup/$(cut  -d '/' -f 2- /proc/$$/cgroup)/memory.*max
/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-62812d17df30424f949531f71d42a32c.scope/memory.max:max
/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-62812d17df30424f949531f71d42a32c.scope/memory.swap.max:max

Thanks for sharing your thoughts.

twelveeighty · 2021-07-06 18:38:23

clupus wrote:

I had a crash of plasmashell 5 min before

I would focus on troubleshooting that first - Plasma has had memory/resource leaks in the past - can you reproduce the issue with a brand-new user, therefore with a fresh Plasma config, without any Plasma widgets, etc.?

Arch Linux

#1 2021-07-05 16:15:43

Erratic failure of desktop environment due to low memory

#2 2021-07-05 21:10:49

Re: Erratic failure of desktop environment due to low memory

#3 2021-07-06 11:34:20

Re: Erratic failure of desktop environment due to low memory

#4 2021-07-06 12:48:15

Re: Erratic failure of desktop environment due to low memory

#5 2021-07-06 15:32:56

Re: Erratic failure of desktop environment due to low memory

#6 2021-07-06 18:38:23

Re: Erratic failure of desktop environment due to low memory

Board footer