Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

frumble · 2014-08-27 19:48:03

Systemd 216 writes lots of GiB dump files from Firefox to /var/lib/systemd/coredump/

See my video. (Ogg Theora)

This is really, really broken! I just start and quit Firefox and systemd writes several hundred MiB to my poor SSD. It just wasted all my space on my /-partition!
I have downgraded to systemd 215 on which this never happened before!

Last edited by frumble (2014-08-28 12:50:06)

okubax · 2014-08-27 21:21:10

Think maybe your firefox installation has a problem. I have some of these coredumps also but all not greater than 50MB. You can change the journal dump behaviour by editing /etc/systemd/coredump.conf

stevenhoneyman · 2014-08-28 00:29:47

Nope, exactly the same problem here -- and my firefox is my own build. 6 in the few hours I've been using 216.

$ ls /var/lib/systemd/coredump/core.*
-rw-r-----+ 1 root root 402K 2014-08-27 21:36 /var/lib/systemd/coredump/core.camerabin.1000.9<SNIP>0.xz
-rw-r-----+ 1 root root  66M 2014-08-27 22:11 /var/lib/systemd/coredump/core.firefox.1000.9<SNIP>0.xz
-rw-r-----+ 1 root root 1.6M 2014-08-27 22:07 /var/lib/systemd/coredump/core.plugin-containe.1000.9<SNIP>0.xz
-rw-r-----+ 1 root root 1.6M 2014-08-27 22:08 /var/lib/systemd/coredump/core.plugin-containe.1000.9<SNIP>0.xz
-rw-r-----+ 1 root root 534K 2014-08-27 00:12 /var/lib/systemd/coredump/core.spacefm.1000.d<SNIP>0.xz
-rw-r-----+ 1 root root 545K 2014-08-27 00:03 /var/lib/systemd/coredump/core.spacefm.1000.d<SNIP>0.xz

(...I've cut those hash numbers or whatever they are out of the filenames - I don't know what's hidden in them and cba finding out.)

also I already have:

[Coredump]
Storage=none

It has also caused X to hang for a couple of mins each, after which somethign finally dies and I can kill systemd-coredump process thats eating half my CPU cores all at 100%

As soon as glibc2.20 and pacman 4.2 are released I'll be nuking systemd and going to nice clean initscripts of some variety.

Last edited by stevenhoneyman (2014-08-28 00:34:06)

Pse · 2014-08-28 00:43:27

Set kernel.core_pattern to core:

sudo systctl -w kernel.core_pattern=core

Please note that you should set this in /etc/sysctl.d for it to get set after a reboot.

stevenhoneyman · 2014-08-28 00:43:56

https://bugs.archlinux.org/task/41728

frumble · 2014-08-29 00:57:08

[Coredump]
Storage=none

doesn't prevents the dumps from getting generated, it just deletes them immediately (I can see this in Dolphin). How can I turn this completely off?!

@Pse: I don't understand what "sudo systctl -w kernel.core_pattern=core" should help, this just specifies the name of the dump files, am I right?

EDIT: Removing /usr/lib/sysctl.d/50-coredump.conf should do it, right?
But before I would like to take a look into the mess. ...But how? I just know journalctl. Are these dumps just memory dumps or is there any more information for a layman in them?

Last edited by frumble (2014-08-29 01:27:36)

branch · 2014-08-29 15:17:17

Use this if you never want a core file to be writtern:

# /etc/sysctl.d/60-coredump.conf
kernel.core_pattern = /dev/null

*edit*
Note that cutting systemd-coredump out of the loop in this manner will prevent the core dump event from being logged.

Last edited by branch (2014-08-29 15:32:12)

qwerty12 · 2014-08-29 15:45:03

frumble wrote:

EDIT: Removing /usr/lib/sysctl.d/50-coredump.conf should do it, right?

It's worth noting that generally for systemd stuff, you can disable a systemd-installed file by symlinking /dev/null to a file of the same name inside a folder of the same name inside /etc. This has the advantage of not having a systemd upgrade bringing back the files you removed. (Admittedly, pacman.conf's NoExtract option would also work but there's only so much you can easily maintain on one line. ). In this case:

sudo ln -s /dev/null /etc/sysctl.d/50-coredump.conf

That said, I'll be doing what branch said (thanks!), as disabling the 50-coredump.conf rule leaves kernel.core_pattern set to core.

Last edited by qwerty12 (2014-08-29 15:45:44)

frumble · 2014-08-29 16:58:18

OK, thank you! No dump files no more under /var/lib/systemd/coredump/.
But what about the warning in the wiki?

The factual accuracy of this article or section is disputed.
Reason: /etc/sysctl.d overwrite files of the same name, however 50-coredump.conf is no longer present in Arch; journal dump behaviour is regulated in /etc/systemd/coredump.conf [2]. Core dumps are by default not written to the journal, but to /var/lib/systemd/coredump.

50-coredump.conf is no longer present in Arch? It was present on my install. O.o

stevenhoneyman · 2014-08-29 17:51:32

Well, apparently freezing your system and filling your rootfs without warning is not a bug...

Great timing by the way Arch devs - I was literally minutes away from installing the lz4 enabled version (as requested for testing the effects on CPU usage)
"Shot yourselves in the foot" once again!

Scimmia · 2014-08-30 01:15:37

Shoot yourselves in the foot? A bug report was filed, it was determined to be expected behavior and the report was closed. How is that shooting yourself in the foot?

jasonwryan · 2014-08-30 04:59:08

stevenhoneyman wrote:

Well, apparently freezing your system and filling your rootfs without warning is not a bug...
Great timing by the way Arch devs - I was literally minutes away from installing the lz4 enabled version (as requested for testing the effects on CPU usage)
"Shot yourselves in the foot" once again!

This is either a spectacularly poorly executed attempt at humour or a troll. The bug report makes it pretty clear the issue isn't with the Arch developers or with systemd, but with (surprise) your browsers.

Reflect for a minute on the fact that however smart you think you are, you aren't the person giving up your spare time as an Arch developer, your name isn't in any of the upstream contributor lists and while posting snide comments here may inflate your own sense of superiority, it doesn't contribute anything to this community; ergo it isn't welcome.

metameta · 2014-08-30 06:54:39

Hi,

> The bug report makes it pretty clear the issue isn't with the Arch developers or with systemd, but with (surprise) your browsers.

No, the bug report makes it pretty clear that some softwares haves issue who created coredump, BUT ALSO, make it clear that something else has happened with a recent update of systemd, who make the log creation eating much more CPU (may be more evident on cheap computer)

If we look at the different bug report, we see that the solution given by Dev are "You could disable coredump creation", or "stopping to use software who create dump".

Fact is, since recent update, this coredump creation seems to eat much more CPU, and hang the system for a while. Dev's could argue that the first thing to do is to fix software issue (and this is a good idea), we could presume that those software issue happened all the time, long before we facing this CPU issue.

Closing bug report without trying to exclude a recent systemd issue (OR a new feature/behaviour with some side effect) is not a good answer.

Regards,

stevenhoneyman · 2014-08-30 10:22:22

Irrespective of whether this is "intended to happen" or not - there is still a bug with cpu usage & also the fact the post-install warning is not shown due to a mistake in the install script.

How is that shooting yourself in the foot?

If it hadn't been closed, I'd have made every effort to help diagnose and/or fix it.

fsckd · 2014-08-30 12:29:24

stevenhoneyman wrote:

Irrespective of whether this is "intended to happen" or not - there is still a bug with cpu usage & also the fact the post-install warning is not shown due to a mistake in the install script.
How is that shooting yourself in the foot?
If it hadn't been closed, I'd have made every effort to help diagnose and/or fix it.

You can still do that.

FWIW the post_upgrade message is in testing.

Scimmia · 2014-08-30 13:47:00

stevenhoneyman wrote:

Irrespective of whether this is "intended to happen" or not - there is still a bug with cpu usage & also the fact the post-install warning is not shown due to a mistake in the install script.
How is that shooting yourself in the foot?
If it hadn't been closed, I'd have made every effort to help diagnose and/or fix it.

The warning is now fixed and the cpu usage is a duplicate of https://bugs.archlinux.org/task/41721, so neither are reasons to keep your report open.

stevenhoneyman · 2014-09-01 17:33:54

Scimmia wrote:

The warning is now fixed and the cpu usage is a duplicate of https://bugs.archlinux.org/task/41721, so neither are reasons to keep your report open.

I don't mind it's closed, it makes no difference to me.
That being said, FYI, or in case anyone else is reading this and has the same problem - LZ4 does not fix the Arch systemd (so it's not a dupe of that "feature request").

I just recompiled systemd and cut out most of the stuff I didn't need (including coredump). Result: problem gone.

kokoko3k · 2014-09-01 18:53:12

Luckily i'm not getting core dumps, so i cannot test, but why you recompiled systemd to disable a feature instead of conguring it at runtime to do the same?

...what's wrong with this?

# cat /etc/sysctl.d/60-coredump.conf
kernel.core_pattern = core

stevenhoneyman · 2014-09-01 19:42:12

kokoko3k wrote:

Luckily i'm not getting core dumps, so i cannot test, but why you recompiled systemd to disable a feature instead of conguring it at runtime to do the same?
...what's wrong with this?
# cat /etc/sysctl.d/60-coredump.conf
kernel.core_pattern = core

I did it initially to test out lz4, and then I saw about 20 other things I could disable! It's only to keep me going until glibc2.20 is out then I'm starting again (getting rid of systemd in the process)

Arch Linux

#1 2014-08-27 19:48:03

Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#2 2014-08-27 21:21:10

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#3 2014-08-28 00:29:47

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#4 2014-08-28 00:43:27

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#5 2014-08-28 00:43:56

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#6 2014-08-29 00:57:08

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#7 2014-08-29 15:17:17

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#8 2014-08-29 15:45:03

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#9 2014-08-29 16:58:18

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#10 2014-08-29 17:51:32

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#11 2014-08-30 01:15:37

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#12 2014-08-30 04:59:08

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#13 2014-08-30 06:54:39

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#14 2014-08-30 10:22:22

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#15 2014-08-30 12:29:24

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#16 2014-08-30 13:47:00

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#17 2014-09-01 17:33:54

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#18 2014-09-01 18:53:12

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

#19 2014-09-01 19:42:12

Re: Systemd 216 writes lots of GiB dump files to /var/lib/systemd/coredump

Board footer