You are not logged in.

#1 2019-12-06 16:18:11

kde35
Member
Registered: 2015-04-13
Posts: 71

systemd-coredump service failed

I got

systemctl status
● stone
    State: degraded
     Jobs: 0 queued
   Failed: 26 units

and it turns out the systemd-coredump@ services failed, that is

systemctl --failed
  UNIT                                 LOAD   ACTIVE SUB    DESCRIPTION                                                                    
● systemd-coredump@113-18197-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@114-18199-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@117-18208-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@118-18206-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@124-18234-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@125-18249-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@126-18251-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@127-18253-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@128-18256-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@145-21414-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@146-21416-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@147-21418-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@148-21421-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@149-21423-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@150-21424-0.service loaded failed failed Process Core Dump                                                              
● systemd-coredump@81-15404-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@82-15407-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@83-15408-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@88-15502-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@89-15504-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@91-15513-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@92-15511-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@93-15515-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@94-15519-0.service  loaded failed failed Process Core Dump                                                              
● systemd-coredump@95-15520-0.service  loaded failed failed Process Core Dump 
LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

26 loaded units listed.

If I try to find out more, for example, I get

systemctl status systemd-coredump@95-15520-0
● systemd-coredump@95-15520-0.service - Process Core Dump
     Loaded: loaded (/usr/lib/systemd/system/systemd-coredump@.service; static; vendor preset: disabled)
     Active: failed (Result: timeout) since Sun 2019-12-01 00:25:03 CET; 5 days ago
       Docs: man:systemd-coredump(8)
   Main PID: 15526 (code=killed, signal=TERM)

Dec 01 00:20:03 stone systemd[1]: Started Process Core Dump (PID 15520/UID 0).

what can I do with this?

Offline

#2 2019-12-06 22:04:09

xerxes_
Member
Registered: 2018-04-29
Posts: 665

Re: systemd-coredump service failed

Read only file system? - https://unix.stackexchange.com/question … ile-system

Print 'journalctl' output from boot when it happened. If you have it all the time, print output of 'systemctl --failed', also you may try 'systemctl isolate rescue.target' and see if there it happens.

Offline

#3 2019-12-07 09:05:48

kde35
Member
Registered: 2015-04-13
Posts: 71

Re: systemd-coredump service failed

No, there is no issue of writing rights.

An inspection of journalctl shows that apache segfaulted just before, creating multiple occurrence of core dump service, which led to

Dec 01 00:00:53 stone systemd[1]: Started Process Core Dump (PID 4760/UID 0).
Dec 01 00:00:53 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:53 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:54 stone audit[4768]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4768 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:54 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:54 stone systemd-coredump[4769]: Failed to send coredump fd: Connection reset by peer
Dec 01 00:00:55 stone audit[4770]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4770 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:56 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:56 stone audit[4772]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4772 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:57 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:57 stone systemd-coredump[4773]: Failed to send coredump fd: Connection refused
Dec 01 00:00:57 stone audit[4774]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4774 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:58 stone kernel: kauditd_printk_skb: 4 callbacks suppressed
Dec 01 00:00:58 stone kernel: audit: type=1701 audit(1575154857.995:961): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4774 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:58 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:58 stone audit[4778]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4778 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:59 stone kernel: audit: type=1701 audit(1575154858.998:962): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4778 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:59 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:01:00 stone audit[4780]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4780 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:01:00 stone kernel: audit: type=1701 audit(1575154860.001:963): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4780 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:01:00 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:01:00 stone audit[4782]: USER_AUTH pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_permit acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: USER_ACCT pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_unix acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: CRED_ACQ pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_permit acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: USER_START pid=4782 uid=0 auid=0 ses=27 msg='op=PAM:session_open grantors=pam_permit,pam_loginuid acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone fcron[4782]: Job '/usr/bin/run-parts /etc/cron.hourly' started for user systab (pid 4783)
Dec 01 00:01:00 stone kernel: audit: type=1100 audit(1575154860.058:964): pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_permit acct="root" exe=

What I don't understand is that
httpd segfaulted only twice before systemd-coredump service fails.  Basically there is one service created for each occurrence of httpd, which doesn't segfault...

Anyway, what can I do to get the status out of "degraded", apart from rebooting?

Offline

#4 2019-12-07 12:00:26

xerxes_
Member
Registered: 2018-04-29
Posts: 665

Re: systemd-coredump service failed

Maybe coredumps were too big or there is some bug in systemd-coredump that they triggered.

kde35 wrote:

Anyway, what can I do to get the status out of "degraded", apart from rebooting?

You can try commands:

systemctl restart systemd-coredump.socket
systemctl reset-failed

After 'reset-failed' may be service name to be failed-reseted or none to reset all failed services (details in 'man systemctl').

Don't worry too much, because degraded status means (according to 'man systemctl'): "The system is operational but one or more units failed."

Last edited by xerxes_ (2019-12-07 12:32:22)

Offline

#5 2019-12-07 14:39:02

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: systemd-coredump service failed

systemd-coredump has limits for the core size it will store / process see man 5 coredump.conf.
Would appear to be a bug in systemd-coredump.

Offline

#6 2019-12-07 14:52:04

kde35
Member
Registered: 2015-04-13
Posts: 71

Re: systemd-coredump service failed

Thank you all.  Anyway I had to reboot for a reason unrelated, so for the time the issue is gone.  I will try

systemctl restart systemd-coredump.socket
systemctl reset-failed

next time this happens.

Offline

Board footer

Powered by FluxBB