You are not logged in.
Pages: 1
I got
systemctl status
● stone
State: degraded
Jobs: 0 queued
Failed: 26 units
and it turns out the systemd-coredump@ services failed, that is
systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-coredump@113-18197-0.service loaded failed failed Process Core Dump
● systemd-coredump@114-18199-0.service loaded failed failed Process Core Dump
● systemd-coredump@117-18208-0.service loaded failed failed Process Core Dump
● systemd-coredump@118-18206-0.service loaded failed failed Process Core Dump
● systemd-coredump@124-18234-0.service loaded failed failed Process Core Dump
● systemd-coredump@125-18249-0.service loaded failed failed Process Core Dump
● systemd-coredump@126-18251-0.service loaded failed failed Process Core Dump
● systemd-coredump@127-18253-0.service loaded failed failed Process Core Dump
● systemd-coredump@128-18256-0.service loaded failed failed Process Core Dump
● systemd-coredump@145-21414-0.service loaded failed failed Process Core Dump
● systemd-coredump@146-21416-0.service loaded failed failed Process Core Dump
● systemd-coredump@147-21418-0.service loaded failed failed Process Core Dump
● systemd-coredump@148-21421-0.service loaded failed failed Process Core Dump
● systemd-coredump@149-21423-0.service loaded failed failed Process Core Dump
● systemd-coredump@150-21424-0.service loaded failed failed Process Core Dump
● systemd-coredump@81-15404-0.service loaded failed failed Process Core Dump
● systemd-coredump@82-15407-0.service loaded failed failed Process Core Dump
● systemd-coredump@83-15408-0.service loaded failed failed Process Core Dump
● systemd-coredump@88-15502-0.service loaded failed failed Process Core Dump
● systemd-coredump@89-15504-0.service loaded failed failed Process Core Dump
● systemd-coredump@91-15513-0.service loaded failed failed Process Core Dump
● systemd-coredump@92-15511-0.service loaded failed failed Process Core Dump
● systemd-coredump@93-15515-0.service loaded failed failed Process Core Dump
● systemd-coredump@94-15519-0.service loaded failed failed Process Core Dump
● systemd-coredump@95-15520-0.service loaded failed failed Process Core Dump
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
26 loaded units listed.
If I try to find out more, for example, I get
systemctl status systemd-coredump@95-15520-0
● systemd-coredump@95-15520-0.service - Process Core Dump
Loaded: loaded (/usr/lib/systemd/system/systemd-coredump@.service; static; vendor preset: disabled)
Active: failed (Result: timeout) since Sun 2019-12-01 00:25:03 CET; 5 days ago
Docs: man:systemd-coredump(8)
Main PID: 15526 (code=killed, signal=TERM)
Dec 01 00:20:03 stone systemd[1]: Started Process Core Dump (PID 15520/UID 0).
what can I do with this?
Offline
Read only file system? - https://unix.stackexchange.com/question … ile-system
Print 'journalctl' output from boot when it happened. If you have it all the time, print output of 'systemctl --failed', also you may try 'systemctl isolate rescue.target' and see if there it happens.
Offline
No, there is no issue of writing rights.
An inspection of journalctl shows that apache segfaulted just before, creating multiple occurrence of core dump service, which led to
Dec 01 00:00:53 stone systemd[1]: Started Process Core Dump (PID 4760/UID 0).
Dec 01 00:00:53 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:53 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:54 stone audit[4768]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4768 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:54 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:54 stone systemd-coredump[4769]: Failed to send coredump fd: Connection reset by peer
Dec 01 00:00:55 stone audit[4770]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4770 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:56 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:56 stone audit[4772]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4772 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:57 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:57 stone systemd-coredump[4773]: Failed to send coredump fd: Connection refused
Dec 01 00:00:57 stone audit[4774]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4774 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:58 stone kernel: kauditd_printk_skb: 4 callbacks suppressed
Dec 01 00:00:58 stone kernel: audit: type=1701 audit(1575154857.995:961): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4774 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:58 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:00:58 stone audit[4778]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4778 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:59 stone kernel: audit: type=1701 audit(1575154858.998:962): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4778 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:00:59 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:01:00 stone audit[4780]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4780 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:01:00 stone kernel: audit: type=1701 audit(1575154860.001:963): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=4780 comm="httpd" exe="/usr/bin/httpd" sig=11 res=1
Dec 01 00:01:00 stone systemd[1]: systemd-coredump.socket: Too many incoming connections (16), dropping connection.
Dec 01 00:01:00 stone audit[4782]: USER_AUTH pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_permit acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: USER_ACCT pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_unix acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: CRED_ACQ pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_permit acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone audit[4782]: USER_START pid=4782 uid=0 auid=0 ses=27 msg='op=PAM:session_open grantors=pam_permit,pam_loginuid acct="root" exe="/usr/bin/fcron" hostname=? addr=? terminal=? res=success'
Dec 01 00:01:00 stone fcron[4782]: Job '/usr/bin/run-parts /etc/cron.hourly' started for user systab (pid 4783)
Dec 01 00:01:00 stone kernel: audit: type=1100 audit(1575154860.058:964): pid=4782 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_permit acct="root" exe=
What I don't understand is that
httpd segfaulted only twice before systemd-coredump service fails. Basically there is one service created for each occurrence of httpd, which doesn't segfault...
Anyway, what can I do to get the status out of "degraded", apart from rebooting?
Offline
Maybe coredumps were too big or there is some bug in systemd-coredump that they triggered.
Anyway, what can I do to get the status out of "degraded", apart from rebooting?
You can try commands:
systemctl restart systemd-coredump.socket
systemctl reset-failed
After 'reset-failed' may be service name to be failed-reseted or none to reset all failed services (details in 'man systemctl').
Don't worry too much, because degraded status means (according to 'man systemctl'): "The system is operational but one or more units failed."
Last edited by xerxes_ (2019-12-07 12:32:22)
Offline
systemd-coredump has limits for the core size it will store / process see man 5 coredump.conf.
Would appear to be a bug in systemd-coredump.
Offline
Thank you all. Anyway I had to reboot for a reason unrelated, so for the time the issue is gone. I will try
systemctl restart systemd-coredump.socket
systemctl reset-failed
next time this happens.
Offline
Pages: 1