Arch Linux

TheMohawkNinja

Member

Registered: 2017-07-11

Posts: 47

coredumpctl doesn't see latest coredumps

Hello,

I am trying to debug why my daemon (which works perfectly fine when manually executed) immediately crashes when using systemctl start. When I run coredumpctl gdb it errors out stating

File "/var/lib/systemd/coredump/core.urxvt.1000.dd44f3c29bfb4854bd02885c500a7f0d.276825.1601511135000000000000.lz4" is not readable: No such file or directory

This is a truthful statement as I wiped the contents of /var/lib/systemd/coredump to make it easier to know which core dumps were actually relevant (didn't realize that using the i3wm keybinding to kill terminals caused a core dump each time, so my core dump list was huge), however that core dump IS NOT the latest core dump. Even when I run coredumpctl list and go all the way to the bottom, the last core dump it sees is from yesterday. However, when I run ls -lrt while at /var/lib/systemd/coredump, I see the core dumps I made today.

If I try and point coredumpctl to the PID of the latest core dump, it doesn't find it.

Any ideas?

loqs

Member

Registered: 2014-03-06

Posts: 19,043

Re: coredumpctl doesn't see latest coredumps

Corrupted journal causing coredumpctl to have no record of the new core dumps?

TheMohawkNinja

Member

Registered: 2017-07-11

Posts: 47

Re: coredumpctl doesn't see latest coredumps

Corrupted journal causing coredumpctl to have no record of the new core dumps?

Sounds like that could be the case. Would I have to just run a "pacman -S systemd" (or whatever the package name for systemd is called), or is there something smaller I can do to rebuild the journal?

Thank you,

merlock

Member

Registered: 2018-10-30

Posts: 262

Re: coredumpctl doesn't see latest coredumps

You can check with

journalctl --verify

---

Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

TheMohawkNinja

Member

Registered: 2017-07-11

Posts: 47

Re: coredumpctl doesn't see latest coredumps

Well, "journalctl --verify" does not seem to have found an issue:

Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac1764e8b5f1-6f718b9cfcd7479f.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ab73c5b4a4d3-56fce60f1c926602.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac1784011478-77ad21b34c62abca.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005acf023a51033-c637d7e014c26f9a.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005b08f1374a252-6dfa95be4dd21166.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005abb0adbc1e3a-3441c6478b872166.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005abefc7cb7846-2cda50f0d82df6b4.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ab9eb04dac92-c9df19b78a2c7bde.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ab5b2e2f8222-4fa9b90e5a9e8406.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005abb10aa03679-6d361186deffe9d6.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac00f5fa13af-36e1dc0a3dfa78f9.journal~                          
7fffef0: Unused data (entry_offset==0)                                                                                                
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@e21c7e3e75d6425fbad44ddf1d352f2a-000000000000046b-0005afbfc942a3a3.journal
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005afbfc942bca4-ce888827fcd00755.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005abb11ef11fbd-7195a82bb4d468a9.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ad1a709d2cc0-bfc46ede74d178a9.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005abd49658b97b-312daf48b84e7813.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac170a1f2d0d-82b5f3051d750a22.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000.journal                                                             
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@d05ced35750d4f2aa2aec1272ae18e34-0000000000000469-0005ad1a709d1580.journal
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac1777c0c63c-9c94ba13a7a3c3bc.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005ac17065169e7-ee9ed26da18a6f95.journal~                          
PASS: /var/log/journal/2cee06fc32e74b53ad289c0d1ca12366/user-1000@0005afac6e179e34-9259868ece961414.journal~

When I run "coredumpctl list" today, I still get a big list, and going to the end of the list shows the last couple of core dumps from yesterday. I still can't point "coredumpctl gdb" to the PIDs from any newer core dumps than those over 24 hours ago.

Given it only shows a couple of core dumps from yesterday, and the timestamps are within 24 hours, is it possible that there is some configuration set to only count any core dumps older than one day?

loqs

Member

Registered: 2014-03-06

Posts: 19,043

Re: coredumpctl doesn't see latest coredumps

Check the timestamp on a coredump file from today.  Does the journal contain a boot which starts before that timestamp and ends after it?

TheMohawkNinja

Member

Registered: 2017-07-11

Posts: 47

Re: coredumpctl doesn't see latest coredumps

Update:

So, I am seeing some core dumps from some earlier crashes today while developing the program that did not involve systemctl invocation.

It looks like even though the "systemctl start" error states "failed because a fatal signal was delivered causing the control process to dump core.", no core dumps are getting made.

It's probably some issue with the service file since the program will work fine if executed manually, so is it the case that issues caused by .service file errors don't actually dump the core?

loqs

Member

Registered: 2014-03-06

Posts: 19,043

Re: coredumpctl doesn't see latest coredumps

so is it the case that issues caused by .service file errors don't actually dump the core?

Not that I am aware of.

TheMohawkNinja

Member

Registered: 2017-07-11

Posts: 47

Re: coredumpctl doesn't see latest coredumps

Figured out the issue.

Systemctl runs under root privileges, so the coredumps could not be accessed when running "coredumpctl gdb" as a non-root user. Prepending "sudo" to the coredumpctl command pulled the ACTUAL latest coredump.