Systemd journal - a design problem apparent at system crash

mcloaked · 2013-09-18 14:49:02

In recent times when I have been testing kernels, particularly self-compiled or testing kernels, there have been occasions when the kernel has crashed, resulting in corrupted systemd journal files if the system fails without a clean shutdown of the systemd-journald service. This has led to a situation where the log files containing important lines that might lead to vital information about what could underlie the kernel crash or system freeze has then become unrecoverable.

After such a system freeze or crash doing

 # journalctl --verify

then shows errors in the journal files and using journalctl will then usually or often be unable to read the lines that were produced in the run-up to the system fail.

I have been unable to find any reference for a method to recover the information when this happens and the only method that I know to then get the systemd journal working sanely again is to stop the systemd-journald service, delete or move the /var/log/journal directory and create a new one - then reboot the system to create new journal files.

The route cause of this problem seems to be the use of binary files in the journal instead of ascii files.

Does anyone know of a method to recover the journal files once they have been corrupted due to an unclean shutdown? If not, then is there a way to get systemd to write ascii log files instead of binary files?

karol · 2013-09-18 14:58:02

Have a look at https://bugs.archlinux.org/task/32191

Have you tried redirecting the journal via syslog to the good ol' plaintext logs?

mcloaked · 2013-09-18 15:17:20

karol wrote:

Have a look at https://bugs.archlinux.org/task/32191
Have you tried redirecting the journal via syslog to the good ol' plaintext logs?

Thanks - it is a long-standing problem - and yes I have used syslog on one of my machines to generate a plaintext log file - but having the main system use binary files and having to work around to get a second level copy in ascii seems not the most efficient way forward! The better way would be for systemd to generate plaintext log files in the first place! However I expect that any plea in that direction upstream will likely not receive a positive response. So the problem actually still remains.

cookies · 2013-09-20 17:08:21

As far as I know you can tell journald not to store any binary log files and let syslog generate plaintext logs instead. Or you can just start/enable the syslog service while you do some testing and stop/disable it when you're done.

Zucca · 2013-10-24 11:41:54

cookies wrote:

As far as I know you can tell journald not to store any binary log files and let syslog generate plaintext logs instead. Or you can just start/enable the syslog service while you do some testing and stop/disable it when you're done.

You mean editing /etc/systemd/journald.conf and then installing some syslogger daemon?

I had a crash few days ago... That time journald really showed its weakness. The whole log file was corrupted.
If you ask me for one thing that I would modify in systemd it's the removal of binary log files. They tend to take more space than normal log files and compress poorly. Btrfs with compression on /var/log just screams for plain text log files.

From this day on I will set up journald to store logs only in memory and syslogger to store plain text log files to /var/log.

Arch Linux

#1 2013-09-18 14:49:02

Systemd journal - a design problem apparent at system crash

#2 2013-09-18 14:58:02

Re: Systemd journal - a design problem apparent at system crash

#3 2013-09-18 15:17:20

Re: Systemd journal - a design problem apparent at system crash

#4 2013-09-20 17:08:21

Re: Systemd journal - a design problem apparent at system crash

#5 2013-10-24 11:41:54

Re: Systemd journal - a design problem apparent at system crash

Board footer