Arch Linux

tachy

Member

From: Denmark

Registered: 2008-07-24

Posts: 24

[SOLVED] Failed to open system journal: Invalid argument

I am running Arch Linux on a LVM-LUKS-LVM-btrfs setup (yes, I know... ).
Just want to share my little experience with systemd-journald today.

Symptoms & Investigation

When looking through my system logs using journalctl, I discovered that nothing had been logged after a certain date up until the most recent reboot.
Executing journalctl --list-boots only showed the most recent boot and old ones up to the said date. Everything in-between was missing.
After every reboot, the log entries from the previous session would disappear - but why?

Attempt 1: Checking integrity
I discovered some corruption with journalctl --verify and deleted the affected files, but the system's journal would still not persist after reboot.

Attempt 2: Checking configuration
I checked that /var/log/journal/ was present, and I had nothing but defaults in /etc/systemd/journald.conf (e.g. #Storage=auto), so the systemd-journald daemon should be logging to persistent storage.
For some obscure reason, the systemd-journald daemon had decided to create /run/log/journal/ and write its system.journal file there, which of course would disappear upon reboot?!

Clue no. 1: By looking through dmesg or journalctl, I found this error message:

Failed to open system journal: Invalid argument

Clue no. 2: When I listed the files in /var/log/journal/*/, the system.journal was only 240 bytes.

Solution

Upon deleting /var/log/journal/*/system.journal and rebooting, logging to persistent storage was working again.

Seeing a clean, no-corruptions-found output of journalctl --verify, I had failed to notice that system.journal was completely missing from the list of verified files.
The system.journal must have truncated in a way that made journalctl "blind" to it.

Further exploration

If anyone's interested, here's a base64 encode of the old truncated system.journal:

TFBLU0hIUkgAAAAAAgAAAAAAAAAAAAAADN8CmRioQpiVX2PAVUudUcXM+XN4nUiUlkqN9g/luM9m
E2LGmt5GVYBIi2p9HIU/DN8CmRioQpiVX2PAVUudUfAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAA

If you specifically run journalctl --verify --file path_to_file on the above file, you will get something like "Failed to open files: Invalid argument".
If you run journalctl --verify [-D path_to_dir], the command will only attempt to validate recognized journal files, which excludes the truncated one above.

Last edited by tachy (2015-06-15 14:35:35)