systemd journal corruption

olive · 2013-01-14 09:35:24

About one out of 10 times I reboot, journalctl --verify mention a corrupted file. I have all Archlinux on a single ext4 partition (on a normal IDE disk). I do not have the shutdown hook, but can it really help me? Anyway why systemd is not able to properly close the journal before shutting down? The problem is that this is difficult to diagnose, it does not always appear.

anonymous_user · 2013-01-14 17:05:48

If including the shutdown hook does not help, then its likely an upstream problem:

https://bugs.archlinux.org/task/32191

cfr · 2013-01-15 01:43:39

I also see this with depressing regularity. I'm not sure whether it is a sign of a hardware problem or not. My laptop does sometimes suddenly shutdown in which case all bets are off but I see the corruption even after a normal shutdown. (reboot never seems to work - also not sure if this is software or hardware.) I'm not even sure I have to reboot or shutdown for it to corrupt a file.

I've just taken to deleting them as there seems no hope of figuring out the trigger. I don't even stop journalctld any longer. It just happens too often.

For whatever it is worth, I already have the shutdown hook.

Last edited by cfr (2013-01-15 01:44:42)

WonderWoofy · 2013-01-15 01:47:39

On my laptop, I just do what cfr does and delete the offending files (actually I just delete up to the newest bad file). But I have a home file server, and there I have made the journal volatile and then use syslog-ng. This seems to work just fine.

@cfr, I thought you were using syslog-ng as well? I remember when you first switched, you were using it (I think).

cfr · 2013-01-15 01:49:57

I'm using syslog-ng as well. However, it doesn't seem to actually capture very much. Maybe I should make the journal volatile again. I guess I'm reluctant to do that as I feel I need all the info I can get about what's going on right now (still).

WonderWoofy · 2013-01-15 01:53:36

Isn't there a separate option for the level of logging for syslog as there is for the journal in journald.conf?

anonymous_user · 2013-01-15 01:58:17

Are you referring to the MaxLevelSyslog option? By default its set to debug.

WonderWoofy · 2013-01-15 02:00:06

So it is....

cfr · 2013-01-15 02:48:40

I don't mean it doesn't log stuff. I should have said that it doesn't seem to log stuff which is equivalent to what the journal shows. And I'm not sure why. (Or whether it is just arranged differently in some way I don't understand.)

anonymous_user · 2013-01-15 02:50:09

syslog-ng produces several different logs. Did you check them all?

cfr · 2013-01-15 02:57:16

So for example, ntp.log seems not to have been updated since the middle of December. Yet the journal shows ntp connections from yesterday.

olive · 2013-01-15 08:03:02

In my case, I am pretty sure it is not a hardware problem. I can use the disk intensively (by compiling something that generate a lot of small file) and I do not see any problem; it is only the journal that show this symptom. The journal seems to work nevertheless, I delete the offending files from time to time (in fact, I often delete the whole journal, because it is easier).

This may be a separate post, but I see also various problems and instability that appear rarely. Once I had a hibernate that fails for no reason or a hang at shut down, but this appears rarely and I cannot reproduce it.

I nevertheless have seen something that could be called a bug. I had reformatted the swap partition and the swap could not be mounted, so far this was normal. But what was not was that the whole boot process hang completely; I have had to use a rescue disk to fix the problem. Not being able to mount an non root filesystem, and in particular the swap should not cause the boot to hang.

Last edited by olive (2013-01-15 08:03:36)

cfr · 2013-01-15 21:16:33

How long did you wait? If you did not wait at least 5-10 minutes it may not really have hung. systemd waits a *long* time before giving up on a device and continuing with the boot. But it should eventually continue. So, yeah, if you waited that long and it didn't obviously that would show a problem.

I'm not convinced the journal corruption issue is hardware in my case either. I suspect any hardware issue I have is unrelated. (Except where it actually causes an immediate shutdown - I expect problems then, obviously.) I don't see other file system issues. (Again, obviously, where it has shutdown uncleanly, I do, but not the rest of the time. Yet the corruption of the journal is constant.)

I find the journal the least satisfactory part of systemd. What bugs me is that I cannot look at any part of those files because they are not just text files. I'm not convinced binary formats are good for log files...

olive · 2013-01-15 22:52:25

@cfr 5-10 minutes ?! No, I didn't wait that long. I waited 20-30s. 5-10minutes is a very long time in computing. Is there an option to reduce the timeout to a more reasonable time. I would say, anything that does not happen in 20-30s is unlikely to happen at all.

For the journal, at least the output of journalctl is text. The corruption cannot be an hardware problem in my case. How could the disk so precisely corrupt only the journal files of systemctl?

Last edited by olive (2013-01-15 22:53:01)

cfr · 2013-01-15 22:56:45

I don't know of a way to reduce it. Sorry. Doesn't mean there isn't one - I mean I didn't look for one, really, but just solved the issue causing the hang.

I don't think it is hardware. In my case, I only wondered if it might be a side-effect of something - not hardware causing that file to be corrupted but a hardware issue causing something weird to get written to the journal. But I think it is more likely a bug in the journal feature of systemd.

Arch Linux

#1 2013-01-14 09:35:24

systemd journal corruption

#2 2013-01-14 17:05:48

Re: systemd journal corruption

#3 2013-01-15 01:43:39

Re: systemd journal corruption

#4 2013-01-15 01:47:39

Re: systemd journal corruption

#5 2013-01-15 01:49:57

Re: systemd journal corruption

#6 2013-01-15 01:53:36

Re: systemd journal corruption

#7 2013-01-15 01:58:17

Re: systemd journal corruption

#8 2013-01-15 02:00:06

Re: systemd journal corruption

#9 2013-01-15 02:48:40

Re: systemd journal corruption

#10 2013-01-15 02:50:09

Re: systemd journal corruption

#11 2013-01-15 02:57:16

Re: systemd journal corruption

#12 2013-01-15 08:03:02

Re: systemd journal corruption

#13 2013-01-15 21:16:33

Re: systemd journal corruption

#14 2013-01-15 22:52:25

Re: systemd journal corruption

#15 2013-01-15 22:56:45

Re: systemd journal corruption

Board footer