You are not logged in.

#26 2012-10-26 21:38:33

dontbugme
Banned
Registered: 2011-11-04
Posts: 166
Website

Re: Corruption of system.journal

ball wrote:

Where do I have to search for shutdown errors? I can't find the right log...

On my embedded box, I'am using console (over serial line). It's done by this settings in /etc/systemd/system.conf:

LogLevel=debug
LogTarget=console

(BTW, I'am using systemd v195. And this is not Arch, but system Angstrom generated by OpenEmbedded.)

Offline

#27 2012-10-28 21:17:06

kenny96
Member
Registered: 2012-08-18
Posts: 17

Re: Corruption of system.journal

I use ext3 on /root and /home and ext2 on /boot so /var is in the same partition of root.
The strange thing is that after the shutdown hook this problem appears sometimes but not always... and i don't know why.
Before the introduction of shutdown hook there were lvm that aren't correctly umounted, but, as i show with one of my previous post, after that seems that all partitions are correctly umounted before poweroff.

Offline

#28 2012-10-28 21:43:21

brain0
Developer
From: Aachen - Germany
Registered: 2005-01-03
Posts: 1,382

Re: Corruption of system.journal

Good, this rules out a potential problem.

Unfortunately, I still have no idea what your problem is.

Offline

#29 2012-11-03 16:06:41

Bevan
Member
Registered: 2009-09-08
Posts: 45

Re: Corruption of system.journal

I'm experiencing the same problem. Now I did some tests that let me believe that the problem is not related to an unclean shutdown:

I ran

kill -USR2 `pidof systemd-journald`

to force journald to rotate its logs. After that I waited a bit until some log entries could be found in the new log file. Now it's getting interesting:

root # journalctl --verify --verify-key="my-key-is-secret"
PASS: /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system.journal                                                                   
=> No sealing yet, 10min 26s 594ms 119us of entries not sealed.

root # kill -USR2 `pidof systemd-journald`

root # journalctl --verify --verify-key="my-key-is-secret"
Tag failed verification at 567368███████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  48%
File corruption detected at /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system@a52a0139951b453ebb061bfed00d2d73-0000000000000516-0004cd9923c1b74c.journal:567368 (of 569344, 99%).
FAIL: /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system@a52a0139951b453ebb061bfed00d2d73-0000000000000516-0004cd9923c1b74c.journal (Bad message)

So if I tell journald to rotate its logs the old log file gets corrupted near the end. The same thing could happen when journald closes its files on shutdown.

Can anyone confirm these observations?

---
edit:
I just had to see that the logs not only get corrupted when they are rotated but also without any obvious reason after some time:

root # journalctl --verify --verify-key="my-key-is-secret"
Tag failed verification at 562280███████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  48%
File corruption detected at /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system.journal:562280 (of 569344, 98%).                    
FAIL: /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system.journal (Bad message)
Tag failed verification at 567368███████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  48%
File corruption detected at /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system@a52a0139951b453ebb061bfed00d2d73-0000000000000516-0004cd9923c1b74c.journal:567368 (of 569344, 99%).
FAIL: /var/log/journal/82d11945e2770cf18c6a25454a5b797f/system@a52a0139951b453ebb061bfed00d2d73-0000000000000516-0004cd9923c1b74c.journal (Bad message)

---
edit 2:
I just disabled forward sealing and can't reproduce the problem anymore. So if you don't use that feature this seems to be a different problem.

Last edited by Bevan (2012-11-03 16:54:00)

Offline

#30 2012-11-06 12:29:10

berny99
Member
From: Canary Islands (Spain)
Registered: 2010-10-05
Posts: 14

Re: Corruption of system.journal

Hi, I had the same problem, some times the journal went corrupted. What I noticed is that on my system it happen only with the last kernel 3.6. If I use for example the 3.4.18 lts this don't happen so may be is something connected with last kernel ?
All kernels I use are custom kernels but on my other laptop where I use only arch kernel the same problem went away installing linux-lts from the repository and using that version instead of the last one 3.6.x.
Now I'm using only the 3.4.18 lts and the problem never happened again, it is one week now.
Just my two cents.

Cheers,
berny

Offline

#31 2012-11-06 13:43:58

luz1
Member
From: Germany
Registered: 2011-12-14
Posts: 20

Re: Corruption of system.journal

@berny99
To me, the error occurred also since last kernel-update. So I can confirm your notice.

Offline

#32 2012-11-06 16:34:14

berny99
Member
From: Canary Islands (Spain)
Registered: 2010-10-05
Posts: 14

Re: Corruption of system.journal

After one week of no corruptions it happened again even with kernel 3.4 lts so I guess that it was not the problem, it still happen even if a lot less frequently.
Cheers.

Offline

#33 2012-11-07 08:02:46

luz1
Member
From: Germany
Registered: 2011-12-14
Posts: 20

Re: Corruption of system.journal

I noticed following:

If I do a "sudo halt" I get error messages like "Not all DM devices detached, 2 left." etc. and the system is haltet (no power off as before).
If I do a "systemctl poweroff" everything works fine.

Offline

#34 2012-11-19 18:38:39

dontbugme
Banned
Registered: 2011-11-04
Posts: 166
Website

Re: Corruption of system.journal

Hi all,

I'am thinking, that this issue is not related to unclean shutdown. As I'am working on embedded box, I switched off all disk (NAND) writes and I'am using volatile memory as a journal storage. Obviously, message during start disappeared. But  when I did "journalctl --verify", there were some bad messages. Strange thing is that this happened with 2 MB journal runtime size. After boot, there were ~800 kB used (journalctl --disk-usage).
When I switched runtime size to 50 MB, "journalctl --verify" was successful after reboot.
I'am using 3.3 kernel with some additions for our own embedded board, systemd 195, Angstrom distribution build by OpenEmbedded.

S.

Offline

#35 2013-01-28 03:09:10

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,661

Re: Corruption of system.journal

I have clean shutdowns followed by corruption.

I also now have unclean shutdowns followed by no corruption.


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

Board footer

Powered by FluxBB