You are not logged in.
I have a problem with journald, my system.journal file's is corrupted at every boots
dmesg:
[ 35.565257] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[ 35.587096] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[ 36.117750] EXT4-fs (dm-3): mounting ext3 file system using the ext4 subsystem
[ 36.171892] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[ 36.374606] systemd-journald[168]: Received SIGUSR1
[ 36.591060] systemd-journald[168]: File /var/log/journal/02903fbaca7aea3444c839ac000006dc/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 38.016227] ip_tables: (C) 2000-2006 Netfilter Core Team
Inside my /var/log/journal/02903fbaca7aea3444c839ac000006dc/ i have this:
-rw-r----- 1 root adm 4603904 11 ott 01.01 system.journal
-rw-r----- 1 root adm 6180864 7 ott 22.25 system@0004cb7de8496f40-f7a891d93261d74b.journal~
-rw-r----- 1 root adm 5591040 10 ott 22.27 system@0004cbba4a2197dd-389d22cc7f3650a9.journal~
I have already tried to delete folder with corrupted files, but, at the next boot the problem persist.
My systemd is updated, vers. 194-1
I already have made migration to pure systemd
Can you help me to indentify the origin of problem?
Offline
Faulty drive?
To know or not to know ...
... the questions remain forever.
Offline
mmm i don't think, the only corruption's that I can see are system.journal files. I think is unlikely that every time only these files are corrupted if is an hard disk problem. Plus this error started after migration to systemd (and relative journal system).
Is possible that journal service doesn't stop correctly when I shutdown arch linux? And is there a way to check this?
Offline
I've got this problem as well and it's a bit weird. The corrupt, renamed journal files are still accessed/readable when you do journalctl -a, there's nothing wrong with them.
Is possible that journal service doesn't stop correctly when I shutdown arch linux?
I think so because on every shutdown/reboot I have this at the end of the journal:
Oct 11 07:34:51 localhost systemd[1]: Unmounting /var...
Oct 11 07:34:51 localhost umount[2060]: umount: /var: target is busy.
Oct 11 07:34:51 localhost umount[2060]: (In some cases useful info about processes that use
Oct 11 07:34:51 localhost umount[2060]: the device is found by lsof(8) or fuser(1))
Oct 11 07:34:51 localhost systemd[1]: Unmounting /tmp...
Oct 11 07:34:51 localhost systemd[1]: Stopping Forward Password Requests to Wall Directory Watch.
Oct 11 07:34:51 localhost systemd[1]: Stopped Forward Password Requests to Wall Directory Watch.
Oct 11 07:34:51 localhost systemd[1]: var.mount mount process exited, code=exited status=32
Oct 11 07:34:51 localhost systemd[1]: Failed unmounting /var.
Oct 11 07:34:51 localhost systemd[1]: Unmounted /tmp.
Oct 11 07:34:51 localhost systemd[1]: Unmounted /home.
Oct 11 07:34:51 localhost systemd[1]: Starting Unmount All Filesystems.
Oct 11 07:34:51 localhost systemd[1]: Reached target Unmount All Filesystems.
Oct 11 07:34:51 localhost systemd[1]: Stopping Local File Systems (Pre).
Oct 11 07:34:51 localhost systemd[1]: Stopped target Local File Systems (Pre).
Oct 11 07:34:51 localhost systemd[1]: Stopping Remount Root and Kernel File Systems...
Oct 11 07:34:51 localhost systemd[1]: Stopped Remount Root and Kernel File Systems.
Oct 11 07:34:51 localhost systemd[1]: Starting Shutdown.
Oct 11 07:34:51 localhost systemd[1]: Reached target Shutdown.
Oct 11 07:34:51 localhost systemd[1]: Starting Final Step.
Oct 11 07:34:51 localhost systemd[1]: Reached target Final Step.
Oct 11 07:34:51 localhost systemd[1]: Starting Power-Off...
Oct 11 07:34:51 localhost systemd[1]: Shutting down.
Oct 11 07:34:52 localhost systemd-journal[127]: Journal stopped
----- Reboot -----
So systemd-journal looks like it's stopping after "Reached target Unmount All Filesystems" and after "Shutting down".
Seems odd to me.
Offline
Add the 'shutdown' hook to mkinitcpio.conf, and re-create your initramfs. Reboot twice and profit.
Offline
Add the 'shutdown' hook to mkinitcpio.conf, and re-create your initramfs. Reboot twice and profit.
Thanks for the suggestion brain0 but it made no difference for me. Perhaps kenny96 will have more luck.
Offline
Add the 'shutdown' hook to mkinitcpio.conf, and re-create your initramfs. Reboot twice and profit.
impressive.
Offline
brain0 wrote:Add the 'shutdown' hook to mkinitcpio.conf, and re-create your initramfs. Reboot twice and profit.
Thanks for the suggestion brain0 but it made no difference for me. Perhaps kenny96 will have more luck.
At least it should now cleanly umount your drives. Can you use 'halt' (instead of 'poweroff') to shut down? Then you will see if anything goes wrong during shutdown and if everything is properly umounted. Make a picture of your screen.
Offline
Without shutdown hook:
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /var.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/debug.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
[2633.536575] System halted.
With shutdown hook:
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /var.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/debug.
Unmounted /dev/hugepages.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Successfully changed into root pivot.
Detaching loop devices.
Unmounting all devices.
Disassembling stacked devices.
[183.950345] System halted.
Looks OK to me. The journal has the same entries as the one I posted previously despite the shutdown hook.
If you put "system.journal corrupted or uncleanly shut down" into google you'll see that a few people seem to have it in their dmesg output.
There was no problem with earlier versions of systemd but I've no idea which update caused it.
Thanks.
Offline
Okay, unclean umounting is not your problem, otherwise we would have seen errors here.
Offline
shutdown hook doesn't resolve the journal problem but using halt to shutdown the system I discover a problem with 2 unmounted devices that shutdown hook seems to solve. Here the code after applied shutdown hook:
Sending SIGTERM to remaining process...
Sending SIGKILL to remaining process...
Unmounting file systems.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/debug.
Unmounted /dev/hugepages.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Could not delete dm /dev/dm-0: Device or resource busy
Not all DM devices detached, 2 left.
Detaching DM devices.
Could not delete dm /dev/dm-0: Device or resource busy
Not all DM devices detached, 2 left.
Cannot finalize remainig file systems and devices, giving up.
Successfully changed into root pivot.
Detaching loop devices.
Unmounting all devices.
Disassembling stacked devices.
0 logical volume(s) in volume group "vgroup" now active
System halted.
Shutdown hook doesn't solve journal problem but something is changed. I have tried to reboot system 4 times after mkinitcpio change and the first 3 times I hadn't any corruption, fourth time the error came back
Offline
Any other suggestion? Is there a way to force journald to shutdown early?
Offline
Hmmm, I have the same problem but never figured it was anything until I read this post.
[ 1.933672] systemd-journald[238]: File /var/log/journal/a83c09bbec854118b18e67bb8cb4bc20/system.journal corrupted or uncleanly shut down, renaming and replacing.
On every boot, shutdown hook made no difference, one thing I do have though is a separate /var partition.
meyithi in /var/log/journal
$ cd a83c09bbec854118b18e67bb8cb4bc20/
meyithi in /var/log/journal/a83c09bbec854118b18e67bb8cb4bc20
$ ls
total 280978
drwxr-xr-x 2 root root 4440 Oct 23 23:56 .
drwxr-xr-x 3 root root 96 Oct 16 01:58 ..
-rw-r----- 1 root adm 4640768 Oct 16 14:15 system@0004cc2cf3c6e50f-6fab44e5e4d0dfd5.journal~
-rw-r----- 1 root adm 4575232 Oct 16 15:18 system@0004cc2dd3200d56-065c082bce334494.journal~
-rw-r----- 1 root adm 4575232 Oct 17 01:05 system@0004cc360691a43e-4af30f0ee644066b.journal~
-rw-r----- 1 root adm 4575232 Oct 17 01:05 system@0004cc360897b033-5cd763c2028b422c.journal~
-rw-r----- 1 root adm 4591616 Oct 17 01:11 system@0004cc361c49b253-179a1cbdf92aebb4.journal~
-rw-r----- 1 root adm 4579328 Oct 17 01:12 system@0004cc361e94cac7-b03349cdb9e42ce5.journal~
-rw-r----- 1 root adm 4743168 Oct 17 01:54 system@0004cc36b81389a3-3aff8277dcdbd6ee.journal~
-rw-r----- 1 root adm 4583424 Oct 17 01:59 system@0004cc36c9b7067d-7ae77546e4804d17.journal~
-rw-r----- 1 root adm 4616192 Oct 17 14:36 system@0004cc415d5e1904-18e3204857f1241b.journal~
-rw-r----- 1 root adm 4608000 Oct 17 14:52 system@0004cc4194ed0455-d92b7569f79fdc6e.journal~
-rw-r----- 1 root adm 4595712 Oct 17 15:01 system@0004cc41b3f95e8f-8756c32b82220670.journal~
-rw-r----- 1 root adm 4640768 Oct 17 15:16 system@0004cc41e98301d5-dad597ff49ccea96.journal~
-rw-r----- 1 root adm 4579328 Oct 17 15:35 system@0004cc422f9a1771-16cd191c2aa59da7.journal~
-rw-r----- 1 root adm 4857856 Oct 17 21:37 system@0004cc473cb583bf-8cc20efbdb981ba1.journal~
-rw-r----- 1 root adm 4808704 Oct 17 23:31 system@0004cc48d47e003a-425f724e0ad15cbe.journal~
-rw-r----- 1 root adm 4595712 Oct 17 23:43 system@0004cc490098940a-d9085e2c14b11dc2.journal~
-rw-r----- 1 root adm 4612096 Oct 17 23:47 system@0004cc490efe701f-1194f03fa4c8fd55.journal~
-rw-r----- 1 root adm 4644864 Oct 17 23:49 system@0004cc4915e6fb78-4cde56951f5da9ce.journal~
-rw-r----- 1 root adm 4698112 Oct 18 00:00 system@0004cc493be242f4-3b0e08d0a22303ea.journal~
-rw-r----- 1 root adm 4632576 Oct 18 00:07 system@0004cc495702d3c5-e6c85b54ba38d682.journal~
-rw-r----- 1 root adm 4677632 Oct 18 01:12 system@0004cc4a3ee0bb16-bf63b98591f9ef3c.journal~
-rw-r----- 1 root adm 4665344 Oct 18 01:52 system@0004cc4acbbab89f-ee1aeacb17c527da.journal~
-rw-r----- 1 root adm 4788224 Oct 18 12:32 system@0004cc53bce4f189-8e3946c15d64431a.journal~
-rw-r----- 1 root adm 5095424 Oct 19 12:15 system@0004cc679f8d8c6d-227ebac1b13b3069.journal~
-rw-r----- 1 root adm 4665344 Oct 19 12:37 system@0004cc67ecccdd1b-3d1d76473076c03f.journal~
-rw-r----- 1 root adm 4714496 Oct 19 13:42 system@0004cc68d57f0d58-7f52eb4c04feee99.journal~
-rw-r----- 1 root adm 4657152 Oct 19 13:43 system@0004cc68da047fcc-18f92cfee75a59bb.journal~
-rw-r----- 1 root adm 5009408 Oct 19 15:46 system@0004cc6a8ffe163a-b0e97d7167d41d7f.journal~
-rw-r----- 1 root adm 5320704 Oct 20 14:27 system@0004cc7d94b1d5ae-d3f710c560ba9f72.journal~
-rw-r----- 1 root adm 4730880 Oct 20 15:49 system@0004cc7eb91df3eb-b849853c21c5a955.journal~
-rw-r----- 1 root adm 4841472 Oct 20 22:49 system@0004cc8497530704-d7e45a7ce4da7f9a.journal~
-rw-r----- 1 root adm 4620288 Oct 20 22:56 system@0004cc84b099c64e-8f61104d671a0cdc.journal~
-rw-r----- 1 root adm 4931584 Oct 21 05:13 system@0004cc89f4c01d9a-35ad7b44b14df133.journal~
-rw-r----- 1 root adm 4595712 Oct 21 05:16 system@0004cc89ffd247b5-b6986c8e9785dcb8.journal~
-rw-r----- 1 root adm 4644864 Oct 21 14:49 system@0004cc9202c508fc-f4a46b3ca694ef35.journal~
-rw-r----- 1 root adm 5242880 Oct 21 23:04 system@0004cc98ea098fb3-aaec097ec5f6e1e5.journal~
-rw-r----- 1 root adm 4706304 Oct 22 01:12 system@0004cc9ab630bc20-a27e631c05586e29.journal~
-rw-r----- 1 root adm 4579328 Oct 22 01:17 system@0004cc9ac802f879-51803319a05abf81.journal~
-rw-r----- 1 root adm 4648960 Oct 22 03:11 system@0004cc9c5e35b9a6-6d4e8f06acfdabef.journal~
-rw-r----- 1 root adm 4620288 Oct 22 03:27 system@0004cc9c9a0dd370-1dacb693dbdbb60d.journal~
-rw-r----- 1 root adm 4837376 Oct 22 06:31 system@0004cc9f296681c7-eecd304082538d6b.journal~
-rw-r----- 1 root adm 5128192 Oct 22 14:50 system@0004cca621d6f73d-ecb98df6fb6c0f66.journal~
-rw-r----- 1 root adm 5115904 Oct 22 17:42 system@0004cca88a35a510-0a2470da0f249ee4.journal~
-rw-r----- 1 root adm 4669440 Oct 22 21:28 system@0004ccabb3e21df0-2262dfda170e6a5b.journal~
-rw-r----- 1 root adm 4583424 Oct 22 21:30 system@0004ccabb7fff40f-ef75fcb7f0892c16.journal~
-rw-r----- 1 root adm 4603904 Oct 22 22:08 system@0004ccac40062456-30c9ac91deb14253.journal~
-rw-r----- 1 root adm 4751360 Oct 22 22:46 system@0004ccacca48e715-c9dc3d637e09ceee.journal~
-rw-r----- 1 root adm 5951488 Oct 23 21:01 system@0004ccbf6e384ee2-87512f6b3f12436b.journal~
-rw-r----- 1 root adm 4608000 Oct 23 21:24 system@0004ccbfc1eb67e2-affb2b40fbecc5c4.journal~
-rw-r----- 1 root adm 4620288 Oct 23 21:45 system@0004ccc00b5676c4-8817abec9320f44f.journal~
-rw-r----- 1 root adm 4624384 Oct 23 21:47 system@0004ccc013cea1ec-078371363cec1009.journal~
-rw-r----- 1 root adm 4612096 Oct 23 21:51 system@0004ccc02417648e-4db8132d72002a4c.journal~
-rw-r----- 1 root adm 4620288 Oct 23 21:55 system@0004ccc031353025-9f0d0a9319f898ac.journal~
-rw-r----- 1 root adm 4673536 Oct 23 23:55 system@0004ccc1deb3af4e-aa5ef78d58abb011.journal~
-rw-r----- 1 root adm 4591616 Oct 23 23:56 system@0004ccc1e105ad91-7c5055d006d24afb.journal~
-rw-r----- 1 root adm 4509696 Oct 23 23:57 system.journal
-rw-r----- 1 root adm 3739648 Oct 19 15:48 user-1000@0004cc6a9722e217-7911792d2764214f.journal~
-rw-r----- 1 root adm 3739648 Oct 21 04:34 user-1000@0004cc896ae0408d-e114a5fc9bb7e819.journal~
-rw-r----- 1 root adm 3739648 Oct 21 20:17 user-1000@0004cc9695e57804-19dcd504533e199e.journal~
-rw-r----- 1 root adm 3743744 Oct 22 17:06 user-1000@0004cca80a357c01-3c2d25399459d10f.journal~
-rw-r----- 1 root adm 3739648 Oct 23 22:19 user-1000@0004ccc0866ff384-8678d71df2570e86.journal~
-rw-r----- 1 root adm 3739648 Oct 23 22:19 user-1000.journal
I take it this isn't good then?
EDIT/ full dmesg @ www.meyithi.com/dmesg.log
Last edited by Meyithi (2012-10-23 23:21:31)
Offline
There seems to be some problem when /var is on a different partition. I guess we'll have to ask the systemd guy(s).
Offline
There seems to be some problem when /var is on a different partition. I guess we'll have to ask the systemd guy(s).
I have this problem too and my /var is on the same partition.
Cheers.
Scientia et sapientia patrimonium humanitatis sunt.
Offline
[ALERT] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3
https://bbs.archlinux.org/viewtopic.php?id=151341
Offline
[ALERT] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3
https://bbs.archlinux.org/viewtopic.php?id=151341
My filesystem is JFS.
Scientia et sapientia patrimonium humanitatis sunt.
Offline
rasertux wrote:[ALERT] EXT4 Data Corruption Bug Linux 3.6.2 & 3.6.3
https://bbs.archlinux.org/viewtopic.php?id=151341My filesystem is JFS.
Mine is reiserfs, but it's separate.
Offline
Hi all,
I don't use Arch, but maybe this will help...
I have the same problem on embedded box (Linux build by OpenEmbedded). I think, that this bug appeared somewhere in v189/v190. So we went back to the v188, which is OK. I tested in on v195 just now and the problem persist. BTW, I'am able to read logs from previous boots.
/var is on the same partition. I'am using ext3 on SD card, there are almost no writes to the card and all reboots are clean (using "reboot").
Stoupa
Last edited by dontbugme (2012-10-24 17:08:46)
Offline
I've found something that seems to solve it for me. If I boot, run:
sudo systemctl daemon-reload
then reboot, the problem goes away.
After running the above, the only difference in the output of "systemctl" is the addition of a "dev-sda3.swap" unit.
You could all try that and see what happens.
If anyone knows why dev-sda3.swap isn't looking after itself I'd love to know
Maybe it'll only load if a process starts using swap?
Offline
I've found something that seems to solve it for me. If I boot, run:
sudo systemctl daemon-reload
then reboot, the problem goes away.
After running the above, the only difference in the output of "systemctl" is the addition of a "dev-sda3.swap" unit.
You could all try that and see what happens.If anyone knows why dev-sda3.swap isn't looking after itself I'd love to know
Maybe it'll only load if a process starts using swap?
Unfortunately, this didn't help me. I had no swaps, so I created the 10 MB swapfile and tried again, but also without any success. Moreover, new message about not unmounted drives appeared during shutdown.
Unmounting file systems.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/debug.
Could not remount as read-only /: Device or resource busy
Not all file systems unmounted, 1 left.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Unmounting file systems.
All filesystems, swaps, loop devices, DM devices detached.
But this is different story.
S.
Offline
I've accidentally solved my issue - https://bugs.archlinux.org/task/32191#comment101107 - I don't know why or how though
Offline
bump.
when
halt
gets me this:
sending sigterm to remaining processes...
sending sigkill to remaining processes...
unmounting filesystems
unmounted /var
unmounted /sys/kernel/debug
unmounted /dev/hugepages
unmounted /dev/mqueue
disabling swaps
detaching loop devices
detaching DM devices
could not delete dm /dev/dm-0: device or resource busy
not all DM devices detached, 2left
could not delete dm /dev/dm-0: device or resource busy
not all DM devices detached, 2left
cannot finalize remaining file systems and devices, giving up
successfully changed into root pivot
detaching loop devices
unmounting all devices
disassembling stacked devices
0 logical volume(s) in volume "vgroup" now active
system halted
but it doesn't power off on its own i have to hold the power button and that is annoying especially on laptop. is there a FIX?
NOTE: i have new arch install and this started to occur somewhere between post install config and pacman -S xorg sudo awesome. i cant tell for sure when it started, but my first halt after successful install was OK: it powered off my pc
NOTE2: shutdown hook didn't help
NOTE3: my disk setup is:
sda1 - ext4/boot syslinux+gpt
sda2 - luks ( lvm (separate partitions for /root /home /var /temp swap ) )
please help.
a.
Offline
Use "poweroff" instead of "halt" if you want it to power off by itself.
Offline
I have the same problems:
My journal is corrupted
[ 27.326195] uhci_hcd 0000:00:1a.0: power state changed by ACPI to D2
[ 27.329735] uhci_hcd 0000:00:1a.2: power state changed by ACPI to D2
[ 27.343413] ehci_hcd 0000:00:1a.7: power state changed by ACPI to D3cold
[ 27.360073] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D3cold
[ 27.650798] ehci_hcd 0000:00:1a.7: power state changed by ACPI to D0
[ 27.753607] ehci_hcd 0000:00:1a.7: setting latency timer to 64
[ 27.930481] ehci_hcd 0000:00:1d.7: power state changed by ACPI to D0
[ 28.033623] ehci_hcd 0000:00:1d.7: setting latency timer to 64
[ 28.088768] uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
[ 28.088936] uhci_hcd 0000:00:1a.0: setting latency timer to 64
[ 28.133556] uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
[ 28.133718] uhci_hcd 0000:00:1a.2: setting latency timer to 64
[ 36.704964] systemd-journald[210]: File /var/log/journal/f663e6519fbbc5da304be4a50000135d/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.
[ 92.824132] fuse init (API version 7.20)
And like others in this thread I get the error on shutdown/reboot that dm devices cannot be detached. (Where do I have to search for shutdown errors? I can't find the right log...)
As both of these errors occurred right after my switch to systemd today I will switch back to initscripts and hope that my system will shutdown properly again...
EDIT: I have everything on one partition except /boot and /home; furthermore I use ext4.
Last edited by ball (2012-10-26 21:12:33)
Offline