Systemd error when using separate /var partition

heman · 2012-12-05 12:28:42

I recently moved /var to a new partition and have noticed this error in journalctl log

Dec 05 07:48:00 archlinux systemd[1]: Started Console System Shutdown Logging.
Dec 05 07:48:00 archlinux systemd[1]: var.mount mount process exited, code=exited status=32
Dec 05 07:48:00 archlinux systemd[1]: Failed unmounting /var.

I added an entry in fstab for /var

/dev/sda4 /var  ext4 defaults,noatime,discard 0 1

Drive is ssd and I simply added the options I have for / is this correct?

10PinkPanther · 2012-12-05 13:21:33

Did you only add this to fstab or did you actualy made /var partition with cfdisk or fdisk?

krest · 2012-12-05 13:22:18

Probably not relevant, but change

/dev/sda4 /var  ext4 defaults,noatime,discard 0 1

to

/dev/sda4 /var  ext4 defaults,noatime,discard 0 2

heman · 2012-12-05 15:36:06

@10PinkPanther Yes I created partition and it is mounted ok.

Jristz · 2012-12-05 18:07:14

why not use UUID, probably the reboots meed the order of the drives (orchange the name or anithynk like it)

Foucault · 2012-12-07 17:49:06

I have the exact same problem. I can not even umount /var manually even though lsof returns nothing. The relevant fstab line

# /dev/sdb2 LABEL=var
UUID=a24cb81b-a0b1-4c4d-a625-55d1c9c2a88a       /var            reiserfs        rw,relatime     0 2

Does anyone have any insight on this issue? I've also tried adding the shutdown hook to mkinitcpio as per this post but no luck.

Last edited by Foucault (2012-12-07 18:02:20)

nixpunk · 2012-12-10 23:00:09

Same issue here with SATA drive and separate /var partition. Shutdown hook didn't help either. I remember having this issue in the past before using systemd, but I can't remember how I fixed it.

Last edited by nixpunk (2012-12-10 23:00:52)

hunterthomson · 2012-12-11 04:13:49

Yep, this is a problem with journald

Systemd can not unmount /var becuase journald is still writing log files to /var/log/journal/

The easy way to go is to delete the folder /var/log/journal
Then journald will write it's logs to /run

The correct way to fix this is to edit the UNIT files so it dose not try to umount /var until after journald has been stopped. However, I do not know if this is possible.

Last edited by hunterthomson (2012-12-11 04:14:49)

falconindy · 2012-12-11 04:21:10

No, the correct "fix" for this is to use mkinitcpio's shutdown hook -- consider it a requirement for having /var on a separate partition (and this isn't particularly anything new). Deleting /var/log/journal is futile, as it's packaged with systemd.

hunterthomson · 2012-12-11 04:27:19

Well, it is not futile, when you delete it journald dose not try to recreate it. It just writes the logs to /run by default.

Awe okay, so like I should change my HOOKS= array from this

HOOKS="base udev autodetect block keymap encrypt lvm2 filesystems usbinput fsck"

To this?

HOOKS="base udev autodetect block keymap encrypt lvm2 filesystems usbinput fsck var shutdown"

Or what should the order be?

Last edited by hunterthomson (2012-12-11 04:29:10)

WonderWoofy · 2012-12-11 04:31:29

I don't think there is a var hook. Only a usr hook.

If you want to stop persistent journaling, I think the better solution would be to use /etc/systemd/journald.conf.

hunterthomson · 2012-12-11 04:47:21

Well falconindy is an arch dev. After reading his post again... I think he said I just need to add the "shutdown" hook, not anything ellse.

I'll give it a shot and report back.

But ya, WonderWoofy. I think I could like add "Storage=/run/journal" to /etc/systemd/journald.conf would work too.

falconindy · 2012-12-11 04:51:28

journald.conf(5) would tell you the correct value is "volatile" if you want the journal stored in /run/log, rather than /var/log

HungGarTiger · 2012-12-11 04:56:37

same problem here, I followed the advice in the other thread posted but the problem persists. Here is my mkinitcpio..

HOOKS="base udev autodetect modconf block filesystems usbinput fsck shutdown"

Also checked journald.conf, but couldn't really find any relevent options. Mostly just detailing speed/where to write to journal

Last edited by HungGarTiger (2012-12-11 04:57:45)

hunterthomson · 2012-12-11 05:16:00

falconindy wrote:

journald.conf(5) would tell you the correct value is "volatile" if you want the journal stored in /run/log, rather than /var/log

Awe yes, after a few reboots. I decided reading the man page would be a good way to go
man journald.conf

Adding shutdown to the end of the HOOKS array and rebuilding the initramfs did not work. I still got the failed to umount /var

But, adding Storage=volatile to /etc/systemd/journald.conf did the trick and I deleted /var/log/journal Now my logs are in /run/log/journal which is fine with me.

Last edited by hunterthomson (2012-12-11 05:17:12)

falconindy · 2012-12-11 05:17:17

Right, systemd will still try to unmount it. It's still going to fail. The shutdown hook will succeed, but it isn't going to tell you about it.

hunterthomson · 2012-12-11 05:31:07

Okay, that makes complete sense. Journald is stoped at that time so how could it write that /var was latter umounted correctly to a log on a filesytem that is not even mounted anymore.

Last edited by hunterthomson (2012-12-11 05:46:35)

HungGarTiger · 2012-12-11 05:55:55

hunterthomson wrote:

Okay, that makes complete sense. Journald is stoped at that time so how could it write that /var was latter umounted correctly to a log on a filesytem that is not even mounted anymore.

Hmmmm, I never thought of it so logically. So leaving things as they are is fine then? And the error message is no problem?

The error message is actually showing a successful unmount by journald not being able to write to the log file....

hunterthomson · 2012-12-11 06:17:08

HungGarTiger wrote:

The error message is actually showing a successful unmount by journald not being able to write to the log file....

What? I don't understand.

The log says "umounting /var" then "Failed to umount /var"

What I think falconindy is saying is that, yes systemd will still try and fail to umount /var. However, if you add "shutdown" to the end of your HOOKS="" array in /etc/minitcpio.conf and rebuild the initramfs. That shutdown hook will umount /var after all the systemd and journald stuff is no longer running.

Personaly, I'm just going have journald write the journal to /run/log/journal. I run AIDE before and after every upgrade, so I will know if anything funky happens to my system. A log would be usefull, but as long as I know what files have been changed I can recover them from my backups.

Last edited by hunterthomson (2012-12-11 06:22:36)

HungGarTiger · 2012-12-11 06:42:28

hunterthomson wrote:

HungGarTiger wrote:
The error message is actually showing a successful unmount by journald not being able to write to the log file....
What? I don't understand.
The log says "umounting /var" then "Failed to umount /var"
What I think falconindy is saying is that, yes systemd will still try and fail to umount /var. However, if you add "shutdown" to the end of your HOOKS="" array in /etc/minitcpio.conf and rebuild the initramfs. That shutdown hook will umount /var after all the systemd and journald stuff is no longer running.

Ok I get it now, thanks for the insight dude.

nixpunk · 2012-12-11 14:01:45

falconindy wrote:

Right, systemd will still try to unmount it. It's still going to fail. The shutdown hook will succeed, but it isn't going to tell you about it.

Ahh. That makes sense.

cfr · 2013-03-18 23:30:25

Does this only work with newer kernels? With the LTS kernel, I get

device-mapper: remove ioctl on lvm failed: Device or resource busy

repeated many times (more than a screen) on shutdown and I find "Failed unmounting /var." in the journal. A bit later, I then get "kernel: watchdog watchdog0: watchdog did not stop!" in the journal and the next message notes that journald has stopped.

I don't see the same chaos usually. (I'm not about to risk rebooting with the current kernel as I have problems booting right now but I certainly didn't see this prior to today's update with stable current kernels.)

mkkot · 2015-11-09 19:32:51

Hello all,

I have /var on separate partition and the same problem with Failed unmounting /var message when powering off my system. Within few months I had 2 times problems with filesystem on /var (which was xfs) so I finally got angry and re-formatted it to ext4 (as I can't really check if xfs filesystem was cleanly unmounted or not because of it's utilities which won't tell me that or I can't force them to do that).

After that I made a simple test: I reset on reset button my pc, booted from pendrive and checked filesystem via fsck. On each partition there was a message "Replaying journal" and then filesystem clean. There obviously should be no Replaying journal message beforehand when filesystem gets unmounted in proper manner.

So in next step I started my Archlinux and rebooted it. I saw a message that /var was not unmounted properly, so I booted up again live archlinux from pendrive and checked all partitions with fsck.ext4. The result is that all of them were clean, including /var. So there is nothing to be worried about, as in my case the filesystem is really being unmounted in proper manner.

Arch Linux

#1 2012-12-05 12:28:42

Systemd error when using separate /var partition

#2 2012-12-05 13:21:33

Re: Systemd error when using separate /var partition

#3 2012-12-05 13:22:18

Re: Systemd error when using separate /var partition

#4 2012-12-05 15:36:06

Re: Systemd error when using separate /var partition

#5 2012-12-05 18:07:14

Re: Systemd error when using separate /var partition

#6 2012-12-07 17:49:06

Re: Systemd error when using separate /var partition

#7 2012-12-10 23:00:09

Re: Systemd error when using separate /var partition

#8 2012-12-11 04:13:49

Re: Systemd error when using separate /var partition

#9 2012-12-11 04:21:10

Re: Systemd error when using separate /var partition

#10 2012-12-11 04:27:19

Re: Systemd error when using separate /var partition

#11 2012-12-11 04:31:29

Re: Systemd error when using separate /var partition

#12 2012-12-11 04:47:21

Re: Systemd error when using separate /var partition

#13 2012-12-11 04:51:28

Re: Systemd error when using separate /var partition

#14 2012-12-11 04:56:37

Re: Systemd error when using separate /var partition

#15 2012-12-11 05:16:00

Re: Systemd error when using separate /var partition

#16 2012-12-11 05:17:17

Re: Systemd error when using separate /var partition

#17 2012-12-11 05:31:07

Re: Systemd error when using separate /var partition

#18 2012-12-11 05:55:55

Re: Systemd error when using separate /var partition

#19 2012-12-11 06:17:08

Re: Systemd error when using separate /var partition

#20 2012-12-11 06:42:28

Re: Systemd error when using separate /var partition

#21 2012-12-11 14:01:45

Re: Systemd error when using separate /var partition

#22 2013-03-18 23:30:25

Re: Systemd error when using separate /var partition

#23 2015-11-09 19:32:51

Re: Systemd error when using separate /var partition

Board footer