[solved] Boot from ZFS root successful but rest of pool won't mount

iliveinapark · 2013-04-04 06:25:08

TL;DR for thread: If you separate out "core" filesystems (/var, /usr, /etc) on a ZFS root system, set them to have mountpoint=legacy and mount them using /etc/fstab, else they won't mount, and your machine won't boot properly.

I have arch installed on a ZFS raidz, with /boot on an SSD which I intend to use the split for L2ARC/ZIL. See below for all the details.

The machine boots just fine, however it only mounts the bootfs:

storage/root

from my zpool, but not

storage/root/var

An empty /home and a sparse /var are created.
Running:

rm -rf /home /var 
zfs mount -a

sets everything right, however.

I've tried changing the kernel option to

zfs=storage

but the root filesystem does not mount in this case, and I get dumped to busy box.

Any help would be greatly appreciated.

ZFS details:

# zpool status
pool: storage
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Thu Apr 4 00:56:08 2013
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5219192 ONLINE 0 0 0
ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5220136 ONLINE 0 0 0
ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5210619 ONLINE 0 0 0
ata-ST31000524AS_9VPBVZAK ONLINE 0 0 0
errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 1.00G 2.59T 221K none
storage/home 6.20M 2.59T 6.20M /home
storage/root 823M 24.2G 823M /
storage/root/var 194M 9.81G 194M /var
# zpool get bootfs
NAME PROPERTY VALUE SOURCE
storage bootfs storage/root local
# cat /boot/syslinux/syslinux.cfg
...
LABEL arch
MENU LABEL Arch Linux
LINUX ../vmlinuz-linux
APPEND zfs=bootfs zfs_force=1 ro
INITRD ../initramfs-linux.img

Edit: Updated to my current, tweaked setup.

Last edited by iliveinapark (2013-04-10 08:28:17)

ezzetabi · 2013-04-04 07:38:36

did you activate the zfs daemon? if you did double check it creates the cache file in the right place: https://wiki.archlinux.org/index.php/ZF … atic_Start

iliveinapark · 2013-04-06 03:14:34

Thanks for the idea. I hadn't started the daemon when I posted. Having done it now, storage/home is successfully mounting, but storage/var cannot mount, and the zfs daemon is failing.

I think the issue is that /var is being created by one of the hooks before the zfs hook, and so zfs cannot mount to /var.

zfs.service - Zettabyte File System (ZFS)
Loaded: loaded (/usr/lib/systemd/system/zfs.service; enabled)
Active: failed (Result: exit-code) since Sat 2013-04-06 08:24:59 EST; 1min 1s ago
Docs: man:zfs(8)
man:zpool(8)
Process: 638 ExecStart=/usr/sbin/zfs mount -a (code=exited, status=1/FAILURE)
Process: 503 ExecStart=/usr/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
Process: 489 ExecStart=/sbin/modprobe zfs (code=exited, status=0/SUCCESS)
Apr 06 08:24:59 bamboozle zpool[503]: no pools available to import
Apr 06 08:24:59 bamboozle zfs[638]: cannot mount '/var': directory is not empty
Apr 06 08:24:59 bamboozle systemd[1]: zfs.service: main process exited, code=exited, status=1/FAILURE
Apr 06 08:24:59 bamboozle systemd[1]: Failed to start Zettabyte File System (ZFS).
Apr 06 08:24:59 bamboozle systemd[1]: Unit zfs.service entered failed state
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

/var:
total 60
drwxr-xr-x 5 root root 5 Apr 6 08:25 .
drwxr-xr-x 18 root root 20 Apr 6 08:24 ..
drwxr-xr-x 3 root root 3 Apr 6 08:25 cache
drwxr-xr-x 2 root root 3 Apr 6 08:24 lib
drwxrwxrwt 2 root root 2 Apr 6 08:25 tmp
/var/cache:
total 36
drwxr-xr-x 3 root root 3 Apr 6 08:25 .
drwxr-xr-x 5 root root 5 Apr 6 08:25 ..
drwxr-xr-x 2 root root 2 Apr 6 08:25 man
/var/cache/man:
total 24
drwxr-xr-x 2 root root 2 Apr 6 08:25 .
drwxr-xr-x 3 root root 3 Apr 6 08:25 ..
/var/lib:
total 31
drwxr-xr-x 2 root root 3 Apr 6 08:24 .
drwxr-xr-x 5 root root 5 Apr 6 08:25 ..
-rw------- 1 root root 512 Apr 6 08:24 random-seed
/var/tmp:
total 24
drwxrwxrwt 2 root root 2 Apr 6 08:25 .
drwxr-xr-x 5 root root 5 Apr 6 08:25 ..

# grep "^HOOKS" /etc/mkinitcpio.conf
HOOKS="base udev autodetect modconf block zfs filesystems keyboard"

ezzetabi · 2013-04-08 08:07:47

I had a similar problem with /etc, but I had to give up.

cngn · 2013-04-08 09:16:30

iliveinapark wrote:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 1.00G 2.59T 221K none
storage/home 6.20M 2.59T 6.20M /home
storage/root 823M 24.2G 823M /
storage/var 194M 9.81G 194M /var

Did you try storage/root/var if it changes anything?

iliveinapark · 2013-04-08 09:46:01

Yeah, did that today after reading on an old Sun mailing list archive that /var, /usr, and so on, the "core" filesystems, should be decendents of the root filesystem. No difference, still can't mount /var because it already exists.

I've seen quite a few posts that use the format:

<pool>/ROOT/<rootfs>
<pool>/ROOT/<rootfs>/var

As far as I can tell, though, that's just a convention, and there should be no difference to my setup.

ezzetabi · 2013-04-08 13:55:26

Control that if in the / filesystem there is a directory called /var it might. If it is present it will stop the /var filesystem to be mounted.

iliveinapark · 2013-04-09 00:04:18

@ezzetabi: Not sure exactly what you mean, mate. The problem does seem to be that /var is getting created by something before the zfs daemon kicks in. Deleting it and starting the zfs daemon allows it to mount no problem.

If storage/root/var would mount at the same time as storage/root, I think it'd work fine. Just gotta figure out how to get it to do so.

Last edited by iliveinapark (2013-04-09 01:04:19)

ezzetabi · 2013-04-09 07:18:39

That is exactly what I had to do with my /etc problem.
I booted with an USB key and mounted the filesystem hosting /etc in a directory, mounted the filesystem hosting / in another, copied the former in the latter and finally removed the filesystem hosting /etc.

cngn · 2013-04-09 08:13:22

You may try setting the mountpoint properties to legacy and delegating mounts to fstab.

Last edited by cngn (2013-04-09 08:38:22)

ezzetabi · 2013-04-10 07:34:20

That is indeed a good idea, fstab is parsed very soon in the boot process and it might solve the problem. If it works it deserves a note in the wiki.

iliveinapark · 2013-04-10 08:23:52

Eureka! I don't like the fact that it's so customised, but am happy enough with this result.

# zfs set mountpoint=legacy storage/root/var
# cat /etc/fstab
UUID=61ea9c33-fbd7-4de1-9b72-e70dd45ef012 /boot ext2 rw,noatime 0 2
storage/root/var /var zfs defaults 0 2

It seems that the zfs hook mounts / with no worries, /etc/fstab mounts /var, and the zfs daemon mounts /home. I suppose I could set all three to mountpoint=legacy and do it in /etc/fstab, but I'm going to stay with what's working.

My thanks to all who contributed.

Last edited by iliveinapark (2013-04-10 08:24:25)

ezzetabi · 2013-04-10 08:34:20

Update the wiki! Write somewhere that using a separate zfs filesystem for /etc or /var (and possibly other directories needed during the boot) require legacy mountpoint and /etc/fstab editing.

iliveinapark · 2013-04-10 09:47:30

Will do, when I get some spare time. Thanks for your inputs mate.

Zazzman · 2013-08-22 13:53:56

iliveinapark wrote:

Will do, when I get some spare time. Thanks for your inputs mate.

You didn't update the wiki :'(

iliveinapark · 2013-08-22 14:00:33

Zazzman, you're absolutely correct. I'm sorry, buddy. Life happened, and I forgot all about it. As it happens, I've got the next week off, so I will look into this stuff again, and update it with whatever info I can.

Thanks for reminding about it.

Arch Linux

#1 2013-04-04 06:25:08

[solved] Boot from ZFS root successful but rest of pool won't mount

#2 2013-04-04 07:38:36

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#3 2013-04-06 03:14:34

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#4 2013-04-08 08:07:47

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#5 2013-04-08 09:16:30

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#6 2013-04-08 09:46:01

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#7 2013-04-08 13:55:26

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#8 2013-04-09 00:04:18

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#9 2013-04-09 07:18:39

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#10 2013-04-09 08:13:22

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#11 2013-04-10 07:34:20

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#12 2013-04-10 08:23:52

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#13 2013-04-10 08:34:20

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#14 2013-04-10 09:47:30

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#15 2013-08-22 13:53:56

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

#16 2013-08-22 14:00:33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Board footer