You are not logged in.

#1 2013-04-04 06:25:08

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

[solved] Boot from ZFS root successful but rest of pool won't mount

TL;DR for thread: If you separate out "core" filesystems (/var, /usr, /etc) on a ZFS root system, set them to have mountpoint=legacy and mount them using /etc/fstab, else they won't mount, and your machine won't boot properly.

I have arch installed on a ZFS raidz, with /boot on an SSD which I intend to use the split for L2ARC/ZIL. See below for all the details.

The machine boots just fine, however it only mounts the bootfs:

storage/root

from my zpool, but not

storage/root/var

An empty /home and a sparse /var are created.
Running:

rm -rf /home /var 
zfs mount -a

sets everything right, however.

I've tried changing the kernel option to

zfs=storage

but the root filesystem does not mount in this case, and I get dumped to busy box.

Any help would be greatly appreciated.

ZFS details:

# zpool status
  pool: storage
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Apr  4 00:56:08 2013
config:

    NAME                                          STATE     READ WRITE CKSUM
    storage                                       ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5219192  ONLINE       0     0     0
        ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5220136  ONLINE       0     0     0
        ata-WDC_WD10EFRX-68JCSN0_WD-WMC1U5210619  ONLINE       0     0     0
        ata-ST31000524AS_9VPBVZAK                 ONLINE       0     0     0

errors: No known data errors

# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
storage       1.00G  2.59T   221K  none
storage/home  6.20M  2.59T  6.20M  /home
storage/root   823M  24.2G   823M  /
storage/root/var    194M  9.81G   194M  /var

# zpool get bootfs
NAME  PROPERTY  VALUE         SOURCE
storage bootfs           storage/root  local

# cat /boot/syslinux/syslinux.cfg
...
LABEL arch
    MENU LABEL Arch Linux
    LINUX ../vmlinuz-linux
    APPEND zfs=bootfs zfs_force=1 ro
    INITRD ../initramfs-linux.img

Edit: Updated to my current, tweaked setup.

Last edited by iliveinapark (2013-04-10 08:28:17)

Offline

#2 2013-04-04 07:38:36

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

did you activate the zfs daemon? if you did double check it creates the cache file in the right place: https://wiki.archlinux.org/index.php/ZF … atic_Start

Offline

#3 2013-04-06 03:14:34

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Thanks for the idea. I hadn't started the daemon when I posted. Having done it now, storage/home is successfully mounting, but storage/var cannot mount, and the zfs daemon is failing.

I think the issue is that /var is being created by one of the hooks before the zfs hook, and so zfs cannot mount to /var.

zfs.service - Zettabyte File System (ZFS)
          Loaded: loaded (/usr/lib/systemd/system/zfs.service; enabled)
          Active: failed (Result: exit-code) since Sat 2013-04-06 08:24:59 EST; 1min 1s ago
            Docs: man:zfs(8)
                  man:zpool(8)
         Process: 638 ExecStart=/usr/sbin/zfs mount -a (code=exited, status=1/FAILURE)
         Process: 503 ExecStart=/usr/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
         Process: 489 ExecStart=/sbin/modprobe zfs (code=exited, status=0/SUCCESS)

Apr 06 08:24:59 bamboozle zpool[503]: no pools available to import
Apr 06 08:24:59 bamboozle zfs[638]: cannot mount '/var': directory is not empty
Apr 06 08:24:59 bamboozle systemd[1]: zfs.service: main process exited, code=exited, status=1/FAILURE
Apr 06 08:24:59 bamboozle systemd[1]: Failed to start Zettabyte File System (ZFS).
Apr 06 08:24:59 bamboozle systemd[1]: Unit zfs.service entered failed state
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

/var:
total 60
drwxr-xr-x  5 root root  5 Apr  6 08:25 .
drwxr-xr-x 18 root root 20 Apr  6 08:24 ..
drwxr-xr-x  3 root root  3 Apr  6 08:25 cache
drwxr-xr-x  2 root root  3 Apr  6 08:24 lib
drwxrwxrwt  2 root root  2 Apr  6 08:25 tmp

/var/cache:
total 36
drwxr-xr-x 3 root root 3 Apr  6 08:25 .
drwxr-xr-x 5 root root 5 Apr  6 08:25 ..
drwxr-xr-x 2 root root 2 Apr  6 08:25 man

/var/cache/man:
total 24
drwxr-xr-x 2 root root 2 Apr  6 08:25 .
drwxr-xr-x 3 root root 3 Apr  6 08:25 ..

/var/lib:
total 31
drwxr-xr-x 2 root root   3 Apr  6 08:24 .
drwxr-xr-x 5 root root   5 Apr  6 08:25 ..
-rw------- 1 root root 512 Apr  6 08:24 random-seed

/var/tmp:
total 24
drwxrwxrwt 2 root root 2 Apr  6 08:25 .
drwxr-xr-x 5 root root 5 Apr  6 08:25 ..

#  grep "^HOOKS" /etc/mkinitcpio.conf
HOOKS="base udev autodetect modconf block zfs filesystems keyboard"

Offline

#4 2013-04-08 08:07:47

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

I had a similar problem with /etc, but I had to give up. sad

Offline

#5 2013-04-08 09:16:30

cngn
Member
Registered: 2010-03-20
Posts: 65

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

iliveinapark wrote:

# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
storage       1.00G  2.59T   221K  none
storage/home  6.20M  2.59T  6.20M  /home
storage/root   823M  24.2G   823M  /
storage/var    194M  9.81G   194M  /var

Did you try storage/root/var if it changes anything?

Offline

#6 2013-04-08 09:46:01

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Yeah, did that today after reading on an old Sun mailing list archive that /var, /usr, and so on, the "core" filesystems, should be decendents of the root filesystem. No difference, still can't mount /var because it already exists.

I've seen quite a few posts that use the format:

<pool>/ROOT/<rootfs>
<pool>/ROOT/<rootfs>/var

As far as I can tell, though, that's just a convention, and there should be no difference to my setup.

Offline

#7 2013-04-08 13:55:26

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Control that if in the / filesystem there is a directory called /var it might. If it is present it will stop the /var filesystem to be mounted.

Offline

#8 2013-04-09 00:04:18

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

@ezzetabi: Not sure exactly what you mean, mate. The problem does seem to be that /var is getting created by something before the zfs daemon kicks in. Deleting it and starting the zfs daemon allows it to mount no problem.

If storage/root/var would mount at the same time as storage/root, I think it'd work fine. Just gotta figure out how to get it to do so.

Last edited by iliveinapark (2013-04-09 01:04:19)

Offline

#9 2013-04-09 07:18:39

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

That is exactly what I had to do with my /etc problem.
I booted with an USB key and mounted the filesystem hosting /etc in a directory, mounted the filesystem hosting / in another, copied the former in the latter and finally removed the filesystem hosting /etc.

Offline

#10 2013-04-09 08:13:22

cngn
Member
Registered: 2010-03-20
Posts: 65

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

You may try setting the mountpoint properties to legacy and delegating mounts to fstab.

Last edited by cngn (2013-04-09 08:38:22)

Offline

#11 2013-04-10 07:34:20

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

That is indeed a good idea, fstab is parsed very soon in the boot process and it might solve the problem. If it works it deserves a note in the wiki.

Offline

#12 2013-04-10 08:23:52

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Eureka! I don't like the fact that it's so customised, but am happy enough with this result.

# zfs set mountpoint=legacy storage/root/var
# cat /etc/fstab
UUID=61ea9c33-fbd7-4de1-9b72-e70dd45ef012    /boot         ext2          rw,noatime    0 2
storage/root/var                /var        zfs        defaults    0 2

It seems that the zfs hook mounts / with no worries, /etc/fstab mounts /var, and the zfs daemon mounts /home. I suppose I could set all three to mountpoint=legacy and do it in /etc/fstab, but I'm going to stay with what's working.

My thanks to all who contributed.

Last edited by iliveinapark (2013-04-10 08:24:25)

Offline

#13 2013-04-10 08:34:20

ezzetabi
Member
Registered: 2006-08-27
Posts: 947

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Update the wiki! Write somewhere that using a separate zfs filesystem for /etc or /var (and possibly other directories needed during the boot) require legacy mountpoint and /etc/fstab editing.

Offline

#14 2013-04-10 09:47:30

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Will do, when I get some spare time. Thanks for your inputs mate.

Offline

#15 2013-08-22 13:53:56

Zazzman
Member
Registered: 2011-12-26
Posts: 34

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

iliveinapark wrote:

Will do, when I get some spare time. Thanks for your inputs mate.

You didn't update the wiki :'(

Offline

#16 2013-08-22 14:00:33

iliveinapark
Member
Registered: 2013-03-18
Posts: 33

Re: [solved] Boot from ZFS root successful but rest of pool won't mount

Zazzman, you're absolutely correct. I'm sorry, buddy. Life happened, and I forgot all about it. As it happens, I've got the next week off, so I will look into this stuff again, and update it with whatever info I can.

Thanks for reminding about it.

Offline

Board footer

Powered by FluxBB