Failed mount /sysroot - zfs on lvm on luks with sd-boot

m4ks!m · 2021-06-24 13:28:54

Hi everyone,

i'm trying in the last 2 weeks to set this up in a new installation. I always end up getting the same error. Searching for this error I only find wrong configured kernel startup parameters and IMHO mine seem to be ok. I don't know what else can be done.

This is my setup:

sda                         
└─c1                    crypt
  ├─systemVG-rootLV     lvm   
   |   └─zfs pool                           /mnt
  └─systemVG-swapLV   lvm   

sdc (USB)                   
  └─sdc1                                     /mnt/boot
                                                       └─luksheader

zfs mounts:

# zfs get mountpoint
NAME                                    PROPERTY    VALUE                     SOURCE
rpool                                   mountpoint  /mnt/root                 local
rpool/arch                              mountpoint  none                      local
rpool/arch/DATA                         mountpoint  none                      local
rpool/arch/DATA/default                 mountpoint  /mnt/root                 local
rpool/arch/DATA/default/home            mountpoint  /mnt/root/home            inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/nix             mountpoint  /mnt/root/nix             inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/root            mountpoint  /mnt/root/root            inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/srv             mountpoint  /mnt/root/srv             inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/usr             mountpoint  /mnt/root/usr             inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/usr/local       mountpoint  /mnt/root/usr/local       inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var             mountpoint  /mnt/root/var             inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/games       mountpoint  /mnt/root/var/games       inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/lib         mountpoint  /mnt/root/var/lib         inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/lib/docker  mountpoint  /mnt/root/var/lib/docker  inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/log         mountpoint  /mnt/root/var/log         inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/spool       mountpoint  /mnt/root/var/spool       inherited from rpool/arch/DATA/default
rpool/arch/DATA/default/var/www         mountpoint  /mnt/root/var/www         inherited from rpool/arch/DATA/default
rpool/arch/ROOT                         mountpoint  none                      local
rpool/arch/ROOT/default                 mountpoint  /mnt/root                 local

I also set bootfs:

zpool get bootfs rpool
NAME   PROPERTY  VALUE                    SOURCE
rpool  bootfs    rpool/arch/ROOT/default  local

/etc/crypttab:

# Configuration for encrypted block devices.
# See crypttab(5) for details.

# NOTE: Do not list your root (/) partition here, it must be set up
#       beforehand by the initramfs (/etc/mkinitcpio.conf).

# <name>       <device>                                     <password>              <options>
# home         UUID=b8ad5c18-f445-495d-9095-c9ec4f9d2f37    /etc/mypassword1
# data1        /dev/sda3                                    /etc/mypassword2
# data2        /dev/sda5                                    /etc/cryptfs.key
# swap         /dev/sdx4                                    /dev/urandom            swap,cipher=aes-cbc-essiv:sha256,size=256
# vol          /dev/sdb7                                    none
c1 /dev/disk/by-id/ata-SAMSUNG_MZ7TD256HAFV-000L7_S16GNEAD406956 - header=/boot/trezor/header.img

cat /boot/loader/entries/arch.conf 
## This is just an example config file.
## Please edit the paths and kernel parameters according to your system.

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options root=ZFS=rpool/arch/ROOT/default

# cat /boot/loader/loader.conf 
default arch
timeout 3
#console-mode keep

/etc/mkinitcpio.conf:

FILES=(/boot/trezor/header.img)
HOOKS=(base systemd autodetect keyboard sd-vconsole modconf block sd-encrypt lvm2 sd-zfs filesystems)

Using all this i get this error while booting.

 Failed to mount /sysroot

I don't understand why i don't get a password request before it's trying to mount /sysroot.

Before I tried to use sd-zfs I tried a custom hook and the regular 'zfs' hook but it resulted the very same situation.

Last edited by m4ks!m (2021-06-24 13:30:28)

Mr Victory · 2021-06-24 16:27:34

arch.conf wrote:

options root=ZFS=rpool/arch/ROOT/default

In these parameters, there is nothing to specify which partition to boot, unless you labeled your root partition ZFS. For a basic setup, for instance an unencrypted ext4 partition, a parameter like this is required(these are just examples, don't copy paste them):

root=UUID=<the uuid>

or

root=/dev/sda1

You need something specifying the partition but I do not know how exactly you should do that.
What errors do you get alongside

Failed to mount /sysroot

Just in case, do you have a partition or a partitionless disk?

m4ks!m · 2021-06-24 16:39:18

Mr Victory wrote:

In these parameters, there is nothing to specify which partition to boot

I understood that crypttab has this task. I specified my disk by Id there.

do you have a partition or a partitionless disk?

Partitionless I guess, I encrypted /dev/sda without partitioning it and using a detached header.

Last edited by m4ks!m (2021-06-24 16:48:24)

Mr Victory · 2021-06-24 17:27:56

At the moment you got stuck, nothing is mounted so crypttab is not accessible!
Also, the crypttab is not designed to unlock your root partition.

Arch Wiki wrote:

Note that crypttab is read after the system has booted up, therefore it is not a replacement for unlocking encrypted partitions by using mkinitcpio hooks and boot loader options as in the case of encrypting the root partition.

https://wiki.archlinux.org/title/Dm-cry … n#crypttab

m4ks!m · 2021-06-24 21:07:36

Yes, that slipped my eyes. It should be crypttab.initramfs.

Though the only thing that changed is I get an extra line with

[ OK ]Found device SAMSUNG_M7....
Starting cryptography setup for c1

And then it switches to emergency mode where I cannot access the console.

The failed to mount /systoot is still there.

Here's a screenshot of my boot screen since I can't copy paste it. (Sorry for the quality)

https://firealpaca.com/get/DO6rE7El

Last edited by m4ks!m (2021-06-24 21:18:51)

Mr Victory · 2021-06-25 12:00:30

First, LUKS should be unlocked: https://wiki.archlinux.org/title/Dm-cry … UKS_header
From the live image, plug in your USB drive and identify your USB drive and your hard drive with

ls /dev/disk/by-id/

Chroot to the ZFS.
Create the file /etc/initcpio/hooks/customencrypthook containing this:

run_hook() {
    modprobe -a -q dm-crypt >/dev/null 2>&1
    modprobe loop
    [ "${quiet}" = "y" ] && CSQUIET=">/dev/null"

    while [ ! -L '/dev/disk/by-id/[i]id of usb[/i]-part1' ]; do
     echo 'Waiting for USB'
     sleep 1
    done

    mkdir -p /mnt
    mount /dev/disk/by-id/[i]id of usb[/i]-part1 /mnt
    cryptsetup --header /mnt/boot/trezor/header.img open /dev/disk/by-id/[i]id of hard drive[/i] c1
    umount /mnt
}

Then, run this so customencrypthook is used:

cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/customencrypthook

On /etc/mkinitcpio.conf add customencrypthook lvm2 between block and filesystems to the HOOKS array. Remove systemd from HOOKS.
Also, move /etc/crypttab.initramfs to somewhere else. It is not needed.
Regenerate the initcpio.

Secondly, LVM and ZFS should be accessed. On /boot/loader/entries/arch.conf change the options line to

options root=/dev/c1/systemVG-rootLV rootflags=zfs=rpool/arch/ROOT/default

m4ks!m · 2021-06-25 15:36:23

Ok I managed to boot. With systemd and sd-zfs (zfs doesn't work with systemd). ^^

Your kernel parameters made me wondering.
Why are you specifing the roots and the pool in rootflags. Using just the root flag I got a prompt for password (wow) but gpt-auto was hanging. Same is the case when I leave 'root='.

After some tweaking it turned out that the only syntax that works is root=zfs:pool/dataset.
not zfs=Pool/dataset (like said in the wiki)
not root=ZFS=Pool/dataset (like said on sd-zfs readme)

Than I had an error with zfs-import-cache failing. Luckily the emergency shell worked from here on and I found out from the logs that the zfs modules where not loaded (guess that's a bug from sd-zfs?). I added zfs in mkinitcpio.conf and boom it boots.

The only problems I didn't figured out yet is why my root pool is mounted in read-only mode and why it fails when I set /boot in fstab as noauto. I want to be able to pull the usb after booting.

Last edited by m4ks!m (2021-06-25 15:51:49)

Mr Victory · 2021-06-26 06:51:24

For readonly mount, add rw to your kernel parameters. I forgot it
For the USB, you may label your USB as "BOOT" and add a line to fstab like this so the USB is automaticly mounted.

LABEL=BOOT    /boot    auto    nofail,x-systemd.device-timeout=500ms    0 0

Manually unmount the USB before removing.

I just wonder, how exactly did you make it boot? Did you use the customencrypthook? Could you send the full list of the HOOKS in mkinitcpio? What are your kernel parameters? Giving thess info may help others in the future.

Last edited by Mr Victory (2021-06-26 08:40:56)

m4ks!m · 2021-07-12 18:40:35

Sorry for the super late reply. I haven't had time to login into my installation since. I used the sd-zfs hook from here
https://github.com/dasJ/sd-zfs

I installed it through AUR. Interestingly the new syntax for the kernel parameter that someone states in a issue is not a problem in the aur build. Furthermore the old Syntax is indeed the only way it works. I tried to describe it in my earlier post. So again
The right/working syntax is 'zfs:'
Like in

root=zfs:pool/dataset

My mkinitcpio.conf is the same like in my original post

Arch Linux

#1 2021-06-24 13:28:54

Failed mount /sysroot - zfs on lvm on luks with sd-boot

#2 2021-06-24 16:27:34

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#3 2021-06-24 16:39:18

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#4 2021-06-24 17:27:56

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#5 2021-06-24 21:07:36

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#6 2021-06-25 12:00:30

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#7 2021-06-25 15:36:23

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#8 2021-06-26 06:51:24

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

#9 2021-07-12 18:40:35

Re: Failed mount /sysroot - zfs on lvm on luks with sd-boot

Board footer