[CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

LunarLambda · 2024-11-24 09:08:31

For a future PC build I am planning on running my root filesystem on a ZFS raidz1 pool, with the drives being LUKS-encrypted.

The reason for this being that ZFS' native encryption as of 2.2.6 has several major problems as well as playing poorly with other features such as compression.

Unfortunately I have been completely unable to produce a working initramfs that can unlock the drives and mount the pool correctly.

I tried both mkinitcpio and dracut, both with and without systemd. In both cases I am creating a UKI I am booting using systemd-boot. ZFS is provided by the zfs-dkms and zfs-utils AUR packages (both tagged to 2.2.6), with the LTS kernel.

1. mkinitcpio (busybox): The crypt hook does not support unlocking multiple disks, nor does it support the use of systemd-cryptsetup or keyctl for caching passwords so I only need to type it in once, not 3 times.
2. mkinitcpio (systemd): The ZFS hook shipped with zfs-utils in the AUR does not support systemd, which results in ordering issues when the hook is run before the sd-encrypt hook has unlocked the disks. The sd-zfs AUR package has been unmaintained for about 3 years and as such I don't really want to rely on it.
3. dracut (non-systemd/bash): The dracut ZFS module provided by OpenZFS should support a non-systemd initramfs, but does not work correctly without systemd and causes booting to fail due to a udev issue, see https://github.com/openzfs/zfs/issues/16802
4. dracut (systemd): There seems to be an issue possibly related to systemd v256 that's causing a kernel panic on shutdown (see https://github.com/dracut-ng/dracut-ng/issues/1023) as well as some semi-recent regressions with correctly unlocking LUKS drives in systemd-based initramfs.

I'm not really sure what my best option here is.
Write my own mkinitcpio encrypt hook, possibly based on debian's `decrypt_keyutil` keyscript?
Port OpenZFS's dracut zfs module to mkinitcpio as a replacement for sd-zfs?
See if I can somehow get dracut to behave despite having only a few days of experience using it (from trying to accomplish this very setup in a VM)?
Ditch encryption? (don't want to...)
Ditch booting off of ZFS and put in a non-ZFS boot drive?

If someone has a working install of this kind, or can walk me through how to debug the dracut-related issues beyond what I was able to accomplish in the 2 linked github issues, please let me know.

Last edited by LunarLambda (2024-11-24 17:47:01)

cryptearth · 2024-11-24 13:03:42

first of all: do NOT layer zfs inside anything else!
zfs is specificlly designed to be the lowest layer and to have direct acfess to the drives
zfs MIGHT work when layered inside other containers or using partitions instead of drives but it's not designed to be used in such an environment

as for booting: yes, ditch the idea of having root on any kind of array - if you really need encryption and the pool buy a cheap small ssd to boot from with a single ext4 partition for root and have the pool on a separate mountpoint
as for encryption: if you really have the need for it but don't want to or can't use zfs' native encryption (is there a bug report upstream for the issues you have?) I'm not sure if you can use luks or lvm on a zfs dataset but you could create a zfs volume and use it the same way as any other virtual volume (basically what lvm/md provides)

LunarLambda · 2024-11-24 13:23:00

As for issues with native encryption on ZFS:
https://github.com/openzfs/openzfs-docs/issues/494
https://wiki.gentoo.org/wiki/ZFS#Caveats

I've not heard any particular issues about layering ZFS in e.g. a plain dm-crypt or even LUKS. The wiki even has a section suggesting it (https://wiki.archlinux.org/title/ZFS#En … g_dm-crypt), although the caveats it mentions about playing better with deduplication or compression seem to contradict with the openzfs docs link which suggests that these features play mostly fine together. Bit confusing but it also doesn't seem outright harmful to use LUKS, especially if it sidesteps the apparently many unsolved bugs with ZFS native encryption.

If having a separate boot SSD is the way, that's fine. But I do want to have encryption, and as it stands, everyone I've spoken to says to avoid the ZFS encryption.

WorMzy · 2024-11-24 15:30:49

Mod note: moving to AUR Issues

cryptearth · 2024-11-24 17:31:35

to put it this way: what is the actual threat scenario you want to protect against?
is this a mobile device? then, sorry not sorry, my honest point is: if you lose it's your own fault
if this is a regular desktop: any sort of attack you can protect against is physical access - or: someone already broke into your home - and although I do see this can be a real threat the only real protection against that is off-site storage

as for why zfs is adviced to have direct access and be the lowest layer: zfs is designed to overcome shortcommings of MD or LVM in terms of error detection and correction
in a regular MD or LVM the array depends on the drive reporting itself to have encountered an error - but if it just replies with garbage data in the worst case the array controller can't tell who's lying and in the end overwrites good data with bad data - a proper option are drives with 520byte sectors which store a 8byte checksum along the 512byte data (there's an 4k equivalent but I don'T know its specs from the top of my head) - but such drives are usually SAS and require special controllers - at least I don'T know any SATA 520byte/sector consumer grade drive - if someone does: please let me know
zfs comes with its own i/o stack and writes additional block checksums so it can tell which drives returned garbage and can reconstruct the data from parity and prevent bad data overwriting good data - layering a drive behind luks/lvm/md loses you that advantage - at that point using zfs doesn't make any sense anymore

also: https://gitlab.com/cryptsetup/cryptsetu … ns#2-setup - 2.2 (3) and (4):

(3) Encrypted RAID: Create your RAID from partitions and/or full devices. Put LUKS on top of the RAID device, just if it were an ordinary block device. Applications are just the same as above, but you get redundancy. (Side note as many people seem to be unaware of it: You can do RAID1 with an arbitrary number of components in Linux.) See also Item 2.8.
(4) Now, some people advocate doing the encryption below the RAID layer. That has several serious problems. One is that suddenly debugging RAID issues becomes much harder. You cannot do automatic RAID assembly anymore. You need to keep the encryption keys for the different RAID components in sync or manage them somehow. The only possible advantage is that things may run a little faster as more CPUs do the encryption, but if speed is a priority over security and simplicity, you are doing this wrong anyways. A good way to mitigate a speed issue is to get a CPU that does hardware AES as most do today.

as well as 2.8:

  Filesystem     <- top
  |
  Encryption (LUKS)
  |
  RAID
  |
  Raw partitions (optional)
  |
  Raw disks      <- bottom

as for how to setup lvm or luks on top of zfs - I'm not quite sure - but it doesn't work with a simple zfs dataset with mountpoint set to none or legacy - so I guess a zfs volume might be do the trick here - but that's a question for zfs upstream: https://github.com/openzfs/zfs/discussions

LunarLambda · 2024-11-24 17:46:22

I simply encrypt my devices whenever possible as a matter of habit, since I was mainly using a laptop for the last few years (though this is a desktop), and at my work it is mandatory.

But I do see your point that trying to put LUKS above or below ZFS is inadvisable.

So I will wait for the showstoppers with ZFS native encryption to be fixed rather than try and force a configuration that is quite clearly more pain than gain. Thank you for the advice.

Last edited by LunarLambda (2024-11-24 17:50:35)

cryptearth · 2024-11-24 18:12:12

well - I played around a bit - but the only way I was able to get something to work was to create a zfs volume as a mockup virtual drive and use it as base for any further commands of lvm or luks
I was not able to get either a lvm or a luks container working directly on a zfs dataset
so you could go

zfs create -V <size> pool/volume
cryptsetup luksFormat /dev/zvol/pool/volume
cryptsetup open /dev/zvol/pool/volume crypt

but depend on which zfs vdev / pool layout you use this can waste a lot of space due to volume overhead - and if you want to automate it you have to ensure to do that crypt open after zfs-volumes.target is reached to have the zfs volume available

WorMzy · 2024-11-24 23:58:57

Closing, on implied request, for deletion undeleting, on third-party appeal.

Last edited by WorMzy (2024-11-25 00:41:50)

Arch Linux

#1 2024-11-24 09:08:31

[CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#2 2024-11-24 13:03:42

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#3 2024-11-24 13:23:00

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#4 2024-11-24 15:30:49

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#5 2024-11-24 17:31:35

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#6 2024-11-24 17:46:22

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#7 2024-11-24 18:12:12

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

#8 2024-11-24 23:58:57

Re: [CLOSED] ZFS raidz1 on LUKS - cannot get a working initramfs

Board footer