Arch Linux

cmtonkinson

Member

Registered: 2013-11-15

Posts: 18

non-root-ZFS over LUKS: failure of systemd understanding

I have a RAID-Z2 pool on six HDDs which is not used for my root partition, and I'd like to encrypt each of them. My process is to wipe a drive, format the dm-crypt device via LUKS, append appropriately to /etc/crypttab, and then replace the now stale unencrypted drive entry in the zpool with the new /dev/mapper name and wait for resilvering to finish. Having RAID-Z2 allows me to safely add this encryption "in-place" (i.e. without blowing the entire pool away) as long as I do it one disk at a time.

My strategy worked well for the first drive, until I rebooted my system as a sanity check before I moved on to the second drive. Though the new LUKS device is available ZFS, is not bringing it online when it imports the pool (the device is listed UNAVAIL and thus the pool DEGRADED). I've come to believe that ZFS is importing the pool before dm-crypt has unlocked any/all devices, but I'm stuck on how to configure this dependency - this is the first time I've really worked with systemd.

From what I can tell, both ZFS and dm-crypt are being managed by systemd:

    ~ $ systemctl list-units | grep "crypt\|zfs"
    systemd-cryptsetup@locker\x2d1.service  loaded active exited  ...
    zfs-import-cache.service                loaded active exited  ...
    zfs-mount.service                       loaded active exited  ...
    zfs-share.service                       loaded active exited  ...
    system-systemd\x2dcryptsetup.slice      loaded active active  ...
    cryptsetup.target                       loaded active active  ...
    zfs.target                              loaded active active  ...

    ~ $ find /usr/lib/systemd/ -name *crypt*
    /usr/lib/systemd/system-generators/systemd-cryptsetup-generator
    /usr/lib/systemd/systemd-cryptsetup
    /usr/lib/systemd/system/cryptsetup.target
    /usr/lib/systemd/system/sysinit.target.wants/cryptsetup.target

    ~ $ find /usr/lib/systemd/ -name *zfs*
    /usr/lib/systemd/system-preset/50-zfs.preset
    /usr/lib/systemd/system/zfs.target
    /usr/lib/systemd/system/zfs-share.service
    /usr/lib/systemd/system/zfs-import-cache.service
    /usr/lib/systemd/system/zfs-import-scan.service
    /usr/lib/systemd/system/zfs-mount.service

Yet I see no boot log entries for cryptsetup (even though the encrypted block device is available under /dev/mapper as soon as I log in):

    ~ $ journalctl -b | grep 'zed\['
    Jul 09 05:20:03 localhost.example.net zed[720]: ZFS Event Daemon 0.6.3-1
    Jul 09 05:20:03 localhost.example.net zed[720]: Processing events since eid=0
    Jul 09 05:20:03 localhost.example.net zed[720]: Invoking "all-syslog.sh" eid=1 pid=726
    Jul 09 05:20:03 localhost.example.net zed[727]: eid=1 class=statechange
    Jul 09 05:20:03 localhost.example.net zed[720]: Finished "all-syslog.sh" eid=1 pid=726 exit=0
    Jul 09 05:20:03 localhost.example.net zed[720]: Invoking "all-syslog.sh" eid=2 pid=729
    ...

    ~ $ journalctl -b | grep 'crypt'
    ~ $

I attempted to customize the ZFS import service to add cryptsetup as a dependency:

    ~ $ cat /etc/systemd/system/zfs-import-cache.service.d/luks.conf
    [Unit]
    Requires=systemd-cryptsetup@locker\x2d1.service
    After=systemd-cryptsetup@locker\x2d1.service

but no joy. I did the same (same file contents) for the zfs-import-scan.service as well - again, no luck. However, systemd-delta does see these types of changes:

    ~ $ systemd-delta
    [EXTENDED]   /run/systemd/system/session-c2.scope → /run/systemd/system/session-c2.scope.d/90-SendSIGHUP.conf
    [EXTENDED]   /run/systemd/system/session-c2.scope → /run/systemd/system/session-c2.scope.d/90-After-systemd-user-sessions\x2eservic
    [EXTENDED]   /run/systemd/system/session-c2.scope → /run/systemd/system/session-c2.scope.d/90-After-systemd-logind\x2eservice.conf
    [EXTENDED]   /run/systemd/system/session-c2.scope → /run/systemd/system/session-c2.scope.d/90-Description.conf
    [EXTENDED]   /run/systemd/system/session-c2.scope → /run/systemd/system/session-c2.scope.d/90-Slice.conf
    [EXTENDED]   /usr/lib/systemd/system/zfs-import-cache.service → /etc/systemd/system/zfs-import-cache.service.d/luks.conf

From my reading it doesn't look like I can define dependencies on targets, so I'm not really sure where to go for further research. I even went as far as adding encrypt to mkinitcpio HOOKS before filesystems as suggested for encrypted root partitions, but that didn't seem to matter either.

Any direction would be greatly appreciated. All of the documentation I've found (mailing list archives, forums, web search) between the communities of systemd, Arch, ZFS, and dm-crypt (even though I'm fairly sure this is all due to a lack of systemd knowledge on my part) all seem to assume either that you have your root filesystem in ZFS or that you're using LVM on top of LUKS and none of this advice seems to translate directly to what I'm doing.

ostkind

Member

Registered: 2014-07-11

Posts: 2

Re: non-root-ZFS over LUKS: failure of systemd understanding

Running into the same problem here with a RAIDZ1 pool of 2 harddisks on top of dm-crypt with LUKS. I tried the trick with a:

ostkind@somerandomarchbox ~ % cat /usr/lib/systemd/system/zfs.target

[Unit]
Description=ZFS startup target
Requires=zfs-mount.service
Requires=zfs-share.service
After=cryptsetup.target
Wants=zed.service

[Install]
WantedBy=multi-user.target

but no success.

cmtonkinson

Member

Registered: 2013-11-15

Posts: 18

Re: non-root-ZFS over LUKS: failure of systemd understanding

@ostkind, did you try the same with After *and* Requires? The docs claim you need both to get systemd dependencies working the way you "expect".

Last edited by cmtonkinson (2014-07-11 20:06:34)

ostkind

Member

Registered: 2014-07-11

Posts: 2

Re: non-root-ZFS over LUKS: failure of systemd understanding

@cmtonkinson, tried right know to have cryptsetup.target in After and Requires. No success: After reboot zfs pool still not mounted and dm-crypt drives ready in /dev/mapper/.

ukhippo

Member

From: Non-paged pool

Registered: 2014-02-21

Posts: 366

Re: non-root-ZFS over LUKS: failure of systemd understanding

You should be able to see if cryptsetup.target is before zfs.target with:

systemd-analyze critical-chain --fuzz=1m zfs.target

float

Member

Registered: 2012-10-05

Posts: 23

Re: non-root-ZFS over LUKS: failure of systemd understanding

I had the same problem and I used

systemd-analyze plot > sys.svg

to reveal that a service named systemd-cryptsetup@cryptstorage.service (where cryptstorage is my LUKS mapped device) is started before zfs but takes about 3 seconds to finish.
The service zfs-import-cache.service which imports the zpools using /etc/zfs/zpool.cache does not wait for this to finish so cryptroot is not available and the import fails silently.

Here is what I did to get that straight:

mkdir /etc/systemd/system/zfs-import-cache.service.d
echo "[Unit]
After=systemd-cryptsetup@cryptstorage.service" >/etc/systemd/system/zfs-import-cache.service.d/cryptstorage.conf

You have to replace "cryptstorage" by the actual LUKS device, of course.
If you can confirm this solution or even have a more general one please let me know so I can update the wiki to include (this hopefully temporary) workaround.
Google gives a lot of hits on this problem but at least I didn't find any working solution so I started debugging the problem myself.

Edit:
More general solution (pretty obvious):

mkdir /etc/systemd/system/zfs-import-cache.service.d
echo "[Unit]
After=cryptsetup.target" >/etc/systemd/system/zfs-import-cache.service.d/cryptsetup.conf
mkdir /etc/systemd/system/zfs-import-scan.service.d
echo "[Unit]
After=cryptsetup.target" >/etc/systemd/system/zfs-import-scan.service.d/cryptsetup.conf

Last edited by float (2014-09-04 17:41:50)