Encrypting ZFS (with LUKS)

hobarrera · 2013-11-11 05:37:43

I'm about to set up a home NAS/media centre, and I'm considering using ZFS.
I'm pretty familiar with LUKS already, so my intention was to use LUKS+ZFS.

My doubt however, is how to handle this: unlike LVM (which I've used up to now), ZFS isn't a container, but a proper filesystem, so I can't create a huge ZFS pool, and a single encrypted volume inside.
Is my only choice to create a single encrypted LUKS volume in each drive, unlock them, and add each unlocked volume to the ZFS pool?

Since ZFS handles mounting by itself upon startup, but LUKS will most likely start unlock the volumes once the OS is already starting user-space services, will ZFS kick in by itself, or will I have to hack something around this?

Is there any saner choice? Four individual encrypted volumes just sounds like a hack, IMHO.

Finally, has anyone done something similar, and do you have any advice for me?

WonderWoofy · 2013-11-11 06:18:15

While I think taht zfs is awesome, and works quite well, I also think that using zfs on Arch specifically is kinda shaky. This is because it does not enjoy official support in the repos. So when there is a kernel update, you have to rebuild the spl and zfs modules manually, which lead to circular dependencies (or at least used to, I haven't tried in a while). Demizerone was running an unofficial repo for a while, and maybe it still exists. I used it for a bit, but decided that it despite his pretty great upkeep, there were still points at which the kernel would be updated and his repo would follow a couple days later. This, to me, is just asking for trouble.

But to answer your question more directly, you have the right idea about how you would need to do that. You would encrypt each device and then add each unlocked device to the pool. This is not as problematic as it may seem. As long as you don't run rootfs off the zfs pool, all should be fine. The encrypted containers will be decrypted by systemd automatically during boot, the zfs pool will be assembled, and it will be mounted.

If I'm not mistaken, although there is the ability to use zfs automounting magic, this is not actually a requirement. I seem to recall being able to use the fstab as normal if you so desired.

The problem with running zfs over multiple encrypted devices for root would not be with zfs itself, but rather that the current state of the mkinitpcio encrypt hook is only able to decrypt one disk/partition, so if you have it over several devices... no bueno.

Still it has been some time since I have played around with zfs, so maybe someone else who is a currrent user might be able to step in and give you more precise answers.

hobarrera · 2013-11-11 06:59:43

First off, thanks for all your feedback!

WonderWoofy wrote:

While I think taht zfs is awesome, and works quite well, I also think that using zfs on Arch specifically is kinda shaky. This is because it does not enjoy official support in the repos. So when there is a kernel update, you have to rebuild the spl and zfs modules manually, which lead to circular dependencies (or at least used to, I haven't tried in a while). Demizerone was running an unofficial repo for a while, and maybe it still exists. I used it for a bit, but decided that it despite his pretty great upkeep, there were still points at which the kernel would be updated and his repo would follow a couple days later. This, to me, is just asking for trouble.

I think I can handle this by having some sort of monitor build rebuild the packages whenever there's a kernel update (BEFORE I install the update!). Cron and some scripting will probably fix this, or at least email me in case of required attention.

This is a home media server/nas, so, while keeping data intact is important, a short downtime while upgrading is not a critical issue.

WonderWoofy wrote:

But to answer your question more directly, you have the right idea about how you would need to do that. You would encrypt each device and then add each unlocked device to the pool. This is not as problematic as it may seem. As long as you don't run rootfs off the zfs pool, all should be fine. The encrypted containers will be decrypted by systemd automatically during boot, the zfs pool will be assembled, and it will be mounted.
If I'm not mistaken, although there is the ability to use zfs automounting magic, this is not actually a requirement. I seem to recall being able to use the fstab as normal if you so desired.
The problem with running zfs over multiple encrypted devices for root would not be with zfs itself, but rather that the current state of the mkinitpcio encrypt hook is only able to decrypt one disk/partition, so if you have it over several devices... no bueno.
Still it has been some time since I have played around with zfs, so maybe someone else who is a currrent user might be able to step in and give you more precise answers.

It all looks a bit simpler now really, I guess.
I've no intention of keeping the rootfs inside the ZFS volume, so I won't be using the encrypt hook, but rather /etc/crypttab. Having / inside the RAID-Z is not really important for my scenario.

Also, yes, I've dealt with separate LUKS volumes and the encrypt hook, and it's not pleasent at all when you need to unlock several LUKS volumes for boot. I certainly don't wan't to deal with that scenario again.

WonderWoofy · 2013-11-11 16:32:44

I think I eventually resorted to building spl and zfs in a clean chroot for a while. That way you can update the chroot first and don't end up with the circular dependencies.

In fact, I would imagine you could probably create the chroot in your zfs pool and leverage the power of the filesystem to clean the chroot. This is like what devtools does with btrfs where it creates a chroot, then snapshots it to build. When it comes time to update the chroot, it only updates the one to be snapshotted, simply 'btrfs sub del'ing the "dirty" chroot.

Also, I think that putting linux, linux-headers, and any other 3rd party kernel modules into 'IgnorePkg' of pacman.conf would at least print you out a warning when there is a new kernel and headers to be updated that it is skipping. But until pacman hooks are implemented, I think there is no real automated solution for this.

hobarrera · 2013-11-11 17:52:59

WonderWoofy wrote:

I think I eventually resorted to building spl and zfs in a clean chroot for a while. That way you can update the chroot first and don't end up with the circular dependencies.
In fact, I would imagine you could probably create the chroot in your zfs pool and leverage the power of the filesystem to clean the chroot. This is like what devtools does with btrfs where it creates a chroot, then snapshots it to build. When it comes time to update the chroot, it only updates the one to be snapshotted, simply 'btrfs sub del'ing the "dirty" chroot.
Also, I think that putting linux, linux-headers, and any other 3rd party kernel modules into 'IgnorePkg' of pacman.conf would at least print you out a warning when there is a new kernel and headers to be updated that it is skipping. But until pacman hooks are implemented, I think there is no real automated solution for this.

I was unaware of the circular dependency issue.
I like your solution, but in my case, I guess I can:
a) Have ZFS built on a different machine that's not using ZFS and scp the built packages(s).
b) Have a tiny downtime while NAS ZFS builds, which is fine in my scenario (this can even be automated for mondays 6am ).

In any case, for reference, there are some hints for automating this, and an unofficial repo here.

Have you tried zfs-dkms?

WonderWoofy · 2013-11-11 18:18:23

I have not tried the dkms package. I did however try using demizerone's repo for a while. It worked pretty well, and he stays pretty on top of the updates (he runs it himself). But every so often it takes him a couple days to get around to doing the update after a kernel is pushed. Ultimately I stopped using it, and in turn zfs, because I run [testing] and he stopped building [testing] packages some time ago.

Building the packages on a separate machine would be ideal.

Arch Linux

#1 2013-11-11 05:37:43

Encrypting ZFS (with LUKS)

#2 2013-11-11 06:18:15

Re: Encrypting ZFS (with LUKS)

#3 2013-11-11 06:59:43

Re: Encrypting ZFS (with LUKS)

#4 2013-11-11 16:32:44

Re: Encrypting ZFS (with LUKS)

#5 2013-11-11 17:52:59

Re: Encrypting ZFS (with LUKS)

#6 2013-11-11 18:18:23

Re: Encrypting ZFS (with LUKS)

Board footer