Multiple Disk Encryption with single pass phrase at boot?

oni5115 · 2014-03-19 03:01:43

I have been working on setting a home NAS/MythTV server, and learning about encryption while I do it. I've read most of the pertinent Arch Wiki articles, including: DM-Crypt, Partitioning, File Systems, ZFS Installation, probably a few I missed. I have also read some other pages about setting up Linux systems with ZFS / RAID / Encryption. In particular, the following two articles:

http://www.0x00.to/post/2013/04/28/Buil … Arch-Linux
https://help.ubuntu.com/community/encryptedZfs

My first attempt, using a USB stick for the OS, was rather sluggish and was no where near as user friendly as I thought Ubuntu would be. While reading up on LUKS I also found out that Ubuntu installers apparently have a tendency to destroy LUKS headers... So I decided I would use Arch.

What I have right now:
64 Gb, SanDisk SSD, x1
3 TB, WD RED, x7

The intent, is to run Arch on the SSD. ZFS Raid10, 6 of the 3TB drives, and keep 1 hot spare.

I'd set it up that way on the USB stick previously, and it was working quite well, sluggishness aside. However, I did not encrypt the OS, only using the Ubuntu installers default of encrypting the home partition. This time, I'd like to fully encrypt the Arch install, and see if there is a way to have it automatically decrypt the other 7 drives, and mount the ZFS pool. The Ubuntu guide really only confused me, so I am hoping someone can clarify how to setup derived keys (correct term?) to automatically unlock the other drives (if possible).

My understanding is that I will need to partition my SSD with:
/boot - 200+ Mb, non-encrypted, boot-able partition.
/ - encrypted partition on the rest of the SSD.

I am not planning to mess with using separate partitions for /home, /var, /etc. KISS. All the real data will be stored in the ZFS raid. So the only writes to the SSD will be Arch package upgrades, configuration changes, and anything MythTV / MythWeb save/change/store. (I intend to configure as much as of that to save in the zpool as I can.) I expect the SSD should last a while in this setup, correct?

So TLDR;
What filesystem should I use for /boot ? [ext2?]
What filesystem should I use for / ? [ext4?]
How do I setup Arch to decrypt 8 drives with 1 pass phrase prompt? (Or, is that impossible?)

gegenschall · 2014-03-19 10:24:21

Yes, it is possible. Use ext4 or btrfs on / and /boot. Do not encrypt /boot. Encrypt / with a password.
For your 8 drives you may do the following:
Encrypt using a password and use another keyslot with a keyfile, s.t. you have either 8 individual keyfiles (i.e. /etc/keys/sd[b,c,d,...].key) and your /etc/crypttab points to those keyfiles. If you unlock your / partition at boot, all other drives (as mounted per systemd or fstab) will also be unlocked.

But read up on that in the wiki: Cryptsetup and keyfiles

Slithery · 2014-03-19 11:18:46

I'd question the choice of running the RAID10 setup you've outlined above, as you're only guaranteed to be able to lose 1 drive. If the wrong 2 drives fail then you've lost your array.

If you went for a RAIDZ3 with all 7 drives not only would you get more space, but you could also have any 3 drives file and still have a working array.

Do you have a backup strategy in place for your 9TB of data?

Do you have enough RAM (a rule of thumb for ZFS is 1GB RAM per TB of data)?

Last edited by Slithery (2014-03-19 11:40:53)

frostschutz · 2014-03-19 12:33:08

For 7 disks, RAID5 is fine, RAID6 is luxury, and RAIDZ3 is wasteful. Test your disks regularly for read errors, reallocated sectors and the like, so you won't get surprises during rebuilds.

As for decryption in initramfs, I did it like this (but that's Gentoo):

https://wiki.gentoo.org/wiki/Custom_Ini … ed_Keyfile
https://wiki.gentoo.org/wiki/Custom_Initramfs/Examples

In that example, md0 is my SSD - it's RAID1 only so I can mirror it to HDD-LV from time to time.

md1-md8 is a 7x2TB disk RAID5, each disk with 8x 250G partitions. I find several smaller RAID to be easier to handle than one big one; if a disk develops bad sectors in a specific region, the other regions may stay redundant; accidental resyncs (after a power loss) finish faster as it usually doesn't have to resync all of them; grows, checks etc. can be distributed over several days, etc. And it doesn't have the performance penalty of a write intent bitmap.

Using crypttab and keyfiles on / would certainly be the simpler solution, if you have nothing relevant to root / booting on those HDDs.

Last edited by frostschutz (2014-03-19 12:34:42)

Strike0 · 2014-03-19 18:50:19

frostschutz wrote:

[snip]
As for decryption in initramfs, I did it like this (but that's Gentoo):
https://wiki.gentoo.org/wiki/Custom_Ini … ed_Keyfile
[snip]
Using crypttab and keyfiles on / would certainly be the simpler solution, if you have nothing relevant to root / booting on those HDDs.

For above bit, I'd be +1 for the standard method of using crypttab with keyfiles on the encrypted root to unlock the non-root disks. On top of the link Gegenschall quoted for the keyfiles, look at this one how to use crypttab: https://wiki.archlinux.org/index.php/Dm … #Partition
The Gentoo example is a nice hack, but what does it really do being non-standard? It encrypts your key-files in the initramfs, so the entropy of the key.luks passphrase decides key-security for the rest. I'd argue If you use the same entropy for your root passphrase, the hack brings nothing but obfuscation and rolling-release troubles.

oni5115 · 2014-03-19 23:22:43

Thank you for the responses. I will check out the wiki on crypttab, cryptsetup, and keyfiles.

I went for RAID10 for 2 main reasons.
1: If I went RAIDZ3, I wouldn't really gain much/if any space. IRRC, the break even point was between 6-8 3TB drives before you started to gain drive space.
2: If I went RAIDZ3, it would take a very long time to rebuild the array, possibly causing another drive to die from the increases stress during the rebuild.

I've read a few anecdotes about raid2/3 arrays taking days to recover. With the RAID10, rebuilds should be rather quick (comparatively speaking).

I do plan to setup a backup drive in addition to the raid, at least for critical files. I can live if I lose last night's hockey game, or some TV season episodes, etc. But the family photos, receipts. etc, will all be backed up to another drive. That will be my next project once I get the main array setup; learning snapshot's, backups, etc, using ZFS.

frostschutz · 2014-03-19 23:46:26

The Gentoo example is a nice hack, but what does it really do being non-standard?

It's an USB stick (so /boot can't be tampered with as long as you take it with you) with encrypted keyfiles (so a hardware keylogger alone isn't sufficient, as it only gives you the keyfile passphrase, not the HDD itself). Gentoo doesn't have crypttab anyhow.

oni5115 wrote:

possibly causing another drive to die from the increases stress during the rebuild.

"stress during rebuild" is largely a myth. A rebuild is a linear read (for the old disks) and a linear write (for the new disks). Apart from being idle, that's the most boring operation a disk can possibly have. At the same time you have disks in database servers that are hot with random I/O 24/7 and they last years without dying. But that's not the kind of setup you find at home.

If you have an error during rebuild, maybe you just had bad luck; or maybe you just never tested your disks for read errors before. Or you didn't replace disks despite them having reallocated sectors and such.

oni5115 wrote:

I've read a few anecdotes about raid2/3 arrays taking days to recover. With the RAID10, rebuilds should be rather quick (comparatively speaking).

Rebuilds take long regardless of raid level. It depends on speed/size of the disk; it takes a while for a 3TB disk to be fully rewritten.

Maybe ZFS is faster, if it just resyncs individual files instead of "everything". Maybe ZFS is better at resuming, if you want to shut down the box over night. I don't have any experience with ZFS so I don't know. Regarding encryption that setup sounds strange to me. Especially if you intend to encrypt each disk individually - you'll end up encrypting data twice then, that should cost you write performance (unless you have an AES-NI box where you don't notice encryption overhead).

Arch Linux

#1 2014-03-19 03:01:43

Multiple Disk Encryption with single pass phrase at boot?

#2 2014-03-19 10:24:21

Re: Multiple Disk Encryption with single pass phrase at boot?

#3 2014-03-19 11:18:46

Re: Multiple Disk Encryption with single pass phrase at boot?

#4 2014-03-19 12:33:08

Re: Multiple Disk Encryption with single pass phrase at boot?

#5 2014-03-19 18:50:19

Re: Multiple Disk Encryption with single pass phrase at boot?

#6 2014-03-19 23:22:43

Re: Multiple Disk Encryption with single pass phrase at boot?

#7 2014-03-19 23:46:26

Re: Multiple Disk Encryption with single pass phrase at boot?

Board footer