[SOLVED] Replacing existing GPG-LUKS-ZFS system

Orionis · 2018-01-19 21:06:37

I have a ridiculously daunting task ahead of me. I have a Gentoo system that is so far out-of-date that it can no longer be salvaged. It runs, I just can no longer update it. Rather than reinstalling Gentoo, my experiences with Arch on the Raspberry Pi make me feel I should switch to Arch.

But here's the problem:
- /boot is on a USB stick.
- The initramfs unlocks two system drives which are LUKS-encrypted with a GPG-encrypted keyfile on said USB stick.
- These two (now unlocked) LUKS devices form a ZFS mirror that gets imported to enable root and swap.
- After that, the changeroot takes place and the system comes online.

So far, all the instructions I can find to create a setup like this are years old. I need help.

My thinking is to first create an ArchIso with ZFS support so I can do the install. I can probably figure that one out.
I could, for the time being, decrypt the GPG-keyfiles or use a passphrase to unlock the volumes to lower the complexity. Although I'd love to get the GPG part back at some point.

My ZFS pool currently contains filespaces for / , /home, /usr/src and some for portage (Gentoo-specific). I could rename those and create new ones for the Arch.

Another problem is that there is also data in the same pool that hosts my system that is not as easy to backup, so I'd need to attempt to continue using the existing ZFS pools.

Did I mention I needed help? Anything you can offer would be welcome. I document everything, so I could write a new guide for it afterwards.

PS No uEFI, just basic BIOS.

Last edited by Orionis (2018-02-04 21:49:06)

Tarqi · 2018-01-20 00:20:01

This might help: Installing Arch Linux on ZFS and Using GPG, LUKS, or OpenSSL Encrypted Keyfiles.

Orionis · 2018-01-20 09:29:12

You guys really don't make this easy, never posting anything more than once, preferring instead to just link to one wiki page to the next trusting I'd catch everything. It would have been nice to at least have things like enabling dhcpcd and openssh as well as grub in the install guide.

Anyway, now I need to get around this one while building the archiso:

error: failed to update archzfs (invalid or corrupted database (PGP signature))

null · 2018-01-20 09:57:23

That's part of the Arch way. You (usually) don't get general advice on how to install your system here but only answers to specific problems. For everything else there is the very good wiki which is always worth a look. You never said that you need ssh or dhcp running, how should we know? There are a lot of users here who don't require neither. The bootloader on the other hand is mentioned in the official installation guide, which you should have read if this is your first Arch installation

Did you import the maintainers key? If so, how is your pacman.conf looking?
https://wiki.archlinux.org/index.php/Un … es#archzfs

I've also got a couple of computers that run zfs in luks encrypted devices and it is very possible to do that with the documentation in the wiki. But I'll be very happy to help if you've got specific questions like this one.

Orionis · 2018-01-20 10:35:11

Yes, the boot loader is indeed mentioned. Then after three clicks leading me to two different pages, I can see grub's grub-install command, 4 clicks for grub-mkconfig, and that's not mentioning that in grub's case there's a need for an extra partition. It's not that I didn't manage, just that it took me unneeded extra time. I had to search through almost a dozen pages to discover the 5 commands to make it work. Anyway, I'll adapt, I always do.

After Googling, I first tried:

pacman-key --init && pacman-key --populate archlinux

and

SigLevel = Never

After your question about importing the key, I found and followed the instruction at https://wiki.archlinux.org/index.php/Pa … icial_keys.
ArchISO is building now.

Still, same issue in this case. There are instructions on how to add ZFS to archiso, but nowhere does it mention the key needs importing. The Arch way is frustrating...

Thank you for providing me with the google keywords (arch import repo key) that led me to the correct page. Trust me, I will do some searching for myself before posting, but I'd appreciate if you checked this thread every now and then. I'll be as specific as I can.

Last edited by Orionis (2018-01-20 10:35:51)

null · 2018-01-20 11:34:31

That's probably because there a so many boot loaders and there is no preference on which to use. I went to syslinux from grub because I find it to be much simpler to configure by hand and I think a lot of people are using gummiboot/systemd-boot for EFI enabled systems.. Not to have default packages means that the users need to know which one they want to use (or figure it out).

The missing part about adding the key is probably because zfs is not a common choice for a file system here and additionally not anyone builds their own ArchISO. I never did.. But don't be harsh on the wiki for this. In most cases it is one of the best sources of general troubleshooting with Linux systems in my experience.

Good luck with the installation!

Trilby · 2018-01-20 12:33:30

Orionis wrote:

You guys really don't make this easy, never posting anything more than once, preferring instead to just link to one wiki page to the next trusting I'd catch everything.

We're not trusting that you are some sort of master sleuth. We are trusting (or rather expecting and requiring) that you are literate, and willing to not entirely disengage your brain when setting up your system.

Orionis wrote:

Then after three clicks leading me to two different pages, I can see grub's grub-install command, 4 clicks for grub-mkconfig, and that's not mentioning that in grub's case there's a need for an extra partition. It's not that I didn't manage, just that it took me unneeded extra time.

It is needed. You need to understand, not blindly copy and paste instructions.

If reading a couple paragraphs, learning, and clicking a couple links is too much of a strain, you really need to rethink what you are doing. You want easy? Here, one click:

https://www.ubuntu.com/download/desktop

I find this frustration with having to follow good and complete documentation partiuclarly odd given your, useless and hyperbolic, thread title thinking you have some special elaborate and unique needs. Zfs can be a bit tricky on linux - but it's gaining good community support now with it's own repos and such. But beyond that, there's nothing so "complex" about your setup that would not have been addressed by properly following the standard installation guide in our wiki.

Last edited by Trilby (2018-01-20 12:41:02)

Orionis · 2018-01-20 14:10:15

If I insulted you, I apologize. I think the wiki is extensive and well-maintained and one of the better ones I've seen on any distribution of Linux. It's just frustrating sometimes to find everything one needs. I had that on the R-Pi installs and I experience it now. It may keep you from having to field n00b-questions, but it also scares people away that could be great contributors in the future.

As for the thread title, as @null mentions, ZFS is not a common choice. Nor are GPG-coded keyfiles. I can find guides on all the topics, but now I have to blend them all together. The last time I did that, I still had to design and build the kernel and initramfs manually. The software has grown since then and I have some catching up to do. It may not be complex to you, but to me it is.

TL;DR:
I'm a Windows admin first, VMware second and Red Hat third. I'm very good at it. I prefer working without a GUI and am an expert in PowerShell and decent with Python. I built several Linux NAS systems from source because commercial solutions didn't do everything I wanted. Then I got swamped with work and now one of those is near death because I didn't update it every month, hence my self-imposed need for expediency.

That said, to learn any new distribution of Linux, I start out with the most basic setup, creating a step-by-step guide for myself, then start adding complexity. It wouldn't hurt to have a basic step-by-step page that just states 'do this and then that', that uses commonly used packages like dhcpcd, grub, openssh, but has links to alternatives for further exploration. Being able to successfully install any configuration gives one the confidence to experiment further.

Anyway, back on topic. I have now done (and documented) a basic setup. From that, I have compiled a ZFS archiso and booted from it. Next is an unencrypted ZFS setup.

When it comes to partitioning, am I correct in this:
Grub2 has native support for encrypted boot and for ZFS boot, but not when used together?
In other words, I should create /boot on a separate partition, which I may or may not choose to encrypt, but I can't place /boot inside ZFS if said ZFS will be on top of LUKS?
Mainly because I would need a readable initramfs somewhere along the way for decrypting the ZFS.

Scimmia · 2018-01-20 17:05:07

The issue here is that you did not ask a specific question. Nobody's going to sit here and type out a step by step guide for you, pointing you to the documentation is the best answer possible. When you run into a specific roadblock, you ask a question and we go from there.

graysky · 2018-01-20 17:08:11

Back up your data to an external drive, wipe your complex setup, repartition, and restore the data.

jasonwryan · 2018-01-20 17:45:35

The thread title says "complex system" and yet you are complaining there isn't a copy and paste guide for you?

it also scares people away that could be great contributors in the future.

This is a fallacy. Arch is no different from any other community: there is nothing magical about it that can transform someone that lacks basic motivation (to read the wiki) into an active contributor.

Orionis · 2018-01-20 20:05:22

I apologized for the misunderstanding and will stick to direct questions.

I asked a question about the partitioning scheme. Also, I must have made a mistake somewhere during the install, since I get dropped into busybox on boot. I can then import my ZFS root by doing:

zpool import -R /new_root system

It boots fine after that.

I followed the instructions as best as possible, had to compensate for the zpool being mounted when it wasn't supposed to and the dkms package having no dependency on linux-headers (as stated in the wiki-page).

Can you tell me where I went wrong?

The error in initramfs is:

filesystem 'ZFS=system/ROOT/default' cannot be mounted, unable to open the dataset

My mkinitcpio.conf contains:

HOOKS=(base udev autodetect modconf block keyboard zfs filesystems fsck)

This is my zfs list.

NAME                  USED  AVAIL  REFER  MOUNTPOINT
system               3.02G  11.9G    24K  none
system/ROOT           912M  11.9G    24K  none
system/ROOT/default   912M  11.9G   912M  /new_root/
system/data           235K  11.9G    24K  none
system/data/home       24K  11.9G    24K  legacy
system/data/var_log   187K  11.9G   187K  /new_root//var/log
system/swap          2.13G  14.0G    16K  -

The associated entries in mount are:

system/ROOT/default on / type zfs (rw,noatime,xattr,noacl)
system/data/var_log on /var/log type zfs (rw,xattr,posixacl)
system/data/home on /home type zfs (rw,noatime,xattr,noacl)
/dev/sda2 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)

And my fstab:

system/ROOT/default     /               zfs             defaults,rw,xattr,noacl,noatime 0 0
UUID=53d8c1da-2fd5-427a-acb1-bf7ba6c3e888       /boot           ext2            rw,relatime,block_validity,barrier,user_xattr,acl       0 2
system/data/home        /home           zfs             defaults,rw,relatime,xattr,noacl,noatime        0 0
/dev/zvol/system/swap                   none            swap            defaults,pri=-2,discard 0 0

Anything else you need?

Last edited by Orionis (2018-01-20 20:47:39)

Trilby · 2018-01-20 20:49:35

Unfortunately I don't use zfs and am unable to help - but I will note that this last post seems much more likely to result in productive solutions.

However, to ensure that those with good zfs experience see this you may still want to edit the title to mention zfs specifically, possibly "ZFS on LUKS cannot be mounted" or "LUKS on ZFS..." whichever is more accurate : you can do this by clicking "edit" on your first post.

Orionis · 2018-01-21 02:55:55

I tried twice now, just to see if I did something wrong the first time. Same result. It seems as if the initramfs is trying to mount the root filsystem before importing the ZFS pool.

The hooks are set in the correct order (see above), so what's going wrong?

null · 2018-01-21 10:08:28

Please post your bootloader configuration

Orionis · 2018-01-21 11:46:29

I managed to make it work by using a simple entry into grub like this:

# (0) Arch Linux
menuentry "Arch Linux" {
    linux /vmlinuz-linux zfs=primary/system/archzfs/ROOT/default rw
    initrd /initramfs-linux.img
}

The entry that grub self-generated (and removing "root=" as the guide states) didn't work.

menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-92ba58723f6f1d2c' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  463b593f-38e7-46ff-93eb-5370647059f8
        else
          search --no-floppy --fs-uuid --set=root 463b593f-38e7-46ff-93eb-5370647059f8
        fi
        echo    'Loading Linux linux ...'
        linux   /vmlinuz-linux ZFS=primary/system/archzfs/ROOT/default rw
        echo    'Loading initial ramdisk ...'
        initrd  /initramfs-linux.img
}

EDIT: Okay, it appears to work if I make ZFS int lowercase zfs. Silly, goofy, grub. I'm beginning to understand why people use other boot loaders. Although I wonder if any other boot loaders can load an encrypted /boot, should I choose to use that.

Last edited by Orionis (2018-01-21 11:49:18)

null · 2018-01-21 14:59:04

I don't think that there are other boot loaders that can enrypt /boot. But I've seen very simple grub configurations. Maybe you'll be better off not using auto-generated stuff from grub at all? I did this a short time before I changed to syslinux and found it much easier to deal with.

Glad you could figure this out. Please mark your thread as solved if everything works now.

Orionis · 2018-01-21 15:45:40

I'm not there yet. I have a basic setup, a ZFS setup, now I need a LUKS-ZFS setup. It's why I was a bit reluctant to make the thread topic about any one thing. But making a new thread for every issue would have me repeat the end goal in every thread, hence the one topic.

I'm going to add LUKS in now.

By the way, I agree grub2 is complex, almost everybody does. It has a lot of possibilities though.

Orionis · 2018-01-24 18:04:53

I managed to get ZFS on top of dm-crypt. But it throws me into the rescue shell because although '/' was mounted in the correct location, the other mount points (/var, /usr, /opt & /home) were mounted in the root of the rescue shell instead of the /new_root folder. So no init-binary and a kernel panic. I will try again, maybe I did something incorrect during initial deployment.

Orionis · 2018-01-26 19:36:44

After a lot of re-installs it appears that even with the usr-hook, ZFS won't mount /usr. It has to be done using a legacy-mountpoint in combination with fstab.

Furthermore, the initramfs fills /var before ZFS has the chance tomount it. So /var also needs a legacy mount.

So far /home, /root and /opt seem unaffected, ZFS is able to mount them.

Now I have found instructtions spread across the internet where all ZFS datasets are defined as "legacy", it just doesn't state anywhere what the purpose is. I figured that ZFS native mounts would be preferable over legacy mounts. Guess I found the exception.

Now I have to try installing it correctly in a single shot, ZFS on top of LUKS. Then it's time to add the keyfiles.

Orionis · 2018-01-30 08:00:43

Okay, so encrypted keyfiles are more difficult than I was hoping for. There is still no way around the GnuPG 1/2 issue that requires GnuPG 1.x, and an OpenSSH keyfile won''t work on a newer version of OpenSSH either. Maybe I should just leave the keyfile unencrypted, use a LUKS-encrypted /boot and turn off automount for /boot, assuming Arch support not having /boot mounted.

By the way, is there any way to stop pacman from running mkinitcpio every time I install a package that affects the kernel? I prefer to run it manually afterwards when I'm installing multiple such packages.

Also, is there a way to run sudo from a chrooted environment? Otherwise I can't install packages from AUR until I'm booted in the new install, since makepkg doesn't run as root.

null · 2018-01-30 10:02:22

Mkinitcpio:
There doesn't seem to be a pacman option but you can probably use "--hookdir <dir>" (pacman(8)) and overwrite the kernel hooks from /usr/share/libalpm/hooks/ in another directory.

Sudo:
You can install sudo in the environment and just run it. Why shouldn't you? But you really don't need to: just build the package and install with "pacman -U /path/to/package.pkg.tar.xz"

If you have this mkinitcpio problem only because you install all those aur packages after another I would recommend just to build them and then install them by pacman -U all together.

Last edited by null (2018-01-30 10:02:37)

Orionis · 2018-02-04 21:45:01

Thanks for pointing me in the right direction. It's the "90-linux.hook" file in that directory, which keeps getting triggered while I'm staging. I suppose I could temporarily remove it, and put it back when I'm done.

As for sudo, the error has something to do with tty not being correct in chroot, making it impossible to ask for a password. So it fails. I did as you said and split the process into "makepkg" as a user and "pacman -U" as root. I'm learning as I go.

Another unfortunate thing is that the automated dkms process keeps building zfs before spl, even though the former depends on the latter. So it ends up only building spl. Strange it doesn't pick up on that dependency. No matter, manually running the module build a second time will build the other one.

Anyway, I managed to get the whole thing to work on a VM without depending on older versions of GPG or OpenSSL by encrypting the cryptsetup keyfile in another (password protected) dm-crypt container in combination with a custom hook. And of course that itself is in an encrypted partition that can be opened through grub. A nice proof-of concept, and it fits my needs. Although I have yet to install it outside of a virtual environment, I now have the process completely documented. As such, I can mark this topic as solved.

Thank you for answering my queries, null. Although I would have found the answers eventually in the Arch-way, it would have taken so much more time, time which I simply don't have available.

Last edited by Orionis (2018-02-04 21:46:36)

Arch Linux

#1 2018-01-19 21:06:37

[SOLVED] Replacing existing GPG-LUKS-ZFS system

#2 2018-01-20 00:20:01

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#3 2018-01-20 09:29:12

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#4 2018-01-20 09:57:23

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#5 2018-01-20 10:35:11

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#6 2018-01-20 11:34:31

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#7 2018-01-20 12:33:30

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#8 2018-01-20 14:10:15

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#9 2018-01-20 17:05:07

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#10 2018-01-20 17:08:11

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#11 2018-01-20 17:45:35

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#12 2018-01-20 20:05:22

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#13 2018-01-20 20:49:35

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#14 2018-01-21 02:55:55

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#15 2018-01-21 10:08:28

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#16 2018-01-21 11:46:29

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#17 2018-01-21 14:59:04

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#18 2018-01-21 15:45:40

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#19 2018-01-24 18:04:53

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#20 2018-01-26 19:36:44

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#21 2018-01-30 08:00:43

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#22 2018-01-30 10:02:22

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

#23 2018-02-04 21:45:01

Re: [SOLVED] Replacing existing GPG-LUKS-ZFS system

Board footer