Arch Linux

NachoGoro

Member

Registered: 2020-10-31

Posts: 24

[SOLVED] Can't decrypt ZFS pool after unexpected poweroff

Yesterday power went off in my house while pacman was installing some updates. Now I am trying to boot again and I get to the point when it asks for my decryption password for my non-boot partition, and I get a "Failed to start Load encryption keys service". It happens regardless of whether I boot with my linux or linux-lts kernel.

The installation is 3 years old, more complex than it should have been for my skill level, and ran smoothly since then, so I haven't had to boot into archiso basically in years. This all means I am struggling quite a lot to do the first basic steps in a reparation process.

My intention is to chroot into the system and downgrade all packages from yesterday to see if that fixes anything, but I can't figure out how to mount the drives I have due to my boot partition being in a RAID1 array and the rest in a ZFS pool.

I have two separate drives, nvme0n1 and nvme2n1. Each has three partitions: 2GB at the beginning, a 60GB partition called md127 of type RAID1 which should hold /boot and a third partition of 870GB used for a ZFS pool with all non-boot stuff.

I tried first mounting /boot with

mdadm --assemble /dev/md127 /dev/nvme0n1p2 /dev/nvme2n1p2

, but I get: "mdadm: /dev/nvme0n1p2 is busy - skipping" for both partitions.
I also tried importing the ZFS pool with

zpool import -d /dev/disk/by-id/nvme-eui.... tank

and getting "Cannot import 'tank': no such pool available".

The ZFS pool should be protected with a password. I have a backup of the key from back when I installed the system, in case it's necessary. I unfortunately don't remember all of the details of the technical decisions taken while installing, but I hope we can figure it out by prodding around the system once I can actually mount it!

Thanks.

Last edited by NachoGoro (2024-01-30 19:20:23)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,637

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

Your iso is up to date? lsblk, cat /proc/partitions, cat /proc/mdstat, anything in dmesg, ...?

NachoGoro

Member

Registered: 2020-10-31

Posts: 24

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

Here's the information as outputted from archiso:

# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0 693.4M  1 loop  /run/archiso/airootfs
sda           8:0    0   3.6T  0 disk  
└─sda1        8:1    0   3.6T  0 part  
sdb           8:16   0 447.1G  0 disk  
└─sdb1        8:17   0 447.1G  0 part  
sdc           8:32   1  57.8G  0 disk  
├─sdc1        8:33   1   798M  0 part  
└─sdc2        8:34   1    15M  0 part  
nvme2n1     259:0    0 931.5G  0 disk  
├─nvme2n1p1 259:2    0     2G  0 part  
├─nvme2n1p2 259:3    0    60G  0 part  
│ └─md127     9:127  0    60G  0 raid1 
└─nvme2n1p3 259:4    0 869.5G  0 part  
nvme0n1     259:1    0 931.5G  0 disk  
├─nvme0n1p1 259:5    0     2G  0 part  
├─nvme0n1p2 259:6    0    60G  0 part  
│ └─md127     9:127  0    60G  0 raid1 
└─nvme0n1p3 259:7    0 869.5G  0 part  
nvme1n1     259:8    0 931.5G  0 disk  
├─nvme1n1p1 259:9    0    16M  0 part  
├─nvme1n1p2 259:10   0 930.9G  0 part  
└─nvme1n1p3 259:11   0   639M  0 part  

sda is a separate only-data drive, and I'm 99% sure nvme1n1 and sdc are separate Windows drives.

# cat /proc/partitions
major minor  #blocks  name

   8        0 3907018584 sda
   8        1 3907016704 sda1
   8       16  468851544 sdb
   8       17  468797440 sdb1
 259        0  976762584 nvme2n1
 259        2    2097152 nvme2n1p1
 259        3   62914560 nvme2n1p2
 259        4  911749831 nvme2n1p3
 259        1  976762584 nvme0n1
 259        5    2097152 nvme0n1p1
 259        6   62914560 nvme0n1p2
 259        7  911749831 nvme0n1p3
 259        8  976762584 nvme1n1
 259        9      16384 nvme1n1p1
 259       10  976089256 nvme1n1p2
 259       11     654336 nvme1n1p3
   8       32   60653568 sdc
   8       33     817120 sdc1
   8       34      15360 sdc2
   7        0     710000 loop0
   9      127   62880768 md127

# cat /proc/mdstat
Personalities : [raid1] 
md127 : active (auto-read-only) raid1 nvme0n1p2[0] nvme2n1p2[1]
      62880768 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

dmesg is in Pastebin.

Also, trying to mount md127 spits this:

mount /dev/md127 /mnt/boot
mount: /mnt/boot: unknown filesystem type "crypto_LUKS".

Ta!

Last edited by NachoGoro (2023-07-22 09:23:19)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,637

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

your md raid seems fine, it's LUKS encrypted so... cryptsetup open it?

maybe your ZFS is LUKS encrypted too? then you'd likewise have to cryptsetup open before you can proceed.

blkid, file -s /dev/nvme* might tell you more

once you can mount everything, you should be able to arch-chroot in and run pacman to fix things from the botched update

NachoGoro

Member

Registered: 2020-10-31

Posts: 24

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

I have been able to mount my /dev/md127 partition by doing:

# cryptsetup luksOpen /dev/md127 my_encrypted_partition
# mount /dev/mapper/my_encrypted_partition /mnt/mypoint

Turns out it isn't the /boot partition, but basically everything else apart from /boot and /home.

EDIT: Missed the previous post, looking into that...

Last edited by NachoGoro (2023-07-22 09:34:53)

NachoGoro

Member

Registered: 2020-10-31

Posts: 24

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

I didn't need to do anything with my ZFS pool, md127 was the partition I could chroot into. After chrooting in, I changed mirrorlist to point to last week and ran pacman -Syyuu, which downgraded a bunch of packages and kernel modules (in particular ZFS ones).

Still no luck, though. After rebooting into my system I get the following:

A password is required to access the cryptroot volume:
Enter passphrase for /dev/md127: [I ENTER IT]
rootarray: clean, 586144/3932160 files, 12350037/15716096 blocks
[FAILED] Failed to mount /boot.
[DEPEND] Dependency failed for Local File Systems.
[FAILED] Failed to start Load encryption keys.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):
[root@socrates ~]#

The "Load encryption keys" service is the one which loads all keys from this section: https://wiki.archlinux.org/title/ZFS#Un … e:_systemd

Manually running

/usr/bin/zfs load-key -a

to see what goes on, it says that the ZFS modules are not loaded.

Last edited by NachoGoro (2023-07-22 09:59:17)

NachoGoro

Member

Registered: 2020-10-31

Posts: 24

Re: [SOLVED] Can't decrypt ZFS pool after unexpected poweroff

Solved! I think when I reverted back to an arbitrary point in the past week, I somehow got some strange state with the kernel version (as there was no directory with my current kernel version in /lib/module/). I re-chroot'ed into the system again and re-run pacman against the latest mirrorlist, and that fixed it.

For future reference to myself:
* The 2G partition in nvme0n1 (nvme0n1p1) is /boot (and seems to be wasted in the other drive);
* The 60 GB partition in both disks is a RAID1 array (encrypted with LUKS with a passphrase), for /;
* The 870GB partition in both disks is an encrypted ZFS pool (tank, with datasets tank/data and tank/data/home), which take the hex key from the / filesystem.

Thank you very much for all your help, frostschutz.