[Solved] Luks - Failing to boot

Soultrigger · 2023-12-10 19:34:59

I am posting this from my smartphone, so I cant really post commands results at this time, I did some system updates (at least 3) and after rebooting after a whie (weeks) the system hangs when I put my Luks2 password.

When I insert the password it say

Found device /dev/gpt-auto-root
Reached target Initrd Boot Device
Finished Cryptographic Setup for root
Reached target Local Encrypted Volumes
a start job is running "device description by uuid

And stays there forever, the device by uuid is the swap. (I thought it was the root, but it was actually the swap)

My setup looks like this:

/dev/sda1- EFI
/dev/sda2 - Luks2/BTFS (root on devicemapper)
/dev/sda3 - Arch Rescue (basically a copy of the installation media)

/dev/sdb1 - Luks2/BTRFS (storage on devicemapper)
/dev/sdb2 - Luks2/BTRFS (var on devicemapper)
/dev/sdb3 - Luks2/SWAP (swap on devicemappe)

I also use the HOOK systemd on MKINITCPIO so its inherits the password for all devices.

The password I am inserting is right for sure, if I log in the Arch Resuce I am able to open all devices with cryptsetup luksOpen, and I mount root /mnt and arch-chroot into it and use mount -a to mount all partitions without issues.

I even tried to run pacman -Syu to see if it would fix my boot, but didnt happen.

I even excluded the recovery key and pcr0+7 entry and its tokens from the Luks2 headers to see if I only having the"main/first" password only if it would fix anything, no lucky.

With or without secureboot.

I am at a loss on what to do. (There are no hardware changes or UUID or whatever) and I dont see the logic for it to hang up as if trying to unencrypt the device, even though it gets the right key,-

Last edited by Soultrigger (2023-12-17 15:09:58)

Soultrigger · 2023-12-10 20:28:55

Since it is fhe swap, I arch-chroot into it, I did comment it on crypttab and fstab, but no luck. Also the uuid is not from /dev/sda3, but from /dev/mapper/swap (by this I mean the opened Luks2).

Last edited by Soultrigger (2023-12-10 20:29:20)

Soultrigger · 2023-12-10 22:11:28

I also tried redoing the "mkinitcpio -P" and later "pacman -S linux" to try and recreate the UKI with updated information, but somehow it still tries to mount the swap.

Soultrigger · 2023-12-10 23:21:53

It seems anything I try to install with pacman I get an errir on Refreshing Packagekit, is it related somehow?

Soultrigger · 2023-12-11 12:18:04

Well, I decided it was easier to just reinstall than try to fix this. But after a "clean" install it still asks for the same device with that UUID from the swap partition that I removed the partition to try and see if it was the systemd auto detecting it.

By "clean" I mean that I tried keeping the /etc, but it still was like that, so I removed every single folder and file from the partition except home.

I formatted the EFI to start from scratch

I used efibootmgr to remove the boot entry.

I remounted every folder where it belonged and reisntalled from scratch. (I didnt format my btrfs partitions, but I went in every single subvol, removed all files and even looked for hidden .dotfile .dotdirs)

I pacstrapped, and redid the UKI, systemd-boot install and everything. And it still asked for the swap partition... (I also changed secureboot to setup mode effectively wiping it and redoing secureboot too)

Since there is no old ETC and no old files pointing to it, how can it still asks for something doesnt even exist anymore?

I even umounted home and commented it on fstab, and then tried mkinitcpio -P, sbctl sign-all, bootctl install --esp-path=/efi to recreate the whole "boot" to make sure and it still asks for the swap partition that no longer exists.

I am at a loss after all this, can it be saved in the UEFI somehow for using PCR 0+7 to check if there was UEFI changes? (turn off secureboot or updated the UEFI).

Soultrigger · 2023-12-11 14:48:53

I am using systemd-boot as bootloader, it is not configured in anyway and auto finding the UKIs .efi files/images

The root partition is auto mounted for having the root_x86_64 partition ID.

The other 3 BTRFS partitions are mounted trough /etc/cryptab and they all are just with something similar to "storage UUID=abcdef"

I dont have the crypttab.initramfs (and If I had, the file was deleted when I removed /etc and reinstalled).

The /etc/kernel/cmdline only has "splash" as a command line so that plymouth loads/works.

Is there any way I can inspect the .efi file? and read the fstab or whatever it uses to try and mount volumes?

Last edited by Soultrigger (2023-12-11 14:50:15)

ranurag · 2023-12-12 05:54:00

Have you searched these topics in arch wiki ?

Head_on_a_Stick · 2023-12-12 06:06:01

Remove the swap line from /etc/fstab, systemd will autodetect it on a GPT disk. No idea about LUKS though, I've never used that (sorry).

Soultrigger · 2023-12-12 16:27:02

Thanks for the feedback.

/etc/fstab will have mount points for our regular partitions.

/etc/crypttab will have your crypto partitions which you will choose a "nickname" to mount it later, such as UUID will have the name storage, and later on I will use /dev/mapper/storage to refer to it in fstab.

/etc/crypttab.initramfs can have the same information from the previous one, but will be mounted early trough systemd using UKI (unified kernel images).

The issue is that the UUID it is trying to mount whenever I run the boot process was from a previous partition used as swap that no longer exist.

I reinstalled my system and it still asks for it, and there is no file pointing to it at all.

After doing that I also grabbed another HDD and created a new EFI partition and a new root partition and reinstalled encrypted and non encrypted, and as long as I use systemd hook in mkinitcpio it still tries to look for that partition that no longer exists.

What my logic and limited knowledge guess is that since I used the command:

systemd-cryptenroll --tpm2-device=auto /dev/sdX --tpm2-pcrs=0+7 --tpm2-with-pin=yes

It somehow saved the information for the unlocking in the TPM2? So to understand it supposedly saves the computer state to PCR so it can compare later to create a trusted everionment, PCRS=0+7 should refuse the password if the secure boot was turned off or if the UEFI was updated.

I have no idea why, but my only guess is that the configuration was saved on the UEFI and that information is being recovered by systemd and he is forcefully trying to unlock the swap partition that no longer exists.

I would like to know if there is anyway I could wipe that information on the UEFI, or if anyone has any idea how to fix this or what I could look into? Even wiki articles would help a lot. Right now I am stuck with installation media and cant really run commands to paste the results here.

The only measure of sucess is that if I use udev hook instead of systemd in mkinitcpio I could get the system running, I did a brief test and since I didnt configure sd-boot it stopped in the emergency roofs.

Looking online how to reset TPM2, someone said in a forum that Windows 11 Instalation would wipe it, but I would like to know how to do it from Linux and how to prevent this in the future. (but it is a wide guess, and I might try if I dont find another technical solution trough Linux).

As for my crypto devices, I removed keys and tokens (not trough cryptenroll) associated with TPM2 and recovery key. I just left alone Key slot 0 which I inserted when creating the crypto device header to still have access to those partitions.

I am at a loss on what to do, and I though I was at least good enough on partitions, dm-crypt, linux boot and so on, but nothing like an issue like this to humble us.

Or my computer is possessed by a demonic entity and I have to look for a exorcist instead.

Last edited by Soultrigger (2023-12-12 16:54:02)

seth · 2023-12-12 20:46:35

https://bbs.archlinux.org/viewtopic.php … 9#p2136989 ?

Soultrigger · 2023-12-13 02:02:38

Thanks, you are always so kind.

It is the same issue it seems

If I understood your suggestion, I did add the following line to my fstab:

UUID=92ba8c70-ef4a-4d5f-b591-1363ca1848d7     /mnt   btrfs  noauto

Didnt fix it... The real issue is, I nuked my install and reinstalled, and still asks for it. I did another install on separate storage device, and it still asks for it (I even created a new efi partition on this device to be completely serparate).

I also tried that fstab change without and with mkinitcpio -P to generate the UKIs again.

But this UUID=92ba8c70-ef4a-4d5f-b591-1363ca1848d7 was from a swap, but not the normal UUID, but the UUID it would get when the luks2 is open. (so it is the uuid for /dev/mapper/swap when it still existed and not for the /dev/sdb3 which was the luks2 encrypted partition) But nonethless even with the right password for TPM2 or the original key slot 0 password it would refuse to continue the boot process and unlock it. Even nuking my installation to do it from scratch it still haunts me...

Is there anyway to wipe clean whatever systemd saves in the UEFI/TPM2? That seems the solution for me. (Because I dont see how it is getting this information from anywhere else).

kermit63 · 2023-12-13 02:46:41

How about going the other way and create a swap with the exact same UUID the system is looking for just to test if it will boot all the way through?

seth · 2023-12-13 07:44:56

If that also doesn't work, try to downgrade systemd, it might "detect" a subvolume or something like that.

Soultrigger · 2023-12-13 12:10:18

kermit63 wrote:

How about going the other way and create a swap with the exact same UUID the system is looking for just to test if it will boot all the way through?

I created a new swap partition and changed it UUID with gdisk to the one systemd wants, but when I ran mkswap it generated another UUID. (bothe UUID are valid, the one I changed on gdisk is PARTUUID and the UUID is the one from the swap)This way it didnt work. Maybe if I could manipulate the UUID for an opened Luks2 header that would work, but my wild guess is that since there is no longer a key and a token for TPM2 in the partition it wouldnt work.

Last edited by Soultrigger (2023-12-13 12:12:04)

Soultrigger · 2023-12-13 12:11:14

seth wrote:

If that also doesn't work, try to downgrade systemd, it might "detect" a subvolume or something like that.

I will try to learn that and I will report back as soon as I am able to do it.

kermit63 · 2023-12-13 12:30:53

Soultrigger wrote:

I created a new swap partition and changed it UUID with gdisk to the one systemd wants, but when I ran mkswap it generated another UUID. (bothe UUID are valid, the one I changed on gdisk is PARTUUID and the UUID is the one from the swap)This way it didnt work. Maybe if I could manipulate the UUID for an opened Luks2 header that would work, but my wild guess is that since there is no longer a key and a token for TPM2 in the partition it wouldnt work.

mkswap has a -U option that allows you to set the UUID manually.

seth · 2023-12-13 12:42:46

You'd have to downgrade at least systemd and systemd-libs to the (last) 254 version and can use either the packages from your pacman cache or the https://wiki.archlinux.org/title/Arch_Linux_Archive

Soultrigger · 2023-12-13 14:34:47

I found this in the systemd 255 release notes:

* A new component "systemd-pcrlock" has been added that allows managing
  local TPM2 PCR policies for PCRs 0-7 and similar, which are hard to
  predict by the OS vendor because of the inherently local nature of
  what measurements they contain, such as firmware versions of the
  system and extension cards and suchlike. pcrlock can predict PCR
  measurements ahead of time based on various inputs, such as the local
  TPM2 event log, GPT partition tables, PE binaries, UKI kernels, and
  various other things. It can then pre-calculate a TPM2 policy from
  this, which it stores in an TPM2 NV index. TPM2 objects (such as disk
  encryption keys) can be locked against this NV index, so that they
  are locked against a specific combination of system firmware and
  state. Alternatives for each component are supported to allowlist
  multiple kernel versions or boot loader version simultaneously
  without losing access to the disk encryption keys. The tool can also
  be used to analyze and validate the local TPM2 event log.
  systemd-cryptsetup, systemd-cryptenroll, systemd-repart have all been
  updated to support such policies. There's currently no support for
  locking the system's root disk against a pcrlock policy, this will be
  added soon. Moreover, it is currently not possible to combine a
  pcrlock policy with a signed PCR policy. This component is
  experimental and its public interface is subject to change.

And found this: https://archlinux.org/packages/extra/x8 … ols/files/

And this: https://manpages.ubuntu.com/manpages/fo … set.1.html

Which contains:

    tpm2_pcrreset(1) - Reset one or more PCR banks

So could it be so simple as to run "tpm2_pcrreset 0 7"?

Since you are all far more wise than me, would there be any risks or issues if I reset my pcr states for PCR 0 and 7? (If that wipes out whatever is saved in the nv index)

I am at work right now, so I can only think about solutions.

Last edited by Soultrigger (2023-12-13 14:36:40)

seth · 2023-12-13 15:27:03

I'd look at the first.
https://wiki.archlinux.org/title/Truste … _registers

Soultrigger · 2023-12-13 17:18:23

seth wrote:

I'd look at the first.
https://wiki.archlinux.org/title/Truste … _registers

Humm, even though it checks PCR 0+7 states, it seems the one I would need to reset is the PCR 15?

PCR 15

systemd-cryptsetup optionally measures the volume key of activated LUKS volumes into this PCR. systemd-pcrmachine.service measures the machine-id into this PCR. systemd-pcrfs@.service measures mount points, file system UUIDs, labels, partion UUIDs of the root and /var/ filesystems into this PCR.

Either way, it seems to be far more complex than I guessed. Battery reseting the UEFI might do the trick? Or is this data more persistent?

seth · 2023-12-13 19:53:58

Have you checked "systemd-analyze pcrs" or "tpm2_pcrread" to see whether the UUID shows up anywhere?

Soultrigger · 2023-12-14 13:29:08

seth wrote:

Have you checked "systemd-analyze pcrs" or "tpm2_pcrread" to see whether the UUID shows up anywhere?

Both give similar outputs, but the systemd is more detailed.

PCR 0-9 had data (encrypted I would assume, so I couldnt read anything about the UUID), PCR 10-15 were only zeroes and PCR 16-23 were only Fs;

The tpm2_pcrreset was a complete failure, I tried all slots to reset for the sake of research and I just got all errors and no resets atl all. Probably this is working as intended to dont have external changes for the sake of security. I did run as root and if tpm2_pcrreset does reset something or I used it wrong or it is intended for other scenarios.

About downgrading systemd, it is still something I want to learn and do for the sake of being a better sysadmin someday, but I thought it was not a pragmatic solutions for the future. I would rather wait for a new version with a fix in my mind. But I will dedicate time to learn this because I know it might save me from issues in the future. My guess is that since they changed so much about TPM2 and crypto partitions in release 255 that reverting to 254 would fix it. But I think we got a solution below.

I am thankful for all the input so far, helped me learn and have more options.

Last edited by Soultrigger (2023-12-14 14:14:10)

Soultrigger · 2023-12-14 14:01:50

kermit63 wrote:

Soultrigger wrote:
I created a new swap partition and changed it UUID with gdisk to the one systemd wants, but when I ran mkswap it generated another UUID. (bothe UUID are valid, the one I changed on gdisk is PARTUUID and the UUID is the one from the swap)This way it didnt work. Maybe if I could manipulate the UUID for an opened Luks2 header that would work, but my wild guess is that since there is no longer a key and a token for TPM2 in the partition it wouldnt work.
mkswap has a -U option that allows you to set the UUID manually.

When you suggested it I already though about doing this, but I though it wouldnt work and I didnt know how to manipulate UUID besides gdisk, and it would be a long time before I did deep dive in mkswap command to look on how to change the UUID because I would be fixed on how to manipulate that on a open luks partition.

The first attempt with gdisk was a completely failure just changing the PARTUUID.

But changing the UUID with mkswap -U seems to be a success.

I used the swap space to generate a EFI and a root partition, so I tried to resize the BTRFS partition with fdsik for the sake of research, but it corrupted the filesystem (which I thought it would) and the freed space was created a new swap with the UUID that systemd requested, and it followed the boot procedure, but failed to mount the BTRFS partition.

Thus, I reformated the root partition and pacstraped the new installation, but now it requested the UUID of the previous root, but not as an unlimited time, it just ran for 1:30m and resulted in an error.

So I removed the fstab (that was not there originally, but I created for the sake of testing), and it didnt fix it.

After this I formated the EFI, did bootctl install and remade the UKI with mkinitcpio -P.

And it booted without issues.

After this I did run mkswap again to random the UUID and it booted fine again.

I will have to try to reinstall my system at night, but it seems that it will now be possible. So I am really thankful for your input.

It seems finding a device to mount with the right UUID made systemd "panic" go away.

I thoguht it was locked because of using cryptenroll with PCR 0+7, but it seems it actually used some extra information that might have been saved in PCR 5 wich had the last valid boot and systemd locked over it with release 255 (and may or may not have anything to do with cryptenroll PCR 0+7). I dont know if I am happy or not, because it was supposed to protect a Luks2 partiton, and since the new swap with the same UUID was not locked under a Luk2 header, is it a breach in security?

Eitther way I am happy that this situation let me learn a bit more about Linux, thanks for your time and help.

Thanks to all who did use some of your free time to give some feedback.

I know it was not the proper way of asking for help, that I should have some commands outputs and be more technical about it, but I was limited in resources. I tried to use lynx to post here some commands outputs, but I didnt find a way to run commands within it, even though I could come to the forum fine. (I was hopeful I could use it as nano or vim to run commands to capture its output within the app).

Do you know if there is a terminal browser that would allow to run commands and post here? It might be useful in the future.

I will change to Solved if I can install later today without issues.

Last edited by Soultrigger (2023-12-14 14:12:34)

kermit63 · 2023-12-14 16:21:09

Soultrigger wrote:

Do you know if there is a terminal browser that would allow to run commands and post here? It might be useful in the future.

I haven't used one of those in more than a decade, so I'm not qualified to recommend any. What I do have though, is a bootable custom arch iso with GUI, browser and system tools for partitioning, backing up/cloning disks and general troubleshooting, which I was able to make with archiso. It does take a bit of a learning curve to make one, An alternative is to have a bootable arch-based USB like EndeavourOS available for those times when your arch install won't boot and you need to get on here to ask for help.

seth · 2023-12-14 16:22:31

1st link below, basicall

echo foo | curl -F 'file=@-' 0x0.st

Arch Linux

#1 2023-12-10 19:34:59

[Solved] Luks - Failing to boot

#2 2023-12-10 20:28:55

Re: [Solved] Luks - Failing to boot

#3 2023-12-10 22:11:28

Re: [Solved] Luks - Failing to boot

#4 2023-12-10 23:21:53

Re: [Solved] Luks - Failing to boot

#5 2023-12-11 12:18:04

Re: [Solved] Luks - Failing to boot

#6 2023-12-11 14:48:53

Re: [Solved] Luks - Failing to boot

#7 2023-12-12 05:54:00

Re: [Solved] Luks - Failing to boot

#8 2023-12-12 06:06:01

Re: [Solved] Luks - Failing to boot

#9 2023-12-12 16:27:02

Re: [Solved] Luks - Failing to boot

#10 2023-12-12 20:46:35

Re: [Solved] Luks - Failing to boot

#11 2023-12-13 02:02:38

Re: [Solved] Luks - Failing to boot

#12 2023-12-13 02:46:41

Re: [Solved] Luks - Failing to boot

#13 2023-12-13 07:44:56

Re: [Solved] Luks - Failing to boot

#14 2023-12-13 12:10:18

Re: [Solved] Luks - Failing to boot

#15 2023-12-13 12:11:14

Re: [Solved] Luks - Failing to boot

#16 2023-12-13 12:30:53

Re: [Solved] Luks - Failing to boot

#17 2023-12-13 12:42:46

Re: [Solved] Luks - Failing to boot

#18 2023-12-13 14:34:47

Re: [Solved] Luks - Failing to boot

#19 2023-12-13 15:27:03

Re: [Solved] Luks - Failing to boot

#20 2023-12-13 17:18:23

Re: [Solved] Luks - Failing to boot

#21 2023-12-13 19:53:58

Re: [Solved] Luks - Failing to boot

#22 2023-12-14 13:29:08

Re: [Solved] Luks - Failing to boot

#23 2023-12-14 14:01:50

Re: [Solved] Luks - Failing to boot

#24 2023-12-14 16:21:09

Re: [Solved] Luks - Failing to boot

#25 2023-12-14 16:22:31

Re: [Solved] Luks - Failing to boot

Board footer