Arch Linux

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

LUKS on LVM no mounting and not asking for passphrase after update

My system won't unlock my LUKS volume during boot like it used to and gets stuck instead with a message like "A start job is running for /dev/mapper/root" without asking for a passphrase

It's a LUKS on LVM configuration with UKIs generated by mkinitcpio

I triple checked my cryptroot UUID even though I didn't touch it and also tried changing mkinitcpio.conf from systemd to udev based but no luck.

mkinitcpio.conf hooks:

HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block lvm2 encrypt filesystems fsck)

luksDump:

cryptsetup luksDump /dev/cid/cryptroot
...
UUID:           85886087-dc66-4eb9-8c3c-2824402b5380

cat /etc/kernel/cmdline :

cryptdevice=UUID=85886087-dc66-4eb9-8c3c-2824402b5380:root root=/dev/mapper/root rw

I'm out of ideas on how to troubleshoot

EDIT: I should note that I had this issue since a week or two and this is my fresh reinstall shown above, exact same issue though

Last edited by mongaikan (2025-01-25 15:52:20)

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

I now have my second system affected.

It's a LVM on LUKS in this case

Firmware update reset my secure boot keys and my passphrase wouldn't work anymore (used to be auto unlocked with secure boot + tpm)

Rebooted on arch iso, mounted the luks volume with my passphrase, reset all tokens just in case with just my (same) passphrase + tpm

Reenrolled everything, reboot into my install and crypsetup still won't auto unlock with TPM nor accepts my passphrase.

Pretty sure it's a bug at this point.

My only two systems are now borked

Any help would be greatly appreciated

frostschutz

Member

Registered: 2013-11-15

Posts: 1,513

Re: LUKS on LVM no mounting and not asking for passphrase after update

"A ... job is running for ..." should be a systemd message. If you see this in initramfs phase then you have systemd-based initramfs, however your initcpio hooks line looks like the non-systemd one.

Is it possible you changed your mkinitcpio configuration and forgot to rebuild the initcpio afterward, or your bootloader is trying to load the wrong initrd file? (Or your UKI builder uses the wrong one.)

With systemd based initramfs the parameter cryptdevice=UUID= would most likely be ignored altogehter, and you'd need options described in systemd-cryptsetup-generator. (mkinitcpio -H encrypt vs. -H sd-encrypt)

Last edited by frostschutz (2025-01-30 09:10:41)

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

I did regenerate the UKIs after changing mkinitcpio.conf from systemd based to the udev one as a first step but it didn't work

I then went ahead and modified mkinitcpio conf again to generate standard non UKI images, removed all traces of them, but got the same symptom with non uki images

systemd-crytenroll or luks or something just did something similar on my laptop, slight difference actually because it asked for a passphrase but it wouldn't accept it (I even added one with just one character just to be sure it wasn't me or my keyboard layout or something) nor did it even try the tpm key

I'm still pretty sure it's a bug somewhere in the systemd-cryptenroll / cryptsetup / mkinitcpio stack as it appeared out of nowhere without any conf change on my side after recent update

I couldn't find how to log persistently during boot to provide more info unfortunately

Thanks anyway!

frostschutz

Member

Registered: 2013-11-15

Posts: 1,513

Re: LUKS on LVM no mounting and not asking for passphrase after update

And you're 140% sure you're not booting the wrong file? You got your /boot /efi /boot/efi whatever you use mounted properly?

You could check your initrd with lsinitcpio (will show you its contents, config, can also unpack it in an empty directory).

If you're using multiple initrd files (in grub, e.g for early microcode) also remove everything but the primary mkinitcpio one. But that shouldn't be it unless someone overwrite the early microcode with an old initcpio image. Which hopefully nobody ever does but you never know.

If you're very familiar with qemu/kvm (or any other virtualization) it's also possible to test kernel+initrd in a VM. It would complain about missing modules / drives / files. (If you pass your actual drive, it must be done in hard read only, no network mode, i.e. fully isolated). Sometimes that helps me with debugging grub and initrd stuff since rebooting on bare metal is quite annoying.

PS: is it possible there is an error in the mkinitcpio.conf? for example if you uncommented the HOOKS line twice, the one you posted might not be used at all. What's the output when you run mkinitcpio?

Last edited by frostschutz (2025-01-30 12:48:05)

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

Those are all very interesting ideas I didn't think of or didn't even know of, awesome.

I'm gonna need some time to explore them, also need to do a simpler temp install on another drive to do the work first, and do my best to document where my mistake was because it seems really sneaky to me.

Thank you for taking the time and for all the ideas, I really appreciate!

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

Can confirm everything from mkinitcpio.conf and the content of my UKIs are correct and identical.
Which isn't too surprising since I haven't touched it before the upgrade that broke it.

I don't see many options for me to investigate further with a VM or something unfortunately as I would need at least one machine working.

I'm gonna do a fresh install giving up on luks, uki, secure boot and all that in favor of a simpler fscrypt on my home for now and test out my previous setup next time I feel like doing a fresh install.

Hopefully someone else is able to catch the bugs I mentioned before in another way sometime.

Thanks again,
Cheers

frostschutz

Member

Registered: 2013-11-15

Posts: 1,513

Re: LUKS on LVM no mounting and not asking for passphrase after update

If you don't mind, you could share your kernel image / initrd file / and I'll take a blink at it. No idea what's going on there, though.

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

Of course! I'll send you a link to the archive in a PM if that's ok.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,513

Re: LUKS on LVM no mounting and not asking for passphrase after update

"A ... job is running for ..." should be a systemd message. If you see this in initramfs phase then you have systemd-based initramfs, however your initcpio hooks line looks like the non-systemd one.

What you posted above:

mkinitcpio.conf hooks:

HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block lvm2 encrypt filesystems fsck)

What's inside the file you sent me: ( /boot/EFI/Linux/arch-linux-{lts,zen}{,-fallback}.efi :: buildconfig )

HOOKS=(systemd autodetect microcode modconf kms keyboard sd-vconsole block sd-encrypt lvm2 filesystems fsck)

So based on what you sent me, you're using systemd based initramfs, which requires systemd specific kernel parameters (man systemd-cryptsetup-generator).

If you rebuilt your images with changed mkinitcpio configuration, then for some reason those changes didn't take, or it somehow ended up in a different location.

mongaikan

Member

From: France

Registered: 2023-03-08

Posts: 8

Re: LUKS on LVM no mounting and not asking for passphrase after update

Yes, to recap I used to use only systemd based configuration / images on both machines before they stopped booting and it's only after things went bad a few days/weeks ago that I tried reverting to udev based images.
The archive contained the the entire /boot that was part of my full laptop system back up before reinstalling it by the way.
I was pretty sure I did regenerate the images after changing the configuration at the time but honestly at this point I can't be 100% certain of all the things I've tried I'm afraid.

I wish I had a way to pin point what update broke my installs in case it could be useful to others but I guess I'll never know. (unencrypted back up is now shred-ed anyway)
Well at least it seems I was the only one affected so probably a bad configuration somewhere that happened to reveal itself only at some point I suppose.