[SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

geekmiki · 2023-06-05 19:13:16

Hi,

Updating systemd to 253.5-1 breaks booting on my system. After reaching the [encrypt] hook the device it's supposed to boot from is not found neither by device name, nor by UUID.

:: running hook [encrypt}
ERROR: Failed to open encryption mapping: The device /dev/nvme1n1p2 is not a LUKS volume and the crypto= parameter was not specified.
Waiting 10 seconds for device /dev/disk/by-uuid/...
ERROR: Device 'UUID=...' not found.

Downgrading to systemd 253.4-1 solves the issue.

Has anything changed with systemd that I am not aware of? Anyone experiencing this issue?

Thanks in advance.

Last edited by geekmiki (2023-06-06 18:48:11)

nicktee · 2023-06-06 03:56:45

Yep I just had a very similar issue. That was pretty rude. I couldn't boot and couldn't even use the keyboard in the emergency prompt.

Downgrading systemd(-*) to 253.4-1 fixed it

Last edited by nicktee (2023-06-06 03:57:17)

DerTee · 2023-06-06 08:07:09

I experienced the same thing however I'm NOT using LUKS - in fact, my root-partition was not able to be mounted at all on boot.
Solution was to boot from arch-install usb-key, then mount my root-directory, then boot directory and using arch-chroot and pacman to downgrade to a previous version of systemd.

For all those unfortunate enough to not have the packages in local cache (usually found in /var/cache/pacman/pkg/) you will need to get the packages from the online archive.
Here's what I did - to at least get my system into a running state again:

pacman -U https://archive.archlinux.org/packages/s/systemd/systemd-253.4-1-x86_64.pkg.tar.zst
pacman -U https://archive.archlinux.org/packages/s/systemd-libs/systemd-libs-253.4-1-x86_64.pkg.tar.zst
pacman -U https://archive.archlinux.org/packages/s/systemd-sysvcompat/systemd-sysvcompat-253.4-1-x86_64.pkg.tar.zst
pacman -U https://archive.archlinux.org/packages/l/lib32-systemd/lib32-systemd-253.4-1-x86_64.pkg.tar.zst

(edit: ancient bbcode is ancient)

Last edited by DerTee (2023-06-06 08:08:12)

schard · 2023-06-06 13:11:47

Strange, I have no such issues, though running ext4 on a LUKS volume.
Do you folks use udev or systemd in your initramfs?
I use the latter btw.

Ort · 2023-06-06 13:29:21

I had the same problem and I am not using any kind of encryption, keyboard did not work in emergency shell. I did boot from initramfs in GRUB menu and that worked for me. I downgraded systemd version and after reboot I could enter Arch the usual way again.

geekmiki · 2023-06-06 13:31:48

schard wrote:

Strange, I have no such issues, though running ext4 on a LUKS volume.
Do you folks use udev or systemd in your initramfs?
I use the latter btw.

Yes, using udev here.

I can also confirm that keyboard was not working in the emergency shell.

loqs · 2023-06-06 14:32:01

geekmiki wrote:

Yes, using udev here.

Can you switch to systemd to test if the issue is limited to udev?

geekmiki · 2023-06-06 15:08:26

loqs wrote:

Can you switch to systemd to test if the issue is limited to udev?

I would need a little bit of help on that. I guess it's not as simple as removing udev from mkinitcpio.conf? What do I replace it with?
Thanks in advance.

loqs · 2023-06-06 15:22:45

geekmiki wrote:

loqs wrote:
Can you switch to systemd to test if the issue is limited to udev?
I would need a little bit of help on that. I guess it's not as simple as removing udev from mkinitcpio.conf? What do I replace it with?

Please post the contents of mkinitcpio.conf or at least the hooks array.

Last edited by loqs (2023-06-06 15:23:00)

geekmiki · 2023-06-06 15:30:59

loqs wrote:

Please post the contents of mkinitcpio.conf or at least the hooks array.

mkinitcpio.conf looks like this:

MODULES=(vmd btrfs amdgpu)
BINARIES=(/usr/bin/btrfs)
FILES=()
HOOKS=(base udev autodetect keyboard keymap modconf block encrypt filesystems fsck)

loqs · 2023-06-06 15:41:41

HOOKS=(base systemd autodetect keyboard sd-vconsole modconf block encrypt filesystems fsck)

You would also need to change the boot parameter cryptdevice=device:dmname to rd.luks.name=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX=name see System_configuration#rd.luks.name or drop cryptdevice and create /etc/crypttab.initramfs as covered in System_configuration#Using_systemd-cryptsetup-generator.

schard · 2023-06-06 16:00:41

loqs wrote:

HOOKS=(base systemd autodetect keyboard sd-vconsole modconf block encrypt filesystems fsck)
You would also need to change the boot parameter cryptdevice=device:dmname to rd.luks.name=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX=name see System_configuration#rd.luks.name or drop cryptdevice and create /etc/crypttab.initramfs as covered in System_configuration#Using_systemd-cryptsetup-generator.

Also replace encrypt with sd-encrypt.

geekmiki · 2023-06-06 18:47:55

Thanks for all your help.
Switching to systemd and removing udev solved the issue with systemd-253.5-1

schard · 2023-06-06 19:17:47

Well, partially. We now know what the issue is, but I believe, that the system should™ still work for users that prefer a udev based image (though I am not one of them).

Update: You may also check whether the update to udisks2-2.9.4-4 solved this issue.

Last edited by schard (2023-06-07 07:19:27)

nicktee · 2023-06-11 02:54:36

So to avoid breakages like this, I usually like to "run with the herd" on my system config, I don't have any particular requirements or strong enough opinions about systemd/udev to warrant running anything "non-standard"

Should I be switching to systemd from udev as described in this thread? Am I doing something weird by mixing them? I believe when I installed Arch (about 3 or 4 years ago) I just followed the instructions from the wiki.

kunev · 2023-09-05 12:09:40

This seems to have happened again with 254-1.1.

Is this actually a desired thing and should we adapt, or is it just a tricky problem that keeps creeping in as a regression?

loqs · 2023-09-05 12:27:14

kunev wrote:

This seems to have happened again with 254-1.1.
Is this actually a desired thing and should we adapt, or is it just a tricky problem that keeps creeping in as a regression?

Was it ever reported on the Arch bugtracker yet alone fixed for systemd-253.5-1?

nicktee · 2023-09-06 01:02:05

An issue that I've encountered a few times recently and has similar symptoms to this one is this https://bbs.archlinux.org/viewtopic.php?id=284757

Might be worth looking at your pacman logs to ensure it's not that issue you're seeing, you can resolve it by running mkinitcpio -p linux again

sekret · 2023-09-06 07:17:28

Strange, I use udev and haven't run in to either of those issues.

At least I think so. It's happened twice in the last ~2 months that I couldn't boot. Unfortunately I didn't write down the error message, but I'm pretty sure it wasn't the one in the OP. What I then did was to boot the system with the fallback image, ran mkinitcpio again and all was well after that.

To be honest, I only use udev because it was the default when I installed my system 2 years ago. These issues seem like an indicator that we should switch to systemd. What do you think?

Last edited by sekret (2023-09-06 07:17:55)

nicktee · 2023-09-06 08:24:32

@sekret you can tell if the error I linked was the cause because you'll see lines like

[ALPM-SCRIPTLET] sort: cannot read: '/sys/devices/pci0000:00/0000:00:1b.0/0000:02:00.0/0000:03:01.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/usb3/3-1/3-1.4/3-1.4.2/3-1.4.2.3/3-1.4.2.3.1/3-1.4.2.3.1:1.2/ep_02/ueven
t': No such file or directory

in /var/log/pacman.log in the updates where it occurred.

I also just installed udev because it was the default so many years ago when I installed arch. I also wonder if it's worth switching.

Arch Linux

#1 2023-06-05 19:13:16

[SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#2 2023-06-06 03:56:45

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#3 2023-06-06 08:07:09

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#4 2023-06-06 13:11:47

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#5 2023-06-06 13:29:21

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#6 2023-06-06 13:31:48

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#7 2023-06-06 14:32:01

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#8 2023-06-06 15:08:26

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#9 2023-06-06 15:22:45

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#10 2023-06-06 15:30:59

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#11 2023-06-06 15:41:41

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#12 2023-06-06 16:00:41

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#13 2023-06-06 18:47:55

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#14 2023-06-06 19:17:47

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#15 2023-06-11 02:54:36

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#16 2023-09-05 12:09:40

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#17 2023-09-05 12:27:14

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#18 2023-09-06 01:02:05

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#19 2023-09-06 07:17:28

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

#20 2023-09-06 08:24:32

Re: [SOLVED]Systemd 253.5-1 update breaks boot from LUKS volume

Board footer