[SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

beachcoder · 2023-12-12 13:43:30

Edit: Problem meanwhile fixed by systemd 255.2-3.

Hello!

After a recent

pacman -Suy

I find my system in a stucked boot screen that waits forever for a device by uuid (xxxxxxxx-e955-4d53-91ce-a78dd3abcc2b) that I've seen nowhere before in my system.

I'm having a full-encrypted system partition. Here's the mount layout:

$ lsblk -f
NAME       FSTYPE      FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1     crypto_LUKS 2           xxxxxxxx-1b25-4005-a77d-15c1c2a879fc
│ └─system btrfs                   xxxxxxxx-2bbc-4f18-beba-101863678cd6    4.2G    98% /mnt/btrfs/system
│                                                                                      /home
│                                                                                      /
├─sda2     vfat        FAT32 EFI   4FC3-B8C3                             510.6M     0% /boot/efi
└─sda3

/etc/mkinitcpio.conf

MODULES=""
BINARIES="/usr/bin/btrfs"
FILES=(/root/boot-unlock-keyfile.bin)
HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block sd-encrypt btrfs filesystems fsck)

/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX="rootflags=subvol=@root rd.luks.name=xxxxxxxx-1b25-4005-a77d-15c1c2a879fc=system rd.luks.key=/keyfile.bin rd.luks.options=discard"
GRUB_ENABLE_CRYPTODISK=y
GRUB_PRELOAD_MODULES="part_gpt part_msdos"
GRUB_TERMINAL_INPUT=console
GRUB_GFXMODE=auto
GRUB_GFXPAYLOAD_LINUX=keep
GRUB_DISABLE_RECOVERY=true

I've tried adding

rd.break=pre-trigger

or

rd.break=initqueue

to the kernel parameters to get into a rescue shell, but no way.
I don't know how to diagnose this anymore. I've restored my root from a backup snapshot for now.

Last edited by beachcoder (2024-01-29 10:36:32)

seth · 2023-12-12 14:24:57

Smells somehow like https://bbs.archlinux.org/viewtopic.php?id=290953 - all the way down to btrfs…
The UUIDs are not sensitive data, their purpose is to prevent reference collisions.

pacman -Suy

Ignoring the heresy (-Syu), what was updated?

beachcoder · 2023-12-12 20:28:42

seth wrote:

Smells somehow like https://bbs.archlinux.org/viewtopic.php?id=290953 - all the way down to btrfs…

Thanks for your reply, and pointing to the other thread.
Indeed, this seems to be a similar problem, especially with the unknown UUID.

seth wrote:

The UUIDs are not sensitive data, their purpose is to prevent reference collisions.

Okay... IMHO they could also be used to identify a computer ;-)

seth wrote:

Ignoring the heresy (-Syu), what was updated?

From /var/log/pacman.log I've extracted the package upgrade information from below.
I've tried to downgrade to linux-6.6.4 again but this didn't help. I'm always on stock kernel.
Maybe it could also be the systemd version upgrade?

[2023-12-12T12:55:29+0100] [ALPM] upgraded aom (3.7.1-1 -> 3.8.0-2)
[2023-12-12T12:55:29+0100] [ALPM] upgraded util-linux-libs (2.39.2-2 -> 2.39.3-1)
[2023-12-12T12:55:29+0100] [ALPM] upgraded systemd-libs (254.6-2 -> 255-1)
[2023-12-12T12:55:29+0100] [ALPM] upgraded curl (8.4.0-2 -> 8.5.0-1)
[2023-12-12T12:55:29+0100] [ALPM] upgraded glib2 (2.78.1-1 -> 2.78.3-1)
[2023-12-12T12:55:29+0100] [ALPM] upgraded libxml2 (2.12.1-1 -> 2.12.2-1)
[2023-12-12T12:55:29+0100] [ALPM] upgraded archlinux-keyring (20231130-1 -> 20231207-1)
[2023-12-12T12:55:32+0100] [ALPM] upgraded blas (3.12.0-2 -> 3.12.0-3)
[2023-12-12T12:55:32+0100] [ALPM] upgraded bluez (5.70-1 -> 5.70-2)
[2023-12-12T12:55:32+0100] [ALPM] upgraded bluez-libs (5.70-1 -> 5.70-2)
[2023-12-12T12:55:32+0100] [ALPM] upgraded cblas (3.12.0-2 -> 3.12.0-3)
[2023-12-12T12:55:32+0100] [ALPM] upgraded xdg-utils (1.2.0r17+g21fb316-1 -> 1.2.0r25+g0f49cf5-1)
[2023-12-12T12:55:32+0100] [ALPM] upgraded hwdata (0.376-1 -> 0.377-1)
[2023-12-12T12:55:33+0100] [ALPM] upgraded util-linux (2.39.2-2 -> 2.39.3-1)
[2023-12-12T12:55:33+0100] [ALPM] upgraded systemd (254.6-2 -> 255-1)
[2023-12-12T12:55:35+0100] [ALPM] upgraded chromium (119.0.6045.199-1 -> 120.0.6099.71-1)
[2023-12-12T12:55:35+0100] [ALPM] upgraded chrpath (0.16-4 -> 0.17-1)
[2023-12-12T12:55:37+0100] [ALPM] upgraded cmake (3.27.9-1 -> 3.28.0-1)
[2023-12-12T12:55:38+0100] [ALPM] upgraded libnl (3.8.0-1 -> 3.9.0-1)
[2023-12-12T12:55:38+0100] [ALPM] installed rubberband (3.3.0-1)
[2023-12-12T12:55:38+0100] [ALPM] upgraded v4l-utils (1.24.1-2 -> 1.26.0-1)
[2023-12-12T12:55:38+0100] [ALPM] upgraded ffmpeg (2:6.1-1 -> 2:6.1-2)
[2023-12-12T12:55:39+0100] [ALPM] upgraded code (1.84.2-1 -> 1.85.0-1)
[2023-12-12T12:55:39+0100] [ALPM] upgraded containerd (1.7.10-1 -> 1.7.11-1)
[2023-12-12T12:55:40+0100] [ALPM] upgraded discord (0.0.36-1 -> 0.0.37-1)
[2023-12-12T12:55:40+0100] [ALPM] upgraded dotnet-host (8.0.0.sdk100-2 -> 8.0.0.sdk100-3)
[2023-12-12T12:55:40+0100] [ALPM] upgraded fltk (1.3.8-2 -> 1.3.9-1)
[2023-12-12T12:55:40+0100] [ALPM] upgraded frei0r-plugins (2.3.1-1 -> 2.3.2-1)
[2023-12-12T12:55:40+0100] [ALPM] upgraded glib2-docs (2.78.1-1 -> 2.78.3-1)
[2023-12-12T12:55:43+0100] [ALPM] upgraded go (2:1.21.4-1 -> 2:1.21.5-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded imagemagick (7.1.1.22-1 -> 7.1.1.23-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded thin-provisioning-tools (1.0.7-1 -> 1.0.9-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded libkcddb5 (23.08.3-4 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded k3b (1:23.08.3-2 -> 1:23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded signon-kwallet-extension (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded kaccounts-integration (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded libkexiv2 (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded kio-extras (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded kio-fuse (5.0.1-2 -> 5.1.0-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded konsole (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lapack (3.12.0-2 -> 3.12.0-3)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-curl (8.4.0-2 -> 8.5.0-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-util-linux (2.39.2-1 -> 2.39.3-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-glib2 (2.78.1-1 -> 2.78.3-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-libnl (3.8.0-1 -> 3.9.0-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-libxml2 (2.12.1-1 -> 2.12.2-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded lib32-systemd (254.6-1 -> 255-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded libdecor (0.2.1-1 -> 0.2.1-2)
[2023-12-12T12:55:44+0100] [ALPM] upgraded libnautilus-extension (45.2-1 -> 45.2.1-1)
[2023-12-12T12:55:44+0100] [ALPM] upgraded zxing-cpp (2.1.0-1 -> 2.2.0-2)
[2023-12-12T12:55:49+0100] [ALPM] upgraded libreoffice-fresh (7.6.3-3 -> 7.6.4-1)
[2023-12-12T12:55:49+0100] [ALPM] upgraded lightdm (1:1.32.0-4 -> 1:1.32.0-5)
[2023-12-12T12:55:51+0100] [ALPM] upgraded linux (6.6.4.arch1-1 -> 6.6.6.arch1-1)
[2023-12-12T12:55:51+0100] [ALPM] upgraded linux-firmware-whence (20231110.74158e7a-1 -> 20231211.f2e52a1c-1)
[2023-12-12T12:55:52+0100] [ALPM] upgraded linux-firmware (20231110.74158e7a-1 -> 20231211.f2e52a1c-1)
[2023-12-12T12:55:52+0100] [ALPM] upgraded luajit (2.1.1700008891-1 -> 2.1.1702233742-1)
[2023-12-12T12:55:52+0100] [ALPM] upgraded netstandard-targeting-pack (8.0.0.sdk100-2 -> 8.0.0.sdk100-3)
[2023-12-12T12:55:53+0100] [ALPM] upgraded nodejs (21.3.0-1 -> 21.4.0-1)
[2023-12-12T12:55:53+0100] [ALPM] upgraded npm (10.2.4-1 -> 10.2.5-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded okular (23.08.3-1 -> 23.08.4-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded perl-io-tty (1.17-2 -> 1.18-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded pkgconf (2.0.3-1 -> 2.1.0-2)
[2023-12-12T12:55:54+0100] [ALPM] upgraded postgresql-libs (15.4-2 -> 16.1-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded pybind11 (2.11.1-1 -> 2.11.1-2)
[2023-12-12T12:55:54+0100] [ALPM] upgraded pyenv (1:2.3.33-1 -> 1:2.3.35-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-pathspec (0.11.2-1 -> 0.12.1-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-typing_extensions (4.8.0-1 -> 4.9.0-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-platformdirs (4.0.0-1 -> 4.1.0-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-pygments (2.16.1-1 -> 2.17.2-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-setuptools (1:68.2.1-1 -> 1:69.0.0-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-virtualenv (20.24.6-1 -> 20.25.0-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded python-websocket-client (1.6.4-1 -> 1.7.0-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded qpdf (11.6.3-1 -> 11.6.4-1)
[2023-12-12T12:55:54+0100] [ALPM] upgraded systemd-sysvcompat (254.6-2 -> 255-1)
[2023-12-12T12:55:55+0100] [ALPM] upgraded valgrind (3.21.0-4 -> 3.22.0-1)
[2023-12-12T12:55:55+0100] [ALPM] upgraded virtualbox-host-modules-arch (7.0.12-9 -> 7.0.12-11)
[2023-12-12T12:55:55+0100] [ALPM] upgraded wasmer (4.2.2-1 -> 4.2.3-1)
[2023-12-12T12:55:56+0100] [ALPM] upgraded webkit2gtk (2.42.2-1 -> 2.42.3-1)
[2023-12-12T12:55:56+0100] [ALPM] upgraded webkit2gtk-4.1 (2.42.2-1 -> 2.42.3-1)

seth · 2023-12-12 20:42:57

Maybe it could also be the systemd version upgrade?

Yes, there seem to be more issues w/ SD255…

If the downgrade doesn't help, try to add the UUID to your fstab as "noauto" so hopefully systemd will just ignore the device it made up.

IMHO they could also be used to identify a computer ;-)

It pseudonymizes (is that a word?) the system, ie. if you post that information in multiple locations, you can be tracked across those locations.
If you exclusively post it here along your user account, it's meaningless.

beachcoder · 2023-12-26 13:31:16

seth wrote:

If the downgrade doesn't help, try to add the UUID to your fstab as "noauto" so hopefully systemd will just ignore the device it made up.

Yesterday, I did a new try with systemd 255.2 but the problem still exists - anyway, the boot does not get stuck forever, but continues after 2minutes.

I've then tried your advise, and added this to my /etc/fstab, but it didn't change anything.

# systemd BUG workaround:
UUID=xxxxxxxx-e955-4d53-91ce-a78dd3abcc2b	none	ext2	noauto

I've downgraded again to systemd 254.6 and put it on the IgnorePkg-list in pacman.conf, anyway this isn't a good solution. Should I file this to upstream?

Head_on_a_Stick · 2023-12-26 13:51:08

beachcoder wrote:

Should I file this to upstream?

Only if you want it to be fixed

seth · 2023-12-26 19:38:00

As a mitigation, adding "systemd.gpt_auto=0 systemd.swap=0 noresume" will likely prevent it.
An interesting question to gage whether this is any different from https://bbs.archlinux.org/viewtopic.php?id=290990 would be whether only "systemd.gpt_auto=0 systemd.swap=0" will work (but I doubt it)

https://github.com/systemd/systemd/issues/30557
https://github.com/systemd/systemd/issues/30395

beachcoder · 2024-01-11 13:37:26

Hello all,

first of all, thanks for additional comments and I'm sorry for the delayed reply.

Today, I've updated my Arch Linux system again with the ignored systemd.

After reboot, it totally broke with several other FAILURES and start jobs running, I think it has to deal with the D-BUS integration. The boot entirely got stuck here, I wasn't able to resolve it (and also don't have the time for it!).
Then I booted into an installation medium, removed systemd from IgnorePkg and ran pacman -Suy again, so everything was up-to-date.
This gave me again the version where I needed to wait 2min and the boot was normal (as reported above).
I then added the proposed boot parameters from seth to the kernel parameters and it worked for now.

seth wrote:

As a mitigation, adding "systemd.gpt_auto=0 systemd.swap=0 noresume" will likely prevent it.

Yup, that worked. Unfortunately, I don't have any swap now. I will try without systemd.swap=0 in a moment.

Head_on_a_Stick wrote:

beachcoder wrote:
Should I file this to upstream?
Only if you want it to be fixed

I think this is necessary.

Ok that's the current state for now. I'm reporting back if the systemd.swap=0 remove normally boots and gives me back the swap-space in a moment.

beachcoder · 2024-01-11 13:41:57

beachcoder wrote:

seth wrote:
As a mitigation, adding "systemd.gpt_auto=0 systemd.swap=0 noresume" will likely prevent it.
Yup, that worked. Unfortunately, I don't have any swap now. I will try without systemd.swap=0 in a moment.

Ok, that worked. I now have swapspace and no interrupt during boot. Therefore systemd.gpt_auto=0 noresume satisfies.
I think this information is useful for the bug report to file to upstream.

seth · 2024-01-11 14:30:24

Upstream claims this is fixed.
Is it actually still a problem w/ the latest systemd update?

beachcoder · 2024-01-11 17:16:23

seth wrote:

Is it actually still a problem w/ the latest systemd update?

Well, yes it happened today with this version:

local/systemd 255.2-2
    system and service manager
local/systemd-libs 255.2-2
    systemd client libraries
local/systemd-sysvcompat 255.2-2
    sysvinit compat for systemd

Can you provide a link to the ticket where it's flagged to be fixed?

seth · 2024-01-12 08:58:15

Links are in post #7
https://bbs.archlinux.org/viewtopic.php … 0#p2139430
But might only apply to systemd-256 then

beachcoder · 2024-01-29 10:35:17

Hey, thanks all your help.
Meanwhile, the problem seems to be fixed with systemd 255.2-3.

Arch Linux

#1 2023-12-12 13:43:30

[SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#2 2023-12-12 14:24:57

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#3 2023-12-12 20:28:42

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#4 2023-12-12 20:42:57

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#5 2023-12-26 13:31:16

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#6 2023-12-26 13:51:08

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#7 2023-12-26 19:38:00

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#8 2024-01-11 13:37:26

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#9 2024-01-11 13:41:57

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#10 2024-01-11 14:30:24

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#11 2024-01-11 17:16:23

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#12 2024-01-12 08:58:15

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

#13 2024-01-29 10:35:17

Re: [SOLVED] Boot stuck "start job is running for /dev/disk/by-uuid/..."

Board footer