[solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

sedrubal · 2023-07-21 08:37:18

I'm not sure, what changed, however, systemd stucks at boot while waiting for the root file system to come available. It is waiting 1:30min for /dev/disk/by-uuid/<root fs UUID> and then fails, because it is a dependency for sysroot.mount. Thing is, the root file system /dev/disk/by-uuid/<root fs UUID> IS available.

I'm using:

- UEFI (boot entry exists and starts bootlaoder)
- systemd-boot (successfully finds uki and starts the kernel with the initramfs)
- uki (uniform kernel image) which starts successfully
- luks with btrfs as root partition
- systemd inside initramfs (I tried also busybox but the result is the same)
- systemd-cryptenrolled encryption of luks partition (works and gets unlocked automatically when using systemd inside initramfs)

I tried:

- rebuilding initramfs (which was the suggestion in https://bbs.archlinux.org/viewtopic.php?id=287042)
- downgrading systemd (rebuilding initramfs afterwards)
- using busybox instead of systemd in initramfs
- using linux and linux-lts kernel instead of linux-hardened

When I run mount -a and systemctl start graphical.target inside the systemd resuce shell, the system is continue booting and reaches grpahical.target

Does anyone have any further ideas?

Last edited by sedrubal (2023-07-22 10:15:24)

schard · 2023-07-21 10:27:55

Please post your /etc/mkinitcpio.conf and /proc/cmdline from the emergency shell.

sedrubal · 2023-07-21 11:37:27

/etc/mkinitcpio.conf:

MODULES=()
BINARIES=(/usr/bin/btrfs)
FILES=()
HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block sd-encrypt filesystems fsck)

/proc/cmdline:

pti=on page_alloc.shuffle=1 root=UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e rootflags=subvol=@ rw apparmor=1 lsm=lockdown,yama,apparmor brt_disable intel_iommu=on

The UUID given here is exactly the same systemd is waiting for and which is already available.

sedrubal · 2023-07-21 11:42:24

Maybe some more context:

/etc/mkinitcpio.d/linux-hardened.preset:

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux-hardened"
ALL_microcode=(/boot/*-ucode.img)
PRESETS=('default' 'fallback')
default_uki="/efi/EFI/Linux/arch-linux-hardened.efi"
default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"
fallback_uki="/efi/EFI/Linux/arch-linux-hardened-fallback.efi"
fallback_options="-S autodetect"

/etc/initcpio/post/uki-sbsign:

#!/usr/bin/env bash
set -euo pipefail

uki="$3"
[[ -n "$uki" ]] || exit 0

keypairs=(/etc/efi-keys/db.key /etc/efi-keys/db.crt)

for (( i=0; i<${#keypairs[@]}; i+=2 )); do
    key="${keypairs[$i]}" cert="${keypairs[(( i + 1 ))]}"
    if ! sbverify --cert "$cert" "$uki" &>/dev/null; then
        sbsign --key "$key" --cert "$cert" --output "$uki" "$uki"
    fi
done

/etc/fstab:

UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /                       btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,subvol=/@        0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /home                   btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,subvol=/@home       0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /var/swap               btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,noexec,subvol=/@swap        0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /var/cache/pacman/pkg   btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,noexec,subvol=/@pkg 0       0
UUID=7697-FD30                                  /efi                    vfat            rw,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,nosuid,nodev,noexec,errors=remount-ro        0       2
/var/swap/swapfile                              none                    swap            defaults,noatime,nofail                         0       0

/etc/crypttab and /etc/crypttab.initramfs:

root    /dev/sda2                                       none                    discard,tpm2-device=auto

schard · 2023-07-21 11:55:47

sedrubal wrote:

pti=on page_alloc.shuffle=1 root=UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e rootflags=subvol=@ rw apparmor=1 lsm=lockdown,yama,apparmor brt_disable intel_iommu=on

sedrubal wrote:

UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /                       btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,subvol=/@        0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /home                   btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,subvol=/@home       0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /var/swap               btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,noexec,subvol=/@swap        0       0
UUID=e58f9254-4bd2-4143-bf19-64cc6a6d9d1e       /var/cache/pacman/pkg   btrfs           rw,noatime,compress=zstd,ssd,space_cache=v2,autodefrag,nosuid,nodev,noexec,subvol=/@pkg 0       0
UUID=7697-FD30                                  /efi                    vfat            rw,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,nosuid,nodev,noexec,errors=remount-ro        0       2
/var/swap/swapfile                              none                    swap            defaults,noatime,nofail                         0       0

I'm not too familiar with btrfs, but rootflags=subvol=@ vs. subvol=/@ might be a problem.

seth · 2023-07-21 13:07:22

The UUID given here is exactly the same systemd is waiting for and which is already available.

You're asserting that but haven't actually demonstrated it.
In the emergency shell, what's the output of "lsblk -f" and "uname -a"?

sedrubal · 2023-07-21 18:05:13

@schard Good catch! But unfortunately using subvol=/@ instead of subvol=@ and rebuilding the uki this doesn't change anything.

@seth

Here comes the proof that the UUIDs are really the same. I wrote the following script. It does not fail so everything should match:

#!/bin/bash

set -euo pipefail

echo Check if sysroot is available
ls $(systemctl show -P What sysroot.mount)

echo Check if root as specified in kernel args is available
ls /dev/disk/by-uuid/$(cat /proc/cmdline | sed "s/.*root=UUID=//" | cut -d" " -f1)

echo Check if root as specified in fstab is available
ls /dev/disk/by-uuid/$(cat /etc/fstab | grep "\s/\s" | cut -f1 | sed "s/UUID=//")

echo Check if disk, for which systemd is waiting, is available
TIMED_OUT_UNIT=dev-disk-by\\x2duuid-e58f9254\\x2d4bd2\\x2d4143\\x2dbf19\\x2d64cc6a6d9d1e.device
journalctl -u "${TIMED_OUT_UNIT}" -b
ls $(systemctl show -P Description "${TIMED_OUT_UNIT}")

echo "All good"

---

However, I might have found the root problem: Somehow initrd-switch-root.service seems to be executed twice or even three times! But I'm not 100% sure. Can you confirm my assumption?

Here is the journal output of relevant units:

$ journalctl -u /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e
Jul 21 19:27:58 archlinux systemd[1]: Found device /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e.
Jul 21 19:29:36 laptop systemd[1]: dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device: Job dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device/start timed out.
Jul 21 19:29:36 laptop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e.
Jul 21 19:29:36 laptop systemd[1]: dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device: Job dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device/start failed with result 'timeout'.
Jul 21 19:31:06 laptop systemd[1]: dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device: Job dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device/start timed out.
Jul 21 19:31:06 laptop systemd[1]: Timed out waiting for device /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e.
Jul 21 19:31:06 laptop systemd[1]: dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device: Job dev-disk-by\x2duuid-e58f9254\x2d4bd2\x2d4143\x2dbf19\x2d64cc6a6d9d1e.device/start failed with result 'timeout'.

$ journalctl -u initrd-switch-root.service
Jul 21 19:27:59 archlinux systemd[1]: Starting Switch Root...
Jul 21 19:28:05 laptop systemd[1]: Starting Switch Root...  <--- WHY AGAIN? The hostname has already changed so I assume we already switched root.
Jul 21 19:29:36 laptop systemd[1]: Starting Switch Root...  <--- AND AGAIN? (maybe this one comes from my debugging)
Jul 21 19:29:36 laptop @ystemctl[514]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
Jul 21 19:29:36 laptop systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
Jul 21 19:29:36 laptop systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
Jul 21 19:29:36 laptop systemd[1]: Failed to start Switch Root.
Jul 21 19:29:36 laptop systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.

If I'm right, why does systemd try to switch root again and how can I prevent it?

sedrubal · 2023-07-21 18:21:09

@seth

Here is uname -a

Linux laptop 6.4.4-hardened1-1-hardened #1 SMP PREEMPT_DYNAMIC Wed, 19 Jul 2023 19:31:16 +0000 x86_64 GNU/Linux

And lsblk -f captured at the time of waiting on /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e

NAME     FSTYPE      FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                       
|-sda1   vfat        FAT32 EFI        7697-FD30                              62.8M    79% /efi
`-sda2   crypto_LUKS 2     crypt-root 0384eb7c-13d9-471d-8f24-af33f96c6128                
  `-root                                                                    346.6G    27% /

This is how lsblk -f should look like after booting

NAME     FSTYPE      FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                       
├─sda1   vfat        FAT32 EFI        7697-FD30                              62,8M    79% /efi
└─sda2   crypto_LUKS 2     crypt-root 0384eb7c-13d9-471d-8f24-af33f96c6128                
  └─root                                                                    346,5G    27% /var/cache/pacman/pkg
                                                                                          /var/swap
                                                                                          /home
                                                                                          /

Last edited by sedrubal (2023-07-21 18:22:26)

loqs · 2023-07-21 18:36:15

sedrubal wrote:

If I'm right, why does systemd try to switch root again and how can I prevent it?

Does /etc/initrd-release exist on the root file-system?

seth · 2023-07-21 20:04:17

So sda2 is 0384eb7c-13d9-471d-8f24-af33f96c6128 - and where is e58f9254-4bd2-4143-bf19-64cc6a6d9d1e ?

stat /dev/disk/by-uuid/e58f9254-4bd2-4143-bf19-64cc6a6d9d1e

The two root switches are 90s apart - the systemd default timeout.

sedrubal · 2023-07-22 10:14:49

@loqs Thank you <3 I created /etc/initrd-release because I misunderstood some tutorial. Deleting that file solved the issue.

@seth This is the UUID of the root btrfs. I'm not sure, why lsblk -f didn't show that UUID in the emergency state. After booting properly lsblk -f shows the UUID of the btrfs again.

Thank you all. What I did was stupid, but I learned a lot by debugging the issue with you.

Arch Linux

#1 2023-07-21 08:37:18

[solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#2 2023-07-21 10:27:55

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#3 2023-07-21 11:37:27

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#4 2023-07-21 11:42:24

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#5 2023-07-21 11:55:47

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#6 2023-07-21 13:07:22

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#7 2023-07-21 18:05:13

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#8 2023-07-21 18:21:09

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#9 2023-07-21 18:36:15

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#10 2023-07-21 20:04:17

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

#11 2023-07-22 10:14:49

Re: [solved] Timed out waiting for device /dev/disk/by-uuid/<root fs UUID>

Board footer