You are not logged in.

Pages: 1

#1 2023-10-05 01:21:32

JoshWVS
Member
Registered: 2023-10-05
Posts: 3

[SOLVED] Can't boot after system upgrade ("ERROR: device not found")

Hello all! After performing an upgrade via pacman -Suy about a month ago, my system became unable to boot. Specifically, it shows the error message below (and I can't type any commands here, everything is "frozen" until I shutdown by holding the power button).

ERROR: device 'UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401' not found. Skipping fsck.
mount: /new_root: can't find UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401.
ERROR: failed to mount '1dd743ed-e0a5-4b0c-80ee-186e89eea401' on real root
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ~]# _

I have been able to (temporarily) resolve the issue by booting into the installation media, mounting my root/boot partitions, arch-chroot-ing in, and finally running mkinitcpio -p linux. After that procedure, I can boot normally again; however, any time I run pacman -Suy, I face the same problem above after rebooting.

I've seen a few forum posts that discuss the same symptom ("device ... not found. Skipping fsck"), but none seemed mkinitcpio-related (as I believe my problem is?). Does anyone have thoughts on what could be going wrong here? I'm not sure what logs/other diagnostics to provide--I've included a few guesses below.

Here's a snippet from /var/log/pacman for my last pacman -Suy (eliding the hundreds of package lines and jumping to the hooks output)--it looks like there's some errors in mkinitcpio, but it still ends with "Image generation successful"?

[2023-10-04T19:14:44-0400] [ALPM] transaction completed
[2023-10-04T19:14:44-0400] [ALPM] running '20-systemd-sysusers.hook'...
[2023-10-04T19:14:45-0400] [ALPM] running '30-systemd-daemon-reload.hook'...
[2023-10-04T19:14:45-0400] [ALPM] running '30-systemd-tmpfiles.hook'...
[2023-10-04T19:14:45-0400] [ALPM] running '30-systemd-udev-reload.hook'...
[2023-10-04T19:14:45-0400] [ALPM] running '30-systemd-update.hook'...
[2023-10-04T19:14:45-0400] [ALPM] running '30-update-mime-database.hook'...
[2023-10-04T19:14:46-0400] [ALPM] running '40-update-ca-trust.hook'...
[2023-10-04T19:14:46-0400] [ALPM] running '90-mkinitcpio-install.hook'...
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img --microcode /boot/*-ucode.img
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET] ==> Starting build: '6.5.5-arch1-1'
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2023-10-04T19:14:46-0400] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] sort: cannot read: '/sys/devices/pci0000:00/0000:00:08.1/0000:2f:00.3/usb3/3-3/3-3.4/3-3.4.3/3-3.4.3:1.1/ep_81/uevent': No such file or directory
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] modprobe: ERROR: missing parameters. See -h.
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [kms]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect --microcode /boot/*-ucode.img
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET] ==> Starting build: '6.5.5-arch1-1'
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2023-10-04T19:14:48-0400] [ALPM-SCRIPTLET]   -> Running build hook: [kms]
[2023-10-04T19:14:50-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'ast'
[2023-10-04T19:14:52-0400] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2023-10-04T19:14:52-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'bfa'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qed'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'aic94xx'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla2xxx'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla1280'
[2023-10-04T19:14:53-0400] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'wd719x'
[2023-10-04T19:14:54-0400] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2023-10-04T19:14:55-0400] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2023-10-04T19:14:56-0400] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-10-04T19:14:56-0400] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
[2023-10-04T19:14:57-0400] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-10-04T19:14:57-0400] [ALPM] running 'dbus-reload.hook'...
[2023-10-04T19:14:57-0400] [ALPM] running 'detect-old-perl-modules.hook'...
[2023-10-04T19:14:57-0400] [ALPM] running 'fontconfig.hook'...
[2023-10-04T19:14:57-0400] [ALPM] running 'ghc-register.hook'...
[2023-10-04T19:15:03-0400] [ALPM] running 'glib-compile-schemas.hook'...
[2023-10-04T19:15:03-0400] [ALPM] running 'gtk-update-icon-cache.hook'...
[2023-10-04T19:15:03-0400] [ALPM] running 'texinfo-install.hook'...
[2023-10-04T19:15:04-0400] [ALPM] running 'update-desktop-database.hook'...

/etc/fstab (root/boot entries; this matches what I expect from lsblk -f or /dev/disk/by-uuid)

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p4 LABEL=arch
UUID=1dd743ed-e0a5-4b0c-80ee-186e89eea401	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p1
UUID=DE8E-62EA      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

/etc/mkinitcpio.conf (I don't think I've ever touched this by hand)

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(usbhid xhci_hcd)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No RAID, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect modconf block filesystems fsck)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev modconf block filesystems fsck)
#
##   This setup assembles a mdadm array with an encrypted root file system.
##   Note: See 'mkinitcpio -H mdadm_udev' for more information on RAID devices.
#    HOOKS=(base udev modconf keyboard keymap consolefont block mdadm_udev encrypt filesystems fsck)
#
##   This setup loads an lvm2 volume group.
#    HOOKS=(base udev modconf block lvm2 filesystems fsck)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr and fsck hooks.
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

# MODULES_DECOMPRESS
# Decompress kernel modules during initramfs creation.
# Enable to speedup boot process, disable to save RAM
# during early userspace. Switch (yes/no).
#MODULES_DECOMPRESS="yes"

/etc/mkinitcpio.d/linux.preset

# mkinitcpio preset file for the 'linux' package

#ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"
ALL_microcode=(/boot/*-ucode.img)

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-linux.img"
#default_uki="/efi/EFI/Linux/arch-linux.efi"
#default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-linux-fallback.img"
#fallback_uki="/efi/EFI/Linux/arch-linux-fallback.efi"
fallback_options="-S autodetect"

Last edited by JoshWVS (2023-10-05 23:25:56)

Offline

#2 2023-10-05 01:41:30

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,635

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

That error when generating the initramfs was critical. It generated the image, but was missing some important things. There's an issue reading something on the USB bus, I would look into that dir and a dir or two above it, see what you can see.

Offline

#3 2023-10-05 05:42:01

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

JoshWVS wrote:
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] sort: cannot read: '/sys/devices/pci0000:00/0000:00:08.1/0000:2f:00.3/usb3/3-3/3-3.4/3-3.4.3/3-3.4.3:1.1/ep_81/uevent': No such file or directory

That's mkinitcpio#177, or more accurately FS#77789. I was under the impression that it would be finally fixed in systemd 254.4-3.  hmm

Offline

#4 2023-10-05 06:56:12

amhairghin
Member
Registered: 2015-10-14
Posts: 63

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

nl6720 wrote:

That's mkinitcpio#177, or more accurately FS#77789. I was under the impression that it would be finally fixed in systemd 254.4-3.  hmm

Nope, I had exactly the same error yesterday with systemd 254.5-1 installed:

  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img --microcode /boot/*-ucode.img
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] ==> Using drop-in configuration file: 'nvidia.conf'
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] ==> Starting build: '6.5.5-arch1-1'
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] sort: no se puede leer: '/sys/devices/pci0000:00/0000:00:08.3/0000:17:00.0/usb11/11-1/11-1.2/11-1.2:1.1/ep_81/uevent': No existe el fichero o el directorio
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET] modprobe: ERROR: missing parameters. See -h.
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
  [2023-10-04T18:04:35+0200] [ALPM-SCRIPTLET]   -> Running build hook: [kms]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
  [2023-10-04T18:04:36+0200] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Image generation successful
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect --microcode /boot/*-ucode.img
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Using drop-in configuration file: 'nvidia.conf'
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET] ==> Starting build: '6.5.5-arch1-1'
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET]   -> Running build hook: [base]
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
  [2023-10-04T18:04:38+0200] [ALPM-SCRIPTLET]   -> Running build hook: [kms]
  [2023-10-04T18:04:40+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'ast'
  [2023-10-04T18:04:42+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET]   -> Running build hook: [keymap]
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET]   -> Running build hook: [consolefont]
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET]   -> Running build hook: [block]
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'aic94xx'
  [2023-10-04T18:04:43+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'bfa'
  [2023-10-04T18:04:44+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qed'
  [2023-10-04T18:04:44+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla1280'
  [2023-10-04T18:04:44+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla2xxx'
  [2023-10-04T18:04:44+0200] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'wd719x'
  [2023-10-04T18:04:44+0200] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
  [2023-10-04T18:04:45+0200] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
  [2023-10-04T18:04:47+0200] [ALPM-SCRIPTLET] ==> Generating module dependencies
  [2023-10-04T18:04:47+0200] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
  [2023-10-04T18:04:48+0200] [ALPM-SCRIPTLET] ==> Image generation successful

Today my machine didn't boot, giving a device not found error (in my case using PARTUUID). I almost had a heart attack thinking that I had just broken my hard disk (which is new), but when I pulled the thread I found this same problem.

Offline

#5 2023-10-05 07:56:20

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,866

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

nl6720 wrote:
JoshWVS wrote:
[2023-10-04T19:14:47-0400] [ALPM-SCRIPTLET] sort: cannot read: '/sys/devices/pci0000:00/0000:00:08.1/0000:2f:00.3/usb3/3-3/3-3.4/3-3.4.3/3-3.4.3:1.1/ep_81/uevent': No such file or directory

That's mkinitcpio#177, or more accurately FS#77789. I was under the impression that it would be finally fixed in systemd 254.4-3.  hmm

At least we have the file opt out option now.

Can all the affected try to create a file "/etc/systemd/do-not-udevadm-trigger-on-update" and then check whether they can reproduce? simplest to retrigger this is reinstalling the systemd package

Offline

#6 2023-10-05 07:56:42

seth
Member
Registered: 2012-09-03
Posts: 52,140

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

https://gitlab.archlinux.org/archlinux/ … ote_128870

Abuse a cat.

mapfile -t mods < <(find /sys/devices -name uevent \
        -exec cat {} + | sort -u | awk -F= '$1 == "MODALIAS" && !_[$0]++')

HWVp.png

Last edited by seth (2023-10-05 08:01:01)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#7 2023-10-05 07:58:15

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

I'm trying to think of way to make the autodetect hook give up on autodetection and add all modules if it encounters this error.

Offline

#8 2023-10-05 08:04:05

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

seth wrote:

Abuse a cat.

I'm not sure how that will help. The process substitution masks the exit code as well.
Best I've come up this far is:

uevents="$(find /sys/devices -name uevent -exec sort -u {} +)" || return 1
mapfile -t mods < <(printf '%s\n' "$uevents" | awk -F= '$1 == "MODALIAS" && !_[$0]++')

Now I just need the autodetect hook to fail in:

mapfile -t mods < <(auto_modules "$KERNELVERSION")

Edit: oh, wait, if auto_modules fails, then $mods in autodetect will be empty. This could work.

Last edited by nl6720 (2023-10-05 08:06:05)

Offline

#9 2023-10-05 08:08:21

seth
Member
Registered: 2012-09-03
Posts: 52,140

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

cats don't care whether there's litt… file, so you get sort to deal w/ the stdin produced by cat and not fail on non-existing files.

I wrote that before I saw you're all-hands-abandon-ship suggestion - is that actually desireable/necessary?
Would one not rather re-trigger auto_modules (at least once after a short delay, given we know there's pot. for a race condition)?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#10 2023-10-05 08:13:30

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

The thought of retrying didn't even occur to me, it might be worth a try. I was thinking of adding `udevadm settle` there, but since it doesn't work in the pacman hook, I had no hope for it.

Offline

#11 2023-10-05 08:15:36

amhairghin
Member
Registered: 2015-10-14
Posts: 63

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

A question (it may be a bit offtopic, but it's to know). Would this problem happen using dracut instead of mkinitcpio?

Offline

#12 2023-10-05 08:52:06

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,866

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

... maybe. It's ultimately a race condition getting produced by a device file not being present when the mkinitcpio hook runs. Whether that same file will be present when dracuts generation runs/whether it has better mechanisms to wait for this is not entirely clear. The "simplest" mitigation while still doing the trigger would be to just add an explicit sleep to the script getting invoked by the udev systemd hook.

Last edited by V1del (2023-10-05 08:52:39)

Offline

#13 2023-10-05 09:18:40

amhairghin
Member
Registered: 2015-10-14
Posts: 63

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

V1del wrote:

... maybe. It's ultimately a race condition getting produced by a device file not being present when the mkinitcpio hook runs. Whether that same file will be present when dracuts generation runs/whether it has better mechanisms to wait for this is not entirely clear. The "simplest" mitigation while still doing the trigger would be to just add an explicit sleep to the script getting invoked by the udev systemd hook.

Ok. I understand. Thank you very much for your answer.

Offline

#14 2023-10-05 10:04:04

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

I created mkinitcpio!266 (untested) to hopefully fix this. It would be good if someone could give it a try.

A success would be if you still see the

sort: cannot read: '/sys/devices/*/uevent': No such file or directory

message, but the built initramfs boots without issue.

Last edited by nl6720 (2023-10-05 10:04:44)

Offline

#15 2023-10-05 12:16:41

seth
Member
Registered: 2012-09-03
Posts: 52,140

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

Repeating the command is not the prettiest thing to do, but any loops I could think of would look even uglier.

local attempts_left=3
local race_gap
while ((attempts_left--)); do
    uevents="$(find /sys/devices -name uevent -exec sort -u {} +)" && attempts_left=0
    ((attempts_left)) && sleep $((race_gap - attempts_left))
done

I'm sure this pipe can be avoided somehow.

By the power of bashisms… wink

awk -F= '$1 == "MODALIAS" && !_[$0]++' <<< "$uevents"

----

This tries thrice and on failure sleeps 1, then 2 seconds to cover the race:

local attempts_left=3
local race_gap=$attempts_left
local uevents
while ((attempts_left--)); do
    uevents="$(find /sys/devices -name uevent -exec sort -u {} +)" && attempts_left=0
    ((attempts_left)) && sleep $((race_gap - attempts_left))
done
mapfile -t mods < <(awk -F= '$1 == "MODALIAS" && !_[$0]++' <<< "$uevents")

inb4

Due to us not being able to disable the bloody scratchpad, you gotta beg for an account => here

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#16 2023-10-05 23:25:11

JoshWVS
Member
Registered: 2023-10-05
Posts: 3

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

V1del wrote:

Can all the affected try to create a file "/etc/systemd/do-not-udevadm-trigger-on-update" and then check whether they can reproduce? simplest to retrigger this is reinstalling the systemd package

Awesome, that worked for me--thank you! After creating that file, I was able to pacman -Suy and restart successfully; the mkinitcpio logs were clean.

nl6720 wrote:

I created mkinitcpio!266 (untested) to hopefully fix this. It would be good if someone could give it a try.

I'm happy to try testing this if helpful, but could you please give me some pointers on how to build it? Also I haven't been able to reproduce my USB bus error again (even after removing /etc/systemd/do-not-udevadm-trigger-on-update and trying a few rounds of sudo pacman -S systemd and sudo udevadm trigger; sudo mkinitcpio -p linux), so I might not be a very useful test subject anymore...

Offline

#17 2023-10-07 13:43:29

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

JoshWVS wrote:

I'm happy to try testing this if helpful, but could you please give me some pointers on how to build it?

Try this PKGBUILD.

seth wrote:

By the power of bashisms… wink

If https://unix.stackexchange.com/a/584788 is correct, then herestrings originate from the superior zsh.  big_smile

Last edited by nl6720 (2023-10-07 13:56:30)

Offline

#18 2023-10-21 01:39:18

JoshWVS
Member
Registered: 2023-10-05
Posts: 3

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

nl6720 wrote:

Sorry, I've tried a few more times and I'm still unable to reproduce my original issue. :-/ I've removed /etc/systemd/do-not-udevadm-trigger-on-update--if I ever see this again, I'll try your PKGBUILD and report back. Thanks for taking the time to write the patch!

Offline

#19 2023-10-22 07:06:36

nl6720
The Evil Wiki Admin
Registered: 2016-07-02
Posts: 611

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

seth wrote:

This tries thrice and on failure sleeps 1, then 2 seconds to cover the race:

local attempts_left=3
local race_gap=$attempts_left
local uevents
while ((attempts_left--)); do
    uevents="$(find /sys/devices -name uevent -exec sort -u {} +)" && attempts_left=0
    ((attempts_left)) && sleep $((race_gap - attempts_left))
done
mapfile -t mods < <(awk -F= '$1 == "MODALIAS" && !_[$0]++' <<< "$uevents")

Thanks! I also need it to `return 1` if all attempts fail.

Offline

#20 2023-10-22 07:52:08

seth
Member
Registered: 2012-09-03
Posts: 52,140

Re: [SOLVED] Can't boot after system upgrade ("ERROR: device not found")

local attempts_left=3
local race_gap=$attempts_left
local uevents
local success=false
while ((attempts_left--)); do
    uevents="$(find /sys/devices -name uevent -exec sort -u {} +)" && attempts_left=0 && success=true
    ((attempts_left)) && sleep $((race_gap - attempts_left))
done
$success || return 1
mapfile -t mods < <(awk -F= '$1 == "MODALIAS" && !_[$0]++' <<< "$uevents")

One could also play w/ the math and encode the result in attempts_left, but this here should be more readable wink

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

Pages: 1

Board footer

Powered by FluxBB