Arch Linux

gilbs

Member

Registered: 2016-12-28

Posts: 11

[SOLVED] "modules.devname not found" at boot after each kernel upgrade

Hi,

I've been facing the following issue for a few months: each time I perform a system upgrade that triggers mkinitcpio, it ends up showing the following warning at boot:

Warning: /lib/modules/5.8.8-arch1-1/modules.devname not found - ignoring"
                      ^ ^ ^ notice old kernel version, why?

Then it asks for my LUKS password as expected, but the keyboard does not respond. My current (cumbersome) workaround when this happens is to make a arch-chroot from live USB and reinstall the latest kernel available in pacman cache like so:

pacman -U /var/cache/pacman/pkg/linux-5.11.4.arch1-1-x86_64.pkg.tar.zst

Note that running "mkinitcpio -p linux" within the chroot instead of reinstalling the kernel does not have any effect on the issue, I really have to reinstall the kernel to be able to boot again. I've seen a few threads on BBS that relates similar issues, but it seems that they were all able to fix the issue permanently when for me it keeps coming back over and over again.

I don't think my setup is overly complex: I boot through EFISTUB on a fully encrypted drive and sign kernel images using sbupdate for secure boot. Unless I'm missing something, everything else is vanilla. I'm a bit clueless, so I'll post some random snippets, but tell me if you need more info to help:

$ sudo fdisk -l
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: KXG50ZNV512G NVMe TOSHIBA 512GB         
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F7270836-83BA-4690-94E9-3FD39FCA4581

Device             Start        End   Sectors  Size Type
/dev/nvme0n1p1      2048    2099199   2097152    1G EFI System
/dev/nvme0n1p2   2099200  991954943 989855744  472G Linux filesystem
/dev/nvme0n1p3 991954944 1000215182   8260239  3.9G Linux swap


Disk /dev/mapper/cryptroot: 472 GiB, 506801946624 bytes, 989847552 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/swap: 3.94 GiB, 4229242368 bytes, 8260239 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

$ cat /etc/fstab 
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/cryptroot LABEL=root
UUID=1ec0d8a6-a459-4455-9a2e-1cda5be3ec4d	/         	btrfs     	rw,relatime,compress=lzo,ssd,space_cache,subvolid=257,subvol=/@,subvol=@	0 0

# /dev/mapper/cryptroot LABEL=root
UUID=1ec0d8a6-a459-4455-9a2e-1cda5be3ec4d	/var         	btrfs     	rw,relatime,compress=lzo,ssd,space_cache,subvolid=264,subvol=/@var,subvol=@var0 0

# /dev/mapper/cryptroot LABEL=root
UUID=1ec0d8a6-a459-4455-9a2e-1cda5be3ec4d	/home     	btrfs     	rw,relatime,compress=lzo,ssd,space_cache,subvolid=258,subvol=/@home,subvol=@home	0 0

# /dev/mapper/cryptroot LABEL=root
UUID=1ec0d8a6-a459-4455-9a2e-1cda5be3ec4d	/.snapshots	btrfs     	rw,relatime,compress=lzo,ssd,space_cache,subvolid=259,subvol=/@snapshots,subvol=@snapshots	0 0

# /dev/nvme0n1p1
UUID=8929-D507      	/boot 	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

/dev/mapper/swap              none          swap      defaults              0      0

$ sudo efibootmgr -v
BootCurrent: 0007
Timeout: 0 seconds
BootOrder: 0007,0009,0000,0008,000A
Boot0000* KeyTool	PciRoot(0x0)/Pci(0x1d,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-08-0D-02-00-38-12-B7)/HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(\EFI\KeyTool-signed.efi)
Boot0001* Diskette Drive	BBS(Floppy,Diskette Drive,0x0)..BO
Boot0002* M.2 PCIe SSD	BBS(HD,P1: P0: KXG50ZNV512G NVMe TOSHIBA 512GB,0x0)..BO
Boot0003* USB Storage Device	BBS(USB,Corsair Slider 3.0 000D,0x0)..BO
Boot0004* CD/DVD/CD-RW Drive	BBS(CDROM,CD/DVD/CD-RW Drive,0x0)..BO
Boot0005* UEFI: KXG50ZNV512G NVMe TOSHIBA 512GB, Partition 1	HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(EFI\boot\bootx64.efi)..BO
Boot0006* UEFI: Corsair Slider 3.0 000D, Partition 1	PciRoot(0x0)/Pci(0x14,0x0)/USB(0,0)/HD(1,MBR,0x126c99e0,0x2bf01c,0x1200)..BO
Boot0007* ArchLinux	PciRoot(0x0)/Pci(0x1d,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-08-0D-02-00-38-12-B7)/HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(\EFI\Arch\linux-signed.efi)
Boot0008* KeyTool (unsafe)	PciRoot(0x0)/Pci(0x1d,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-08-0D-02-00-38-12-B7)/HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(\EFI\KeyTool.efi)
Boot0009* ArchLinux (unsafe)	PciRoot(0x0)/Pci(0x1d,0x0)/Pci(0x0,0x0)/NVMe(0x1,00-08-0D-02-00-38-12-B7)/HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot000A* Linux Firmware Updater	HD(1,GPT,6d30e36b-08a5-6e41-a643-adb6d49aeefc,0x800,0x200000)/File(\EFI\arch\fwupdx64.efi)

$ cat /etc/sbupdate.conf
# Configuration file for sbupdate
#
# Commented-out variables show default values.

# Common settings
#
# KEY_DIR          Directory with the custom Secure Boot keys
# ESP_DIR          EFI System Partition location
# OUT_DIR          Relative path on ESP for signed kernel images
# SPLASH           Splash image file. Use "/dev/null" to disable splash.
# BACKUP           Whether to back up old signed kernel images
# EXTRA_SIGN       An array of additional files to sign
# CMDLINE_DEFAULT  Default kernel command line (REQUIRED)

#KEY_DIR="/etc/efi-keys"
KEY_DIR="/secureboot"
ESP_DIR="/boot"
OUT_DIR="EFI/Arch"
#SPLASH="/usr/share/systemd/bootctl/splash-arch.bmp"
BACKUP=1
#EXTRA_SIGN=()
#CMDLINE_DEFAULT=""
CMDLINE_DEFAULT="rw cryptdevice=UUID=9742bff9-45e7-42df-860e-a1336c573527:cryptroot root=/dev/mapper/cryptroot rootflags=subvol=@"

# Per-kernel configuration
#
# For each kernel <NAME> the following optional settings may be added:
#
#   CMDLINE[<NAME>]="..."  # Alternative kernel command line
#                          # Default value: ${CMDLINE_DEFAULT}
#   INITRD[<NAME>]="..."   # Alternative initramfs file
#                          # Default value: /boot/initramfs-<NAME>.img
#
# Each kernel can be mapped to multiple configs using the CONFIGS
# variable, in which case CMDLINE and INITRD are set per config.
#
# Examples:
#   CMDLINE["linux-lts"]="custom command line"
#   INITRD["linux-lts"]="/boot/initramfs-linux-custom.img"
#
#   # Generates variants with default and fallback initramfs
#   CONFIGS["linux"]="linux linux-fallback"
#
# Note: Intel and AMD microcode updates are handled automatically and
# do not need to be specified in INITRD.

I made two changes in mkinitcpio.conf:
- I added a few standard hooks
- I had to set lz4 compression recently because they changed the default compression to zstd which caused a kernel panic at boot in my case

$ cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=()

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base udev autodetect modconf block filesystems keyboard keymap consolefont modconf block encrypt filesystems fsck shutdown)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

I also noticed that running fsck on the boot partition shows this prompt, but I'm unsure if it means the boot partition is corrupted or something (that sounds unrelated to the problem anyways, but maybe it is?):

$ sudo fsck /dev/nvme0n1p1
fsck from util-linux 2.36.2
fsck.fat 4.2 (2021-01-31)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
[123?q]? 

$ sudo mkinitcpio -p linux
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.11.4-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> Generating module dependencies
==> Creating lz4-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.11.4-arch1-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: xhci_pci
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
==> WARNING: Possibly missing firmware for module: qat_4xxx
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> Generating module dependencies
==> Creating lz4-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

Thank you in advance.

Last edited by gilbs (2021-03-12 16:25:51)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,726

Re: [SOLVED] "modules.devname not found" at boot after each kernel upgrade

The warning you're seeing is because you're loading an old kernel. I would say that your custom signed image isn't getting created/copied to the place where the firmware is loading it from.

gilbs

Member

Registered: 2016-12-28

Posts: 11

Re: [SOLVED] "modules.devname not found" at boot after each kernel upgrade

The warning you're seeing is because you're loading an old kernel. I would say that your custom signed image isn't getting created/copied to the place where the firmware is loading it from.

Thanks, that's very relevant.

After taking a closer look at how sbupdate gets the kernel image to sign, it starts to make sense: https://github.com/andreyv/sbupdate/blo … pdate#L102 . I had dangling kernel modules in "/usr/lib/modules/5.8.8-arch1-1/". Not sure where that came from, but pacman told me it wasn't referenced by any package so I removed it. So my hypothesis is that sbupdate copied the signed kernel to the intended location, but signed the old (dangling) kernel image when running from the pacman hook instead of the updated version. As a consequence, the old kernel was unable to load the new modules, hence the crash. Therefore, I'm confident removing that dangling kernel will actually solve my issue, still I think it's a buggy behavior from sbupdate. I might consider raising an issue. Thank you again.

Last edited by gilbs (2021-03-12 21:51:55)