[SOLVED] Can't boot after kernel update. Versioning troubles?

sguertin · 2016-01-21 16:36:22

I just updated (using yaourt), including a kernel update, and now booting fails. It says

:: running early hook [udev]
Warning: /lib/modules/4.3.3-3-ARCH/modules.devname not found - ignoring
starting version 228
:: running hook [udev]
:: Triggering uevents...
Waiting 10 seconds for device /dev/sda5...
ERROR: device '/dev/sda5' not found. Skipping fsck.
ERROR: Unable to find root device '/dev/sda5'.
You are being dropped to a recovery shell
    Type 'exit' to try and continue booting
sh: can't access tty; job control turned off
[rootfs /]#

But I can't type anything at the prompt. (/dev/sda5 is where Arch is installed.)

I have access to a working Debian install on the same box, which is how I got the pacman logs below. It's a Thinkpad X240.

I should mention that I have previously had troubles with my bootloader configuration interacting with kernel updates as described here, but I've checked that by hand, and I don't think that's the problem now.

I have openafs and openafs-modules-dkms installed from the AUR. I also have linux-headers, which openafs forgot to list as a dependency. In past installations, I've had to reinstall at least one of openafs and openafs-modules-dkms when the kernel updates in order to get afs to work, but this go around this is the first kernel update since I installed them. I haven't otherwise done anything fancy with the kernel.

My pacman log from the update:

[2016-01-21 10:25] [PACMAN] Running 'pacman --color auto -Sy'
[2016-01-21 10:25] [PACMAN] synchronizing package lists
[2016-01-21 10:25] [PACMAN] Running 'pacman --color auto -S -u'
[2016-01-21 10:25] [PACMAN] starting full system upgrade
[2016-01-21 10:26] [ALPM] transaction started
[2016-01-21 10:26] [ALPM] upgraded avahi (0.6.32rc-3 -> 0.6.32rc-4)
[2016-01-21 10:26] [ALPM] upgraded lib32-elfutils (0.164-1 -> 0.165-1)
[2016-01-21 10:26] [ALPM] upgraded linux (4.3.3-2 -> 4.3.3-3)
[2016-01-21 10:26] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2016-01-21 10:26] [ALPM-SCRIPTLET] >>> Generating initial ramdisk, using mkinitcpio. Please wait...
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> -k /boot/caroline/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/caroline/initramfs-linux.img
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Starting build: 4.3.3-2-ARCH
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `openafs'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `usbhid'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: No modules were added to the image. This is probably not what you want.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/caroline/initramfs-linux.img
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during the build. The image may not be complete.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> -k /boot/caroline/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/caroline/initramfs-linux-fallback.img -S autodetect
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Starting build: 4.3.3-2-ARCH
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `openafs'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `usbhid'
[2016-01-21 10:26] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: No modules were added to the image. This is probably not what you want.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/caroline/initramfs-linux-fallback.img
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during the build. The image may not be complete.
[2016-01-21 10:26] [ALPM] upgraded linux-headers (4.3.3-2 -> 4.3.3-3)
[2016-01-21 10:26] [ALPM] upgraded pandoc (1.16.0.2-6 -> 1.16.0.2-7)
[2016-01-21 10:26] [ALPM] transaction completed
[2016-01-21 10:47] [PACMAN] Running 'pacman --color auto -U /tmp/yaourt-tmp-sguertin/PKGDEST.6S6/google-chrome-48.0.2564.82-1-x86_64.pkg.tar.xz'
[2016-01-21 10:47] [ALPM] transaction started
[2016-01-21 10:47] [ALPM] upgraded google-chrome (47.0.2526.111-1 -> 48.0.2564.82-1)
[2016-01-21 10:47] [ALPM-SCRIPTLET] ==> Updating desktop MIME database...
[2016-01-21 10:47] [ALPM-SCRIPTLET] ==> Updating icon cache..
[2016-01-21 10:47] [ALPM-SCRIPTLET] ==> NOTE: Custom flags should be put directly in: ~/.config/chrome-flags.conf
[2016-01-21 10:47] [ALPM-SCRIPTLET] ==> NOTE: The launcher is called: 'google-chrome-stable'
[2016-01-21 10:47] [ALPM] transaction completed

Some excerpts that seem meaningful:

[2016-01-21 10:26] [ALPM] upgraded linux (4.3.3-2 -> 4.3.3-3)
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Starting build: 4.3.3-2-ARCH
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `openafs'
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `usbhid'
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: No modules were added to the image. This is probably not what you want.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during the build. The image may not be complete.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> Starting build: 4.3.3-2-ARCH
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `openafs'
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> ERROR: module not found: `usbhid'
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: No modules were added to the image. This is probably not what you want.
[2016-01-21 10:26] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered during the build. The image may not be complete.
[2016-01-21 10:26] [ALPM] upgraded linux-headers (4.3.3-2 -> 4.3.3-3)

From Debian, I ran

lsinitramfs -l /path/to/arch/initramfs-linux.img

.
./VERSION
./bin -> usr/bin
./buildconfig
./config
./dev
./etc
./etc/fstab
./etc/initrd-release
./etc/ld.so.cache
./etc/ld.so.conf
./etc/modprobe.d
./etc/modprobe.d/50-alsa.conf
./etc/mtab -> /proc/self/mounts
./hooks
./hooks/udev
./init
./init_functions
./lib -> usr/lib
./lib64 -> usr/lib
./new_root
./proc
./run
./sbin -> usr/bin
./sys
./tmp
./usr
./usr/bin
./usr/bin/[ -> busybox
...
./usr/bin/yes -> busybox
./usr/lib
./usr/lib/ld-linux-x86-64.so.2
./usr/lib/libacl.so.1
./usr/lib/libattr.so.1
./usr/lib/libblkid.so.1
./usr/lib/libc.so.6
./usr/lib/libcap.so.2
./usr/lib/libcom_err.so.2
./usr/lib/libe2p.so.2
./usr/lib/libext2fs.so.2
./usr/lib/libkmod.so.2
./usr/lib/liblzma.so.5
./usr/lib/libmount.so.1
./usr/lib/libpthread.so.0
./usr/lib/librt.so.1
./usr/lib/libuuid.so.1
./usr/lib/libz.so.1
./usr/lib/modprobe.d
./usr/lib/modprobe.d/usb-load-ehci-first.conf
./usr/lib/modules
./usr/lib/modules/4.3.3-2-ARCH
./usr/lib/modules/4.3.3-2-ARCH/kernel
./usr/lib/systemd
./usr/lib/systemd/systemd-udevd
./usr/lib/udev
./usr/lib/udev/ata_id
./usr/lib/udev/rules.d
./usr/lib/udev/rules.d/50-udev-default.rules
./usr/lib/udev/rules.d/60-persistent-storage.rules
./usr/lib/udev/rules.d/64-btrfs.rules
./usr/lib/udev/rules.d/80-drivers.rules
./usr/lib/udev/scsi_id
./usr/lib64 -> lib
./usr/local
./usr/local/bin -> ../bin
./usr/local/lib -> ../lib
./usr/local/sbin -> ../bin
./usr/sbin -> bin

The version numbers don't make sense to me. Pacman says linux is going from 4.3.3-2 -> 4.3.3-3, but then builds 4.3.3-2, which is what's in the initramfs. Then linux-headers upgraded to 4.3.3-3.

Any idea what's going on, and how I can fix it from Debian/live media?

Last edited by sguertin (2016-01-21 18:37:11)

ukhippo · 2016-01-21 17:30:44

Are you using the standard linux.preset or an edited one?

Edit: see https://wiki.archlinux.org/index.php/Change_root for how to rerun mkinitcpio.

Edit2: inside an arch-chroot what is the output from the following:

grep kver /etc/mkinitcpio.d/linux.preset
file /boot/vmlinuz-linux

Last edited by ukhippo (2016-01-21 17:45:29)

sguertin · 2016-01-21 18:24:27

I edited linux.preset in order to change the paths for initramfs, as recommended in the bootloader configuration thread I linked to above.

# mkinitcpio preset file for the 'linux' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/caroline/vmlinuz-linux"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/caroline/initramfs-linux.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/caroline/initramfs-linux-fallback.img"
fallback_options="-S autodetect"

file /boot/caroline/vmlinuz-linux
vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 4.3.3-3-ARCH (builduser@tobias) #1 SMP PREEMPT Wed Jan 20 08:12, RO-rootFS, swap_dev 0x4, Normal VGA

Also, here's my /etc/mkinitcpio.conf (sans comments)

MODULES=""
BINARIES=""
FILES=""
HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

When should I need to run mkinitcpio? I assumed pacman would take care of running it whenever it's needed.

sguertin · 2016-01-21 18:35:06

Using an old live media to run mkinitcpio -p linux from arch-chroot fixed the problem.

Do you expect that this will happen on every kernel update, or will running mkinitcpio fix the problem until the next configuration change?

Thanks!

ukhippo · 2016-01-21 23:37:52

This will happen on every kernel update, since you're using the "wrong" location for the kernel. Kernel gets installed to /boot, so until you copy it mkinitcpio will generate incorrect initramfs files.
However, Head_on_a_Stick has pointed to a solution in your other thread.

A slight modification would be to combine that solution with a modified linux.preset:
• revert "ALL_kver" to default location (/boot/vmlinuz-linux) in your linux.preset
>> ensures correct kernel is used for building the initramfs files
• use your location of the fallback initramfs in efistub-update.path
• the efistub-update service just needs to copy the kernel
>> the initramfs files were already built into your preferred location

Arch Linux

#1 2016-01-21 16:36:22

[SOLVED] Can't boot after kernel update. Versioning troubles?

#2 2016-01-21 17:30:44

Re: [SOLVED] Can't boot after kernel update. Versioning troubles?

#3 2016-01-21 18:24:27

Re: [SOLVED] Can't boot after kernel update. Versioning troubles?

#4 2016-01-21 18:35:06

Re: [SOLVED] Can't boot after kernel update. Versioning troubles?

#5 2016-01-21 23:37:52

Re: [SOLVED] Can't boot after kernel update. Versioning troubles?

Board footer