[solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

nhasian · 2018-11-02 23:40:00

Both my root and home partitions use btrfs. I don't know if that has anything to do with why it's no longer booting with the latest kernel. NOTE: I can still boot fine with the 4.14 LTS kernel.

When attempting to boot the 4.18 kernel I am dropped to an emergency shell. It says fsck error 2 no such file or directory while executing fsck.ext2 for /dev/sdb2. mount: /new_root: wrong fs type, bad option, bad superblock on /dev/sdb2.
It shouldn't run fsck.ext2 against /dev/sdb2 because that is a BTRFS filesystem, not ext. Manually typing "mount /dev/sdb2 /new_root" also fails, I imagine because it doesn't understand what btrfs is. On a hunch, I added the module btrfs to the mkinitcpio.conf and ran "mkinitcpio -p linux" but that did not resolve the issue.

my mkinitcpio.conf:

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=(btrfs nvidia nvidia_modeset nvidia_uvm nvidia_drm)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base udev resume autodetect modconf block filesystems keyboard fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

here's my /etc/fstab:

# <file system> <dir> <type> <options> <dump> <pass>

# /dev/sdb2 LABEL=root
UUID=a7f536bf-069a-444a-aba7-6535c29fd779	/         	btrfs     	rw,noatime,compress=lzo,ssd,discard,space_cache,subvolid=5,subvol=/	0 0

# /dev/sdb1
UUID=A4F3-B8AD      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro	0 2

# /dev/sdb3 LABEL=home
UUID=8c7e9502-0b13-48e7-9c57-56379f630060	/home     	btrfs     	rw,noatime,compress=lzo,ssd,discard,space_cache,subvolid=5,subvol=/	0 0

# /dev/sdb5
UUID=0fc45c56-7800-489b-b080-13f1469edf9f	none      	swap      	defaults  	0 0

# /dev/sdb4
UUID=0549D8381CAA47C2	/Data	ntfs-3g uid=1000,gid=1000,dmask=022,fmask=133	0 0 

# mounting music folder
/Data/Music		/home/username/Music	none	bind

output of file /boot/* && file /boot/**/* | grep linux:

/boot/EFI:                              directory
/boot/initramfs-linux-fallback.img:     gzip compressed data, last modified: Fri Nov  2 23:09:00 2018, from Unix, original size 120985600
/boot/initramfs-linux.img:              gzip compressed data, last modified: Fri Nov  2 23:08:52 2018, from Unix, original size 48762880
/boot/initramfs-linux-lts-fallback.img: gzip compressed data, last modified: Fri Nov  2 21:50:42 2018, from Unix, original size 116932608
/boot/initramfs-linux-lts.img:          gzip compressed data, last modified: Fri Nov  2 21:50:35 2018, from Unix, original size 49868288
/boot/intel-ucode.img:                  ASCII cpio archive (SVR4 with no CRC)
/boot/loader:                           directory
/boot/vmlinuz-linux:                    Linux kernel x86 boot executable bzImage, version 4.18.16-arch1-1-ARCH (builduser@heftig-28760) #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018, RO-rootFS, swap_dev 0x5, Normal VGA
/boot/vmlinuz-linux-lts:                Linux kernel x86 boot executable bzImage, version 4.14.78-1-lts (builduser@andyrtr) #1 SMP Sun Oct 21 14:04:15 CEST 2018, RO-rootFS, swap_dev 0x4, Normal VGA

output of /lib/modules/4.18.16-arch1-1-ARCH/:

extramodules  modules.alias      modules.builtin      modules.dep      modules.devname  modules.softdep  modules.symbols.bin
kernel        modules.alias.bin  modules.builtin.bin  modules.dep.bin  modules.order    modules.symbols

output of lsblk -o +UUID:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT                                  UUID
sda      8:0    0 238.5G  0 disk                                             
├─sda1   8:1    0   300M  0 part                                             BA4E-B214
├─sda2   8:2    0   128M  0 part                                             
├─sda3   8:3    0 237.2G  0 part                                             00964F14964F099C
└─sda4   8:4    0   900M  0 part                                             0292503792503201
sdb      8:16   0 465.8G  0 disk                                             
├─sdb1   8:17   0   512M  0 part /boot                                       A4F3-B8AD
├─sdb2   8:18   0    25G  0 part /                                           a7f536bf-069a-444a-aba7-6535c29fd779
├─sdb3   8:19   0   220G  0 part /home                                       8c7e9502-0b13-48e7-9c57-56379f630060
├─sdb4   8:20   0 212.3G  0 part /Data                                       0549D8381CAA47C2
└─sdb5   8:21   0     8G  0 part [SWAP]                                      0fc45c56-7800-489b-b080-13f1469edf9f
sdc      8:32   0   3.7T  0 disk                                             
├─sdc1   8:33   0   128M  0 part                                             
└─sdc2   8:34   0   3.7T  0 part /run/media/username/Seagate Backup Plus Drive 349CE0319CDFEB7E
sr0     11:0    1  1024M  0 rom

Last edited by nhasian (2018-11-03 03:39:28)

nhasian · 2018-11-02 23:46:15

Welp, I included a ton of information in my initial post and after reviewing the post after I submitted it I noticed that my external USB Seagate hard disk was connected. That reminded me of a bug I came across while searching for my issue https://bugs.archlinux.org/task/60272. I wasn't booting FROM a usb drive. this wouldn't affect me right? Wrong. I disconnected my USB drive and then the problem instantly vanished. I'll mark the issue as solved, just wanted to leave this up in case someone else runs into this issue. Looks like I was bitten by that bug as well.

Scimmia · 2018-11-03 00:33:11

And as I said in the reopen denial, that bug has nothing to do with your problem. Fix your bootloader config.

nhasian · 2018-11-03 03:27:47

Hello Scimmia,

Thanks for your response. I did some further testing and adding "usb_storage" and "uas" to the MODULES list in /etc/mkinitcpio.conf did not fix the problem. I use systemd-boot for my bootloader. You still think it's a bootloader issue if it works with the LTS kernel but not the latest kernel? Perhaps it's a kernel bug?

EDIT: Nevermind. I fixed it by updating systemd-boot with

# bootctl update

TIL: systemd-boot does not automatically get updated when updating your system with pacman -syu. I created a hook for pacman to run it again if systemd is updated.

Scimmia wrote:

And as I said in the reopen denial, that bug has nothing to do with your problem. Fix your bootloader config.

Last edited by nhasian (2018-11-03 03:38:41)

Scimmia · 2018-11-03 04:25:16

It's "fixed" because the device node created is random. Fix your *config*.

nhasian · 2018-11-03 23:41:14

Scimmia wrote:

It's "fixed" because the device node created is random. Fix your *config*.

After some more digging, I think I have finally figured out what you are talking about. I noticed the options in the arch.conf pointed to /dev/sdb2. I changed it to the partuuid.

old /boot/loader/entries/arch.conf:

title		Arch Linux
linux		/vmlinuz-linux
initrd		/intel-ucode.img
initrd		/initramfs-linux.img
options		root=/dev/sdb2 rw resume=UUID=0fc45c56-7800-489b-b080-13f1469edf9f nvidia-drm.modeset=1

here is my new /boot/loader/entries/arch.conf:

title		Arch Linux
linux		/vmlinuz-linux
initrd		/intel-ucode.img
initrd		/initramfs-linux.img
options		root=PARTUUID=8d304e16-e203-491b-8693-0335624c7952 rw resume=UUID=0fc45c56-7800-489b-b080-13f1469edf9f nvidia-drm.modeset=1

Arch Linux

#1 2018-11-02 23:40:00

[solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

#2 2018-11-02 23:46:15

Re: [solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

#3 2018-11-03 00:33:11

Re: [solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

#4 2018-11-03 03:27:47

Re: [solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

#5 2018-11-03 04:25:16

Re: [solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

#6 2018-11-03 23:41:14

Re: [solved] Unable to boot with latest linux kernel 4.18.16 with btrfs

Board footer