Arch Linux

bigbrettt

Member

Registered: 2018-11-19

Posts: 7

fsck failure on boot

When I powered on my laptop (Dell Precision 5520) this morning, I was greeted with the following boot error:

:: performing fsck on 'UUID=<number>`
fsck.ext2: Input/output error while trying to open /dev/nvm0n1p3
/dev/nvme0n1p3:
The superblock could not be read or does not describe a valid ext2/ext3/ext filesystem. If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else), then the superblock is corrupt and you might try running e2fsck with an alternate superblock:

e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>

ERROR: fsck failed on 'UUID=<number>'
:: mounting 'UUID=<number>' on real root
mount: /new_root: no valid filesystem type specified.
ERROR: failed to mount 'UUID=<number>' on real root
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off

However, when I boot into a live USB (Kubuntu) I'm able to mount the rootFS without issues.

From the live USB, if I run

e2fsck -f -C 0 /dev/nvme0n1p3

it completes without errors, leading me to think the filesystem is intact and there aren't issues with it.

Next I tried to chroot into the rootFS and tried to rebuild the initramFS and rebuild/reinstall GRUB, using the following procedure:

sudo mount /dev/nvme0n1p3 /mnt/
sudo mount /dev/nvme0n1p1 /mnt/boot/EFI
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys

sudo chroot /mnt

mkinitcpio -P

grub-install --target=x86_64-efi --efi-directory=/boot/EFI --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg

# unmount and exit

Unfortunately this made no difference, and the same issue persists. Note that when I rebuilt the initramFS without rebuilding GRUB, the BIOS was not able to find a bootable image anymore and I never even hit the GRUB menu.

I checked to see if the label for /dev/nvm30n1p3 in my fstab was wrong, which I know can cause issues, but I don't think this is the case.

Here is the output from blkid:

/dev/nvme0n1p1: UUID="66E1-49EB" TYPE="vfat" PARTUUID="0912101a-9637-8a4c-aa2f-d5fe4d27af31"
/dev/nvme0n1p3: UUID="103e7a13-91ff-49fa-bd60-de67a83d7de6" TYPE="ext4" PARTUUID="b4615deb-97dd-ca45-b84b-27502367d6f6"
/dev/sda1: UUID="2020-04-23-07-59-42-00" LABEL="Kubuntu 20.04 LTS amd64" TYPE="iso9660" PTUUID="6511e92e" PTTYPE="dos" PARTUUID="6511e92e-01"

And my fstab:

# Static information about the filesystems.
# See fstab(5) for details.
 
# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p3
UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6	/         	ext4      	rw,relatime	0 1
 
# /dev/nvme0n1p2
UUID=41502da4-b96c-4a4c-888e-b17ec143f50c	none      	swap      	defaults,pri=-2	0 0

And my grub config is here:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
 
### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="save"
fi
 
if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi
 
export menuentry_id_option
 
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi
 
function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
 
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}
 
if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
    font="/usr/share/grub/unicode.pf2"
fi
 
if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###
 
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	savedefault
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
	echo	'Loading Linux linux-lts ...'
	linux	/boot/vmlinuz-linux-lts root=UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6 rw  loglevel=3
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initramfs-linux-lts.img
}
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	menuentry 'Arch Linux, with Linux linux-lts' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-advanced-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
		echo	'Loading Linux linux-lts ...'
		linux	/boot/vmlinuz-linux-lts root=UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6 rw  loglevel=3
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-lts.img
	}
	menuentry 'Arch Linux, with Linux linux-lts (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-fallback-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
		echo	'Loading Linux linux-lts ...'
		linux	/boot/vmlinuz-linux-lts root=UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6 rw  loglevel=3
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-lts-fallback.img
	}
	menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
		echo	'Loading Linux linux ...'
		linux	/boot/vmlinuz-linux root=UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6 rw  loglevel=3
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux.img
	}
	menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-103e7a13-91ff-49fa-bd60-de67a83d7de6' {
	savedefault
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root 103e7a13-91ff-49fa-bd60-de67a83d7de6
		echo	'Loading Linux linux ...'
		linux	/boot/vmlinuz-linux root=UUID=103e7a13-91ff-49fa-bd60-de67a83d7de6 rw  loglevel=3
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-fallback.img
	}
}
 
### END /etc/grub.d/10_linux ###
 
### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###
 
### BEGIN /etc/grub.d/25_bli ###
if [ "$grub_platform" = "efi" ]; then
  insmod bli
fi
### END /etc/grub.d/25_bli ###
 
### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###
 
### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
	fwsetup --is-supported
	if [ "$?" = 0 ]; then
		menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
			fwsetup
		}
	fi
fi
### END /etc/grub.d/30_uefi-firmware ###
 
### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###
 
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###
 
### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

I'm at a loss for how to fix this, and would really appreciate any help. This is my work machine so I'm stuck until I can boot it.

jl2

Member

From: Switzerland

Registered: 2022-06-01

Posts: 209

Website

Re: fsck failure on boot

Any recent kernel update? (Tip: /var/log/pacman.log)
Run dmesg after running e3fsck, at least look at the last lines. (It might give a hint on the error)

---

cheese for everyone!

bigbrettt

Member

Registered: 2018-11-19

Posts: 7

Re: fsck failure on boot

No recent kernel update, but wouldn't all of this happen even before the kernel is loaded?  Or am I misunderstanding something? If I did have a kernel issue would you mind explaining how it would affect things at this stage?

I can't run e2fsck in the recovery shell because /dev/nvm0n1p3 isn't present in the filesystem. I ran it in the live USB image. And nothing was present in dmesg in the liveUSB about e2fsck there.

loqs

Member

Registered: 2014-03-06

Posts: 16,838

Re: fsck failure on boot

Can you reproduce the issue with the kernel parameters

module_blacklist=rtsx_pci,rtsx_pci_sdmmc

https://bugs.archlinux.org/task/79439

Have you booted successfully since the last kernel update?

The bootloader or firmware loads the kernel and initrd,  the kernel then probes for hardware where it appears to be failing to find the NVME containing the root file-system.

bigbrettt

Member

Registered: 2018-11-19

Posts: 7

Re: fsck failure on boot

Have you booted successfully since the last kernel update?

Yes

I'm using LTS kernel so don't believe that issue would affect me, right? output of mkinitcpio -P shows "Starting build: 6.1.46-1-lts".

That said, I do see a bunch of "WARNING: possibly missing firmware for module: 'XXX'" warnings showing up when I run mkinitcpio -P. Would this cause any related issues?

loqs

Member

Registered: 2014-03-06

Posts: 16,838

Re: fsck failure on boot

I'm using LTS kernel so don't believe that issue would affect me, right?

The bad commit was also back ported to 6.1.46.  From the rescue prompt are there any rtsx modules loaded?
If not if you boot using the fallback initrd can you mount a USB thumb drive and write the contents of dmesg to it then post that to a pastebin?

Last edited by loqs (2023-09-18 19:24:53)