You are not logged in.

Pages: 1

#1 2021-03-12 07:16:11

cmonty14
Member
Registered: 2019-10-05
Posts: 56

[solved] Boot with latest kernel fails - Failed to mount /sysroot

Hello,

after system upgrade (on Friday, 02/26/2021) the system fails to boot with this error
Failed to mount /sysroot

I have attached a photo of the monitor documenting the failure.
https://ibb.co/vxDG9jC

The system was running w/o issues before this upgrade.

With regards to the system setup the root filesystem is using Btrfs with multiple devices (2 identical SSDs):
mkfs.btrfs -d raid0 /dev/sdb1 /dev/sdc1

I can boot a Live-CD and chroot into the system w/o any problems.
This means the root partition mounts w/o any error incl. all Btrfs subvolumes.

I have evaluated that this issue is related to the kernel, because I can boot the system using LTS kernel.
On my system I have installed both kernel, LTS and standard:
linux 5.11.5.arch1-1
linux-api-headers 5.10.13-1
linux-headers 5.11.5.arch1-1
linux-lts 5.10.22-2
linux-lts-headers 5.10.22-2

Can you please advise how to analyse the root cause for this error when booting standard kernel?

Based on my findings I would exclude
- HW
- Grub
- /boot
- filesystem
as the root cause of this issue.

I have upgraded the system to latest kernel release, but the issue is not solved.

THX

Last edited by cmonty14 (2021-04-04 19:06:07)

Offline

#2 2021-03-12 08:31:05

seth
Member
Registered: 2012-09-03
Posts: 59,786

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

Did you see the message in your screenshot which says "do stuff for details"? Did you do stuff? What does stuff tell you?

Also check "uname -a" to ensure you're not booting an old kernel, try to boot the failsafe initramfs and lsinitcpio the initramfs to see whether the btrfs module is added.

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#3 2021-03-12 11:04:41

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

seth wrote:

Did you see the message in your screenshot which says "do stuff for details"? Did you do stuff? What does stuff tell you?

Also check "uname -a" to ensure you're not booting an old kernel, try to boot the failsafe initramfs and lsinitcpio the initramfs to see whether the btrfs module is added.

Can you please advise how to "do stuff for details"?

I cannot boot standard kernel, therefore I cannot do 

uname -a

.
However, I have installed the latest standard and LTS kernel, and when I boot from Grub the latest kernel is always used.

Booting failsafe standard kernel fails with the same error.

This is the output of lsinitcpio:

$ sudo lsinitcpio /boot/initramfs-linux.img | grep btrfs
usr/bin/btrfs
usr/lib/modules/5.11.5-arch1-1/kernel/btrfs.ko
usr/lib/udev/rules.d/64-btrfs.rules

$ sudo lsinitcpio /boot/initramfs-linux-lts.img | grep btrfs
usr/bin/btrfs
usr/lib/modules/5.10.22-2-lts/kernel/btrfs.ko
usr/lib/udev/rules.d/64-btrfs.rules

The modules are built for any kernel equally, there's no difference in config.

Offline

#4 2021-03-12 12:34:15

seth
Member
Registered: 2012-09-03
Posts: 59,786

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

You're dumped into a rescue shell, are you not?
And on top of that there's a message that says "See yaddayaddayadda for details", is there? So you do that in the rescue shell.

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#5 2021-03-13 09:17:12

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

seth wrote:

You're dumped into a rescue shell, are you not?
And on top of that there's a message that says "See yaddayaddayadda for details", is there? So you do that in the rescue shell.

Actually I'm not... the rescue shell is not opening.

Offline

#6 2021-03-13 09:23:16

seth
Member
Registered: 2012-09-03
Posts: 59,786

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

When you press enter??

In that case you'll have to boot a different system (eg. install iso) and can try to access the journal of the installed system, https://wiki.archlinux.org/index.php/Sy … al_to_view

You should also inspect the supposed boot path (mounted and unmounted) for present kernel versions ("file /boot/vmlinuz-linux etc") and the grub config in comparism to "lsblk -f" (uuid mismatches)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#7 2021-03-25 19:48:07

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

seth wrote:

When you press enter??

Yes.

seth wrote:

In that case you'll have to boot a different system (eg. install iso) and can try to access the journal of the installed system, https://wiki.archlinux.org/index.php/Sy … al_to_view

Do you mean to chroot into the system after booting an iso?
Or do you recommend to boot an iso and then display the journal using this command: 

journalctl -D /patch/to/journal -xe

If yes, what is the default location where journal is stored?

seth wrote:

You should also inspect the supposed boot path (mounted and unmounted) for present kernel versions ("file /boot/vmlinuz-linux etc") and the grub config in comparism to "lsblk -f" (uuid mismatches)

In my opinion there's no error with UUID and grub.config.

[root@pc1-desktop ~]# file /boot/vmlinuz-linux
/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.11.9-arch1-1 (linux@archlinux) #1 SMP PREEMPT Wed, 24 Mar 2021 18:53:54 +0000, RO-rootFS, swap_dev 0x8, Normal VGA

[root@pc1-desktop ~]# file /boot/vmlinuz-linux-lts 
/boot/vmlinuz-linux-lts: Linux kernel x86 boot executable bzImage, version 5.10.25-1-lts (linux-lts@archlinux) #1 SMP Sat, 20 Mar 2021 19:37:44 +0000, RO-rootFS, swap_dev 0x8, Normal VGA

[root@pc1-desktop ~]# blkid | grep sda
/dev/sda1: LABEL="archlinux" UUID="78462a70-55ad-4444-9d91-e71e42cce51c" UUID_SUB="c257c349-41d2-475e-b52e-801d7f7a4b24" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="0914a19b-01"
/dev/sda2: LABEL="swap" UUID="f500539a-35cd-46b6-88be-3f027c45a6c6" TYPE="swap" PARTUUID="0914a19b-02"

root@pc1-desktop ~]# ack 78462a70-55ad-4444-9d91-e71e42cce51c /boot/grub/grub.cfg
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-78462a70-55ad-4444-9d91-e71e42cce51c' {
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
	  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
	linux	/@/boot/vmlinuz-linux-lts root=UUID=78462a70-55ad-4444-9d91-e71e42cce51c rw rootflags=subvol=@  loglevel=3
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-78462a70-55ad-4444-9d91-e71e42cce51c' {
	menuentry 'Arch Linux, with Linux linux-lts' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-advanced-78462a70-55ad-4444-9d91-e71e42cce51c' {
		  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
		  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
		linux	/@/boot/vmlinuz-linux-lts root=UUID=78462a70-55ad-4444-9d91-e71e42cce51c rw rootflags=subvol=@  loglevel=3
	menuentry 'Arch Linux, with Linux linux-lts (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-fallback-78462a70-55ad-4444-9d91-e71e42cce51c' {
		  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
		  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
		linux	/@/boot/vmlinuz-linux-lts root=UUID=78462a70-55ad-4444-9d91-e71e42cce51c rw rootflags=subvol=@  loglevel=3
	menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-78462a70-55ad-4444-9d91-e71e42cce51c' {
		  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
		  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
		linux	/@/boot/vmlinuz-linux root=UUID=78462a70-55ad-4444-9d91-e71e42cce51c rw rootflags=subvol=@  loglevel=3
	menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-78462a70-55ad-4444-9d91-e71e42cce51c' {
		  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  --hint-bios=hd1,msdos1 --hint-efi=hd1,msdos1 --hint-baremetal=ahci1,msdos1  78462a70-55ad-4444-9d91-e71e42cce51c
		  search --no-floppy --fs-uuid --set=root 78462a70-55ad-4444-9d91-e71e42cce51c
		linux	/@/boot/vmlinuz-linux root=UUID=78462a70-55ad-4444-9d91-e71e42cce51c rw rootflags=subvol=@  loglevel=3

Offline

#8 2021-03-25 21:04:49

loqs
Member
Registered: 2014-03-06
Posts: 18,124

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

It will be easier to chroot in check the journal.  The default location is /var/log/journal and you would need to prepend the mounts path when accessing it from outside the chroot.

If you want to unlock the root account in the initrd from the chroot copy /usr/lib/initcpio/install/systemd to /etc/initcpio/install/systemd then change line 176 from

    echo 'root:*:::::::' >"$BUILDROOT/etc/shadow"

to

    echo 'root::::::::' >"$BUILDROOT/etc/shadow"

then remake the initrd.

Offline

#9 2021-03-27 18:59:29

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

loqs wrote:

It will be easier to chroot in check the journal.  The default location is /var/log/journal and you would need to prepend the mounts path when accessing it from outside the chroot.

If you want to unlock the root account in the initrd from the chroot copy /usr/lib/initcpio/install/systemd to /etc/initcpio/install/systemd then change line 176 from

    echo 'root:*:::::::' >"$BUILDROOT/etc/shadow"

to

    echo 'root::::::::' >"$BUILDROOT/etc/shadow"

then remake the initrd.

Can you please advise why I should want (or need) to unlock root account in the initrd?

I can boot a live CD / ISO and chroot into the system w/o problems.

However, I see the following issue:
Is journal including information of the boot process failing?
I mean the boot process is failing in an early stage when /sysroot is mounted.

If the journal includes records that are only relevant for the successfull boot process using LTS kernel, than it is useless to start investigation in the journal.

Offline

#10 2021-03-27 19:06:47

loqs
Member
Registered: 2014-03-06
Posts: 18,124

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

cmonty14 wrote:

Can you please advise why I should want (or need) to unlock root account in the initrd?

From the rescue shell you can check the journal for the failing boot,  which covers your point about the on disk journal not recording relevant entries.
You can also check if the device that should be mounted to /sysroot has an entry under /dev and try manually mounting the device to sysroot.

Offline

#11 2021-04-02 09:20:46

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

I can now enter the rescue shell and display journal using command journal -xb.

I have documented the screen content in attached pictures.
pic1
pic2
pic3
pic4

One can identify these error messages:
BTRFS info (device sda1): has skinny extents
BTRFS error (device sda1): failed to read chunk tree
BTRFS error (device sda1): open_ctree failed

Now I can compare this content with journal booting LTS kernel:

Apr 02 10:51:50 archlinux kernel: Btrfs loaded, crc32c=crc32c-generic
Apr 02 10:51:50 archlinux kernel: BTRFS: device label archlinux devid 1 transid 145177 /dev/sda1 scanned by systemd-udevd (161)
Apr 02 10:51:50 archlinux kernel: BTRFS: device label archlinux devid 2 transid 145177 /dev/sdb1 scanned by systemd-udevd (173)
Apr 02 10:51:50 archlinux systemd[1]: Found device SanDisk_SD9TB8W256G1001 archlinux.
░░ Subject: A start job for unit dev-disk-by\x2duuid-78462a70\x2d55ad\x2d4444\x2d9d91\x2de71e42cce51c.device has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit dev-disk-by\x2duuid-78462a70\x2d55ad\x2d4444\x2d9d91\x2de71e42cce51c.device has finished successfully.
░░ 
░░ The job identifier is 29.
Apr 02 10:51:50 archlinux systemd[1]: Reached target Initrd Root Device.
░░ Subject: A start job for unit initrd-root-device.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit initrd-root-device.target has finished successfully.
░░ 
░░ The job identifier is 28.
Apr 02 10:51:50 archlinux systemd[1]: Starting File System Check on /dev/disk/by-uuid/78462a70-55ad-4444-9d91-e71e42cce51c...
░░ Subject: A start job for unit systemd-fsck-root.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit systemd-fsck-root.service has begun execution.
░░ 
░░ The job identifier is 32.
Apr 02 10:51:50 archlinux systemd-fsck[187]: sda1: fsck.btrfs doesn't exist, not checking file system.
Apr 02 10:51:50 archlinux systemd[1]: Finished File System Check on /dev/disk/by-uuid/78462a70-55ad-4444-9d91-e71e42cce51c.
░░ Subject: A start job for unit systemd-fsck-root.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit systemd-fsck-root.service has finished successfully.
░░ 
░░ The job identifier is 32.
Apr 02 10:51:50 archlinux audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck-root comm="systemd" exe="/init" hostname=? addr=? ter>
Apr 02 10:51:50 archlinux systemd[1]: Mounting /sysroot...
░░ Subject: A start job for unit sysroot.mount has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit sysroot.mount has begun execution.

░░ The job identifier is 31.
Apr 02 10:51:50 archlinux kernel: audit: type=1130 audit(1617353510.814:6): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-fsck-root comm="systemd" exe="/i>
Apr 02 10:51:50 archlinux kernel: random: fast init done
Apr 02 10:51:50 archlinux kernel: BTRFS info (device sda1): using free space tree
Apr 02 10:51:50 archlinux kernel: BTRFS info (device sda1): has skinny extents
Apr 02 10:51:50 archlinux kernel: BTRFS info (device sda1): bdev /dev/sdb1 errs: wr 2749920, rd 2772064, flush 0, corrupt 6, gen 0
Apr 02 10:51:50 archlinux kernel: sr 3:0:0:0: Attached scsi CD-ROM sr0
Apr 02 10:51:50 archlinux systemd[1]: Mounted /sysroot.
░░ Subject: A start job for unit sysroot.mount has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit sysroot.mount has finished successfully.
░░ 
░░ The job identifier is 31.
Apr 02 10:51:50 archlinux kernel: BTRFS info (device sda1): enabling ssd optimizations
Apr 02 10:51:50 archlinux systemd[1]: Reached target Initrd Root File System.
░░ Subject: A start job for unit initrd-root-fs.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit initrd-root-fs.target has finished successfully.
░░ 
░░ The job identifier is 30.
Apr 02 10:51:50 archlinux systemd[1]: Starting Reload Configuration from the Real Root...
░░ Subject: A start job for unit initrd-parse-etc.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit initrd-parse-etc.service has begun execution.
░░ 
░░ The job identifier is 34.

The error messages
BTRFS error (device sda1): failed to read chunk tree
BTRFS error (device sda1): open_ctree failed
are not shown in LTS kernel journal.

I'm aware that there's an issue with my BTRFS Raid 1 on 2 SSDs.
However I don't understand why this should be the root cause for failure booting standard kernel.

The kernel files for standand and LTS kernel reside on the same device, so why does booting standard kernel fail when booting LTS kernel is successfull?

Last edited by cmonty14 (2021-04-02 09:22:37)

Offline

#12 2021-04-02 09:49:52

seth
Member
Registered: 2012-09-03
Posts: 59,786

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

Please don't use "-x", it's de-facto spam. Also, journalctl at its best: "Does the terminal actually support escape codes? Nahhh, gonna be fine!"…

On topic:
https://bbs.archlinux.org/viewtopic.php?id=264142

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#13 2021-04-04 19:07:45

cmonty14
Member
Registered: 2019-10-05
Posts: 56

Re: [solved] Boot with latest kernel fails - Failed to mount /sysroot

Issue is fixed.

The root cause was an inconsistency with the BTRFS partitions on device /dev/sda + /dev/sdb.
Once this inconsistency is solved the linux kernel boots as expected.

Thanks for you support!

Offline

Pages: 1

Board footer

Powered by FluxBB