[SOLVED] Kernel Panic and No Booting after BIOS Upgrade

FrustratedMSIuser · 2021-05-23 10:14:49

Hello everyone!

Problem

I have the MSI motherboard Z490-A PRO and upgraded the BIOS to 7C75v29.
Since then I am not able to boot my Arch Linux installation any more.

...
ACPI BIOS Error (bug): Could not resolve symbol [\_PR.PR00._CPC], AE_NOT_FOUND (20210105/psargs-330)
ACPI Error: Aborting method \_PR.PR14._CPC due to previous error (AE_NOT_FOUND) (20210105/psparse-529)
ACPI BIOS Error (bug): Could not resolve symbol [\_PR.PR00._CPC], AE_NOT_FOUND (20210105/psargs-330)
ACPI Error: Aborting method \_PR.PR15._CPC due to previous error (AE_NOT_FOUND) (20210105/psparse-529)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 15 PID: 1 Comm: swapper/0 Not tainted 5.12.5-arch-1 #1
Hardware name: Micro-Star International Co., Ltd. MS-7C75/Z490-A PRO (MS-7C75), BIOS 2.90 04/20/2021
Call Trace:
 dump_stack+0x76/0x94
 panic_0x112/0x2e8
 mount_block_root+0x310/0x31f
 ? rest_init+0xbe/0xbe
 prepare_namespace+0x136/0x165
 ? rest_init+0xbe/0xbe
 kernel_init+0xa/0x112
 ret_from_fork+0x1f/0x30
Kernel Offset: 0x600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

However, I am still able to boot my Windows installation.
Note that the BIOS bugs were also present before the BIOS upgrade and are probably related to this bug.
So, despite of the bugs I was able to boot before the BIOS upgrade.

My Setup

I have four partitions (/dev/sdb): EFI/Boot, SWAP (unused), enc. root, enc. /home
lsblk -f

NAME          FSTYPE      FSVER            LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0         squashfs    4.0                                                                              
sda                                                                                                        
├─sda1        vfat        FAT32                        9092-57FC                                           
├─sda2                                                                                                     
├─sda3        ntfs                                     086C92E86C92CFB4                                    
└─sda4        ntfs                                     D430A26930A25272                                    
sdb                                                                                                        
├─sdb1        vfat        FAT32            EFI         576A-5BE0                             453.8M    11% /boot
├─sdb2        swap        1                SWAP        23f0ff72-10e9-4662-bb06-97c633588e60                
├─sdb3        crypto_LUKS 2                            aac36f28-d230-41c2-877b-740caf0a46b6                
│ └─cryptroot ext4        1.0              CRYPTROOT   2d9dffb2-0e13-4e58-9838-75d7c99e9f90   90.9G    22% /
└─sdb4        crypto_LUKS 2                            c6ce2052-f4f3-499e-8040-2c3ddf7cf7c5

I am booting directly with EFISTUB.
Changes to the NVRAM of my motherboard, using efibootmgr, are not persistent.
So, I use the UEFI Shell v2 from the Arch Linux Installer to create boot entries, like explained here.
efibootmgr -v

BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0002,0000,0001,0003,0004
Boot0000* Windows Boot Manager	HD(1,GPT,36ed0845-8bca-4e1a-b45c-5451d97a34be,0x800,0x32000)/File(\EFI\MICROSOFT\BOOT\BOOTMGFW.EFI)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...,................
Boot0001* Arch Linux	PciRoot(0x0)/Pci(0x17,0x0)/Sata(3,65535,0)/HD(1,GPT,f36ce601-041e-7343-9126-d90afe34ca0e,0x800,0x100000)/File(\vmlinuz-linux)    cryptdevice=UUID=aac36f28-d230-41c2-877b-740caf0a46b6:cryptroot root=LABEL=CRYPTROOT rw initrd=\intel-ucode.img initrd=\initramfs-linux.img.
Boot0002* Arch Fallback	PciRoot(0x0)/Pci(0x17,0x0)/Sata(3,65535,0)/HD(1,GPT,f36ce601-041e-7343-9126-d90afe34ca0e,0x800,0x100000)/File(\vmlinuz-linux)    cryptdevice=UUID=aac36f28-d230-41c2-877b-740caf0a46b6:cryptroot root=LABEL=CRYPTROOT rw initrd=\intel-ucode.img initrd=\initramfs-linux-fallback.img.
...

/etc/fstab

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/mapper/cryptroot LABEL=CRYPTROOT
UUID=2d9dffb2-0e13-4e58-9838-75d7c99e9f90	/         	ext4      	rw,relatime,commit=60	0 1

# /dev/mapper/crypthome LABEL=CRYPTHOME
UUID=ec6e77c4-320f-4e61-9717-0d34e31150b0	/home     	ext4      	rw,relatime,commit=60	0 2

# /dev/sda1 LABEL=EFI
UUID=576A-5BE0      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2

My custom mkinitcpio.conf /etc/mkinitcpio-custom.conf

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=(hid)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=()

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base udev autodetect keyboard keymap modconf block encrypt filesystems fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

Things I Tried

Regenerating initramfs via arch-chroot
```
mkinitcpio -P
```
Upgrading my system via arch-chroot
```
pacman -Syu
```
Booting into fallback
Removing the Intel microcode from my boot options, i.e. removing initrd=\intel-ucode.img
Looking into similar looking problems in this forum, e.g 1 and 2
Downgrading my BIOS to default, but this is not possible because MSI only allows upgrades ...
Reinstalling the kernel
```
pacman -S linux
```

But the kernel panic remains.

In the post 2 the problem was due to wrongly configured /etc/fstab and /etc/crypttab.
In my case, I have not touched them since my initial installation. So, my guess is that the problem lies elsewhere.

But I really hope it is just a dumb and small mistake, as in the post.
Thank you all in advance!

Last edited by FrustratedMSIuser (2021-05-29 10:12:36)

FrustratedMSIuser · 2021-05-29 10:11:15

Solution

Before the BIOS upgrade I created the UEFI boot entry using the UEFI shell and bcfg.
Ater the BIOS upgrade this entry was gone. So I had to recreate it.
I followed the Wiki guide thoroughly but I however always used a file to store the options for the kernel.

bcfg boot dump -v

showed me that the entries were correct. I even compared the hexdump to the hexdump of another Arch Linux system of mine.

Since none of that worked, I started trying to boot however possible.
One thing that worked was creating a script

\vmlinuz-linux cryptdevice=UUID=aac36f28-d230-41c2-877b-740caf0a46b6:cryptroot root=LABEL=CRYPTROOT rw initrd=\intel-ucode.img initrd=\initramfs-linux.img

and runing it from the UEFI shell by first navigating into the EFI partition and then executing it.
Since this always requires booting into the UEFI shell, it is not a satisfactory solution.

So, I tried to create an UEFI boot entry by using efibootmgr again. This time with the option -e 3. See here.

sudo efibootmgr --create --disk /dev/sdb --part 1 --label "Arch Linux" --unicode "cryptdevice=UUID=aac36f28-d230-41c2-877b-740caf0a46b6:cryptroot root=LABEL=CRYPTROOT rw initrd=\intel-ucode.img initrd=\initramfs-linux.img" --loader "/vmlinuz-linux" --verbose -e 3

This time the entry was persistent and I could boot into Arch without any kernel panic.

Conclusion

I still don't know if it is related to a BIOS bug or if I did something wrong while using bcfg to create a boot entry.
Especially since I have already successfully created entries using bcfg and compared them to my other systems boot entries, I don't think that it is the latter.
Nevertheless, running efibootmgr with option -e 3 made the entry persistent and Arch Linux bootable again.

Last edited by FrustratedMSIuser (2021-05-29 10:37:03)

Arch Linux

#1 2021-05-23 10:14:49