[SOLVED] unable to boot after update - error: device 'UUID' not found

jack_slapped · 2022-06-27 21:53:51

So I recently updated my computer and upon reboot i get the error below. it kinda looks like it cant see my nvme drive which is really weird. shows up fine when i arch-chroot in.

ERROR: device 'UUID=uuid of root parition' not found. Skipping fsck.
mount: /new_root: can't find UUID=uuid of root partition.
you are now being dropped into an emergency shell. 
sh: can't access tty; job control off.

i think the issue might have to do with uname -r output not matching pacman -Q linux. pacman -Q linux returns kernel version 5.18.7 and uname -r returns kernel version 5.18.1. also not sure if it matters but i ran those 2 commands while being arch-chrooted from the installation media. i saw some posts mentioned not having /boot mounted during upgrade, so i tried reinstalling the kernel and grub from arch install media while being arch-chrooted in with proper mount points for /mnt and in /mnt/boot. for some reason tho uname -r always returns the kernel version 5.18.1.

in my research i found a lot of people posting about similar issues when using a nvme drive which i am. im not dual booting but i do have a windows 10 vm setup with gpu passthrough for gaming. im passing through my other nvme drive directly to the vm as well. not sure if that really matters but i also saw some posts of people who do dual boot windows, im not dual booting per say but i thought it might be relevant. other things that i have tried. sata setting in bios is set to AHCI and is NOT on RAID. tried the module setting for vmd in mkinitcpio.conf. im completly stumped. been researching and rebooting for hours now. if anyone could help that would be greatly appreciated.

Last edited by jack_slapped (2023-04-14 03:12:55)

seth · 2022-06-28 06:50:36

uname -r output not matching pacman -Q linux … while being arch-chrooted from the installation media

is insignificant ("normal", uname shows the kernel from the install iso, because that's the one you booted)
From the emergency shell, run "lsblk -f" and "lsmod | grep nvme"
Also remove the "quiet" parameter from the kernel options and pay attention to errors ahead of the root failure, https://wiki.archlinux.org/title/Solid_ … leshooting

If the device shows up in lsblk, you can try passing "rootwait" (waits forever) or eg. "rootdelay=10" (seconds) to the kernel.

jack_slapped · 2022-06-29 15:41:24

thanks so much for taking a look and trying to help me out. for some reason the emergency shell is only letting me run certain commands.

output for lsblk -f is

 sh: lsblk: not found

output for lsmod | grep nvme

nvme                                 61440         0
nvme_core                            176128        1 nvme

i edited grub and removed the quiet parameter, but no additional output. i think whatever issue is occurring is happening extremely early in the boot process perhaps during initramfs.

Last edited by jack_slapped (2022-06-29 15:55:54)

seth · 2022-06-29 19:51:25

the emergency shell is only letting me run certain commands

It's a limited shell, check

ls /dev/nvm*

The nvme modules are certainly loaded at this point, did you try either of the rootdelay/rootwait options to wait for the device?
Also make absolutely sure that the error has the correct UUID and there's not some typo in your grub.cfg and you're trying to boot the wrong device.

loqs · 2022-06-29 19:57:45

If the machine contains two NVME drives of the same make and model looks in dmesg for the error 'globally duplicate IDs' https://bugs.archlinux.org/task/74916

jack_slapped · 2022-06-29 21:39:05

no luck with setting rootdelay. its waiting to mount that uuid but that device isnt showing up under /dev anymore. output from ls /dev/nvm* is no such file or directory which makes sense as none of my devices are showing up when trying to boot from the drive with arch installed. UUID's are correct as well and error matches whats listed in /etc/fstab.

jack_slapped · 2022-06-29 22:51:08

loqs wrote:

If the machine contains two NVME drives of the same make and model looks in dmesg for the error 'globally duplicate IDs' https://bugs.archlinux.org/task/74916

for a second i thought this was going to be the answer, but i ran both of these and got no output. although i think they are the same make/model so i suppose its possible.

dmesg | grep duplicate*
dmesg | grep global*

loqs · 2022-06-29 22:57:06

If you install linux-lts (using the live media and chrooting in) does that boot correctly? If linux-lts boots correctly can you post the full dmesg output from a failed boot?

schard · 2022-06-29 23:03:53

I'm not sure whether I missed the elephant in the room, but did you literally specify UUID=uuid as your root device?
Since, well, "uuid" is not a valid UUID.

jack_slapped · 2022-06-30 01:40:00

schard wrote:

I'm not sure whether I missed the elephant in the room, but did you literally specify UUID=uuid as your root device?
Since, well, "uuid" is not a valid UUID.

no, fair question lol. there is an actual UUID there and its the correct one. i just cant figure out a way to get output this early in the boot process so i shortened it. the prob is when i try and boot from the nvme drive. its like it doesnt see any of the UUIDs it should be seeing or maybe booting from a different device that nothing exists on. at least i think it is, im not sure.

If you install linux-lts (using the live media and chrooting in) does that boot correctly? If linux-lts boots correctly can you post the full dmesg output from a failed boot?

Im going to try this in a minute and let you know how it goes.

jack_slapped · 2022-06-30 02:03:38

loqs wrote:

If you install linux-lts (using the live media and chrooting in) does that boot correctly? If linux-lts boots correctly can you post the full dmesg output from a failed boot?

LEGEND. fixed it. booted up instantly. THANK YOU EVERYONE. how exactly do i post the failed dmesg? also how do i go forward from here? do i have to do anything special to my current install if need to stay on LTS, and should i stay on LTS?

ewaller · 2022-06-30 03:59:34

jack_slapped wrote:

how exactly do i post the failed dmesg?

# dmesg | curl -F 'f:1=<-' ix.io

Post the link it reports when it is finished.

Extra points when you decode how that works

Last edited by ewaller (2022-06-30 03:59:58)

seth · 2022-06-30 06:28:25

I don't think there'll be a network (certainly no wifi) in the rescue shell and absent the root device, the journal will also not be preserved.
You'll then have to usb walk it.
Boot the fallback initramfs (to get access to most kernel modules) for the bad kernel, run "dmesg -w" and plug a usb key. Hope that the usb_storage module kicks in and you get a device node (eg. /dev/sda1)
Try to mount that (mkdir -p /mnt/usb; mount /dev/sda1 /mnt/usb) and "dmesg > /mnt/usb/dmesg.txt", umount and reboot.

If you cannot mount any FS but have a device node (make SURE! that it's the usb key!) and there's nothing relevant on it, you can also write directly onto the device (dmesg > /dev/sda1) and later on directly read from it.
NOTITCE that this will shred the filesystem on the usb key and you lose all data, soe MAKE ABSOLUTELY SURE THAT IT'S AN EMPTY USB KEY AND NOT SOME HDD!

ewaller · 2022-06-30 13:45:03

True. I had been skimming and missed that this was the rescue shell.

jack_slapped · 2022-07-01 00:08:46

run "dmesg -w" and plug a usb key. Hope that the usb_storage module kicks in and you get a device node (eg. /dev/sda1)

weird. when i run this i get invalid option. options that are available are -c, -n LEVEL, -s SIZE, -r

i went ahead and was able to get the usb mounted and got the output from dmesg tho. lets see if i did this right. never posted output through a pastebin client before so let me know if i just need to post the actual text in here. it just seemed like way to much to post. dmesg.txt

loqs · 2022-07-01 00:34:56

Hmm nothing in the dmesg output from the nvme module. Please post the dmesg output from linux-lts for comparison and the output of lspci.

jack_slapped · 2022-07-01 03:13:33

loqs wrote:

Hmm nothing in the dmesg output from the nvme module. Please post the dmesg output from linux-lts for comparison and the output of lspci.

output for linux-lts dmesg
dmesglts.txt

and for lscpi (this is output from linux-lts as well since lspci isnt available when i get kicked to the emergency shell)

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:03.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01)
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
07:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
09:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XTX [Radeon R7 260X/360]
0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tobago HDMI Audio [Radeon R7 360 / R9 360 OEM]
0b:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
0b:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
0d:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller
0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
0e:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0e:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller

loqs · 2022-07-01 04:21:23

[    0.898085] pci 0000:01:00.0: [144d:a808] type 00 class 0x010802
[    0.898100] pci 0000:01:00.0: reg 0x10: [mem 0xfcc00000-0xfcc03fff 64bit]
....
[    0.901493] pci 0000:0c:00.0: [144d:a808] type 00 class 0x010802
[    0.901507] pci 0000:0c:00.0: reg 0x10: [mem 0xfca00000-0xfca03fff 64bit]

The devices show up then nothing.

seth · 2022-07-01 06:42:27

LTS has

[    0.977022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

From the overall context, try passing "iommu=soft" to the kernel.

jack_slapped · 2022-07-01 13:42:07

seth wrote:

LTS has
[    0.977022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
From the overall context, try passing "iommu=soft" to the kernel.

will this still allow for me to run vm's and do gpu passthrough? also do i need to remove this kernel parameter "amd_iommu=on" or can i keep them both? this is one of the parameters required to do gpu passthrough.

Last edited by jack_slapped (2022-07-01 14:30:38)

seth · 2022-07-01 14:31:17

Well, "hopefully" - but for the moment we should first and unconditionally figure whether it's relevant at all.

jack_slapped · 2022-07-01 14:41:55

very true. im at work right now but ill give it a shot when i get home and let you know. side note. should i even bother running the regular linux kernel? i mean bleeding edge updates are nice but the chance of breaking my system hardly seems worth it tbh. stability sounds way more my speed and i cant imagine that the benefits of the most updated kernel are really worth it.

seth · 2022-07-01 15:06:57

You "need" the main kernel if you've newer HW that lacks support in older kernels, however your premise is a fallacy.
At some point a main kernel will become the LTS kernel and if it still has that problem, because it never got addressed - and there's no guarantee that it will be - there's no easy way back (but downgrading)

If you're not short on boot partition space, I'd simply have the main and the LTS kernel installed at all times and if an update welcomes you with an unpleasent surprise, you have a fallback solution to rely on until the problem in the other kernel is resolved.

jack_slapped · 2022-07-01 22:04:21

seth wrote:

LTS has
[    0.977022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
From the overall context, try passing "iommu=soft" to the kernel.

ok so i have some more information. i added "iommu=soft" and it did actually allow the new kernel to boot, buuuut it broke my vm setup saying something about it not being supported. Strange that the vm still works under the linux-lts kernel tho without the "iommu=soft" parameter, and just fyi im using grub to set the kernel parameters. not sure if that matters or not tho. is it possible some of the modules/hooks being loaded in mkinicpio.conf would be affecting the boot process? here are the modules im using and their order. I feel like we might be zoning in on the issue bc breaking the ability to run a vm seemed to have some affect on successfully booting. i did rearrange the order of the hooks based on one post. it had recommended putting block before autodetect. not sure if the order really matters or not on that.

MODULES=(vfio vfio_iommu_type1 vfio_pci vfio_virqfd amdgpu radeon)
HOOKS=(base udev block autodetect modconf filesystems keyboard fsck)

and while im at it here are the grub parameters im using as well.

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 amd_iommu=on hugepages=2048"

i did notice some lines pertaining to the vm in the dmesg output, but i suppose that could just be from the modules being loaded during the initramfs part. i only have a very basic understanding of the early boot process so this is quite possibly is nothing, but thought it worth mentioning.

seth wrote:

...If you're not short on boot partition space, I'd simply have the main and the LTS kernel installed at all times and if an update welcomes you with an unpleasent surprise, you have a fallback solution to rely on until the problem in the other kernel is resolved.

this makes so much sense. i didnt realize you could have 2 kernels installed at the same time. i will def be doing this from now on. should i go ahead and post dmesg output for when it boots successfully under the latest kernel with iommu=soft parameter?

Last edited by jack_slapped (2022-07-01 22:09:38)

loqs · 2022-07-01 22:10:05

What about iommu=pt which allows IOMMU for VMs but not the host?

Arch Linux

#1 2022-06-27 21:53:51

[SOLVED] unable to boot after update - error: device 'UUID' not found

#2 2022-06-28 06:50:36

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#3 2022-06-29 15:41:24

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#4 2022-06-29 19:51:25

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#5 2022-06-29 19:57:45

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#6 2022-06-29 21:39:05

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#7 2022-06-29 22:51:08

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#8 2022-06-29 22:57:06

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#9 2022-06-29 23:03:53

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#10 2022-06-30 01:40:00

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#11 2022-06-30 02:03:38

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#12 2022-06-30 03:59:34

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#13 2022-06-30 06:28:25

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#14 2022-06-30 13:45:03

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#15 2022-07-01 00:08:46

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#16 2022-07-01 00:34:56

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#17 2022-07-01 03:13:33

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#18 2022-07-01 04:21:23

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#19 2022-07-01 06:42:27

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#20 2022-07-01 13:42:07

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#21 2022-07-01 14:31:17

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#22 2022-07-01 14:41:55

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#23 2022-07-01 15:06:57

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#24 2022-07-01 22:04:21

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

#25 2022-07-01 22:10:05

Re: [SOLVED] unable to boot after update - error: device 'UUID' not found

Board footer