[SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

Cbhihe · 2023-08-20 15:42:57

Hello,
At first reboot and after the usual GRUB2 menu, I get to choose between regular boot or advanced boot options.

Choosing the latter I get:

/root:clean, 462327/2052000 files, 6108880/8192000 blocks
   systemd:caught <BUS> from PID -803962496
   systemd:caught <BUS>, core dump failed (child 207, code=killed, statu=7/BUS).
   systemd:freezing execution.

Choosing the former I get:

loading Linux linux ....
loading initial ramdisk....
....
ERROR: resume: hibernation device '/dev/disk/by-uuid/c1e1198....' not found
ERROR: device 'UUID=0f2397...' not found.
mount: /new_root: can't find UUID=0f2397...
ERROR: Failed to mount 'UUID=0f2397...' on real root
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off
[rootfs ~]#

Looking at recent forum posts (e.g. here), i saw that this may be related to a systemd 253.7 buggy update. This is pretty much out of my league.
How to recover from this ?

Last edited by Cbhihe (2023-08-21 14:18:33)

seth · 2023-08-20 18:44:09

https://bbs.archlinux.org/viewtopic.php?id=288095 ?

Cbhihe · 2023-08-21 10:02:57

Thank you, @seth; i saw your pointer yesterday and have tried to boot from live usb ever since.

Following the link you provided, I want to be able to chroot into the host, effect the change ( in /etc/modprobe.d/blacklist_rtsx.conf) and rebuild the initramfs so that a normal reboot becomes possible again.

It seems the problem I have now (and for the 1st time ever) is that the host just does not reboot from either one of the two functional Archlinux usb iso I have. Both are UEFI iso I have used before arch linux 2022-10 and 2022-12 .Will keep trying although i am a bit at a loss as to why the specified boot sequence options, in which "USB iso" comes first, is ignored.

EDIT: just managed booting from USB iso now. Don't know why now and not before...

Last edited by Cbhihe (2023-08-21 10:17:56)

seth · 2023-08-21 12:57:27

Fwwi, you don't have to blacklist the module from some chroot, you can just do so at the kernel commandline in the bootloader.

module_blacklist=rtsx_pci,rtsx_pci_sdmmc

would do if you're hitting that bug.
(The usb ky might have struggled if it's also booting 6.4.11?)

Cbhihe · 2023-08-21 14:17:40

ok. So I could probably have achieved the same result from the GRUB2 command line... but I saw your reply above too late. It would havebeen somewhat less involved than rebuilding initramfs, but then chroot'ing is usually no big hassle, except in this very case (cause unknown <- faulty usb iso ?)

Just for future reference and others who may stumble on the same bug, what I did was to reboot from Archlinux ISO x86_64 (versión 2022-10, so it's was probably some kernel version in the low 6.y)

# mount /dev/nvme0nXpY /mnt                    # where "nXpY" is that host's root volume on its own partition
# touch  /mnt/etc/modprobe.d/blacklist_modules.conf
# printf "%s\n" "blacklist rtsx_pci" "blacklist rtsx_pci_sdmmc"  >> /mnt/etc/modprobe.d/blacklist_modules.conf
# mount /dev/nvme0nXpY /mnt/boot           # where "nXpY" is that host's /boot volume on its own partition
# mount /dev/nvme0nXpY /mnt/boot/efi      # where "nXpY" is that host's EFI (FAT32) volume on its own partition
# mount /dev/nvme0nXpY /mnt/var             # where "nXpY" is that host's /var volume on its own partition
# arch-chroot /mnt mkinitcpio -P
    [... output ...]
# exit
# umount --recursive /mnt
# reboot now

SOLVED.

PS: @admins: I don't know what the policy is for publishing announcements on problematic updates, but perhaps this could have justified a note to all info list subscribers from arch-announce-request@archlinux.org. Just saying. :-) Thanks.

Last edited by Cbhihe (2023-08-22 07:46:32)

vityafx · 2023-09-05 06:08:20

How do you guys were able to do all of this? I just updated yesterday and have the same thing, except for the fact that the whole USB subsystem doesn’t work for me at all. The devices seem to be powered, but don’t work. Another really weird thing is that my keyboard doesn’t work even in BIOS!!! I am so surprised some Linux update could break something at BIOS time. I can’t believe it, how could this even happen? There is no way a Linux update can break things at BIOS time, except, perhaps, a CPU microcode update which broke something there somehow, I guess. So, that said, I literally can’t do anything now - there is no way to fix anything, as my input devices don’t work and I can’t even select a different boot drive to boot from anything else, nor I can’t do anything at the GRUB second stage and in this emergency shell. Is it so broke just for me or was for you too?
I have an AMD 5950x and ASUS Dark Hero motherboard.

UPD: so i was able to get my keyboard working but blacklisting the rtsx modules in the kernel cmd line didn’t help.

Last edited by vityafx (2023-09-05 06:22:57)

seth · 2023-09-05 06:48:28

I can’t believe it, how could this even happen?

i was able to get my keyboard working

So apparently it just didn't happen?

Did you add the parameters from #4?
Try to also add "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" ( https://bugs.archlinux.org/task/79439#comment221327 ) and then post the ouputs of

uname -a; cat /proc/cmdline

from the emergency shell (link a photo of your monitor in doubt)

vityafx · 2023-09-05 07:46:15

After I enabled my keyboard back, I tried the kernel cmdline arguments, but it didn’t work. Then I booted from a liveusb, blacklisted the modules, updated the initramfs image, rebooted and only then it worked. Thank you for your response anyway! And thanks everyone for the help here! Solved on my side!

P.S. the keyboard issue is probably due to BIOS enabling the devices sequentially and it just wasn’t getting too fast to enable my keyboard before it went to booting the kernel. I unplugged a few USB devices and then the keyboard was working again.

Last edited by vityafx (2023-09-05 07:47:31)

seth · 2023-09-05 08:03:05

The kernel parameters in #4 /do/ blacklist the relevant modules.
Typo?

Arch Linux

#1 2023-08-20 15:42:57

[SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#2 2023-08-20 18:44:09

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#3 2023-08-21 10:02:57

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#4 2023-08-21 12:57:27

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#5 2023-08-21 14:17:40

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#6 2023-09-05 06:08:20

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#7 2023-09-05 06:48:28

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#8 2023-09-05 07:46:15

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

#9 2023-09-05 08:03:05

Re: [SOLVED] dropped to emergency rootfs after 6.4.10->6.4.11 update

Board footer