You are not logged in.

Pages: 1

#1 2022-09-23 23:42:23

b0ss_
Member
Registered: 2021-01-16
Posts: 83

[ISSUE] Arch throws me into rootfs after GRUB starts

Hello guys. I don't know if I should post this in Newbie Corner, but I didn't find anywhere else to put this in.
My issue is the following:
After the last update on my homemade-server (an old AIO Compaq) Arch didn't boot. It didn't throw me into GRUB rescue or anything, it took me to the BIOS screen. I tried booting from my disk and it led to the same results as booting normally. I tried using some CDs I have lying around to boot into live mode and check if it was just a GRUB error. I tried re-installing and making a new config, but then it started taking me to the rootfs emergency terminal.

So, I'll try to expand what I've done with the PC, the specs, filesystem and else.

This PC is an old Compaq with a super old AMD CPU and AMD aGPU. It has about 2 gigs of ram, runs on Btrfs with about 4 subvolumes + snapper configured to take snaps after every update (yes, I'll come back to this later on). The error started happening right after I updated my system and installed UFW, unmasked it and restarted because I'd started having kernel issues after not restarting for like 6 months.

So the first thing I did was trying to disable fast boot to try and pick something. It kinda worked, it showed me the GRUB welcoming page for less than a second, but I was able to see it; after that it sent me to the BIOS options.
Second I thought "hey, I have snaps on my PC, I should go live mode and try to restore them!" and I did. I picked the latest ISO I had on a DVD (2021_9), booted into live mode, and started toying around.

Now for context: my disk is partitioned like this:

/dev/sda1 /boot
/dev/sda2 [SWAP]
/dev/sda3 /
/dev/sda3 /home
/dev/sda3 /.snapshots
/dev/sda3 /var_log

At first I thought I was using an old install I had on this pc that used LUKS encryption, so I tried using cryptsetup tools to decrypt it. It didn't lead to anywhere since none of those partitions are LUKS encrypted. After that I started to simply mount the partitions. I first mounted /dev/sda3 to /mnt, arch-chrooted into /mnt/@/ and there I ran mount -a to mount all my subvolumes and other partitions. I ran "btrfs subvolume delete /.snapshots/9306/snapshot" to get rid of the last snapshot my system took after the update to go back a couple days ago when my system was working. I also reinstalled GRUB using the default installation method found in the Wiki (grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB). After that I ran grub-mkconfig -o /boot/grub/grub.cfg but it didn't work. It gave me a weird error I had never seen before. I followed this guide to fix it, and it worked. It created a new grub.cfg, so I got out of live mode and rebooted my system. I was then dropped into the GRUB menu and I picked the first option, but right after that it gave me this message:

"error: no such device: root"

And right into rootfs. I will copy and paste exactly what it shows me in the initial prompt:

Starting version 251.4-1-arch
mount: /new_root: unknown filesystem type 'vfat'.
dmesg(1) may have more information after failed mount system call.
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off.

And from now on I can't do much really. I've tried "mount -t btrfs -o subvol=.snapshots,var_log,home /dev/disk/by-uuid/uuid OR /dev/sda3". I've also tried just mounting /dev/sda3 and chrooting into it and mounting everything with mount -a and quitting, but it usually gives me a kernel panic. Sometimes it says that it can't find any partition or volume whatsoever. 

I don't know what else to do right now. I'll be answering your replies ASAP. Thanks in advance.

Offline

#2 2022-09-24 11:37:29

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 25,263

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

You get this issue if you aren't booting the kernel that's actually installed. From a chroot and things mounted like you think they are correct what do you get for 

pacman -Q linux
file /boot/vmlinuz-linux
lsinitcpio /boot/initramfs-linux.img | grep module
cat /boot/grub/grub.cfg
lsblk -f

Offline

#3 2022-09-24 16:45:13

b0ss_
Member
Registered: 2021-01-16
Posts: 83

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

V1del wrote:

You get this issue if you aren't booting the kernel that's actually installed. From a chroot and things mounted like you think they are correct what do you get for 

pacman -Q linux
file /boot/vmlinuz-linux
lsinitcpio /boot/initramfs-linux.img | grep module
cat /boot/grub/grub.cfg
lsblk -f

Hi V1del, thanks for your answer. I haven't seen you in quite a while. I hope you've been doing fine!

# mount /dev/sda3 new_root/
# mount /dev/sda3 new_root/@/home
# mount /dev/sda3 new_root/@/var_log
# mount /dev/sda3 new_root/@/.snapshots
# mount /dev/sda1 new_root/@/boot
mount: /new_root/@/boot: unknown filesystem type 'vfat'.
dmesg(1) may have more information after failed mount system call.
# chroot new_root/@/
# pacman -Q linux
linux 5.19.10.arch1-1
# file /boot/vmlinuz-linux
/boot/vmlinuz-linux: cannot open '/boot/vmlinuz-linux' (No such file or directory) (Makes sense, since I couldn't mount the boot partition.)
# lsinitcpio /boot/initramfs-linux.img | grep module
==> ERROR: No such file: /boot/initramfs-linux.img
# cat /boot/grub/grub.cfg
cat: /boot/grub/grub.cfg: No such file or directory.
# lsblk -f
lsblk: failed to access sysfs directory: /sys/dev/block: No such file or directory.
# mount -t sysfs sysfs /sys
# lsblk -f
NAME     FSTYPE   FSVER    LABEL   UUID   FSAVAIL   FSUSE%   MOUNTPOINTS
sda
|-sda1
|-sda2
\-sda3
sr0

It gave no other info, not even the mountpoints. If I use lsblk it gives me the usual info: names, sizes, RM, RO, types, but nothing else.
When I try to mount, say, /dev/sda1 into boot/, (while in chroot) it says.

mount: /boot: mount failed: Operation not permitted.

I'm thinking about live booting again... but I'll just wait, since last time I started toying around with one of my systems I completely anhilated the LUKS encryption it had and lost all my data smile so I'll be patient! Thanks for your help.

Offline

#4 2022-09-24 22:10:27

Head_on_a_Stick
Member
From: The Wirral
Registered: 2014-02-20
Posts: 9,003
Website

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

It looks like those commands were run from your installed system. V1del asked you to boot from a live ISO image.

Jin, Jîyan, Azadî

Offline

#5 2022-09-26 19:50:02

b0ss_
Member
Registered: 2021-01-16
Posts: 83

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

Head_on_a_Stick wrote:

It looks like those commands were run from your installed system. V1del asked you to boot from a live ISO image.

Sorry for the late response. I thought that by "chroot" V1del meant to get into it from rootfs and chroot into the installed system. My fault! Now I'll paste the outputs.

# pacman -Q linux
linux 5.19.10.arch-1
# file /boot/vmlinuz-linux
/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.19.10-arch-1 (linux@archlinux) #1 SMP PREEMPT_DYNAMIC Tue, 20 Sep 2022 15:17:59 +0000, RO-rootFS, swap_dev 0XA, Normal VGA
# lsinitcpio /boot/initramfs-linux.img | grep module
usr/lib/modules/ 
usr/lib/modules/5.19.10-arch-1-1/
usr/lib/modules/5.19.10-arch-1-1/kernel/blake2b_generic.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/btrfs.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/cdrom.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/cqhci.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/crc32c_generic.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/crc32c-intel.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/firewire-core.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/firewire-spb2.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/libcrc32e.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/mmc_block.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/raid6_pq.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/sdhci-pci.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/sha256-ssse3.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/sr_mod.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/usbhid.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/usb-storage.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/virtio_blk.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/xhci-pci.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/xhci-pci-renesas.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/xor.ko
usr/lib/modules/5.19.10-arch-1-1/kernel/xxhash_generic.ko
usr/lib/modules/5.19.10-arch-1-1/modules.alias.bin
usr/lib/modules/5.19.10-arch-1-1/modules.builtin.alias.bin
usr/lib/modules/5.19.10-arch-1-1/modules.builtin.bin
usr/lib/modules/5.19.10-arch-1-1/modules.dep.bin
usr/lib/modules/5.19.10-arch-1-1/modules.devname
usr/lib/modules/5.19.10-arch-1-1/modules.softdep
usr/lib/modules/5.19.10-arch-1-1/modules.symbols.bin
# cat /boot/grub/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod ext2
search --no-floppy --fs-uuid --set=root 
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod ext2
	search --no-floppy --fs-uuid --set=root 
	echo	'Loading Linux linux-lts ...'
	linux	/vmlinuz-linux-lts root=/dev/sda1 rw  loglevel=3 quiet
	echo	'Loading initial ramdisk ...'
	initrd	/amd-ucode.img /initramfs-linux-lts.img
}
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-' {
	menuentry 'Arch Linux, with Linux linux-lts' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-advanced-' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 
		echo	'Loading Linux linux-lts ...'
		linux	/vmlinuz-linux-lts root=/dev/sda1 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/amd-ucode.img /initramfs-linux-lts.img
	}
	menuentry 'Arch Linux, with Linux linux-lts (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-fallback-' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 
		echo	'Loading Linux linux-lts ...'
		linux	/vmlinuz-linux-lts root=/dev/sda1 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/amd-ucode.img /initramfs-linux-lts-fallback.img
	}
	menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 
		echo	'Loading Linux linux ...'
		linux	/vmlinuz-linux root=/dev/sda1 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/amd-ucode.img /initramfs-linux.img
	}
	menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 
		echo	'Loading Linux linux ...'
		linux	/vmlinuz-linux root=/dev/sda1 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/amd-ucode.img /initramfs-linux-fallback.img
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
fwsetup --is-supported
if [ "$grub_platform" = "efi" -a "$?" = 0 ]; then
	menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
		fwsetup
	}
fi
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/41_snapshots-btrfs ###
### END /etc/grub.d/41_snapshots-btrfs ###
# lsblk -f
NAME   FSTYPE      FSVER            LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  squashfs    4.0                                                                              
sda                                                                                                 
├─sda1 vfat        FAT32                        1EF8-1143                             381.8M    23% /boot
├─sda2 swap        1                            c16fc8a4-f2cb-4fe7-879b-bea324072561                
└─sda3 btrfs                                    0ad994d8-4404-4033-a7f2-06d59cc3c684  280.4G    70% /home
                                                                                                    /var_log
                                                                                                    /.snapshots
sdb                                                                                                 
├─sdb1 btrfs                                    06f86eb1-ee84-4210-a46b-8489edd3ef46                
├─sdb2 ntfs                         Nuevo vol   E8967B1C967AEB08                      121.7G    76% /mnt
└─sdb3 crypto_LUKS 2                            f6712bb9-af48-41a6-8570-f0a485c9e885                
sr0    iso9660     Joliet Extension ARCH_202109 2021-09-01-11-25-35-00

There we go! Now I have more time, so I think I'll be able to answer quicker than before.

Offline

#6 2022-09-26 20:12:59

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 25,263

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

Kernel files are present as expected, your grub.cfg uses non-persistent device names to identify the root partition which isn't good, which should also not be the default.  is 

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

uncommented in your /etc/default/grub for some reason? In any case you need to fix your GRUB to lookup the rootfs via persistent names https://wiki.archlinux.org/title/Persis … ice_naming whether you get grub-mkconfig to behave or manually adjust this in the file.

Offline

#7 2022-09-26 21:22:59

b0ss_
Member
Registered: 2021-01-16
Posts: 83

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

V1del wrote:

Kernel files are present as expected, your grub.cfg uses non-persistent device names to identify the root partition which isn't good, which should also not be the default.  is 

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

uncommented in your /etc/default/grub for some reason? In any case you need to fix your GRUB to lookup the rootfs via persistent names https://wiki.archlinux.org/title/Persis … ice_naming whether you get grub-mkconfig to behave or manually adjust this in the file.

Hey, I just checked if it was uncommented and no, it isn't. I will try and to what the wiki says, thank you. Although I already have persistant naming in my fstab:

# /dev/sda3
UUID=0ad994d8-4404-4033-a7f2-06d59cc3c684       /               btrfs           rw,noatime,compress=lzo,space_cache=v2,subvolid=257,subvol=/@   0 0

# /dev/sda1
UUID=1EF8-1143          /boot           vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro   0 2

# /dev/sda3
UUID=0ad994d8-4404-4033-a7f2-06d59cc3c684       /home           btrfs           rw,noatime,compress=lzo,space_cache=v2,subvolid=258,subvol=/@home       0 0

# /dev/sda3
UUID=0ad994d8-4404-4033-a7f2-06d59cc3c684       /.snapshots     btrfs           rw,noatime,compress=lzo,space_cache=v2,subvolid=259,subvol=/@snapshots  0 0

# /dev/sda3
UUID=0ad994d8-4404-4033-a7f2-06d59cc3c684       /var_log        btrfs           rw,noatime,compress=lzo,space_cache=v2,subvolid=260,subvol=/@var_log    0 0

# /dev/sda2
UUID=c16fc8a4-f2cb-4fe7-879b-bea324072561       none            swap            defaults        0 0

I ran cat /etc/fstab | grep 1EF8-1143 and it matched with my boot partition (/dev/sda1). Either way, I will configure the kernel parameters to make it persistent using UUID.

menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod ext2
        search --no-floppy --fs-uuid --set=root
        echo    'Loading Linux linux ...'
        linux   /vmlinuz-linux root=UUID=1EF8-1143 rw  loglevel=3 quiet
        echo    'Loading initial ramdisk ...'
        initrd  /amd-ucode.img /initramfs-linux.img
}

So I tried booting into the installed system and it didn't work. At first it didn't work because I forgot to add UUID= after root=, so I booted to the CD again and fixed that. But after trying to boot again from the main install, it showed me the same message as before. 

Starting version 251.4-1-arch
mount: /new_root: unknown filesystem type 'vfat'.
dmesg(1) may have more information after failed mount system call.
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off.

So using the UUID for the boot partition didn't work. So I got back to the wiki and found out that the root partition is not the boot partition, so I used the UUID of /dev/sda3 (0ad994d8-4404-4033-a7f2-06d59cc3c684) in the root= parameter. Didn't work either, although it was able to mount the root partition.

May it had to do with the fact that I'm using BTRFS and the root partition is technically a subvolume? Who knows. I'll be answering ASAP.

UPDATE: Tried BTRFS subvol UUID. Didn't work.

Last edited by b0ss_ (2022-09-26 21:42:18)

Offline

#8 2022-09-27 05:05:36

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,178

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

b0ss_ wrote:

May it had to do with the fact that I'm using BTRFS and the root partition is technically a subvolume? Who knows. I'll be answering ASAP.

UPDATE: Tried BTRFS subvol UUID. Didn't work.

https://wiki.archlinux.org/title/Btrfs# … me_as_root

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#9 2022-09-27 07:27:26

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 76,484

Re: [ISSUE] Arch throws me into rootfs after GRUB starts

mount: /new_root: unknown filesystem type 'vfat'.

You're not booting the installed kernel, typically because you forgot to mount the /boot partition during the update or you're (likely) in fact not booting from the /boot partition

umount /boot
ls -lR /boot

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

Pages: 1

Board footer

Powered by FluxBB