[SOLVED] Arch not booting...

Der Chefkoch · 2023-03-09 17:42:11

As much as I love Arch, now every month after some update its unbootable.

Ran a full upgrade today, now running in an issue I have not encountered so far:

After it shows the message "Starting systemd-udevd version 253.1-3-arch" it says it wouldnt find the device with UUID... and then I am dropped into an emergency shell that I cannot use as it is not recognizing the keyboard.

Thx

Last edited by Der Chefkoch (2023-03-09 18:04:10)

Der Chefkoch · 2023-03-09 18:03:51

... but at least the fixing gets better:
1. use livecd and uncrypt (cryptlvm) the drive
2. properly mount the boot partition
3. regenerate initramfs and grub (when used)
4. restart

Stunts · 2023-03-10 18:50:42

I should report that this has happened to me too.
Last week this happened once, and I thought it could have been a fluke. I used the same method as the OP to restore functionality.
But today this has happened again. I "fixed" the boot problem again in the same fashion, but something is clearly not right.
I frequently (as in daily) use 3 machines with archlinux installs, and this has only occurred on my laptop (Thinkpad x395). All machines have a similar partition schemes, all boot from (similar) nvme drives, and all are frequently updated.
Has anyone else had similar issues and found the root cause? I'm not even sure how I can further debug the problem since when the issue occurs, like the OP, the keyboard does not respond (I guess I can try using a USB keyboard, but even then I'm not sure what I should be looking for).

ewaller · 2023-03-10 20:09:46

Sounds like your boot partition is not mounted when you perform the upgrades. Post the output of lsblk.

Edit: Both of you

Last edited by ewaller (2023-03-10 20:10:15)

Stunts · 2023-03-10 23:04:47

@ewaller, thank you for the follow-up.
Below is the result of the requested command on the problematic machine:

╭──francisco@SideSwipe [22:58] [~]
╰─$ lsblk                                                                     0
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 476,9G  0 disk
├─nvme0n1p1 259:1    0   512M  0 part /ESP
├─nvme0n1p2 259:2    0   512M  0 part /boot
├─nvme0n1p3 259:3    0    35G  0 part /
├─nvme0n1p4 259:4    0   425G  0 part /home
└─nvme0n1p5 259:5    0  15,9G  0 part [SWAP]

As far as I can tell, it looks OK.
Also, my /etc/fstab contains:

UUID=7bdb1369-d1a8-466d-86fb-f495c346c6d8	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p2
UUID=6495e10b-1f5c-4e5d-ac82-ad037bcf3a04	/boot     	ext4      	rw,relatime	0 2

# /dev/nvme0n1p1
UUID=98A8-8AB0      	/ESP      	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro
    0 2

# /dev/nvme0n1p4
UUID=62b15981-c1de-404d-afe3-2147f1346bfa	/home     	ext4      	rw,relatime	0 2

# /dev/nvme0n1p5
UUID=d2fc6b30-80ed-48ab-af29-86ac488d3f70	none      	swap      	defaults  	0 0

I guess that looks OK too?

ewaller · 2023-03-11 01:44:48

I see you have a /boot mount that has nvme0n1p2 mounted on it, but you also have an an /ESP mount point that has nvme0n1p1 mounted on it.
I will bet one adult beverage that you are booting from nvme0n1p1, but your kernel and initramfs are being updated on nvme0n1p2 (As it is mounted on /boot)

Where did the ESP mount point come from? It appears that ESP is used interchangeably with EFI meaning the partition the system will boot from on uEFI firmware.

I'd be curious about the contents of the two mount points -- What are the output of ls -l /EFI /boot ?

I want to verify this, but you probably want to get rid of the /ESP mount point, and remove nvme0n1p2 from your /etc/fstab.

Edit:

# /dev/nvme0n1p2
UUID=6495e10b-1f5c-4e5d-ac82-ad037bcf3a04	/boot     	ext4      	rw,relatime	0 2

And ^^ that is a problem. /boot should be a vfat partition it is going to be an EFI system partition. Note that the volume on /ESP is.

Again, I think you may have added the ESP mountpoint by hand and have mounted the actual EFI partition on it. You have a different ext4 partition mounted on /boot that is receiving the kernel updates, but is not being used to start the system. The unloved partition mounted on /ESP (from which you boot) still has the old kernel installed until you fix it by hand.

Last edited by ewaller (2023-03-11 01:50:11)

Stunts · 2023-03-11 16:59:01

Well, I think I owe you one adult beverage then. You are spot on. If you are ever in the Lisbon area, just PM me, and I will make good in this promise!
Here is what, up until now, I thought my boot process looked like:
1. System firmware finds the EFI binary to start grub on /dev/nvme0n1p1 (vFAT).
2. Grub runs, and boots using the initramfs and linux kernel, which are located on /dev/nvme0n1p2 (ext4).
3. Once initramfs and the kernel are loaded, the root partition (/dev/nvme0n1p3) is mounted on "/" (ext4).
4. The OS gets loaded and the remaining two partitions get mounted under /home and as swap respectively. Simultaneously, /dev/nvme0n1p1 gets mounted under /ESP and /dev/nvme0n1p2 gest mounted under /boot.

I thought this was "The Right Way" (TM) to handle the boot process.

Below is the result of the requested command (actually split in two for the sake of completeness):

╭──francisco@SideSwipe [16:22] [~]
╰─$ ls -l /ESP/*/*                                                            0
/ESP/EFI/arch:
total 68
drwxr-xr-x 2 root root  4096 dez  9  2021 fw
-rwxr-xr-x 1 root root 62205 jun 19  2021 fwupdx64.efi

'/ESP/EFI/ArchLinux GRUB':
total 132
-rwxr-xr-x 1 root root 135168 ago  6  2019 grubx64.efi

/ESP/EFI/GRUB:
total 148
-rwxr-xr-x 1 root root 151552 set  5  2022 grubx64.efi

And here it is for /boot:

╭──francisco@SideSwipe [16:24] [~]
╰─$ ls -l /boot/                                                                                                                                                    0
total 115944
-rw-r--r-- 1 root root    51200 fev 16 15:55 amd-ucode.img
drwxr-xr-x 6 root root     4096 set  5  2022 grub
-rw------- 1 root root 73041804 mar 10 13:59 initramfs-linux-fallback.img
-rw------- 1 root root 33096196 mar 10 13:58 initramfs-linux.img
drwx------ 2 root root    16384 ago  6  2019 lost+found
-rw-r--r-- 1 root root 12506208 mar 10 13:58 vmlinuz-linux
╭──francisco@SideSwipe [16:24] [~]
╰─$ ls -l /boot/grub                                                                                                                                                0
total 44
drwxr-xr-x 2 root root  4096 ago  6  2019 fonts
-rw------- 1 root root  4659 set  5  2022 grub.cfg
-rw-r--r-- 1 root root  1024 ago  6  2019 grubenv
drwxr-xr-x 2 root root  4096 set  5  2022 locale
drwxr-xr-x 3 root root  4096 ago  6  2019 themes
drwxr-xr-x 2 root root 20480 set  5  2022 x86_64-efi

Now, both these commands did yield some surprises for me. Namely, the 2019 files (grubenv and grub themes directory), which were created on the date I got this laptop. Then I also seem to have an older grub EFI binary leftover under /ESP, from the same day which was unexpected (I can't remember, but I must have messed up my initial install attempt somehow). I guess I should get rid of these files. Other than that, this was exactly what I expected to find.

I went back to the Archwiki partitioning entry, and my! Have best practices changed since 2019 (or maybe I had just misinterpreted the wiki)!
So, according to the wiki and your recommendation, I should be mounting /dev/nvme0n1p1 under /boot, have it used for both housing the grub EFI binary, kernel & initramfs and all the GRUB files, and simply getting rid of /dev/nvme0n1p2? (maybe just delete it and merge the resulting free space with /dev/nvme0n1p1 as vFAT?) I was not aware that GRUB could be installed in a vFAT partition.
This should be as "simple" as booting a live environment, modifying the partitions, chrooting into the system with the new partition scheme, change /etc/fstab and re-install the bootloader, correct?

If this is so, then I guess the right question is how come I only experienced the issue that brought me here after almost 4 years of daily use? And as such, I suppose I should aim for the same change on my other two machines with a similar boot mechanism.

Arch Linux

#1 2023-03-09 17:42:11

[SOLVED] Arch not booting...

#2 2023-03-09 18:03:51

Re: [SOLVED] Arch not booting...

#3 2023-03-10 18:50:42

Re: [SOLVED] Arch not booting...

#4 2023-03-10 20:09:46

Re: [SOLVED] Arch not booting...

#5 2023-03-10 23:04:47

Re: [SOLVED] Arch not booting...

#6 2023-03-11 01:44:48

Re: [SOLVED] Arch not booting...

#7 2023-03-11 16:59:01

Re: [SOLVED] Arch not booting...

Board footer