[S] emergency shell, can't mount /efi - how to fix without live stick?

069cho · 2021-08-13 19:40:21

Hello fellow arch users,
I would appreciate your assistance. Normally, I would know how to fix this issue myself by using a live USB stick and chroot, but unfortunately I can't do so right now, since I'm on vacation and don't have a USB drive (nor a hub or adapter, since the device only has one USB c port)

When booting my system I am prompted to enter the password for my Luks root partition (as usual) but afterwards I'm being put into emergency shell since /efi can't be mounted. I know why (and I'll describe the setup in the next paragraph), I just don't know how to fix this on the go.

My device has one ssd with three partitions:
nvme0n1p1 - efi partition, mounted to /efi
nvme0n1p2 - crypted root partition
nvme0n1p3 - swap

During the initial installation of arch on this device, I was a bit confused with the setup of the boot loader (using systemd boot) and the kernel hooks etc. and made some mistake: somehow when generating initramfs etc. It is not placed in /efi but rather in /boot. Eventually I was lazy and 'fixed' this issue by just copying initramfs and vmlinuz(big mistake, I know) to /efi manually.

Now, after forgetting about this issue and carelessly updating the system (and thus the kernel), I get this error message saying that I'm not able to mount /efi for vfat is an unknown filesystem.

Thus, I can't even 'fix' it for now by just pasting the files over again, since I can't access that boot partition at all.

All I can think of right now would be formatting the boot partition while in the emergency shell and setting it up again, but I'm not sure if that would be sensible.
Or downgrading the kernel via pacman-cache would be an option? Can't put the files generated by mkinitcpio in the right spot, though?

Any help would be appreciated. Thanks! (Please excuse bad formatting and lack of logs etc., I'm posting this from my phone's browser). Any further information needed?

Last edited by 069cho (2021-08-16 17:31:05)

loqs · 2021-08-13 19:59:47

Do you have the old version of the kernel package in pacman's cache ( /var/cache/pacman/pkg ) ? If so you can downgrade the kernel package so the modules match the kernel version.

069cho · 2021-08-13 20:05:01

Thanks for your reply. According to uname -r I'm on 5.13.7 while the pacman cache has only 5.13.9 present?! Installing it, however, ends in an error message, as the generated initramfs can't be placed in the proper location as /efi can't be mounted

loqs · 2021-08-13 20:07:29

What was the error message? What boot loader is the system using?

069cho · 2021-08-13 20:22:48

loqs wrote:

What was the error message? What boot loader is the system using?

Boot loader is systemd boot.

First error message says
"failed to start load kernel modules"
(See systemctl status systemd-modules.service for details)

Second one says
"Failed to mount /efi"
(See systemctl status efi.mount for details)

Here's the output of the two status commands, hopefully readable: https://ibb.co/zJrPJqN

I think my problem might be identical to this, btw:
https://bbs.archlinux.org/viewtopic.php?id=178358
Just that I don't have access to a live system for an easy fix right now.

Last edited by 069cho (2021-08-13 20:23:31)

loqs · 2021-08-13 20:43:36

Without live media or the old kernel version it becomes more complicated. What if you try and force loading of the fat module for 5.13.9?

# modprobe -f -S 5.13.9-arch1-1 fat

069cho · 2021-08-13 20:54:54

loqs wrote:

# modprobe -f -S 5.13.9-arch1-1 fat

Results in

modprobe: ERROR: could not insert "fat": invalid argument

(modprobe fat without specifying the kernel results in

modprobe: FATAL: module fat not found in directory /lib/modules/5.13.7-arch1-1

as you might have guessed)

loqs · 2021-08-13 21:00:55

You did pass -S 5.13.9-arch1-1 to modprobe? Does /lib/modules/5.13.9-arch1-1 exist?

Slithery · 2021-08-13 21:04:32

loqs wrote:

You did pass -S 5.13.9-arch1-1 to modprobe? Does /lib/modules/5.13.9-arch1-1 exist?

AFAIK the old module directory gets deleted on a kernel upgrade.

loqs · 2021-08-13 21:08:18

Slithery from post #3 the only version of the linux package in pacman's cache was 5.13.9 so reinstalling that package would still leave those modules installed?

Last edited by loqs (2021-08-13 21:08:46)

069cho · 2021-08-13 21:13:13

loqs wrote:

You did pass -S 5.13.9-arch1-1 to modprobe? Does /lib/modules/5.13.9-arch1-1 exist?

I did so and yes, the directory exists, containing multiple folders like kernel and modules.*

loqs · 2021-08-13 21:14:48

What is the output of

# insmod /lib/modules/5.13.9-arch1-1/kernel/fs/fat/fat.ko.zst

069cho · 2021-08-13 21:30:55

loqs wrote:

# insmod /lib/modules/5.13.9-arch1-1/kernel/fs/fat/fat.ko.zst

fat: version magic '5.13.9-arch1-1 SMP preempt mod_unload ' should be '5.13.7-arch1-1 SMP preempt mod_unload '
insmod: ERROR could not insert module /lib/modules/5.13.9-arch1-1/kernel/fs/fat.ko.zst: invalid module format

loqs · 2021-08-13 21:35:40

Lets see if it can be forced

# insmod -f /lib/modules/5.13.9-arch1-1/kernel/fs/fat/fat.ko.zst

069cho · 2021-08-13 21:42:03

loqs wrote:

# insmod -f /lib/modules/5.13.9-arch1-1/kernel/fs/fat/fat.ko.zst

insmod: ERROR: could not insert module /lib/modules/5.13.9-arch1-1/kernel/fs/fat/fat.ko.zst: invalid parameters

loqs · 2021-08-13 21:51:32

I assume networking is not available and you do not have removable media that uses the same file-system as the installed system's root file-system containing the linux 5.13.7 kernel package.
Then I am out of ideas without the use of live media.

Last edited by loqs (2021-08-13 21:51:52)

069cho · 2021-08-13 22:01:35

I'm assuming I can't use network from the emergency shell, but I'm not 100% sure. Thanks for your help nonetheless. Guess I'll have to try finding a USB C to A adapter and an USB drive (and another PC for preparing it).

Reformating the boot partition and reinstalling and -configuring the boot loader wouldn't be an option, right? Don't want to try it unless I'm 100% sure.

loqs · 2021-08-13 22:09:52

The ESP needs to be FAT. Without the kernel fat module you would not be able to install a boot loader onto the newly formatted ESP.
systemd-boot can only read FAT so can not use the kernel in the root file-system's /boot directory even if it was not encrypted.

069cho · 2021-08-16 17:30:05

Thanks for your help, @loqs. In the end, I managed to find a USB C to USB A adapter and a USB drive and fixed the problem with a live stick. So, solved, I guess? Thread title seems to be too long for the tag anyway.

Last edited by 069cho (2021-08-16 17:31:40)

Wild Penguin · 2021-08-17 08:43:42

Shorten the title to accomodate for the tag.

On this problem for future reference: there is a relatively large window where the system is in a non-bootable state, when running pacman to upgrade the system. Namely, pacman will first install the kernel (this will remove old kernel(s), then the rest of the packages (which can take a long time) and finally, after everything else, install the module(s) of the new kernel (or several kernels) and create the initramfs's. If anything happens in between kernel and module installation which forces the system to reboot, the system can not use the modules from initramfs, nor the (old) modules of the (old) Kernel. This may mean the system can not be properly booted, and the system is non-fixable without some other, working boot.

For this kinds of situations, I usually have several boot loaders/managers in the EFI. This includes grub2, which is the most handy tool, since it can (with a bit of hassle) load many Kernels from live ISO images of Linux distributions straight from the HDD (haven't tried the Arch one, though).

As another approach, it might make sense to mount the EFI partition to /efi, however depending on the boot loader that might not be feasible.

Last edited by Wild Penguin (2021-08-17 08:45:29)

Arch Linux

#1 2021-08-13 19:40:21

[S] emergency shell, can't mount /efi - how to fix without live stick?

#2 2021-08-13 19:59:47

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#3 2021-08-13 20:05:01

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#4 2021-08-13 20:07:29

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#5 2021-08-13 20:22:48

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#6 2021-08-13 20:43:36

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#7 2021-08-13 20:54:54

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#8 2021-08-13 21:00:55

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#9 2021-08-13 21:04:32

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#10 2021-08-13 21:08:18

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#11 2021-08-13 21:13:13

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#12 2021-08-13 21:14:48

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#13 2021-08-13 21:30:55

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#14 2021-08-13 21:35:40

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#15 2021-08-13 21:42:03

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#16 2021-08-13 21:51:32

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#17 2021-08-13 22:01:35

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#18 2021-08-13 22:09:52

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#19 2021-08-16 17:30:05

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

#20 2021-08-17 08:43:42

Re: [S] emergency shell, can't mount /efi - how to fix without live stick?

Board footer