[Solved] Failed to mount /boot

RogerRabbit · 2023-05-23 22:53:01

Hello,

Since my latest upgrade to linux 6.3.3 from 6.3.1, I can't boot to Arch, I enter emergency mode with this "Failed to mount /boot" error (unknown filesystem type 'vfat').

Following the other threads regarding similar issues, I booted on the Archlinux install media, mounted my system and chrroted to it using these commands :

mount /dev/nvme0n1p3 /mnt
mount /dev/nvme0n1p1 /mnt/boot
chroot /mnt bash

From there I understand that I should reinstall the kernel, I have the pkg available in /var/cache/pacman, but pacman seems to be broken.
It gives no output whatever I try, either :

pacman -U linux-6.3.3.arch1-1x86_x64.pkg.tar.zst
pacman -Q --help
pacman nonsense

None of these give any output...

Following is some information...

From chroot :

# file /boot/vmlinuz-linux
/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 6.3.3-arch1-1 (linux@archlinux) #1 SMP PREEMPT_DYNAMIC Sun 21 ?ay 2023 16:15:22 +0000, RO-rootFS, swap_dev 0XB, Normal VGA

From emergency mode :

# uname -rv
6.3.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun 21 ?ay 2023 16:15:22 +0000

Kernel versions from boot to system seems aligned, maybe my issue is different...

Last edited by RogerRabbit (2023-05-28 22:33:15)

Scimmia · 2023-05-23 23:00:12

what happens when you `modprobe vfat` in the emergency shell? What's in /lib/modules/? Is /lib still a symlink to usr/lib?

Last edited by Scimmia (2023-05-23 23:00:43)

RogerRabbit · 2023-05-23 23:20:08

# modprobe vfat
modprobe: FATAL: Module vfat not found in directory /lib/modules/6.3.3-arch1-1

Yes, /lib is a symlink to usr/lib.

There is only one folder in /lib/modules/, the folder is "6.3.3-arch1-1"
It contains itself a kernel folder and the following files

modules.alias
modules.alias.bin
modules.builtin
modules.builtin.alias.bin
modules.builtin.bin
modules.builtin.modinfo
modules.dep
modules.dep.bin
modules.devname
modules.order
modules.softdep
modules.symbols
modules.symbols.bin
pkgbase
vmlinuz

Scimmia · 2023-05-23 23:43:43

Just those files, no kernel dir? The module should be in kernel/fs/fat/

RogerRabbit · 2023-05-23 23:54:53

Yes the kernel folder is there, I said it before listing the files.

Sorry, not familiar yet with kernel librairies, the vfat module seems to be present, there is a "vfat.ko.zst" file in the kernel/fs/fat folder, along with fat.ko.zst and msdos.ko.zst.

Scimmia · 2023-05-23 23:56:30

Then the problem would seem to be with those files, which is what tells modprobe where to find things. Try running depmod and then modprobe vfat again.

RogerRabbit · 2023-05-24 00:02:04

depmod gives the same error for every single module

depmod: ERROR: failed to load symbols from /lib/modules/6.3.3-arch1-1/kernel/xxx/xxx.ko.zst: Invalid argument

Scimmia · 2023-05-24 00:06:33

at least we know what lead to this problem, then. Now the question is why depmod has a problem. Invalid argument is suspicious, what does `type -a depmod` show?

I've got to head out for a while, hopefully someone will pick this up. Otherwise I'll continue then.

RogerRabbit · 2023-05-24 00:09:22

# type -a depmod
depmod is /usr/bin/depmod

RogerRabbit · 2023-05-24 00:42:34

I'm reading some documentation on how modules an kernel are built together to try to understand what happens to me.

Does it mean that I have a kernel with no module at all in the emergency mode ?
That would explain the pacman behavior...

But this is gonna make the fix harder... From what I've seen on similar issues, the kernel reinstall seems to be a valid solution, but I have no idea how to do it without packman or dpkg...

Well, it's 2.40 am right now, I need some sleep, I'll follow this tomorrow evening, thanks for your help.
Cheers.

Scimmia · 2023-05-24 02:01:57

dpkg? Please tell me you haven't been running dpkg on Arch.

seth · 2023-05-24 06:34:38

but pacman seems to be broken.

None of these give any output...

https://wiki.archlinux.org/title/Pacman … an_upgrade
However, along the broken on-disk modules you might want to talk about the incident during the last update … or check https://wiki.archlinux.org/title/SMART

RogerRabbit · 2023-05-24 19:44:11

No I did not use dpkg, was just talking about convenience to install a package without a package manager.

About the issue, it happened yesterday, I closed everything on my laptop (I mostly use gnome terminal, chromium, textedit and that's it), typed "sudo pacman -Syu", it downloaded and installed something like 70 packages, including linux6.3.3-arch1-1, without any error.
Then I immediatly initiated a reboot to run the latest kernel, but after gnome is closed, the computer got stuck on a black screen, full of terminal text, some overlapping other, that I did not picture. At this stage the computer was frozen and the capslock key was blinking, the SysRq keys to reboot did not seem to work, I waited for like 1 minute and then tured it off.

The issue happened right after that.

seth · 2023-05-24 19:51:13

After fixing pacman and kernel, run

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

Head_on_a_Stick · 2023-05-24 19:57:43

RogerRabbit wrote:

The issue happened right after that.

What does fsck(8) say about the FAT filesystem on the EFI system partition? That filesystem doesn't handle cold shutdowns very gracefully.

I prefer to leave the ESP unmounted and keep /boot/ on the root partition for this reason.

RogerRabbit · 2023-05-24 20:06:27

# fsck /dev/nvme0n1p1
fsck from util-linux 2.39
fsck.fat 4.2 (2021-01-31)
/dev/nvme0n1p1: 374 files, 27510/130804 clusters

RogerRabbit · 2023-05-24 20:15:00

@Seth, I'll test with the latest install media to be aligned with the kernel. But I don't think the pacman db is broken, pacman would give some error output, whatever pacman command I try, there is absolutely no output, even just to query --help. There was no error during pacman -Syu neither (before the crash).

@Head_on_a_Stick great advice regarding not mounting the boot partition, does it mean you mount it manually when you need to upgrade grub/kernel ?

Last edited by RogerRabbit (2023-05-24 20:17:50)

seth · 2023-05-24 20:19:51

You can ignore the context of the wiki paragraph.
The point is that it explains how you can fix the system for a (generally) broken pacman by using the pacman copy on the install iso.

Head_on_a_Stick · 2023-05-24 20:27:09

RogerRabbit wrote:

@Head_on_a_Stick great advice regarding not mounting the boot partition, does it mean you mount it manually when you need to upgrade grub/kernel ?

I don't use GRUB myself but if I did I would only have to mount the ESP if the grub package was updated, and even then only for the subsequent grub-install command. I keep the kernel & initramfs images on the root partition. GRUB doesn't need to have the ESP mounted under /boot/.

RogerRabbit · 2023-05-24 20:47:40

Yep Seth that's what I was doing, and now there is news... not sure good or not...

so I mounted everything (including /mnt/boot /mnt/proc, /mnt/dev and /mnt/sys), then ran pacman with --sysroot to install the linux package available in the cache, it failed saying that the files were already there, so I added --overwrite \* to force the installation anyway, it succeeded with no warning, but now after a restart, there is a new error :

mount: /new_root: can't find UUID=XXXXXXXXXXXXXXXXXX.
You are now being dropped into emergency shell.
sh: can(t access tty: job control turned off

Input is not working anymore in emergency shell, so I'd say it's worse now
But maybe this is something I can fix in fstab regarding the UUID because of some weird context ?

Here is the command I used to reinstall the kernel :

pacman --sysroot /mnt -U --overwrite \* /var/cache/pacman/pkg/linux-6.3.3.arch1-1-x86_64.pkg.tar.zst

P. S. I am manually transcriving most of the stuff, that's why I replaced the actual UUID with "XXXXXXXXXXXXXXXXXX"

Last edited by RogerRabbit (2023-05-24 20:51:29)

seth · 2023-05-24 21:01:11

Check "lsblk -f" and compare that to your grub.cfg and fstab

This would not have happened w/ the kernel installation.
Also "/var/cache/pacman/pkg/linux-6.3.3.arch1-1-x86_64.pkg.tar.zst" cannot have worked unless you chrooted into the syste (but then pacman would have failed)

RogerRabbit · 2023-05-24 21:12:08

Well it did, I was not sure if I needed to use the /mnt path or not, knowing that I was using the --sysroot parameter, I tried without it first and it worked.

What do you mean abou the kernel, I should not have tried to reinstall it ?

fstab is okay, it shows the same uuids I can see in blkid (or lsblk -f), grub.cfg also, I can only find the right uuids (nvme0n1p1 and nvme0n1p3, p2 being the swap partition).

seth · 2023-05-24 21:27:37

And is that the same UUID as the error message?
The creation of the initramfs might have failed.

Don't try "-U whatever", actually run "pacman --sysroot /mnt -Syu"
If you run into issues w/ the cache, add "--cachedir /mnt/var/cache/pacman/pkg"

RogerRabbit · 2023-05-24 21:43:26

"pacman --sysroot /mnt -Syu" only offers me to update some packages that were updated in between.

seth · 2023-05-24 21:51:10

pacman --sysroot /mnt -Syu`linux pacman

Arch Linux

#1 2023-05-23 22:53:01

[Solved] Failed to mount /boot

#2 2023-05-23 23:00:12

Re: [Solved] Failed to mount /boot

#3 2023-05-23 23:20:08

Re: [Solved] Failed to mount /boot

#4 2023-05-23 23:43:43

Re: [Solved] Failed to mount /boot

#5 2023-05-23 23:54:53

Re: [Solved] Failed to mount /boot

#6 2023-05-23 23:56:30

Re: [Solved] Failed to mount /boot

#7 2023-05-24 00:02:04

Re: [Solved] Failed to mount /boot

#8 2023-05-24 00:06:33

Re: [Solved] Failed to mount /boot

#9 2023-05-24 00:09:22

Re: [Solved] Failed to mount /boot

#10 2023-05-24 00:42:34

Re: [Solved] Failed to mount /boot

#11 2023-05-24 02:01:57

Re: [Solved] Failed to mount /boot

#12 2023-05-24 06:34:38

Re: [Solved] Failed to mount /boot

#13 2023-05-24 19:44:11

Re: [Solved] Failed to mount /boot

#14 2023-05-24 19:51:13

Re: [Solved] Failed to mount /boot

#15 2023-05-24 19:57:43

Re: [Solved] Failed to mount /boot

#16 2023-05-24 20:06:27

Re: [Solved] Failed to mount /boot

#17 2023-05-24 20:15:00

Re: [Solved] Failed to mount /boot

#18 2023-05-24 20:19:51

Re: [Solved] Failed to mount /boot

#19 2023-05-24 20:27:09

Re: [Solved] Failed to mount /boot

#20 2023-05-24 20:47:40

Re: [Solved] Failed to mount /boot

#21 2023-05-24 21:01:11

Re: [Solved] Failed to mount /boot

#22 2023-05-24 21:12:08

Re: [Solved] Failed to mount /boot

#23 2023-05-24 21:27:37

Re: [Solved] Failed to mount /boot

#24 2023-05-24 21:43:26

Re: [Solved] Failed to mount /boot

#25 2023-05-24 21:51:10

Re: [Solved] Failed to mount /boot

Board footer