Boot failure after update [SOLVED]

paulti · 2024-10-22 13:40:15

Hi !

I have been using arch without major issue since spring, but today i tried booted and ended up in emergency mode. I did a pacman update yesterday before shutting down. I also did a timeshift backup, but probably after the update so I guess it is useless.
So I get asked my root password and then drop in a shell, which advises me to look at journalctl which I did.

I think the most relevant error I get is:

Oct 22 14:51:18 archlinux mountI662]: mount: boot: unknown filesystem type 'vfat'
Oct 22 14:51:18 archlinux mount [662]:
dmesg(1) may have more information after failed mount systen call.

The information I think is relevant to this problem can be seen below (i tried ocr but worked very badly)

But to sum it up I have my EFI partition on nvme0n1p1, and then a root partition which contains the decryption key for the home partition.

I used to have /dev/nvme0n1p1 instead of the UUID=C88E-3DEA in my fstab but I changed it because I read it could cause an issue, but the error stayed the same.

Also, I tried booting using archiso and mounting partitions manually and I was able to mount /dev/nvme0n1p1 in /boot, and I could see the usual folders in there so it doesn't seem to be broken ?

I have read other threads with similar issues and some people mentioned downgrading the linux kernel ? Does this apply here ?

If you need more information to understand the problem don't hesitate to ask I will gladly provide it !

Thanks a lot in advance for your help !

Last edited by paulti (2024-10-25 12:39:09)

cryptearth · 2024-10-22 14:01:22

usually this happens when the to-be-booted kernel in /boot gets out of sync with the rest of the system - usually by bad mountpoints or have the esp not mounted during kernel update

way to solve: boot arch install - mount all correctly - reinstall kernel and check boot config
also make sure the chroot setuo matches the system

sainishwanth · 2024-10-22 14:02:57

paulti wrote:

I have read other threads with similar issues and some people mentioned downgrading the linux kernel ? Does this apply here ?
!

The kernel recently got updated in the repos (couple hours), which most likely caused the issue, as it loooks like whatever kernel version you have that is being booted, does not match the existing modules in /usr/lib/modules. Could be bad mountpoints during installation.

You can try downgrading for now, as a temporary fix. Other threads like this one - https://bbs.archlinux.org/viewtopic.php?id=178358
might offer a proper solution.

Last edited by sainishwanth (2024-10-22 14:06:41)

Omarz2012 · 2024-10-22 14:04:30

maybe you can try to mount your partitions manually and regenerate fstab? if not then reconfigure your encryption.

paulti · 2024-10-22 19:44:53

cryptearth wrote:

way to solve: boot arch install - mount all correctly - reinstall kernel and check boot config
also make sure the chroot setuo matches the system

Ok I want to try this but I have a couple of questions before I start tinkering:
1. How can I see the kernel version my system is trying to boot and the one that is installed ? Can I see it from the archiso environment ?
2. How to reinstall the kernel ?
I found this article that gives a way to do it.
-> Chroot, then pacman -S <kernel name>
-> Is the rest of the article also necessary (reinstall grub, recreate linux image, regen fstab) ? It makes sense that I have to rebuild the linux image with the clean kernel.
3. Do I reinstall the same version of the kernel ? If not, which version ?
4. How do I prevent this from happening next time I update ?
5. What do I need to check in boot options (assuming this refers to /etc/default/grub)
6. Finally, what do you mean exactly by making sure that the setuo matches the system ?

Sorry this is a bit heavy but I think this is all I need to solve the problem, thanks again for your speedy answers !

seth · 2024-10-22 20:04:43

You can just temporarily comment the boot partition entry in your fstab and that will probably allow you to boot the system

1
"pacman -Qs kernel", for the booting attempt you'll have to check the system journal of the failing boot - it'll also tell you whether you're booting from a dedicated partition.
"uname -a" won't work from the iso (it'll show the iso kernel) and file /boot/vmlinuz-linux doesn't work because you don't know what kernel file is being loaded…

2
"pacman -S <kernel name>" will do, but oc. you need to arch-chroot into the system and don't forget to mount the /boot partition - assuming you're booting from one.

3
the only thing that really matters is that the installed and booting kernel are the same so you've access to the relevant on-disk modules

4
don't forget to mount the /boot partition before updating the kernel or if you're actually booting from the root partition, don't mount a spurious /boot partition

5
the kernel commandline (cat /proc/cmdline - but again: not possible from the install iso boot)

cryptearth · 2024-10-22 20:09:21

in addition to seth (damn - F5 and such ...): https://bbs.archlinux.org/viewtopic.php … 2#p2203932 summarizes your issue - for some reason the location your bootloader loads the kernel and initrd from wasn't available during the update and hence not updated - now the boot mismatches the system and crashes as it can't load required modules

paulti · 2024-10-22 21:08:44

Alright, it seems like I managed to solve the issue, I rebooted multiple times successfully and everything seems to be normal !! Thank you so much

Here is what I did in case anyone runs into the same issue:

1. Boot into archiso
2. Decrypt my root partition, then my home partition
3. Mount them were they have to be, i.e. at /mtn/ and /mnt/home
4. Mount my boot partition at /mnt/boot (in my case it was /dev/nvme1n1p1 but this is dependent on the system).
5. arch-chroot into the system
6. Check the kernel version with pacman -Qs kernel (it is the version of the linux package), even though it is not necessary.
7. Reinstall the kernel with

pacman -S linux

8. Exit chroot and reboot

To avoid this happening again, whenever I see that the linux package will be updated, I check that the boot partition is mounted, is that correct ? I feel like I updated multiple times without issues, I wonder what happened this time....

Thanks again I feel so much better now

P.S. I also tried to comment the boot line in fstab and it managed to boot, whytf is that even working hahaha. However it seemed to be in a weird state, my mouse and keyboard were working erratically

seth · 2024-10-22 21:15:54

I wonder what happened this time....

The boot partition wasn't mounted, wa unmounted, was re/mounted read-only or the mkinitcpio hook that copied over the kernel failed for reasons™.
Always pay attention to the update process.
Do you have files in the mountpoint (ie /boot when the boot partition isn't mounted?)
Is there a parallel OS (windows)?

whytf is that even working hahaha

The mounting of the boot partition isn't necessary to boot the system, but systemd will shortcut to the rescue.target whenever an entry in the fstab cannot be mounted.
The input issues were likely because you also lacked a couple of other modules.

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

cryptearth · 2024-10-22 23:42:22

paulti wrote:

To avoid this happening again, whenever I see that the linux package will be updated, I check that the boot partition is mounted, is that correct ?

that pretty much sums it up, yes

paulti wrote:

I feel like I updated multiple times without issues, I wonder what happened this time....

can be anything from cosmic rays flipping a bit in memory (although many people say that's not possible) to hardware malfunction due to unstable power
even following the install guide and having done several times it just happens from time to time - and mosty just goes unnoticed because having a non-critical mount fail ususally doesn't prevent the system from boot (as in: wasn't able to mount because the drive wasn't physical present)
using systemd-boot one can get away without any fstab at all and have systemd auto-mount /, /boot, /efi, /home and up to several others all by itself - but it also can fail at any given point in time for no obvious reasons
I played around with this in a qemu kvm - and sometimes it worked sometimes it failed - from what I was able to find by a search systemd heavily relies on the uefi to set a few variabls so it can find its root to perform all that auto-moun magic - and for some reason in some version of qemu this somehow got borked and hence just failed - but was fixed in the next version
same can happen on realy hardware: for some reason the uefi fails to correctly set the required stuff at boot and BANG - systemd-boot auto-mount without fstab fails completely - during a warm reboot all of the sudden fixes it out of nowhere and the error is not reproduceable
so - although you maybe not actively did anything causing this it still somehow failed due to whatever only god knows - but now you learned from it and experienced one of the hot topics yourself and how to get out of it ...

.. *cut for being offensive*

Arch Linux

#1 2024-10-22 13:40:15

Boot failure after update [SOLVED]

#2 2024-10-22 14:01:22

Re: Boot failure after update [SOLVED]

#3 2024-10-22 14:02:57

Re: Boot failure after update [SOLVED]

#4 2024-10-22 14:04:30

Re: Boot failure after update [SOLVED]

#5 2024-10-22 19:44:53

Re: Boot failure after update [SOLVED]

#6 2024-10-22 20:04:43

Re: Boot failure after update [SOLVED]

#7 2024-10-22 20:09:21

Re: Boot failure after update [SOLVED]

#8 2024-10-22 21:08:44

Re: Boot failure after update [SOLVED]

#9 2024-10-22 21:15:54

Re: Boot failure after update [SOLVED]

#10 2024-10-22 23:42:22

Re: Boot failure after update [SOLVED]

Board footer