[SOLVED] Best way to recover from "failed to mount /boot"?

Cheeto · 2023-05-20 11:36:15

Restored a btrfs snapshot, now I'm getting that error on boot. What actually happened was I made the standard snapshot to restore to, then ran `yay-Syyu` and ran all the experiments I needed, then went to restore as usual. I've never had any issues restoring snapshots before but I guess there's a first time for everything.

NOTE: I CAN boot into some of the newer snapshots (presumably ones that occurred after the kernel upgrade) and use the system that way, thanks to grub-btrfs--that's how I'm writing this.

I read some things here:
https://bbs.archlinux.org/viewtopic.php?id=284263
https://bbs.archlinux.org/viewtopic.php?id=269588
https://bbs.archlinux.org/viewtopic.php?id=279804
https://wiki.archlinux.org/title/downgr … the_kernel

What I gather is that I need to downgrade the kernel. I'm asking for advice (or clarification of the existing advice on whether that's appropriate and how to do it with respect to my situation.

- It's an archinstall installation, with the home subvolume deleted because I prefer home to revert when I restore snapshots
- I don't really clear any caches or anything so if the previous kernel is in there then that may be helpful
- Again, I can boot into some newer snapshots and use the GUI environment with no issue
- I can boot into the older snapshot that I need to get back to, but I'm stuck at the CLI there
- Fstab looks like I would expect it to:

cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p2
UUID=320d4560-699c-4348-be74-9e1236f0b6ce	/         	btrfs     	rw,noatime,s
sd,discard=async,space_cache=v2,subvol=/@	0 0

# /dev/nvme0n1p1
UUID=0CC0-5BB7      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

# /dev/nvme0n1p2
UUID=320d4560-699c-4348-be74-9e1236f0b6ce	/.snapshots	btrfs     	rw,noatime,ssd,discard=async,space_cache=v2,subvol=/@.snapshots	0 0

# /dev/nvme0n1p2
# UUID=320d4560-699c-4348-be74-9e1236f0b6ce	/home     	btrfs     	rw,noatime,ssd,discard=async,space_cache=v2,subvol=/@home	0 0

# /dev/nvme0n1p2
UUID=320d4560-699c-4348-be74-9e1236f0b6ce	/var/cache/pacman/pkg	btrfs     	rw,noatime,ssd,discard=async,space_cache=v2,subvol=/@pkg	0 0

# /dev/nvme0n1p2
UUID=320d4560-699c-4348-be74-9e1236f0b6ce	/var/log  	btrfs     	rw,noatime,ssd,discard=async,space_cache=v2,subvol=/@log	0 0

---

EDIT: I tried restoring the old snapshot (by "old" I mean like 2 hours old, just distinguishing it from the "newer" ones that automatically happen hourly after I upgraded the system) and running `pacman -S linux-zen linux-zen-headers` then `mkinitcpio-P` and rebooting, but I still get the same error.

Last edited by Cheeto (2023-05-24 02:12:27)

seth · 2023-05-20 11:52:08

You're booting a kernel newer than the one that was installed during that snapshot and lack (at least) the vfat module.
Boot the fallback image, it should™ have most modules.

What I gather is that I need to downgrade the kernel.

Yes, but what actually is the point of using the older snapshot?
You can also just boot (w/ the fallback image) or chroot (from eg. the install iso using arch-chroot) into the older snapshot, don't forget to mount the /boot partition and udpate the kernel (or rather everything) there.

Edit

running `pacman -S linux-zen linux-zen-headers` then `mkinitcpio-P` and rebooting, but I still get the same error.

See the highlighted part.

Last edited by seth (2023-05-20 11:53:13)

Cheeto · 2023-05-20 12:10:23

seth wrote:

You're booting a kernel newer than the one that was installed during that snapshot and lack (at least) the vfat module.
Boot the fallback image, it should™ have most modules.
What I gather is that I need to downgrade the kernel.
Yes, but what actually is the point of using the older snapshot?
You can also just boot (w/ the fallback image) or chroot (from eg. the install iso using arch-chroot) into the older snapshot, don't forget to mount the /boot partition and udpate the kernel (or rather everything) there.
Edit
running `pacman -S linux-zen linux-zen-headers` then `mkinitcpio-P` and rebooting, but I still get the same error.
See the highlighted part.

What I gather is that I need to downgrade the kernel.

Yes, but what actually is the point of using the older snapshot?

Sorry, I don't understand. Is it a rhetorical question? If so I don't know what you mean?

> You can also just boot (w/ the fallback image) or chroot (from eg. the install iso using arch-chroot) into the older snapshot, don't forget to mount the /boot partition and udpate the kernel (or rather everything) there.

Booting the fallback unfortunately returns the same error. Note, that for each case, uname -a returns the current linux-zen kernel according to the arch site.

Apologies for the formatting, I'm typing this from my phone and the post editor doesn't have many options.

seth · 2023-05-20 12:34:59

Sorry, I don't understand. Is it a rhetorical question? If so I don't know what you mean?

Why are you trying to boot an older snapshot?
You'll update the system from there anyway, yesno?

I do have the vfat module in the fallback initramfs - maybe the older fstab just references the wrong UUID?
Compare it to "lsblk -f" and in doubt just comment the /boot line in the fstab (nb. that you won't have the /boot partition mounted and kernel or grub updates will go into the wrong location!)

Cheeto · 2023-05-20 13:14:26

I experiment with new packages and after a year of doing that, I concluded that no matter how much you uninstall, they always leave files and, importantly, often PERMANENT changes which can accumulate and conflict with things.

It was recommended to use BTRFS for the snapshots feature, and it's worked brilliantly so far. This kernel thing is new, but it's like the home issue I had before. I needed to find a way to include home in the snapshots, because if it doesn't revert then it defeated the purpose of restores.

---

I see your posts so much, it's insane what you do for people here. However, I may need more stepwise clarification on what you just described.

It sounds like you're saying:

- you have the vfat module somewhere, and you're suggesting that I don't but it seems like I do based on the fstab I supplied
- therefore, I should compare the UUID in the fstab to the output of fsblk -rf
- (it's not clear to what help comparing them would do)
- I presume there are steps you omitted, and you're saying that if they don't resolve things, then I can try to comment out the /boot line in the fstab, but you're saying that would cause other complications (that are over my head)

Last edited by Cheeto (2023-05-20 13:14:53)

seth · 2023-05-20 14:25:53

The vfat module is supposed to be in the fallback initramfs, if it is (on your copy) it's likely not the cause for the failure to mount the boot partition (the journal will likely tell that)

If the UUIDs are different that means that teh fstab references the wrong partition (eg. because that has changed between the snapshot and now) and fixing that would allot the boot partition to mount.
You can also try to manuall "mount /dev/nvme0n1p1 /boot" - if /dev/nvme0n1p1 is a vfat partition and the module available through the initramfs, it'll mount w/o error.

COmmenting the entry in the fstab will prevent the attempt to mount it and allow the boot to continue, but you cannot write into the boot partition what will prevent you from fixing any files there and installing the kernel will install it into the /boot directory on the root partition (which is supposed to be an empty mountpoint only)

HOWEVER:
If you've an install iso at hand, you can just boot that, mount the root and /boot partitions, arch-chroot into the system and update it to align system and kernel.
https://wiki.archlinux.org/title/Pacman … _rebooting
"pacman -Syu"

Cheeto · 2023-05-22 05:53:54

seth wrote:

The vfat module is supposed to be in the fallback initramfs, if it is (on your copy) it's likely not the cause for the failure to mount the boot partition (the journal will likely tell that)
If the UUIDs are different that means that teh fstab references the wrong partition (eg. because that has changed between the snapshot and now) and fixing that would allot the boot partition to mount.
You can also try to manuall "mount /dev/nvme0n1p1 /boot" - if /dev/nvme0n1p1 is a vfat partition and the module available through the initramfs, it'll mount w/o error.
COmmenting the entry in the fstab will prevent the attempt to mount it and allow the boot to continue, but you cannot write into the boot partition what will prevent you from fixing any files there and installing the kernel will install it into the /boot directory on the root partition (which is supposed to be an empty mountpoint only)

HOWEVER:
If you've an install iso at hand, you can just boot that, mount the root and /boot partitions, arch-chroot into the system and update it to align system and kernel.
https://wiki.archlinux.org/title/Pacman … _rebooting
"pacman -Syu"

Ok I booted into the fallback option and it failed. Ran `journalctl -b` and got a bunch of stuff, but this pic seems to be the relevant part:

https://i.imgur.com/NbDdIAn.jpg

Sorry, I don't know how to get that as text onto my phone.

I do have access to an Arch iso, but mountain was always incredibly confusing to me, and it got worse with btrfs. I thought this pic https://i.imgur.com/BWFQB5x.jpg would help me understand where things are mounted but that never worked when I tried to UNMOUNT the home subvolume for example, which I had to unmount via ID as a workaround or else comment it out of fstab. Could you possibly give me instructions or direct me to something that I could understand and follow?

seth · 2023-05-22 06:32:32

Yeah, you're missing the vfat module.
Not sure how that's possible w/ the fallback image, but we'll take that sa given.

"Mounting", "mountains" are done over at alpine linux.

mount  -o 'subvol=/@' /dev/nvme0n1p2 /mnt # yo probably don't even have to specify the subvol
ls /mnt # shows your root partition?
mount  /dev/nvme0n1p1 /mnt/boot
# if there're no errors above
arch-chroot /mnt # arch-chroot sets up the pseudo FS for you
pacman -Syu # update the system

it got worse with btrfs

Then why did you opt for a complex and still somewhat experimental FS?

Cheeto · 2023-05-22 07:16:25

seth wrote:

Yeah, you're missing the vfat module.
Not sure how that's possible w/ the fallback image, but we'll take that sa given.
"Mounting", "mountains" are done over at alpine linux.
mount  -o 'subvol=/@' /dev/nvme0n1p2 /mnt # yo probably don't even have to specify the subvol
ls /mnt # shows your root partition?
mount  /dev/nvme0n1p1 /mnt/boot
# if there're no errors above
arch-chroot /mnt # arch-chroot sets up the pseudo FS for you
pacman -Syu # update the system
it got worse with btrfs
Then why did you opt for a complex and still somewhat experimental FS?

Thanks autocorrect ?

It may be possible due to me earlier running `pacman -S linux-zen linux-zen-headers`? Even the fallback image appears to load the regular current zen kernel.

I went with btrfs because...
- enough reliable sources said its mature enough
- there's a large enough support community for it that it's not necessarily more difficult to manage than ext4 despite being more complex
- everything is complex as an early-stage linux user/learner, so btrfs is not significantly different
- you don't know what you don't know, so you study what you determine to be a reasonable amount but at some point you make a choice and learn from the consequences
---- trying different things and learning in general, including learning the boundaries, is inherent in the process of learning linux for probably the vast majority of people who don't have IRL mentors to teach us
- I learned enough after about a year trying a trillion different linux setups that I trusted my intuition that I would probably be OK trying btrfs
- I struggled through some challenges, and confirmed that learning from mistakes is inherent in learning linux and I was able to succeed with some help from the community
- ChatGPT is a thing now and it blew my mind how helpful it is for learning so much about linux, it misses the mark occasionally but for a HUGE amount of stuff it has completely replaced my reliance on the community and skyrocketed my capabilities, teaching me things happily where the linux community is notoriously....mean and anti-helpful (this is not a commentary about you--god bless you--just a literal response to a question you had)
---- with respect to btrfs specifically, it's the biggest reason I've had success
- until this moment, btrfs has been entirely manageable, this current issue was not something I've come across or anticipated based on what I've learned from what I've read (reminder that as a learner I don't necessarily retain everything I learn, let alone everything I simply read)
- I need btrfs because despite its difficulties, the benefits of allowing me to experiment more than before are worth it for the learning process
- you guys will, by definition, only see my struggles, not my successes, so its not apparent here how much trouble I've NOT had, or how much of the trouble I HAVE had that I've been able to manage successfully and learn and grow from

But ya it's a fair question. I'll report back ASAP as soon as I can get back to my machine.

Last edited by Cheeto (2023-05-22 07:38:10)

seth · 2023-05-22 07:34:38

Even the fallback image appears to load the regular current zen kernel.

That's normal and expected, it would just be expectable to have the matching vfat module from the fallback initramfs.

ChatGPT is a thing now and it blew my mind how helpful it is for learning

You better drop that idea asap.
ChatGPT is an authorative bullshit generator - there's no way telling whether the answer it gives you is reasonable, correct or complete nonsense.
It is NOT intelligent, but a glorified ELIZA.
You're better off googling for stack-overflow threads.

Cheeto · 2023-05-22 14:58:16

seth wrote:

Even the fallback image appears to load the regular current zen kernel.
That's normal and expected, it would just be expectable to have the matching vfat module from the fallback initramfs.
ChatGPT is a thing now and it blew my mind how helpful it is for learning
You better drop that idea asap.
ChatGPT is an authorative bullshit generator - there's no way telling whether the answer it gives you is reasonable, correct or complete nonsense.
It is NOT intelligent, but a glorified ELIZA.
You're better off googling for stack-overflow threads.

I booted into the iso and ran the commands above. I wasn't ignoring all the errors, I just wanted to be sure to collect them all so I could report back here:

https://i.imgur.com/bpBHsoM.jpg

seth · 2023-05-22 15:00:00

seth wrote:

mount -o 'subvol=/@' /dev/nvme0n1p2 /mnt

Cheeto · 2023-05-22 15:16:05

seth wrote:

seth wrote:
mount -o 'subvol=/@' /dev/nvme0n1p2 /mnt

Is it random? The time before it was 1n1, so I did it by habit, and this time it's back to 1n1 again...?

https://i.imgur.com/7nFoepD.jpg

Will report back ASAP.

---

Ok I ran the commands successfully (though it was 1n1 this most recent time) all with no errors, and was able to chroot in and update. However, I'm still getting the same `failed to mount /boot` problem when booting normally (which should be the restore of the pre-experiments snapshot), and when booting into the pre-experiments snapshot itself, and when booting into the fallback img version of the same snapshot.

I don't know if it's relevant, but keep in mind I did run those commands way earlier (installing linux-zen and linux-zen-headers) on a normal boot where I'm getting the error. I don't know if that caused any unexpected changes or complications.

Last edited by Cheeto (2023-05-22 15:30:53)

ewaller · 2023-05-22 15:32:06

seth wrote:

ChatGPT is a thing now and it blew my mind how helpful it is for learning
You better drop that idea asap.
ChatGPT is an authorative bullshit generator - there's no way telling whether the answer it gives you is reasonable, correct or complete nonsense.
It is NOT intelligent, but a glorified ELIZA.
You're better off googling for stack-overflow threads.

For instance, ask ChatGPT to provide you with references and see what it says. Then ask it about the age of its knowledge base.

Last edited by ewaller (2023-05-22 15:32:28)

seth · 2023-05-22 15:33:50

The first number is not stable, your newest screenshot would be for nvme1n1p2
The second number (the partition) is stable, it cannot have been *n1p1 before.
That's the vfat boot partition.

Edit: why

Ok I ran the commands successfully (though it was 1n1 this most recent time) all with no errors

this is *impossbile*

Document everything you do.

Last edited by seth (2023-05-22 15:36:03)

Cheeto · 2023-05-22 16:31:06

seth wrote:

The first number is not stable, your newest screenshot would be for nvme1n1p2
The second number (the partition) is stable, it cannot have been *n1p1 before.
That's the vfat boot partition.
Edit: why
Ok I ran the commands successfully (though it was 1n1 this most recent time) all with no errors
this is *impossbile*
Document everything you do.

https://youtu.be/bkKtoAC3it8

Apologies for the potato quality while YouTube processes the video.

Basically, I booted into the arch iso and connected to the internet then cleared the terminal and began recording.

Ran the commands with no errors, update showed no updates available because the update I had completed per the last post, then I rebooted, got the same error, ran journalctl -b | grep vfat to show that it isn't finding that or something (can't remember the output but it's the same as earlier, really hard to multitask to check on the phone), then ran journalctl -b and cycled through the pages in case there's something helpful, ending on the error on the last page, which has the /boot error which should also be the same as before.

Also apologies for the angle. I checked it 400 times before recording but I think it cropped part of the outside for...phone camera reasons?

Last edited by Cheeto (2023-05-22 16:41:54)

seth · 2023-05-22 18:41:49

I don't need to ses you typing, just what you typed.
Photos of some screens after the effect would be fine/better.

When the boot fails into the rescue.target, check "uname -a" and, if you've access to the root partition, the directories in /lib/modules.
If you successfully updated, the kernel and the modules should align and be 6.3.3.zen1-1

jl2 · 2023-05-22 18:52:38

have you tried loading the vfat module after a normal boot? (modprobe vfat) I don't see this anywhere here.
if it succeeds, post cat /etc/mkinitcpio.

EDIT: first try from within seth's rescue.target.

Last edited by jl2 (2023-05-22 19:00:42)

Cheeto · 2023-05-23 02:49:47

Thank you guys for your patience. @jl2 I ran the second command just in case something useful was in it despite the first one "failing".

https://i.imgur.com/5yGI2IA.jpg

Scimmia · 2023-05-23 03:17:29

So the problem is that the bootloader is loading an old kernel. Where is it getting that? That's the question you need to figure out

Booting the fallback won't help, because by the time you're trying to use vfat, you're past the initramfs and what's in it isn't available.

Last edited by Scimmia (2023-05-23 03:20:02)

seth · 2023-05-23 06:41:09

OP restored an old btrfs snapshot and is loading a new kernel from the current vfat /boot on the old system.
The current question is why updating the system from there does align the versions.

Scmimmia wrote:

Booting the fallback won't help, because by the time you're trying to use vfat, you're past the initramfs and what's in it isn't available.

Try

modules_load=vfat

https://wiki.archlinux.org/title/Kernel_parameters

Edit: ah, screenshot.
You didn't mount the /boot partition when updating the system during the chroot.
Or you mounted the wronng partition or into the wrong location.

Last edited by seth (2023-05-23 06:43:41)

Cheeto · 2023-05-23 09:24:17

EDIT: yes I see the mistake about linux-zen-headers, I'll retry it if anyone thinks it would help the situation. Pacman says it's already installed and I'm not sure if the headers error is referring to that package.

---

To check why it might be using the wrong kernel, I chrooted in (@seth this should confirm the command I'm using to mount the boot partition) and checked if the kernel needed updating. I am getting some errors about headers and mkinitcpio -P fails, though I don't know how relevant that is.

Here's a screenshot:
https://i.imgur.com/DYvNAYy.jpg

Also, before that I did the following on a regular boot:
- checked the grub config at /boot/grub/grub.cfg and I don't really know exactly what to look for but it seems normal (it doesn't mention any kernel versions anywhere that I saw)
- ran grub-mkconfig -o /boot/grub/grub.cfg and it didn't appear to have errors.
- confirmed the boot partition is correctly located in the fstab.

@Seth should I try modules_load=vfat on a regular boot, or when booting the snapshot from the fallback img?

Last edited by Cheeto (2023-05-23 09:31:04)

seth · 2023-05-23 10:57:01

"linux-zen-headers"
You're not reinstalling the kernel because of the pacman error and then mkinitcpio fails because you're trying to build for the (still) old kernel w/ the updated on-disk modules.
That doesn't explain why you're in the current situation (I suspect you forgot to mount the boot partition) but explains why you didn't get out of that.

try modules_load=vfat … when booting the snapshot from the fallback img

Cheeto · 2023-05-23 15:17:28

> at grub, choose arch Linux snapshots
> choose the snapshot
> highlight the fallback.img one and press e
> add modules_load=vfat to the end and press ctrl+x
> get this screen (same error as always): https://i.imgur.com/JzcuOZs.jpg

Regarding not mounting the boot partition, when do you feel I may have forgotten it? At least in the video I made and in the screenshot after, both show me mounting the boot partition with your command (which seems appropriate as I have this command in my personal "arch way" notes: `mount /dev/nvmeXnYpZ /mnt/boot` which suggests that's where I mount it typically during install).

seth · 2023-05-23 16:19:56

You're missing the point.
Your most recent effort to install the kernel failed because you passed an invalid package list.

We can speculatze all day long what errors existed on your previous attempts (i could not read anything at least in the beginning of the video), but we know what right now was your most recent flub.
Irobn that out to hopefully return to a working systen.

Arch Linux

#1 2023-05-20 11:36:15

[SOLVED] Best way to recover from "failed to mount /boot"?

#2 2023-05-20 11:52:08

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#3 2023-05-20 12:10:23

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#4 2023-05-20 12:34:59

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#5 2023-05-20 13:14:26

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#6 2023-05-20 14:25:53

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#7 2023-05-22 05:53:54

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#8 2023-05-22 06:32:32

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#9 2023-05-22 07:16:25

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#10 2023-05-22 07:34:38

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#11 2023-05-22 14:58:16

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#12 2023-05-22 15:00:00

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#13 2023-05-22 15:16:05

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#14 2023-05-22 15:32:06

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#15 2023-05-22 15:33:50

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#16 2023-05-22 16:31:06

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#17 2023-05-22 18:41:49

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#18 2023-05-22 18:52:38

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#19 2023-05-23 02:49:47

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#20 2023-05-23 03:17:29

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#21 2023-05-23 06:41:09

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#22 2023-05-23 09:24:17

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#23 2023-05-23 10:57:01

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#24 2023-05-23 15:17:28

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

#25 2023-05-23 16:19:56

Re: [SOLVED] Best way to recover from "failed to mount /boot"?

Board footer