You are not logged in.
Hello,
Super newbie here who has been thrilled with using Arch for the past several months but ran into an issue early today that has put my primary laptop out of commission for the moment.
I've been successfully updating regularly and figuring out rollbacks and troubleshooting mostly when things have cropped up in the context of specific configs/standalone apps I have to update/build myself, but this is the first time my general package updates have failed and I have seemingly worked myself into more of a pickle by booting a snapshot and 'solving' it. I'm worried about messing things up further/irreversibly, so I wanted to reach out for help from more experienced users.
The lead up: after failing to successfully update any packages due to 'GPGME error: No data' and 'x.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)' issues, I attempted several of the solutions on the wiki and on related posts (clearing pacman/sync/, fresh mirror list, re-initializing/populating keys, the DISPLAY value quirk breaking the GPG-Flow, etc.) without luck.
I then tried to boot an earlier snapshot to see if that helped/turned up anything, but I had forgotten the scope/depth of my backup configuration didn't include these details anyway as I haven't used snapper recently/frequently (first facepalm of the day, certainly not the last). While booted into the snapshot, I continued troubleshooting by reverting the fresh mirror list I had been experimenting with back to what I had originally in the spirit of keeping as much the same as I could. I then simply commented out the mirror list entry I suspected of returning markdown or whatever was leading to the "error: could not open file /var/lib/pacman/sync/[core/extra/multilib].db: Unrecognized archive format" errors. That seemed to help, as the update proceeded as expected (not sure why so many servers are listed if this kind of issue with one mirror going down isn't seemingly detected/handled and then a fallback used already?). I thought that was the end of it, but to try and keep things mostly the same once more I remembered I had booted a snapshot and decided to revert back to the most recent state to keep things as current as I could.
This appears to be the point at which I went wrong, as now booting fails and I am in emergency mode where inspection of the journalctl -xb contents indicates:
mount: /boot: unknown filestystem type 'vfat'
and that, based on other posts I've seen, that the boot partition was not mounted when the update occurred. It dawned on me, with increasing palming of the face, that I did seem to remember needing to confirm rollbacks and after checking, that 'sudo snapper rollback' was definitely skipped...
attempting to update the kernel with:
pacman -S linuxstirs up the previously seen:
error: could not open file /var/lib/pacman/sync/core.db: Unrecognized archive format
regardless of the mirrorlist edit that worked before and no target is found. What am I missing and how can I unscrew myself without losing data? I'm now anxiously second-guessing even things I've done before in case I am forgetting other wrinkles again. I think I have the installation media around still, this bootable iso can be used to fix the kernel version while leaving everything else intact, correct? Although I assume the version on it currently is outdated as well, and I'll need to recreate it. For future reference, besides just not doing the obvious goofs, what strategies do you use to avoid kernel issues like this and make unwinding mistakes smoother?
Additional information to flesh this out better:
uname -rgives:
6.13.6-arch1-1
while,
pacman -Q linuxgives:
6.13.2-arch1-1
Other examples of kernel issues stemming from updates seem to have it the other way around... where attempting to bump up the version leads to 'uname -a' showing an older version. Any ideas how I managed this?
I attempted to dump the 'journalctl -xb' output to the one other USB drive I have available but the inability to recognize the 'vfat' filesystem got in the way here too, going to have to look for another drive to try another format for now I guess. Any slick tips for gathering output text while in emergency mode and sharing that are appreciated.
The /etc/fstab/ entry for /boot:
# /dev/nvme0n1p1
UUID=4C58-0010 /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii, shortname=mixed,utf8,errors=remount-ro 0 2
and,
cat /proc/cmdlineyields,
BOOT_IMAGE=/vmlinuz-linux root=UUID=XXXX rw rootflags=subvol=@ zswap,enabled=0 rootfstype=btrfs loglevel=3 quiet
where UUID does indeed match the root/snapshot btrfs entries in fstab.
I apologize for not getting the journalctl and other info explicitly copied and published for review, trying to figure that out currently but wanted to get the conversation started now as I am fairly pressed to get this working as soon as I can without any additional bungling on my part.
Thank you for taking the time to read this and any help/related information you can provide.
Cheers,
aa
Offline
classic update fail resulting in mismatch between booting kernel and installed packaged
boot install media
chroot
reinstall kernel (should already run mkinitcpio and install kernel)
reboot
Offline
Thanks for the quick reply! I was getting the sense this was a classic goof, which is both heartening on the one hand but embarrassing on the other as I haven't yet pieced together the solution for my case with what is already searchable haha
Following that process, I booted my original install media (a bit older, kernel version 6.11.5-arch1-1), setup a connection with iwctl and then, because I originally followed the wiki's suggested snapper setup, I have to do the following to mount the filesystem:
mount -o subvol=@ /dev/nvme0n1p2 /mntbefore I can chroot successfully with,
arch-chroot /mnt /bin/bashAfter,
pacman -Sythe needed target is found so I can finally:
pacman -S linuxBut... at this point it shows the old version as 6.13.2.arch1-1 and the new version available as 6.13.6.arch1-1, with no sign of the lagging 6.12.9.arch1-1 that was originally out of step. I guess things can't get much messier if I go ahead with that, but what stale kernel was 'pacman -Q linux' possibly referencing? Could a boot configuration relic be pointing things the wrong way? Now that I'm touching the filesystem again I'd like to be more confident/clear about changes I make to avoid muddying the water further but maybe I'm just being gun shy now.
Thanks again.
Offline
cryptearth forgot to tell you to mount the /boot/ partition so mount the root subvolume again but add this after chrooting and before reinstalling the kernel package(s):
mount -aAlso:
pacman -Sy
EDIT: what happens if you run fsck(8) on the EFI system partition? Is it corrupted?
Last edited by Head_on_a_Stick (2025-03-12 07:45:03)
Jin, Jîyan, Azadî
Offline
Oof, just now seeing this but thank you for this important addendum. I should of stepped away sooner as it'd been a long day and I think I got myself turned around - that older kernel version must have been referencing something else because I do think the ones in the original post were correct in the end. Perhaps more telling, you'd think after the first goof I'd think twice about proceeding without mounting the /boot/ partition.. guess that's what I get for moving hastily instead of snail-style, slow but avanzo.
After running the above ill-advised partial update, as written, things managed to boot and I eventually got a full upgrade in successfully. However, does proceeding without /boot mounted mean there is something potentially lurking I should clean up/check/repair further? Didn't bother with fsck after successful boot thinking it was baked into that process.
I say 'eventually' I got a full upgrade in only because immediately after boot the same
error: GPGME error: No data
cropped up as before. In the end what did the trick was removing the full sync directory as I tried before:
sudo rm -r /var/lib/pacman/sync/but instead of trying to update things immediately afterward and having things recreated in seemingly the same error state, making sure to do an actual system restart before running a now successful:
sudo pacman -SyuI know rebooting ASAP after updates are successful is advised for things to be properly/fully implemented, but is this need for a system restart between such changes fairly standard as well?
It hasn't been very long and it's only at a surface level I can say things seem to be behaving as expected, but unless I'm cautioned to the contrary I think the original issues I posted about have been resolved. Of course, any further insight is very much appreciated, but I'd like to thank you both for the help and information you've already provided.
Cheers,
aa
Offline
does proceeding without /boot mounted mean there is something potentially lurking I should clean up/check/repair further?
If you now have the /boot/ partition mounted correctly there may be another /boot/ directory hidden by the mountpoint. It won't cause any harm being there but to remove the excess you can un-mount the /boot/ partition, clean up /boot/ on the root partition, then re-mount.
Didn't bother with fsck after successful boot thinking it was baked into that process.
I'm just wondering why we keep seeing this problem on the forums. The /boot/ partition is always listed in fstab but people still manage to upgrade the kernel without it mounted. I would really like to find out why.
Is your /boot/ partition configured to fsck at boot in /etc/fstab? Does your mkinitcpio.conf have the fsck hook? Are there any messages in the journal about corruption on the /boot/ partition?
Jin, Jîyan, Azadî
Offline
I'm just wondering why we keep seeing this problem on the forums. The /boot/ partition is always listed in fstab but people still manage to upgrade the kernel without it mounted. I would really like to find out why.
I've asked that question before, it's usually been because they decided to change their layout far too late in the process.
IE, "oh, I forgot to mount my boot partition, better do it now"
Last edited by Scimmia (2025-03-12 16:05:03)
Offline