Arch Linux

cazuza

Member

Registered: 2014-05-04

Posts: 34

Repairing a broken update on an encrypted btrfs drive [SOLVED]

Hello,

Something went really bad.

On my laptop I have an archlinux installed on a new ssd bought few month ago. The disk was encrypted at the installation and everything was working flawlessly for the last few months. The filesystem is btrfs and I have weekly snapshots.
Yesterday I tried to update (including linux 6.13.6 and linux headers) and got weird error messages, right after the kernel hooks : "impossible to write the file [path to the updated programs] in [install path]" multiple times, for every updated program.
I tried to relaunch a "pacman -Syu" but this time I immediately got another message. something like "impossible to run the command, unknown error" (sorry I don't remember the exact message, never saw it before)

I then opened timeshift to check that the btrfs snapshots were fine and eventually reboot on a snapshot but it said "the selected disk is not a system disk, please choose another one". (i'm translating and it's from memory so it might not be exactly this message"
I opened gnome-disk which said the disk was ok.

A little in panic I closed the gnome session and got a black screen. I could still access a tty that I used to reboot.

At reboot I can access grub but arch refuses to boot, as well as any of the btrfs snapshots.
Here's the error message (translated from french):

loading linux linux
error : file "/vmz-linux" not available
loading initial memory disk
error : the kernel must be loaded first

It's really not the right time for me since I won't really have time before monday and I will need it next week. But I'm asking for help now because this is beyond my knowledge...

To repair (or in the worst case to access my ssd to backup important files) I have another desktop computer running fedora (not mine, I'm in holidays at a friend's place) and a usb stick. I'm downloading arch now to have a repair live-usb and try to arch chroot...

Thank you in advance if you can give me clues to what happened or how to investigate !

Last edited by cazuza (2025-03-15 19:37:27)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,513

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

boot the install disk, mount everything, arch-chroot in, reinstall all packages that were part of the update.

cazuza

Member

Registered: 2014-05-04

Posts: 34

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

Thank you, I got the install disk ready, I'll do this when I have a moment and report back here.

cazuza

Member

Registered: 2014-05-04

Posts: 34

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

Thank you @Scimmia for making me understand that I shouldn't have panicked, and that is was most likely a failed update than a drive malfunction... I kinda freaked out because I have no idea what caused the updated to fail and the btrfs snapshots to be lost. But I can at least now save all my important data somewhere else (I usually do it on a regular basis but was pretty busy in the last months)...

Just a reminder for next time (for myself or for those who have the same config : encrypted btrfs installed from archinstall). I found all the info in the arch wiki as usual :

1. boot on the arch live_usb

2. load the non-english keymap if needed

# loadkeys YOUR_LANG_CODE

3. connect to the wifi with iwctl

4. create the /mnt directory and find the LUKS volume to unenncrypt :

# mdkir mnt
# lsblk
# blkid | grep "crypto_LUKS"
# cryptsetup luksOpen /dev/sda2 my_drive

5. mount the btrfs subvolumes and the /boot partition

# mount -o subvol=@ /dev/mapper/my_drive /mnt
# mount -o subvol=@home /dev/mapper/my_drive /mnt/home
# mount -o subvol=@pkg /dev/mapper/my_drive /mnt/var/cache/pacman/pkg
# mount -o subvol=@log /dev/mapper/my_drive /mnt/var/log
# mount -o subvol=@.snapshots /dev/sdXY /mnt/.snapshots
# mount /dev/sda1 /mnt/boot

6. chroot and repair

# arch-chroot /mnt
# rm /var/lib/pacman/db.lck
# pacman -Sy linux linux-headers
# pacman -Syu
## invalid/corrupted packages/key > delete all of them
# pacman -Syu 
# exit # exists the arch-chroot
# umount -R /mnt
# reboot

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,513

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

while reinstalling the kernel and headers was enough to get you back into the system, you should really reinstall all packages that were part of the failed transaction. You don't know what post transaction hooks were missed, what kind of database corruption could have occurred, or even file corruption.

ReDress

Member

From: Nairobi

Registered: 2024-11-30

Posts: 125

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

while reinstalling the kernel and headers was enough to get you back into the system, you should really reinstall all packages that were part of the failed transaction.

How is that supposed to work? Reinstalling all packages?

pacman -Syu

The above shouldn't be enough?

cazuza

Member

Registered: 2014-05-04

Posts: 34

Re: Repairing a broken update on an encrypted btrfs drive [SOLVED]

Actually I'm pretty sure 'pacman -Syu' did reinstalled all the failed packages since the failed upgraded wasn't successful.

But feel free to correct me if I'm wrong, I'd be down to know why and how could I retrieve the list of packages to reinstall. There is nothing in /var/log/pacman/log (the log of the day the updated failed stops at "[2025-03-10T22:34:53-0400] [PACMAN] starting full system upgrade")

Anyway thank you for the help