Lost pacman, checking its dependencies

PMR021 · 2020-02-05 16:36:21

I wrote this post just in case the same thing happens to you; my own experience of using Arch suggests that you would be very unlucky if it did ever happen to you.

I had a curious pacman crash. I was doing a normal pacman -Syu but the whole run crashed near the end of upgrading several hundred packages, and in particular it seemed to be upgrading pacman itself when it happened. But I could not resume the run or check for package consistency because pacman itself was now missing.

I tried rebooting, but discovered that vmlinuz-linux and the initramfs files were missing from the boot directory (as was linux.conf, since I use rEFInd). So I booted up a recent installation
DVD and after mounting the relevant partitions and doing arch-chroot /mnt, I tried running mkinitcpio, only to find that I could not because /etc/mkinitcpio.d/linux.preset was also missing.

I am unsure why the crash happened at all: I ran fsck -f on the relevant unmounted partitions but they were all clean; my disk is an SSD but the partitions were nowhere near full. I have been using Arch for years, and have been updating regularly without problems. I don't have any obvious explanation, but I do still trust pacman: it's an outstandingly good piece of work!

Advice on the web suggested that one could re-install pacman without using pacman, by first checking all its dependencies and then using tar with suitable arguments -- see the sticky posting at the top of the 'Pacman and Package Upgrade Issues' section of the Arch forums. Fortunately I have a laptop that also runs Arch and which I had very recently updated, so I could generate a list of pacman's dependencies: ninety of them! So I created the following small perl script to check all these dependencies, after transferring a list of them to my broken system via USB stick:

#!/bin/perl -w
#
# This reads a file of pkgfiles, each in full-pathname form,
# and for each one, checks that the files in the pkg file exist
# and have the right size -- only a BASIC check. Useful only
# if you have lost pacman. Pkg names typically end with .xz or .zst

use strict;

my $infile = shift or die("need a file that lists pkgs to check\n");
my ( $pkgfile, $fh, $fg, $filename, $pkgsize );
my @fields;
my $realsize;

open($fg, "<", $infile) or die("could not read $infile\n");
while( <$fg> ) {
  chomp;
  next if /^\s*$/; # skip empty/whitespace lines
  $pkgfile = $_;
  printf("==== checking %s\n", $pkgfile);
  open($fh, "-|", "tar -tvf $pkgfile --exclude .BUILDINFO --exclude .MTREE --exclude .PKGINFO --exclude .INSTALL --exclude .Changelog")
     or die("quitting: cannot run tar on $pkgfile\n"); 
  while( <$fh> ) {
    chomp;
    @fields = split;
    $tarsize = $fields[2]; # tar -v: third field is the size
    next if $#fields > 5;  # probably symbolic link: ignore it
    next if /\/$/;         # a directory: ignore it
    $filename = "/$fields[5]"; # put a / in front of the name
    if( ! -e $filename) {
      printf(" .... does not exist: %s\n", $filename);
      next;
    }
    $realsize = (stat($filename))[7];
    if($pkgsize != $realsize) {
      printf(" size mismatch: pkg has %d, filesystem has %d for %s\n", $pkgsize, $realsize, $filename);
    }
  }
  close($fh);
}
close($fg);

It turned out that most of the files in the pacman package were missing, and otherwise the only dependency files with unexpected sizes were ones that you might expect, such as /etc/passwd. However, it was disturbing that essential boot files had also gone missing, so rather than check everything on my system I opted to do a full re-install: fortunately, besides a backup I also had /home on a separate partition. It took some time but I'm now back iup and running.

V1del · 2020-02-05 16:50:54

Let me guess, you symlinked your /var/cache/pacman directory? Don't do that, that directory is a part of the pacman package. If you replace this with a symlink, pacman will replace that symlink with the directory it knows about, by consequence your cache directory is empty and it won't find files to continue with anymore.

vertex · 2020-03-02 20:02:13

I just ran into the same problem and yes, I've been stupid enough to not look at the pacman.conf to realize I could just change the cache directory there, but went ahead and symlinked /var/cache/pacman/ to my large HDD raid volume, since my system SSD is samewhat limited in size while I still enjoy having access to past versions of packages in the cache.

Now I changed the Cachedir in pacman.conf and tried

tar -xvpf pacman-5.1.3.-1-x86_64.pkg.tar.xz -C /

in order to restore it.

Didn't help tho - trying to pacman -S pacman yields

looking for conflicting packages...
error: could not open file /var/lib/pacman/local/pacman-5.2.1-4/desc: no such file or directory
warning: could not fully load metadata for package pacman-5.2.1-4
error: failed to prepare transaction (invalid or corrupted package)

Plus I'm running zfs for my raid.. a pacman -Syu yields the usual "installing linux X breaks dependency Y required by zfs-linux". Usually I would just wait for this to resolve on its own (once zfs-linux catches on), but right now that's a circumstance I think I should mention.

I'm running arch since August 2018 and so far (even if this is my first posting) I could always resolve my issues on my own (huge "Thanks!" and "Hello!" to this forum) - but right now I seem to have messed up and don't really know how to resolve the situation. Am I screwed?

eschwartz · 2020-03-03 01:16:42

Well, if the pacman installed database got broken, then those errors will just keep on happening until you actively fix it. Yes, this takes some hacking around, so it's not surprising that it isn't something you were able to solve the issue on your own -- it is a weird case, and you're not supposed to edit databases manually, but... that's what you need to do here, fortunately it should be fairly simple.

So...

run 'rm -rf /var/lib/pacman/local/pacman-5.2.1-4/' as root, since you already know the database is corrupted here. Now pacman will think the pacman package is not installed, rather than thinking it is installed with broken metadata.

You've verified the program itself is on disk and works due to tar -xf, even if it aborts with database errors. So, then, you can install pacman using: pacman -S pacman --overwrite "*"

(This will install that one package, and tell it to ignore untracked files which already exist, since you know they are there and they are broken. Do not worry about your /etc/pacman.conf, if it is different, the package will not overwrite it but will create a .pacnew file.)

At this point, you should now have a pacman package that is definitely properly installed, and which is tracked in the pacman installation database as such.

vertex · 2020-03-03 02:33:45

Thanks. I'll backup and try that. Tho I'd like to understand how the database got corrupted at this point: https://photos.app.goo.gl/N9ZU29iQp2ui1L6d6 (call me crazy, but I like to capture that process, since the warnings and notices always rush by so fast that I can't keep up)

It kinda just said "oops, file gone" and "Errors occured, no packages were upgraded." - so what did pacman do to itself prior trying to install from that package, apart from obviously replacing my symlink with an empty directory? Could I somehow pick the process up again at this point, now that the cache is back?

If I delete that and rebuild the database, does that imply pacman won't know any of the packages that are installed and I should re-install everything in order to rebuild the database and not encounter more dependency problems in the future?

Again, thanks alot for the assistance. Just wanna make sure I know what I'm getting into

eschwartz · 2020-03-03 03:01:48

vertex wrote:

Thanks. I'll backup and try that. Tho I'd like to understand how the database got corrupted at this point: https://photos.app.goo.gl/N9ZU29iQp2ui1L6d6 (call me crazy, but I like to capture that process, since the warnings and notices always rush by so fast that I can't keep up)
It kinda just said "oops, file gone" and "Errors occured, no packages were upgraded." - so what did pacman do to itself prior trying to install from that package, apart from obviously replacing my symlink with an empty directory? Could I somehow pick the process up again at this point, now that the cache is back?

It removed the old version of the package in preparation for installing the new version in its place. That includes removing the database entry so it can insert a new one.

Then, partway through this process, it turned out that the filesystem entry /var/cache/pacman/pkg (owned by the pacman package, which was the current transaction target) got removed from the filesystem and reinstalled as part of the new package... except no, wait, it could not install the new version.

This is because in order to extract the new version, it needed to read the file /var/cache/pacman/pkg/pacman-5.2.1-4-x86_64.pkg.tar.zst but it could not find it anymore. At that point, pacman crashed due to "something completely unexpected just happened".

At that point, it was left in an inconsistent state, and the single package which it was in the middle of transacting, was borked and who even knows what might have happened.

As you can see, at that point, at least the file /var/lib/pacman/local/pacman-5.2.1-4/desc had been unlinked and not yet recreated.

If I delete that and rebuild the database, does that imply pacman won't know any of the packages that are installed and I should re-install everything in order to rebuild the database and not encounter more dependency problems in the future?
Again, thanks alot for the assistance. Just wanna make sure I know what I'm getting into

"any of the packages", no. "the pacman package, specifically", yes. That's why I said "Now pacman will think the pacman package is not installed, rather than thinking it is installed with broken metadata."

To put this into context, the database structure of pacman is a flat-file database. The directory "/var/lib/pacman/local/" is the database, and each subdirectory of the form ${pkgname}-${fullpkgver} represents a package. Each subdirectory then contains the desc file, with metadata about the package, the "files" file, which lists the files the package owns (and backup files + checksums for their unmodified state), and the "mtree" file which contains some data used for verifying pacman -Qkk.

So, it's a very easily hand-modifiable database. You could open a desc file in your text editor and make pretend that a package has whatever dependencies, groups, etc. you want. (I don't advise doing this, obviously.) Most importantly, you can remove a single package from the database simply by removing its directory. This will leave its files as untracked files, which is typically bad...

In this case, you know this one single database entry is already corrupted, so you're removing it from the database manually, then using pacman -S --overwrite "*" to rebuild it. This will reset the install reason to "explicitly installed", but that is the only information that gets lost (and it was already lost).

vertex · 2020-03-03 09:04:57

Awesome explanation! Really helps me to know/understand what I'm doing

Very interesting insight into package management. Until now I thought it's a merge/update and didn't expect simple delete/write transactions. Makes sense tho, I guess.

I proceeded and at first ran into some issues, since I extracted the v5.1 package yesterday (thought to restore the previous state). After extracting v5.2 the pacman installation went through ok. Guess now I only have to compare the contents of both versions to find and remove any orphans I created by extracting one after the other.

Thanks alot for taking the time. Cheers!

/edit/ps
"pacman -Syu" (with IgnorePkg linux, since zfs...) went through ok too. Nice!

Last edited by vertex (2020-03-03 09:18:35)

Arch Linux

#1 2020-02-05 16:36:21

Lost pacman, checking its dependencies

#2 2020-02-05 16:50:54

Re: Lost pacman, checking its dependencies

#3 2020-03-02 20:02:13

Re: Lost pacman, checking its dependencies

#4 2020-03-03 01:16:42

Re: Lost pacman, checking its dependencies

#5 2020-03-03 02:33:45

Re: Lost pacman, checking its dependencies

#6 2020-03-03 03:01:48

Re: Lost pacman, checking its dependencies

#7 2020-03-03 09:04:57

Re: Lost pacman, checking its dependencies

Board footer