[SOLVED by xyne] Search for packages I have but are NOT installed?

ngoonee · 2009-06-26 00:53:05

Hi all,

I run a couple of computers on Arch (both 64-bit), was just wondering if I could copy the /var/cache/pacman/pkg from one computer to the other, and then run some command to install from the cache all packages that are not currently installed. Basically it would be something like "go through all the packages in /var/cache/pacman/pkg, if any are installed ignore them, if a newer version is installed ignore them, but if no version of this package is installed then install it.

Last edited by ngoonee (2009-06-26 05:17:35)

whoops · 2009-06-26 01:32:24

You can try to write a line that looks a bit like this...:

for file in $(pacman -Qql $(pacman -Qqs)); do ls $file > /dev/null || pacman -S $(pacman -Qqo $file); done;

... but is actually working.

arkham · 2009-06-26 01:50:14

In my opinion, a better approach would be to setup a local repo on one computer[1] and add it to the other computer pacman.conf; after that you will be able to install everything you miss by running:

pacman -Sl custom-repo | awk '{ print $2 }' | xargs pacman -S

In this way, you will download the updates on the main computer, and eveytime you need to update the second computer you'll just have to run "pacman -Syu"

[1] http://wiki.archlinux.org/index.php/Cus … repository

ngoonee · 2009-06-26 01:50:46

Ooh, by file... What would that actually do? My pseudo-code for that looks something like:-

for every file in every package you have installed
EITHER
check if it exists with ls
OR
install it
end

Is that just about it? Isn't there an 'ifexists' of some sort in Bash, I sort of remember writing a script once which checks that, a -z or something.

Xyne · 2009-06-26 02:02:25

I think you're approaching this the wrong way. If you did something like that, all dependencies would be explicitly installed.

If your goal is to install all packages from computer A on computer B, you should get a list of explicitly installed packages from A and use that to install on B after moving the packages over to B's cache:

(on A)

pacman -Qqet >pkglist

(on B)

pacman -S --needed <pkglist

If your main goal is to avoid redownloading packages from the external network for each machine, you could use pkgd to avoid manually copying them.

ngoonee · 2009-06-26 02:42:14

Hi Xyne. I see you're still on your quest for totally mastery of Arch . Thanks for your response, but there's a 'BUT' here...

I really should have made this explicit in OP, but what happened was my laptop died on me, and this desktop I'm using was supposed to be a sort-of mirror of the laptop. Data-wise I had daily syncs, so that was fine, but package wise I use my laptop a whole lot more, for a whole different set of things than just the work I get done in the office (where this desktop is). Upshoot of that is that the installed packages, which started off almost identical, started drifting apart. I've looked at pkgd before, but never really managed to get it working (admittedly with only about 10 minutes of fiddling before getting to other stuff).

So, anyway, I took out my laptop's hard disk and just copied /var/cache/pacman/pkg, so that's all I currently have, no possibility of pacman spitting out a pkglist for me right now (though I hope to get it back today). So, really, all I have are the packages, hence the question here. I'm aware that explicitly installing dependant packages is a bad idea in terms of keeping your system free from cruft (uninstalling main package doesn't uninstall dependencies) but is there any other downside?

Xyne · 2009-06-26 04:29:02

Try this:

#!/usr/bin/perl
use strict;
use warnings;

exit(1) if not @ARGV;

my $cachedir = $ARGV[0];
$cachedir = substr($cachedir,0,-1) if substr($cachedir,-1) eq '/';

opendir(my $dh, $ARGV[0]) or die $!;
my @cache_pkgs = map { s/-[^-]+-[^-]+(?:-i686|-x86_64|-any)?\.pkg\.tar\.gz$//; $_ }
                 grep {/\.pkg\.tar\.gz$/}
                 readdir($dh);
closedir($dh);

my @installed_pkgs = split(/\s*\n\s*/, `pacman -Qq`);

my %cache_pkgs;
$cache_pkgs{$_} = 1 foreach @cache_pkgs;

my %installed_pkgs;
$installed_pkgs{$_} = 1 foreach @installed_pkgs;

foreach (keys(%cache_pkgs))
{
  delete($cache_pkgs{$_}) if exists($installed_pkgs{$_});
}

# uncomment these lines if you just want a list of uninstalled packages
# without filtering dependencies
# @cache_pkgs = keys(%cache_pkgs);
# print "$_\n" foreach @cache_pkgs;
# exit();

my $pkg_files;
foreach my $pkg (keys(%cache_pkgs))
{
  opendir(my $dh, $cachedir) or die $!;
  my @pkg_files = grep {/^\Q$pkg\E/} readdir($dh);
  closedir($dh);

  foreach my $pkg_file (@pkg_files)
  {
    if ( $pkg_file =~ m/^(.+)-([^-]+-[^-]+)(?:-i686|-x86_64|-any)?\.pkg\.tar\.gz$/ )
    {
      $pkg_file = $cachedir . '/' . $pkg_file;
      if (not exists($pkg_files->{$1}))
      {
        $pkg_files->{$1}->{'ver'} = $2;
        $pkg_files->{$1}->{'file'} = $pkg_file;
      }
      else
      {
        my $ver = $pkg_files->{$1}->{'ver'};
        if (&compare_versions($2,$ver))
        {
          $pkg_files->{$1}->{'ver'} = $2;
          $pkg_files->{$1}->{'file'} = $pkg_file;
        }
      }
    }
  }
}

foreach my $pkg (keys(%{$pkg_files}))
{
  my $pkg_file = $pkg_files->{$pkg}->{'file'};
  my $pkginfo = `tar -O -zxf $pkg_file .PKGINFO`;
  foreach my $line (split(/\s*\n\s*/, $pkginfo))
  {
    if ($line =~ /^depend\s+=\s+([^>=\s]+)/)
    {
      delete($cache_pkgs{$1}) if exists($installed_pkgs{$1});
    }
  }
}



@cache_pkgs = keys(%cache_pkgs);
print "$_\n" foreach @cache_pkgs;





sub compare_versions
{
  my ($a,$b) = @_;
  return -1 if not defined($a) and defined($b);
  return 1 if defined($a) and not defined($b);
  return 0 if not defined($a) and not defined($b);

  my ($rel_a,$rel_b) = (0,0);
  if ($a =~ m/(.*)-(.+)$/) {($a,$rel_a)=($1,$2);}
  if ($b =~ m/(.*)-(.+)$/) {($b,$rel_b)=($1,$2);}

  my @a = split(/\./,$a);
  my @b = split(/\./,$b);
  my $len_a = scalar @a;
  my $len_b = scalar @b;
  my $n = ($len_a < $len_b) ? $len_a : $len_b;
  for (my $i=0; $i<$n; $i++)
  {
    my ($int_a,$alph_a,$dash_a) = ($a[$i] =~ m/^(\d*)(.*?)(_.*)?$/);
    my ($int_b,$alph_b,$dash_b) = ($b[$i] =~ m/^(\d*)(.*?)(_.*)?$/);
    $int_a = 0 if $int_a eq '';
    $int_b = 0 if $int_b eq '';
    if ($int_a != $int_b)
    {
      return $int_a <=> $int_b;
    }
    if (length($alph_a) and not length($alph_b))
    {
      return -1;
    }
    if (not length($alph_a) and length($alph_b))
    {
      return 1;
    }
    if ($alph_a ne $alph_b)
    {
      return $alph_a cmp $alph_b;
    }
    return 1 if defined($dash_a) and not defined($dash_b);
    return -1 if not defined($dash_a) and defined($dash_b);
    if (defined($dash_a) and defined($dash_b))
    {
      my $dash_cmp = $dash_a cmp $dash_b;
      return $dash_cmp if $dash_cmp != 0;
    }
  }
  return $len_a <=> $len_b if not $len_a == $len_b;
  return $rel_a <=> $rel_b;
}

Save it as "list_pkgs", make it executable and then run it with

./list_pkgs /var/cache/pacman/pkg

It should spit back a list which contains all packages in the cache which are currently uninstalled, minus any packages which are dependencies of others. To install those packages, combine it with pacman:

pacman -S $(./list_pkgs /var/cache/pacman/pkg)

This doesn't completely solve the issue of explicit versus implicit installation, but I expect that this would be more manageable than installing everything explicitly. I've commented on lines which can be uncommented to get a simple list which ignores dependency relationships.

Btw, the script isn't really as long as it might look. The compare_versions function, which I copied over from another module, nearly doubles the size of the script.

*edited for typos*

Last edited by Xyne (2009-06-26 04:30:12)

ngoonee · 2009-06-26 05:16:59

You, my good sir, are a hero. Epic SOLVED

shining · 2009-06-26 11:12:09

Why does it matter whether they are installed or not?
Reinstalling does not hurt, it just takes more time.
So you could just have done : pacman -U /var/cache/pacman/pkg/*
(btw, install reason of packages are preserved).

If you really don't want to reinstall, I have this suggestion :
pacman -S --needed $(pacman -Qqp /var/cache/pacman/pkg/* | uniq)

However this does not work for the packages not in sync db. So either you create your own db (not too hard), or you just need to exclude the following packages from the previous list :
pacman -Qmqp /var/cache/pacman/pkg/* | uniq

I know your specific problem has been solved, so this is just FYI.

ngoonee · 2009-06-27 00:43:53

As Xyne has mentioned, one thing is that it takes longer (the lesser problem) the other problem is that everything becomes explicitly installed. So, for example, if gnumeric pulled in Gnome components on a KDE system, reinstalling in this manner means that uninstalling gnumeric leaves everything it pulled in, even with -Rsn.

Thank you for your suggestion, I'm bookmarking it

shining · 2009-06-27 10:59:09

ngoonee wrote:

As Xyne has mentioned, one thing is that it takes longer (the lesser problem) the other problem is that everything becomes explicitly installed. So, for example, if gnumeric pulled in Gnome components on a KDE system, reinstalling in this manner means that uninstalling gnumeric leaves everything it pulled in, even with -Rsn.
Thank you for your suggestion, I'm bookmarking it

I got confused by this :

Xyne wrote:

This doesn't completely solve the issue of explicit versus implicit installation, but I expect that this would be more manageable than installing everything explicitly.

By reading the code quickly, I couldn't find anything which tried to solve the issue.
After looking again, I think I found the key part, which only keeps the top-level packages.

foreach my $pkg (keys(%{$pkg_files}))
{
  my $pkg_file = $pkg_files->{$pkg}->{'file'};
  my $pkginfo = `tar -O -zxf $pkg_file .PKGINFO`;
  foreach my $line (split(/\s*\n\s*/, $pkginfo))
  {
    if ($line =~ /^depend\s+=\s+([^>=\s]+)/)
    {
      delete($cache_pkgs{$1}) if exists($installed_pkgs{$1});
    }
  }
}

So yeah, that's probably a better way. Maybe you will have too many packages installed as deps though.
For instance, mplayer is a dep of other packages, but I would always install it explicitly

ngoonee · 2009-06-27 15:58:06

I guess from my point of view (and, it seems, Xyne's), having TOO MANY dependencies is just fine, since when you Rsn you see the list of packages which will also be removed and can manually keep what you want. However, having TOO FEW or NO dependencies means -Rsn just removes one package, and you'd have to go to the ABS PKGBUILD to see what dependencies you should remove, if any.

shining · 2009-06-27 16:13:57

ngoonee wrote:

I guess from my point of view (and, it seems, Xyne's), having TOO MANY dependencies is just fine, since when you Rsn you see the list of packages which will also be removed and can manually keep what you want. However, having TOO FEW or NO dependencies means -Rsn just removes one package, and you'd have to go to the ABS PKGBUILD to see what dependencies you should remove, if any.

You just need to do pacman -Si <pkg> to see the dependencies of a pkg.

And pacman -Qt shows you the packages not required by any others, so any orphan will be in that list.

ngoonee · 2009-06-27 23:19:01

Ah, yes I forgot about the -i parameter. 'Running' dependencies may still be a problem however, if A depends on B, C, D, when you uninstall A if B, C, D have been explicitly installed only A will be uninstalled. So if you have to manually uninstall B, C, and D, what about E and F which are dependencies of C? And so on.

I've never really understood the logic of 'orphaned' packages, surely every 'main' package is an orphan not 'required by others'? For example, if I do pacman -S firefox, isn't firefox not required by others? If explicitly installed packages are excluded from the orphan list, then the above problem(s) come into play...

shining · 2009-06-28 10:43:52

ngoonee wrote:

Ah, yes I forgot about the -i parameter. 'Running' dependencies may still be a problem however, if A depends on B, C, D, when you uninstall A if B, C, D have been explicitly installed only A will be uninstalled. So if you have to manually uninstall B, C, and D, what about E and F which are dependencies of C? And so on.
I've never really understood the logic of 'orphaned' packages, surely every 'main' package is an orphan not 'required by others'? For example, if I do pacman -S firefox, isn't firefox not required by others? If explicitly installed packages are excluded from the orphan list, then the above problem(s) come into play...

Have a look at pacman -Qh or man pacman
pacman -Qt : orphan (include explicit and deps)
pacman -Qtd : orphan and deps
pacman -Qte : orphan and explicit

So when all packages are correctly marked, you can use pacman -Qtd, but when they are not, you can do pacman -Qt and filter manually that list.

ngoonee · 2009-06-28 12:24:52

Ah, I saw the 'filter manually' part. Many thanks for all your assistance, however. What started as a simple question (with a slightly less simple answer) is now a veritable wiki .

Arch Linux

#1 2009-06-26 00:53:05

[SOLVED by xyne] Search for packages I have but are NOT installed?

#2 2009-06-26 01:32:24

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#3 2009-06-26 01:50:14

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#4 2009-06-26 01:50:46

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#5 2009-06-26 02:02:25

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#6 2009-06-26 02:42:14

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#7 2009-06-26 04:29:02

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#8 2009-06-26 05:16:59

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#9 2009-06-26 11:12:09

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#10 2009-06-27 00:43:53

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#11 2009-06-27 10:59:09

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#12 2009-06-27 15:58:06

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#13 2009-06-27 16:13:57

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#14 2009-06-27 23:19:01

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#15 2009-06-28 10:43:52

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

#16 2009-06-28 12:24:52

Re: [SOLVED by xyne] Search for packages I have but are NOT installed?

Board footer