You are not logged in.

#1 2014-12-24 21:11:30

gsingh93
Member
Registered: 2013-07-19
Posts: 96

Quick way to see if a list of packages exist in the repos or the AUR

For a script I'm writing, I'd like to check if a list of packages actually exists in the enabled repos or in the AUR. Currently, I'm looping through all packages in the list and runing `pacman -Ss package_name`, but there are two problems with this. The first is  pacman can return multiple packages here, whereas I only want it to return the package that exactly matches `package_name`. I can fix this by using grep to find an exact match for the name. The second problem is that this method is very slow, but I want this to take no more than a second for about 100 packages. Is there a faster way I can figure out which packages actually exist?

Offline

#2 2014-12-24 21:19:36

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 16,280
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

gsingh93 wrote:

The first is  pacman can return multiple packages here, whereas I only want it to return the package that exactly matches `package_name`. I can fix this by using grep to find an exact match for the name.

A better solution for that problem (as stated) is simply to use a proper regex.  The following returns only the single exact match:

pacman -Ss ^firefox$
gsingh93 wrote:

The second problem is that this method is very slow, but I want this to take no more than a second for about 100 packages. Is there a faster way I can figure out which packages actually exist?

Then a much better approach would be something like the following:

comm -23 your_list_file <(pacman -Ssq | sort)

EDIT: I just tested the above command using a random selection of 100 package names from the repos with 40 of the modified (so as not to match) and then I ran the command through `time`:

real	0m0.283s
user	0m0.227s
sys	0m0.030s

If you want this to work for the AUR as well, we'd just need a list of all package names in the AUR.  This should be easy to get from the web-interface (standby I'll see if I can get it).  Then basically you'd do the following:
1) pacman -Ssq > pkglist
2) cmd_to_get_aur_names >> pkglist
3) comm -23 <(sort your_list_file) <(sort pkglist)

EDIT 2: I don't know of any efficient way to get a list of all AUR packages.  The best idea I had was to "simply" wget and parse this page: http://pkgbuild.com/git/aur-mirror.git/tree/ but at 26MB that takes well more than a second to download even on a good connection.  I suspect it will be far more efficient to get the `comm` output from above for a list of packages not in the repos, then for each of those search the aur as a fallback (with something like cower).

Last edited by Trilby (2014-12-24 21:38:08)

Offline

#3 2014-12-24 21:52:52

gsingh93
Member
Registered: 2013-07-19
Posts: 96

Re: Quick way to see if a list of packages exist in the repos or the AUR

Thanks for the response. I saw the regex feature in the man page, but it's not working for me. That command you posted gives no results.

In the year I've been using Arch, I never thought to run `pacman -Ss` without any packages. I never thought it'd return everything. That'll work great, thanks!

If anyone thinks of an efficient way to get a list of AUR packages, let me know. I'm going to use the fallback method for now.

Offline

#4 2014-12-24 21:57:25

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 21,805
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

gsingh93 wrote:

If anyone thinks of an efficient way to get a list of AUR packages, let me know. I'm going to use the fallback method for now.

https://wiki.archlinux.org/index.php/AurJson


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5 2014-12-24 22:25:52

karol
Archivist
Registered: 2009-05-06
Posts: 25,438

Re: Quick way to see if a list of packages exist in the repos or the AUR

Trilby wrote:

The following returns only the single exact match:

pacman -Ss ^firefox$

Nope:

$ pacman -Ss ^firefox$
extra/firefox 34.0.5-1 [installed]
    Standalone web browser from mozilla.org
archlinuxcn/firefox-kde-opensuse 34.0.5-1
    Standalone web browser from mozilla.org with OpenSUSE patch, integrate better with KDE
$ pacman -Si firefox-kde-opensuse | grep firefox$
Conflicts With : firefox

'pacman -Ss' searches all fields, that's why you have to weed out false positives with e.g. grep or use expac:

$ expac "%n" -S firefox
firefox

Edit: For AUR, try https://aur.archlinux.org/packages/aurlist/

Last edited by karol (2014-12-24 22:33:34)

Offline

#6 2014-12-24 22:34:53

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,071
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

I've added a script named pkg-simple_search to my pkg_scripts package. It will search for exact matches in the repos and the AUR. You can use the options to either return a simple list of the packages found or a list of packages along with their locations.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#7 2014-12-24 22:38:33

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 21,805
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

Xyne wrote:

I've added a script named pkg-simple_search to my pkg_scripts package. It will search for exact matches in the repos and the AUR. You can use the options to either return a simple list of the packages found or a list of packages along with their locations.

OT: Is there anywhere these are hosted where I can browse the source?


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#8 2014-12-24 22:40:40

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,071
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

jasonwryan wrote:

OT: Is there anywhere these are hosted where I can browse the source?

You can download the archives here: http://xyne.archlinux.ca/projects/pkg_scripts/src/


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#9 2014-12-24 22:43:44

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 21,805
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

Xyne wrote:
jasonwryan wrote:

OT: Is there anywhere these are hosted where I can browse the source?

You can download the archives here: http://xyne.archlinux.ca/projects/pkg_scripts/src/

Ta: I was being lazy and hoping for a web type thing smile


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#10 2014-12-24 22:52:07

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,071
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

jasonwryan wrote:
Xyne wrote:
jasonwryan wrote:

OT: Is there anywhere these are hosted where I can browse the source?

You can download the archives here: http://xyne.archlinux.ca/projects/pkg_scripts/src/

Ta: I was being lazy and hoping for a web type thing smile

Yeah, I figured. There is no web type thing because I'm lazy too. wink


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#11 2014-12-24 23:33:35

karol
Archivist
Registered: 2009-05-06
Posts: 25,438

Re: Quick way to see if a list of packages exist in the repos or the AUR

Xyne wrote:

I've added a script named pkg-simple_search to my pkg_scripts package.

Would it be possible to mention such changes in the news on your website and the feed?

I don't think xyne spelled backwards says 'lazy' ;P

Offline

#12 2014-12-25 00:27:26

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,071
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

karol wrote:
Xyne wrote:

I've added a script named pkg-simple_search to my pkg_scripts package.

Would it be possible to mention such changes in the news on your website and the feed?

I don't think xyne spelled backwards says 'lazy' ;P

http://xyne.archlinux.ca/news/#update smile

And with that the derailment of this thread is complete.

So, who's tried poutine? tongue


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#13 2015-01-04 21:47:13

helix
Member
Registered: 2013-02-17
Posts: 166

Re: Quick way to see if a list of packages exist in the repos or the AUR

Another way:

pacman -Ss package_name | grep -v '    ' | gawk -F '/' '{print $2}' | gawk '{print $1}' > result
grep "^package_name$" result
rm -f result

Last edited by helix (2015-01-04 21:49:17)

Offline

#14 2015-01-05 05:56:41

karol
Archivist
Registered: 2009-05-06
Posts: 25,438

Re: Quick way to see if a list of packages exist in the repos or the AUR

I have no idea why are you doing it this way, helix ...
Why not just

pacman -Ssq firefox | grep "^firefox$"

?

Offline

#15 2015-01-05 06:01:28

helix
Member
Registered: 2013-02-17
Posts: 166

Re: Quick way to see if a list of packages exist in the repos or the AUR

The original line included on a script of my own (debtap) uses package-query instead of pacman to search in AUR as well

Last edited by helix (2015-01-05 06:02:07)

Offline

#16 2015-01-06 18:02:04

Herk
Member
Registered: 2013-01-20
Posts: 4

Re: Quick way to see if a list of packages exist in the repos or the AUR

gsingh93 wrote:

If anyone thinks of an efficient way to get a list of AUR packages, let me know. I'm going to use the fallback method for now.

Maybe not needed anymore, but a complete list of AUR packages is available at https://aur.archlinux.org/packages.gz

Seems to be regenerated every hour.

Offline

#17 2015-01-06 19:16:14

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 16,280
Website

Re: Quick way to see if a list of packages exist in the repos or the AUR

Nice Herk.  Adding that to my previous post works well:

comm -23 <(sort -u needed_pkg_list) <(sort -u <(wget -q -O - https://aur.archlinux.org/packages.gz | gunzip) <(pacman -Ssq))

Time says:

real	0m1.569s
user	0m0.023s
sys	0m0.013s

Of course this time will be highly dependent on your internet connection - I'm on a fairly mediocre connection at the moment.  The plus side is it will not be affected noticably by the size of the "needed_pkg_list" file.

Offline

Board footer

Powered by FluxBB