[SOLVED] Get the base package of a split package, and install from ABS

thefiercerabbit · 2016-10-16 18:03:11

I wanted to develop a simple bash script getting package names as argument and retrieving the corresponding PKGBUILD from ABS tree. Then, each package would be created with makepkg.
It sounds very easy to do, but I'm struggling with split packages. These packages share a PKGBUILD file with other packages. Then, their name doesn't correspond to a PKGBUILD file in the ABS tree. Instead, the name of the corresponding base package is used.
From pacman, we do not really see the difference between these packages and the classic ones.

For example, let's consider the bind-tools package. If I look for it with pacman, here is what I get :

Repository      : extra
Name            : bind-tools
Version         : 9.11.0-1
Description     : The ISC DNS tools
Architecture    : x86_64
URL             : http://www.isc.org/software/bind/
Licenses        : custom:ISC
Groups          : None
Provides        : dnsutils=9.11.0
Depends On      : glibc  libcap  libseccomp  libxml2  zlib  krb5  e2fsprogs  openssl  readline  geoip  idnkit  dnssec-anchors  json-c  python
Optional Deps   : python: for python scripts
Conflicts With  : dnsutils
Replaces        : dnsutils  host
Download Size   : 1629.34 KiB
Installed Size  : 5993.00 KiB
Packager        : Sébastien Luttringer <seblu@seblu.net>
Build Date      : Sun 09 Oct 2016 12:22:48 BST
Validated By    : MD5 Sum  SHA-256 Sum  Signature

Since now, the only way I have found to get the base package is to get on the package description URL : https://www.archlinux.org/packages/extr … ind-tools/

Does someone know how to get the base package name ?

Last edited by thefiercerabbit (2016-10-20 10:37:26)

lolilolicon · 2016-10-16 18:15:38

Probably not easy to do, see discussion at https://github.com/falconindy/asp/issues/8

Xyne · 2016-10-16 18:50:48

The current state of accessibility of metadata for the official packages is a mess. It's all available, but most of it needs to be scraped together from different resources (sync dbs, online json api, abs web interface), depending on what metadata you need. I have written modules in Python for Powerpill, Bauerbill and Pacboy to manage this conveniently.

With the current release of python3-xcpf, you can use this script to retrieve the pkgbase of a package:

#!/usr/bin/env python3

import sys
import XCPF.common

opi = XCPF.common.OfficialPkgInfo(resolve_pkgbases=True)
for pkgname in sys.argv[1:]:
  pkginfo = opi.pkginfo_by_pkgname(pkgname)
  print(pkginfo['pkgbase'])

Usage:

$ ./test.py  bind-tools
bind

Incidentally, both Bauerbill and Pacboy can be used to do full system upgrades with automatic rebuilds of the official packages (either rebuilding everything or a selected subset).

Trilby · 2016-10-16 18:54:21

I don't quite follow how you say it's not easy to do and you link to a discussion showing exactly how to do it. In this case:

curl -s  https://www.archlinux.org/packages/extra/x86_64/bind-tools/json/ | sed 's/.*"pkgbase": "\([^"]*\).*/\1/'

Xyne · 2016-10-16 18:58:50

Trilby wrote:

I don't quite follow how you say it's not easy to do and you link to a discussion showing exactly how to do it. In this case:
curl -s  https://www.archlinux.org/packages/extra/x86_64/bind-tools/json/ | sed 's/.*"pkgbase": "\([^"]*\).*/\1/'

For use in a script, that needs to be wrapped in a function to determine which repo the package is in and, if it's in community, change "packages" to "community".

thefiercerabbit · 2016-10-16 19:10:52

Indeed the curl query seems to be as light as it could be. And since my script is pure bash, it's kind of cleaner than to use a Python callback.
However, if this information could be displayed via pacman it would be way easier. I'm quite sure it stores it since it knows which package it needs to build...

Thanks to all of you.

Trilby · 2016-10-16 20:25:34

thefiercerabbit wrote:

I'm quite sure it stores it since it knows which package it needs to build...

pacman doesn't build the packages.

thefiercerabbit · 2016-10-16 20:27:50

That's quite true... Well, it could know the base package even though.
Thank you for the correction.

Trilby · 2016-10-16 20:34:12

It turns our your suspicion was still right: the "base" is in the sync database, perhaps pulling it from there would be easier. To confirm this you can uncompress /var/lib/pacman/sync/extra.db and there is a folder for each package with a .desc file which includes the %BASE% field.

lolilolicon · 2016-10-16 20:38:32

@Trilby pardon the wording, by which I meant "no straightforward way".
I'm pretty sure Xyne has covered all the ground here in his library... but as my apology, I'll present a solution here,

#!/bin/bash
# Download PKGBUILD & source files by pkgname

arch=$(uname -m) || exit 42

get_meta() {
    curl -s -G 'https://www.archlinux.org/packages/search/json/' \
        -d "arch=any" -d "arch=$arch" -d "name=$pkgname" |
    jq -r '.results[]|.repo, .pkgbase' | sed 'N;s/\n/ /;q'
}

download() {
    read -r repo pkgbase || return
    [[ $repo == community ]] || repo=packages
    local host='https://git.archlinux.org'
    curl -s "$host/svntogit/$repo.git/tree/trunk?h=packages/$pkgbase" |
    grep -Po "(?<=<a class='button' href=')[^']+(?='>plain</a>)" | sed "s#^#$host#" |
    wget -P "$pkgbase" -i- --content-disposition
}

for pkgname; do
    printf 'package: %s => ' "$pkgname" >&2
    get_meta | tee /dev/stderr | download
done

Last edited by lolilolicon (2016-10-16 20:47:23)

Trilby · 2016-10-16 20:45:04

Here's another one liner with no need to curl from the web, just use the local sync db:

zcat /var/lib/pacman/sync/extra.db | awk '/%NAME%/ { getline; pkg=$1 } /%BASE%/ { getline; if (pkg == "bind-tools") print $1; }'

You could easily zcat /var/lib/pacman/sync/*.db into the same awk command, so you wouldn't even need to know the repo before hand.

So for a more general use in a script:

pkg=bind-tools
base=$(zcat /var/lib/pacman/sync/*.db | awk '/%NAME%/ { getline; pkg=$1 } /%BASE%/ { getline; if (pkg == "'$pkg'") print $1; }')

eschwartz · 2016-10-20 05:38:05

pacinfo (from pacutils) prints the pkgbase as well.

I think at some point there was actually some sort of intention to have pacman do that natively -- ISTR that %BASE% was only added to the desc file recently, as a preliminary step towards making the pkgbase more discoverable?

Alad · 2016-10-20 06:18:38

expac -S '%e' bind-tools
bind

lolilolicon · 2016-10-20 07:31:25

TIL thx m8.

thefiercerabbit · 2016-10-20 10:47:06

Alad wrote:

expac -S '%e' bind-tools
bind

Well, definitely the easiest solution, but maybe not the fastest...
On my system, the zcat+awk solution takes less than 0.100s (if in a for loop, otherwise around 0.300s -- someone can explain this ?).
However, the expac solution takes almost 0.500s.

Depending on what you want to achieve, you may have to choose between simplicity and performance.

Trilby · 2016-10-20 11:39:40

I suspect expac will be faster than 0.5s if you run it a second time. But in several repetitions here, the zcat/awk command still remains faster than expac.

This is almost certainly due to expac being a much more powerful and flexible tool. It has some startup overhead, and the logic it uses to run doesn't use the same simplifying assumptions. The zcat / awk line is tailored to exactly this task and would be completely useless for other tasks. With awk and zcat, no database data is read into memory, files are just dumped to a pipe, and lines are read one at a time and discarded immediately after being read.

The awk solution could be even faster by adding an exit after printing the pkgbase:

zcat /var/lib/pacman/sync/*.db | awk '/%NAME%/ { getline; pkg=$1 } /%BASE%/ { getline; if (pkg == "'$pkg'") { print $1; exit; }}'

But another important difference is that expac is maintained. And it is maintained by an active arch developer. One of the simplfying assumptions behind my zcat/awk approach is that the database format will always remain consistent across packages and over time. It might change. If the database format changes, expac will get a revision to work with the new format. Don't come back to me looking for a revision of the awk command

lolilolicon · 2016-10-20 11:41:48

Well expac can search multiple targets at once, which should reduce the relative overhead.

Trilby · 2016-10-20 11:51:46

lilicon, so can awk, just check if "pkg" is in the list of packages:

pkgs='%bind-tools%linux%vim%'
zcat /var/lib/pacman/sync/*.db | awk '/%NAME%/ { getline; pkg="%" $1 "%";} /%BASE%/ { getline; if (index("'$pkgs'",pkg) > 0) print $1; }'

I thought I found a good reason to prefer expac: some packages are missing a %BASE% field, openbox and namcap are examples. But it turns out expac fails with these too. Perhaps it's the entire community repo that doesn't (yet) have the %BASE% field. In which case none of the above solutions would be viable.

EDIT: hmm, the community db seems to have %BASE% for some packages, but not others, perhaps only for those where the pkgbase doesn't match the pkgname (speculation).

EDIT2: the core repo also is missing some %BASE% entries - eg, pacman.

Unfortunately it would seem scraping the web interface would currnently be the only really viable solution. Hopefully the %BASE% field will continue to be added to the databases.

lolilolicon · 2016-10-20 12:21:10

trilbibi, I guess Eschwartz had a point with the word "preliminary"
Also, "probably not easy to do" turns out to be true, right? I SAID SO, DUDE.

Trilby · 2016-10-20 12:22:45

*hangs head in shame*

You were indeed right.

*takes hat off of parrot - eats hat*

Alad · 2016-10-20 12:22:53

Trilby wrote:

I thought I found a good reason to prefer expac: some packages are missing a %BASE% field, openbox and namcap are examples. But it turns out expac fails with these too. Perhaps it's the entire community repo that doesn't (yet) have the %BASE% field. In which case none of the above solutions would be viable.

expac -Sv '%n\t%e' namcap bind-tools | awk -F'\t' '{if ($2 == "(null)") print $1; else print $2;}'
namcap
bind

Last edited by Alad (2016-10-20 12:23:16)

Trilby · 2016-10-20 12:28:28

Ha, I did give up to soon:

pkgs='%linux%pacman%bind-tools%'
zcat /var/lib/pacman/sync/*.db | awk '/%NAME%/ { getline; if (index("'$pkgs'",$1) > 0) { base=$1; while (getline) { if ($1 == "%NAME%") break; if ($1 == "%BASE%") { getline; b
ase=$1; } } print base; }}'

Still no need for expac

*coughs up hat, hands it to lolilolicon*

lolilolicon · 2016-10-20 13:24:30

Aaarrgghhh, so much saliva...

Good job but *cough* that's not very *cough* straight *cough* forward *cough*

I need to go to the toilet now.

eschwartz · 2016-10-20 22:10:41

Yeah, not much need for a pkgbase field in the package's .PKGINFO if it isn't a split package... especially because it isn't actually used by anything except peoples' eyeballs, so it's not like it makes a difference.

If people cared a lot more about pkgbases then the tools which make it easier to read this information could be taught how to fallback on a pkgname if the pkgbase doesn't exist. Which I consider a whole lot more useful than being told the pkgbase is "(null)".
Of course, I don't really care all that much myself.

Arch Linux

#1 2016-10-16 18:03:11

[SOLVED] Get the base package of a split package, and install from ABS

#2 2016-10-16 18:15:38

Re: [SOLVED] Get the base package of a split package, and install from ABS

#3 2016-10-16 18:50:48

Re: [SOLVED] Get the base package of a split package, and install from ABS

#4 2016-10-16 18:54:21

Re: [SOLVED] Get the base package of a split package, and install from ABS

#5 2016-10-16 18:58:50

Re: [SOLVED] Get the base package of a split package, and install from ABS

#6 2016-10-16 19:10:52

Re: [SOLVED] Get the base package of a split package, and install from ABS

#7 2016-10-16 20:25:34

Re: [SOLVED] Get the base package of a split package, and install from ABS

#8 2016-10-16 20:27:50

Re: [SOLVED] Get the base package of a split package, and install from ABS

#9 2016-10-16 20:34:12

Re: [SOLVED] Get the base package of a split package, and install from ABS

#10 2016-10-16 20:38:32

Re: [SOLVED] Get the base package of a split package, and install from ABS

#11 2016-10-16 20:45:04

Re: [SOLVED] Get the base package of a split package, and install from ABS

#12 2016-10-20 05:38:05

Re: [SOLVED] Get the base package of a split package, and install from ABS

#13 2016-10-20 06:18:38

Re: [SOLVED] Get the base package of a split package, and install from ABS

#14 2016-10-20 07:31:25

Re: [SOLVED] Get the base package of a split package, and install from ABS

#15 2016-10-20 10:47:06

Re: [SOLVED] Get the base package of a split package, and install from ABS

#16 2016-10-20 11:39:40

Re: [SOLVED] Get the base package of a split package, and install from ABS

#17 2016-10-20 11:41:48

Re: [SOLVED] Get the base package of a split package, and install from ABS

#18 2016-10-20 11:51:46

Re: [SOLVED] Get the base package of a split package, and install from ABS

#19 2016-10-20 12:21:10

Re: [SOLVED] Get the base package of a split package, and install from ABS

#20 2016-10-20 12:22:45

Re: [SOLVED] Get the base package of a split package, and install from ABS

#21 2016-10-20 12:22:53

Re: [SOLVED] Get the base package of a split package, and install from ABS

#22 2016-10-20 12:28:28

Re: [SOLVED] Get the base package of a split package, and install from ABS

#23 2016-10-20 13:24:30

Re: [SOLVED] Get the base package of a split package, and install from ABS

#24 2016-10-20 22:10:41

Re: [SOLVED] Get the base package of a split package, and install from ABS

Board footer