Announcing some additional pacman utils + replacement for ABS

kata198 · 2017-06-08 02:33:28

Hey yall,

I wanted to share some programs/scripts I've been using and have recently made public, especially since abs is dead.

These deal with what I feel are the few deficiencies in the pacman/archlinux environment, and allow very simple abs-like, intuitive, fully commandline driven procedures and patterns for us power users that like to build packages.

The Repos / Packages

This section will cover the git repos for scripts and the AUR packages I've made available for the tools described below.

pacman-utils git repo
https://github.com/kata198/pacman-utils

This provides the tools (which are also available as a package, I'll cover in a sec), and also provides "extras" which are not found in the package.

The "extras" are makepkg.conf additions which are intended to simplify package building for native, lto, and most importantly: PGO (profile-guided optimization).

There is a guide at the bottom of that page on how easily do PGO on packages with these additions. On average, I get about 20-50% performance increase doing native + lto + PGO vs stock build.

https://github.com/kata198/pacman-utils-data

Data, where I generate a "providesDB" (covered more later)

pacman-utils packages

https://aur.archlinux.org/packages/pacman-utils

^ Package for pacman-utils

https://aur.archlinux.org/packages/pacman-utils-data

^ Package for pacman-utils-data

Again note, that the "extras" found in pacman-utils are NOT part of this package, until I have some more time to implement a /etc/makepkg.conf.d which can inherit additional functions.

The Tools

whatprovides_upstream

The biggest pain for me on archlinux (which I've been using for over a decade) is that there's no "whatprovides" functionality, as with yum.

For example:

yum --whatprovides '*/libc.so.6

has no pacman equivalent. You can query against currently-installed packages, but not against available (uninstalled) packages.

whatprovides_upstream adds this functionality (requires pacman-utils-data).

You can query for a single file:

[myuser@myhostname ~]$ whatprovides_upstream '/usr/lib/libgvpr.so'
graphviz

Otherwise, you have to resort to the web to figure out what "gvpr.so" is and to which package it belongs..

Or you can query with glob-expressions (* and ?), which is especially useful for finding a random executable or lib.

In glob mode, you will get a list of packages (sorted), followed by a tab character, followed by the file that matched.

Example:

[myuser@myhostname ~]$ whatprovides_upstream '*/libssl.so*'
openssl /usr/lib/libssl.so
openssl /usr/lib/libssl.so.1.1
openstack-guest-agents-unix     /usr/share/nova-agent/1.39.1/lib/libssl.so.1.0.0
vagrant-substrate       /opt/vagrant/embedded/lib/libssl.so
vagrant-substrate       /opt/vagrant/embedded/lib/libssl.so.1.0.0

I update the "providesDB" which feeds whatprovides_upstream via pacman-utils-data quite often. It would be "better" if mirrors themselves generated / synced it, but that's outside my scope.

You can generate your own, for example if you have an offline mirror (like at a previous employer, I forked archlinux and managed an internal repo). You can find that info out in the git README.

whatprovides

This checks what provides a file on the local system, just for the sake of a common interface (rather than pacman -Sl | grep). Does not require any external DB.

pacman-mirrorlist-optimize

Yet another pacman mirrorlist optimization script. There was a web-based one a while back, which seemed to use ping-times to the various server (which measures latency, NOT throughput).

This tool scans your /etc/pacman.d/mirrorlist, by default using all mirrors listed (even commented), but can be changed ( see --help ).

It downloads a package from each of them, and measures the real throughput between you and that server, and sorts according to such. Defaults to glibc, but you can pick a different package with a parameter. Again, see --help.

It will display a ratio next to each server (in a comment), to let you know the relative speed to the fastest.

abs2

Since abs and mass-updates have been removed, this tool takes a gentler approach. It will download/update info on a single package (provided as an argument), and output the path to where it put it (/var/abs2).

First time run as root, after that anyone in "users" can do it (as oppose to old abs, which default design and usage was to always be run as root)

Not very useful on its own, but provides the backing for the next two tools:

archsrc-getpkg

Gets the latest PKGBUILD etc info for a provided package (via argument), and puts it in current directory ( I recommend using /usr/src/arch for this, other tools like that directory as well ).

If a directory already exists by the given name, it will be renamed to "${NAME}.bak". If "${NAME}.bak" already exists, you will be prompted to either have it remove that dir or abort.

This tool obsoletes the old pattern of:

sudo abs;
cd /usr/src/arch;
rm -Rf "${PKGNAME}.bak"
[ -d "${PKGNAME}" ] && mv "${PKGNAME}" "${PKGNAME}.bak"
cp -R /var/abs/REPO/PACKAGE .

Into a much simpler pattern:

cd /usr/src/arch
archsrc-getpkg PKGNAME

archsrc-buildpkg

If you use my "extras" and set the DEFAULT_USE_CFLAGS to your desired profile, or just use the stock and modify CFLAGS/CXXFLAGS therein to your liking, and don't want to do PGO, this is the tool for you.

It's a one-stop shop to build a package for you.

You run it as your user, and it will sudo to root.
This is to ensure /usr/src/arch is created, 2775, and group=users, and to install the package after build.

Then it fetches the package build info into /usr/src/arch, builds package, and installs it. On any failure it stops and you see why.

aur-getpkg

Same as archsrc-getpkg, except fetches the latest snapshot of an AUR package into current dir

aur-buildpkg

Same as archsrc-buildpkg, except builds a package from AUR instead of standard arch repos

Some utils for lazy folks

I included a few in pacman-utils, but I have a lot more..

mkgcdatar

For use with my PGO stuff, see the README on https://github.com/kata198/pacman-utils for more info

getpkgs

Gets the names of packages, with no args in current directory, otherwise in a directory, or unrolls a glob and only prints packages.

installpackage

Installs packages matching a pattern. Very useful if you have several versions of the same package, or similar names that tab doesn't make things quick, you can do like:

installpackage 6-2

To install all packages in current dir with "6-2" somewhere in it.

Anyway, please check them out if you're into building your own packages. It's good stuff!

Last edited by kata198 (2017-06-08 03:42:18)

jasonwryan · 2017-06-08 02:48:42

kata198 wrote:

whatprovides_upstream
The biggest pain for me on archlinux (which I've been using for over a decade) is that there's no "whatprovides" functionality, as with yum.
For example:
yum --whatprovides '*/libc.so.6
has no pacman equivalent. You can query against currently-installed packages, but not against available (uninstalled) packages.
whatprovides_upstream adds this functionality (requires pacman-utils-data).
You can query for a single file:
[myuser@myhostname ~]$ whatprovides_upstream '/usr/lib/libgvpr.so'
graphviz
Otherwise, you have to resort to the web to figure out what "gvpr.so" is and to which package it belongs..

Ummm...

┌─[Veles ~]
└─╼ pacman -Fo /usr/lib/libgvpr.so
usr/lib/libgvpr.so is owned by extra/graphviz 2.38.0-11
┌─[Veles ~]
└─╼ pkgfile /usr/lib/libgvpr.so
extra/graphviz
┌─[Veles ~]
└─╼ pacman -Q graphviz
error: package 'graphviz' was not found

Moving to Community Contributions

kata198 · 2017-06-08 03:24:21

Oh, and I just updated and added two more tools:

aur-getpkg

Same as archsrc-getpkg, except fetches the latest snapshot of an AUR package into current dir

aur-buildpkg

Same as archsrc-buildpkg, except builds a package from AUR instead of standard arch repos

kata198 · 2017-06-08 03:40:10

I see. I didn't know pacman supported that. It doesn't work for me at all, any ideas why?

pacman -Fo /usr/lib/libc.so
warning: database file for 'core' does not exist
warning: database file for 'extra' does not exist
warning: database file for 'community' does not exist
warning: database file for 'multilib' does not exist

But I obviously have said databases.

Also, based on help, I don't get the impression that it support glob expressions, like '*/whatever', is that the case?

EDIT: I did install pkgfile as well, which does seem to support glob expressions (with pkgfile -v -g), supporting all that whatprovides_upstream provides.

So thanks for that info! Probably 13 years ago I did my first archlinux install, and had never heard of it. Learn something new every day!

Anyway, some of the other utils may be useful (archsrc-buildpkg does kinda what "srcpac" used to do, although srcpac seems to no longer exist.

Enjoy!

Last edited by kata198 (2017-06-08 03:50:10)

apg · 2017-06-08 03:44:56

-Fy

x33a · 2017-06-08 07:02:53

To add to what apg mentioned, the pacman files dbs are separate from the main dbs. So you have to keep them synced separately.

jasonwryan · 2017-06-08 07:18:59

kata198 wrote:

I see. I didn't know pacman supported that. It doesn't work for me at all, any ideas why?

If you are announcing additional pacman tools, it wouldn't hurt if you knew how pacman actually worked... Or just read the man page.

eschwartz · 2017-06-09 02:38:10

Your reimplementation of pacman -F and pkgfile is not the only thing that already had a solution.

kata198 wrote:

pacman-mirrorlist-optimize
Yet another pacman mirrorlist optimization script. There was a web-based one a while back, which seemed to use ping-times to the various server (which measures latency, NOT throughput).
This tool scans your /etc/pacman.d/mirrorlist, by default using all mirrors listed (even commented), but can be changed ( see --help ).
It downloads a package from each of them, and measures the real throughput between you and that server, and sorts according to such. Defaults to glibc, but you can pick a different package with a parameter. Again, see --help.
It will display a ratio next to each server (in a comment), to let you know the relative speed to the fastest.

Historically, pacman has shipped with the rankmirrors script, which also tests the servers based on the speed of downloading, in this case, the database for the core repo.
Additionally, reflector can download a refreshed mirrorlist and sort it according to several criteria including the download speed of the core repo database.

reflector is very featureful and I would tend to go with that.

abs2
Since abs and mass-updates have been removed, this tool takes a gentler approach. It will download/update info on a single package (provided as an argument), and output the path to where it put it (/var/abs2).
First time run as root, after that anyone in "users" can do it (as oppose to old abs, which default design and usage was to always be run as root)

I feel that asp (the official Arch Linux replacement for abs) does a much better job of this. No need for root at all, and you can run `asp export $pkgname` to both clone it into the asp cache and export it into $(pwd)/$pkgname

archsrc-buildpkg
If you use my "extras" and set the DEFAULT_USE_CFLAGS to your desired profile, or just use the stock and modify CFLAGS/CXXFLAGS therein to your liking, and don't want to do PGO, this is the tool for you.
It's a one-stop shop to build a package for you.
You run it as your user, and it will sudo to root.
This is to ensure /usr/src/arch is created, 2775, and group=users, and to install the package after build.
Then it fetches the package build info into /usr/src/arch, builds package, and installs it. On any failure it stops and you see why.

This is already as simple as `asp export $pkgname && cd $pkgname && makepkg -i`.

aur-getpkg
Same as archsrc-getpkg, except fetches the latest snapshot of an AUR package into current dir
aur-buildpkg
Same as archsrc-buildpkg, except builds a package from AUR instead of standard arch repos

Okay... Yet Another AUR Helper (YAAH).

Some utils for lazy folks
I included a few in pacman-utils, but I have a lot more..

mkgcdatar
For use with my PGO stuff, see the README on https://github.com/kata198/pacman-utils for more info
getpkgs
Gets the names of packages, with no args in current directory, otherwise in a directory, or unrolls a glob and only prints packages.

This is a very complex wrapper for `echo *.pkg.tar.xz`, or I suppose since you value newline-delimited output, `printf '%s\n' *.pkg.tar.xz`

installpackage
Installs packages matching a pattern. Very useful if you have several versions of the same package, or similar names that tab doesn't make things quick, you can do like:
installpackage 6-2
To install all packages in current dir with "6-2" somewhere in it.

This is literally a shellscript that runs `pacman -U *.pkg.tar.xz` after possibly reinvoking itself with su.

These belong as aliases/functions in your bashrc...

Last edited by eschwartz (2017-06-09 02:40:15)

Arch Linux

#1 2017-06-08 02:33:28

Announcing some additional pacman utils + replacement for ABS

#2 2017-06-08 02:48:42

Re: Announcing some additional pacman utils + replacement for ABS

#3 2017-06-08 03:24:21

Re: Announcing some additional pacman utils + replacement for ABS

#4 2017-06-08 03:40:10

Re: Announcing some additional pacman utils + replacement for ABS

#5 2017-06-08 03:44:56

Re: Announcing some additional pacman utils + replacement for ABS

#6 2017-06-08 07:02:53

Re: Announcing some additional pacman utils + replacement for ABS

#7 2017-06-08 07:18:59

Re: Announcing some additional pacman utils + replacement for ABS

#8 2017-06-09 02:38:10

Re: Announcing some additional pacman utils + replacement for ABS

Board footer