Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Xyne · 2015-12-04 18:19:30

Project page: http://xyne.archlinux.ca/projects/bauerbill/
AUR page: https://aur.archlinux.org/packages/bauerbill/

Bauerbill is a versatile ABS and AUR helper. It is an extension of Powerpill that can build packages from AUR and ABS with full dependency resolution. It provides trust management (e.g. trust some users for all packages, trust others for certain packages, trust others for particular releases that you have verified) and provides extensive support for user hooks to apply patches and affect other changes while building packages. It correctly handles all metadata (e.g. split packages, architecture-specific metadata): see the table.

Being an extension of Powerpill, it also speeds up downloads of databases and packages.

Original Post
It's been a while since I've posted anything new but 'tis this season! I've finally managed to rewrite a (imo) worthy successor to Bauerbill.

For those of you who've never heard of the original Bauerbill*, it was an extension of Powerpill that could build packages from ABS and the AUR (nearly) transparently. It was in the same class of AUR helpers as yaourt. This iteration is slightly different. It generates build scripts that can be configured with custom commands and hooks per package.

It's fresh off the keyboard so expect some bugs. Please report what you find (or just give feedback in general). Check the project page for information about what it can do and how you can configure it. Things will likely change a bit in the coming weeks as I play around with it and get feedback. The main work is done but there is a lot of polishing left.

* Rhymes with Powerpill, means Bill the Builder... the name is clunky but it amuses me, and you can alias it to "bb"

Last edited by Xyne (2016-04-05 16:17:15)

SanskritFritz · 2015-12-08 10:06:22

WOW. Welcome back, Bauerbill.
Thanks Xyne.

SanskritFritz · 2015-12-14 22:24:49

How do I set the working directory? I would like to reuse my existing one.

Xyne · 2015-12-14 22:25:33

I have added some usage examples to the webpage here.

I have also added a "--build-vcs" option to rebuild VCS packages. The patterns for VCS packages can be configured in bauerbill.json. You can easily add a custom package to the array to force it to always be rebuilt with that option.

SanskritFritz wrote:

How do I set the working directory? I would like to reuse my existing one.

What's the structure of your working directory? If it just contains the package directories then you should be able to run the download and build scripts in it. You would also use the --build-dir option in that case with the wrapper script.

If you want custom SRCDEST and PKGDEST then you can create a custom makepkg configuration file and add it to the makepkg commands (either the default or a per-package command) in bauerbill.json.

I will probably add some more configuration options for per-package build locations. I need to think about how to do it.

Feel free to make suggestions. Same for anything else in the package, e.g. the bb-wrapper script. I may create a second one without "sudo", or make it optional in the script via a command line argument.

SanskritFritz · 2015-12-15 12:17:05

--build-dir was my solution, thanks, Xyne.

SanskritFritz · 2015-12-15 21:07:03

I get an error when running this:

~> bauerbill -Su --aur-only

WARNING: multiinfo query (kde-wallpapers): no results
...
WARNING: Installed orphans: kde-wallpapers ...
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.5/site-packages/Bauerbill.py", line 1362, in <module>
    run_main()
  File "/usr/lib/python3.5/site-packages/Bauerbill.py", line 1346, in run_main
    return main(args)
  File "/usr/lib/python3.5/site-packages/Bauerbill.py", line 1324, in main
    scripts['download'] = download_script(bb_config, pargs, build_pkgs)
  File "/usr/lib/python3.5/site-packages/Bauerbill.py", line 836, in download_script
    'info' : sh_info_comments(pkgs),
  File "/usr/lib/python3.5/site-packages/Bauerbill.py", line 645, in sh_info_comments
    maintainers=' '.join(pkg.maintainers()),
TypeError: sequence item 0: expected str instance, NoneType found

Xyne · 2015-12-15 21:52:08

fixed (I think)

SanskritFritz · 2015-12-15 23:32:13

Xyne wrote:

fixed (I think)

Confirmed, thanks!

SanskritFritz · 2015-12-16 10:01:17

There is a false warning, when checking for installed non existing packages.
Example:

WARNING: multiinfo query (pzl_games): no results

This is probably because pzl_games is a base package, but it does exist, but of course it cannot be installed directly, for it provides plz_* packages (https://aur.archlinux.org/pkgbase/pzl_games/).

Xyne · 2015-12-16 13:40:46

That's not a false warning. When running a system upgrade, the query arguments are taken directly from the names of installed foreign packages. If you have a package named "pzl_games" installed then it will search for it on the AUR if it can't be found in the repos. Even if it coincides with an existing package base, there is no package with that name.

If there is no way to upgrade the package via pacman or bauerbill then you could just add it to the list of ignored packages to avoid the warning (assuming that you want to keep it installed).

SanskritFritz · 2015-12-16 14:55:47

Oh, you're right, sorry. I checked it again, and it was indeed installed. I accidentally installed it long time ago according to pacman.log. Sorry for the noise.

rlevArch · 2015-12-18 02:57:33

Hi Xyne,

Just getting back into using my arch system after many years. I'm having trouble with bauerbill, getting the following error:

~  bauerbill -Syu
ERROR: LockError(/var/lib/pacman/db.lck): failed to lock database [[Errno 13] Permission denied: '/var/lib/pacman/db.lck']

db.lck does not exist in that directory, and pacman is not running elsewhere. If I run pacman -Syu or use pacaur -Syu, everything updates correctly.

_________
Edit: nvm, looks like using bb-wrapper instead of bauerbill alone does the trick.

Last edited by rlevArch (2015-12-18 03:00:39)

Xyne · 2015-12-18 20:39:42

Hi rlevArch,

The error says that you do not have permission to create the file, not that it exists. "bauerbill -Syu" requires root privileges, just like "pacman -Syu'.

Xyne · 2016-01-12 23:34:30

Made some changes:

Renamed option --build-targets to --build.
Added option --nobqd for generating build scripts without pre-emptive dependency installation. The scripts can now be created without ever needing root.
Added possibility to trust maintainer-pkgbase combinations with bb-query_trust.
Added colorized output to bb-query_trust (screenshot below).

SanskritFritz · 2016-01-14 04:58:39

Handling of orphaned packages is a bit problematic. When I run ./download.sh I get this:

INFO: searching AUR
INFO: found gtk-theme-orion at https://aur4.archlinux.org/cgit/aur.git/snapshot/gtk-theme-orion.tar.gz
INFO: found lib32-libpng12 at https://aur4.archlinux.org/cgit/aur.git/snapshot/lib32-libpng12.tar.gz
INFO: found libpng12 at https://aur4.archlinux.org/cgit/aur.git/snapshot/libpng12.tar.gz
usage: bb-query_trust [-h] [-c <path>] [-l <path>]
                      <package base> <modification time> <maintainer>
                      [<maintainer> ...]
bb-query_trust: error: the following arguments are required: <maintainer>

It would be nice to know which one is orphan and honestly I don't know what bauerbill should do. What should happen with orphan packages?

Also, when I press ctrl-c at the prompt, I'd expect bauerbill to exit, instead it continues:

Untrusted combination:
  Package Base:         lib32-libpng12
  Modification Time:    1451369309 [2015-12-29 06:08:29 GMT]
  Maintainers:
    arthurzam

You may inspect the files at the following location before proceeding:
  /home/frank/build/lib32-libpng12

Options:
  P: trust this combination once and proceed
  M: add maintainer(s) to list of trusted users
  C: add combination to list of trusted combinations

  Press any other key if you do not trust this combination.

[PMC] ^CUntrusted combination:
  Package Base:         libpng12
  Modification Time:    1451369269 [2015-12-29 06:07:49 GMT]
  Maintainers:
    arthurzam

You may inspect the files at the following location before proceeding:
  /home/frank/build/libpng12

Options:
  P: trust this combination once and proceed
  M: add maintainer(s) to list of trusted users
  C: add combination to list of trusted combinations

  Press any other key if you do not trust this combination.

Xyne · 2016-01-14 07:54:57

Orphans and ctrl+c should now be handled correctly. Thanks for reporting.

edit
Added support for "-Qu --aur" and new option `--bb-quiet` to suppress AUR query warnings.

Last edited by Xyne (2016-01-14 10:26:35)

tedbell · 2016-01-23 20:12:20

When updatign aur packages I get this error:

Running makepkg as root is not allowed as it can cause permanent,
catastrophic damage to your system.

How can I update AUR packages with bauerbill as user?

Xyne · 2016-01-24 18:52:37

How are you running bauerbill? It doesn't run the generated AUR scripts automatically so you must be trying to run them directly as root.

Please see the usage examples on the project page. Basically there are two main ways to do it. Either run bauerbill as root then run the generate build scripts as a normal user, or just use the bb-wrapper script, e.g.

bb-wrapper -Syu --aur

tedbell · 2016-01-25 00:52:55

Xyne wrote:

How are you running bauerbill? It doesn't run the generated AUR scripts automatically so you must be trying to run them directly as root.
Please see the usage examples on the project page. Basically there are two main ways to do it. Either run bauerbill as root then run the generate build scripts as a normal user, or just use the bb-wrapper script, e.g.
bb-wrapper -Syu --aur

I was running it as

sudo bb-wrapper -Syu --aur

thanks for the suggestions

Xyne · 2016-01-25 21:30:37

Since Friday's version bb-wrapper should drop privileges when necessary when run as root.

Alad · 2016-02-13 17:12:43

Hi Xyne,

When retrieving the AUR deps for plasma-desktop-git with bauerbill --aur --nobqd plasma-desktop-git -S, it seems to miss two makedepends:

extra-cmake-modules-git
kdoctools-git

Depends list from download.sh: https://ptpb.pw/Ei96
Depends list from tsort: https://ptpb.pw/j7PW
download.sh: https://ptpb.pw/XV7l/sh

Last edited by Alad (2016-02-13 17:14:54)

Xyne · 2016-02-14 06:23:11

It should be fixed now. It will also generate the clean-up script now for nobqd to remove leftover deps and makedeps.
edit: Both bauerbill and pm2ml need to be updated.

Last edited by Xyne (2016-02-14 06:28:22)

Spyhawk · 2016-02-14 11:38:38

I know you have some concern about putting your work on github or similar social forges for some reason, but I really feel the biggest drawback of your projects is that they're developed being closed doors. Sometimes, I wish I could have a look at what the last commit is and what it brings, how you fixed a certain issue without having to diff the provided tarballs, or just have a look at your release map and planned features, or other similar stuff as done by developers and enthusiasts in the 2010'. This is specially true for project that could be built-on like your python3-aur package.

This said, thank you for all your work. I'm a very happy user of aurtomatic among other tools, and I'm also quite impressed by bauerbill too.

Edit: Also, I just found out that bauerbill has some difficulties installing split packages. The "python-novaclient" PKGBUILD provides both python-novaclient and python2-novaclient. I tried installing python-novaclient with bb-wrapper -S --aur --nobqd python-novaclient, but get many errors along the way:

=> Removing installed dependencies...
checking dependencies...
error: failed to prepare transaction (could not satisfy dependencies)
:: python-oslo-i18n: removing python-pbr breaks dependency 'python-pbr>=1.6'
:: python2-oslo-i18n: removing python2-pbr breaks dependency 'python2-pbr>=1.6'
:: python2-oslo-i18n: removing python2-six breaks dependency 'python2-six>=1.9.0'
==> WARNING: Failed to remove installed dependencies.

then:

warning: cannot resolve "python2-debtcollector>=0.3.0", a dependency of "python2-oslo-utils"
:: The following package cannot be upgraded due to unresolvable dependencies:
      python2-oslo-utils

:: Do you want to skip the above package for this upgrade? [y/N] y

and later on:

==> Installing python-novaclient package group with pacman -U...
loading packages...
resolving dependencies...
warning: cannot resolve "python-oslo-serialization>=1.10.0", a dependency of "python-novaclient"
warning: cannot resolve "python-oslo-utils>=3.2.0", a dependency of "python-novaclient"
warning: cannot resolve "python2-oslo-serialization>=1.10.0", a dependency of "python2-novaclient"
warning: cannot resolve "python2-oslo-utils>=3.2.0", a dependency of "python2-novaclient"
:: The following packages cannot be upgraded due to unresolvable dependencies:
      python-novaclient  python2-novaclient

:: Do you want to skip the above packages for this upgrade? [y/N] y
looking for conflicting packages...
 there is nothing to do
~/build
error: target not found: python-oslo-serialization
error: target not found: python-oslo-utils

On a side note, it seems bauerbill is currently unable to install only one of the provided split packages (either python-novaclient or python2-novaclient) and wants to install both of them.

Last edited by Spyhawk (2016-02-14 15:30:53)

Xyne · 2016-02-14 19:13:33

The warnings are due to makepkg failing to distinguish between deps and makedeps and therefore trying to remove them all after the package has been installed. It correctly detects the deps and leaves them installed but issues the warning nevertheless. You can ignore the warnings or file a bug report on the pacman bug tracker.

The issue with split packages is that makepkg no longer recognized the "--pkg" option to select which split package to install. I'm looking in to it and will file a bug report if it really has been removed.

edit
Bug report: https://bugs.archlinux.org/task/48183

I will post an update if I find a reliable work-around.

Last edited by Xyne (2016-02-14 19:21:59)

Spyhawk · 2016-02-14 20:03:17

Xyne wrote:

The issue with split packages is that makepkg no longer recognized the "--pkg" option to select which split package to install. I'm looking in to it and will file a bug report if it really has been removed.
edit
Bug report: https://bugs.archlinux.org/task/48183
I will post an update if I find a reliable work-around.

Yes, this has been known for a few months now. See pacman-dev and following messages for the explanation.
The workaround is to simply use "pacman -U" to install the selected subpackage instead of using "makepkg -i --pkg subpackage".

Arch Linux

#1 2015-12-04 18:19:30

Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#2 2015-12-08 10:06:22

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#3 2015-12-14 22:24:49

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#4 2015-12-14 22:25:33

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#5 2015-12-15 12:17:05

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#6 2015-12-15 21:07:03

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#7 2015-12-15 21:52:08

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#8 2015-12-15 23:32:13

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#9 2015-12-16 10:01:17

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#10 2015-12-16 13:40:46

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#11 2015-12-16 14:55:47

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#12 2015-12-18 02:57:33

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#13 2015-12-18 20:39:42

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#14 2016-01-12 23:34:30

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#15 2016-01-14 04:58:39

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#16 2016-01-14 07:54:57

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#17 2016-01-23 20:12:20

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#18 2016-01-24 18:52:37

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#19 2016-01-25 00:52:55

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#20 2016-01-25 21:30:37

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#21 2016-02-13 17:12:43

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#22 2016-02-14 06:23:11

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#23 2016-02-14 11:38:38

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#24 2016-02-14 19:13:33

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#25 2016-02-14 20:03:17

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Board footer