Automatically compiling several AUR packages in parallel

Wild Penguin · 2016-09-29 15:57:34

Hi,

Is it possible to compile several packages from AUR in parallel, via some AUR helper? If it is, I seem to have trouble finding the right search words...

Currently, I'm using several git/svn/etc. packages from AUR, some of which take a long time to compile. Some packages don't scale well with '-j 8' set in MAKEFLAGS in makepkg.conf, hence the CPU is sitting idle, while it could be compiling another package that is unrelated / not dependent on the package currently being compiled - or make rest of the AUR upgrades before the huge package has finished compiling (so the impatient user can use the upgraded software while the rest are being compiled).

Essentially, I'm looking for a similar option to "EMERGE_DEFAULT_OPTS" in Gentoo.... the said feature would prepare & compile all non-dependent packages in the queue up-to '-jX' packages and/or '--lX.Y' load average (note: these should NOT be passed on to make us such, but used by the AUR helper itself - so it should be totally separate from MAKEFLAGS).

x33a · 2016-09-29 16:32:52

I am not sure if any such option is available for use with AUR packages. You can try making your own solution, perhaps by using GNU parallel or such.

Wild Penguin · 2016-09-29 17:31:01

I was guessing that is the situation, just wanted to confirm I'm not missing something.

I believe the most easy way would be to modify some of the existing helpers so that it would understand the proposed cmd (or config file) parameters and fork to install those packages in parallel, for which dependencies are met. But that is not a trivial task (at least not for me). I don't have enough knowledge of pacman databases to achieve this (currently, you can not run several helpers - well not at least pacaur - at the same time, since one of the processes will bail out). As a related note, none of the helpers I've tried can resume a failed install, either - say, if the compile phase succeeds but some trivial thing (for example pacman db lock) prevents the installation, they will start the compile phase from the beginning on the next try.

This just shows there is a lot to be done for such a helper... running things parallel is trivial, but automating the task (of installing packages), taking into consideration that no two packages can be actually installed at the same time, is not. Main job is in making a helper that can stop after compiling, if the database is already in use - and also lock it only when it actually needs it. I'm not sure the PKGBUILD spec actually has enough information to achieve this, either - but perhaps, if it is enough for the helper to wait after pre_X if the database is locked.

I'm not sure this is going to happen, unless someone else finds it usefull (there is not that much packages that actually compile something, than for example in Gentoo, where everything is compiled) and is motivated and skilled enough to do it.

Last edited by Wild Penguin (2016-09-29 17:33:11)

WorMzy · 2016-09-29 17:48:21

I'm not sure if I'm missing something here, but you can have as many instances of makepkg running concurrently as you like. If you're using clean chroots, then you can have as many chroots running as you like. You can only have one pacman instance running at a time, however, so you can only install (make)dependencies/the finished products in serial, but unless you're trying to set off the makepkg instances simultaneously, this isn't likely to be an issue.

Wild Penguin · 2016-09-29 18:10:08

I believe I didn't explain the problem clearly enough.

I can run makepkgs in parallel (that is trivial) - however, this is about automating repetitive, boring tasks. Current helpers are already quite good at that, this is just one point I think they could be improved. Problem with current helpers is that they have no features to do pre-installation phases in parallel (and lock the database only before package() - forget what I said about pre_x before, I understod the MAKEPKG documentation wrongly, I think). If there are huge source packages for which compiling takes a long time, there could be some time to be gained here...

Trilby · 2016-09-29 18:37:19

As you said it is trivial. So if you want it automated, automated it ... with a script for shell function:

find ~/builds -maxdepth 1 -type d -exec bash -c "cd '{}'; makepkg &" \;
while [[ $(pgrep makepkg) ]]; do sleep 10; done
find ~/builds -maxdepth 1 -type d -exec bash -c "cd '{}'; makepkg -i" \;

Awebb · 2016-09-29 18:39:51

What do you usually do, when you want to run things in parallel from a bash script?

Spyhawk · 2016-09-29 20:31:14

That's something I had a look at long time ago (for pacaur), but deemed the complexity of the implementation simply not worth it.

Xyne · 2016-09-30 01:10:35

After seeing this thread tonight, I tweaked Pacboy to support parallel builds (it was on my TODO list and was actually easy to do). In the testing directory, take a look at snippets/example-parallel_build-prologue.sh for notes on how to use it. Basically, just add the configuration lines to config-pacboy.json and you're good to go.

For those curious about the implementation, Pacboy constructs an internal dependency graph to order the builds. Each package is assigned a level in the graph and all packages with the same level should be independent of each other. In order to support parallel builds, I have added the graph level to the variables that will be interpolated in the user-provided code snippets to generate the build script. All of the magic happens in the build script which is fully customizable by the user.

The example implementation uses all 4 snippets to set up functions, launch builds in the background for each graph level, wait for them to complete when the next level is reached, then repeat until the end and clean up. I wrote the snippets in less than 10 minutes. Feel free to suggest ways to improve them or notify me of any bugs.

Incidentally, Pacboy can also pre-emptively download all source files in parallel prior to the build.

To test it quickly:

Download, untar and cd into the pacboy test directory.

Add the following to the "snippets" section of config-pacboy.json:

"prologue": "snippets/example-parallel_build-prologue.sh",
"epilogue": "snippets/example-parallel_build-epilogue.sh",
"build": "snippets/example-parallel_build-build.sh",
"pacman": "snippets/example-parallel_build-pacman.sh"

Generate a build script with
```
./testrun.sh ./pacboy --pb-np -Su
```
cd into the build directory and, after inspection, run it with ../testrun.sh ./build.sh

The testrun.sh script is just a wrapper to manage paths in the testing directory.

Wild Penguin · 2016-10-30 13:36:49

Hi all,

thanks for your replies (had some boring stuff like work to do, hence no replies from my part). I could have been more clear in my OP - I was specifically looking for an AUR helper, that can compile in parallel; i.e. something removing the need for me to use somewhat complex bash one-liners or writing my own scripts - both of which have their uses, and I appreciate the examples. I used to use Gentoo, where this is quite a mandatory feature - as everything is compiled. In AUR, there actually aren't that many source packages - which probably explains why such a helper has not surfaced as of yet: there has been too little to gain to trigger someone to write such a feature...

However, I will take a look at pacboy!

Spyhawk · 2016-10-30 16:41:38

Yes, pacboy is what you are looking for. I however do not plan to add support parallel builds in pacaur, mostly because I have no use for it, and that this is somewhat more complex to add than in bauerbill/pacboy due to a different design.

Arch Linux

#1 2016-09-29 15:57:34

Automatically compiling several AUR packages in parallel

#2 2016-09-29 16:32:52

Re: Automatically compiling several AUR packages in parallel

#3 2016-09-29 17:31:01

Re: Automatically compiling several AUR packages in parallel

#4 2016-09-29 17:48:21

Re: Automatically compiling several AUR packages in parallel

#5 2016-09-29 18:10:08

Re: Automatically compiling several AUR packages in parallel

#6 2016-09-29 18:37:19

Re: Automatically compiling several AUR packages in parallel

#7 2016-09-29 18:39:51

Re: Automatically compiling several AUR packages in parallel

#8 2016-09-29 20:31:14

Re: Automatically compiling several AUR packages in parallel

#9 2016-09-30 01:10:35

Re: Automatically compiling several AUR packages in parallel

#10 2016-10-30 13:36:49

Re: Automatically compiling several AUR packages in parallel

#11 2016-10-30 16:41:38

Re: Automatically compiling several AUR packages in parallel

Board footer