Programmatically getting build order for AUR packages?

Slithery · 2015-07-09 23:27:06

Is it possible to get a list of AUR packages and all of their AUR dependencies in the correct build order, given an initial AUR package list?

I'm trying to automate building my custom repo using clean-chroot-manager, and this is the part that I'm stuck on.

How do the rest of you that run your own repos handle this?

pypi · 2015-07-10 01:09:18

This is... difficult. I'm no dev, but I have hacked together systems before that created a dependency graph and fed it through tsort. Or created a makefile. Or recursively dug through the packages until every one was installed/built. But I've hit a few issues; mostly because Arch is not supposed to be a build from source distro. For instance, you have to install a package to get a full dependency list. All packages effectively depend on those in base, which is a bit ugly, although it can be worked around.

I'd be fascinated to hear how everyone else does it!

More usefully, I believe Allan has a few scripts that do automate part of the process. Unfortunately, I couldn't find it again... I think it might have been linked to from Allan's TODO list on the wiki?

Spyhawk · 2015-07-22 22:54:40

This is basically what aur helpers do, so it is definitely possible.

Lone_Wolf · 2015-07-23 14:33:40

pypi wrote:

For instance, you have to install a package to get a full dependency list.

For aur packages, parsing .SRCINFO should be enough to get all top level dependencies.

mauritiusdadd · 2015-07-23 17:37:16

Lone_Wolf wrote:

pypi wrote:
For instance, you have to install a package to get a full dependency list.
For aur packages, parsing .SRCINFO should be enough to get all top level dependencies.

Or you can use cower for AUR packages

cower -i $pkgname --format "%D"

and expac for the ones available in the official repos

expac -S "%D" $pkgname

progandy · 2015-07-23 17:48:38

I have implemented something similar in my AUR helper in deprecurse where I collect all AUR dependencies and toposort to create the topological order.
https://github.com/progandy/lauri/blob/ … lauri#L700
Edit: There is no failsafe/error for a version mismatch and dependency cycles are simply ignored, but the basic ordering should be working.
For local PKGBUILDs, simply change the source from the AurJson to mksrcinfo and SRCINFO parsing.
The topological sorting is generic enough: I used the standard algorithm and went through the Depends and MakeDepends arrays. Members of Depends and MakeDepends that are not part of the given package list are discarded since they are no AUR packages.

Last edited by progandy (2015-07-23 18:16:28)

pypi · 2015-07-23 20:37:36

mauritiusdadd wrote:

Lone_Wolf wrote:
pypi wrote:
For instance, you have to install a package to get a full dependency list.
For aur packages, parsing .SRCINFO should be enough to get all top level dependencies.
Or you can use cower for AUR packages
cower -i $pkgname --format "%D"
and expac for the ones available in the official repos
expac -S "%D" $pkgname

Split packages can override the depends fields. mksrcinfo will not note any dependencies that are specified in the packaging stage of split PKGBUILDs, so you would have to parse the SRCINFO, build the package, and then query the package itself to get the full dependency list.
@mauritiusdadd: Thanks for pointing out that you don't actually have to "install" a given package Just downloading it should be enough, or using an external helper.

@progandy's approach also looks pretty sane, albeit missing a few checks for corner cases. It'd probably be interesting to see how all the different AUR helpers advertising dependency resolution do it.

To be sure that everything was being built in order, you'd probably have to use a post-order DFS, and double check that all the required packages where built and available before trying to install a given package, in order to catch dependencies added in split packages. It'd be a bit clunky, though - in theory, a DFS would let you do parallel builds, but in practice it would probably be too complicated to be worth it. Topo sorting has the advantage of being simple

Last edited by pypi (2015-07-23 20:40:20)

progandy · 2015-07-23 22:59:34

pypi wrote:

Split packages can override the depends fields. mksrcinfo will not note any dependencies that are specified in the packaging stage of split PKGBUILDs, so you would have to parse the SRCINFO, build the package, and then query the package itself to get the full dependency list.

If dependencies are added during package() they should be unnecessary for the build process. You will only get problems with faulty packages that build inside package().

Last edited by progandy (2015-07-23 23:00:43)

pypi · 2015-07-26 01:21:36

progandy wrote:

pypi wrote:
Split packages can override the depends fields. mksrcinfo will not note any dependencies that are specified in the packaging stage of split PKGBUILDs, so you would have to parse the SRCINFO, build the package, and then query the package itself to get the full dependency list.
If dependencies are added during package() they should be unnecessary for the build process. You will only get problems with faulty packages that build inside package().

It's problematic for recursive builds - the sort which this question asks about. Let's say that a split package has an added dependency for at least one extra custom package, on an interpreter, for instance. And the split package is required for building another package. When you try to build the top level package, you will first build the split package (which will work). However, when you try to install the split package to build the top level package, it won't install - you would have to also build the required interpreter.
I've possibly got the wrong end of the stick, so if I've explained it badly or there's some reason why it would work, I'd be interested to hear.

progandy · 2015-07-26 02:16:23

mksrcinfo parses the extra dependencies for split packages:
PKGBUILD

pkgname=('pkg1' 'pkg2')
pkgbase=BASE
pkgver=1.2.3
pkgrel=1
pkgdesc="split pkg"
arch=(any)
license=('GPL')
depends=(both)
makedepends=(both_make)

build() {
  :
}

package_pkg1() {
  # options and directives that can be overridden
  pkgdesc="first pkg"
  depends+=(dep1)
  :
}

package_pkg2() {
  # options and directives overrides
  pkgdesc="second pkg"
  :
}

.SRCINFO

pkgbase = BASE
	pkgdesc = split pkg
	pkgver = 1.2.3
	pkgrel = 1
	arch = any
	license = GPL
	makedepends = both_make
	depends = both

pkgname = pkg1
	pkgdesc = first pkg
	depends = both
	depends = dep1

pkgname = pkg2
	pkgdesc = second pkg

pypi · 2015-07-28 05:49:30

Hmm. I can reproduce that now, although I did test it out earlier, and it didn't work. Unfortunately, I've deleted my test case...

That's useful though, thanks progandy! - I'll investigate parsing .SRCINFO files when I have an opportunity.

Arch Linux

#1 2015-07-09 23:27:06

Programmatically getting build order for AUR packages?

#2 2015-07-10 01:09:18

Re: Programmatically getting build order for AUR packages?

#3 2015-07-22 22:54:40

Re: Programmatically getting build order for AUR packages?

#4 2015-07-23 14:33:40

Re: Programmatically getting build order for AUR packages?

#5 2015-07-23 17:37:16

Re: Programmatically getting build order for AUR packages?

#6 2015-07-23 17:48:38

Re: Programmatically getting build order for AUR packages?

#7 2015-07-23 20:37:36

Re: Programmatically getting build order for AUR packages?

#8 2015-07-23 22:59:34

Re: Programmatically getting build order for AUR packages?

#9 2015-07-26 01:21:36

Re: Programmatically getting build order for AUR packages?

#10 2015-07-26 02:16:23

Re: Programmatically getting build order for AUR packages?

#11 2015-07-28 05:49:30

Re: Programmatically getting build order for AUR packages?

Board footer