Optional Build Time Packages

ewaller · 2015-09-11 20:17:21

I've been upping my game where it comes to Autotools. I can proudly state that I can now detect the presence (or lack thereof) of things like doxygen and, if it is available at build time, create a suite of documentation for my packages. If it is not present, then I ignore it and move on. Nice.

Taking it a step further, I tried a PKGBUILD to deal with one of my packages that has the optional build dependency of doxygen. I realized there is optional run time dependency support in makepkg, but there is no support for optional build time dependencies that I can see. This might make sense -- the build won't fail without it, and if present, I think it will use it. But, might it not be nice to be able to indicate to the end user that, if a package like doxygen is available, they will get documentation to go along with it?

Further, if one user built it on a machine without doxygen, and another user used a machine with doxygen, would they not get two different packages that have the same name and revision that had different sizes and contents? Is this not a bit nondeterministic? Or am I missing some better practice?

Scimmia · 2015-09-11 22:17:12

Since PKGBUILDs really should be built in a clean chroot, optional build-time deps don't make a lot of sense. The package is either installed in the chroot or it's not.

ewaller · 2015-09-11 22:45:41

I'm not sure I fully understand here, mostly because have honestly never build in a clean chroot. Mostly because I have not needed to or have cared.

So, if I get this right, the contents of a clean chroot are not controlled by the PKGBUILD, but rather one has to install packages into the chroot prior to the build.
So, I would conceivably create the chroot, install the packages I think I need, then do the build in the chroot. The build won't see things outside the jail. Can one make any assumptions as to what is in the chroot, like stuff from core?

Most users, I'm sure, have never used a clean chroot. I imagine most people who have created an AUR package have not either. So --- If an Autotools configured configure exists that looks for an optional build dependency, and the packager did not include that optional build dependency in their chroot, and someone else, who has the optional package installed uses the PKGBUILD, they could end up with different results then the original packager.

Last edited by ewaller (2015-09-11 22:47:43)

Scimmia · 2015-09-11 23:01:55

Creating a clean chroot installs the base-devel group. Building then installs all packages listed in the depends and makedepends arrays before building.

This is the entire problem with building packages on a live system. Too many build systems will try to autodetect something and give you completely different results. Building in a clean chroot is what gives you the consistent environment to be able to predict how the final package will turn out. All package in the binary repos are built in a clean chroot for this reason. Can you imagine how things would be if dependencies were determined by what the packager had installed?

WorMzy · 2015-09-11 23:08:57

ewaller wrote:

So, if I get this right, the contents of a clean chroot are not controlled by the PKGBUILD, but rather one has to install packages into the chroot prior to the build.

Not quite. Only base-devel (or multilib-devel if you're building a multilib package) is installed into a clean chroot by default, then anything that the PKGBUILD demands (deps and makedeps). Nothing else.

The main benefit of using a clean chroot is that "greedy" configure scripts don't pull in extra dependencies, so you have full control over what libraries the compiled binaries link against.

So --- If an Autotools configured configure exists that looks for an optional build dependency, and the packager did not include that optional build dependency in their chroot, and someone else, who has the optional package installed uses the PKGBUILD, they could end up with different results then the original packager.

Exactly. This is why everyone should build in clean chroots. It goes a long way in making reproduceable builds. https://bugs.archlinux.org/task/43407

I agree with you that most users do not though. I think running the PKGBUILD through one should be a prerequisite to submitting a package however.

EDIT: Beaten by Scimmia again. I got distracted.

Last edited by WorMzy (2015-09-11 23:10:05)

Scimmia · 2015-09-11 23:15:22

Wow, twice in one day. I guess there's an advantage to being a bit terse.

FWIW, I completely agree with you that every AUR maintainer should be doing, at minimum, a test build in a clean chroot. If nothing else, it helps a lot in figuring out dependencies.

ewaller · 2015-09-11 23:46:53

Okay thanks guys. I'll be going off to play with this for awhile and will come back with more questions after making and resolving a few mistakes.

Allan · 2015-09-12 00:12:44

Optional build time dependencies can be indicated by a comment in the PKGBUILD. Everyone reads these scripts before running them on their system... right?

ewaller · 2015-09-12 18:45:12

Okay, I did it using extra-x86_64-build. What a brilliant bit of code. If I required Doxygen, it did what I would have expected -- except carping about my package being mostly documentation -- and If I did not require it, I get no documentation in the package. I'll the while, Doxygen being installed in the real system.

I'm not finding a lot of data on this aside from the link in post #3. Any suggested reading? I have a bunch of artifacts left over in /var/lib/archbuild/extra-x86_64. There is a root branch and an ewaller branch. The ewaller branch has more files in it, including the program built by my PKGBUILD. The root branch seems to be pristine. I imagine that the root branch is a vanilla installation with that WorMzy described, whereas the ewaller branch has my dependencies that are declared by the PKGBUILD, plus the package files themselves. Do these take care of themselves? Next month when I use extra-x86_64-build on a different PKGBUILD does the root branch update by itself? Using pacman -Syu? or the packages installed in my system? Is the ewaller branch purged and rebuilt from the root branch and the PKGBUILD? Or is maintained incrementally through pacman?

Last edited by ewaller (2015-09-12 20:12:18)

WorMzy · 2015-09-12 20:00:40

Mmm, the devtool scripts are really handy.

To answer your questions: the root copy stays clean, and updates when you next build a package, and the ewaller copy is destroyed and remade from the root copy each time you build a package. If you have /var/lib/archbuild/extra-x86_64 on a btrfs partition, it will automagically use a snapshot of "root" for "ewaller", saving space and time.

Lone_Wolf · 2015-09-13 22:50:03

devtools / nspawn have 1 big flaw :

they only work when arch is booted with systemd .

Allan · 2015-09-13 23:45:40

Lone_Wolf wrote:

devtools / nspawn have 1 big flaw :
they only work when arch is booted with systemd .

Makes sense, given Arch developers only support systemd...

Lone_Wolf · 2015-09-14 07:12:09

I am aware of that,
devtools and netctl are both on my list of projects to try to adjust them so they can work with any init system.
I started looking at netctl, found 20+ files where systemd functionality is used.
I'm trying to split those parts out into bash functions, but sofar with little success.

Allan · 2015-09-14 07:37:01

Both devtools and netctl used to be systemd free. So your best bet is looking at old versions of devtools and netcfg. Have fun...

Slithery · 2015-09-14 22:18:33

ewaller wrote:

Okay, I did it using extra-x86_64-build.

Take a look at graysky's clean-chroot-manager

Arch Linux

#1 2015-09-11 20:17:21

Optional Build Time Packages

#2 2015-09-11 22:17:12

Re: Optional Build Time Packages

#3 2015-09-11 22:45:41

Re: Optional Build Time Packages

#4 2015-09-11 23:01:55

Re: Optional Build Time Packages

#5 2015-09-11 23:08:57

Re: Optional Build Time Packages

#6 2015-09-11 23:15:22

Re: Optional Build Time Packages

#7 2015-09-11 23:46:53

Re: Optional Build Time Packages

#8 2015-09-12 00:12:44

Re: Optional Build Time Packages

#9 2015-09-12 18:45:12

Re: Optional Build Time Packages

#10 2015-09-12 20:00:40

Re: Optional Build Time Packages

#11 2015-09-13 22:50:03

Re: Optional Build Time Packages

#12 2015-09-13 23:45:40

Re: Optional Build Time Packages

#13 2015-09-14 07:12:09

Re: Optional Build Time Packages

#14 2015-09-14 07:37:01

Re: Optional Build Time Packages

#15 2015-09-14 22:18:33

Re: Optional Build Time Packages

Board footer