[SOLVED] How can I incrementally build *-git AUR packages?

bertulli · 2022-11-12 11:50:24

Hi all!

I'm trying to wrap my mind around development AUR packages, say the one ending like '*-git'. Specifically, I want to build emacs-git. What I am not sure I understood is: how can I build a development package, starting from a PKGBUILD (which IIUC is updated only once in a while?). I gathered a bit of information (partly with my Google-fu, partly by inspecting the wiki on PKGBUILDs and VCS packages, partly inspecting the scripts). Please, tell me if I understood correctly:

*-git packages doesn't contain instruction on how to build a specific release. Instead, each time the package is built, makepkg will fetch the new commits from the upstream repo, and build that version.
For that reason, the way to distinguish packages is the pkgver. Normally it's a variable, but in VCS packages is usually a function that generates an auto-incrementing integer. However, this information doesn't show up in 'pacman -Qi' output
To build such a package incrementally (thus leveraging the git+make-avoid-recompilation-cache), it's not necessary to clean the directory. makepkg will automatically pull the latest commit, and reperform all the necessary steps. However, to do so, it's necessary not to perform the configure build phase. For makepkg, this translate to invoking 'makepkg -ef' (and thus skipping the prepare() function)
As a confirmaion, reading the package version from Emacs I see a change
from
GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6) of 2022-11-12
to
GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6) of 2022-07-31 (note the build number)

Now, I don't understand

Reading the PKGBUILD, I can't figure out where does it pull the latest commit/changes, since it never calls 'git pull'?
You can see at the beginning of the PKGBUILD there are some options you can set, to enable various features. If I change those between recompilations, will the cache be invalidated? Should it be?
If *-git packages are (in a sense), always updated, why does the AUR package sometimes gets updated? I suppose it's in case of major changes. If that is so, is a complete recompilation (deleting the cache) necessary?

Thanks for your help!

Last edited by bertulli (2022-11-13 16:54:35)

Slithery · 2022-11-12 12:17:47

bertulli wrote:

Reading the PKGBUILD, I can't figure out where does it pull the latest commit/changes, since it never calls 'git pull'?

The makepkg script parses the source array in the PKGBUILD (the git+ before the URL) and therefore knows that it should be pulling the source from the git repo.

bertulli wrote:

If *-git packages are (in a sense), always updated, why does the AUR package sometimes gets updated?

PKGBUILD's for AUR packages only need to be updated when any of their defined functions are altered, for example a change in the build process.

Trilby · 2022-11-12 13:52:29

(Referencing number of bullets from your post below)

1) Correct.

2) Mostly correct. But your point about it not showing up in `pacman -Qi` output is unclear and seems incorrect. The version number reported by pacman -Qi will be the result of the pkgver function at the time the package was built. This is the same as any other AUR package (or repo package), pacman -Qi tells you the version you installed, but contains no information about whether new versions are available.

3) The first half is correct. The second half actually seems reversed. Just rerun `makepkg`. That's it. Neither the -e or -f flags are needed and the former may even cause problems. You do not want to use the existing source, you want to pull new code. If some modules have not changed, the build system (e.g., make) will recognize the object file's timestamp as newer than the source files and will properly not rebuild that module. But any modules with changed source files do need to be rebuilt. The -f flag is mostly harmless, but it could cause needless (but harmless) work. If there is new code and thus a new version generated by the pkgver, then there is no need for the -f as a newly named pkg.tar.xz is being built. If the -f would have been necessary, then you shouldn't be rebuilding at all: if you run `makepkg -f` in a git aur package directory and there are no updates, you force makepkg to rebuild it anyways; if you use just `makepkg`, it will check if the pkgver has changed, and if so it will rebuild, if not, it will properly stop and do nothing more. (In other words, makepkg would do the right thing on it's own, but if you use the -f flag, you force it to do the wrong thing a fair portion of the time).

4) You seem to be comparing the emacs-reported versions (e.g., from --version output perhaps). Those are not package versions. The latter would be reported by pacman -Qi.

---

5) makepkg calls (an equivalent of) `git pull` internally (specifically in `/usr/share/makepkg/source/git.sh`).

6) There is no singular "cache" - personally I think that's a misleading concept in this context (unless of course you have independently set up something else like ccache). There are just source files and object files and build scripts (e.g., makefiles). Make is an impressively intelligent bit of software that is capable of detecting when any given object file needs to be rebuilt or when it does not. However, this ability depends on the upstream authors writing proper makefiles. In ths specific case of whether changing those variables will cause a rebuild, the ideal answer is that any change in those variables should only trigger a rebuilt of the modules specifically needing a rebuild due to that variable change while all others should be left untouched (the existing object files reused, or the "cache" in your terminology maintained). Any deviation from this behavior is A) a bug in their makefile and / or build scripts to report upstream, and B) highly likely - like I said, make depends on authors writing informative makefiles, and this is sadly a dying art form.

7) The PKGBUILD of a git package can be updated for any number of reasons. The most likely (good) reason is if changes in upstream development require changes in the actual build commands. Perhaps a new option is added and an environment variable needs to be passed to make to use it. This requires a change in the PKGBUILD. Then there is the most common bad reason for -git PKGBUILDS to change, and that's because the AUR maintainer thinks they should bump the version number to "help" other users know there is an update (this is wrong, stupid, and annoying ... but common). When the PKGBUILD changes you do not need to delete anything (again "cache" is misleading in this context, there is no such thing here); just get the new PKGBUILD and any associated files from the aur git repo (git pull in the toplevel directory from aur.archlinux.org) then as above, just run `makepkg`. That's it. Don't make it more complicated.

Last edited by Trilby (2022-11-12 13:54:33)

bertulli · 2022-11-13 15:50:39

As always, thanks a lot for the super explicative answers!

To Trilby:
2) Sure pacman cannot know about new versions. What I meant is: does pacman track down the incremental build number? Unfortunately, I didn't write down what was the output of pacman before updating, but right now I see from 'pacman -Qi emacs':

[alessandro@commodoroII ~]$ pacman -Qi emacs
Name            : emacs-git
Version         : 29.0.50.158619-1
Description     : GNU Emacs. Development master branch.
[...]

Moreover, the PKGBUILD contains:

pkgver=29.0.50.158619
pkgrel=1

[...]

pkgver() {
  cd "$srcdir/emacs-git"

  printf "%s.%s" \
    $(grep AC_INIT configure.ac | \
    awk -F',' '{ gsub("[ \\[\\]]","",$2); print $2 }') \
    $(git rev-list --count HEAD)
}

Pkgver should be override with the output of the function pkgver(), IIUC. In normal circumstances, this is what should track the effective development of the package, right? On the other hand, the wiki says 'pgkrel' should reflect the difference between consecutive builds. Another question is, is this incremented automatically each time, or should it be modified in the PKGBUILD manually?

3) Thanks, now it's clearer. However, I see in the PKGBUILD:

# There is no need to run autogen.sh after first checkout.
# Doing so, breaks incremental compilation.
prepare() {
  cd "$srcdir/emacs-git"
  [[ -x configure ]] || ( ./autogen.sh git && ./autogen.sh autoconf )
  mkdir -p "$srcdir/emacs-git/build"
}

What does this warning mean then? I mean, if the 'configure' script exist, the function doesn't do anything anyway. Is the '-e' flag also preventing the new sources to be pulled?
I think my confusion arises from the fact I cannot see where is the repo pulled: according to Slithery, is this automatically done by makepkg when it sees a 'git+https://...' source? If yes, I assume '-e' also blocks that (thus not downloading anything)?

4) As in 2)

5) Thanks, I suppose this answer my comment on 3) too?

6) You're right, bad choice of words from me, sorry. Yes, I wasn't referring to external build tools like Ccache, but to "compiled object files usually properly handled by make dependency detection, to produce the final executable". In this specific case, those variables are used to build the 'depends' and '_conf' PKGBUILD vectors, so may I be reasonably confident it's working as planned, I suppose?

7) Perfectly clear, thanks!

---------------------

P.S. I also see in the PKGBUILD, inside build():

################################################################################

################################################################################

  ../configure "${_conf[@]}"

  # Using "make" instead of "make bootstrap" enables incremental
  # compiling. Less time recompiling. Yay! But you may
  # need to use bootstrap sometimes to unbreak the build.
  # Just add it to the command line.
  #
  # Please note that incremental compilation implies that you
  # are reusing your src directory!
  #

Since, IIUC, makepkg doesn't accept targets like 'makepkg bootstrap', does this comment mean to tweak the PKGBUILD?

Thanks again!

Trilby · 2022-11-13 16:46:25

2) Yes, pacman tracks the (incremental) build number. When makepkg runs on a VCS (e.g., -git) PKGBUILD, the pkgver function is run, then makepkg itself edit's the PKGBUILD file and replaces the pkgver variable value with the result of the pkgver function. In your example, 29.0.50.158619 was the result of the pkgver function last time you built that package, and that value was placed in the pkgver variable field of the PKGBUILD at that time.

bertulli wrote:

'pgkrel' should reflect the difference between consecutive builds.

You missed quoting the relevant part of the wiki there "... consecutive builds of the same version of a package". If you were to run makepkg in your emacs-git directory and if there had been absolutely no activity upstream, makepkg would properly report that there was nothing to do as the relevant pkgver-pkgrel.pkg.tar.xz already exists. If there have been new commits upstream, there will be a new pkgver, and so a new different pkgver-pkgrel.pkg.tar.xz would be created (which is also why there is no need for the -f flag in these cases). If you really needed to rebuild a new package for the same source code, then you'd either need to use the -f flag (which is what a user of an AUR package would / should do) or increment the pkgrel number (which is what the maintainer of an AUR package would / should do). Pkgrel increments are exceedingly rare in VCS packages to the point that as a rule, they'd never be anything other than 1 (there may be extremely rare exceptions ... but as soon as there was upstream activity, they could still be reset back to 1 again). Pkgrel values are primarily relevant for non-vcs packages.

3) I'm not sure what the packager meant by the comment - whether it means after the very first checkout, or after each subsequent checkout. The code implies the former as that's what it implements. In either case, that comment appears informative: it's telling you what that code is doing, not indicating any intervention you need to handle. As for the -e flag, yes, it prevents new sources from being retrieved. The -e flag tells makepkg to build using just what is currently in $srcdir as it is.

bertulli wrote:

6) ... so may I be reasonably confident it's working as planned, I suppose?

Reasonably ... sure. Fully, I'm not sure. Note that my comments have focused on how the AUR and VCS PKGBUILDs can / should work. But there are ample examples of AUR packages doing things counter to the guidelines and best practices. I've never used emacs and know nothing about the emacs-git AUR package. But it is quite popular and widely used / reviewed - so it's a pretty safe bet that the packager either really knows what they are doing or are getting feedback if / when anything is off.

bertulli wrote:

Since, IIUC, makepkg doesn't accept targets like 'makepkg bootstrap', does this comment mean to tweak the PKGBUILD?

Yes, that would be my reading of the intent of that comment: in contrast to the other descriptive comment, this one is telling you how you could edit to the PKGBUILD if needed.

Last edited by Trilby (2022-11-13 16:47:05)

bertulli · 2022-11-13 16:54:18

Thanks really a lot! I think I know understand reasonably well what (the major part of) makepkg does. Kudos to you and to all this wonderful community!

Arch Linux

#1 2022-11-12 11:50:24

[SOLVED] How can I incrementally build *-git AUR packages?

#2 2022-11-12 12:17:47

Re: [SOLVED] How can I incrementally build *-git AUR packages?

#3 2022-11-12 13:52:29

Re: [SOLVED] How can I incrementally build *-git AUR packages?

#4 2022-11-13 15:50:39

Re: [SOLVED] How can I incrementally build *-git AUR packages?

#5 2022-11-13 16:46:25

Re: [SOLVED] How can I incrementally build *-git AUR packages?

#6 2022-11-13 16:54:18

Re: [SOLVED] How can I incrementally build *-git AUR packages?

Board footer