aurutils: building in a chroot, signing packages, adding to local repo

MountainX · 2020-10-21 01:15:54

EDIT: please skip down to My Steps. I'm hoping to determine if they are correct.

I've been using extra-x86_64-build (as described here) and other places to build AUR packages for my own use (and to share with my team in a shared local repo).

I am able to build simple packages just fine, but packages with complex dependencies (such as some dependencies that are in the regular repos and some dependencies that are in AUR give me trouble. nordvpn-bin, pcloudcc and x2godesktopsharing are a couple examples that come to mind, but there are others. Therefore, I'm now learning about and exploring aurutils.

In the case of x2godesktopsharing, I'm getting the same error with aurutils that I got with my script using extra-x86_64-build. (I also left a comment at the project AUR page.)

  -> Extracting x2godesktopsharing-3.2.0.0.tar.gz with bsdtar
==> Starting build()...
Info: creating stash file /build/x2godesktopsharing/src/x2godesktopsharing-3.2.0.0/.qmake.stash
Project ERROR: Unknown module(s) in QT: svg
==> ERROR: A failure occurred in build().
    Aborting...
==> ERROR: Build failed, check /mnt/chroots/test1/extra-x86_64/myuser/build

I tried this solution by adding the missing dependency to the PKGBUILD, but it did not help.

depends=('x2goserver>=4.0.1' 'qt5-base' 'qt5-svg')

I'm wondering if I am doing something wrong or if there is something wrong with this package.

My steps

I prepared a fairly detailed set of steps. These might be useful to other noobs who are interested in using aurutils to build in a chroot, sign packages and add them to a local repository. However, it would be very useful for me (and, I assume, others) to know if these steps are all correct. Here they are:

Preliminary general steps:
1. read wiki pages (such as [https://wiki.archlinux.org/index.php/Creating\_packages](https://wiki.archlinux.org/index.php/Creating_packages))
2. install dev tools. Some of these (e.g., repoctl) are optional:

sudo pacman -Syu --needed base-devel devtools namcap shellcheck haveged repoctl
sudo systemctl status haveged # make sure it is active

Preliminary GPG steps (if you don't already have a key configured:

gpg --full-gen-key #see man gpg
gpg --armor --output mygpg.key --export-keys YOURKEYFINGERPRINT
sudo pacman-key -a mygpg.key
sudo pacman-key --finger YOURKEYFINGERPRINT
sudo pacman-key --lsign-key YOURKEYFINGERPRINT

Preliminary signing steps:

Edit /etc/makepkg.conf:

BUILDENV=(!distcc color !ccache check sign) # make sure sign is active; by default it is not.
PACKAGER="Your Name <you@email.tld>"
GPGKEY="YOURKEYFINGERPRINT" # use your key full fingerprint uppercase, no whitespace

Define my locations, etc.

chrootpath="mnt/chroots/test1"
CHROOT="${chrootpath}/extra-x86_64"
reporoot="path/to/repos"
pkgsroot="path/to/packages"
reponame="test1" # I use the same name for the project top level directory and the respository directory (and the repository db).
echo $pkgsroot # check it

I wanted to keep the tmpfs and chroot locations the same as they had been with extra-x86_64-build.

Create and mount RAM disk if it doesn't already exist (optional)

devtempfs() {
    echo "mounting tmpfs at /${chrootpath}..."
    if [[ ! -d /${chrootpath} ]] ; then 
      sudo install -dm755 /${chrootpath} -o $USER
    fi;
    findmnt /${chrootpath};
    if [[ $? -ne 0 ]]; then
      sudo mount -t tmpfs -o defaults,size=20G tmpfs /${chrootpath};
    fi
    xbarg="-r /${chrootpath}"; #used by some other devtools
    echo "finished mounting tmpfs at /${chrootpath}."

    # optional:
    # make sure the base chroot (/$CHROOT/root) is up to date: 
    arch-nspawn /"$CHROOT"/root pacman -Syu
}
devtempfs

Create the project root, repository root and repo database:

sudo install -dm751 /"$pkgsroot"/"$reponame" -o $USER
sudo install -dm751 /"$reporoot"/"$reponame" -o $USER
repo-add /"$reporoot"/"$reponame"/"$reponame".db.tar.gz #https://wiki.archlinux.org/index.php/Pacman/Tips_and_tricks#Custom_local_repository
  # alternative if using repoctl:
  # repoctl conf new /"$reporoot"/"$reponame"/"$reponame".db.tar.gz 
  # then edit the config file ... (see man page)
ls -la /"$reporoot"/"$reponame"
sudo install -d /var/cache/pacman/"$reponame" -o $USER
mount --bind /"$reporoot"/"$reponame" /var/cache/pacman/"$reponame"
mount -o remount,bind,ro /"$reporoot"/"$reponame" /var/cache/pacman/"$reponame"

Edit pacman.conf

sudo tee -a /etc/pacman.conf >/dev/null << ENDPC

[$reponame]
Server = file:///$reporoot/\$repo/
ENDPC

Obtain the PKGBUILDs and other required files for the AUR packages of interest.

cd /"$pkgsroot"/"$reponame"
for project in anydesk aurutils cherrytree firefox-extension-arch-search geteltorito hsxkpasswd joplin kim4 nordpy nordvpn-bin openaudible-bin pcloudcc perl-file-share qgnomeplatform repoctl scrcpy speedtest-cli-git systemd-kcm ungoogled-chromium vivaldi x2godesktopsharing yay zfs-linux zoom zotero; do \
  git clone https://aur.archlinux.org/"$project".git; \
done
tree /"$pkgsroot"/"$reponame"

This key is needed for zfs-linux:

$ gpg --recv-keys 6AD860EED4598027
gpg: key 6AD860EED4598027: public key "Tony Hutter (GPG key for signing ZFS releases) <hutter2@llnl.gov>" imported
gpg: Total number processed: 1
gpg:               imported: 1

The main steps are:

cd /"$pkgsroot"/"$reponame"
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'pwd; makepkg --noprepare -od; echo -e "\n";' \; #(effectively running an existing pkgver() function.
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'makepkg --printsrcinfo > .SRCINFO' \; 
aur graph */.SRCINFO | tsort | tac > tobuild.txt # TODO: manually edit if you wish to exclude any packages
xargs -a tobuild.txt aur sync --chroot --sign --upgrades --noconfirm --database="$reponame" --directory=/$CHROOT

Be sure to read the aurutils man pages for more info.

As a side note, I am trying to decide whether I need this in the first find command above: "makepkg --noprepare -od".
Or whether a simple "git pull" is sufficient.

One example in the aurutils man pages suggests that "git pull" is OK. But I have run into package version mismatches with VCS packages, so I liked the idea of "effectively running an existing pkgver() function." Any thoughts on this?

Last edited by MountainX (2020-10-21 20:13:30)

MountainX · 2020-10-21 20:09:02

My confusion comes down to this:

The steps I listed in the OP give me problems. I have tried the usual approaches with both extra-x86_64-build and aurutils, and when I build in a chroot, about half the packages I try to build fail (usually due to dependency problems).

However, I created my own chroot using "pacstrap -i" (similar to the Arch install steps, but leaving unnecessary things out). Then I chrooted into this env and I was able to build all those same packages without errors.

I listed my steps above in detail. Did I miss anything or did I get something wrong?

Alad · 2020-10-23 18:13:27

Obtain the PKGBUILDs and other required files for the AUR packages of interest.

You can also use aur fetch to do some of the heavy lifting, including git rebase if you need it.

The steps I listed in the OP give me problems. I have tried the usual approaches with both extra-x86_64-build and aurutils, and when I build in a chroot, about half the packages I try to build fail (usually due to dependency problems).

For the dependency problems, most AUR packages are not tested in (devtools) chroots, and thus don't build in chroots. In my experience that's about ~90-95% of AUR packages.

The extra-x86_64-build and aurutils chroot only includes base-devel by default. Perhaps yours included some additional packages.

https://github.com/AladW/aurutils/blob/ … ot#L80-L87

As a side note, I am trying to decide whether I need this in the first find command above: "makepkg --noprepare -od".

As indicated you'd only use it for packages with a pkgver function, like VCS packages. Generally this has no impact on dependencies or their order (in tobuild.txt) - those are defined in depends/makedepends.

This is outside of pathological PKGBUILDs using versioned dependencies that depend on pkgver.

cd /"$pkgsroot"/"$reponame"
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'pwd; makepkg --noprepare -od; echo -e "\n";' \; #(effectively running an existing pkgver() function.
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'makepkg --printsrcinfo > .SRCINFO' \; 
aur graph */.SRCINFO | tsort | tac > tobuild.txt # TODO: manually edit if you wish to exclude any packages
xargs -a tobuild.txt aur sync --chroot --sign --upgrades --noconfirm --database="$reponame" --directory=/$CHROOT

In the last step, I guess you meant aur build instead of aur sync.

MountainX · 2020-10-23 22:51:32

Thanks so much for looking at my question, Alad. From now until Monday morning I plan to do nothing but learn more about building all kinds of packages in chroots, especially AUR packages that depend on both other AUR packages and official Arch packages, and my own custom packages. I plan to post several follow ups on this topic.

Alad wrote:

The extra-x86_64-build and aurutils chroot only includes base-devel by default. Perhaps yours included some additional packages.

Yes, I did include some additional packages that I am 100% sure will be present on all the systems I'm responsible for. I wonder if I could go into $CHROOT/root and make some changes similar to what I did when I created my own chroot? Is there anything unique about the chroots created by devtools that makes modifying them a bad idea?

Alad wrote:

https://github.com/AladW/aurutils/blob/ … ot#L80-L87

You have a very nice style of writing bash code. Can you suggest some things I can read to start learning to write bash scripts the way you do? I like your bash style better than probably any I have seen before. I can implement some of your style, but you do a number of things I don't know how to do. I'd like to learn more.

Alad wrote:

cd /"$pkgsroot"/"$reponame"
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'pwd; makepkg --noprepare -od; echo -e "\n";' \; #(effectively running an existing pkgver() function.
find . -mindepth 2 -maxdepth 2 -name PKGBUILD -execdir sh -c 'makepkg --printsrcinfo > .SRCINFO' \; 
aur graph */.SRCINFO | tsort | tac > tobuild.txt # TODO: manually edit if you wish to exclude any packages
xargs -a tobuild.txt aur sync --chroot --sign --upgrades --noconfirm --database="$reponame" --directory=/$CHROOT

In the last step, I guess you meant aur build instead of aur sync.

If I understand correctly from reading the aurutils man pages, I can use "aur sync" for my AUR packages and "aur build" for my own custom packages. I've been working on both. Probably confused myself during the learning process by working with both types of packages at the same time. In the beginning I did not understand some of the differences in workflows for my custom packages vs AUR packages.

What I think I really want is something just like "aur sync" that I can use for my custom packages. Unfortunately, some of the code in aurutils is beyond my present understanding. If I create a script to do that, my code will be of much lower quality than aurutils. But I plan to give it my best shot.

Alad · 2020-10-28 17:44:29

MountainX wrote:

Yes, I did include some additional packages that I am 100% sure will be present on all the systems I'm responsible for. I wonder if I could go into $CHROOT/root and make some changes similar to what I did when I created my own chroot? Is there anything unique about the chroots created by devtools that makes modifying them a bad idea?

Not really, you can install packages inside the chroot like in any arch environment. I use arch-nspawn to do so: https://github.com/AladW/aurutils/blob/ … oot.1#L172

You have a very nice style of writing bash code. Can you suggest some things I can read to start learning to write bash scripts the way you do? I like your bash style better than probably any I have seen before. I can implement some of your style, but you do a number of things I don't know how to do. I'd like to learn more.

Thanks. I guess my bash code is somewhat unorthodox. Some things I adhere to:

* I use (or try to) use bash as a command language. That means I leave data processing to other tools and save the results in files, if necessary. Piping commands is also very easy in bash, so I use it where it makes sense. aur-sync probably makes the most use of both concepts.
* I sometimes redefine functions or commands depending on options (a rudimentary form of polymorphism). aur-search is an example.
* Maintaining large bash scripts is difficult, so I make several smaller utilities that handle specific tasks. If the utilities need to communicate, I use text (example: aur fetch --results).

Other than that it's been lots of trial and error, discarding concepts and finding new ones. I also see documenting limitations, rather than write code around them, as a viable approach.

If I understand correctly from reading the aurutils man pages, I can use "aur sync" for my AUR packages and "aur build" for my own custom packages. I've been working on both. Probably confused myself during the learning process by working with both types of packages at the same time. In the beginning I did not understand some of the differences in workflows for my custom packages vs AUR packages.
What I think I really want is something just like "aur sync" that I can use for my custom packages. Unfortunately, some of the code in aurutils is beyond my present understanding. If I create a script to do that, my code will be of much lower quality than aurutils. But I plan to give it my best shot.

In the end all of the packages will end up in (one or several) local repositories. What aur-sync does, roughly speaking, is:

* resolve AUR dependencies with aur-depends;
* retrieve (or update) the results as git repositories with aur-fetch;
* print diffs if there were changes (e.g. after an AUR update);
* show all of these in a file manager;
* build the packages with aur-build.

For your own custom packages you'd have to see which parts of the workflow you want. The parts where'd you'd sync changes from another source are probably not applicable. The file manager stuff should be fairly straightforward - I found vifm the easiest one to work with. This is the relevant part in aur-sync:

https://github.com/AladW/aurutils/blob/ … #L405-L414

Last edited by Alad (2020-10-28 17:44:57)

Arch Linux

#1 2020-10-21 01:15:54

aurutils: building in a chroot, signing packages, adding to local repo

My steps

#2 2020-10-21 20:09:02

Re: aurutils: building in a chroot, signing packages, adding to local repo

#3 2020-10-23 18:13:27

Re: aurutils: building in a chroot, signing packages, adding to local repo

#4 2020-10-23 22:51:32

Re: aurutils: building in a chroot, signing packages, adding to local repo

#5 2020-10-28 17:44:29

Re: aurutils: building in a chroot, signing packages, adding to local repo

Board footer