aurutils - helper tools for the aur

ngoonee · 2020-10-16 01:10:23

A bit of a... conceptual? question

I've been using aurutils for a while now on my laptop (the sole active Arch machine I use, the rest are just servers or the like where I hardly need AUR packages). Now I have a new desktop which I'll be using more actively and will need AUR packages (including duplicates of what's on my laptop).

Both the laptop and desktop will not be running 24/7, so sharing packages may be a bit of a pain. Both are fast enough that compiling on machine isn't an issue, but if I have already compiled something I'd like to just be able to install it.

I do have a raspberry pi sitting around doing some service hosting, with a huge hard disk. I guess hosting a repo on that would be the right way to go?

So my current setup (duplicated on two machines) is:-
Compilation happens in ~/.cache/aurutils/sync
The repo itself is in /var/cache/pacman/aurrepo

The setup I envision is:-
Compilation happens in ~/.cache/aurutils/sync
The repo itself is hosted on my raspberry pi

Would this work? Internet connection is fast (300Mbps) so I'm not too concerned about sharing .cache/aurutils/sync between my laptop and desktop. Is it better for the repo to be just a shared folder (samba) or some properly hosted folder behind nginx?

Alad · 2020-10-17 10:28:05

ngoonee wrote:

Would this work? Internet connection is fast (300Mbps) so I'm not too concerned about sharing .cache/aurutils/sync between my laptop and desktop. Is it better for the repo to be just a shared folder (samba) or some properly hosted folder behind nginx?

You could update the files on the rpi with rsync/ssh after each build, and then use an HTTP server. If you use something like samba, you have to take care of symlinks:

https://github.com/AladW/aurutils/issues/700

HeptaSean wrote:

There is some processing happening, which is described by man aur-chroot:

The only processing aur-chroot does is read file:// entries and pass them as bind-rw arguments to makechrootpkg and arch-nspawn. All the strange edits made to the pacman configuration is by complex logic inside devtools:

https://git.archlinux.org/devtools.git/ … awn.in#n67

There is no way to disable this functionality:

https://bugs.archlinux.org/task/38641
https://bugs.archlinux.org/task/27544

Last edited by Alad (2020-10-17 10:30:11)

ngoonee · 2020-10-18 00:14:12

Alad wrote:

ngoonee wrote:
Would this work? Internet connection is fast (300Mbps) so I'm not too concerned about sharing .cache/aurutils/sync between my laptop and desktop. Is it better for the repo to be just a shared folder (samba) or some properly hosted folder behind nginx?
You could update the files on the rpi with rsync/ssh after each build, and then use an HTTP server.

I'll just ignore the samba option then. Looks like I can either:-

1. Host a proper repo with some http server (I'd still need to push to it though)

or

2. Sync /var/cache/pacman/aurrepo between the machines regularly.

Both of these need a bit more busy work than I like. Would something like sshfs mounting of /var/cache/pacman/aurrepo from both machines to a folder on the pi work as well? Repos are self-contained folders (with internal symlinks)?

progandy · 2020-10-18 00:32:47

ngoonee wrote:

I'll just ignore the samba option then. Looks like I can either:-

Doesn't samba also have some options to support symlinks? mfsylinks or sfu mount parameter. The arm server doesn't have need to follow the symlink anyways, right?

Alad · 2020-10-18 19:02:36

It should work with the correct mount options, see the last comment in the linked issue.

Slithery · 2020-10-18 19:07:07

ngoonee wrote:

I'll just ignore the samba option then. Looks like I can either:-
1. Host a proper repo with some http server (I'd still need to push to it though)
or
2. Sync /var/cache/pacman/aurrepo between the machines regularly.
Both of these need a bit more busy work than I like. Would something like sshfs mounting of /var/cache/pacman/aurrepo from both machines to a folder on the pi work as well? Repos are self-contained folders (with internal symlinks)?

Setting up a web server takes literally seconds. Install and enable nginx, upload the repo to /usr/share/nginx/html/ and you're done.

ngoonee · 2020-10-20 01:45:48

Slithery wrote:

ngoonee wrote:
I'll just ignore the samba option then. Looks like I can either:-
1. Host a proper repo with some http server (I'd still need to push to it though)
or
2. Sync /var/cache/pacman/aurrepo between the machines regularly.
Both of these need a bit more busy work than I like. Would something like sshfs mounting of /var/cache/pacman/aurrepo from both machines to a folder on the pi work as well? Repos are self-contained folders (with internal symlinks)?
Setting up a web server takes literally seconds. Install and enable nginx, upload the repo to /usr/share/nginx/html/ and you're done.

The local setup on a machine is fine. I hate the reverse proxying part (as I already have a few web services and pages running on a Pi for my home network so I'd have to figure out traefik syntax for redirecting the right URL...)

Easiest for me now is just syncthing on the repo. Will get round to web server at some point (maybe).

Alad · 2020-11-09 13:40:08

aurutils 3 has been released. For an overview of changes, see:

https://pkgbuild.com/~alad/aurutils_3.html

After initial feedback the current version is aurutils 3.1.2 with kinks ironed out and chroot pacman.conf defaulting to /etc/aurutils/pacman-<repository>.conf.

sshaikh · 2020-11-27 18:33:50

After falling into the Arch rabbit hole I've faced what is probably a common situation for those with itchy fingers - how to manage the ever increasing number of custom PKGBUILDs based off those existing in the ABS and AUR. I'm sold on the idea of aurutils and local repos, so thank you for these tools. I have some questions regarding the use of aurutils as well as the workflow of which it will be a part of. I need someone to check my homework, or alternatively whack me over the head with the over-designed hammer of doom.

1. The organisation of local repos. The man page for aur(1) encourages the use of multiple repos for different purposes, eg VCS packages. Why is that? In the absence of knowing why, I've identified the following "classes" and will create local repos for each (even though some if not all will remain sparse). Although I understand that I will need a different strategy to update packages in these classes, I don't understand why they need to live in separate package repos too:

a) non-vcs AUR PKGBUILDs
b) vcs AUR PKGBUILDs
c) modified (by me) non-vcs AUR PKGBUILDs
d) modified vcs AUR PKGBUILDs
e) modified non-vcs ABS PKGBUILDs
f) modified vcs ABS PKGBUILDs (do these exist?)

2. For c-d, managing updates to AUR PKGBUILDs. aur-sync aside (which I don't want to use), I suspect this is "out of scope" for aurutils but I'm interested in how others have solved this. My general plan is to have some kind of "update-merge" check. This is easy enough for the AUR as each package has its own repo, and mirroring and forking repos is git's forte. I will end up with a repo per PKGBUILD, just like the AUR, which is nice, and know when I need to do a merge.

3. For e-f, managing updates to ABS PKGBUILDS. I'm still figuring out how to do it for these, but it will probably involve using asp to maintain mirrors of repos, either one per ABS PKGBUILD, or one for all of them - the former will make them look like my AUR repos (again, nice), the latter will more closely resemble the ABS PKGBUILD repos which is easier, but means handling the update and build steps differently.

4. For b, d and f, I will need a vcs check too. This is where I'm starting to spiral a bit as I see there are strategies and tools (eg aur-srcver) for particular situations but none which can cover all three. In my mind it shouldn't really make a difference where the PKGBUILD came from. EDIT: although now I understand that aur-srcver only needs source dirs (containing PKGBUILDs?) so maybe that is actually enough for all cases.

I guess the reason why I consider this an aurutils question is that since I want to use that as the final step before pacman, it would be of value to have a "standard interface/organisation of PKGBUILDs" that can be fed into whatever script I'll be using to invoke aurutils, so I'm seeking advice as to what that standard organisation should look like. I suspect "one repo per PKGBUILD" would work the best, but I see different approaches elsewhere.

I suppose that if I was just interested in aur things I would use aur-sync, but a) I don't think it would work with all the usecases and b)... where's the fun in that?

Alad · 2020-11-28 15:50:56

Maybe I should remove that remark on multiple repositories since it seems to lead more to confusion than enlightening ideas. Anyway:

1. One reason for keeping VCS packages in a separate repo is that it gives an exact definition on what a VCS package is (the package suffix does not necessarily, see https://bugs.archlinux.org/task/56602): it is whatever you put in the VCS repo. That again can be used with aur-srcver.

Keeping ABS packages in a separate repo allows you to put it above any other repositories in pacman.conf, which is something you wouldn't want to do with an AUR repo.

I'd not classify further than the above. For modified AUR pkgbuilds, it's unfortunate pacman/vercmp supports no "~user" prefix like dpkg but as ABS pkgbuilds, you can use a custom group (e.g group=custom) to distinguish them.

2. You can use the underlying tools aur-sync uses, i.e. aur-fetch, aur-vercmp, aur-repo, aur-depends. You can also use git submodules or git subtree.

3. archweb has an API you can query for package updates. I use kittypack to query it: https://aur.archlinux.org/packages/kittypack/

4. aur-srcver covers basic needs but it doesn't scale well to a high amount of VCS packages. See also https://github.com/AladW/aurutils/pull/617

I suppose that if I was just interested in aur things I would use aur-sync, but a) I don't think it would work with all the usecases and b)... where's the fun in that?

Finally someone gets it.

Last edited by Alad (2020-11-28 15:57:14)

sshaikh · 2020-11-28 20:47:04

Thank you for the informative reply. I've since made some progress on tracking modified upstream PKGBUILDs wherever they are (for AUR, mirror the repo, for ABS script this: https://bbs.archlinux.org/viewtopic.php … #p1940448).

I noticed that aur-vercmp-devel had disappeared. aurvcs looks good and is actually how I planned to script my own check.

Regarding multiple repos, your explanation makes sense. It didn't occur to me to use the contents of a repo as an index of what packages are being built (I was going to use a list somewhere). I'll follow your suggestion and stick to ABS, AUR (custom or otherwise) and AUR-VCS (custom or otherwise) repos.

I don't have any custom ABS-VCS PKGBUILDs yet so I'll omit that for now. I'm still not sure they exist, but I suppose that would be another repo if they do.

sshaikh · 2020-12-03 17:43:17

I have built and deposited a pair of packages to a local repo. This was from a split package, and I renamed both the pkgnames and pkgbase to make it clear this was built from a custom PKGBUILD.

I'm now trying to (programatically) update the packages, but can't figure out how to determine the basepkg from the list given by "aur repo" (so I know what to build).

"aur sync" appears to do this by querying the AUR, but this is obviously not an option for non-AUR packages. But does this imply that I'm to keep my own registry of PKGBUILDs to check?

If so, I'm starting to wonder why we keep separate repos for VCS builds if we need to maintain external metadata anyway (I don't mind that per se, just wondering once more what the value of separate VCS repos are)

Last edited by sshaikh (2020-12-03 17:47:04)

Alad · 2020-12-04 10:01:12

There's no need to maintain external metadata. If you need pkgbase, you can retrieve that from the local repo, e.g. with aur repo --table.

zpg443 · 2020-12-04 12:26:35

sshaikh wrote:

I suppose that if I was just interested in aur things I would use aur-sync, but a) I don't think it would work with all the usecases and b)... where's the fun in that?

I get the point, but it is also nice to run sync with noview, especially when you have no use cases requiring custom builds and you rarely encounter an install fail. Simpler just to deal with exceptions when they arise.

Who needs fun when easy and functional is good enough?

aur sync -u --noview

Alad, thanks for aurutils.

Last edited by zpg443 (2020-12-04 12:27:41)

Alad · 2020-12-05 12:18:32

Just a reminder that AUR code is fully untrusted and has full access to your user account and root (i.e. through PKGBUILD and pacman install files, respectively). At minimum, you should check that the maintainer of the PKGBUILD has not changed before building an update. See e.g. the infamous case of acroread being orphaned and a maintainer adopting it and adding malware.

sshaikh · 2020-12-13 19:26:07

Are packages created with a normal "makepkg" supposed to be identical to those created by aur build?

I have created a package based off the PKGBUILD here https://aur.archlinux.org/packages/xrdp-git/. I have included an extra patch and changed the package name.

However when I build with makepkg vs aur build (using a chroot), I have the following diff in files:

diff aurbuild makepkg
6,9d5
< etc/default/
< etc/default/xrdp
< etc/init.d/
< etc/init.d/xrdp
82a79,82
> usr/lib/systemd/
> usr/lib/systemd/system/
> usr/lib/systemd/system/xrdp-sesman.service
> usr/lib/systemd/system/xrdp.service

I'm assuming it's the chroot that's doing something but I can't quite figure out how to fix it - or more to the point whether the chroot is supposed to behave this way.

EDIT:

So! Seems like the upstream PKGBUILD has a typo that causes the systemd units not to be created. Fixing the typo fixes it. Why it works outside of a chroot I don't know (perhaps some kind of autodetect of systemd), but it seems that the chroot was doing its job and breaking correctly.

Last edited by sshaikh (2020-12-13 20:42:50)

zpg443 · 2020-12-13 19:29:32

Alad wrote:

Just a reminder that AUR code is fully untrusted and has full access to your user account and root (i.e. through PKGBUILD and pacman install files, respectively). At minimum, you should check that the maintainer of the PKGBUILD has not changed before building an update. See e.g. the infamous case of acroread being orphaned and a maintainer adopting it and adding malware.

Appreciate the caution and sound advice. I was unaware of the acroread event.

Unb0rn · 2021-01-05 08:29:41

After upgrading to 3+ (3.1.2) version is there any way to pass makepkg.conf into aur-sync for chroot build?
I think it used to be -M /path/to/makepkg.conf and -C /path/to/pacman.conf, now pacman.conf is static and is taken from /etc/aurutils/pacman-<somesuffix>.conf but what about makepkg?
Man gives the same -M option for aur-chroot, but no such options for aur-sync...

Alad · 2021-01-05 08:52:11

aur-sync supports any aur-build options, and aur-build --makepkg-conf passes the option on to aur-chroot. I didn't document this explicitly in man aur-sync, but it's mentioned in man aur-build.

You can also set a different path for the pacman configuration with aur-build --pacman-conf.

Looks like I forgot to mention the respective short options are only in aur-chroot now.

Last edited by Alad (2021-01-05 08:56:30)

Unb0rn · 2021-01-05 08:55:28

Alad wrote:

aur-sync supports any aur-build options, and aur-build --makepkg-conf passes the option on to aur-chroot. I didn't document this explicitly in man aur-sync, but it's mentioned in man aur-build.

Oh, great! Thanks!

merlock · 2021-01-10 00:17:59

Alad wrote:

aur-sync supports any aur-build options

For the life of me, I can't figure that out.

aur build -d custom -c -f -S --root $repo -D $CHROOT -N --margs n,s,r,clean

Builds packages in the chroot, and installs them into my custom repo, then I manually install the package using pacman -Syu.

What I can't figure out is how to use that build string using aur-sync (which, I think, would eliminate the extra pacman install step).

Last edited by merlock (2021-01-10 03:16:47)

Alad · 2021-01-10 16:31:17

There seem to be some redundant flags in that above command-line. -n is an aur-build option but not a makepkg option; makechrootpkg already runs makepkg -s --noconfirm; and r/clean are covered by makechrootpkg -c which syncs the container template on each build. -N is implied by -c and apparently the one option above not wrapped by aur-sync.

When using margs you should also use hyphens, as the original makepkg options.

Builds packages in the chroot, and installs them into my custom repo, then I manually install the package using pacman -Syu.

When using -c, you always have to use pacman -Syu - that's the idea when building in a container. If you don't build in a container, you still have to run pacman -S, unless the package was already installed on the host. (aur-build includes a mechanism to run an -Syu "restricted" to your local repo.)

So you have the following options:

aur sync -d custom -cS --root "$repo" -D "$CHROOT" <targets>
pacman -Syu <targets>

aur sync -d custom -S --root "$repo" <targets>
pacman -S <targets> # on first install

Note I left out -f in the above. If you want to leave out version checks with aur-sync, you should use --rebuild (or -f --no-ver-argv).

Last edited by Alad (2021-01-10 16:32:41)

merlock · 2021-01-10 18:48:45

Thanks, Alad. I really appreciate it.

ngoonee · 2021-03-27 22:18:37

I realize this is more of a pacman question than aurutils, but I'm not able to find a way to update only one repo (and I guess there wouldn't be such a way since in most cases it would lead to partial upgrades). This seems like it would make sense with aurutils in particular...

Alad · 2021-03-27 22:33:41

aur-build already does something like that by using a pacman.conf with only a single repo inside:

https://github.com/AladW/aurutils/blob/ … #L371-L374

Arch Linux

#401 2020-10-16 01:10:23

Re: aurutils - helper tools for the aur

#402 2020-10-17 10:28:05

Re: aurutils - helper tools for the aur

#403 2020-10-18 00:14:12

Re: aurutils - helper tools for the aur

#404 2020-10-18 00:32:47

Re: aurutils - helper tools for the aur

#405 2020-10-18 19:02:36

Re: aurutils - helper tools for the aur

#406 2020-10-18 19:07:07

Re: aurutils - helper tools for the aur

#407 2020-10-20 01:45:48

Re: aurutils - helper tools for the aur

#408 2020-11-09 13:40:08

Re: aurutils - helper tools for the aur

#409 2020-11-27 18:33:50

Re: aurutils - helper tools for the aur

#410 2020-11-28 15:50:56

Re: aurutils - helper tools for the aur

#411 2020-11-28 20:47:04

Re: aurutils - helper tools for the aur

#412 2020-12-03 17:43:17

Re: aurutils - helper tools for the aur

#413 2020-12-04 10:01:12

Re: aurutils - helper tools for the aur

#414 2020-12-04 12:26:35

Re: aurutils - helper tools for the aur

#415 2020-12-05 12:18:32

Re: aurutils - helper tools for the aur

#416 2020-12-13 19:26:07

Re: aurutils - helper tools for the aur

#417 2020-12-13 19:29:32

Re: aurutils - helper tools for the aur

#418 2021-01-05 08:29:41

Re: aurutils - helper tools for the aur

#419 2021-01-05 08:52:11

Re: aurutils - helper tools for the aur

#420 2021-01-05 08:55:28

Re: aurutils - helper tools for the aur

#421 2021-01-10 00:17:59

Re: aurutils - helper tools for the aur

#422 2021-01-10 16:31:17

Re: aurutils - helper tools for the aur

#423 2021-01-10 18:48:45

Re: aurutils - helper tools for the aur

#424 2021-03-27 22:18:37

Re: aurutils - helper tools for the aur

#425 2021-03-27 22:33:41

Re: aurutils - helper tools for the aur

Board footer