aurutils - helper tools for the aur

Alad · 2016-03-27 03:14:27

Why another AUR helper

With the AUR4 update quartering available AUR helpers, I found none which fitted the following criteria:

- Be modular, instead of monolithic
- Handle most user PKGBUILDs reliably
- Use a local repository

As well as few more uncommon features, such as topological sorting, PCRE searches, and devtools (systemd-nspawn) support.

In 2016, there's a lot of tools to simplify managing packages in Arch, so "starting late" has its benefits. Taking a modular approach also allows uses outside the AUR, such as building package chains from github.

Why a local repository

Some benefits of using a repository, rather than foreign packages:

- It handles the issue of installing split packages trivially
- pacman --files support
- All packages work with --asdeps
- Multiple repos for different tasks, e.g. version control packages
- Allows to easily distribute packages across machines
- AUR packages can be updated without having them installed
- You get to learn the methods of Arch Linux package maintainers

How to use it

Two packages are available on the AUR: aurutils and aurutils-git. I've kept the PKGBUILD simple, so you may not need all dependencies (for example, aurqueue only relies on coreutils and awk). See aurutils(7) for example uses, and details on setting up a local repository.

Caveats

The documentation needs improvements, but I've tried to keep the code clear and accessible. See the issue tracker for progress:

https://github.com/AladW/aurutils/issues

Cheers

Last edited by Alad (2016-05-18 14:12:25)

Alad · 2016-04-09 13:32:51

aurutils is at 0.5.1 now. Some highlights since the initial release:

git diffs are now displayed together with build files
Update local repo packages in one step with aursync -u
Validation of tsort input via datamash
pkgbase, long format and raw support for aursearch
Ability to ignore packages with aursync -i
New script: aurstranger (update wrapper for other helpers)
Improved error reporting and general reliability

Last edited by Alad (2016-04-09 13:40:46)

Spyhawk · 2016-04-10 20:29:54

Still no love for aurutils out there? Well, let me be the first to add a comment in this thread then.

Aurutils has been handily crafted after careful analysis of existing helpers and thorough consideration of what a "helper done right" should be. Helpers come and go, aurutils is still quite young but I'm pretty sure it will be around for a very long time. Definitely worth a look.

Alad · 2016-04-14 11:46:47

Thanks spyhawk. Listening to common complaints on IRC also helped much.

I wanted to try asciinema, so I've made two screencasts previewing the 0.7.0 release:

https://ptpb.pw/t/t58p
https://ptpb.pw/t/jBt1

runical · 2016-04-14 13:55:04

I finally got around to testing this and it works quite well (I already had a local repo set up, but this works better than my solution). Only the documentation seems to be lacking. Any way I could help with that?

Alad · 2016-04-14 14:10:13

Yeah, there's an open issue for that:

https://github.com/AladW/aurutils/issues/76

Would be great to get some help on this (feel free to use whatever format you prefer, I can typeset it to groff).

runical · 2016-04-14 14:17:04

I'll see what I can do later today then. Seems pandoc also handles groff, so that should be alright then

Earnestly · 2016-04-14 15:17:24

Try to avoid converting a format to groff with tools like pandoc, it will make a mess of it. Just use whatever format you like and it'll be converted into a style defined by man-pages(7), no problem.

runical · 2016-04-14 16:01:45

Noted. I'll write things in markdown then and look into man-pages then. Thanks for the warning.

Alad · 2016-04-19 13:44:05

0.7.0 was released:

https://github.com/AladW/aurutils/issue … s%3Aclosed

* man pages for most programs (thanks to runical)
* improved error reporting and structure (thanks to Earnestly)
* repofind -c removes local repo packages from /var/cache/pacman (built packages are kept in PKGDEST)
* repofind -u warns on local repo packages which are not in the AUR
* repoman respects packages in IgnorePkg and IgnoreGroup
* aurchain warns on dependencies that are both missing in AUR and the official repositories
* aurbuild passes all options after -- to either makepkg or makechrootpkg

Alad · 2016-05-11 19:44:09

0.9.0 was released. This is a pre-release and currently requires aria2-git (1.23 should be released May 15th). Testing/bug reports welcome.

https://github.com/AladW/aurutils/releases/tag/0.9.0

Note: aursync now assumes the package pool is in /var/cache/pacman/<reponame>/. If you use it, see the new man page aurutils(7) for information on how to migrate.

Last edited by Alad (2016-05-11 19:47:11)

runical · 2016-05-16 20:21:00

Hey Alad, I appreciate the work you are putting in, but is there a solid reason why you would need to use two directories for one repository? This makes making a repository available across multiple computers a less straight cut use case (especially for me, as I want to sync the repo to several computers).

Last edited by runical (2016-05-16 20:21:18)

Alad · 2016-05-16 21:06:30

repose creates symbolic links in the root of the repository to all packages, or reflinks if you prefer. Are the packages not accessible this way? As to the reasoning, see:

https://github.com/AladW/aurutils/releases/tag/0.9.0
https://github.com/AladW/aurutils/issues/129

It solved the issue of mismatches between pool and cache when rebuilding (same pkgver) in an elegant way, and you get some added benefits like pacman -Sc removing old versions, or packages not being stored twice twice (e.g. for packages like Darkmod at 2.6 GB this would be especially noticeable).

That said, this is only "enforced" in aursync, and disabling it means commenting a single line, replacing with pool=$root :

https://github.com/AladW/aurutils/blob/ … rsync#L190

However as I removed the code which deleted cached packages while building (makepkg installs needed dependencies during the process), you'd have to increment pkgrel with setconf when rebuilding. See:

https://github.com/AladW/aurutils/issues/125

If you have a better suggestion to the above problematic, I'm happy to hear it as well.

Last edited by Alad (2016-05-16 21:11:58)

Alad · 2016-05-17 02:54:25

Actually, it might be possible to have both the root and pool in /var/cache/pacman/foo. If this actually works, please open a github issue.

Last edited by Alad (2016-05-17 02:55:14)

runical · 2016-05-17 09:46:12

I'll take a look tonight. It just struck me as strange to separate the pool from the database. Thanks for the quick overview!

Alad · 2016-05-17 11:23:18

It works for me:

https://github.com/AladW/aurutils/commi … e85ad85a9c

So when you test, please do so with the latest git checkout.

Alad · 2016-05-18 11:23:37

I've went ahead and released 1.0.0, which also includes GPG signing.

As above, aursync now assumes Server matches CacheDir; when updating from 0.8.x, just uncomment CacheDir in /etc/pacman.conf, and add a second entry CacheDir = /path/to/your/repo.

CleanMethod = KeepCurrent is also a good idea to easily remove old versions from the pool with pacman -Sc.

Last edited by Alad (2016-05-18 11:23:46)

runical · 2016-05-18 15:46:00

I didn't get around to testing yesterday, but the new version works great! I have added the repositories as CacheDirs and everything works, including the syncing. Thanks for the work you are putting in!

beelzebub · 2016-05-19 19:46:38

installed it and used it today...really nice work

Alad · 2016-05-28 05:19:58

Thanks, glad I didn't break it.

1.0.1 was released.

https://github.com/AladW/aurutils/releases/tag/1.0.1

Note that with systemd 230 release, you now need to add a machine id to an existing nspawn container:

sudo systemd-machine-id-setup --root=/var/lib/aurbuild/root

Last edited by Alad (2016-05-28 05:32:03)

Alad · 2016-07-28 15:02:00

1.2.3 now supports i686 containers on x86_64 (or generally, any supported architecture by setarch) with aurbuild -m <arch>. To avoid creating a new container for the current architecture, move the files:

# mkdir /var/lib/aurbuild/$(uname -m)
# mv /var/lib/aurbuild/* /var/lib/aurbuild/$(uname -m)

Last edited by Alad (2016-07-28 15:28:02)

PUE · 2016-08-27 21:04:04

I tried to use aurutils, but without success so far. Can you please write up a 'getting started' guide?
I followed the man page and ended up with the following error when I tried to use aursync -c (after installing the required 'devtools'):

ERROR: '/var/lib/aurbuild/x86_64/root' does not appear to be an Arch chroot.

I tried to specify CHROOT (according to the devtools wiki page), but as far as I could tell, it had no effect.

I tried without '-c' but aborted when aursync showed me the dependencies. On each subsequent run, aursync claimed that the package was already up-to-date. The only solution I could come up with was to delete the cache directory and create it again. Back to square one.

I was hoping for a better cower.

Alad · 2016-08-27 21:59:46

Well, man 7 aurutils is supposed to be the getting started guide...

Regarding the error, it may happen when the chroot wasn't set up properly (e.g., the process was interrupted). Note that aurbuild - which does the builds for aursync - ignores the CHROOT variable.

Please remove the /var/lib/aurbuild/ directory, and post the output of the following:

pacconf
bash -x aursync <insert aur packages you want to build>
bash -x aursync -c <insert aur packages you want to build>

Last edited by Alad (2016-08-27 22:00:39)

PUE · 2016-08-28 14:07:42

Thanks Alad, simply removing /var/lib/aurbuild/ did the trick.
I did follow 'man 7 aurutils' but it still seems a bit overwhelming for a simple user.
How the wrong setup occured I do not know, it is a fresh Arch install and I only follwed instructions.
The improper package that showed up after I aborted the build also worries me a bit and smells like a bug. After all, I aborted before installing dependencies, yet there was a package in the cache.

Alad · 2016-08-30 02:25:32

I did follow 'man 7 aurutils' but it still seems a bit overwhelming for a simple user.

The page uses terminology similar to repose, so it might help to read man repose first. Help:Reading might also help, as I follow a style similar to the wiki's. If you have particular suggestions on any parts that are unclear, feel free to share them.

The improper package that showed up after I aborted the build also worries me a bit and smells like a bug. After all, I aborted before installing dependencies, yet there was a package in the cache.

Well it's hard to tell what it is if you don't give me some logs, like the ones mentioned above. At this time, I don't even know which AUR package we're talking about.

Arch Linux

#1 2016-03-27 03:14:27

aurutils - helper tools for the aur

#2 2016-04-09 13:32:51

Re: aurutils - helper tools for the aur

#3 2016-04-10 20:29:54

Re: aurutils - helper tools for the aur

#4 2016-04-14 11:46:47

Re: aurutils - helper tools for the aur

#5 2016-04-14 13:55:04

Re: aurutils - helper tools for the aur

#6 2016-04-14 14:10:13

Re: aurutils - helper tools for the aur

#7 2016-04-14 14:17:04

Re: aurutils - helper tools for the aur

#8 2016-04-14 15:17:24

Re: aurutils - helper tools for the aur

#9 2016-04-14 16:01:45

Re: aurutils - helper tools for the aur

#10 2016-04-19 13:44:05

Re: aurutils - helper tools for the aur

#11 2016-05-11 19:44:09

Re: aurutils - helper tools for the aur

#12 2016-05-16 20:21:00

Re: aurutils - helper tools for the aur

#13 2016-05-16 21:06:30

Re: aurutils - helper tools for the aur

#14 2016-05-17 02:54:25

Re: aurutils - helper tools for the aur

#15 2016-05-17 09:46:12

Re: aurutils - helper tools for the aur

#16 2016-05-17 11:23:18

Re: aurutils - helper tools for the aur

#17 2016-05-18 11:23:37

Re: aurutils - helper tools for the aur

#18 2016-05-18 15:46:00

Re: aurutils - helper tools for the aur

#19 2016-05-19 19:46:38

Re: aurutils - helper tools for the aur

#20 2016-05-28 05:19:58

Re: aurutils - helper tools for the aur

#21 2016-07-28 15:02:00

Re: aurutils - helper tools for the aur

#22 2016-08-27 21:04:04

Re: aurutils - helper tools for the aur

#23 2016-08-27 21:59:46

Re: aurutils - helper tools for the aur

#24 2016-08-28 14:07:42

Re: aurutils - helper tools for the aur

#25 2016-08-30 02:25:32

Re: aurutils - helper tools for the aur

Board footer