aurutils - helper tools for the aur

ngoonee · 2018-04-24 22:37:42

Ah the conversation there has sprung back to life. Will follow github developments then.

TaZeR · 2018-04-27 10:14:36

Wow aurutils is amazing, once you get over the initial setup you realize what it does and setting up a local repository for your aur packages and having them managed by pacman's regular -S, Q, etc.. commands instead of the ugly -U forcing foreign package. Thank you for the amazing hard work and dedication your passion for the archlinux aur truly shows.

tmike · 2018-05-03 14:14:30

Thanks for putting together aurutils! I've been looking for a halfway decent way to manage AUR packages, and these tools have been extremely helpful.

However, I recently switched to aurutils-git, and I'm now running into some oddities:

==> Leaving fakeroot environment.
==> Finished making: gonvim 0.22-1 (Thu May  3 10:02:27 EDT 2018)
==> Cleaning up...
==> Extracting database to a temporary location...
bsdtar: Removing leading '/' from member names
==> ERROR: Repository file '/var/cache/pacman/custom//custom.files.tar' is not a proper pacman database.

It doesn't matter what package, whether it's a -git package. They all do it. I can't even upgrade aurutils-git to see if that fixes it

Any thoughts on how I can give aurutils a nudge to get it to behave again?

Alad · 2018-05-03 17:28:03

TaZeR wrote:

Wow aurutils is amazing, once you get over the initial setup you realize what it does and setting up a local repository for your aur packages and having them managed by pacman's regular -S, Q, etc.. commands instead of the ugly -U forcing foreign package. Thank you for the amazing hard work and dedication your passion for the archlinux aur truly shows.

Cheers

tmike wrote:

Any thoughts on how I can give aurutils a nudge to get it to behave again?

Take a look at this post: https://bbs.archlinux.org/viewtopic.php … 9#p1707649

PS: While aurutils is very stable, both in the "it doesn't break" and the "it doesn't change" sense, aurutils-git is the complete opposite. You're more than welcome to use it, but I wanted to give a fair warning in any case.

Last edited by Alad (2018-05-03 17:29:12)

Lucki · 2018-05-03 22:31:55

Hi Alad,

I'm using aurutils-git and have some problems with --makepkg-conf. Following the man page it should be used in the following way but fails:

LANG=C GPGKEY=xxxxx aur sync -cs --repo aur --makepkg-conf=/etc/makepkg.conf aurutils-git
sync: option '--makepkg-conf' doesn't allow an argument

Using the same with --makepkg-conf FILE continues but gives an error later on and also ignores every other argument given after --makepkg-conf.

LANG=C GPGKEY=xxxxx aur sync -csf --repo aur --no-ver --makepkg-conf /etc/makepkg.conf aurutils-git
  -> Using [aur] repository

[…]

cp: missing destination file operand after '/var/lib/aurbuild/x86_64/root/etc/makepkg.conf'
Try 'cp --help' for more information.

[…]

Is this known or expected behavior?

Alad · 2018-05-04 02:50:08

Fixed with c5656082b2030b31a12e8febadc8d309dffccb6f

TaZeR · 2018-05-04 08:24:18

Alad wrote:

Support for repose is removed in master, see https://github.com/AladW/aurutils/pull/238.
As repose is incompatible to repo-add, I'm giving people some time to rebuild their databases before tagging a new release. To do so, run:
cd /var/cache/pacman/myrepo/
rm myrepo.db* myrepo.files*
repo-add -n myrepo.db.tar *.pkg.tar.xz
If you want to maintain automatic database management, you can switch to a compatible tool such as repoctl:
https://github.com/cassava/repoctl

When do you think this will be coming to the release version, and how will we be informed on when to perform these changes on managing the database with ripose/repoctl? Also can we switch from ripose to repoctl right now on the release version in order to get ready for the future change?

Alad · 2018-05-04 09:49:32

Yes, you can just uninstall repose to have the release version work with repo-add.

Unb0rn · 2018-05-04 18:37:54

Hello! Had some reading on aurutils and it seems like a great idea, but there are no guides at all. (Although, those who are able to use man should have no problems)
I am here for a little advice, as it is possible that someone tried something like this. It may be a good idea to write a little guide on building images in nspawn containers (and even better-on BTRFS or ZFS)
We can create a BTRFS subvolume, mount it like in /var/lib/machines/<something>, populate it as a rootfs for nspawn container with all the basic dependencies (and even configure auto updating via cron or something), and then "fork" our building containers from it, with systemd-nspawn
--ephemeral, making use of BTRFS powerful features like snapshots, eliminating the need to re-download all the dependencies or store everything in one single big container or even need to "clean up", as newly created subvolume would be automatically destroyed after shutdown.
Are there any "best practices"? Where to mount nspawn images (/var/lib/machines or somewhere else), what packages to use in base container, anything else? Thanks!

ngoonee · 2018-05-05 00:26:57

Unb0rn wrote:

Hello! Had some reading on aurutils and it seems like a great idea, but there are no guides at all. (Although, those who are able to use man should have no problems)

Uh... the man page is a step-by-step guide... Unless you're saying that that content should be duplicated somewhere (with obvious problems in syncing with latest version etc.).

Man pages ARE the canonical place to look for documentation.

Alad · 2018-05-05 11:11:38

Yes, read the man pages. These have had significant effort put in by me and other community members.

We can create a BTRFS subvolume (...)

What you're describing is already close to what makechrootpkg does, though with rsync instead of systemd --ephemeral. If you want it to use latter, you'll need to bring it up with arch-projects since makechrootpkg is part of devtools and not aurutils.

Last edited by Alad (2018-05-05 11:12:31)

Cody Learner · 2018-05-06 02:47:57

Unb0rn wrote:

Hello! Had some reading on aurutils and it seems like a great idea, but there are no guides at all. (Although, those who are able to use man should have no problems)
I am here for a little advice, as it is possible that someone tried something like this. It may be a good idea to write a little guide on building images in nspawn containers (and even better-on BTRFS or ZFS)
We can create a BTRFS subvolume, mount it like in /var/lib/machines/<something>, populate it as a rootfs for nspawn container with all the basic dependencies (and even configure auto updating via cron or something), and then "fork" our building containers from it, with systemd-nspawn
--ephemeral, making use of BTRFS powerful features like snapshots, eliminating the need to re-download all the dependencies or store everything in one single big container or even need to "clean up", as newly created subvolume would be automatically destroyed after shutdown.
Are there any "best practices"? Where to mount nspawn images (/var/lib/machines or somewhere else), what packages to use in base container, anything else? Thanks!

This may have some of what you're asking about. https://github.com/Cody-Learner/Arch-Image

eschwartz · 2018-05-06 03:23:52

Unb0rn wrote:

Hello! Had some reading on aurutils and it seems like a great idea, but there are no guides at all. (Although, those who are able to use man should have no problems)
I am here for a little advice, as it is possible that someone tried something like this. It may be a good idea to write a little guide on building images in nspawn containers (and even better-on BTRFS or ZFS)
We can create a BTRFS subvolume, mount it like in /var/lib/machines/<something>, populate it as a rootfs for nspawn container with all the basic dependencies (and even configure auto updating via cron or something), and then "fork" our building containers from it, with systemd-nspawn
--ephemeral, making use of BTRFS powerful features like snapshots, eliminating the need to re-download all the dependencies or store everything in one single big container or even need to "clean up", as newly created subvolume would be automatically destroyed after shutdown.
Are there any "best practices"? Where to mount nspawn images (/var/lib/machines or somewhere else), what packages to use in base container, anything else? Thanks!

There's no need to write a guide, this is already the case even if people don't try doing anything in particular.

The base container *requires* the base-devel package, and nothing else. This is automatically accomplished via the devtools, specifically mkarchroot. extra-x86_64-build uses mkarchroot and directs both that and makechrootpkg to use /var/lib/archbuild by default, but you can use whatever you want and IIRC aurutils sets /var/lib/aurbuild.

Note that running cron jobs doesn't really accomplish much, since you could simply ensure the base image is updated when you start building things, this slows down the first use of the nspawn container but subsequent uses require no updating... and you don't need to constantly hit the network for hefty downloads, when you're not even building anything that day.

packages do not need to be redownloaded as they use the pacman cache of the host machine, which is bind-mounted into the nspawn container and used for persistent download caching, most of the time simply reusing the packages you installed/upgraded on your main system.

Cleaning up the chroot after building doesn't seem especially useful, as it is usually quite fast to clean up right before you use it, and makechrootpkg is designed for flexibility so you can rebuild in the same dirty chroot if you're just building the same thing again, or if you're building a connected set of packages. I don't think --ephemeral is very well suited for this... it does support btrfs transparently which is nice, but deleting the chroot immediately after exiting it is problematic since e.g. if the build fails we explicitly tell you to go look in the /build directory of the chroot to investigate what happened, inspect the logs from /logdest, etc. Also we don't move the logs and built packages until after we know everything is okay, rather than mounting /pkgdest and /logdest directly from the host system and potentially clobbering interesting stuff by broken, failed builds.

Arch does not support zfs. At all.

Alad wrote:

We can create a BTRFS subvolume (...)
What you're describing is already close to what makechrootpkg does, though with rsync instead of systemd --ephemeral. If you want it to use latter, you'll need to bring it up with arch-projects since makechrootpkg is part of devtools and not aurutils.

makechrootpkg autodetects btrfs filesystems and uses btrfs subvolume instead of rsync.

Cody Learner · 2018-05-07 22:30:52

Hypothetically speaking:

I have an AUR package that I prefer to build manually, say ttf-ms-win10. I gather the requirements, run 'updpkgsums', 'makepkg -s'

Now I have a built package ready to install. I'd prefer to add it to my local aur database, so copy the package to </path/to/local/aur/repo>

Then run 'repose aur' from within local aru repo, 'pacman -Syyu,' ' pacman -S ttf-ms-win10'

Now it's as if I used 'aurt -S' to build and install it.

Other than modifying the PKGBUILD to do some of the above when prompted during the install,
am I missing a more straight forward way to have ttf-ms-win10 installed as a local aur package?

aurt -S :

install () {

	aursync --no-confirm "${pkglist[@]}"
	sudo pacman -S --noconfirm --color=always "${pkglist[@]}"

	verify=$(pacman -Qq "${pkglist[@]}"|fmt -1000)

if 	[[ "${verify}" = "${pkglist[*]}" ]]; then

	. aurt-info --installprint1
    else
	. aurt-info --installprint2
fi
}

EDIT: Add info

Did a bit of testing and came up with this after the post. Probably use some variation of it in aurt as a new '-U ' operation. Unless I am missing something that could simplify things.

PKG=$(find -maxdepth 1 | awk -F/  '/pkg.tar/ {print $2}' | awk -F- '{print $1}')
cp *.pkg.tar* /var/cache/pacman/aur 2>/dev/null || { echo ":: ERROR: Must run in directory containing <package>.pkg.tar.gz" ; exit ; }
echo ":: Installing $PKG "
repose -p /var/cache/pacman/aur -r /var/cache/pacman/aur aur
sudo pacsync aur &>/dev/null
sudo pacman -S $PKG

Last edited by Cody Learner (2018-05-08 04:58:41)

Alad · 2018-05-08 08:50:31

Just run aurbuild from a directory containing a PKGBUILD instead of makepkg and all this is handled for you.

In any case I would recommend moving to repo-add/repoctl since aurutils 2 will not support repose anymore.

Last edited by Alad (2018-05-08 08:55:43)

lamarpavel · 2018-05-18 21:08:05

Hi. I've been playing around with aurutils and wanted to thank you, Alad, for creating it. It's a fascinating project and I enjoyed using it so far.

A situation I am not entirely clear on is how to influence the repositories available to pacman within a container (spawned by aurbuild).
Specifically, I would like to build mesa-git, but avoid having to build llvm-svn. Before using aurutils I added a custom repo to my main pacman.conf

[...]
[llvm-svn]
Server = http://repos.uni-plovdiv.net/archlinux/$repo/$arch
[...]

But now that I have the power of aurutils at my disposal, I would like to get rid of that extra repo in my hostmachines pacman.conf and only have it in the container that is building mesa-git.
From reading the manpages I gathered that the container is created with a copy of the hosts pacman.conf, and changes to the containers pacman.conf may be overwritten.
However, I am not clear on when things are overwritten in the container. Is there a preferred way to keep a different pacman.conf in the container?
I'm using aurutils stable, not -git.

And again, thanks for creating these tools.

Alad · 2018-05-18 21:55:22

The -git version has a flag for a custom pacman config, see https://github.com/AladW/aurutils/blob/ … ot#L55-L57

If you can test it and ensure it works 100% I can probably backport it to the 1.5 version

Last edited by Alad (2018-05-18 21:55:46)

lamarpavel · 2018-05-19 07:40:42

Alad wrote:

The -git version has a flag for a custom pacman config, see https://github.com/AladW/aurutils/blob/ … ot#L55-L57
If you can test it and ensure it works 100% I can probably backport it to the 1.5 version

I'm still figuring out some of the basics using the 1.5 release and I can't predict the effect of switching to the -git release. Could that screw with my already set-up repos? How easy is it to roll back to stable afterwards (I've read in this thread that you consider the development branch to be very unstable, so I'd rather not use that for more than this test.

I guess I could just try aurutils-git on a different machine, though...

In any case, aur-chroot seems to be new in this version, do I have to call it manually or will it be called by aur-build? And if so, can I pass the -C through aur-build to aur-chroot?
Disregard that, I should have just read the manpage.

Edit: Not sure if I'm misunderstanding the manpage or if there is a bug; I'm running into the following problem:
Calling

aur build -d aurpak -c -a /tmp/queue -- -C /var/cache/pacman/aurpak/custom_pacman.conf

results in ==> ERROR: Failed to change to directory /var/cache/pacman/aurpak//home/lamarpavel/.cache/aurutils/sync/mesa-git

The content of /tmp/queue is /home/lamarpavel/.cache/aurutils/sync/mesa-git and that directory is the result of

aur fetch mesa-git

To check that the repo is set up properly, I tried

aur sync -d aurpak --pacman-conf=/var/cache/pacman/aurpak/custom_pacman.conf -c bgs-git

and that worked without a hitch.

Last edited by lamarpavel (2018-05-19 09:03:38)

Alad · 2018-05-19 13:43:50

How easy is it to roll back to stable afterwards

If you didn't install repo-add it's equivalent. Otherwise AURDEST has a different default value script names/options have changed for the most part.

results in ==> ERROR: Failed to change to directory /var/cache/pacman/aurpak//home/lamarpavel/.cache/aurutils/sync/mesa-git

In this case you've used the end-of-options marker -- with -c, so you've passed on -C to makechrootpkg instead of aur-chroot; simply leave it out.

aur build -d aurpak -c -a /tmp/queue -C /var/cache/pacman/aurpak/custom_pacman.conf

Last edited by Alad (2018-05-19 13:44:36)

lamarpavel · 2018-05-19 16:37:30

Alad wrote:

In this case you've used the end-of-options marker -- with -c, so you've passed on -C to makechrootpkg instead of aur-chroot; simply leave it out.
aur build -d aurpak -c -a /tmp/queue -C /var/cache/pacman/aurpak/custom_pacman.conf

I'm still getting the same error:

$ aur build -d aurpak -c -a /tmp/queue -C /var/cache/pacman/aurpak/custom_pacman.conf
[sudo] password for lamarpavel:
:: Synchronizing package databases...
 core is up to date
 extra                                                                                     1605.2 KiB  3.65M/s 00:00 [######################################################################] 100%
 community                                                                                    4.4 MiB  4.95M/s 00:01 [######################################################################] 100%
 aurpak is up to date
:: Starting full system upgrade...
 there is nothing to do
container path: /var/lib/aurbuild/x86_64
/usr/share/makepkg/util/util.sh: line 80: cd: /var/cache/pacman/aurpak//home/lamarpavel/.cache/aurutils/sync/mesa-git: No such file or directory
==> ERROR: Failed to change to directory /var/cache/pacman/aurpak//home/lamarpavel/.cache/aurutils/sync/mesa-git
    Aborting...

Alad · 2018-05-19 17:01:38

What are the contents of the /tmp/queue file and what directory are you running aur build from?

Last edited by Alad (2018-05-19 17:02:21)

lamarpavel · 2018-05-19 17:15:21

Alad wrote:

What are the contents of the /tmp/queue file and what directory are you running aur build from?

$ cat /tmp/queue
/home/lamarpavel/.cache/aurutils/sync/mesa-git

$ pwd
/var/cache/pacman/aurpak

edit: I think we are on the right track, if I execute it from my home directory, I get a different prefix to the path:

==> ERROR: Failed to change to directory /home/lamarpavel//home/lamarpavel/.cache/aurutils/sync/mesa-git

Last edited by lamarpavel (2018-05-19 17:17:51)

Alad · 2018-05-19 18:03:57

Right, aur-build does the following:

cd <directory from which aur-build was run>/<name of line in queue file>

So it works if you run it from the parent directory that contains AUR repos (like ~/.cache/aurutils/sync) and feed it a queue file only containing the package names.

I should probably improve it to not rely on this

Alad · 2018-05-19 18:20:40

Please try this patch: https://paste.xinu.at/Y891G/

lamarpavel · 2018-05-19 20:26:22

Alad wrote:

Please try this patch: https://paste.xinu.at/Y891G/

Indeed, this patch solves that problem.

The build is now getting to the point I was initially trying to reach (build mesa-git but using the specified pacman.conf).
However, the pacman.conf that I've specified with -C isn't being used (or not properly used?). In that file I have

[llvm-svn]
Server = http://repos.uni-polvdiv.net/archlinux/$repo/$arch

but the build is aborted with

$ aur build -d aurpak -c -a /tmp/queue -C /var/cache/pacman/aurpak/custom_pacman.conf
:: Synchronizing package databases...
 core is up to date
 extra is up to date               0.0   B  0.00B/s 00:00 [-------------------------------]   0%
 community is up to date           0.0   B  0.00B/s 00:00 [-------------------------------]   0%
 aurpak                           11.0 KiB  0.00B/s 00:00 [###############################] 100%
:: Starting full system upgrade...
 there is nothing to do
container path: /var/lib/aurbuild/x86_64
==> Synchronizing chroot copy [/var/lib/aurbuild/x86_64/root] -> [lamarpavel]...done
:: Synchronizing package databases...
 core is up to date                0.0   B  0.00B/s 00:00 [-------------------------------]   0%
 extra is up to date
 community is up to date
 aurpak is up to date
:: Starting full system upgrade...
 there is nothing to do
==> Making package: mesa-git 18.2.0_devel.102242.dace607245-3 (Sat May 19 22:13:10 CEST 2018)
==> Retrieving sources...
  -> Cloning mesa git repo...
Cloning into bare repository '/home/lamarpavel/.cache/aurutils/sync/mesa-git/mesa'...
remote: Counting objects: 1003781, done.
remote: Compressing objects: 100% (148209/148209), done.
remote: Total 1003781 (delta 857494), reused 995623 (delta 850590)
Receiving objects: 100% (1003781/1003781), 175.49 MiB | 4.56 MiB/s, done.
Resolving deltas: 100% (857494/857494), done.
  -> Found LICENSE
==> Validating source files with sha512sums...
    mesa ... Skipped
    LICENSE ... Passed
==> Making package: mesa-git 18.2.0_devel.102242.dace607245-3 (Sat May 19 22:14:45 CEST 2018)
==> Checking runtime dependencies...
==> Installing missing dependencies...
error: target not found: llvm-libs-svn
==> ERROR: 'pacman' failed to install missing dependencies.
==> ERROR: Build failed, check /var/lib/aurbuild/x86_64/lamarpavel/build
==> ERROR: build: a package failed to build ()
remaining packages: (none)
successfully processed packages: (none)

I see two possible causes right now: Either the specified custom_pacman.conf isn't used, or it's used but the pacman.db in the container isn't updated accordingly (eg. I have to do sudo pacman -Syy after editing the pacman.conf when doing it manually)

Arch Linux

#176 2018-04-24 22:37:42

Re: aurutils - helper tools for the aur

#177 2018-04-27 10:14:36

Re: aurutils - helper tools for the aur

#178 2018-05-03 14:14:30

Re: aurutils - helper tools for the aur

#179 2018-05-03 17:28:03

Re: aurutils - helper tools for the aur

#180 2018-05-03 22:31:55

Re: aurutils - helper tools for the aur

#181 2018-05-04 02:50:08

Re: aurutils - helper tools for the aur

#182 2018-05-04 08:24:18

Re: aurutils - helper tools for the aur

#183 2018-05-04 09:49:32

Re: aurutils - helper tools for the aur

#184 2018-05-04 18:37:54

Re: aurutils - helper tools for the aur

#185 2018-05-05 00:26:57

Re: aurutils - helper tools for the aur

#186 2018-05-05 11:11:38

Re: aurutils - helper tools for the aur

#187 2018-05-06 02:47:57

Re: aurutils - helper tools for the aur

#188 2018-05-06 03:23:52

Re: aurutils - helper tools for the aur

#189 2018-05-07 22:30:52

Re: aurutils - helper tools for the aur

#190 2018-05-08 08:50:31

Re: aurutils - helper tools for the aur

#191 2018-05-18 21:08:05

Re: aurutils - helper tools for the aur

#192 2018-05-18 21:55:22

Re: aurutils - helper tools for the aur

#193 2018-05-19 07:40:42

Re: aurutils - helper tools for the aur

#194 2018-05-19 13:43:50

Re: aurutils - helper tools for the aur

#195 2018-05-19 16:37:30

Re: aurutils - helper tools for the aur

#196 2018-05-19 17:01:38

Re: aurutils - helper tools for the aur

#197 2018-05-19 17:15:21

Re: aurutils - helper tools for the aur

#198 2018-05-19 18:03:57

Re: aurutils - helper tools for the aur

#199 2018-05-19 18:20:40

Re: aurutils - helper tools for the aur

#200 2018-05-19 20:26:22

Re: aurutils - helper tools for the aur

Board footer