aurutils - helper tools for the aur

johnramsden · 2017-12-30 00:35:21

I'm really impressed with this tool and how it can build packages in containers. I like the idea of keeping build dependencies separate from the final packages. Thanks for your work on this great utility Alad.

I'm wondering what the recommended way would be to migrate all packages that one might have had installed from the AUR to the local repository. Is there a recommended way to do this besides getting a list of all installed AUR packages and rebuilding, and reinstalling them?

eschwartz · 2017-12-31 02:01:31

If your previous AUR packages were saved via makepkg.conf(5) PKGDEST, or alternatively via some AUR helper that offers a separate option for saving the built package files, then all you need to do is move the the saved packages to the custom repo directory configured via aurutils, and `repo-add` the old files.

aurutils will then check the new custom repo entries for updates, just like any other AUR package in its repo.

If you allowed the older packages to be purged, for example building and storing them in /tmp and allowing them to be deleted at the next reboot, then you will have to rebuild them with aurutils in order to add them to your custom repo.

johnramsden · 2017-12-31 02:16:32

Okay thanks, I'll just get a list and rebuild them. I was using pacaur and purging the older packages.

Ichimonji10 · 2018-01-01 19:51:56

This is a nice set of tools, Alad. Thanks for making aurutils.

Several folks (e.g. in this thread and on aurutils #247) have expressed a desire for more or different "getting started" documentation. I understand that aurutils(7) is supposed to fulfill that role, and in the end, it worked for me. But I think I understand where other folks are coming from. I think people are wanting a story, e.g. "aurutils can populate a local repository with packages from the AUR, which pacman can then install. Here's how that might be done..."

Just something to chew on.

I whipped up an Ansible role that packages and installs software from the AUR, using aurutils. Y'all might find it of interest. I'm interested in hearing what I've done wrong.

bulletmark · 2018-01-01 23:21:12

Ichimonji10 wrote:

"aurutils can populate a local repository with packages from the AUR, which pacman can then install."

Oh, so that's what aurutils does!?

Ichimonji10 · 2018-01-01 23:33:53

It's one possible use. They could also be used to populate a remote repository. Or they could be used piecemeal by another application. Or something else.

EDIT: But yes, that's how I use it.

Last edited by Ichimonji10 (2018-01-02 01:32:16)

bgdawes · 2018-01-05 04:51:13

I realize this is a dumb question, before I ask it, but can anyone let me know if I'm missing any steps to completely remove a package? I took the following steps below:

# pacman -Rsn foo

$ repo-remove /var/cache/pacman/custom/custom.db.tar foo

# pacman -Syu

Everything looks great after these steps, package is uninstalled and removed from my custom repository. However, I still see remnants of the package in:

/var/lib/aurbuild/x86_64/username/build

/var/lib/aurbuild/x86_64/username/build/usr/share

/var/lib/aurbuild/x86_64/username/build/var/lib/pacman/local

~/.cache/aursync/foo

Any obvious suggestions about what I'm doing wrong?

eschwartz · 2018-01-05 06:23:27

~/.cache contains the cached PKGBUILD files, in case you decide to redownload and rebuild the package. /var/lib/aurbuild is the transient build directory used for chroot builds; as soon as you build another package with aurutils, it will be purged and used to create the other package -- at which point the reference will go away.
If you want, you can delete the package from ~/.cache/aursync.

tl;dr You didn't miss anything, nor did you do anything wrong.

phw · 2018-01-05 13:57:58

Just wanted to say thanks for these tools. I had previously been using pacaur and manual git clone / makepkg for some packages, and when I first heard about aurutils maintaining a local repository felt a bit overkill. But after having a fresh new install on a new laptop I decided to got with aurutils, and I must say I love it. Due to the setup required and multiple separate tools it is a bit more work to get into it then other aur helpers or even basic manual building, but having the separate tools is quite flexible.

And a custom repository is awesome. I frequently switch between packages I test, and having e.g. both the stable and -git package in the repository makes it super easy to switch between them (as opposed to looking for the package file on my disk and / or rebuilding the package).

bgdawes · 2018-01-05 20:03:39

Thanks for the response Eschwartz! I also really appreciate the background for context as well. I'm still relatively new to the linux world and this is very helpful!

ewaller · 2018-01-07 18:13:16

Alad, I am loving these tools. Thank you.

One question. I note that when I sync things that are from source management systems (git, svn, hg) they sync (I beleive) to the HEAD of the source tree, not the version advertised by the AUR tools. I am on board with this, but what is the behavior of aursync/aurcheck when the HEAD moves, but the AUR is not updated? Do things rebuild in this case, or need this be performed manually?

ewaller@turing ~ 1050 %aurcheck -d my_foreign -a | grep \<\-
                                                   
avrdude-svn 20171129.1399-1 <- 20160328.1388-1
chessx-svn 1.4.8.2420-1 <- 1.3.4.2124-1
lemonbar-xft-git 268.043ad47-1 <- 245.38f69d8-1
libfprint-git 1:0.7.0.r7.gd35da0c-1 <- 1:0.7.0.r4.g69de32f-1
ltspice 4.23k-1 <- 4.23h-1
pyside2-common-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
python-pyside2-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
python-shiboken2-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
python2-pyside2-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
python2-shiboken2-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
shiboken2-git 2.0.0.r5383.2f4bfa56-1 <- 2.0.0.r5325.ad14f649-1
urxvt-resize-font-git 14.97c91ec-1 <- 10.2bbde29-1
ewaller@turing ~ 1051 %

Having recently changed over, I just don't have enough data points to see a pattern based on my observations.

Last edited by ewaller (2018-01-07 18:16:01)

eschwartz · 2018-01-07 19:41:29

aurutils behaves no different from pacaur or yaourt in this respect, in that aursync -u will only detect available updates if the AUR is updated.

Both pacaur and yaourt have --devel flags which essentially say, "try updating the package anyway if it matches some arbitrary heuristic which is probably a VCS package".
aurutils does not (yet) implement special handling of VCS packages. Instead, you can explicitly list them using `aursync --no-ver "${pkglist[@]}"`.

Also see https://github.com/AladW/aurutils/issues/227 and https://github.com/AladW/aurutils/issues/65

ewaller · 2018-01-07 20:33:55

Thanks Eschwartz, Makes sense.
... And congrats on becoming a TU.

Lucki · 2018-01-09 15:38:21

Hi Alad,
I'm trying out these tools and everything worked out so far. But there's something I can't figure out which is related to issue 242 when building in a container an unedited makepkg.conf is used:
The issue is about aursync and you've implemented -M in aurbuild. How can I set the makepkg.conf that aursync should use or forward to aurbuild? I think I need this to set an additional -T0 switch for faster xz compression and to set my GPG-Key while using aursync -cs. (With the stable package it was possible to use GPGKEY=XXXXX aursync -cs, that's not possible with the git version anymore.)

Anyway, thanks for these interesting tools!

carlitos · 2018-01-09 21:29:15

Hi guys,

I've just recently moved from pacaur and now am getting to know these excellent tools Alad made. One question, if a package was updated with aursync -u in my custom repo, why am I still seeing the old pkg? I saw output like removing existing entry (in .db I suppose ) run paccache -rvvvk1 but it is still there. repo-add -R did not help either ( file cannot be found - so it's not in repo db, right? )
Can I simply delete the old pkg at this point?

Thanks

eschwartz · 2018-01-09 21:57:35

Yes, the old package files can be removed. repo-add does not remove them by default, and neither does aurutils (what would you do if you needed to downgrade for some reason?)

carlitos · 2018-01-09 22:31:50

Thanks for the clarification.
I thought tools like paccache ( and pkgcacheclean ) could clean up my custom repo as well as my regular pacman repo. [ I always do paccache -rvvvk1 ( and use downgrade/r if needed or snapper snapshots ) for regular pkgs ] I keep latest 3 of each my custom package in the cloud -to save space and practice cli rclone.

Thanks

Last edited by carlitos (2018-01-09 22:32:46)

eschwartz · 2018-01-09 23:09:40

paccache only cleans directories that are configured as a pacman.conf CacheDir. Which I do for my custom repo, but if you only use it as Server = file:/// then pacman will "download" the package extremely fast and make a copy in /var/cache/pacman/pkg/ and paccache will only see the extra copy.

Having it clean out your custom repo is not one of its design goals, just a happy coincidence.

...

The other option is to use repo-add -R at the time it initially adds the new version. Which would require modifying aurutils itself, as that is not currently exposed via an option.

Alad · 2018-01-10 16:12:11

SolarAquarion wrote:

I'm trying to build habs and i'm having issues with aurqueue, specifically wanting to build the PKGBUILDs in the pkgbuild directories that are created

Circular dependency. Fix your pkgbuilds.

Eschwartz wrote:

If you allowed the older packages to be purged, for example building and storing them in /tmp and allowing them to be deleted at the next reboot, then you will have to rebuild them with aurutils in order to add them to your custom repo.

Should not be required unless the installed packages are damaged, see paccheck + bacman.

Eschwartz wrote:

The other option is to use repo-add -R at the time it initially adds the new version. Which would require modifying aurutils itself, as that is not currently exposed via an option.

The amount of options to aursync/aurbuild is already daunting...

Eschwartz wrote:

aurutils does not (yet) implement special handling of VCS packages. Instead, you can explicitly list them using `aursync --no-ver "${pkglist[@]}"`.
Also see https://github.com/AladW/aurutils/issues/227 and https://github.com/AladW/aurutils/issues/65

I'd rather see https://bugs.archlinux.org/task/56602 fixed than implement hacks client-side.

Lucki wrote:

How can I set the makepkg.conf that aursync should use or forward to aurbuild? I think I need this to set an additional -T0 switch for faster xz compression and to set my GPG-Key while using aursync -cs.

https://github.com/AladW/aurutils/commi … 4de482e129

Lucki wrote:

(With the stable package it was possible to use GPGKEY=XXXXX aursync -cs, that's not possible with the git version anymore.)

Not sure why it would no longer be possible, can you provide some detail?

https://github.com/AladW/aurutils/blob/ … build#L109

Lucki · 2018-01-10 19:27:01

Hi Alad,
thanks for your answer and fast implementation!

I've done some further testing and found an error at your linked line 109. There's

gpg --listkeys

which should actually be

gpg --list-keys

. That's why GPGKEY=XXXXX aursync -cs don't work anymore.

However, if I'm using aursync -cs -M /etc/makepkg.conf where makepkg.conf has the same key set inside as GPGKEY="XXXXX" (and correctly used by makepkg --sign) the wrong key gets selected. Maybe I'm wrong here but shouldn't the same key be used here as with the other command above?
So for your troubleshooting:
1)-4) all ok here, I'm using aurutils-git to test
5) Maybe? Do I have to do special configuration inside the container?

Logs (--list-keys already fixed):

LC_ALL=C aursync -csf --repo aur --nover -M /etc/makepkg.conf aurutils-git |& tee error.log

LC_ALL=C GPGKEY=xxx aursync -csf --repo aur --nover -M /etc/makepkg.conf aurutils-git |& tee error.log

Last edited by Lucki (2018-01-10 20:03:18)

Alad · 2018-01-10 23:50:12

Lucki wrote:

I've done some further testing and found an error at your linked line 109.

Wow, not sure how that one slipped through the cracks... Good catch!

https://github.com/AladW/aurutils/commi … b00ac2391d

Lucki wrote:

However, if I'm using aursync -cs -M /etc/makepkg.conf where makepkg.conf has the same key set inside as GPGKEY="XXXXX" (and correctly used by makepkg --sign) the wrong key gets selected. Maybe I'm wrong here but shouldn't the same key be used here as with the other command above?

You're not wrong, there's however 3 methods available for signing:

1. makepkg --sign
2. manual signing
3. repo-add --sign

Method 2 is used because of FS #49946 which is fixed in pacman 5.1. Method 3 does not respect configuration in makepkg.conf.

I guess you could parse the given makepkg.conf for GPGKEY (using pacini), but then you'd have to consider even more cases: makepkg.conf given through -M or any of the standard locations used by makepkg. It may be better to add something to the man page instead.

Tracked in: https://github.com/AladW/aurutils/issues/272

Last edited by Alad (2018-01-10 23:58:11)

Lucki · 2018-01-11 01:00:42

So, if I understand that right:
You're using method 2 at the moment and want to switch to method 1 when pacman 5.1 with the fix ships out.
So this will be some time on hold. Got it!

Thanks for the explanation

Edit: Seems like I missed that method 3 is also used a few lines later. So setting the key as enviroment variable will also be needed with pacman 5.1 and the change to makepkg --sign…
If makepkg tells us which key is used to sign it would also be possible to catch the output and give that key with --key <key> to repo-add - instead of manually scraping the config file.

Last edited by Lucki (2018-01-11 01:28:18)

eschwartz · 2018-01-11 02:55:06

Alad wrote:

Eschwartz wrote:
If you allowed the older packages to be purged, for example building and storing them in /tmp and allowing them to be deleted at the next reboot, then you will have to rebuild them with aurutils in order to add them to your custom repo.
Should not be required unless the installed packages are damaged, see paccheck + bacman.

That will be difficult once pacman 5.1 removes it, since it has not been ported to pacman-contrib.

Alad wrote:

Eschwartz wrote:
The other option is to use repo-add -R at the time it initially adds the new version. Which would require modifying aurutils itself, as that is not currently exposed via an option.
The amount of options to aursync/aurbuild is already daunting...

I make no judgment calls on whether it *should*, merely on what the concept would entail.

Alad wrote:

Eschwartz wrote:
aurutils does not (yet) implement special handling of VCS packages. Instead, you can explicitly list them using `aursync --no-ver "${pkglist[@]}"`.
Also see https://github.com/AladW/aurutils/issues/227 and https://github.com/AladW/aurutils/issues/65
I'd rather see https://bugs.archlinux.org/task/56602 fixed than implement hacks client-side.

If only I really understood php well enough to implement this (or most anything else, really)...

chicha · 2018-01-11 10:14:07

Hello,

Do you plan to add recursive aur dependency handling within aursync ?
Else is

aurchain foobar | while read -r pkg _; do aursync -c $pkg; done

the recommended way to build an aur package and all its related aur dependencies, please ?

Ichimonji10 · 2018-01-11 16:28:31

aursync does recursive AUR dependency handling. To build $pkg and all of its AUR dependencies, the following should suffice:

aursync "$pkg"

Arch Linux

#101 2017-12-30 00:35:21

Re: aurutils - helper tools for the aur

#102 2017-12-31 02:01:31

Re: aurutils - helper tools for the aur

#103 2017-12-31 02:16:32

Re: aurutils - helper tools for the aur

#104 2018-01-01 19:51:56

Re: aurutils - helper tools for the aur

#105 2018-01-01 23:21:12

Re: aurutils - helper tools for the aur

#106 2018-01-01 23:33:53

Re: aurutils - helper tools for the aur

#107 2018-01-05 04:51:13

Re: aurutils - helper tools for the aur

#108 2018-01-05 06:23:27

Re: aurutils - helper tools for the aur

#109 2018-01-05 13:57:58

Re: aurutils - helper tools for the aur

#110 2018-01-05 20:03:39

Re: aurutils - helper tools for the aur

#111 2018-01-07 18:13:16

Re: aurutils - helper tools for the aur

#112 2018-01-07 19:41:29

Re: aurutils - helper tools for the aur

#113 2018-01-07 20:33:55

Re: aurutils - helper tools for the aur

#114 2018-01-09 15:38:21

Re: aurutils - helper tools for the aur

#115 2018-01-09 21:29:15

Re: aurutils - helper tools for the aur

#116 2018-01-09 21:57:35

Re: aurutils - helper tools for the aur

#117 2018-01-09 22:31:50

Re: aurutils - helper tools for the aur

#118 2018-01-09 23:09:40

Re: aurutils - helper tools for the aur

#119 2018-01-10 16:12:11

Re: aurutils - helper tools for the aur

#120 2018-01-10 19:27:01

Re: aurutils - helper tools for the aur

#121 2018-01-10 23:50:12

Re: aurutils - helper tools for the aur

#122 2018-01-11 01:00:42

Re: aurutils - helper tools for the aur

#123 2018-01-11 02:55:06

Re: aurutils - helper tools for the aur

#124 2018-01-11 10:14:07

Re: aurutils - helper tools for the aur

#125 2018-01-11 16:28:31

Re: aurutils - helper tools for the aur

Board footer