aurutils - helper tools for the aur

Unb0rn · 2018-07-06 18:43:19

Okay, maybe I'm not clear enough, sorry
Let's say I've built gdm-plymouth with aur sync -c gdm-plymouth.
Everything went fine and now it is added to my local repo.
Now, I want to re-build it with system's makepkg.conf (or maybe update a *-git package and the pkgver inside pkgbuild didn't change, but pkgbuild itself uses latest commit in git)
So, I'm trying aur sync -f -c --makepkg-conf=/<path> and getting "nothing to build" reply.
The link to your github issue is helpful, but according to man, aur build does not have --makepkg-conf key, so if I get everything right-it'll just rebuild the package itself.
Sorry, maybe I'm doing something wrong, but manpages seem to be a little bit unfriendly and with few examples...
Thanks!

Alad · 2018-07-06 19:03:27

With aur-sync, the options you want is "-f --no-ver-shallow". That ignores versions for packages specified on the command line (while checking versions of any dependencies). Since this seems to be a common source of confusion, I might add an alias "--rebuild". Related:

https://github.com/AladW/aurutils/tree/ … rcmp-devel

With aur-build, --makepkg-conf is there, it's just the short option "-M". Since all aurutils programs now support long options, expanding it to --makepkg-conf makes sense.

https://github.com/AladW/aurutils/blob/ … chroot#L56

Hope this helps.

Last edited by Alad (2018-07-06 19:07:56)

ngoonee · 2018-07-06 23:48:49

Alad wrote:

With aur-sync, the options you want is "-f --no-ver-shallow". That ignores versions for packages specified on the command line (while checking versions of any dependencies). Since this seems to be a common source of confusion, I might add an alias "--rebuild". Related:

Hope this helps.

Yes this would help. I couldn't grok the documentation so I end up using aur build when trying to do this.

Alad · 2018-07-09 18:29:11

https://github.com/AladW/aurutils/issues/424

Quite simple to implement, but I'm super lazy. PRs welcome.

Alad · 2018-08-04 23:58:14

Alad wrote:

https://github.com/AladW/aurutils/issues/424
Quite simple to implement, but I'm super lazy. PRs welcome.

Implemented.

rasputin · 2018-09-02 13:22:40

I don't have any questions. Just praise.

I insalled this a while back but I couldn't understand it because I was used to the monolithic way of doing things (yaourt, aurman, etc). But it just clicked with me now how powerful separating the certain functions are. Thank you for making this!

Last edited by rasputin (2018-09-02 13:23:18)

Alad · 2018-12-04 15:55:26

After almost 2 years, I've published a new aurutils release. Note 2.0.0rc differs significantly from 1.5.3, and as the rc suffix makes clear, is not as well tested as the previous release.

https://github.com/AladW/aurutils/releases/tag/2.0.0rc

Preview of the 2.0 release. Note that this is all in an early (untested) state.
Non-inclusive list of changes:
A new design based on git(1). Programs are now run with the aur wrapper. For example, instead of /usr/bin/aursync you would run /usr/bin/aur sync, which calls /usr/lib/aurutils/aur-sync.
Support for repose was removed. See https://bbs.archlinux.org/viewtopic.php … 9#p1707649 for migration instructions.
VCS packages are now supported through the aur-vercmp-devel program.
Packages can now be built in a container without using a local repository. The relevant functionality was moved to aur-chroot.
Fixes for known issues in 1.5.3.
Support for bash completion. zsh completions are not yet supported; see #458.
aur-sync now supports any file manager for file review, including GUI file managers. This is done through a new AUR_PAGER environment variable.
aur-search has more colorful output in the "brief" format.
aur-sync now fetches sources in parallel.
aur-sync now supports makepkg -A.
aur-sync now supports diffs for tar snapshots.
aurcheck was replaced by aur-repo, a general tool to handle local repositories.
aurqueue was expanded to support virtual and versioned dependencies, and moved to aur-graph.
Support for aria2 and curl was removed, using wget as replacement.
The officer program was removed; see aur(1) for a replacement.
See the milestone https://github.com/AladW/aurutils/milestone/30?closed=1 and commit list https://github.com/AladW/aurutils/compare/1.5 for a full overview.

Last edited by Alad (2018-12-04 15:56:38)

NuSkool · 2018-12-05 01:42:33

As a user and fan of aurutils, I'd like to express my gratitude towards Alad and everyone else who has contributed to it. This project seems to have taken AUR helpers to the next level, and best part is it's all in bash! Thanks!

Git cloned and in process of checking out the scripts. Looking forward to creating a new wrapper script around these updates.

Edit: correction.

I fetched the new scripts this morning with wget, not git.

Last edited by NuSkool (2018-12-05 02:17:52)

Alad · 2018-12-08 22:43:29

Thanks.

best part is it's all in bash! Thanks!

Well, it's the best and the worst aspect at the same time. Best because it's easy to reuse to combine and reuse existing utilities (jq, pacman, etc) through pipelines. Worst because bash being a command language with extremely loose typing and rudimentary data structures, even implementing basic algorithms can be highly challenging.

Maybe at some point I'll rewrite a few components in different languages (aur-graph is already written in AWK). C and LUA are my top candidates so far.

Last edited by Alad (2018-12-08 22:43:41)

Atraii · 2018-12-11 02:25:23

Either C or Lua sound like great choices.

I have a question about signed package removal. Using

repo-remove /var/cache/pacman/custom/custom.db.tar.xz eaglemode

results in pacman getting a corrupted database. After a package is removed, this is what happens when you run pacman:

error: custom: signature from "Isaac Curtis <iicurtis@outlook.com>" is invalid                                                                                                        
error: failed to update custom (invalid or corrupted database (PGP signature))                                                                                                        
error: failed to synchronize all databases

Is there a better way to do this?

The only way I've found is to remove

/var/cache/pacman/custom/*.sig

diabonas · 2018-12-11 13:18:39

The problem is that the database itself is signed as well, and

repo-remove /var/cache/pacman/custom/custom.db.tar.xz eaglemode

only updates the database file custom.db.tar.xz without generating a new signature file custom.db.tar.xz.sig. This is why pacman refuses to use the new database, as it now has an invalid signature. To solve this, use the "--sign" option of repo-remove to sign the updated database. Optionally, you might be interested in the "--verify" option as well to check that the database file has not been tampered with before updating it.

Atraii · 2018-12-11 19:19:29

diabonas wrote:

To solve this, use the "--sign" option of repo-remove to sign the updated database. Optionally, you might be interested in the "--verify" option as well to check that the database file has not been tampered with before updating it.

This is exactly what I needed, thank you. I wonder if maybe this should go in the man page.

Ichimonji10 · 2018-12-21 17:01:01

Is there a PKGBUILD for aurutils 2.y? I didn't see a branch in the aurutils AUR repository. (It's fine if none exists, of course!)

Last edited by Ichimonji10 (2018-12-21 17:01:30)

eschwartz · 2018-12-21 17:06:27

There are no branches in the AUR. You are looking for a separate package, the "aurutils-git" package.

Ichimonji10 · 2018-12-21 17:18:49

*facepalm*

Thank you.

amerlyq · 2018-12-29 19:21:47

Is there a way to update only custom repo for aurutils? Was it solved in some way?
I know topic is old and unresolved https://bbs.archlinux.org/viewtopic.php?id=170343
My usecase:
1) I do rolling update only once in two weeks -- I have no time or energy to fix ever-breaking configs for 200+ used packages any more often -- and they break each time.
2) Still sometimes I need new additional package for some tasks in this span of two weeks -- and when package becomes 404 in mirrors -- I use arch archive to fetch it.
3) Local development going on all the time -- which involves rebuilding some AUR pkgs on daily basis -- but doing "pacman -Syu" each time conflicts with the above two points. And half a day I'm from laptom and mobile network, so actually doing each hour even "pacman -Sy" is a pain.
Any suggestions? I'm rather tired to comment in-out /etc/pacman.conf each time...

Edit: Nevermind, it seems solution had come to my head immediatly after posting question:
$ sudo pacman --config /etc/pacman.d/aur.conf -Sy <mypkgname>

Last edited by amerlyq (2018-12-29 20:14:55)

ngoonee · 2018-12-29 22:55:52

amerlyq wrote:

1) I do rolling update only once in two weeks -- I have no time or energy to fix ever-breaking configs for 200+ used packages any more often -- and they break each time.

This is what you should be solving first and foremost. Regular breakage indicates something seriously wrong (and it's unlikely to be Arch, since the regular users would notice that).

Sounds like you should set up your own customer repo mirrors rather than using Arch's.

eschwartz · 2018-12-30 00:06:51

Personally, I make /var/lib/pacman/sync/custom.db (my custom repository) a symlink to the on-disk location of my actual custom repository. If you do this, then pacman -Su will always show updates to the custom repository independently of whether you've updated the sync repositories.

Alad · 2018-12-30 10:00:05

Edit: Nevermind, it seems solution had come to my head immediatly after posting question:
$ sudo pacman --config /etc/pacman.d/aur.conf -Syu <mypkgname>

Even on a local repo you can do a partial upgrade, so use -u.

aur-build has used this approach for local builds since its inception in 2016. (I thought this would be clear from the console output, but I guess not.)

With chroot builds it's a partial upgrade, since you'll build against potentially newer packages - unless you use an archive snapshot for both host and container.

edit: it turns out this is all in the man page already.

Last edited by Alad (2019-01-02 02:18:26)

amerlyq · 2019-01-03 09:02:36

ngoonee wrote:

amerlyq wrote:
1) I do rolling update only once in two weeks -- I have no time or energy to fix ever-breaking configs for 200+ used packages any more often -- and they break each time.
This is what you should be solving first and foremost. Regular breakage indicates something seriously wrong (and it's unlikely to be Arch, since the regular users would notice that).

Nope, actually things continuously being broken indicate evolution of the soft -- it always increases its deprecated, removed or reengineered parts.
And I have too many too much customized configs (and scripts based on it) plus temporary workarounds (until upstream catch up with last PR in some branches).
So on each upgrade something must be fixed, it's inevitable if you don't use virgin vanilla packages.

eschwartz wrote:

Personally, I make /var/lib/pacman/sync/custom.db (my custom repository) a symlink to the on-disk location of my actual custom repository. If you do this, then pacman -Su will always show updates to the custom repository independently of whether you've updated the sync repositories.

Nice hack! I will definitely use it in extreme situation. But still it's hack and I would prefer to not interfere with how pacman actually does it's work an to not bypass the public user interface it provides. And still nice hack.

Alad wrote:

Even on a local repo you can do a partial upgrade, so use -u.

You have a point, but concrete usecase was like 30+ continuous rebuilds-reinstalls for single package only (e.g. like when emacspeak package completely breaks from time to time).
So using "-Sy emacspeak" completely suffice.

Alad wrote:

With chroot builds it's a partial upgrade, since you'll build against potentially newer packages - unless you use an archive snapshot for both host and container.

Is there a way around to prevent database sync (until explicitly requested) when using chroot?

Alad wrote:

edit: it turns out this is all in the man page already.

Yep, in the very bottom of "aur" man page. And still I see similarity only now, when I know what and why to do.
So, as to speak, despite man for 2.0rc contains all necessary commands -- my usecase is impossible to find, because there are no exact words present for user to understand what that section at bottom is about.

Alad · 2019-01-03 09:39:46

Is there a way around to prevent database sync (until explicitly requested) when using chroot?

It doesn't sync anything on the local system then.

So, as to speak, despite man for 2.0rc contains all necessary commands -- my usecase is impossible to find, because there are no exact words present for user to understand what that section at bottom is about.

The command presented in aur(1) is only for compability with outdated AUR helpers that don't support local repos. aur-build(1) already takes the necessary steps so a simple "pacman -S" suffices. (As described in the aur-build(1) man page.)

amerlyq · 2019-01-03 10:13:29

Alad wrote:

Is there a way around to prevent database sync (until explicitly requested) when using chroot?
It doesn't sync anything on the local system then.

If it's not, then from where can appear "potentially newer packages" you mentioned?

Alad wrote:

So, as to speak, despite man for 2.0rc contains all necessary commands -- my usecase is impossible to find, because there are no exact words present for user to understand what that section at bottom is about.
The command presented in aur(1) is only for compability with outdated AUR helpers that don't support local repos. aur-build(1) already takes the necessary steps so a simple "pacman -S" suffices. (As described in the aur-build(1) man page.)

Nice. However now I have doubts why it didn't work for me a week ago (and works now). I will try to catch problem if it repeats itself.

Alad · 2019-01-03 11:08:00

If it's not, then from where can appear "potentially newer packages" you mentioned?

The local system has
* pacman databases A (/var/lib/pacman/sync)
and the container
* pacman databases B (/var/cache/aurbuild/x86_64/root/var/lib/pacman/sync).
There's also the
* local repository R (in /var/cache/pacman/custom/custom.db, bind mounted to the chroot)

1) You build the package bloatpkg in the container.
2) B and R are updated (synced) in the process such that necessary makedepends for bloatpkg can be installed to the container.
3) A remains unaffected. To install bloatpkg on the host, you thus need to sync A to match R.
4) However, R assumed B. Thus the package may not work if you don't do a full system upgrade.

Either use both an archive mirror for host and chroot, don't use chroots, or don't use aur-build/aur-chroot.

Last edited by Alad (2019-01-03 11:11:36)

amerlyq · 2019-01-03 12:42:44

Alad wrote:

2) B and R are updated (synced) in the process such that necessary makedepends for bloatpkg can be installed to the container.
3) A remains unaffected. To install bloatpkg on the host, you thus need to sync A to match R.

So, I correctly understood process before, when I asked "Is there a way around to prevent database sync (until explicitly requested) when using chroot?".
I.e. I meant how to "not sync B at all" or more appropriately "how to make B a copy of A" so host and container packages matched each other and there was no need to do full system upgrade each time?
Actually:
1) can it be done with current implementation of aurutils by configuring or extending with scripts?
2) if not, is it possible to implent or extend aurutils itself w/o too much rewrite?
3) is it possible at all (for example due to limitations of arch-nspawn)
4) is it reasonable? (I think "yes, yes, yes", but gladly accept facts if I'm mistaken in that insistence)

Alad · 2019-01-03 12:44:40

I.e. I meant how to "not sync B at all"

If for some weird reason you don't want to sync repos, there's no point in using aurutils at all. Just use devtools directly. (extra-x86_64-build)

or more appropriately "how to make B a copy of A" so host and container packages matched each other and there was no need to do full system upgrade each time?

As I already pointed out, you'd use the same pacman configuration that points to a specific snapshot of the Arch repositories. I know of no other way using devtools.

edit: I guess you could bind mount /var/lib/pacman/sync, remove -u from the makechrootpkg options and adapt the --config/-Syu mechanism for local builds to chroot builds. That's not something for general consumption though.

Last edited by Alad (2019-01-03 13:01:59)

Arch Linux

#226 2018-07-06 18:43:19

Re: aurutils - helper tools for the aur

#227 2018-07-06 19:03:27

Re: aurutils - helper tools for the aur

#228 2018-07-06 23:48:49

Re: aurutils - helper tools for the aur

#229 2018-07-09 18:29:11

Re: aurutils - helper tools for the aur

#230 2018-08-04 23:58:14

Re: aurutils - helper tools for the aur

#231 2018-09-02 13:22:40

Re: aurutils - helper tools for the aur

#232 2018-12-04 15:55:26

Re: aurutils - helper tools for the aur

#233 2018-12-05 01:42:33

Re: aurutils - helper tools for the aur

#234 2018-12-08 22:43:29

Re: aurutils - helper tools for the aur

#235 2018-12-11 02:25:23

Re: aurutils - helper tools for the aur

#236 2018-12-11 13:18:39

Re: aurutils - helper tools for the aur

#237 2018-12-11 19:19:29

Re: aurutils - helper tools for the aur

#238 2018-12-21 17:01:01

Re: aurutils - helper tools for the aur

#239 2018-12-21 17:06:27

Re: aurutils - helper tools for the aur

#240 2018-12-21 17:18:49

Re: aurutils - helper tools for the aur

#241 2018-12-29 19:21:47

Re: aurutils - helper tools for the aur

#242 2018-12-29 22:55:52

Re: aurutils - helper tools for the aur

#243 2018-12-30 00:06:51

Re: aurutils - helper tools for the aur

#244 2018-12-30 10:00:05

Re: aurutils - helper tools for the aur

#245 2019-01-03 09:02:36

Re: aurutils - helper tools for the aur

#246 2019-01-03 09:39:46

Re: aurutils - helper tools for the aur

#247 2019-01-03 10:13:29

Re: aurutils - helper tools for the aur

#248 2019-01-03 11:08:00

Re: aurutils - helper tools for the aur

#249 2019-01-03 12:42:44

Re: aurutils - helper tools for the aur

#250 2019-01-03 12:44:40

Re: aurutils - helper tools for the aur

Board footer