aurutils - helper tools for the aur

SolarAquarion · 2017-05-03 01:24:05

I guess i need to create a local database for aurutils to do it's thing?

runical · 2017-05-03 08:23:57

To use aurutils you'll need to create a database. How to do this is explained in the aurutils man page (man aurutils).

heliumtt · 2017-05-07 13:34:13

Hello,
Great tool once again ! To me it is exactly the simple clean building helper: doing in a chroot with arch-nspawn avoids the build deps all over the system to "clean".

Small clarifications:
On a btrfs filesystem using aursync -c, a subvolume $ARCH is created with a complete chroot inside.
Then the chroot is synced for updates and "copied" in /var/lib/aurbuild/$ARCH/$USER.
1. Is this snapshotted or copied "--reflink" ? it seems pretty quick but I do not see any snapshot happening. And the resulting subvolume does not show the "root" chroot as parent. It says "clean working copy" as a message.

2. I believe due to systemd-nspawn use, a /var/lib/machines subvolume is created. Is that the reason why the building subvol /var/lib/aurbuid/$ARCH/$USER still exists ? ot is it kept until the next build just in case ?

3. Once package created (successfully) I can find all the related build deps in $USER subvol. However, once I start a new build the $USER subvol seems clean. How is that achieved ?
When one tries to delete a btrfs subvol that has subvols, it is rejected; and afik there is no easy way to delete btrfs subvolumes recursively yet (except listing and deleting with a script and sed etc,which I do not call easy...).

NB: It created on my root filesystem /var/lib/machines which I guess is resulting from systemd-nspawn. I can make a folder initially to avoid this subvolume but not in the chroot aurbuild creates...

Note that I avoid nested subvolumes (I just create subvolumes at the top level and mount them) so that reverting back to a snapper snapshot does not create a mess (as per the wiki guidelines).

Thanks for the tool !

heliumtt · 2017-05-07 13:47:28

Apparently it is directly handled by makchrotpkg:

 create_chroot() {

	# Lock the chroot we want to use. We'll keep this lock until we exit.
	lock 9 "$copydir.lock" "Locking chroot copy [$copy]"

	if [[ ! -d $copydir ]] || $clean_first; then
		# Get a read lock on the root chroot to make
		# sure we don't clone a half-updated chroot
		slock 8 "$chrootdir/root.lock" "Locking clean chroot"

		stat_busy "Creating clean working copy [$copy]"
		if is_btrfs "$chrootdir" && ! mountpoint -q "$copydir"; then
			subvolume_delete_recursive "$copydir" ||
				die "Unable to delete subvolume %s" "$copydir"
			btrfs subvolume snapshot "$chrootdir/root" "$copydir" >/dev/null ||
				die "Unable to create subvolume %s" "$copydir"
		else
			mkdir -p "$copydir"
			rsync -a --delete -q -W -x "$chrootdir/root/" "$copydir"
		fi
		stat_done

		# Drop the read lock again
		exec 8>&-
	fi

	# Update mtime
	touch "$copydir"
}

So it seems to clean only at creation by the "non" easy way that I mentionned.
So nspawn is not used as ephemeral on purpose to keep a track for troubleshooting ?

I would just suggest that at installation one creates a @aurbuild subvol at the top level and mounts it there when one is on a btrfs subvol.

Better not to incorporate in the .install ?

Thanks again.

Thanks

Alad · 2017-05-07 16:58:30

I'm not very familiar with the btrfs problematic, but if you want makechrootpkg to use systemd-nspawn --ephemeral, you should mention it on the arch-projects mailing list.

https://lists.archlinux.org/listinfo/arch-projects

I would just suggest that at installation one creates a @aurbuild subvol at the top level and mounts it there when one is on a btrfs subvol.
Better not to incorporate in the .install ?

Adding automation to the aurutils package is not something I'm interested in, but you could add something to the aurutils(7) or aurbuild(1) man page.

Or as above, mention your suggestion on the arch-projects mailing list so we don't have to care in aurutils.

Last edited by Alad (2017-05-07 17:00:17)

heliumtt · 2017-05-08 21:48:25

Hello,
Yes, I understand the setup will remain in the man page. Actually it is good as it is.
Thinking it through, using arch-nspawn as ephemeral is not a good one: in case a build fails, it is good to be able to enter and see what failed. Plus the makechrootpkg deletes it and resnapshots the "template". So it is better than I thought as it is !

Creating the btrfs subvol at the root as @aurbuild is just one particular case of the btrfs + snapper wiki to avoid nested subvolumes hard to manage when restoring:
https://wiki.archlinux.org/index.php/Sn … tem_layout

So it may not deserve a full explaination in the man page not to clutter...

Thanks for the tool,preferred to yaourt!
Cheers

heliumtt · 2017-05-14 01:54:55

Hello, next small problem that I cannot solve...
I have set "Optional TrustOnly" in pacman.conf.
Package builds with

aursync -sc gentoo-bashrc

without concerns.

However it cannot put it in the local repo.
I did sign the database as perthe wiki. My key is "ultimate" trused via pacman-key.

It actualy trusts the database since signed (and resigns it when adding the package ?).
However there seems to be a signature problem with the "custom.files.tar".

Any clue of what I miss ?
Or bug ?

See output below.

Thanks again !

[xxx@xxx ~]$ aursync -c gentoo-bashrc 
  -> Using [custom] repository
==> Resolving dependencies...
  -> gentoo-bashrc 0 -> 2016.0-1
==> Retrieving build files...
Depuis https://aur.archlinux.org/gentoo-bashrc
 = [à jour]          master     -> origin/master
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 custom is up to date
:: Starting full system upgrade...
 there is nothing to do
==> Creating clean working copy [xxx]...done
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 custom is up to date
:: Starting full system upgrade...
 there is nothing to do
==> Making package: gentoo-bashrc 2016.0-1 (Sat May 13 13:51:06 AST 2017)
==> Retrieving sources...
  -> Found gentoo-bashrc-2016.0
==> Validating source files with md5sums...
    gentoo-bashrc-2016.0 ... Passed
==> Making package: gentoo-bashrc 2016.0-1 (Sat May 13 13:51:08 AST 2017)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found gentoo-bashrc-2016.0
==> Validating source files with md5sums...
    gentoo-bashrc-2016.0 ... Passed
==> Extracting sources...
==> Entering fakeroot environment...
==> Starting package()...
==> IMPORTANT!!
==> 
==> To prevent overwriting your existing .bashrc, the file
==> was installed to /usr/share/gentoo-bashrc/bashrc
==> 
==> Please examine that file, adding your own important
==> settings and local variables to it before replacing
==> your existing bashrc.
==> 
==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Compressing man and info pages...
==> Checking for packaging issue...
==> Creating package "gentoo-bashrc"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Adding install file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: gentoo-bashrc 2016.0-1 (Sat May 13 13:51:10 AST 2017)
==> Installing package gentoo-bashrc with pacman -U...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) gentoo-bashrc-2016.0-1

Total Installed Size:  0.00 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                          [########################################] 100%
(1/1) checking package integrity                                        [########################################] 100%
(1/1) loading package files                                             [########################################] 100%
(1/1) checking for file conflicts                                       [########################################] 100%
:: Processing package changes...
(1/1) installing gentoo-bashrc                                          [########################################] 100%

==> IMPORTANT!!!
==> To prevent overwriting your existing .bashrc, the file
==> was installed to /usr/share/gentoo-bashrc/bashrc
==> 
==> Please examine that file, adding your own important
==> settings and local variables to it before replacing your
==> existing bashrc.   Gentoo's bashrc also includes a
==> color scheme for root, so you may consider copying it to
==> /root/.bashrc after also copying any local variables there
==> to the new .bashrc

Optional dependencies for gentoo-bashrc
    bash: Shell needed to make use of this package [installed]
resolving dependencies...
looking for conflicting packages...

Packages (7) elfutils-0.168-1  libelf-0.168-1  licenses-20140629-2  pyalpm-0.8-2  python-3.6.1-1
             python-pyelftools-0.24-2  namcap-3.2.7-2

Total Installed Size:  130.28 MiB

:: Proceed with installation? [Y/n] 
(7/7) checking keys in keyring                                          [########################################] 100%
(7/7) checking package integrity                                        [########################################] 100%
(7/7) loading package files                                             [########################################] 100%
(7/7) checking for file conflicts                                       [########################################] 100%
:: Processing package changes...
(1/7) installing python                                                 [########################################] 100%
Optional dependencies for python
    python-setuptools
    python-pip
    sqlite [installed]
    mpdecimal: for decimal
    xz: for lzma [installed]
    tk: for tkinter
(2/7) installing pyalpm                                                 [########################################] 100%
(3/7) installing licenses                                               [########################################] 100%
(4/7) installing libelf                                                 [########################################] 100%
(5/7) installing elfutils                                               [########################################] 100%
(6/7) installing python-pyelftools                                      [########################################] 100%
(7/7) installing namcap                                                 [########################################] 100%
Checking PKGBUILD
Checking gentoo-bashrc-2016.0-1-any.pkg.tar.xz
'/var/tmp/aurbuild.imFjowth/gentoo-bashrc-2016.0-1-any.pkg.tar.xz' -> '/var/cache/pacman/custom/gentoo-bashrc-2016.0-1-any.pkg.tar.xz' (archive : '/var/cache/pacman/custom/gentoo-bashrc-2016.0-1-any.pkg.tar.xz~')
==> Verifying database signature...
gpg: enabled debug flags: memstat
gpg: assuming signed data in '/var/cache/pacman/custom//custom.db.tar'
gpg: Signature made Sat May 13 12:09:35 2017 AST
gpg:                using RSA key zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
gpg: Good signature from "xxx yyy <xxx.yyy@gmail.com>" [ultimate]
gpg: keydb: handles=2 locks=0 parse=2 get=2
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=2 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=4 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: secmem usage: 0/32768 bytes in 0 blocks
  -> Database signature file verified.
==> Extracting database to a temporary location...
==> Verifying database signature...
gpg: enabled debug flags: memstat
gpg: assuming signed data in '/var/cache/pacman/custom//custom.files.tar'
gpg: Signature made Sat May 13 11:45:41 2017 AST
gpg:                using RSA key zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
gpg: BAD signature from "xxx yyy <xxx.yyy@gmail.com>" [ultimate]
gpg: keydb: handles=2 locks=0 parse=2 get=2
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=2 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=4 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: secmem usage: 0/32768 bytes in 0 blocks
==> ERROR: Database signature was NOT valid!

Alad · 2017-05-14 02:14:05

Looks like https://github.com/AladW/aurutils/issues/236. Please rebuild the database with repo-add (see post https://bbs.archlinux.org/viewtopic.php … 9#p1707649) and try if the git version fixes your issue.

Last edited by Alad (2017-05-14 02:15:04)

heliumtt · 2017-05-14 19:20:25

Hello,

I had done it recently so wierd that this problem still does.
Actually another work around was for me to put in the custom config file:

 PackageRequired TrustedOnly

As long as the package is signed, the database does not matter too much to me...

Cheers

eschwartz · 2017-05-14 23:46:04

heliumtt wrote:

Hello,
I had done it recently so wierd that this problem still does.
Actually another work around was for me to put in the custom config file:
 PackageRequired TrustedOnly 
As long as the package is signed, the database does not matter too much to me...
Cheers

If you are using a local repo, you don't need any kind of signatures at all (unless your threat model involves a non-root user on your computer who can add a package to your world-writable local repo???).

But in the general case, you kind of do want the database to be signed. If the database is not signed then a malicious actor could hold back a specific package to ensure you continue to use a software release with known vulnerabilities, and since the package was at some point signed by a trusted maintainer key, pacman would happily accept it as the correct and latest version.

Alad · 2017-05-15 01:34:16

There's some odd things going on here:

==> Installing package gentoo-bashrc with pacman -U...

or

-> gentoo-bashrc 0 -> 2016.0-1
(...)
'/var/tmp/aurbuild.imFjowth/gentoo-bashrc-2016.0-1-any.pkg.tar.xz' -> '/var/cache/pacman/custom/gentoo-bashrc-2016.0-1-any.pkg.tar.xz' (archive : '/var/cache/pacman/custom/gentoo-bashrc-2016.0-1-any.pkg.tar.xz~')

Please post output from:

pacman -Q aurutils-git
pacman -Qkk aurutils-git
type aurbuild aurcheck aursync gpg repo-add
env
pacconf

and also:

sudo pacsync custom
bash -x aursync -c gentoo-bashrc
bash -x aursync -c <some other package where verifying the db fails>

edit: I'm guessing you weren't using the latest git checkout since that one didn't work at all... should be fixed now.

Last edited by Alad (2017-05-15 20:04:25)

heliumtt · 2017-05-22 20:08:05

Hello,
Sorry i tried to go through this but....:

Depuis https://aur.archlinux.org/gentoo-bashrc
 = [à jour]          master     -> origin/master
:: Synchronizing package databases...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to update custom (unable to lock database)
error: failed to synchronize any databases
error: failed to init transaction (unable to lock database)
error: could not lock database: File exists
  if you're sure a package manager is not already
  running, you can remove /var/lib/pacman/db.lck

I saw your report on github but I have no such file, nothing in /tmp not /var/tmp, and even rebooted to ensure all was flushed...
No way to get that out. It happens while pacman works like a charm...

Sorry !

runical · 2017-05-22 20:24:28

This seems like a problem with pacman. Can you update with pacman?

Alad · 2017-05-23 12:54:20

Well, I can't reproduce it... besides a pacman issue, it could also be a sudo issue. But I don't have much to go with here.

heliumtt · 2017-05-23 19:54:51

Hello.
Well, I wiped everything and reinstalled from scratch and it disappeared...
Thanks anyway !

heliumtt · 2017-05-24 16:45:30

Hello,
Still on the signature of the Database, I found what created my concern:
1. I set up everything as per the man page.
2. Build some packages with aursync -cs xxx.
3. Pacman -Syyu xxx
All brilliant.

However if I build one package forgetting "-s", it obviously does not sign.
So I then get:

pacman -Syyu
 custom                                          8,0 KiB  0,00B/s 00:00 [########################################] 100%
 custom.sig                                    566,0   B  0,00B/s 00:00 [########################################] 100%
erreur: custom: signature from "yyyyyy" is invalid
erreur: la mise à jour de custom a échoué (base de données invalide ou corrompue (signature PGP))
erreur: la base de données « custom » n’est pas valide (base de données invalide ou corrompue (signature PGP))

If one attempts to rebuild anything with "-s" (hoping that the database would succssfully be signed then), it fails from the start: when attempting to update the root subvolume template, one gets the same message so the whole process aborts.

Wiping the database and recreating works, but a bit heavy.
Perhaps aurbuild should not accept to update a database that is signed without resigning ?

Thanks
Bye

Alad · 2017-05-25 13:02:04

I can reproduce this issue. I thought repo-add would remove the old signature, but I guess it doesn't...

Alad · 2017-05-25 13:21:34

sudo rm /var/lib/pacman/sync/custom*.sig
rm /var/cache/pacman/custom/*.sig

Then try again with the latest aurutils git version:

https://github.com/AladW/aurutils/commi … bb10116e8f

edit:

Perhaps aurbuild should not accept to update a database that is signed without resigning

That might be a good idea. For now, it just removes any old signatures if -s was not specified

edit2: https://github.com/AladW/aurutils/commi … db66a0e49a

Last edited by Alad (2017-05-25 13:39:06)

Alad · 2017-06-08 17:46:14

I've dropped support for aria2 in git as it does not support OCSP stapling. For those still using it I suggest to install the parallel package.

Also note that aursearch -r does not output JSON in a merged format anymore, i.e., for those using it with jshon(1). The new -R option reverts this behaviour.

PUE · 2017-06-23 18:57:24

I've seen it mentioned a few times here but it seems to have changed again and again. The question is simple: How can I remove a package from a custom repo?

Alad · 2017-06-23 22:22:35

repo-remove /path/to/repo/repo.db.tar pkgname
sudo pacsync custom

"pkgname" must be the actual pkgname, not the full path of the package.

Last edited by Alad (2017-06-24 08:13:32)

PUE · 2017-06-24 07:04:32

Thanks, but this fails for me:

repo-remove /var/lib/pacman/sync/aur.db acroread
==> ERROR: '/var/lib/pacman/sync/aur.db' does not have a valid database archive extension.

Alad · 2017-06-24 08:13:52

I forgot to add the .tar extension... edited post above.

Last edited by Alad (2017-06-24 08:15:02)

PUE · 2017-06-24 08:39:23

Thanks, that worked now. I also had the path wrong.
For future reference (for myself...) it worked like this for my repo called 'aur':

sudo repo-remove /var/cache/pacman/aur/aur.db.tar acroread
sudo pacsync aur

On a sidenote, I find it a bit odd that aurcheck just hangs if you do not specify a repo using -d <reponame>. I would expect it to succeed if there is only one repo or fail with an error message.

Alad · 2017-06-24 15:41:30

It doesn't hang, it waits for input from stdin, see aurcheck(1):

-d
The name of a pacman repository. If not specified, packages and their
versions are taken from stdin.

Example use:

pacman -Qm | aurcheck

I'd use a hyphen to make this explicit, but getopts doesn't seem to consider this as an option parameter.

https://github.com/AladW/aurutils/issues/229

Arch Linux

#51 2017-05-03 01:24:05

Re: aurutils - helper tools for the aur

#52 2017-05-03 08:23:57

Re: aurutils - helper tools for the aur

#53 2017-05-07 13:34:13

Re: aurutils - helper tools for the aur

#54 2017-05-07 13:47:28

Re: aurutils - helper tools for the aur

#55 2017-05-07 16:58:30

Re: aurutils - helper tools for the aur

#56 2017-05-08 21:48:25

Re: aurutils - helper tools for the aur

#57 2017-05-14 01:54:55

Re: aurutils - helper tools for the aur

#58 2017-05-14 02:14:05

Re: aurutils - helper tools for the aur

#59 2017-05-14 19:20:25

Re: aurutils - helper tools for the aur

#60 2017-05-14 23:46:04

Re: aurutils - helper tools for the aur

#61 2017-05-15 01:34:16

Re: aurutils - helper tools for the aur

#62 2017-05-22 20:08:05

Re: aurutils - helper tools for the aur

#63 2017-05-22 20:24:28

Re: aurutils - helper tools for the aur

#64 2017-05-23 12:54:20

Re: aurutils - helper tools for the aur

#65 2017-05-23 19:54:51

Re: aurutils - helper tools for the aur

#66 2017-05-24 16:45:30

Re: aurutils - helper tools for the aur

#67 2017-05-25 13:02:04

Re: aurutils - helper tools for the aur

#68 2017-05-25 13:21:34

Re: aurutils - helper tools for the aur

#69 2017-06-08 17:46:14

Re: aurutils - helper tools for the aur

#70 2017-06-23 18:57:24

Re: aurutils - helper tools for the aur

#71 2017-06-23 22:22:35

Re: aurutils - helper tools for the aur

#72 2017-06-24 07:04:32

Re: aurutils - helper tools for the aur

#73 2017-06-24 08:13:52

Re: aurutils - helper tools for the aur

#74 2017-06-24 08:39:23

Re: aurutils - helper tools for the aur

#75 2017-06-24 15:41:30

Re: aurutils - helper tools for the aur

Board footer