Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

Xyne · 2011-02-06 19:20:55

@jsgt
You probably need to run "pacman --fix-provides". See the caveat for CPAN users on the Bauerbill project page.

Basically, very few CPAN packages on Arch respect the CPAN versioning scheme. Most maintainers appear to be completely unaware of how it works and so they apply naive version translations.

I'll probably add that last bit to the aforementioned section.

@flan_suse
I have never run into that problem myself, but I don't build that often and my /tmp dir is cleared out about once a day. The "-c" argument might make sense in most cases, but I won't add it to the default arguments because it would clear CVS sources, which could be undesirable most of the time.

*edit*
Can you confirm that leftover files in $pkgdir are never a problem?
I don't have any ideas of how to avoid the problem without using "-c". I mostly feel that this is the packager's problem, but I don't know how makepkg is expected to behave when building a new package in an old directory.

Last edited by Xyne (2011-02-06 19:35:18)

jsgt · 2011-02-06 19:35:53

@Xyne
Using pacpan --fix-provides doesn't help so I think it's a problem with versions. Bauerbill (or pacpan?) finds Apache::Session on CPAN and creates a PKGBUILD with the dependency perl-test-exception>=0.150.0 (it's currently in 0.31 on CPAN, 0.29-1 in community) so isn't a problem in bauerbill (adding the trailing zero) rather than the community package?

Xyne · 2011-02-06 20:24:51

No, the zero is correct. See the output of the following script that uses CPAN's "version" module:

#!/usr/bin/env perl
use strict;
use warnings;

use version;

foreach my $ver ( qw/ 0.31 0.47 0.15 0.082 1.10.0/ )
{
  print "$ver\n  " . (version->parse($ver)->numify) . "\n";
}

The package version should include the trailing 0. As I wrote, most CPAN package maintainers are not aware of how CPAN versions work. I will see if I can find a way to fix this with pacpan, but otherwise you can just install perl-test-exception with baueribll (or pacpan)... the correct version is detected.

Wait a bit though, as I intend to update perl-xyne-arch in my repo shortly to fix another error in the version formatting function.

flan_suse · 2011-02-06 21:13:38

Xyne wrote:

I have never run into that problem myself, but I don't build that often and my /tmp dir is cleared out about once a day. The "-c" argument might make sense in most cases, but I won't add it to the default arguments because it would clear CVS sources, which could be undesirable most of the time.

Not a problem there, and I wanted to know for myself if there was a way to clear out $srcdir before/after building packages. Since bauerbill can pass arguments to makepkg (in this case, -c, thanks to ngoonee), I just added that to bauerbill.conf, which means it should be good going forward when using -Syu --aur with bauerbill. I tend to get compulsive about these things, even if they are only theoretical. It gives me peace of mind to know that an automated process can be configured to eliminate "possible / theoretical" future problems.

Xyne wrote:

Can you confirm that leftover files in $pkgdir are never a problem?
I don't have any ideas of how to avoid the problem without using "-c". I mostly feel that this is the packager's problem, but I don't know how makepkg is expected to behave when building a new package in an old directory.

When I use makepkg manually on a PKGBUILD, I can confirm that no matter how badly I purposely butcher the $pkgdir (before proceeding with makepkg), the resulting $pkgdir is no different than if I had left it alone before running makepkg. While messing around with footest PKGBUILDs, it was only the $srcdir that was vulnerable to old / remnant files, but never the $pkgdir.

To demonstrate this, just use the footest-1.0-1 PKGBUILD I posted above and build it inside a temporary folder. The resulting $pkgdir will look like this:

$pkgdir/.PKGINFO
$pkgdir/opt/foo/foo1.txt

Now do whatever you want inside of the $pkgdir.

touch pkg/YOUSHOULDNTSEETHISAFTERMAKEPKG

And just run makepkg again.

makepkg -f

You'll notice that the resulting $pkgdir is exactly the same as before. The new file you manually dumped inside (to emulate a leftover file from an older version) is no longer present. The same thing can be observed even if the second time you run makepkg in the same folder you instead use a PKGBUILD with a newer pkgver or pkgrel. Using -c or not doesn't make a difference for the resulting $pkgdir.

Last edited by flan_suse (2011-02-06 21:20:07)

Xyne · 2011-02-06 21:20:11

Ok, then I'm going to leave it as-is for now. There is no (obvious?) way that bauerbill could detect which files should not be in the $srcdir on subsequent rebuilds (e.g. using modification times would not help as "old" files might be CVS sources).

Thanks for the examples and other feedback.

Last edited by Xyne (2011-02-06 23:30:44)

ngoonee · 2011-02-07 02:10:36

Xyne wrote:

*edit*
Can you confirm that leftover files in $pkgdir are never a problem?
I don't have any ideas of how to avoid the problem without using "-c". I mostly feel that this is the packager's problem, but I don't know how makepkg is expected to behave when building a new package in an old directory.

$pkgdir is cleared on initial makepkg. Its partially the packager's problem, but the issue is that makepkg simply unpacks the package over $srcdir, which obviously replaces any patched files BUT does not touch files which have been added. It is those files that are the issue.

Most of the kernel26-* would face these issues as well.

flan_suse · 2011-02-07 04:03:26

ngoonee wrote:

Most of the kernel26-* would face these issues as well.

I don't remember 100%, but I think I might have had issues when updating different kernel26-* (pf, lqx, ck, etc...) packages from the AUR with bauerbill in the past. Now it shouldn't be a problem going forward, since bauerbill will pass -c to makepkg since I made the change to bauerbill.conf.

I do remember running bleachbit (as a normal user) to clean out files under /tmp/ in order to try again, in which I would redo bauerbill -Syu --aur. It's a bit extreme, I know, but at the time I hadn't known about the possibility of old $srcdir's being used in updates. So with my "shotgun to a fly" approach, cleaning /tmp/ was good enough for me since it guaranteed a fresh build folder for the packages to be built from the AUR.

jsgt · 2011-02-08 17:47:07

Xyne wrote:

No, the zero is correct. See the output of the following script that uses CPAN's "version" module:
#!/usr/bin/env perl
use strict;
use warnings;

use version;

foreach my $ver ( qw/ 0.31 0.47 0.15 0.082 1.10.0/ )
{
  print "$ver\n  " . (version->parse($ver)->numify) . "\n";
}
The package version should include the trailing 0. As I wrote, most CPAN package maintainers are not aware of how CPAN versions work. I will see if I can find a way to fix this with pacpan, but otherwise you can just install perl-test-exception with baueribll (or pacpan)... the correct version is detected.
Wait a bit though, as I intend to update perl-xyne-arch in my repo shortly to fix another error in the version formatting function.

Thanks for the clarification and the code. I don't, however, see anything wrong with a package maintainer versioning a package as 0.29 if thats what the CPAN page says (http://search.cpan.org/dist/Test-Exception/).

#!/usr/bin/perl

use strict;
use warnings;

use CPAN::Version;

print CPAN::Version->vcmp("0.150.0","0.29")."\n";

prints -1, so I still don't understand why Test::Exception is getting pulled in from CPAN if I already have 0.29 installed (the dependency was >=0.150.0).

Xyne · 2011-02-10 09:51:09

jsgt wrote:

Thanks for the clarification and the code. I don't, however, see anything wrong with a package maintainer versioning a package as 0.29 if thats what the CPAN page says (http://search.cpan.org/dist/Test-Exception/).
/snip
prints -1, so I still don't understand why Test::Exception is getting pulled in from CPAN if I already have 0.29 installed (the dependency was >=0.150.0).

The point is that CPAN versions cannot be directly translated to Pacman versions because Pacman does not compare versions the same way that CPAN does. CPAN recognizes that 0.29 is a later version than 0.150, while Pacman things that 0..150 is the latest version, which is why it fails to resolve >=0.150 when 0.29 is installed.

CPAN support in Bauerbill requires Pacman packages that respect CPAN's versioning system because it uses META information from CPAN (META.yml) to determine dependencies etc. When packagers completely ignore CPANs system, it doesn't work, but it should always be possible to build the package with Bauerbill to ensure that it uses the correct version and thus satisfies dependencies as expected.

flan_suse · 2011-02-21 16:06:08

Noticed something just recently when I did a full system update with bauerbill. It downloaded the updated packages, but did not install them. Instead, it downloaded the updated PKGBUILDs from the AUR, built them, installed them, and then exited. I had to run a full system update again, and this time it went ahead and installed the updated packages from the repo.

Here's the run-down, after I do "bauerbill -Syu --aur":
1) It finds some updated packages from the repos.
2) It downloads these updated packages.
3) It finds, downloads, builds. and installs updated AUR PKGBUILDs, one by one.
4) It exits.

I try again "bauerbill -Syu --aur":
5) It finds the same updated packages from the repos.
6) It installs the updated packages from the repos (already downloaded to the cache from before).

Last edited by flan_suse (2011-02-21 16:10:23)

Xyne · 2011-02-22 00:31:49

flan_suse wrote:

Noticed something just recently when I did a full system update with bauerbill. It downloaded the updated packages, but did not install them. Instead, it downloaded the updated PKGBUILDs from the AUR, built them, installed them, and then exited. I had to run a full system update again, and this time it went ahead and installed the updated packages from the repo.
Here's the run-down, after I do "bauerbill -Syu --aur":
1) It finds some updated packages from the repos.
2) It downloads these updated packages.
3) It finds, downloads, builds. and installs updated AUR PKGBUILDs, one by one.
4) It exits.
I try again "bauerbill -Syu --aur":
5) It finds the same updated packages from the repos.
6) It installs the updated packages from the repos (already downloaded to the cache from before).

Can you post the actual output? If it downloads the packages then it is aware that they should be upgraded, so it's probably exiting with an error. It might also be due to packages in "SyncFirst" (I don't remember right now how those are handled in the presence of a build queue).

flan_suse · 2011-02-22 14:56:52

I don't have bauerbill/powerill configured to save a log file, which I will probably enable going further. I don't have the buffer anymore since I had rebooted this machine.

I have found one hint in my pacman log that might (though unlikely) point to the cause. In the handful of updated packages from the repos, one of the packages was glibc, which is under "HoldPkg" in pacman.conf. However, this only applies to packages that are to be removed. Under "SyncFirst" I have pacman, but this package was not part of the handful of updates.

The pacman log itself reveals nothing else. It just shows a line that reads "upgraded gtk-murrina-black-theme (0.4-1 -> 0.4-2)" followed by a line that shows I manually ran the -Syu command (which I did). gtk-murrina-black-theme is from the AUR.

Xyne · 2011-02-22 19:09:04

The next time it occurs, pastebin the output and post it here. Circumstantial bugs such as these are difficult to reproduce so I would rather just wait until it comes up again.

flan_suse · 2011-02-22 23:34:16

Xyne wrote:

The next time it occurs, pastebin the output and post it here. Circumstantial bugs such as these are difficult to reproduce so I would rather just wait until it comes up again.

I set it to log under /var/log/powerpill.log and the next time this happens, I have in my mind to save the output as well. I don't remember seeing any error message when it happened. It just exited like normal, as it would with any other update. I'll post the output if it occurs again. I might even try to reproduce it later.

SanskritFritz · 2011-03-07 12:49:14

Is there a way to catch maintainer changes in bauerbill? What I mean is this: I run bauerbill -Syu and there is a new version of an aur package. Now if I could be certain that the package wasn't orphaned and adopted by someone else, I would trust the source and would not check the AUR webpage to see if the maintainer has been changed.
Alternate question: does the TrustedUsers option correctly recognize that the maintainer has been changed? If yes, that would solve my problem completely.
It would also be nice if there was a possibility to add the current maintainer to the TrustedUsers list while installing from bauerbill. For example like this:

view [v]  backup [b]  edit [e]  continue [ok]  trust [t]  skip [s]  abort [x]

Apsu · 2011-03-07 16:46:53

I've been having a problem with bauerbill that started only very recently: aria2 successfully downloads packages but bauerbill believes the downloads were unsuccessful. I'm not quite sure what the cause might be and I've removed/reinstalled every package it uses, as well as blowing away all the relevant config files, trying different options, etc.

For example:

:: Proceed with download? [Y/n]

--> successfully downloaded gecko-mediaplayer-1.0.0-2-x86_64.pkg.tar.xz
--> successfully downloaded gnome-mplayer-1.0.2-1-x86_64.pkg.tar.xz
--> successfully downloaded gettext-0.18.1.1-3-x86_64.pkg.tar.xz

Download Results:
gid|stat|avg speed |path/URI
===+====+===========+===========================================================
2| OK| 224.8KiB/s|/var/cache/pacman/pkg/gecko-mediaplayer-1.0.0-2-x86_64.pkg.tar.xz
1| OK| 159.6KiB/s|/var/cache/pacman/pkg/gnome-mplayer-1.0.2-1-x86_64.pkg.tar.xz
3| OK| 23.3KiB/s|/var/cache/pacman/pkg/gettext-0.18.1.1-3-x86_64.pkg.tar.xz

Status Legend:
(OK):download completed.

:: 3 unsuccessful download(s). Would you like to retry? [Y/n]

------

If I hit no to retry, then no to abort, it hands off to pacman and successfully installs the packages. Obviously the downloading is working. My assumption is that something in the way aria2's exit codes work or some other criteria bauerbill/the xyne perl libs look for has changed.

Anyone else having this problem? Enabling logging is completely pointless as neither pacman nor powerpill(.log) show anything relevant at all. Any hints on where to look or what to try, short of digging into the code or wrapping aria2 or similar? I *love* this package manager and it's quite frustrating to have to fool it into installing packages it's successfully downloading

Xyne · 2011-03-07 18:22:34

SanskritFritz wrote:

Alternate question: does the TrustedUsers option correctly recognize that the maintainer has been changed? If yes, that would solve my problem completely.

The maintainer data is retrieved from the AUR at runtime. That data should be pulled directly from the AUR database and thus always be current.

The only way that the reported maintainer would differ is if someone else managed to adopt the package in the short interval between determining the download targets and actually downloading the files.

It would also be nice if there was a possibility to add the current maintainer to the TrustedUsers list while installing from bauerbill. For example like this:
view [v]  backup [b]  edit [e]  continue [ok]  trust [t]  skip [s]  abort [x] 

The problem is that the maintainer is not included in the normal information. To get the maintainers, Bauerbill detects that trusted maintainers have been specified, and then uses a separate query to return all packages owned by those maintainers to cross-reference them against the requested packages. Basically, you can ask "What packages does this user own?" but not "Who owns this package?". That info could be scraped from the webpage, but that's ugly, inefficient, and just adds another section of code that has to be kept in sync with AUR changes.

I've requested the inclusion of the maintainer in the data returned by the "info" method, but so far there hasn't been any progress (that I'm aware of).

Apsu wrote:

:: Proceed with download? [Y/n] 

--> successfully downloaded gecko-mediaplayer-1.0.0-2-x86_64.pkg.tar.xz                                                                                                                                                                        
--> successfully downloaded gnome-mplayer-1.0.2-1-x86_64.pkg.tar.xz                                                                                                                                                                            
--> successfully downloaded gettext-0.18.1.1-3-x86_64.pkg.tar.xz                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                 
Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  2|  OK| 224.8KiB/s|/var/cache/pacman/pkg/gecko-mediaplayer-1.0.0-2-x86_64.pkg.tar.xz
  1|  OK| 159.6KiB/s|/var/cache/pacman/pkg/gnome-mplayer-1.0.2-1-x86_64.pkg.tar.xz
  3|  OK|  23.3KiB/s|/var/cache/pacman/pkg/gettext-0.18.1.1-3-x86_64.pkg.tar.xz

Status Legend:
 (OK):download completed.

:: 3 unsuccessful download(s). Would you like to retry? [Y/n]

That's strange. The "--> successfully downloaded" lines indicate that bauerbill is detecting them during the download. Once the download is complete, it rechecks that all of the packages have been downloaded, ultimately using the same function. I don't see how it could succeed the first time and then fail the second time, and I can't reproduce it.

Check the permissions of the cache directories and files in case something got mixed up. Post the full output too (in code tags, or on a pastebin site) along with the command so I can see how you're running it.
It might be useful to see the metalink (--save-metalink) and the pacman info (pacman -Si ...) for the target packages.

If none of that points me to the problem, I'll upload a debugging version that you can run to help me track this down.

Apsu · 2011-03-07 18:41:58

Xyne wrote:

Check the permissions of the cache directories and files in case something got mixed up.

I had already done this, walking through /var, cache, pacman, pkg, checking the package files, etc, but I did it again just now when you mentioned this and something occurred to me. I changed the umask recently so the package files were only visible to root. Sure enough, su'ing before running bauerbill has no issue while using sudo still does. Best I can figure is that how you're spawning the aria subprocess ends up with a different effective UID than when bauerbill checks the files. Perhaps bauerbill is switching to the UID of the user specified in the "BuildAs" directive before this portion of code?

Whatever the reason, I easily fixed the issue in my particular case; you can decide how/if you'd like to change bauerbill's behavior when the originating UID is the only user with access to the package files. Thanks for the help, even if it wasn't intentional And thanks for the kickass package manager!

SanskritFritz · 2011-03-07 19:46:07

Xyne, thank you, that solves my problem completely. About the automatic addition, don't worry about it, I will whip up a short script that will kind of simulate that effect (not automatically of course).
Thanks again for your work on bauerbill (and all your other very useful utilities)

Xyne · 2011-03-07 22:04:13

@Apsu
the UID/EUID management is tricky, especially when invoking other processes. I would rather keep making the tacit assumption that the privileged user's umask will leave the files readable as I don't want to fiddle with that code, at least not right now.

@Apsu & SanskritFritz
Thank you both for the feedback.

Queseuq · 2011-03-13 01:26:32

Hi, I wanted to edit bauerbill's config file so I copied it to ~/.config/powerpill/bauerbill.conf and changed a load of settings. However when I run bauerbill it doesn't pick up these changes, unless I point it to the altered config file. Am I missing something here?

ngoonee · 2011-03-13 07:03:53

Queseuq wrote:

Hi, I wanted to edit bauerbill's config file so I copied it to ~/.config/powerpill/bauerbill.conf and changed a load of settings. However when I run bauerbill it doesn't pick up these changes, unless I point it to the altered config file. Am I missing something here?

I doubt that's the correct location, in the first place. bauerbill and powerpill are not identical, you may want to try in .config/bauerbill (not that it WILL work for sure, I just edit /etc/bauerbill.conf in place).

Xyne · 2011-03-13 14:44:04

@nboonee
That is the correct user path. As Bauerbill is just a subclass of Powerpill, I decided to keep its configuration files in the same place. It's stated in the man page.

@Queseuq
If you run bauerbill as root (directly or via sudo), your home directory will be /root, not /home/<user>.

Queseuq · 2011-03-13 17:37:13

Ah, thanks Xyne. That was confusing me somewhat.

ngoonee · 2011-03-14 01:01:41

Thanks Xyne. Also, I don't think running bauerbill as root (or through sudo) is really advisable, Queseuq.

Arch Linux

#976 2011-02-06 19:20:55

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#977 2011-02-06 19:35:53

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#978 2011-02-06 20:24:51

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#979 2011-02-06 21:13:38

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#980 2011-02-06 21:20:11

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#981 2011-02-07 02:10:36

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#982 2011-02-07 04:03:26

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#983 2011-02-08 17:47:07

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#984 2011-02-10 09:51:09

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#985 2011-02-21 16:06:08

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#986 2011-02-22 00:31:49

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#987 2011-02-22 14:56:52

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#988 2011-02-22 19:09:04

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#989 2011-02-22 23:34:16

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#990 2011-03-07 12:49:14

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#991 2011-03-07 16:46:53

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#992 2011-03-07 18:22:34

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#993 2011-03-07 18:41:58

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#994 2011-03-07 19:46:07

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#995 2011-03-07 22:04:13

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#996 2011-03-13 01:26:32

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#997 2011-03-13 07:03:53

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#998 2011-03-13 14:44:04

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#999 2011-03-13 17:37:13

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#1000 2011-03-14 01:01:41

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

Board footer