Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

ngoonee · 2010-03-25 13:42:15

Ah, so these lines:-

depends=('wine' 'cabextract')
[ "$CARCH" == "x86_64" ] && depends=('bin32-wine' 'cabextract')

don't parse in bauerbill?

And yes, its no problem for me to edit, just was wondering whether it should be considered a bug. At the very least, nvidia-beta and related packages in the AUR use that sort of logic for their md5sums.

Xyne · 2010-03-25 14:00:50

Yeah, I'm well aware of that. I've mentioned several times that I'm (very slowly) working on another way of dealing with the problem, but it will take a while to implement and I have other things that I need to get done before I can spend time on it.

Last edited by Xyne (2010-03-25 14:01:28)

milomouse · 2010-03-25 18:57:36

Xyne wrote:

@milomouse
I'll look into sanity-checking the version right now.
I won't back up everything in the build directory because it's easy enough to do with a script. There's no reason to add more complexity to the code. Just use a new BuildDir when you run an update, then copy it into the old one when you're done.
If you use the PatchDir option, you can save patches and re-apply them later. It only works with the PKGBUILD and install file right now, but I will extend it to work with other files. I need to determine the best way to create and apply the patches.

Ok, that seems like a fair compromise. Changing BuildDir seems fine too, maybe I'll incorporate that as well. Simple enough for a script like you said but I just wanted to throw it out there for brainstorming ideas or whatnot. Thanks

Xyne wrote:

*edit*
After looking at it, I just realized a problem. The user is inspected to review the PKGBUILD before it's parsed and the only way to get the pkgver from the PKGBUILD it obviously to parse it. This leaves me with two options. Either I parse all PKGBUILDs twice, which seems wasteful, or I leave it up to the user to either update the PKGBUILD (which usually just means changing the pkgver) or skip it at the prompt.

If they dont skip at prompt, and the user tries to update the PKGBUILD and change the pkgver, they would still have to change the md5/sha* sums for it to compile and continue? Although I do agree parsing twice seems a little wasteful.
But yeah, I've been skipping them at prompt so far if I keep seeing the old versions. This way I can see what needs to be updated and then do it myself by fixing the PKGBUILDs and integrity checks or getting them the abs program if they're up-to-date. At least until the version stuff is figured out. I'm looking at the code and starting to learn perl at the same time so I can be a little more constructive with all this, so please bear with me until I'm able to.

ngoonee · 2010-03-25 22:54:07

Xyne wrote:

Yeah, I'm well aware of that. I've mentioned several times that I'm (very slowly) working on another way of dealing with the problem, but it will take a while to implement and I have other things that I need to get done before I can spend time on it.

Understood, thanks .

Wesman26 · 2010-03-26 02:44:55

[wes@Arch ~]$ sudo bauerbill -S w3m --save-metalin test.metalink
Password: 
Sorry, try again.
Password: 
error: --save-metalin does not match any package or group
[wes@Arch ~]$ sudo bauerbill -S w3m --save-metalink test.metalink
--> resolving openssl>=0.9.8e... installed
--> resolving gc>=7.0... found in [extra]
--> resolving ncurses... installed
--> resolving zlib... installed
--> resolving gcc-libs... installed
--> Reflecting... retrieved 0 mirrors.
[wes@Arch ~]$ w3m
bash: w3m: command not found
[wes@Arch ~]$ aria2c -M test.metalink
[#1 SIZE:0B/931.2KiB(0%) CN:1 SPD:0Bs](1more...) [TOTAL SPD:426.1KiBs]                                                                                                                                     
2010-03-25 22:42:30.082978 NOTICE - Verification finished successfully. file=/home/wes/gc-7.1-1-x86_64.pkg.tar.gz

2010-03-25 22:42:30.083014 NOTICE - Download complete: /home/wes/gc-7.1-1-x86_64.pkg.tar.gz
[#1 SIZE:624.0KiB/931.2KiB(67%) CN:1 SPD:341.4KiBs]                                                                                                                                                        
2010-03-25 22:43:02.557294 NOTICE - Verification finished successfully. file=/home/wes/w3m-0.5.2-4-x86_64.pkg.tar.gz

2010-03-25 22:43:02.557323 NOTICE - Download complete: /home/wes/w3m-0.5.2-4-x86_64.pkg.tar.gz

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  2|  OK| 386.0KiB/s|/home/wes/gc-7.1-1-x86_64.pkg.tar.gz
  1|  OK|  27.8KiB/s|/home/wes/w3m-0.5.2-4-x86_64.pkg.tar.gz

Status Legend:
 (OK):download completed.
[wes@Arch ~]$ sudo bauerbill -S w3m
--> resolving openssl>=0.9.8e... installed
--> resolving gc>=7.0... found in [extra]
--> resolving ncurses... installed
--> resolving zlib... installed
--> resolving gcc-libs... installed
--> Reflecting... retrieved 0 mirrors.

:: The following packages will be downloaded:
   [extra]
     gc-7.1-1 w3m-0.5.2-4

:: Number of Packages:                       2
:: Total Download Size:               1.27 MiB
:: Binary Package Install Size:       2.74 MiB

:: Proceed with download? [Y/n] y
--> Reflecting... retrieved 0 mirrors.

Exception caught
Exception: [XML2SAXMetalinkProcessor.cc:141] Cannot parse metalink XML file. XML may be malformed.


:: 2 unsuccessful download(s). Would you like to retry? [Y/n]

This seems strange...

Also, If I tell it I don't want to retry but don't want to abort, it continues on and asks me if I want to install... You probably designed it that way, but just in case.

Last edited by Wesman26 (2010-03-26 02:46:32)

Xyne · 2010-03-26 09:06:46

@Wesman26
Are you using the latest versions of perl-xyne-arch (from my site) and aria2?

It is by design that you can continue after a failed download. Due to limitations of the PKGBUILD parser, the download may miss some source files. When it gets to the makepkg step, makepkg will be able to resolve the missing source files and download them before building the package.

m0nhawk · 2010-03-27 12:25:46

Firstly, thanks for such a good replacement of Yaourt, IMHO.
It satisfies all of my needs, but i miss one feature (maybe someone has request it already OR maybe I just didn't read man-pages carefully) from Yaourt. Moving directly to example:
I install Yaourt and it's dependencies (for me - package-query and yajl *pastebin*).
When I remove Yaourt with Yaourt it show packages that not needed by any other package *pastebin*.
But BauerBill didn't show this *pastebin*.

So i request (if it's not so hard), maybe, to add option which would enable dependency check for unusable packages.

Xyne · 2010-03-27 12:42:13

Just use the standard pacman options to show orphaned dependencies:

bauerbill -Qdt

or

pacman -Qdt

If you want to run that automatically after every removal, create a bash function.

Wesman26 · 2010-03-28 21:44:10

Xyne wrote:

@Wesman26
Are you using the latest versions of perl-xyne-arch (from my site) and aria2?

I have your site added to my /etc/pacman.conf and I'm using aria2. Is there something else I need to do?

Xyne · 2010-03-28 22:52:24

@Wesman26
Save the metalink by appending the option "--save-metalink test.metalink", then try "aria2c -M test.metalink" to see if it generates an error that way. Post what happens along with the metalink. Also post the versions of perl-xyne-arch and aria2 just in case they're somehow out of sync.

Wesman26 · 2010-03-29 00:59:00

Xyne wrote:

@Wesman26
Save the metalink by appending the option "--save-metalink test.metalink", then try "aria2c -M test.metalink" to see if it generates an error that way. Post what happens along with the metalink. Also post the versions of perl-xyne-arch and aria2 just in case they're somehow out of sync.

[wes@Arch ~]$ sudo bauerbill -S bitlbee --save-metalink test.metalink
--> resolving gnutls... installed
--> resolving glib2... installed
--> Reflecting... retrieved 0 mirrors.
[wes@Arch ~]$ aria2c -M test.metalink

2010-03-28 20:49:45.676148 NOTICE - Verification finished successfully. file=/home/wes/bitlbee-1.2.4-1-x86_64.pkg.tar.gz

2010-03-28 20:49:45.676194 NOTICE - Download complete: /home/wes/bitlbee-1.2.4-1-x86_64.pkg.tar.gz

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK| 401.4KiB/s|/home/wes/bitlbee-1.2.4-1-x86_64.pkg.tar.gz

Status Legend:
 (OK):download completed.

I have version 1.8.0 of Aria2 and I'm not sure how to check the version of perl-xyne-arch, though according to my system update it's up to date.

Xyne · 2010-03-29 01:54:13

The latest version of aria2 is 1.9.0. Check that your mirror is in sync and then update. It should resolve the issue.

You can check the version of installed packages with pacman, either using "pacman -Qs <pkgname>" or "pacman -Qi <pkgname>".

The latest versions in each repo at the time of posting:

extra/aria2 1.9.0-1
xyne-any/perl-xyne-arch 2010.03.21.2-1
community/perl-xyne-arch 2010.03.17.1-1

Last edited by Xyne (2010-03-29 01:55:28)

Wesman26 · 2010-03-29 04:04:21

@Xyne
Thank you very much, I should have this fixed in a couple minutes. On a side note, holy crap, my mirror is way behind. 614MB that I need to download.

virus_found · 2010-03-29 13:59:12

The sources of mesa package can't be downloaded (with %bauerbill -S --abs mesa):

source=(#ftp://ftp.freedesktop.org/pub/mesa/${pkgver}/MesaLib-${pkgver}.tar.bz2
    ftp://ftp.freedesktop.org/pub/mesa/${pkgver/.0.902//}RC/MesaLib-${pkgver/0.902/1-rc2}.tar.bz2
        #ftp://ftp.freedesktop.org/pub/mesa/${pkgver}/MesaDemos-${pkgver}.tar.bz2
    ftp://ftp.freedesktop.org/pub/mesa/${pkgver/.0.902//}RC/MesaDemos-${pkgver/0.902/1-rc2}.tar.bz2
        ftp://ftp.archlinux.org/other/mesa/gl-manpages-1.0.1.tar.bz2
        mesa-7.1-link-shared.patch
        intel-revert-vbl.patch
        LICENSE)

jwbirdsong · 2010-03-29 14:31:31

@virus_found.
Your ABS db is up to date? mesa pkgver=7.7.0.902?? <-- Testing version
bauerbill -S --abs mesa works fine here.

I can build both testing and bauerbill -S --abs extra/mesa just fine

EDIT** clarification of versions

Last edited by jwbirdsong (2010-03-29 14:41:40)

bangkok_manouel · 2010-03-31 01:47:45

edit/ nvm, issue already explained few pages earlier.

Last edited by bangkok_manouel (2010-03-31 03:57:41)

Vasher · 2010-04-01 05:43:31

I've noticed that bauerbill doesn't do very well when building multiple kde packages at once. tar fails apparently. An example would be my last attempt to install kdebase.

prompt> sudo bauerbill -S kdebase --abs
:: group kdebase (including ignored packages):
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror kdebase-konsole kdebase-kwrite kdebase-plasma
:: Install whole content? [Y/n] y
:: ABS build queue:
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror kdebase-konsole kdebase-kwrite kdebase-plasma
:: Build all? [Y/n] y
--> Loading ABS taurball data...
--> Checking for abs archive updates for the following repos: [extra]
error: tar exited with 256 (su vasher -c 'bsdtar -zxf /tmp/bauerbill/taurballs/extra.abs.tar.gz -C /tmp/bauerbill/build --no-same-owner --no-same-permissions extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase  2>/dev/null')

It works fine when you build and install them individually, or if you just install them from the prebuilt packages, but if you try multiple at once, then you get an error like the one above.

Xyne · 2010-04-01 10:57:08

It seems I hadn't considered the case of installing multiple packages from the same split package. Oops.

It should be fixed now. I've restructured a bit of code to recognize shared sources and it seems to work, although I didn't go through the full kdebase installation to check it.

Install perl-xyne-arch from [xyne-any] (on my site) and let me know if it works.

Vasher · 2010-04-02 05:28:06

Xyne wrote:

It seems I hadn't considered the case of installing multiple packages from the same split package. Oops.
It should be fixed now. I've restructured a bit of code to recognize shared sources and it seems to work, although I didn't go through the full kdebase installation to check it.
Install perl-xyne-arch from [xyne-any] (on my site) and let me know if it works.

The problem appears to be fixed. Thanks!

The other problem that I've been having relates to aria2, I believe, so I don't know if you can really fix it without hardcoding special cases for certain packages, but aria2 seems to do horribly with any sites which redirect you to a mirror. A prime example would be the kde packages, but I expect that stuff from sourceforge would have similar problems. wget deals with them just fine, but aria2 creates a file that's about 20K and claims that the download was successful (maybe it's the webpage that it's downloading). The result is that when makepkg goes to build the package, the md5 checksum fails. I have to download each of the kde packages myself or use makepkg directly if I want it to work (and of course, the fact that a couple of the most recent kde packages have bad checksums in their PKGBUILD files doesn't help, but that's a separate issue).

Ideally, aria2 would succeed without difficulties. But barring that it would be nice if it would at least not create useless packages with bauerbill continuing thinking that everythings fine. Again, I don't know if there's a nice way for bauerbill to deal with it without special casing certain packages given that it's likely an aria2 problem, but dealing with such packages would be a lot nicer if bauerbill could deal with the problem. Maybe it could check md5sums itself and delete files that fail so that makepkg can use wget to attempt to get them correctly. I don't know. But if you can think of a clean way to deal with the problem, it would be much appreciated. Thanks.

Xyne · 2010-04-02 13:59:46

I would have expected it to handle redirections. There doesn't seem to be any option for this in the man page either. This might be worth reporting to the aria2 dev(s) upstream. In my experience the main dev is friendly and quick to fix reported bugs.

Can you give a specific example so that I can test it? If I find an option to make it work then I'll add it to the defaults. If not then I'll consider other approaches.

Vasher · 2010-04-03 01:54:12

Xyne wrote:

I would have expected it to handle redirections. There doesn't seem to be any option for this in the man page either. This might be worth reporting to the aria2 dev(s) upstream. In my experience the main dev is friendly and quick to fix reported bugs.
Can you give a specific example so that I can test it? If I find an option to make it work then I'll add it to the defaults. If not then I'll consider other approaches.

Normally, I'd say to use kdelibs since you don't appear to be using kde (since you haven't run into this yourself), and it doesn't rely on anything else in kde, but the md5sum for kdelibs-4.4.2.tar.bz2 in its PKGBUILD is wrong. So, even if aria2c and bauerbill work perfectly, makepkg will still fail. However, you can still see the problem with kdelibs, and pretty much any kde package will do as long as they download from http://download.kde.org/stable/4.4.2/src/{file} (for kdelibs that would be http://download.kde.org/stable/4.4.2/sr … .2.tar.bz2 ).

http://download.kde.org/stable/4.4.2/sr … .2.tar.bz2 takes you to a page with a list of mirrors to choose from. Somehow wget figures this out and downloads using one of the mirrors (it even tries another if the first one fails I think). aria2c appears to just download the webpage. So, if you run

wget http://download.kde.org/stable/4.4.2/src/kdelibs-4.4.2.tar.bz2

you get the file. But if you run

aria2c http://download.kde.org/stable/4.4.2/src/kdelibs-4.4.2.tar.bz2

you get download.php. My guess is that bauerbill is running

aria2c http://download.kde.org/stable/4.4.2/src/kdelibs-4.4.2.tar.bz2 -o kdelibs-4.4.2.tar.bz2

so it gets kdelibs-4.4.2.tar.bz2, but it's really a renamed download.php, so it isn't even a valid tarball, let alone have the correct md5sum.

In any case, you can test it with kdelibs, but the md5sum in the PKGBUILD is wrong, so while you'll see the symptoms, it wouldn't work even if bauerbill and aria2c did the right thing. Probably the best package that I can think of at the moment other than that would be kdebase-runtime. I think that that has the smallest number of kde dependencies of the other kde packages, and I know that its md5sum is correct.

Hmm. A thought just occurred to me. Assuming that bauerbill is indeed using aria2c with the -o option, does it need to? Are there any packages which will download with the wrong file name if you don't give the file name to write to? If that's not really an issue, then you could remove the -o option, in which case, the kde downloads would fail (having downloaded download.php instead of the file that they were supposed to), and then makepkg would succeed with wget if you told bauerbill to continue in spite of the failed downloads. That would fix the problem for the moment, and if aria2c ever gets fixed, then it'll just work with the new aria2c without you having to change bauerbill at all.

Edit: I just submitted a bug report to the aria2 folks. Hopefully, they're able to sort it out.

Last edited by Vasher (2010-04-03 02:11:28)

Xyne · 2010-04-03 06:32:58

It doesn't use the "-o" option directly because it uses metalinks, but the metalinks specify the names and there's no way around that. It's actually necessary too when downloading sources and packages to make sure that all the packages end up in the cache and the sources end up in their respective build directories.

I'll wait for a response from aria2's dev. Most likely he'll provide a solution but if he doesn't I can add update bauerbill to remove downloads that fail the checksum.

Last edited by Xyne (2010-04-03 07:22:45)

Vasher · 2010-04-03 12:45:01

Well, the aria folks said

This is because server treats wget special way. If you send request with
User-Agent: Wget, server responds redirection to actual file. Try this:
$ aria2c -U "Wget" "http://download.kde.org/download.php?ur … .2.tar.bz2"
So we cannot do anything about this.

Whether it's a good idea to use aria2c -U "Wget" in all cases, I don't know, but it looks like that would fix the problem. Certainly, it downloads kdelibs correctly that way. So, as long as it's not likely to cause other problems, it seems like a good idea. There are probably other sites that treat wget as a special case as well, in which case it would be that much more desirable. But maybe there's a downside that I'm not aware of.

flamelab · 2010-04-03 12:50:26

Vasher wrote:

I've noticed that bauerbill doesn't do very well when building multiple kde packages at once. tar fails apparently. An example would be my last attempt to install kdebase.

prompt> sudo bauerbill -S kdebase --abs
:: group kdebase (including ignored packages):
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror kdebase-konsole kdebase-kwrite kdebase-plasma
:: Install whole content? [Y/n] y
:: ABS build queue:
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror kdebase-konsole kdebase-kwrite kdebase-plasma
:: Build all? [Y/n] y
--> Loading ABS taurball data...
--> Checking for abs archive updates for the following repos: [extra]
error: tar exited with 256 (su vasher -c 'bsdtar -zxf /tmp/bauerbill/taurballs/extra.abs.tar.gz -C /tmp/bauerbill/build --no-same-owner --no-same-permissions extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase  2>/dev/null')

It works fine when you build and install them individually, or if you just install them from the prebuilt packages, but if you try multiple at once, then you get an error like the one above.

^^^ I encounter the same problem :-/

 ~ $ pacman -Qi bauerbill | grep Version
Version        : 2010.03.14.1-1
flamelab@battlestar ~ $ sudo bauerbill -S kdebase --abs
:: group kdebase (including ignored packages):
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror
     kdebase-konsole kdebase-kwrite kdebase-plasma
:: Install whole content? [Y/n] y
:: ABS build queue:
   [extra]
     kdebase-dolphin kdebase-kappfinder kdebase-kdepasswd kdebase-kdialog kdebase-kfind kdebase-kinfocenter kdebase-konqueror
     kdebase-konsole kdebase-kwrite kdebase-plasma
:: Build all? [Y/n] y
--> Loading ABS taurball data...
--> Checking for abs archive updates for the following repos: [extra]
error: tar exited with 256 (su flamelab -c 'bsdtar -zxf /tmp/bauerbill/taurballs/extra.abs.tar.gz -C /tmp/bauerbill/build --no-same-owner --no-same-permissions extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase extra/kdebase  2>/dev/null')

Xyne · 2010-04-03 14:59:16

I don't want to change the default user agent because that will only encourage others to the same stupid thing that they've done on the KDE mirrors. I recommend contacting the KDE people to request that they provide a solution for aria2c and other download agents in general. In the meantime you can add "user-agent=Wget" to the Aria2Args option in the config file.

I'll make a note of this on the project page.

Arch Linux

#601 2010-03-25 13:42:15

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#602 2010-03-25 14:00:50

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#603 2010-03-25 18:57:36

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#604 2010-03-25 22:54:07

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#605 2010-03-26 02:44:55

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#606 2010-03-26 09:06:46

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#607 2010-03-27 12:25:46

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#608 2010-03-27 12:42:13

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#609 2010-03-28 21:44:10

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#610 2010-03-28 22:52:24

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#611 2010-03-29 00:59:00

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#612 2010-03-29 01:54:13

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#613 2010-03-29 04:04:21

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#614 2010-03-29 13:59:12

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#615 2010-03-29 14:31:31

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#616 2010-03-31 01:47:45

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#617 2010-04-01 05:43:31

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#618 2010-04-01 10:57:08

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#619 2010-04-02 05:28:06

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#620 2010-04-02 13:59:46

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#621 2010-04-03 01:54:12

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#622 2010-04-03 06:32:58

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#623 2010-04-03 12:45:01

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#624 2010-04-03 12:50:26

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#625 2010-04-03 14:59:16

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

Board footer