Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

jowilkin · 2011-01-20 21:08:41

Xyne, been using bauerbill a while and it is getting more and more polished, but I have another little issue.

Went to do a system upgrade including AUR and there were upgrades available for virtualbox_bin and virtualbox-ext-oracle. bauerbill attempted to install virtualbox-ext-oracle first, but it had a dependency for virtualbox >= 4.0.2 and I only had 4.0.0 installed.

==> Missing Dependencies:
  -> virtualbox>=4.0.2

However if virtualbox_bin had been installed before virtualbox-ext-oracle, I would have had virtualbox 4.0.2 because it was in the upgraded virtualbox_bin package.

> bb -Ss virtualbox_bin
AUR/virtualbox_bin 4.0.2-1 [installed: 4.0.0-2]
    Oracle VM VirtualBox Binary Edition

Maybe you could do a sort of the AUR packages to be installed based on dependencies so this doesn't happen.

Edit: Actually this wouldn't have worked in this case, I tried to install virtualbox_bin first, but the virtualbox-ext-oracle package I have installed currently requires virtualbox=4.0.0

Last edited by jowilkin (2011-01-20 21:16:50)

Xyne · 2011-01-21 13:29:35

flan_suse wrote:

Not sure if this has been covered, but is it normal behaviour for bauerbill to invoke sudo when removing a package, but not when installing a package?

It has been covered but it's buried deep inside this thread. The short answer is that it uses sudo when it invokes pacman, but it never invokes itself with sudo. Bauerbill itself needs a writable cache directory. You could use "--cachedir ." to use the cwd, and then it would invoke pacman with sudo when it needs to install something.

@jowilkin
Even if it doesn't solve that particular case, I've updated the function that orders the build queue (which I actually only added within the last two weeks when I noticed a problem with Haskell packages). I had forgotten to take "provides" into account.

I've uploaded the new version to my repo and will move it to [community] in a few days.

Misfit138 · 2011-01-21 14:01:44

I had promised myself to mention this.

I like you, Xyne!

And, sincere thanks for your prolific contribution. Words alone do not convey the appreciation that I, and the community have for you.

Xyne · 2011-01-21 14:23:17

Misfit138 wrote:

I had promised myself to mention this.
I like you, Xyne!
And, sincere thanks for your prolific contribution. Words alone do not convey the appreciation that I, and the community have for you.

That just put a big stupid smile on my face

Thanks!

Misfit138 · 2011-01-21 20:52:27

Xyne wrote:

That just put a big stupid smile on my face

Good. I'm glad.

Thanks!

You are most welcome.

extofme · 2011-01-28 03:33:11

howdy,

i don't see it mentioned anywhere (so it probably doesn't exist :-), but can bauerbill do chroot builds for cross compiling? or has anyone done a setup like this?

basically i have a nice server than runs KVM/libvirt, and i maintain a local mirror. i have oodles of packages that require regular rebuilds for each of my machines (too many...).

i really like the autopatch stuff, and everything about bauerbill really (nice w3rk btw, Xyne), but if it could do chroot'ed builds in the dev tools fashion... well tha'd just be icing on an already delicious looking cake :-)

the other idea was to chroot bauerbill itself... and do it that way somehow. any ideas here? i need clean builds because the pkgs are never for the build machine.

C Anthony

Xyne · 2011-01-28 03:58:17

I just had one of those "oh wow, I'm an idiot" epiphanies. As I was reading your post, I thought to myself "Huh, I have bauerbill32 in the arch32-light-utils package... wait, htf does that work? Ah, it doesn't. *facepalm*". At the time, I basically just ran s/powerpill/bauerbill/g on powerpill32 without giving it too much though.

Moving on....

Cross-compiling: All building is done by makepkg so if you can find a way to get makepkg to cross-compile then bauerbill can cross compile.

In the chroot: If you can install pacman (and makepkg) in the chroot then you can use bauerbill in the chroot. I haven't used bauerbill in a chroot yet, but it should just work.

extofme · 2011-01-28 04:45:04

Xyne wrote:

I just had one of those "oh wow, I'm an idiot" epiphanies. As I was reading your post, I thought to myself "Huh, I have bauerbill32 in the arch32-light-utils package... wait, htf does that work? Ah, it doesn't. *facepalm*". At the time, I basically just ran s/powerpill/bauerbill/g on powerpill32 without giving it too much though.

nice :-)

Xyne wrote:

Cross-compiling: All building is done by makepkg so if you can find a way to get makepkg to cross-compile then bauerbill can cross compile.

yeah ill need to look into mkchrootpkg more, but im sure it uses makepkg too.. except it handles all the union fs stuff for me. i was also thinking that maybe i can make a small wrapper script, inside the chroot, so makechootpkg actually uses bauerbill instead. it'd just be nice for the abs stuff and autopatching; i might be able to work something out with some bind mounts any what have you, we'll see.

Xyne wrote:

In the chroot: If you can install pacman (and makepkg) in the chroot then you can use bauerbill in the chroot. I haven't used bauerbill in a chroot yet, but it should just work.

ok i figured as much; ill play around a bit and throw back whatever i come up with.

thanks again,

C Anthony

extofme · 2011-01-28 08:08:59

ah ha!

i'm not 100% i'm golden, but i'm going to leverage "--makepkg-bin" to either use mkchrootpkg directly or a small wrapper around it. i didn't think it was possible... wasn't in the man pages (tsk tsk! ;-), but nothing gets past a little xargs + grep action.

so whenever ya get around to it, the option "--makepkg-bin" is missing from man page.

it might not be too hard to add this ability to bauerbill directly... but i'm really not a perl guy. tbh, i haven't actually used bauerbill yet, but the feature set is perfect for scripting an automatic build system; clean chroot builds seem to be the only missing piece, but i don't know that for sure yet.

something to consider perhaps. at any rate... bravo sir, for not skimping on the options :-)

C Anthony

pogeymanz · 2011-01-28 11:27:50

I don't know if this has been covered yet, but will bauerbill get support for multi-package PKGBUILDs? Specifically, I use kernel26-ck from AUR and it builds the kernel and kernel-headers. When I use bauerbill, I have to manually install the headers package from /tmp.

If I just grab the PKGBUILD and run makepkg -sic, it installs both packages, so I imagine bauerbill should be able to do the same.

Xyne · 2011-01-28 17:07:51

@extofme
I've updated the man page. Thanks for pointing it out. The bauerbill package is only a front-end for perl-xyne-arch so I sometimes forget to update it when I make changes in the back-end.
I should start suggesting that people alias "man bauerbill" to "bauerbill --print-man-nroff | nroff -man | less".

If you manage to it working, I would definitely be interested to know how, as I would like to re-include bauerbill32 in the arch32-light-utils package. Let me know if I need to add options or make other changes to the code to make it play nicely.

pogeymanz wrote:

I don't know if this has been covered yet, but will bauerbill get support for multi-package PKGBUILDs? Specifically, I use kernel26-ck from AUR and it builds the kernel and kernel-headers. When I use bauerbill, I have to manually install the headers package from /tmp.
If I just grab the PKGBUILD and run makepkg -sic, it installs both packages, so I imagine bauerbill should be able to do the same.

Bauerbill should support split packages. The reason that it fails to handle kernel26-ck from the AUR is that the package uses bash to "trick" the AUR into accepting a split package. You can fix that by using the PatchDir and AutoPatch options to remove sections such as

pkgname=kernel26-ck
true && pkgname=(kernel26-ck kernel26-ck-headers)

and replace them with

pkgname=(kernel26-ck kernel26-ck-headers)

Read the "How does AutoPatch work?" and "Building from source fails." sections of the Bauerbill project page.

ngoonee · 2011-01-31 02:27:14

--> Checking for abs archive updates for the following repos: [testing]
error: could not find PKGBUILD for dovecot in /tmp/bauerbill/taurballs/testing.abs.tar.gz

Asking again, will the SVN PKGBUILDs be supported

Xyne · 2011-01-31 05:07:48

ngoonee wrote:

--> Checking for abs archive updates for the following repos: [testing]
error: could not find PKGBUILD for dovecot in /tmp/bauerbill/taurballs/testing.abs.tar.gz
Asking again, will the SVN PKGBUILDs be supported

Nope. The move to a pseudo-git interface removed the tarball download link. The only way to get it now is to scrape the website, and I will not add that to bauerbill.

Nag the repo maintainers to keep $repo.abs.tar.gz archives in sync with the packages in the repo.

ngoonee · 2011-01-31 05:28:06

Xyne wrote:

ngoonee wrote:
--> Checking for abs archive updates for the following repos: [testing]
error: could not find PKGBUILD for dovecot in /tmp/bauerbill/taurballs/testing.abs.tar.gz
Asking again, will the SVN PKGBUILDs be supported
Nope. The move to a pseudo-git interface removed the tarball download link. The only way to get it now is to scrape the website, and I will not add that to bauerbill.
Nag the repo maintainers to keep $repo.abs.tar.gz archives in sync with the packages in the repo.

Well, by necessity I guess it can't be updated for every new package. Will live with it.

Out of curiosity, by 'repo maintainers' you're referring to the dev team rather than to operators of individual mirrors, right?

Xyne · 2011-01-31 06:10:12

ngoonee wrote:

Well, by necessity I guess it can't be updated for every new package. Will live with it.

You might be able to convince someone to run the cronjob that creates the archive more often. It would make sense to run it once an hour so that mirrors are never more than one cycle behind.

ngoonee wrote:

Out of curiosity, by 'repo maintainers' you're referring to the dev team rather than to operators of individual mirrors, right?

Yes. The mirrors just mirror the repos and have nothing to do with the contents of the ABS archive.

SanskritFritz · 2011-01-31 09:13:33

Xyne wrote:

ngoonee wrote:
Well, by necessity I guess it can't be updated for every new package. Will live with it.
You might be able to convince someone to run the cronjob that creates the archive more often. It would make sense to run it once an hour so that mirrors are never more than one cycle behind.

That would work only with the bzr and svn packages, but with git and others, it would create a new package everytime the script runs, because the git support in makepgk is as simple as updating the PKGBUILD version to today's date, without even looking into the git repo.

ngoonee · 2011-01-31 09:30:21

SanskritFritz wrote:

Xyne wrote:
ngoonee wrote:
Well, by necessity I guess it can't be updated for every new package. Will live with it.
You might be able to convince someone to run the cronjob that creates the archive more often. It would make sense to run it once an hour so that mirrors are never more than one cycle behind.
That would work only with the bzr and svn packages, but with git and others, it would create a new package everytime the script runs, because the git support in makepgk is as simple as updating the PKGBUILD version to today's date, without even looking into the git repo.

AFAIK abs doesn't look into the PKGBUILDs at all.....

SanskritFritz · 2011-01-31 10:16:00

I thought abs is using a makepkg? Dunno, as I use bauerbill -abs anyway...

ngoonee · 2011-01-31 10:27:41

SanskritFritz wrote:

I thought abs is using a makepkg? Dunno, as I use bauerbill -abs anyway...

abs is the Arch Build System, check the wiki. I was asking Xyne about supporting direct SVN download via the svn interface rather than using the abs tarball.

The -abs option simply downloads the tarball using abs, then uses makepkg.

flan_suse · 2011-02-04 18:24:05

I bump into this rare problem once in a while with bauerbill, but it may even be makepkg itself that has the issue.

If I install a package from the AUR, it will build and install. If there is an update to the pkgrel of the same package, sometimes makepkg will error out, telling me that "foo-2.0.0-4.i686.pkg.tar.xz does not exist". If I check under /tmp/bauerbill/.../foo/ I see that the old pkgrel remains (foo-2.0.0-3.i686.pkg.tar.xz) but the new one (foo-2.0.0-4.i686.pkg.tar.xz) does not exist. Makepkg then tells me "foo-2.0.0-3.i686.pkg.tar.xz appears to match. Would you like to install it?" Naturally, I answer "No".

If I rm -rf /tmp/bauerbill/.../foo/* and try again, it succeeds with no problems.

Last edited by flan_suse (2011-02-04 18:28:15)

Xyne · 2011-02-05 00:35:56

@flan_suse
I need to see the actual output and I don't know how to reproduce it myself.

At first I suspected that bauerbill might not overwrite the previous PKGBUILD, because that would explain why removing the build directory resolves the issue, but I double-checked it and it does overwrite it.

From what you posted, it seems that makepkg is failing to build the new package. The "foo-2.0.0-4.i686.pkg.tar.xz does not exist" message is a bauerbill error message, not a makepkg error message. The actual makepkg error is further up in the output, which is why I need to see all of the output. That will let me determine the source of the issue.

flan_suse · 2011-02-06 05:56:55

I think you're right about it being a makepkg issue (where it could not build the package, and so bauerbill found no finished package ready to pass on to pacman to install.) When it comes up again, I can paste the output. Doesn't happen often enough, and I have my /tmp/ cleared out after I reboot anyways.

But while messing around trying to figure this out, among other things, I bumped into something else. You might be able to give more insight into this. Basically, I made two mock PKGBUILDs called footest. One is version 1.0, the other is 2.0. If you run makepkg for the first one (1.0), but do not clean up the srcdir, and then run makepkg for the second one (2.0), you will get remnants from the first build inside of the second build. This is not the case if you clean out the srcdir first. I noticed bauerbill (for me at least) does not clean out the srcdir, even when dealing with a separate package version (such as with an updated PKGBUILD from the AUR).

So in theory, leftover obsolete files in the srcdir can find themselves carried over into the new version of the package. If you inspect the PKGBUILD, it would give no hints to this. And if you build the packages with a clean srcdir each time, this won't happen. But if you leave the srcdir alone, this does happen.

I made two mock PKGBUILDs, with comments, to demonstrate this. I could be wayyyyyyyyyyy off, because I'm not exactly an expert.

Here are links to them:

footest-1.0-1: http://aur.pastebin.com/GqVBMaY9
footest-2.0-1: http://aur.pastebin.com/s1370QUR

First, run makepkg for the first PKGBUILD, and then inspect $pkgdir/opt/foo/. You will find foo1.txt.

Next, without cleaning out $srcdir, run makepkg on the second PKGBUILD, and then inspect $pkgdir/opt/foo/. You will find foo2.txt AND foo1.txt (a remnant and obsolete file from the first version.)

I couldn't figure out how to configure bauerbill to automatically clean out the srcdir before continuing. I realize this isn't a major thing, since most of the time it's the same exact filenames / folder structures, but there is the possibility this could happen.

*** I also want to add that I routinely run BleachBit now and then, which cleans out /tmp, and thus, bauerbill's build folders. So I often do have a clean srcdir anyways, luckily.

Last edited by flan_suse (2011-02-06 06:29:10)

ngoonee · 2011-02-06 09:58:58

If you have a PKGBUILD which applies patches, at least one of which creates a file rather than edits an existing one, you will have this problem. I just set makepkg's flags to include -c....

jsgt · 2011-02-06 12:53:10

Maybe this has been covered already, if so I apologize.

I'm trying to install the Apache::Session Perl module. In the META.yml file (http://cpansearch.perl.org/src/CHORNY/A … 9/META.yml) it says the build requires Test::Exception: 0.15. In the PKGBUILD that Xyne::Arch::CPAN makes, however, this is made into perl-test-exception>=0.150.0. This leads to the following output when I try to install Apache::Session:

--> resolving perl-test-exception>=0.150.0... found on CPAN (Test-Exception -> perl-test-exception)

even though perl-test-exception 0.29-1 is installed from community.

flan_suse · 2011-02-06 15:15:00

ngoonee wrote:

If you have a PKGBUILD which applies patches, at least one of which creates a file rather than edits an existing one, you will have this problem. I just set makepkg's flags to include -c....

I knew someone was going to do that. I read through the makepkg and bauerbill Wiki and man pages, and for some reason I didn't find it. Yet here it is, right there:

-c, --clean
Clean up leftover work files and directories after a successful build.

I think it's because as I read through them again, I conditioned myself to look out for "source" or "srcdir" specifically. I go by example too, which not all man pages provide. Here's just an idea of what I refer to, taken from rsync's man page.

A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the following commands copies the files in the same way, including their setting of the attributes of /dest/foo:
rsync -av /src/foo /dest
rsync -av /src/foo/ /dest/foo

The explanation combined with the part in bold is what made me understand rsync's usage better. If it was just a brief explanation, I might not have quite figured it out.

Thank you, ngoonee! And there you are, Xyne: My eyes just needed a little tuning. I had a hunch that was the case.

Even though this is only "theoretical," I think it's safer to do what you do, ngoonee. I just added MakepkgArgs = -c to bauerbill.conf, and I'm going to leave it from now on to play it safe.

Last edited by flan_suse (2011-02-06 15:44:08)

Arch Linux

#951 2011-01-20 21:08:41

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#952 2011-01-21 13:29:35

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#953 2011-01-21 14:01:44

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#954 2011-01-21 14:23:17

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#955 2011-01-21 20:52:27

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#956 2011-01-28 03:33:11

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#957 2011-01-28 03:58:17

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#958 2011-01-28 04:45:04

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#959 2011-01-28 08:08:59

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#960 2011-01-28 11:27:50

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#961 2011-01-28 17:07:51

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#962 2011-01-31 02:27:14

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#963 2011-01-31 05:07:48

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#964 2011-01-31 05:28:06

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#965 2011-01-31 06:10:12

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#966 2011-01-31 09:13:33

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#967 2011-01-31 09:30:21

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#968 2011-01-31 10:16:00

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#969 2011-01-31 10:27:41

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#970 2011-02-04 18:24:05

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#971 2011-02-05 00:35:56

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#972 2011-02-06 05:56:55

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#973 2011-02-06 09:58:58

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#974 2011-02-06 12:53:10

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

#975 2011-02-06 15:15:00

Re: Bauerbill: Powerpill + ABS, AUR, CPAN and Hackage support + more

Board footer