More patches, this time to makepkg.

tehdely · 2004-09-06 07:17:33

Makepkg needs extensibility.

What sort of extensibility, you ask?

Anything that automates a common task when building packages.
Examples:

Checking sources out of CVS
Applying patches
Doing build activites that are common to a particular set of software (example: the necessary environment initialization which must precede any GNUstep builds)

Should we keep adding stuff to makepkg, and bloat it up? No.

Makepkg needs a pre-build stage.

I see more and more PKGBUILDs where a lot of what goes on in build() has nothing to do with building. Or installing, for that matter. Patching files... checking files out of CVS, etc. "makepkg -o" (another patch of mine) is useless for a lot of packages if the files are being obtained by some method other than ftp or http. Likewise, it's useless if they're going to be heavily patched before they're useable. A packager should be able to type "makepkg -o" and be presented with the source tree exactly as it will be before actual configuration/building begins.

Certain makepkg functions need to be separated from makepkg itself.

The specific examples I have in mind are warn, msg, and error. Why those three? Well, I see plenty of postinstall scripts that kludgily do something like the following:
echo "==> You now need to GNU your linux"
echo "==> Knock on Stallman's door three times"
echo "==> Yell 'freedom'"
Likewise, some packages install an installer that can be used later.
Having functions like warn, msg, and error in a separate file means that a script can simply source that file and, as a result, be able to display messages in the same manner as makepkg does during a build, thus resulting in consistency and saved time.

Now I could sit and whine, but that's pointless..

Here is how my system looks right now:

/etc/makepkg.d
-> functions <-- Contains warn, error, msg, and strip_url
-> patch <-- Contains routines to automate patching
-> cvs <-- Contains routines to automate CVS checkouts

And here's an example PKGBUILD that takes advantage of all this new loveliness:

pkgname=libxentac-cvs
pkgver=20040806
pkgrel=2
pkgdesc="A library which turns any computer into Canada"
depends=(libbritain libfrance)
source=(make-the-tragically-hip-sound-good.diff.gz international-relevance.patch.bz2)
md5sums=('e117f30d93856e0517ef0b2a5dcea383' 'e7169ccd28f81dbfca44add973d7d7eb')
patch=(make-the-tragically-hip-sound-good.diff international-relevance.patch)
cvsroot=":ext:anoncvs@savannah.nongnu.org:/cvsroot/xentac"
cvsmod="libxentac"
cvsrsh="ssh"

prebuild() {
    include cvs
    include patch
    sed -i s/foo/bar/ $startdir/src/$cvsmod/Makefile.in
}

build() {
    cd $startdir/src/$cvsmod
    ./configure --enable-lots-of-wheat --density=minimal
    make || return 1
    make DESTDIR=$startdir/pkg || return 1
}

And here's what running 'makepkg -o' might look like:

==> Entering fakeroot environment
==> Making package: libxentac-cvs  (Mon Sep  6 03:36:56 CDT 2004)
==> Checking Runtime Dependencies...
==> Checking Buildtime Dependencies...
==> Retrieving Sources...
==> Validating source files with MD5sums
==> Extracting Sources...
==> Preparing for checkout over SSH...
==> Fetching sources from CVS...
==> Applying make-the-tragically-hip-sound-good.diff at level 2...
patching file libxentac/src/hearing.c
patching file libxentac/src/taste.c
==> Applying international-relevance.patch at level 0...
patching file libxentac/src/telling-the-us-where-to-shove-it.c
==> Sources are ready.

Now I kid, I kid

I like Canada, the Tragically Hip, and Herr Doktor Chustick.

But the point remains. Those are some cool features, no?

I'm going to do some touch-up work on these, make a nice big diff, and post it up somewhere. Stay tuned :!:

link · 2004-09-06 07:32:39

I'm am wholeheartedly in support.
Patches and info/warning/error messages are two things I would really like to have.

dp · 2004-09-06 07:33:46

prebuild?

hmm

tehdely · 2004-09-06 08:05:41

Diff: http://dely.conio.net/mirror/arch/next-gen-makepkg.diff

I didn't include the "cvs" makepkg extension in this diff for two reasons.

1) The CVS_RSH="ssh" support is broken until a patch I submitted to Dorphell makes its way into Arch's CVS package.
2) In keeping with the Arch way of doing things, the cvs extension should be installed with cvs. Not with pacman. Why would somebody want a CVS extension if they didn't have CVS? I'm hoping that if all this catches on, someone will make a subversion extension to be packaged with svn, etc. I know I'll be packaging my gnustep extension along with gnustep-make

In any case, here are both of them, for your perusal.

patch

#!/bin/bash

for PATCH in ${patch[@]}; do
        if [ ! -f $startdir/src/$PATCH ]; then
                error "$PATCH does not exist."
                msg "Aborting..."
                exit 2
        fi
        for LEVEL in `seq 0 2`; do
        # Find a level at which the patch cleanly applies
                patch -d $startdir/src -fp$LEVEL --dry-run < $startdir/src/$PATCH 2>&1 >/dev/null
                if [ $? -gt 0 ]; then
                        if [ $LEVEL -eq 2 ]; then
                                error "Could not cleanly apply $PATCH at any patch level."
                                msg "Aborting..."
                                exit 2
                        fi
                        continue
                fi
                msg "Applying $PATCH at level $LEVEL..."
                patch -d $startdir/src -p$LEVEL < $startdir/src/$PATCH
        break
        done
done

cvs

#!/bin/bash

if [ "$cvsrsh" = "ssh" ]; then
        # This is an accomodation to the fact that a package build
        # should never affect anything outside of $startdir.
        # This includes the known_hosts file that ssh writes to.
        # Also, since a package build should be non-interactive,
        # It's important to set StrictHostKeyChecking to "no"
        # so you are not prompted to accept an unknown key.
        msg "Preparing for checkout over SSH..."
        [ -d $startdir/src/ssh ] || mkdir $startdir/src/ssh
        cat > $startdir/src/ssh/ssh_config << SSH_CONFIG_END
StrictHostKeyChecking no
UserKnownHostsFile $startdir/src/ssh/known_hosts
SSH_CONFIG_END
        export CVS_RSH="ssh -qF $startdir/src/ssh/ssh_config"
fi

if [ "$cvsroot" != "" -a "$cvsmod" != "" ]; then
        msg "Fetching sources from CVS..."
        if [ "$pkgver" = "cvs" ]; then
                # We're pulling today's CVS
                # This is _not_ recommended for packagers, who should
                # always assign a specific checkout date to $pkgver
                warning "It is preferable to version CVS-based"
                warning "PKGBUILDS with a specific checkout date."
                cvs -Q -d $cvsroot co $cvsmod
        else
                cvs -Q -d $cvsroot co -D $pkgver $cvsmod
        fi
fi

punkrockguy318 · 2004-09-06 14:13:22

Hey! Nice. Very Nice. I like your ideas, I think this will make a great addition to makepkg.

sarah31 · 2004-09-06 20:44:02

pre-build or regular build seems the same to me. Patches are not commonplace so having to add a whole new section to a build just for one small patch is a bit much.

cvs extensions do not belong in makepkg . CVS builds have no place in Extra and Current (there was at least one that had to be CVS because that was the only way it came when i packaged it).

tehdely · 2004-09-11 01:26:46

Sarah, I know we are frequently at odds but I will try to explain myself on this.

I think you misunderstand the main purposes of these additions to makepkg, as well as the nature of the additions itself.

First of all, the biggest change is making makepkg extensible. This can only be a good thing. There is lots of code duplication between PKGBUILDs, and having the ability to consolidate certain tasks into a single makepkg addition is good. My inspiration for this particular change was Portage which allows packagers to create "classes" and have ebuilds inherit from them; I think that's a bit much, but just extensibility as a feature in itself is a good goal.

Second, you need to look beyond whether something is good for packages that go in current/extra. One of the beauties of ABS is that it makes the package building process so simple that the average user is encouraged to package everything they install, instead of just installing it willy-nilly over their file system. ABS is not just a way for people to produce publically-distributable or core-repository-worthy packages; it is a way for them to make sure that their customization of their own system results in something maintainable for years to come. Adding things like a prebuild stage, and a "--pkg-only" target let users be lazier with PKGBUILDs that they make for themselves, and thus can only be a good thing for the casual packager-for-one's-own-system. As of right now, makepkg basically requires that a PKGBUILD be perfect before you can build a proper package; if something messes up during the build stage of a particularly messy build, you're forced to start over from scratch. When this involves CVS checkouts (once again, something many people will want to package for themselves, regardless of whether they belong in current/extra), you are not only wasting time but the bandwidth of the cvs server if you must check out again as you keep improving your PKGBUILD. That is a bad thing. Likewise with -pkg-only, the user doesn't even need his PKGBUILD to be correct at all... he can configure and build the software himself, then stage it into the destroot himself, and then have makepkg package it up into an installable package. I would hope nobody who makes packages for inclusion in any Arch repository does this, but I am lazy, most people are lazy, and I see no reason why makepkg shouldn't accomodate laziness. One of the biggest mistakes of overly-complicated systems like RPM or dpkg is that they make package building a voodoo art, since they focus on packages as solely a distributable thing. I think the pragmatist realizes that most packaging, at least with a simple as system as abs, is for one's own self, and why not make it simpler for them? What flies for my own system obviously doesn't fly for [current], but it should fly nonetheless

Praise Bob.

@@@@@@@^^~~~~~~~~~~~~~~~~~~~~^^@@@@@@@@  
@@@@@@^     ~^  @  @@ @ @ @ I  ~^@@@@@@  
@@@@@            ~ ~~ ~I          @@@@@  
@@@@'                  '  _,w@<    @@@@  
@@@@     @@@@@@@@w___,w@@@@@@@@  @  @@@  
@@@@     @@@@@@@@@@@@@@@@@@@@@@  I  @@@  
@@@@     @@@@@@@@@@@@@@@@@@@@*@[ i  @@@  
@@@@     @@@@@@@@@@@@@@@@@@@@[][ | ]@@@  
@@@@     ~_,,_ ~@@@@@@@~ ____~ @    @@@  
@@@@    _~ ,  ,  `@@@~  _  _`@ ]L  J@@@  
@@@@  , @@w@ww+   @@@ww``,,@w@ ][  @@@@  
@@@@,  @@@@www@@@ @@@@@@@ww@@@@@[  @@@@  
@@@@@_|| @@@@@@P' @@P@@@@@@@@@@@[|c@@@@  
@@@@@@w| '@@P~  P]@@@-~, ~Y@@^'],@@@@@@  
@@@@@@@[   _        _J@@Tk     ]]@@@@@@  
@@@@@@@@,@ @@, c,,,,,,,y ,w@@[ ,@@@@@@@  
@@@@@@@@@ i @w   ====--_@@@@@  @@@@@@@@  
@@@@@@@@@@`,P~ _ ~^^^^Y@@@@@  @@@@@@@@@  
@@@@^^=^@@^   ^' ,ww,w@@@@@ _@@@@@@@@@@  
@@@_xJ~ ~   ,    @@@@@@@P~_@@@@@@@@@@@@  
@@   @,   ,@@@,_____   _,J@@@@@@@@@@@@@  
@@L  `' ,@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

sarah31 · 2004-09-11 08:15:04

a prebuild will not require that a build be any less "perfect". when source is screwed up it is screwed up and pactches are necessary. and whether they are in a prebuild the patching needs to be done.

not sure if i see how your patches fix having to "start all over" if you are patching or using cvs source. I understand not wanting to dl cvs if you are patching and what not but i don't see how the patches fix this.

all i see is that more lines are added to the array section and another buld area is added.

don't get me wrong i am not opposed to making things for lazy people but i just don't see how this is doing that. this way seems just as messy.

dtw · 2004-09-22 14:10:03

hmm - well of course i have come in here from the thread i started. i think tehdely is spot on. i think that currently the building of packages is fairly restricted.

i am all for the cvs patch and being able to continue/start a build straight from the src. i had a frustrating experience the other day where i was building a kernel. one of the patches i was using was gzipped. when i ever i started the build it insisted on extracting all the source tarballs again. it would happily churn out the kernel source, not making any changes and then fail cos the patch had already been untarred and it wouldn't over write it. so i had to start over - again waiting while it extarcted the kernel sources, which of course it didn't cos they were already there.

for this reason i think having a prebuild seperate from a build which can be selected using a commandling switch is an excellent idea and i fail to see how it might complicate things for anyone with half a brain.

i also feel that sarah has, in several posts, showed an apparent unwillingness to consider the requirements of a normal user beyond what may go in the official repos.

only the other day on IRC i saw someone hassling a fluxbox users for an ebuild for some crappy fluxbox extension that no one uses. the absolute beauty of Arch, as tehdely has said, is that anyone can use ABS to create and install all sorts of stuff.

i for one would greatly appreciate extensions in makepkg and i totally agree with tehdely that they should not be a part of makepkg per se but should be included elsewhere - although i am surprised that tehdely has not actually made package builds to patch and install these extentions already

makepkg is simple because it is well designed - not because it has limited functionality. increased, well designed functionality will not reduce it's simplicity

p.s. that is a hint about pkgbuilds

sarah31 · 2004-09-22 16:09:14

well i am a normal user but adding features and new "standards" to makepkg then forces maintainers to adopt those changes. Some of these extensions do not even provide any benefit to the maintainers .... even the continuation option (which is nice idea i might add) should not be an automatic feature as when you have to apply a patch you would likely need to start from scratch building again to have the patch properly applied (ie if configure says a feature is not there and a patch applied later then, regardless of the patch the source will not be built) .

besides adding features to makepkg when there is alot of really sloppy packaging happening is not going to cure the problem. what arch NEEDS more than makepackage extras for those cutting edge people is a set of packaging standards and policies for introducing packages to the repos so we , the users, can avoid the xorg/kde crap and other big gaffs. after this is done then add the features.

of course the ass backwards nature of arch will likely see these patches before standards.

Arch Linux

#1 2004-09-06 07:17:33

More patches, this time to makepkg.

#2 2004-09-06 07:32:39

Re: More patches, this time to makepkg.

#3 2004-09-06 07:33:46

Re: More patches, this time to makepkg.

#4 2004-09-06 08:05:41

Re: More patches, this time to makepkg.

#5 2004-09-06 14:13:22

Re: More patches, this time to makepkg.

#6 2004-09-06 20:44:02

Re: More patches, this time to makepkg.

#7 2004-09-11 01:26:46

Re: More patches, this time to makepkg.

#8 2004-09-11 08:15:04

Re: More patches, this time to makepkg.

#9 2004-09-22 14:10:03

Re: More patches, this time to makepkg.

#10 2004-09-22 16:09:14

Re: More patches, this time to makepkg.

Board footer