You are not logged in.

#1 2013-08-04 14:29:16

jacobgrevald
Member
From: Denmark
Registered: 2013-08-01
Posts: 6
Website

Build option in Pacman

Hi there,

Sorry if this is a double post.
I know of srcpac, but have it been discussed if pacman should have an option to compile packages? It doesn't seam like a particularly big feature---I would even be willing to make it myself. If this has been discussed, what were the main arguments pro/con?

Jacob

Offline

#2 2013-08-04 14:34:48

karol
Archivist
Registered: 2009-05-06
Posts: 25,422

Re: Build option in Pacman

Errr, why not use makepkg for that?


Edit: And wrt to srcpac: https://bugs.archlinux.org/task/36140

Last edited by karol (2013-08-04 14:35:27)

Offline

#3 2013-08-04 14:44:59

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,371
Website

Re: Build option in Pacman

If you're not familiar with ABS, you should read up on it, once your abs tree is up to date just `cp /var/abs/${repo}/$pkg/* ./ && makepkg -si`

If that is too much, you can make a tiny bash function that will - given a package name - find which repo it is in, make sure the abs tree is up to date, make a suitable build dir, and run the above commands.  I have such a function in one of my bashrc's.

Last edited by Trilby (2013-08-04 14:45:13)


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#4 2013-08-04 14:51:37

jacobgrevald
Member
From: Denmark
Registered: 2013-08-01
Posts: 6
Website

Re: Build option in Pacman

karol wrote:

Errr, why not use makepkg for that?

For several reasons---Primarily ease of use. I don't really follow your logic. makepkg is not a package manager. It doesn't handle updates etc.

You could make the argument that it is not the package manager's job to compile the packages. Yet, a similar argument could be made that it is not the package manager's job to fetch the packages, though most would agree that this is a nice feature to have in pacman.

Ease of use:
pacman --build -S package
pacman --build -Syu

That second line could take some time with makepkg and pacman -U!

Jacob.

Last edited by jacobgrevald (2013-08-04 14:57:21)

Offline

#5 2013-08-04 14:55:16

Scimmia
Bug Wrangler
Registered: 2012-09-01
Posts: 4,691

Re: Build option in Pacman

The authors specifically don't want to obfuscate that part of it; if you're going to be building packages, you need to know how to use makepkg, not just a frontend.

Offline

#6 2013-08-04 14:56:21

jacobgrevald
Member
From: Denmark
Registered: 2013-08-01
Posts: 6
Website

Re: Build option in Pacman

Trilby wrote:

If you're not familiar with ABS, you should read up on it, once your abs tree is up to date just `cp /var/abs/${repo}/$pkg/* ./ && makepkg -si`

If that is too much, you can make a tiny bash function that will - given a package name - find which repo it is in, make sure the abs tree is up to date, make a suitable build dir, and run the above commands.  I have such a function in one of my bashrc's.

Yes. And it would be just as easy to make a bash script which uses wget to fetch packages. Yet this functionality is part of pacman.

The ability to do without, does not alone justify leaving the feature out of pacman. My question was not how to do this, but rather if it has been discussed to make it a part of pacman.

Offline

#7 2013-08-04 15:19:23

karol
Archivist
Registered: 2009-05-06
Posts: 25,422

Re: Build option in Pacman

I don't know how about you, but I use aliases, like 'msi' for 'makepkg -si' and 'psyu' for 'pacman -Syu'.

If you want to rebuild many packages, you need a working srcpac or a similar tool.

Offline

#8 2013-08-04 16:17:30

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,092
Website

Re: Build option in Pacman

jacobgrevald wrote:

I don't really follow your logic. makepkg is not a package manager. It doesn't handle updates etc.

And pacman is not a build tool. See how we're separating responsibilities?

Offline

#9 2013-08-04 16:39:58

dolik.rce
Member
From: Czech republic
Registered: 2011-05-04
Posts: 43

Re: Build option in Pacman

Yaourt has exactly this feature, 'yaourt -Sb <package>' or 'yaourt -Syub' work just fine. I use it only occasionally, but I agree with OP that having such thing in pacman would be a good thing.

I don't see any problem with separating responsibilities, as long as pacman just calls makepkg to do the built. Just as 'makepkg -i' calls pacman and nobody has problem with that. If the idea would be to reimplement the package building in its own code, then of course it would be breaking the responsibility separation, but the way OP puts it, it seems OK to me.

I second his question, whether someone actually thought about this and/or discussed it before? Nobody in this thread has even tried to answer that yet...

Offline

#10 2013-08-04 16:52:19

karol
Archivist
Registered: 2009-05-06
Posts: 25,422

Re: Build option in Pacman

I still don't understand the ease of use thing: if you can use an alias, why does it matter if it's makepkg, pacman, yaourt or something else?

Offline

#11 2013-08-04 17:22:37

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,092
Website

Re: Build option in Pacman

dolik.rce wrote:

I second his question, whether someone actually thought about this and/or discussed it before? Nobody in this thread has even tried to answer that yet...

Allan posted on pacman-dev some time ago about the idea of adding a -B option to build via pacman. It didn't really go anywhere. Neither will this thread/idea if it isn't brought up with the right people, cogent reasons presented, and patches offered.

Offline

#12 2013-08-04 17:41:22

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,371
Website

Re: Build option in Pacman

As far as I know (which may not be all that far in that case - so correct me if I'm wrong) pacman doesn't have a currently built in mechanism to get PKGBUILDS.  Built pkg.tar.gz files which are downloaded from repo mirrors do not include the PKGBUILD.  It seems it would be trivial to have pacman call makepkg as they are part of the same package: if you have pacman, you have makepkg.  This is not the case for ABS though.  It seems such an option would *require* abs, or a substitute for it.  So while a command line switch would be easy to add, this would also be adding another dependency and the requirement of maintaining (at least a partial) abs tree on the user's system.

I'm quite happy to have this package, and the abs tree - but I'm also happy that I chose to install it as a separate tool.  I can't see what possible advantage merging the two packages (pacman + abs) would serve.  The *only* thing mentioned so far is the alleged ease of use, but this has been answered with the suggestion of an alias.  Or I'll go one further, a function:

pacman() {
   if [[ "$@" =~ "--build" ]]; then
      # insert previously mentioned abs function code
   else
      shift
      /usr/bin/pacman $@
   fi
}

InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#13 2013-08-04 18:09:28

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,661

Re: Build option in Pacman

What is the advantage of building ABS packages if you aren't customising them at all? (I assume a --build option for pacman would not customise them if it worked in the way suggested i.e. updating abs, copying the relevant build recipe, compiling and installing it.)


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

#14 2013-08-04 18:11:33

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,371
Website

Re: Build option in Pacman

It depends what you mean by "customizing".  One can define compiler flags and machine-specific optimizations in makepkg.conf which would be applied.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#15 2013-08-04 18:13:07

dolik.rce
Member
From: Czech republic
Registered: 2011-05-04
Posts: 43

Re: Build option in Pacman

Trilby wrote:

It seems such an option would *require* abs, or a substitute for it.

The dependency on abs could be optional, using -B without having abs installed would result in error stating that abs should be installed first...

Also, each PKGBUILD can be downloaded separately from the mirrors, e.g.: http://ftp5.gwdg.de/pub/linux/archlinux … pkg.tar.xz. This could be used as a fallback in case abs is not installed.

Offline

#16 2013-08-04 18:48:38

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 947

Re: Build option in Pacman

dolik.rce wrote:

Yaourt has exactly this feature, 'yaourt -Sb <package>' or 'yaourt -Syub' work just fine. I use it only occasionally, but I agree with OP that having such thing in pacman would be a good thing.

I don't see any problem with separating responsibilities, as long as pacman just calls makepkg to do the built. Just as 'makepkg -i' calls pacman and nobody has problem with that. If the idea would be to reimplement the package building in its own code, then of course it would be breaking the responsibility separation, but the way OP puts it, it seems OK to me.

I second his question, whether someone actually thought about this and/or discussed it before? Nobody in this thread has even tried to answer that yet...

First, if you want a robust implementation of source-installs (download src, compile, install), ditch yaourt and go for gentoo because it implements the procedure the right way. However, even their approach has problems IMHO:
1. Make dependencies. Package X needs pkgs M_1, M_2, ... to build which has to be installed on your system. That's either unnecessary bloat or disk writes if you remove M_x after each source-install.
2. Building as root. To avoid this, pacman will have to somehow drop priviliges for building...

Have you guys given thought to addressing these issues? Simply put, building and installing packages in a single step is messy. Even so that at some point FreeBSD people tried to fix this in their ports system,
http://forums.freebsd.org/showthread.php?t=6975 (don't know how the story ended though).

Last edited by Leonid.I (2013-08-04 18:49:19)


Arch Linux is more than just GNU/Linux -- it's an adventure

Offline

#17 2013-08-04 18:49:10

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,661

Re: Build option in Pacman

Trilby wrote:

It depends what you mean by "customizing".  One can define compiler flags and machine-specific optimizations in makepkg.conf which would be applied.

Oh, I see. So is the idea to compile everything rather than just particular packages?


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

#18 2013-08-04 19:00:15

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 947

Re: Build option in Pacman

Trilby wrote:

If you're not familiar with ABS, you should read up on it, once your abs tree is up to date just `cp /var/abs/${repo}/$pkg/* ./ && makepkg -si`

If that is too much, you can make a tiny bash function that will - given a package name - find which repo it is in, make sure the abs tree is up to date, make a suitable build dir, and run the above commands.  I have such a function in one of my bashrc's.

I usually use the SVN tree. The main problem with ABS is that you don't usually know why a given PKGBUILD has a particular entry. So, you have go to svntogit (web interface) and read packager's comments. I guess ABS could include SVN history via a "changelog" but that'll be reinventing the wheel smile

Last edited by Leonid.I (2013-08-04 19:01:05)


Arch Linux is more than just GNU/Linux -- it's an adventure

Offline

#19 2013-08-04 19:59:15

sitquietly
Member
From: Moscow, Tennessee
Registered: 2010-07-12
Posts: 216

Re: Build option in Pacman

jacobgrevald wrote:

Hi there,

...has it been discussed if pacman should have an option to compile packages? It doesn't seam like a particularly big feature.....

I think that it is not only a difficult feature to add, but from my experience it would end up making something other than "Archlinux".  The idea of building our packages from source code on the user's system breaks the Arch development model.  The binary packages that are released as Archlinux are built on developer's machines against the libraries and tools that the developer happens to have installed at that moment.  There is a huge discipline required for the dev's to stage some changes that impact one another and to serialize changes that involve soname bumps in libraries, but that does not guarantee that the day after a binary package is released by Arch you can successfully build it on your machine using a currently up-to-date system and the released PKGBUILD for that package.  For example a recently released dependency may have removed some macros and headers, but not broken the library interface (no soname bump).  The Arch devs will eventually have to correct (patch) all other packages that build against that dependency, if upstream doesn't fix it first, but they don't do it immediately.  They do it the slacker way, "eventually when it's really necessary". 

In other words, there is no Archlinux build server and packages are not routinely rebuilt (Allan routinely rebuilds core and ensures that it builds).  Really that's the price you pay for continuously bleeding-edge packages.  I don't think there is enough time in the day for Arch devs to also ensure that all of their packages are continuously buildable.  It's actually a big task.

That's one of the things that differentiates Arch from the Big Boys like Fedora and OpenSUSE.  When they put out a 6-month or 8-month release they check that everything in the release is rebuildable.  In a rolling-release the only way to guarantee that all of the packages are rebuildable from source code would be to rebuild everything continuously (with a continuous build server or force users to do it like Gentoo does).

I'm not arguing against working from ABS and building packages from source -- I don't have any Arch binaries on my Archlinux system.  Everything on my system is built from abs/aur pkgbuilds but achieving that requires that I do little releases of my own with periods of "freeze".  It forces me to develop my system more like Debian (but faster with my "releases" coming every couple of months).

Summary: pacman should not have a build option because it's bad factoring and it would promote a false expectation.

Offline

#20 2013-08-04 20:46:40

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,371
Website

Re: Build option in Pacman

cfr wrote:

So is the idea to compile everything rather than just particular packages?

Not necessarily everything.  If/when compiler optimizations make a noteworthy difference, they'd make that difference even on single packages.  So some users may opt for compiling "everything" if they are patient and/or have a fast system (though if their system is that fast, I'd ask why they need the optimizations), others may compile just select packages: either those that are used most frequently, most likely to benefit from machine specific optimizations, or otherwise are strategically chosen.

For example, I use urxvt *a lot* - I've compiled my own urxvt with optimizations for my system.  I don't know if it really makes a difference though, and this was not really the purpose of recompiling urxvt, I was actually making configuration changes.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#21 2013-08-04 22:06:18

Allan
Developer
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,327
Website

Re: Build option in Pacman

As falconindy pointed out above, I want this in pacman.   The initial parts would be quite easy to add and would replace the steaming pile of crap that is ABS.

Patches we need:

1) Option in repo-add to add details of the source package (if the coreesponding foo-1-1-i686.src.pkg.tar.gz is in the directory with the binary package, its details including signature get added to the repo db)
2) Add -B option to pacman, initially only supporting -Bw to download the source package.

Those two steps set us up to be able to build packages, but more importantly provide a superior system than the current ABS.  Actual building etc (and patching the files before that build) are all features that require some actual thought.

But as always, we need someone to do the work.  I barely get through reviewing patches these days and this is not in my high priority queue of things to implement.  So if someone steps up and does the work....

Offline

#22 2013-08-04 22:09:56

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 5,661

Re: Build option in Pacman

Hmm... Maybe I should try it for kile/okular. kile/okular are much slower on my Arch machine than my Fedora box at work even though my Arch machine is generally faster and has generally higher specs (better cpu, more memory etc.). Then again, I wonder how much time I would spend to save each second!


How To Ask Questions The Smart Way | Help Vampires

Arch Linux | x86_64 | GPT | EFI boot | grub2 | systemd | LVM2 on LUKS
Lenovo x121e | Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz GenuineIntel | Intel Centrino Wireless-N 1000 | US keyboard with Euro | 320G 7200 RPM Seagate HDD

Offline

#23 2013-08-05 00:45:24

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: Build option in Pacman

This question seems to crop up every few months, for some odd reason.  srcpac has been mentioned already; pacbuilder was another attempt at this, allowing the user to recompile the system and run updates according to the settings in makepkg.conf.  I think srcpac has a configuration file that applies package-specific sed magic to "automatically" apply customizations to PKGBUILDs, and there was some other script that could be paired with pacbuilder to allow for the same thing.  I don't remember that script's name, though.

I'm not sure how much benefit adding such a feature to Pacman would have beyond making things easier on the developers; Arch was never intended to be a source-based distro, pacman is an excelent binary package manager, the and the repositories are very well-maintained to minimize the installation footprint.  I'm with Trillby on this: The ability to compile from the ABS is great, but I doubt the average user customizes more than a few packages, or that recompiling everything would be of great benefit.

Offline

#24 2013-08-05 05:56:09

isacdaavid
Member
From: México
Registered: 2011-11-21
Posts: 76
Website

Re: Build option in Pacman

ANOKNUSA wrote:

This question seems to crop up every few months, for some odd reason.

Because of the ABS and PKGBUILDS it is natural to expect some tool that makes automated compilation, in spite Arch isn't supposed to be source-based by default. But heck, how many binary distros have a ports/ebuild-like system? Arch is unique in that regard and I think it's being underrated; this could become the first officially supported hybrid distribution ever. Nice to hear directly from Allan and the opposed opinions, for instance, sitquietly's argument.

I coincidently found this fresh thread searching the web because I cannot make up my mind with building a GNU/Linux system from source.  Given that I'm a happy Arch user I would like to try out the ABS before experimenting with Gentoo, and I also think Arch possesses certain advantages over the latter. So here I am to ask you: what method would you recommend for massively building and installing packages from its PKGBUILDS? srcpac is not even mentioned in the wiki and hasn't been version-bumped in a while, yet has an official package; whereas pacbuilder has a development entry on the AUR. Maybe raw makepkg and shell scripting, or maybe yaourt?

Edited: didn't know about Sabayon until now

Last edited by isacdaavid (2013-08-13 13:36:57)

Offline

#25 2013-08-05 17:22:26

sitquietly
Member
From: Moscow, Tennessee
Registered: 2010-07-12
Posts: 216

Re: Build option in Pacman

Trilby wrote:

As far as I know (which may not be all that far in that case - so correct me if I'm wrong) pacman doesn't have a currently built in mechanism to get PKGBUILDS.  ..... I'm quite happy to have this package, and the abs tree - but I'm also happy that I chose to install it as a separate tool.  I can't see what possible advantage merging the two packages (pacman + abs) would serve.  .....

I don't see any advantage of adding a -B option to pacman.  The advantage of separating the "build from source" functionality is (imo) greatly increased flexibility.  As others have noticed there is not much reason to just build packages from source with custom CFLAGS.  The real motivation for using the source is to control dependencies and options, and possibly to integrate mutliple upstream sources (abs, aur, ...).

For example I build everything from source using my own "build" command.  It integrates abs and aur (in fact, it optionally integrates the unity, antergos, and manjaro source repos). So for example I can "build firefox-kde-opensuse" and build an aur package.  Or "build unity" and build the unity meta-package.  All "build" knows is that it should follow a search path in a collection of "upstream buildscripts" until it finds the desired package.  A "pacman -B" option is probably not going to be that simple or allow for that integration.  ("build" also needs access to a database of packages so that it knows the "pkgbase" corresponding to any "pkgname").

To further the example:  the function for fetching the pkgbuild is

FROMDIR=""
function from() { # dir -- copy dir to Buildfolder and set FROMDIR
    cp -r --no-clobber $1 "${Buildfolder}"
    FROMDIR=$1
}

# get the Pkgbase directory into $Buildfolder and initialize the dirty Pkgbase_build
function get_pkgbuild() {
    cd "${Buildfolder}"
    [ -d "${Pkgbase}_build" ] && rm -rf "${Pkgbase}_build"
    if [[ $REBUILD == 1 && -d $MASTER/$Pkgbase ]]; then
        from "$MASTER/$Pkgbase"
        cp -r "${Buildfolder}/$Pkgbase"  "${Buildfolder}/${Pkgbase}_build" 
        return 0
    fi
    for repo in ${UPSTREAM[*]}; do
        if [ -d $repo ]; then
            cd $repo
            for folder in */; do
                if [[ $folder == $Pkgbase/ ]]; then
                    from "${repo}/${Pkgbase}"
                    cp -r "${Buildfolder}/$Pkgbase"  "${Buildfolder}/${Pkgbase}_build" 
                    return 0
                fi
            done
        fi
    done
    return 1
}

Just adding directories to the UPSTREAM array gives access to abs/core, abs/extra, abs/community, aur, unity, ... whatever.

The help message for "build" may give an idea of where you may end up by following this kind of factoring that I'm recommending:

└─> build --help
------------------------------------------------------------------------------
A utility for Archlinux to build packages from source code.

Syntax:    build options packages

Build the packages named on the command line (and packages listed in the file
using the -f option). Packages are built in the order listed.  Packages named
on the command line are built after any given in a file.

Use control-c to interrupt the build then "build --resume" when ready to continue.

Options:
        --rebuild    Copy pkgbuilds from the master repository, ignore upstream
    -c, --chroot     Build packages in $MYROOT using makechrootpkg, initializing
                     the root with packages from the mylinux current repo (or from
                     Archlinux upstream if the package is not in the mylinux repo).
    -f file          Read package names from file.  Only one file can be used.
    -m, --merge      Interactively merge in changes master repository.
    -e, --edit       Edit each PKGBUILD before building the package.
    -k, --keep-going Do not stop to show namcap errors.
    -L, --log        Enable makepkg build logging (do not use for interactive builds).
    -r, --resume     Resume the previously interrupted build in this directory.
    -s, --save       Save the buildscript for each package (see build.conf).
    -S, --src        Also save the unpacked and patched source code (./src)
    -h, --help       Print this message.

Examples:
    Compile the kernel, headers, and docs:
        build -mes linux
    Upgrade all packages except those in groups that are in FREEZE:
         pacman -Qq | versions -qu | freeze | order | unify >upgrades
         build -msLf upgrades

Best practice:
  build is intentionally dumb and builds just what you ask for in the way you specify.
  * always use the -s option to accumulate pkgbuilds into a local master collection
  * always use the -m option to merge back into upstream any local changes
  * always use the order command to get packages in correct build order

See also the help messages for:
  broken  freeze  group  need  order  packages  pkgtree  release  requires
  unify  use  versions  update-package-database
------------------------------------------------------------------------------

I know that that is way too much info but maybe the details of an example can stimulate folks who think similarly.

Last edited by sitquietly (2013-08-05 23:59:46)

Offline

Board footer

Powered by FluxBB