Selectively installing split packages

mrfaber · 2018-09-30 03:34:34

Let me start by illustrating my problem:

I would like to build acpi_call against linux-ck, specifically linux-ck-skylake. Right now, I am using this PKGBUILD(shortened):

pkgbase=acpi_call-ck
pkgname=(acpi_call-ck-skylake)

pkgver=1.1.0
pkgrel=1
pkgdesc='A linux kernel module that enables calls to ACPI methods through /proc/acpi/call'
depends=('linux-ck>=4.18' 'linux-ck<4.19')
makedepends=('linux-ck-headers>=4.18' 'linux-ck-headers<4.19')
_extramodules=extramodules-${pkgname#acpi_call-}
source=("${pkgname}-${pkgver}.tar.gz::https://github.com/mkottman/acpi_call/archive/v${pkgver}.tar.gz")
sha256sums=('d0d14b42944282724fca76f57d598eed794ef97448f387d1c489d85ad813f2f0')

prepare() {
  cd ${pkgname%%-*}-${pkgver}

  # Fix build with Linux >= 3.17
  sed -i 's|acpi/acpi.h|linux/acpi.h|' acpi_call.c

  # Fix build with Linux >= 4.12
  sed -i 's|asm/uaccess.h|linux/uaccess.h|' acpi_call.c
}

build() {
  cd ${pkgname%%-*}-${pkgver}
  make KVERSION="$(cat /usr/lib/modules/$_extramodules/version)"
}

package-generic() {
  cd ${pkgname%%-*}-${pkgver}
  install -Dt "$pkgdir/usr/lib/modules/$_extramodules" -m644 *.ko
  find "$pkgdir" -name '*.ko' -exec xz {} +

  echo ${pkgname%%-*} | install -Dm644 /dev/stdin "$pkgdir/usr/lib/modules-load.d/${pkgname}.conf"

  mkdir -p "$pkgdir/usr/share/${pkgname}"
  cp -t "$pkgdir/usr/share/${pkgname}" -dr --no-preserve=ownership examples support
}

# This would fail because /usr/lib/modules/extramodules-ck/ is not present
#package_acpi_call-ck() {
#  package-generic
#}

package_acpi_call-ck-skylake() {
  package-generic
}

Now this could be made much more versatile if one could simply add the other linux-ck versions to pkgname and call something like

package_acpi_call-ck-skylake() {
  package-generic
}
package_acpi_call-ck-haswell() {
  package-generic
}
...

As I understand right now, split packages (i.e. those with a PKGBUILD containing both pkgbase and multiple pkgnames) will be built in their entirety by makepkg.

This makes in unfeasible to provide a package base PKGBUILD, since makepkg would attempt to create a package for all targets and fail because the specific linux-ck-* packages are missing.

So I've patched my makepkg to give it an -O or --only flag that accepts a single pkgname:

diff --git a/usr/bin/makepkg b/makepkg-split
index 359ee0e..f4cf145 100755
--- a/usr/bin/makepkg
+++ b/makepkg-split
@@ -85,6 +85,7 @@ SKIPCHECKSUMS=0
 SKIPPGPCHECK=0
 SIGNPKG=''
 SPLITPKG=0
+SPLITONLYPKG=0
 SOURCEONLY=0
 VERIFYSOURCE=0
 
@@ -1179,6 +1180,7 @@ usage() {
 	printf -- "$(gettext "  -p <file>        Use an alternate build script (instead of '%s')")\n" "$BUILDSCRIPT"
 	printf -- "$(gettext "  -r, --rmdeps     Remove installed dependencies after a successful build")\n"
 	printf -- "$(gettext "  -R, --repackage  Repackage contents of the package without rebuilding")\n"
+	printf -- "$(gettext "  -O, --only <pkg> Build only the specified pkgname of a group of split packages")\n"
 	printf -- "$(gettext "  -s, --syncdeps   Install missing dependencies with %s")\n" "pacman"
 	printf -- "$(gettext "  -S, --source     Generate a source-only tarball without downloaded sources")\n"
 	printf -- "$(gettext "  -V, --version    Show version information and exit")\n"
@@ -1234,12 +1236,12 @@ fi
 ARGLIST=("$@")
 
 # Parse Command Line Options.
-OPT_SHORT="AcCdefFghiLmop:rRsSV"
+OPT_SHORT="AcCdefFghiLmoO:p:rRsSV"
 OPT_LONG=('allsource' 'check' 'clean' 'cleanbuild' 'config:' 'force' 'geninteg'
           'help' 'holdver' 'ignorearch' 'install' 'key:' 'log' 'noarchive' 'nobuild'
           'nocolor' 'nocheck' 'nodeps' 'noextract' 'noprepare' 'nosign' 'packagelist'
           'printsrcinfo' 'repackage' 'rmdeps' 'sign' 'skipchecksums' 'skipinteg'
-          'skippgpcheck' 'source' 'syncdeps' 'verifysource' 'version')
+          'skippgpcheck' 'source' 'only:' 'syncdeps' 'verifysource' 'version')
 
 # Pacman Options
 OPT_LONG+=('asdeps' 'noconfirm' 'needed' 'noprogressbar')
@@ -1281,6 +1283,7 @@ while true; do
 		--noprepare)      RUN_PREPARE='n' ;;
 		--nosign)         SIGNPKG='n' ;;
 		-o|--nobuild)     NOBUILD=1 ;;
+		-O|--only)        shift; SPLITONLYPKG=$1 ;;
 		-p)               shift; BUILDFILE=$1 ;;
 		--packagelist)    PACKAGELIST=1 IGNOREARCH=1;;
 		--printsrcinfo)   PRINTSRCINFO=1 IGNOREARCH=1;;
@@ -1321,7 +1324,7 @@ trap 'trap_exit INT "$(gettext "Aborted by user! Exiting...")"' INT
 trap 'trap_exit USR1 "$(gettext "An unknown error has occurred. Exiting...")"' ERR
 
 # preserve environment variables to override makepkg.conf
-restore_envvars=$(declare -p PKGDEST SRCDEST SRCPKGDEST LOGDEST BUILDDIR PKGEXT SRCEXT GPGKEY PACKAGER CARCH 2>/dev/null || true)
+restore_envvars=$(declare -p PKGDEST SRCDEST SRCPKGDEST LOGDEST BUILDDIR PKGEXT SPLITONLYPKG SRCEXT GPGKEY PACKAGER CARCH 2>/dev/null || true)
 
 # default config is makepkg.conf
 MAKEPKG_CONF=${MAKEPKG_CONF:-$confdir/makepkg.conf}
@@ -1494,6 +1497,11 @@ check_software || exit $E_MISSING_MAKEPKG_DEPS
 if (( ${#pkgname[@]} > 1 )); then
 	SPLITPKG=1
 fi
+# only operate on the specified pkgname if instructed to only build one single
+# package of a split package group
+if (( SPLITONLYPKG )); then
+    pkgname=("${SPLITONLYPKG}")
+fi
 
 # test for available PKGBUILD functions
 if have_function prepare; then

Obviously this is something targeted mainly towards the AUR and would make installing something like notmuch much nicer, and AUR helpers could give this flag to makepkg if they detect a split package(I think yay knows how to do this and some others as well).

But I think this should not only be included in makepkg because it eases AUR helper usage(which I know are very much unsupported) but also helps people using their own PKGBUILDs accross machines.

So, tell me, am I onto something here or should I just write and maintain 20 different PKGBUILDS for the linux-ck optimized packages? I know it is already a pain to parse PKGBUILDs and forcing this much polymorphism into the packaging system feels wrong.

Last edited by mrfaber (2018-09-30 03:40:52)

eschwartz · 2018-09-30 04:15:05

We removed this feature from makepkg.

...

I'm not really positive why we need any AUR packages for it at all, since the repos have a dkms version that automatically adapts to *all* your installed kernels, and it's not like providing a PKGBUILD to compile it and package the result, is any better than providing a dkms routine to compile it and copy it directly to your modules directory.

compiled module packages are prinicipally useful for creating prebuilt binary repositories.

...

Also see my broadcom-wl PKGBUILD in [community], for what I regard as a very good way to generically make a binary package.

mrfaber · 2018-09-30 04:49:26

Eschwartz wrote:

We removed this feature from makepkg.

Interesting, any particular reason?

Eschwartz wrote:

I'm not really positive why we need any AUR packages for it at all, since the repos have a dkms version that automatically adapts to *all* your installed kernels, and it's not like providing a PKGBUILD to compile it and package the result, is any better than providing a dkms routine to compile it and copy it directly to your modules directory.

My main concern is being able to bake dkms modules into the initrd, is this feasible? I also remember really bad experiences with vbox dkms stuff.

Eschwartz wrote:

compiled module packages are prinicipally useful for creating prebuilt binary repositories.

Yup.

Eschwartz wrote:

Also see my broadcom-wl PKGBUILD in [community], for what I regard as a very good way to generically make a binary package.

I see, you mean this PKGBUILD. Anyone wishing to compile a custom version simply appends to pkgname. But with this one would still have to write 20 files if one was to provide a binary repo for linux-ck extra packages. Maybe I should ask graysky how he does it.

It's not that I am just too lazy to edit PKGBUILDs by hand, where this proposal shines is with large "package bases" like notmuch where makepkg -s would pull in dependencies for and build *all* sub-packages.

Or, put another way, this is to make makepkg more potent, by allowing to write and having to maintain less files, through the power of inheritance. One PKGBUILD stays valid across many sub-packages.

Last edited by mrfaber (2018-09-30 04:57:27)

eschwartz · 2018-09-30 05:01:56

makepkg doesn't really do inheritance, there is no way to have per-split-package makedepends, and you really need to build everything in build() once. The only thing we support is packaging them separately. And we no longer support only running some of the package_*() functions.

I believe graysky does this the sane way, by creating a locally modified PKGBUILD. It would be simple, to use my PKGBUILD as an example, to merely sed the PKGBUILD to append the custom pkgname you want, run makepkg to compile the module, sed it again, rebuild with a different kernel, etc.

mrfaber · 2018-09-30 05:13:25

Hmm, that does make sense.
I also just noticed that I would have to move the make KVERSION=...* step to package_*() and that is just plain ugly.

So there goes my vision of putting acpi_call-ck* into the AUR with little pain

mrfaber · 2018-09-30 05:38:21

I've just published the build to the AUR, see acpi_call-ck-generic.

The PKGBUILD is:

# Maintainer: Adam Hirst <adam@aphirst.karoo.co.uk>
# Maintainer: Zuyi Hu <hzy0668808@gmail.com>
# Maintainer: mrfaber <mrfaber@example.com>
# Contributor: Maxime Gauduin <alucryd@archlinux.org>
# Contributor: mortzu <me@mortzu.de>
# Contributor: fnord0 <fnord0@riseup.net>


pkgbase=acpi_call-ck
pkgname=(acpi_call-ck-generic)
# replace -generic with your appropriate optimized ck version
         # acpi_call-ck-atom
         # acpi_call-ck-silvermont
         # acpi_call-ck-core2
         # acpi_call-ck-nehalem
         # acpi_call-ck-sandybridge
         # acpi_call-ck-ivybridge
         # acpi_call-ck-haswell
         # acpi_call-ck-broadwell
         # acpi_call-ck-skylake
         # acpi_call-ck-p4
         # acpi_call-ck-pentm
         # acpi_call-ck-kx
         # acpi_call-ck-k10
         # acpi_call-ck-bobcat
         # acpi_call-ck-bulldozer
         # acpi_call-ck-piledriver
         # acpi_call-ck-zen
pkgver=1.1.0
pkgrel=10
pkgdesc='A linux-ck kernel module that enables calls to ACPI methods through /proc/acpi/call'
url='https://github.com/mkottman/acpi_call'
arch=('x86_64')
license=('GPL')
depends=('linux-ck>=4.18' 'linux-ck<4.19')
makedepends=('linux-ck-headers>=4.18' 'linux-ck-headers<4.19')
provides=('acpi_call')
_extramodules=extramodules-${pkgname#acpi_call-}
source=("${pkgname}-${pkgver}.tar.gz::https://github.com/mkottman/acpi_call/archive/v${pkgver}.tar.gz")
sha256sums=('d0d14b42944282724fca76f57d598eed794ef97448f387d1c489d85ad813f2f0')

prepare() {
  cd ${pkgname%%-*}-${pkgver}

  # Fix build with Linux >= 3.17
  sed -i 's|acpi/acpi.h|linux/acpi.h|' acpi_call.c

  # Fix build with Linux >= 4.12
  sed -i 's|asm/uaccess.h|linux/uaccess.h|' acpi_call.c
}

build() {
  cd ${pkgname%%-*}-${pkgver}
  make KVERSION="$(cat /usr/lib/modules/$_extramodules/version)"
}

package() {
  cd ${pkgname%%-*}-${pkgver}
  install -Dt "$pkgdir/usr/lib/modules/$_extramodules" -m644 *.ko
  find "$pkgdir" -name '*.ko' -exec xz {} +

  echo ${pkgname%%-*} | install -Dm644 /dev/stdin "$pkgdir/usr/lib/modules-load.d/${pkgname}.conf"

  mkdir -p "$pkgdir/usr/share/${pkgname}"
  cp -t "$pkgdir/usr/share/${pkgname}" -dr --no-preserve=ownership examples support
}


# vim:set ts=2 sw=2 et:

This is my first push to the AUR, let me know if there's anything wrong with it.

eschwartz · 2018-09-30 05:46:35

Well, I don't much see the point of the -generic name. Also graysky's optimized packages use native optimizations selected by the kernel gcc patch and encoded in the /usr/lib/modules/*/build/.config for each variant -- in order to actually build them optimized, the installed linux-ck{,-headers} needs to specifically be atom, silvermont, core2, or whatever.

You can do that, by autoselecting it from the end of pkgname.

mrfaber · 2018-09-30 21:03:03

Ok, I've changed it to default to acpi_call-ck instead of -generic.

Eschwartz wrote:

You can do that, by autoselecting it from the end of pkgname.

It did not occur to me that you could use expressions in the depends and makedepends arrays.

Eschwartz wrote:

lso graysky's optimized packages use native optimizations selected by the kernel gcc patch and encoded in the /usr/lib/modules/*/build/.config for each variant

Do you mean I should source the .config file in the PKGBUILD? Or is changing the depends to the specific -ck variant enough?
Right now, the build() function calls

 make KVERSION="$(cat /usr/lib/modules/$_extramodules/version)"

where $_extramodules is the specific -ck variant.

eschwartz · 2018-10-03 13:32:32

mrfaber wrote:

It did not occur to me that you could use expressions in the depends and makedepends arrays.

You can use arbitrary bash.

Although if you try to do it inside the package() function itself, since makepkg does not execute it but simply greps it in order to do preliminary linting checks or generate .SRCINFO files, there are more restrictions...

Eschwartz wrote:
lso graysky's optimized packages use native optimizations selected by the kernel gcc patch and encoded in the /usr/lib/modules/*/build/.config for each variant
Do you mean I should source the .config file in the PKGBUILD? Or is changing the depends to the specific -ck variant enough?
Right now, the build() function calls
 make KVERSION="$(cat /usr/lib/modules/$_extramodules/version)"
where $_extramodules is the specific -ck variant.

No, I mean the kernel build system, which is used by out-of-tree modules, should already ensure the module is built with the same CFLAGS that the kernel itself was built with. It does this by saving the original .config as part of the /usr/lib/modules/${uname_r}/build/ directory (Kconfig, .config, include/ and more).

So, changing the depends to the specific -ck variant should be enough.

mrfaber · 2018-10-03 20:29:47

Ok, thank you!

Arch Linux

#1 2018-09-30 03:34:34

Selectively installing split packages

#2 2018-09-30 04:15:05

Re: Selectively installing split packages

#3 2018-09-30 04:49:26

Re: Selectively installing split packages

#4 2018-09-30 05:01:56

Re: Selectively installing split packages

#5 2018-09-30 05:13:25

Re: Selectively installing split packages

#6 2018-09-30 05:38:21

Re: Selectively installing split packages

#7 2018-09-30 05:46:35

Re: Selectively installing split packages

#8 2018-09-30 21:03:03

Re: Selectively installing split packages

#9 2018-10-03 13:32:32

Re: Selectively installing split packages

#10 2018-10-03 20:29:47

Re: Selectively installing split packages

Board footer