Makepkg simplification, makepkg-optimize; future libmakepkg extensions

quequotion · 2016-03-20 11:00:22

Allan McRae wants to minimize makepkg. I understand his motivation for doing so; nontheless for my own purposes I want makepkg to process as many packages as possible with every meaningful optimization it can.

Christian Hesse suggests a supplement to makepkg, as a better place for upx and image compression optimizations. I'd like to discuss the feasibility of this, options that could be included (with patches if possible), etc.

For example, I found qqbuild's lto and pgo optimizations integrate well but this kind of feature will no longer fit in makepkg's scope.

In response, I have proposed to extend BUILDENV with supplemental scripts in the same manner as OPTIONS is extended by "tidy" scripts. Were my proposal approved, I'd split makepkg-optimize into two supplemental packages for pacman: "libmakepkg-tidy" for post-packaging optimizations and "libmakepkg-buildenv" for build-time optimizations.

Update: I updated my proposal and it has been incorporated upstream; pacman-git's libmakepkg now supports BUILDENV extentions (and extensions to find dependent executables).

Note: It is possible to install these packages, alternatively, in your clean chroot, and use them only for clean-chroot building optimized packages.

makepkg-optimize is a collection of supplemental scripts for optimizing packages that also creates an additional makepkg config at /etc/makepkg-optimize.conf from your existing configuration. Unless you specify this config file to makepkg, packages will be built in the standard way.

How-to now available in the Archwiki !

Last edited by quequotion (2019-07-21 10:13:44)

quequotion · 2016-03-20 15:38:46

Initial effort

This is a bit premature, as the changes to makepkg have not yet landed. For now I've left static some variables that need to be replaced with actual parsing (system arch in makepkg-optimize.conf; libmakepkg dir in upx and optipng scripts), and makepkg-optimize.conf will be overwritten by updates (changes are being made to this file).

This adds back in the tidy scripts, upx and optipng, as well as the build options lto and pgo. It installs the fork makepkg-optimize and a config file makepkg-optimize.conf, but for the moment the scripts are commented out because they are still present in the current release of pacman.

I read that someone had an idea for svg optimization; I'd like to see it.

If any one wants to contribute, or even take over, please post here!

EDIT: Also added a Graphite build option!

Last edited by quequotion (2016-03-21 13:21:40)

Alad · 2016-03-20 16:30:05

Why don't we focus on improving the makepkg codebase (which has numerous bugs and dodgy mechanisms), instead of implementing every possible feature?

Last edited by Alad (2016-03-20 16:32:16)

quequotion · 2016-03-20 16:51:11

Alad wrote:

Why don't we focus on improving the makepkg codebase (which has numerous bugs and dodgy mechanisms), instead of implementing every possible feature?

Surely we can do both. I'm working on the part I know best.

I'm pretty happy with the makepkg codebase actually. There may be numerous bugs and dodgy mechanisms I am unaware of, but it's already the best package creation utility I have ever known. The simplicity of bash is a great delight to me.

I am a little dissapointed though, since my proposal to do these things in makepkg itself was rejected, that packagers will not be able to specify "!lto" or "!graphite" for packages known not to compile with such optimizations (those being the minority).

Last edited by quequotion (2016-03-20 16:51:56)

pypi · 2016-03-20 18:20:37

What's wrong with just providing the tidy scripts - I think that's what Allan had in mind?

ooo · 2016-03-20 19:43:40

Couldn't such optimization routines as upx, pngopt and svg optimization be added as custom pacman hooks as well, if user wants them?

quequotion · 2016-03-20 20:18:20

pypi wrote:

What's wrong with just providing the tidy scripts - I think that's what Allan had in mind?

I have use for both, the build options are the more important ones anyway, and why not? Allan plans to get anything non-essential out of makepkg, those included--I think you mean to cite Christian's proposal.

Here's an example of why it is important to have this in makepkg and why it works better in makepkg, particularly for the PGO* option. Any means by which to reduce the time and effort PGO requires are a good means:

qqbuild's pgo macro:

      if [ ! -d "$PWD/profile" ]; then
        mkdir "$PWD/profile"
        QQPGO="-fprofile-generate -fprofile-dir=$PWD/profile"
        QQPGOLDFLAGS="-lgcov"
      else
        [[ ! -d "$PWD/profile.used" ]] &&  mv "$PWD/profile" "$PWD/profile.used"
        QQPGO="-fprofile-correction -fprofile-use -fprofile-dir=$PWD/profile.used"
      fi
      shift

This puts the .gcda files in a directory called profile in the present working directory (one must be in the package directory) and passing that LD flag is quite problematic: with a wrapper, one may be forced to choose between makepkg.conf's "-Wl,-option,-option" and "-options -spaced -out" at odd points in the script, because white space creeps in somewhere. In this script, there are several reassignments toward the end. It works, but I always thought of it as messy. With it built into makepkg, the flags integrate much more smoothly with the user's choices in makepkg.conf

makepkg-optimize's pgo macro:

	if check_buildoption "pgo" "y"; then
		if [ ! -d "$PROFDEST/$pkgbase.gen" ]; then
			mkdir "$PROFDEST/$pkgbase.gen"
			CFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen"
			CXXFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen"
			LDFLAGS+=" -lgcov"
		else
			[[ ! -d "$PROFDEST/$pkgbase.used" ]] && mv "$PROFDEST/$pkgbase.gen" "$PROFDEST/$pkgbase.used"
			CFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used"
			CXXFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used"
		fi
	fi

Nearly the same code, but now we can have a PROFDEST, and no more swapping variables for variables like what happens later in the other script. The default behavior hasn't changed, .gcda files will be put in the build directory in a folder, now called "$pkgname.gen"--however, being part of makepkg opens the possibility of storing the profiles elsewhere (PROFDEST can be configured in makepkg-optimize.conf, like the other ~DEST variables), so the profiling data could be kept between builds more safely. It's handy, because you could--for example--build all your most frequently used programs with pgo at once--or all your DE's packages, with one yaourt command, use your computer normally for a day, then rebuild those same pacakges with the same command and have all your favorite software running faster than you've ever seen without having to edit each PKGBUILD one at a time, without having to PGO build each package one at a time, etc.

In theory, you could probably PGO build an entire installation--even the kernel--at once, but the collection phase would be rather hard on your CPU. I say in theory because, in practice, a small minority of packages do fail to compile when confronted with PGO or LTO.

EDIT: Another issue no one will ever have to know about if we can do PGO in (lib)makepkg:
qqbuild, as a wrapper script, is blind to what makepkg actually does. It moves the pgo directory to ".used" before makepkg actually runs. makepkg-optimize doesn't attempt this until it is inside build(), so the ".gen" folder stays right where it should be if makepkg doesn't start for some reason (ie, because it was called without -f and a package exists in the directory).

I don't see any of this as being out of place in makepkg at all.

EDIT: I should clarify this statement, as I am thinking about makepkg as more than just the "makepkg" bash script itself. I think these features belong in the package creator; integrating them as extensions to it is just as good.

It seems very logical to me that a package creator should offer some handy package tuning macros, particularly when just one or two flags will not suffice. These things are, after all, only possible during the creation of packages--where else would they go? I'm satisfied that a wrapper script will always be inferior. Furthermore, these options are all disabled by default; if you aren't interested in building packages with greater optimization, all you have to do is not enable the options.

*If you don't know about PGO I reccommend looking it up. You can squeeze quite a bit more performance from many programs with no loss of stability. At the very least it consistenly reduces load time. It takes two passes: build once, use the program for a while, then rebuild (with collected data).

Last edited by quequotion (2016-05-19 03:21:40)

quequotion · 2016-03-20 20:30:06

ooo wrote:

Couldn't such optimization routines as upx, pngopt and svg optimization be added as custom pacman hooks as well, if user wants them?

Maybe, I haven't looked into it. Those are scripts that run after packaging has completed, so it sounds fine.

I wouldn't be opposed to sending the scripts that way, and keeping the build options in makepkg(-optimize).

The "tidy" script that calls them at the end of the packaging process is pretty handy. It calls the scripts with the same filename.sh as the options specified, without limitation. One so inclined could make any number of scripts for every kind of post-process optimization they need.

Allan · 2016-03-21 01:36:37

Alad wrote:

Why don't we focus on improving the makepkg codebase (which has numerous bugs and dodgy mechanisms), instead of implementing every possible feature?

That is what I am doing. Please file bug reports and point out crap code so it is fixed!

I also am making makepkg extendable via libmakepkg - we already can drop in passes for tidying a package (e.g. upx, optipng), for enforcing PKGBUILD correctness, and for doing namcap like checking of packages. This reduces my maintenance burden, and allows focus on improving the core of makepkg. My bet is that it would take only a few of hours to do the same for adding build options like the LTO/PGO ones proposed here, rather than patching makepkg itself.

quequotion · 2016-03-21 06:35:12

Alad wrote:

Why don't we focus on improving the makepkg codebase?

Allan wrote:

That is what I am doing.

I trust Allan's work in this area more than my own. Obviously, he's more familiar with makepkg in general and on top of that I know next to nothing about the libmakepkg API.

Allan wrote:

I also am making makepkg extendable via libmakepkg - we already can drop in passes for tidying a package (e.g. upx, optipng), for enforcing PKGBUILD correctness, and for doing namcap like checking of packages. This reduces my maintenance burden, and allows focus on improving the core of makepkg.

Wonderful things, and certainly not mutually exclusive with a fork that will only live as long as it needs to.

My bet is that it would take only a few of hours to do the same for adding build options like the LTO/PGO ones proposed here, rather than patching makepkg itself.

I would very much like this to be done. As I said earlier, I don't feel it's out of place at all for the packager to assist users in optimizing packages this way, though I understand why you want such options are provided by supplementary packages. If my work can assist you in any way, please make use of it.

In the meantime, I came up with my own SVGO option using nodejs-svgo (and copy->paste, find->replace). Can anyone recommend a package to test this on?

EDIT: Allan was right, doing it his way didn't take that much time. If the patch is approved, makepkg-optimize will become redundant. I already have a package of libmakepkg extensions for the options discussed in this thread, pending release should this come to pass.

EDIT: The tidy scripts can each be given options--the command line parameters of their respective executables--for enhanced compression and other reasons. I have added, and commented out, some examples of the most extreme options in makepkg-optimize.conf

Last edited by quequotion (2016-04-01 06:54:38)

quequotion · 2016-03-21 18:32:27

Allan wrote:

My bet is that it would take only a few of hours to do the same for adding build options like the LTO/PGO ones proposed here, rather than patching makepkg itself.

I took a hack at some of your scripts.

EDIT: It works. I have a patch for pacman and a package of extensions ready-to-go.

At the moment, there isn't a slot for extra build option scripts to get pulled in. I put these lines after the ccache* and distcc* sections of run_build() in makepkg.sh.in

	# Check for BUILDENV extensions, use any that are requested (check buildenv and PKGBUILD opts)
	buildenv_ext

Using tidy.sh as a template, buildenv_ext.sh

[[ -n "$LIBMAKEPKG_BUILDENV_EXT_SH" ]] && return
LIBMAKEPKG_BUILDENV_EXT_SH=1

LIBRARY=${LIBRARY:-'/usr/local/share/makepkg'}


declare -a  exta_buildopts

for lib in "$LIBRARY/buildenv_ext/"*.sh; do
	source "$lib"
done

readonly -a extra_buildopts

buildenv_ext() {
	# options that alter compilation parameters
	for func in ${extra_buildopts[@]}; do
		$func
	done

}

As an example of a BUILDENV extension, using purge.sh as a template, pgo.sh:

[[ -n "$LIBMAKEPKG_PGO_SH" ]] && return
LIBMAKEPKG_PGO_SH=1

LIBRARY=${LIBRARY:-'/usr/local/share/makepkg'}

source "$LIBRARY/util/message.sh"
source "$LIBRARY/util/option.sh"

extra_buildopts+=('pgo')

[[ -n ${PROFDEST} ]] && _PROFDEST=$(canonicalize_path ${PROFDEST})
PROFDEST=${_PROFDEST:-$PROFDEST}
PROFDEST=${PROFDEST:-$startdir} #default to $startdir if undefined
if [[ ! -w $PROFDEST ]] ; then
	error "$(gettext "You do not have write permission to store profiles in %s.")" "$PROFDEST"
	plain "$(gettext "Aborting...")"
	exit 1
fi

pgo() {
	if check_buildoption "pgo" "y"; then
		if [ ! -d "$PROFDEST/$pkgbase.gen" ]; then
			mkdir "$PROFDEST/$pkgbase.gen"
			CFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen"
			CXXFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen"
			LDFLAGS+=" -lgcov"
		else
			[[ ! -d "$PROFDEST/$pkgbase.used" ]] && mv "$PROFDEST/$pkgbase.gen" "$PROFDEST/$pkgbase.used"
			CFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used"
			CXXFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used"
		fi
	fi
}

*You could further simplify the makepkg script by converting the ccache and distcc clauses into BUILDENV extension scripts.

Last edited by quequotion (2016-03-23 05:44:18)

Raziel23 · 2016-03-22 15:56:07

In the meantime, I came up with my own SVGO option using nodejs-svgo (and copy->paste, find->replace). Can anyone recommend a package to test this on?

For example adwaita-icon-theme. It's used by gtk3. Maybe there are more appropriate packages, nevertheless it contains png and svg files.

I'm very interested in those features that you are proposed here, because I have some packages compiled with LTO. I only have one question. Did you test a PGO compilation in a clean chroot environment? I mean whether profile files (.gcda) will be placed in the present working directory? I suspect that it will require modification of the makechrootpkg script (or other modifications). I compile my packages in a clean chroot in order to avoid unwanted linking.

quequotion · 2016-03-22 16:56:10

Raziel23 wrote:

I mean whether profile files (.gcda) will be placed in the present working directory?

qqbuild might have that kind of problem; makepkg-optimize does not.

PROFDEST works exactly like the other ~DEST variables: when nothing is set, they are created in the build directory. This is the reason "src/" and "pkg/" are in the build directory if you haven't set SRCDEST or PKGDEST. "$pkgname.gen" will be created in the build directory, unless you have specified otherwise.

If the build directoy is your present working directory, that is where "$pkgname.gen" should appear.

Last edited by quequotion (2016-04-01 06:55:26)

quequotion · 2016-03-23 05:41:39

Raziel23 wrote:

For example adwaita-icon-theme.

Total Installed Size without optipng or svgo: 25.95 MiB
Total Installed Size with optipng and svgo: 25.78 MiB

optipng was unable to find any that needed optimization, which seems to be the case most of the time. svgo shaved off 0.17 MiB. I find it likely the benefits of image optimization will be similarly minimal for most packages.

Raziel23 · 2016-03-23 19:42:10

quequotion wrote:

If the build directoy is your present working directory, that is where "$pkgname.gen" should appear.

I did some research and read more about PGO. During compilation in a clean chroot build directory is located within a chroot environment. The file path for the *.gcda file gets included (hard-coded) as absolute file name into a binary file. This problem can be overcome by creating the same directory structure in my main environment as is in a chroot, or playing with cross profiling. Both ways will require modification of makechrootpkg script, because it is not prepared for new variable (PROFDEST). Nevertheless thank you for inspiration, maybe I will play more with PGO in the future.

quequotion · 2016-03-23 21:00:22

Raziel23 wrote:

The file path for the *.gcda file gets included (hard-coded) as absolute file name into a binary file. This problem can be overcome by creating the same directory structure in my main environment as is in a chroot, or playing with cross profiling. Both ways will require modification of makechrootpkg script, because it is not prepared for new variable (PROFDEST). Nevertheless thank you for inspiration, maybe I will play more with PGO in the future.

Wait, what? This doesn't make sense. Could you post the exact commands you executed to test this, line by line?

Do you mean to say that a binary file called "$pkgname.gen" appears somewhere?

Last edited by quequotion (2016-03-24 15:50:00)

Raziel23 · 2016-03-24 16:39:26

quequotion wrote:

Do you mean to say that a binary file called "$pkgname.gen" appears somewhere?

You misunderstood me. I will elaborate. You can check where profile files will be written:

$ strings /usr/bin/your_program | grep '.gcda$'

Also I have to elaborate what I mean 'makechrootpkg is not prepared for PROFDEST variable'. It is not only setting this variable correctly, it also requires moving profile files in 'prepare_chroot' and 'move_products' functions. This is not hard, but there is one thing. When you build a package with PGO then during checking phase (when Makefile checks if we have right libraries installed) is created first profile file 'conftest.gcda'. After that it is never updated, even when you run a program, and is not listed in a binary file (command above). I don't know if this file is crucial for PGO. If this file wouldn't be created during checking phase, then I would have a simple case, because then I could set PROFDEST a different path (that I have in my main environment) then I have in a clean chroot.

I hope it is more clear now. I didn't have enough time lately to test more.

quequotion · 2016-03-24 17:03:10

Raziel23 wrote:

You misunderstood me. I will elaborate. You can check where profile files will be written:
$ strings /usr/bin/your_program | grep '.gcda$'
Also I have to elaborate what I mean 'makechrootpkg is not prepared for PROFDEST variable'. It is not only setting this variable correctly, it also requires moving profile files in 'prepare_chroot' and 'move_products' functions. This is not hard, but there is one thing. When you build a package with PGO then during checking phase (when Makefile checks if we have right libraries installed) is created first profile file 'conftest.gcda'. After that it is never updated, even when you run a program, and is not listed in a binary file (command above). I don't know if this file is crucial for PGO. If this file wouldn't be created during checking phase, then I would have a simple case, because then I could set PROFDEST a different path (that I have in my main environment) then I have in a clean chroot.
I hope it is more clear now. I didn't have enough time lately to test more.

After completing a number of convoluted and horribly complex hoop jumps, I was able to test building in a "clean chroot" with makepkg-optmize and found the same result: $pkgname.gen is created, and does get the one .gcda file which is never updated. I have come to a different conclusion however. This isn't a problem with PROFDEST or makepkg-optmize: it is a problem with building packages in "clean chroots" separately from the system upon which they are intended to run.

EDIT: This theory was proven. No modifications to makechrootpkg, makepkg, or makepkg-optimize were needed; only a few extra commands to ensure profiling applications can access PROFDEST on both sides of the chroot.

The reason conftest.gcda never gets updated, and no other .gcda files are generated, is that the running program cannot access the location it expects--"$PROFDEST/$pkgname.gen".

It might be possible if PROFDEST can be defined as a location that is accessible on both sides of the chroot, in the same place and with the same permissions. (ie, a filesystem mounted in the same place on both sides, in a directory with 777 permissions).

The program I tested on was epiphany, and the command you suggest above returns nothing for it...

Last edited by quequotion (2016-04-01 06:59:08)

Raziel23 · 2016-03-24 17:14:53

quequotion wrote:

It might be possible if PROFDEST can be defined as a location that is accessible on both sides of the chroot, in the same place and with the same permissions.

This is exactly what you said, or playing with cross profiling. Thank you for your efforts, maybe in the future when I will have a time I will prepare appropriate solution for this. Now I don't have enough time.

quequotion wrote:

The program I tested on was epiphany, and the command you suggest above returns nothing for it...

I don't know why it isn't working for you, hard to say, for me it works. I tested it with compiled sed program.

Last edited by Raziel23 (2016-03-24 17:25:15)

quequotion · 2016-03-31 20:05:41

Raziel23 wrote:

playing with cross profiling.

PROFDEST is cross-profiling. You're going to need a location that is exactly the same on both sides: the same actual location, the same ownership, the same permissions. TL;DR: It can be done by creating a directory in the same place on both sides and binding them together, I recommend /mnt/pgo/.

These instructions are based on the clean-chroot building wiki, with amendments for getting the most out of makepkg-optimize (lto, pgo, graphite; upx, optipng, svgo).

Note: I recommend installing an AUR helper in your clean chroot, which is certainly not standard policy. The reason for this is it will save you at least a half dozen additional steps per AUR package built within the chroot. Pikaur is used herein.

Prep work:

Install the "devtools" package:
```
# pacman -S devtools
```
Configure /etc/pacman.conf
Configure /etc/makepkg.conf

Chroot setup (First time only):

Create a folder for "clean" chroot building and assign it to CHROOT:

$ mkdir ~/path/to/buildchroot && CHROOT=~/path/to/buildchroot
$ printf "export CHROOT=~/path/to/buildchroot" >> ~/.bashrc

Transfer your pacman.conf and makepkg.conf; install some basics (graphite, openmp, upx, and optipng are optional).
```
# mkarchroot -C /etc/pacman.conf -M /etc/makepkg.conf "$CHROOT"/root base-devel graphite openmp upx optipng
```

Build pikaur locally*, then copy the package into the chroot (to root's home directory)

# pacman -S --needed base-devel git
$ git clone https://aur.archlinux.org/pikaur.git
$ cd pikaur
$ makepkg -fsri
# cp pikaur-1.2.23-1-any.pkg.tar.xz $CHROOT/root/root/

Install pikaur in the chroot:

# systemd-nspawn -D "$CHROOT"/root pacman -U /root/pikaur-1.2.23-1-any.pkg.tar.xz

AUR helper needs an unprivileged user:

# systemd-nspawn -D "$CHROOT"/root useradd "$USER"
# systemd-nspawn -D "$CHROOT"/root passwd "$USER"

Set up sudo for that user**:

# echo "$USER ALL=(root) NOPASSWD: /usr/bin/pacman" >> "$CHROOT"/root/etc/sudoers

Install AUR packages (nodejs-svgo is optional):

$ arch-nspawn -M /etc/makepkg.conf "$CHROOT"/root -u $USER pikaur -S nodejs-svgo pacman-buildenv_ext-git makepkg-optimize

Create a location that will exist on both sides to store profiles:
```
# mkdir -m 777 {"$CHROOT"/root,}/mnt/pgo
```
Edit $CHROOT/root/etc/makepkg-optimize.conf:
Set PROFDEST=/mnt/pgo under "PACKAGE OUTPUT".

Using the chroot:

Keep your chroot up-to-date:

$ arch-nspawn -M /etc/makepkg.conf "$CHROOT"/root -u $USER pikaur -Syua

Edit $CHROOT/root/etc/makepkg-optimize.conf:
Select your preferred optimizations under "BUILD ENVIRONMENT" (pgo, lto, graphite) and "GLOBAL PACKAGE OPTIONS" (upx, optipng, svgo).

Build the package (in the directory containing its PKGBUILD):

$ makechrootpkg -c -r "$CHROOT" -- --config /etc/makepkg-optimize.conf

If you are building a package with PGO:

Bind the pgo folder across the chroot:***

# mount -o bind {,"$CHROOT"/"$USER"}/mnt/pgo

Install the package and test-run its executables; try to use every function!
Remove cruft and rebuild the package, but do not clean the chroot (in the directory containing its PKGBUILD):
```
$ rm -rf *.pkg.tar.xz pkg src; makechrootpkg -r "$CHROOT" -- --config /etc/makepkg-optimize.conf
```

*Because the pikaur available in [archlinuxfr] does not seem to work, and building this package inside the chroot would require many extra steps. Be sure the chroot's packages are up-to-date with your system's packages, or pikaur may fail for not finding the library versions it was built against.

**When using an AUR helper inside the chroot, sudo will complain about the lack of tty or askpass program if pacman requires a password.

***At the initial PGO build, generated profiles will be stored in a directory called "$pkgbase.gen". If they become corrupted, delete this directory to restart generation. Many programs store their profiles in a hidden directory, such as ".libs" within. If there are no files here, the package may not be compatible with PGO or might be ignoring CFLAGS (often the case with cmake packages). When a package is rebuilt with PGO, applied profiles are moved to "$pkgbase.used" and can be deleted or collected for analysis.

Last edited by quequotion (2018-09-02 17:12:03)

Raziel23 · 2016-04-01 06:53:57

@quequotion Yesterday I also solved this. I considered 3 approaches:

1. Create exactly the same path in chroot like in main environment and copy there profile files.

2. Create exactly the same path in chroot like in main environment and bind a profile directory in rw mode.

3. Set and export appropriate GCOV_PREFIX and GCOV_PREFIX_STRIP variables in chroot.

On the beginning I was working on the third approach, but GCOV_PREFIX and GCOV_PREFIX_STRIP didn't work for me (I append and export them in "/path/to/chroot/etc/profile"). I don't know why they don't work for me.

Yesterday I finished the first approach, here are the following changes that I made to makechrootpkg:
https://gist.github.com/Raziel-23/9445d … 0266540981

It works flawlessly for me and it works with any path, I mean it can be your present working directory or other PROFDEST path. However it requires removing those line from makepkg-optimize that you add recently:

if [[ "${0##*/}" = "mkchrootpkg" ]]; then
  PROFDEST=$PWD #default to $PWD for mkchrootpkg; which should only build one package at a time.
else
  PROFDEST=${PROFDEST:-$startdir} #default to $startdir if undefined
fi

I'm not going to make the second approach, because it's more complicated then the first approach and more vulnerable when building a package failed (there is a chance that we can lose our profile directory).

On the side note I didn't test much pgo compilation, but my first impressions were rather disappointing. I don't know like other packages but during profile correction building 'sed' failed without adding '-Wno-error=coverage-mismatch' flag to CFLAGS/CXXFLAGS. This coverage-mismatch problem is not related to chroot (I checked). Also it fails when I compile 'sed' with lto and pgo, with only lto 'sed' compile correctly. I know it is not your fault, but it feels like pgo usage is limited and mixing pgo with lto is not a good idea.

On the side side note why you don't put 'nodejs-svgo', 'optipng' and 'upx' to optdepends? They should be optional, I don't use them.

Edit: I updated gist because I passed the patch twice.

Last edited by Raziel23 (2016-04-01 07:12:20)

quequotion · 2016-04-01 07:34:30

Raziel23 wrote:

@quequotion Yesterday I also solved this.

Thanks for keeping at it!

It works flawlessly for me and it works with any path, I mean it can be your present working directory or other PROFDEST path. However it requires removing those line from makepkg-optimize that you add recently:

I also intend for these lines to be removed; they didn't really serve their intended purpose anyway.

Yesterday I finished the first approach, here are the following changes that I made to makechrootpkg:
https://gist.github.com/Raziel-23/9445d … 0266540981

Not that I doubt your work, but--for entirely political reasons--I think it best to avoid any additional proposals to change existing packages. Proposing one change is hard enough, proposing two will almost certainly result in the permanent dismissal of both out-of-hand.

I don't know like other packages but during profile correction building 'sed' failed without adding '-Wno-error=coverage-mismatch' flag to CFLAGS/CXXFLAGS.

Most packages compile without problems, but not every package. I'm not sure how safe it is to ignore mismatch errors, so probably best to let users specify this flag on their own when needed.
What kind of CPU do you have? -fprofile-correction is intended to fix errors known to happen when profiling on multi-core processors and theoretically neither helps nor harms compliation on a single-core CPU.

mixing pgo with lto is not a good idea.

My experience has been that LTO occasionally breaks compiles, especially when stacked against other options like pgo, graphite, or "-Ofast" in C(XX)FLAGS while PGO only rarely does so, but also more often when stacked against additional optimizations. The gnu compilier devs would probably appreciate a bug report when these things happen, as they expect LTO and PGO to work--though not specified as working together.

On the side side note why you don't put 'nodejs-svgo', 'optipng' and 'upx' to optdepends? They should be optional, I don't use them.

Indeed, done.

I'm not going to make the second approach, because it's more complicated then the first approach and more vulnerable when building a package failed (there is a chance that we can lose our profile directory).

Actually no, this is the best approach. The profile directory will not be lost, even if a package fails to build. Try the procedure I posted above, which takes this approach. The benefit here is that it requires no changes to makepkg, makechrootpkg, or makepkg-optimize (only pacman is modified, hence pacman-buildenv_ext-git).

Last edited by quequotion (2017-03-26 02:22:52)

Raziel23 · 2016-04-01 07:50:03

quequotion wrote:

Not that I doubt your work, but--for entirely political reasons--I think it best to avoid any additional proposals to change existing packages. Proposing one change is hard enough, proposing two will almost certainly result in the permanent dismissal of both out-of-hand.

I understand your motivation. I never counted that my patch will be accepted by devs. I only create it for my own purposes, maybe someone will benefit from it, maybe not. Devs have valid policies and they have all rights to not accept such patches.

Nevertheless thank you for your efforts, we both somehow solved this problem

Actually no, this is the best approach. The profile directory will not be lost, even if a package fails to build. Try the procedure I posted above, which takes this approach. The benefit here is that it requires no changes to makepkg, makechrootpkg, or makepkg-optimize.

You think about binding from command line, but for me it is convenient to bind from makechrootpkg script

EDIT: You have to understand my motivation. I only want some sort of automation. Maybe creating a wrapper script that setup a bind and then call makechrootpkg script is not a bad idea.

EDIT 2: I missed that:

What kind of CPU do you have? -fprofile-correction is intended to fix errors known to happen when profiling on multi-core processors and theoretically neither helps nor harms compliation on a single-core CPU.

I have 2 cores processor (Intel Core 2 Duo) with 2 threads. I found a cause of this. Second compilation of 'sed' with '-fprofile-use' works without '-Wno-error=coverage-mismatch' flag only if I remove 'conftest.gcda' profile file. Here is described what it's a conftest.c file, and here is described very related problem that may happen with it. So it's harmless to remove it before second compilation.

Last edited by Raziel23 (2016-04-01 14:50:07)

quequotion · 2016-04-01 16:50:20

Raziel23 wrote:

You have to understand my motivation. I only want some sort of automation. Maybe creating a wrapper script that setup a bind and then call makechrootpkg script is not a bad idea.

In fact, I completely agree with you that full automation is better and I'd rather not have so many command-line steps involved. I find it very unfortunate that politics have to be considered in open source development. Perhaps, if the proposed extensions to makepkg ever get approval, we could push for additional changes to makechrootpkg later on. In the worst case, I suppose we could simply fork pacman and devtools--but I doubt my own ability to keep up the maintenance on such large projects.

Here is described what it's a conftest.c file, and here is described very related problem that may happen with it. So it's harmless to remove it before second compilation.

Edit: I've added this to the PGO macro in makepkg-optimize.

Last edited by quequotion (2016-04-01 19:07:17)

quequotion · 2016-05-19 03:00:18

There may yet be hope.

Seems my implementation is too crude, but the idea might get in there eventually.

Arch Linux

#1 2016-03-20 11:00:22

Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#2 2016-03-20 15:38:46

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#3 2016-03-20 16:30:05

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#4 2016-03-20 16:51:11

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#5 2016-03-20 18:20:37

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#6 2016-03-20 19:43:40

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#7 2016-03-20 20:18:20

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#8 2016-03-20 20:30:06

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#9 2016-03-21 01:36:37

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#10 2016-03-21 06:35:12

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#11 2016-03-21 18:32:27

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#12 2016-03-22 15:56:07

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#13 2016-03-22 16:56:10

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#14 2016-03-23 05:41:39

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#15 2016-03-23 19:42:10

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#16 2016-03-23 21:00:22

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#17 2016-03-24 16:39:26

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#18 2016-03-24 17:03:10

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#19 2016-03-24 17:14:53

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#20 2016-03-31 20:05:41

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#21 2016-04-01 06:53:57

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#22 2016-04-01 07:34:30

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#23 2016-04-01 07:50:03

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#24 2016-04-01 16:50:20

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

#25 2016-05-19 03:00:18

Re: Makepkg simplification, makepkg-optimize; future libmakepkg extensions

Board footer