You are not logged in.

#1 2017-12-27 13:35:12

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Run prepare() without extracting sources

I'm bisecting the kernel using the linux-git PKGBUILD from AUR. Every time git bisect checks out a new revision I'd need makepkg to run the prepare() routine in PKGBUILD (which updates the kernel config and package version) without extracting the sources, but I can't figure out which makepkg options should be used. As far as I understand, -e does not unpack the source nor run prepare(), while -o does the opposite. The combination -eo seems to not run prepare(). --noprepare is definitely not what I'm searching for.
I didn't find any other prepare()-related flags in man makepkg, so I'd need some help with this. Thanks.

Last edited by snack (2017-12-27 13:47:32)

Offline

#2 2017-12-27 14:07:58

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,783
Website

Re: Run prepare() without extracting sources

man pkgbuild wrote:
       prepare() Function
           An optional prepare() function can be specified in which operations to prepare the
           sources for building, such as patching, are performed. This function is run after the
           source extraction and before the build() function. The prepare() function is skipped
           when source extraction is skipped.

By design, the prepare function is run after source extraction. What you are asking for is not possible. You will either need to prepare the sources manually, or bisect in another location, and add the commit id to the git source declartion in the PKGBUILD.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#3 2017-12-27 14:09:42

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: Run prepare() without extracting sources

. ./PKGBUILD
prepare

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#4 2017-12-27 14:50:01

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@Trilby Nice! But I think that sourcing the PKGBUILD won't define some needed environment variables like srcdir which I think are defined by makepkg. For sure I can define them and then use your trick, I was hoping for a less dirty procedure but it seems there's no way. Thanks to both!

Offline

#5 2017-12-27 14:58:36

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Run prepare() without extracting sources

snack wrote:

@Trilby Nice! But I think that sourcing the PKGBUILD won't define some needed environment variables like srcdir which I think are defined by makepkg. For sure I can define them and then use your trick, I was hoping for a less dirty procedure but it seems there's no way. Thanks to both!

You could edit the pkgbuild and call prepare in the build function or move the relevant commands from prepare to build.

Edit: As far as I know, you should keep the configuration static during bisection in order to prevent full rebuilds. (simply skip prepare and only run makepkg -e)
https://bbs.archlinux.org/viewtopic.php … 3#p1722083

Last edited by progandy (2017-12-27 15:03:40)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#6 2017-12-27 15:00:33

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@progandy This seems a good solution, also to avoid to forget calling prepare() manually in subsequent builds. Thanks!

Offline

#7 2017-12-27 15:03:08

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: Run prepare() without extracting sources

Good point, but srcdir should be the only one.  So slight modification:

. ./PKGBUILD
srcdir=$(readlink -f src)
prepare 

This could readily be made into a shell alias:

alias runprep='. ./PKGBUILD && srcdir=$(readlink -f src) && prepare'

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#8 2017-12-27 15:04:59

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Run prepare() without extracting sources

Even if it works, you probably should not change the configuration during bisection in order to prevent full rebuilds. (simply skip prepare and only run makepkg -e)
https://bbs.archlinux.org/viewtopic.php … 3#p1722083


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#9 2017-12-27 15:06:01

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@Trilby Thanks for the follow-up, I found that also CARCH is not defined, but in my case this and srcdir are the only ones missing.

Offline

#10 2017-12-27 15:09:09

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@progandy I installed ccache to speed-up the recompilation, but it doesn't seem very effective (only 1.65% cache hit rate). Also, running makepkg -e without re-running prepare() has not been very effective in shortening compilation times on previous bisect steps (I have been asked to reconfigure the build every time, and the build took ages with very low cache hit rate).

Offline

#11 2017-12-27 15:09:28

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: Run prepare() without extracting sources

Oh, right, makepkg.conf might be need too.  Ok, that *really* should be it.

alias runprep='. /etc/makepkg.conf && . ./PKGBUIILD && srcdir=$(readlink -f src) && prepare'

Admittedly it's getting a bit ugly, but you only need to create the alias once.

My primary point was that it is far from impossible to do this (if this is what you really want to do), as PKGBUILDs are just shell commands.  So provided you set the environment variables like makepkg does, you'll get the same result from running the prepare function yourself.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#12 2017-12-27 15:11:26

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@Trilby Thanks a lot again for your very detailed answer. I knew that PKGBUILDs are shell scripts but I didn't think about calling the functions directly. Your solution seems quite handy for my needs.

Offline

#13 2017-12-27 20:16:03

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: Run prepare() without extracting sources

snack wrote:

I'd need makepkg to run the prepare() routine in PKGBUILD (which updates the kernel config and package version) without extracting the sources

Which package version is not being updated the version function should still be called by makepkg -e the version as returned by uname -r will not but you can use CONFIG_LOCALVERSION_AUTO=yes to have that auto updated.
You will not notice any speedup between bisection steps until you cease to be prompted to adjust the config at a step.

Offline

#14 2017-12-28 08:08:33

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@loqs understood, albeit at this point I don't understand the purpose of ccache. I thought it would avoid recompiling object files which were eventually deleted in a make clean but whose source files were actually unmodified. Even if the config is regenerated I guess that most files are untouched in nearby commit, and so ccache should do its trick. But nevermind, maybe I need to understand better how ccache works.

I'll try to optimize my procedure as you described, thank you again.

Offline

#15 2017-12-28 11:40:46

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: Run prepare() without extracting sources

@snack https://aur.archlinux.org/cgit/aur.git/ … ux-git#n45 the config file would be different each if you rerun prepare so ccache would not reuse a cached result if the inputs are different.
So to check if the config file is referenced by files during build.  If you add V=1 to the make step you should see as part of the build for most files "-include ./include/linux/kconfig.h"
which means include the contents of that file and that file contains #include <generated/autoconf.h> so the contents of that file is pulled in too.
That file contains #define CONFIG_LOCALVERSION now if you have changed CONFIG_LOCALVERSION by rerunning prepare then ccache detects the change and will not reuse the previous runs result.
Now even if you did not rerun prepare every time a build step triggers a config change then ccache will not be useful for that step.

Offline

#16 2017-12-28 11:46:56

snack
Member
From: Italy
Registered: 2009-01-13
Posts: 861

Re: Run prepare() without extracting sources

@loqs Thanks for the detailed explanation. I suspected that something in the kernel build process detects if there are different options in the freshly checked out commit with respect to the commit used for the last build, and that in that case the config is regenerated, thus imparing the usage of ccache. If I'm correct then ccache is almost useless for speeding up the bisect of the kernel.

Offline

#17 2017-12-28 18:06:03

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: Run prepare() without extracting sources

cd $builddir/src/linux
git bisect $result
cd ../..
makepkg -e

Assuming each iteration you are doing something similar to the above and nothing else.  If the config is regenerated then ccache will not help because the inputs have changed.
If the config is not regenerated then make will detect the existing object files etc and check the timestamps and should only call gcc/ccache for files that git step has altered
again as the inputs have changed for the case make is calling ccache there is no benefit.

Offline

#18 2021-05-03 21:16:00

DAC324
Member
Registered: 2020-03-12
Posts: 80

Re: Run prepare() without extracting sources

Trilby wrote:

Oh, right, makepkg.conf might be need too.  Ok, that *really* should be it.

alias runprep='. /etc/makepkg.conf && . ./PKGBUIILD && srcdir=$(readlink -f src) && prepare'

Admittedly it's getting a bit ugly, but you only need to create the alias once.

Only thing is: How do I include special commands like msg2 that sometimes occur in PKGBUILD files? What do I have to include in order to get such definitions loaded?

Offline

#19 2021-05-04 02:06:33

HalosGhost
Forum Moderator
From: Twin Cities, MN
Registered: 2012-06-22
Posts: 2,089
Website

Re: Run prepare() without extracting sources

DAC324,

You've been a member here for a little while; please do keep in mind the policy on necrobumping. Feel free to take another read through the Code of Conduct again (link in my signature).

Closing.

All the best,

-HG

Offline

Board footer

Powered by FluxBB