[solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

amish · 2017-05-28 13:19:11

Everytime I create a package via makepkg, I notice that it also packages a file named .BUILDINFO inside the package file.

And this file has list of ALL the packages installed in my system (obtained via pacman -Q)

This adds 50KB to all the packages that I create. (and also leaks the information about what is installed in my system)

Not sure why this information is required to be distributed inside the package file but I do not want it to be packaged.

So how do I remove this file without modifying /usr/bin/makepkg? (or make it not have list of all packages in my system)

Shouldn't it be optional?

I googled and searched this forum (for "remove BUILDINFO", "without BUILDINFO") but could not find anything related.

I noticed that official arch packages DO NOT have .BUILDINFO file. So does official packages use different makepkg?

Sorry in advance if I overlooked something somewhere.

Last edited by amish (2017-05-28 15:20:11)

Scimmia · 2017-05-28 13:46:19

Build in a clean chroot and there's no issue.

eschwartz · 2017-05-28 14:11:55

Official Arch Linux packages do indeed have .BUILDINFO files, assuming they were build with a moderately recent version of pacman/makepkg. Though I can't make any promises about what other distros which model themselves after Arch and share a common package manager do.

e.g. Manjaro, Antergos.

As Scimmia said, if you build in a clean chroot (using makechrootpkg from the devtools package) you won't have this issue. The only packages that will be listed in the .BUILDINFO are the base-devel packages and the packages that your package depends on.

amish · 2017-05-28 15:18:29

Yes you are right. New official packages do have .BUILDINFO.

Earlier I checked 2-3 (small sized) packages which were actually old and hence did not have .BUILDINFO.

For clean chroot, I tried following:
https://wiki.archlinux.org/index.php/De … nience_Way

But it appears that it will install lots of things first. And I need to setup lots of things too.

Looked like too much time as well as space consuming.

Its like additional arch system just for the sake of building package. (and not have all packages listed in .BUILDINFO)

I build many packages.

What package X depends on, package Y may not depend on. And yet both packages will have all packages (whether it depends on OR not) in .BUILDINFO. Unless I have separate chroot for each package. (which is kind of crazy)

Finally I gave up and chose easy way out

i.e. commented this line in /usr/bin/makepkg

# printf "installed = %s\n" "${pkglist[@]}"

PS: I still fail to understand why should .BUILDINFO have information about all the packages installed in system? Isnt "depends" in .PKGINFO enough? Atleast it should be optional for packager to decide whether to include list or not.

PS2: It also leaks the directory path where the package was built. So I also changed line which prints builddir in .BUILDINFO

Last edited by amish (2017-05-28 15:21:22)

Lone_Wolf · 2017-05-28 15:53:41

Amish, this is beginning to sound like an x-y problem.

If you build packages for your own system, you already know where you are building and what is installed.

Are you building binary packages intended to be used by other people ?

amish · 2017-05-28 16:47:11

Yes I am building for others.

But even if it was just for me only, I am still wondering why that information is needed to be stored in package?

Because everyone knows that base-devel group is required and rest of the dependency is already stored in depends directives

So why this redundant information? Am curious to know.

Kind of goes against arch philosophy of keeping it minimal and simple

Scimmia · 2017-05-28 19:42:33

It's necessary for reproducible builds.

Allan · 2017-05-29 00:07:07

If you are building for other people, you should definitely be using a clean chroot. It is the only way to ensure you don't add unspecified dependencies.

amish · 2017-05-29 01:23:03

Thats what I am curious to know. How and why is it necessary?

Isnt "makedepends" and "depends" inside .PKGINFO sufficient to re-build?

Isnt "makepkg -s" sufficient?

Even manually:
To reproduce a build ... we install base-devel. (i.e. all packages in base-devel)

And also install package dependencies (which is stored in .PKGINFO or PKGBUILD)

So why is it required to have that information again in .BUILDINFO?

@Allan
I assume the dependencies you are talking about is same as what is put in PKGBUILD.

So instead of:

run_pacman -Q

why cant we have

run_pacman -Q $(run_pacman -Qgq base-devel) ${depends[@]} ${makedepends[@]} ...

This way it will not disclose everything in .BUILDINFO and work in chroot and non-chroot environment.

eschwartz · 2017-05-29 02:06:05

Because in the real world, this is a thing: https://wiki.gentoo.org/wiki/Project:Qu … pendencies

It is important to know the full set of packages that the system knew about at the time the package was compiled. This is also why clean chroot builds are recommended. So too, the builddir is important because software can record the directory it was compiled in.

Last edited by eschwartz (2017-05-29 02:06:54)

fsckd · 2017-05-29 02:22:36

amish wrote:

And also install package dependencies (which is stored in .PKGINFO or PKGBUILD)
So why is it required to have that information again in .BUILDINFO?

Dependencies listed in .PKGINFO are insuffient for reproducable builds.

amish · 2017-05-29 02:55:21

Finally that link explains certain things.

But then as mentioned in that link itself -its more a case of "lazy devs" who do not do (--disable-XXX) in Configure or equivalent scripts.

I am assuming that this forum is right place for discussion (or atleast for my own understanding)

I am wondering few things.

1) What is the purpose of .BUILDINFO?

For developers who may want to rebuild the package - right?

Then shouldn't it be placed in .src.pkg.tar.xz? Why is it required to be placed in binary pkg also?

2) Is the purpose of .BUILDINFO to specify packages that were in build system when it was built?

If yes, then why would "end user" of binary package be interested in it? So why include it in binary package?

OR

3) Is the purpose of .BUILDINFO to specify list of packages "end-user" may require to re-build the package?

If yes, then why does it need to have whole list of packages?
Isnt .PKGINFO (which has list of depends and makedepends and optdepends) file enough?

And even if we have to have .BUILDINFO - then isnt my suggestion above of listing specific packages instead of whole "pacman -Q" more appropriate?

This way it works with or without chroot.

Last edited by amish (2017-05-29 02:56:38)

eschwartz · 2017-05-29 03:12:03

1) src.tar.z is not a built package, src.tar.gz is e.g. what you get in the AUR. Built packages contain the environment in which they were built.

2) Yes. End users may not be interested in it, but most end users are interested in very few things at all. The people working on reproducible builds care though, as do people debugging why packages fail to work. e.g. a recent example is that I built Firefox/Thunderbird with gcc7 without realizing it and didn't know why it wasn't working, but the .BUILDINFO contains the versions of installed packages and could tell me that I had gcc7 installed.

3) No, that's what the .PKGINFO is for, containing the "depends" info.

4) If you think you are going to convince Allan to stop using .BUILDINFO, you have another think coming. We already know all the arguments, after more extensive discussion and brainstorming than you have used in this merely-day-old thread.
The decision to add .BUILDINFO was not made without due consideration.

Also, no one should distribute built packages without using clean chroots. Not because they have something to hide, merely because it is the best way to ensure the packages actually work as expected.

Allan · 2017-05-29 04:41:59

Eschwartz wrote:

4) If you think you are going to convince Allan to stop using .BUILDINFO, you have another think coming.

My opinions are correct.

amish · 2017-05-29 05:47:08

I never said stop using BUILDINFO. (just asked if it is really necessary and why and if I want to remove it how do i do it.)

If you read my first post then I am simply asking if there is way to NOT list packages. (or make it optional).

My own 1 line hack in to "/usr/bin/makepkg" does NOT remove BUILDINFO. It keeps the file but without build time package information.

Not every developer has resources to have separate chroot for each package they are building.

And I am not insisting on anything. I had already found my solution and had marked discussion as [solved]

Rest of the posts were for my own knowledge / understanding only.

fsckd · 2017-05-29 06:03:47

It is perfectly fine and reasonable to ask questions to improve your understanding on how and why. Please continue to do so.

Reproducible builds is something people interested in the security of Arch Linux have been working on for the past year. It is a very useful thing to have, especially from a security perspective, but that may not be obvious to everyone.

amish · 2017-05-29 07:00:59

Ok I read more about BUILDINFO.

(Question I was asking was why BUILDINFO is required but answer I was getting was why chroot is required)

Here is why BUILDINFO is required. (For people who land here in future and have no clue about why it is required)

Main purpose of BUILDINFO is NOT what I thought was. i.e. it is not really to rebuild package or detect dependencies. (because for rebuilding a package PKGBUILD is enough)

But to verify that person who built the package has not altered any source code.

BUILDINFO includes all the packages along with version used to build any package.

If someone wants to verify that the packager has no bad intention.

All they need is to install the packages specified in BUILDINFO (same versions)

And build it. (reproduce it)

If reproduced build and packaged build matches .. that means there is no source code change made by packager.

Otherwise it should raise eyebrows!

Last edited by amish (2017-05-29 07:12:36)

Allan · 2017-05-29 07:56:19

amish wrote:

Not every developer has resources to have separate chroot for each package they are building.

You only need one. It is 620M for the base chroot.

dreieck · 2018-06-13 10:57:39

amish wrote:

Everytime I create a package via makepkg, I notice that it also packages a file named .BUILDINFO inside the package file.
And this file has list of ALL the packages installed in my system (obtained via pacman -Q)

I was also annoyed by this, but was more annoyed by the time and CPU ressources each pacman -Q run consumes. (I have a single core machine.)

I am building just for myself. And do not wantr to use a chroot because of added maintenance requirements.

I solved the problem by patching makepkg. I created a customizepkg-hook, so that when pacman is updated (pacman provides makepkg), it will be automatically rebuilt, applying my patch.

The patch to the pacman source tree is:

diff -r -U1 pacman-5.1.0.orig/scripts/makepkg.sh.in pacman-5.1.0.new/scripts/makepkg.sh.in
--- pacman-5.1.0.orig/scripts/makepkg.sh.in     2018-06-13 12:14:30.313734761 +0200
+++ pacman-5.1.0.new/scripts/makepkg.sh.in      2018-06-13 12:18:43.626567915 +0200
@@ -736,3 +736,3 @@
        write_pkginfo > .PKGINFO
-       write_buildinfo > .BUILDINFO
+       # write_buildinfo > .BUILDINFO # Disabled in user-built variant of 'makepkg'.

And it get's applied by adding a line (patch -Np1 -i <path-to-above-patch>) to the prepare()-function in the PKGBUILD via customizepkg-scripting.

progandy · 2018-06-13 11:25:01

There is a simple solution that works without patching makepkg:

$ cat ~/.config/pacman/makepkg.conf 
function write_buildinfo() {
	msg2 "Don't write .BUILDINFO"
}

eschwartz · 2018-06-13 15:14:23

Whyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

It calls pacman *once*. Well, twice if you count a mild inefficiency which I've already posted a patch to fix.

If you don't explicitly use -dd, it calls pacman -T anyways.
If you use VCS sources, it will unconditionally call pacman -T.
If you use -r, it calls -qq several times to figure out which packages to uninstall.
If you had to install any dependencies, it loads pacman to do a full-blown transaction.

The first time you execute pacman, it reads the entire database, which can be slightly slow, then afterwards it is in the HDD disk cache. The number of cores is not really a factor here, even on my very old single-core laptop (which I actually no longer use). You're almost assuredly running pacman at least once, close enough to the time buildinfo is generated that the database will still be in the disk cache.

dreieck · 2018-06-14 10:43:00

Eschwartz wrote:

Whyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
It calls pacman *once*.

On my system I see a run of pacman -Qi <all packages installed on my system> (a very long list of arguments) when makepkg is generating the .BUILDINFO. And this run consumes a lot of ressources for about 2 minutes on my system. Every time. Even if I build two packages short after each other.

Last edited by dreieck (2018-06-14 10:44:13)

eschwartz · 2018-06-14 22:15:44

Two minutes in order to run pacman -Qi > /dev/null seems quite horrible, but this is I think the first time I heard of it being that slow for anyone, no matter how old/slow their HDD.

dreieck · 2018-06-15 07:59:48

progandy wrote:

There is a simple solution that works without patching makepkg:
$ cat ~/.config/pacman/makepkg.conf 
function write_buildinfo() {
	msg2 "Don't write .BUILDINFO"
}

Thanks a lot, did not know that those functions can be overridden in the makepkg.conf.

This is elegant solution.

Arch Linux

#1 2017-05-28 13:19:11

[solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#2 2017-05-28 13:46:19

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#3 2017-05-28 14:11:55

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#4 2017-05-28 15:18:29

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#5 2017-05-28 15:53:41

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#6 2017-05-28 16:47:11

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#7 2017-05-28 19:42:33

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#8 2017-05-29 00:07:07

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#9 2017-05-29 01:23:03

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#10 2017-05-29 02:06:05

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#11 2017-05-29 02:22:36

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#12 2017-05-29 02:55:21

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#13 2017-05-29 03:12:03

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#14 2017-05-29 04:41:59

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#15 2017-05-29 05:47:08

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#16 2017-05-29 06:03:47

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#17 2017-05-29 07:00:59

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#18 2017-05-29 07:56:19

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#19 2018-06-13 10:57:39

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#20 2018-06-13 11:25:01

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#21 2018-06-13 15:14:23

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#22 2018-06-14 10:43:00

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#23 2018-06-14 22:15:44

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

#24 2018-06-15 07:59:48

Re: [solved][with a hack] Remove .BUILDINFO from pkg.tar.xz file?

Board footer