You are not logged in.

#1 2016-08-10 13:06:39

kalsan
Member
Registered: 2011-10-10
Posts: 119

[solved] PKGBUILD: Compress packages multithreaded

Assume a fast, modern multi-core machine and a nice Internet connection. Using this setup, we install a package from AUR using the PKGBUILD system (yaourt, pacaur, makepkg or whatever).
-> Thanks to the fast Internet connection the download is speedy
-> Thanks to j8 (or j16, ...) option in the global PKGBUILD config, compilation is parallelized and thus optimally fast
-> However one bottleneck remains: The final step is compressing the package before it is given to pacman for installation. This step may take a very long time (for example if the package is 1GB large) and is inefficient because it's just using one core.

In other applications, we can use pbzip(2) to compress in parallel and the performance is awesome (for dd from mechanical HDDs even real-time). So my question is: Is it possible to use pbzip or pbzip2 in order to compress a package for pacman, and, if yes, how is it done?

If no, I might issue a feature request since this is really the bottleneck on modern machines.

Cheers,
Kalsan

Last edited by kalsan (2016-08-10 19:51:59)

Offline

#2 2016-08-10 13:15:52

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,466

Re: [solved] PKGBUILD: Compress packages multithreaded

The default is xz compression, so pbzip isn't going to do it.

Offline

#3 2016-08-10 13:43:38

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [solved] PKGBUILD: Compress packages multithreaded

If you're just building the packages for personal use rather than distribution then you can always just skip compression completely, you just end up with a larger package; you're likely to be decompressing them straight away when installing anyway.

Last edited by Slithery (2016-08-10 13:45:00)


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#4 2016-08-10 13:49:31

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: [solved] PKGBUILD: Compress packages multithreaded

xz can use multiple cores... `xz --threads 0` but as I recall, the resulting xz files are larger.  Not sure how to integrate into makepkg either perhaps with XZ_OPT?

EDIT: does makepkg even use /usr/bin/xz or a libarchive call?  Need to see the code.

Last edited by graysky (2016-08-10 13:50:23)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#5 2016-08-10 14:01:53

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [solved] PKGBUILD: Compress packages multithreaded

See the last two items in `man makepkg.conf`.

You can set the compression command to whatever you want, but really the smart thing would be to do as slithery said and disable compression altogether. Compression is primarily useful for reducing network download speed, which quite obviously assumes you are hosting a repository for public consumption. wink


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#6 2016-08-10 19:51:39

kalsan
Member
Registered: 2011-10-10
Posts: 119

Re: [solved] PKGBUILD: Compress packages multithreaded

Thanks a lot for these answers! In case anyone else lands here, use this link for reading on how to disable compression: https://bbs.archlinux.org/viewtopic.php?id=127894

Offline

#7 2016-08-10 22:20:36

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,466

Re: [solved] PKGBUILD: Compress packages multithreaded

graysky wrote:

xz can use multiple cores... `xz --threads 0` but as I recall, the resulting xz files are larger.  Not sure how to integrate into makepkg either perhaps with XZ_OPT?

EDIT: does makepkg even use /usr/bin/xz or a libarchive call?  Need to see the code.

Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).

Offline

#8 2016-08-10 23:37:20

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [solved] PKGBUILD: Compress packages multithreaded

Scimmia wrote:

Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).

What does it matter if the archive is deterministic? The date in the .PKGINFO is enough to make a package non-deterministic, surely...

I was under the impression that reproducible builds were all about the binaries. Which would presumably tell you more about the actual differences anyway, rather than looking at an additional layer of indirection via xz-compressed binaries.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#9 2016-08-11 04:48:05

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,466

Re: [solved] PKGBUILD: Compress packages multithreaded

Eschwartz wrote:
Scimmia wrote:

Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).

What does it matter if the archive is deterministic? The date in the .PKGINFO is enough to make a package non-deterministic, surely...

I was under the impression that reproducible builds were all about the binaries. Which would presumably tell you more about the actual differences anyway, rather than looking at an additional layer of indirection via xz-compressed binaries.

The point is to reproduce the package exactly.

Offline

Board footer

Powered by FluxBB