You are not logged in.
Assume a fast, modern multi-core machine and a nice Internet connection. Using this setup, we install a package from AUR using the PKGBUILD system (yaourt, pacaur, makepkg or whatever).
-> Thanks to the fast Internet connection the download is speedy
-> Thanks to j8 (or j16, ...) option in the global PKGBUILD config, compilation is parallelized and thus optimally fast
-> However one bottleneck remains: The final step is compressing the package before it is given to pacman for installation. This step may take a very long time (for example if the package is 1GB large) and is inefficient because it's just using one core.
In other applications, we can use pbzip(2) to compress in parallel and the performance is awesome (for dd from mechanical HDDs even real-time). So my question is: Is it possible to use pbzip or pbzip2 in order to compress a package for pacman, and, if yes, how is it done?
If no, I might issue a feature request since this is really the bottleneck on modern machines.
Cheers,
Kalsan
Last edited by kalsan (2016-08-10 19:51:59)
Offline
The default is xz compression, so pbzip isn't going to do it.
Offline
If you're just building the packages for personal use rather than distribution then you can always just skip compression completely, you just end up with a larger package; you're likely to be decompressing them straight away when installing anyway.
Last edited by Slithery (2016-08-10 13:45:00)
Offline
xz can use multiple cores... `xz --threads 0` but as I recall, the resulting xz files are larger. Not sure how to integrate into makepkg either perhaps with XZ_OPT?
EDIT: does makepkg even use /usr/bin/xz or a libarchive call? Need to see the code.
Last edited by graysky (2016-08-10 13:50:23)
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
See the last two items in `man makepkg.conf`.
You can set the compression command to whatever you want, but really the smart thing would be to do as slithery said and disable compression altogether. Compression is primarily useful for reducing network download speed, which quite obviously assumes you are hosting a repository for public consumption.
Managing AUR repos The Right Way -- aurpublish (now a standalone tool)
Offline
Thanks a lot for these answers! In case anyone else lands here, use this link for reading on how to disable compression: https://bbs.archlinux.org/viewtopic.php?id=127894
Offline
xz can use multiple cores... `xz --threads 0` but as I recall, the resulting xz files are larger. Not sure how to integrate into makepkg either perhaps with XZ_OPT?
EDIT: does makepkg even use /usr/bin/xz or a libarchive call? Need to see the code.
Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).
Offline
Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).
What does it matter if the archive is deterministic? The date in the .PKGINFO is enough to make a package non-deterministic, surely...
I was under the impression that reproducible builds were all about the binaries. Which would presumably tell you more about the actual differences anyway, rather than looking at an additional layer of indirection via xz-compressed binaries.
Managing AUR repos The Right Way -- aurpublish (now a standalone tool)
Offline
Scimmia wrote:Yes, but it's not deterministic. Not going to happen if it breaks reproducible packages (not there yet, but people are working on it).
What does it matter if the archive is deterministic? The date in the .PKGINFO is enough to make a package non-deterministic, surely...
I was under the impression that reproducible builds were all about the binaries. Which would presumably tell you more about the actual differences anyway, rather than looking at an additional layer of indirection via xz-compressed binaries.
The point is to reproduce the package exactly.
Offline