Set system wide default parallel compression

kozaki · 2016-08-06 17:50:45

I did some tests on single and multi-core systems and noted that parallel xz, pigz and lbzip2, interestingly give much compression speed improvement even on a single core with HT and that increases with the available number of cores. With e.g. pigz close to lz4 at decompression, lbzip2 winning over gzip for compression (!), and parallel being 1.4 to 2 × faster on Atom.

As parallel xz, pigz and lbzip2
1) are backward compatible with standard xz, gzip and bzip2, and
2) I dunno of an Arch substitute for Gentoo or Calculate GNU/Linux

echo "app-arch/pigz symlink" >> /etc/portage/package.use/custom

I went I went for building `tar` with the base PKGBUILD from Allan McRae and Andreas Radke, and started [2] with

./configure \
  --with-gzip=pigz \      # use PROG as gzip compressor program                       
  --with-bzip2=lbzip2   # use PROG as bzip2 compressor program

as a limited but usefull and probably less hacky substitute to the "full aliases" hack quoted in this forum [1].

But my `tar-parallelized` build fails on the test #29

ERROR: 183 tests were run,
1 failed unexpectedly.
21 tests were skipped.

Both for i686 and x86_64, with testsuite.log

How do you use threaded compression/decompression on your systems?

[1]: Replace bzip2 and gzip with pbzip2 and pigz system wide? 2011
[2]: For xz I use the `$XZ_DEFAULT=` envirionnement variable to use all the locally available threads. And may shift to `pixz` (that one comes close to gzip / lbzip2 for decompression time). Which would add a third substitute to the tar build options.
As for `lzo` I dunno if `lza`might be used in place yet.

adamlau · 2016-11-17 07:08:50

Drop in pigz to replace gzip. No issues other than tar [core] will fail its PKGBUILD check. lbzip2/pbzip2 as a direct replacement is trickier since so many repo apps which depend on bzip2 are built against the bzip2 library. I have bzip2 built out of all of my packages with lbzip2 --> bzip2. libarchive/bsdtar works fine this way. The caveat is that you may run into a package with a hard dependency on the bzip2 library.

EDIT 11/26: I discovered a few bzip2 packages which requires libarchive be built against the bzip2 libs.

Last edited by adamlau (2016-11-27 02:36:21)

regid · 2018-08-09 15:28:23

Quoting https://www.gnu.org/software/tar/manual/tar.html#SEC136, which is from December, 17 2017:

Recent versions of lbzip2 are mostly command line compatible with bzip2, which makes it possible to automatically invoke it via the `--bzip2' GNU tar command line option.

Perhaps not using tar with

--with-bzip2=lbzip2

, as well as having libarchive depends on bzip2, should be reconsidered?

Arch Linux

#1 2016-08-06 17:50:45

Set system wide default parallel compression

#2 2016-11-17 07:08:50

Re: Set system wide default parallel compression

#3 2018-08-09 15:28:23

Re: Set system wide default parallel compression

Board footer