Replace bzip2 and gzip with pbzip2 and pigz system wide?

uncholowapo · 2010-10-10 22:22:18

I just now got word of these multithreaded ports and would like to know if theres anyway to replace their single threaded brethren? pigz is already in AUR and it wouldn't take much to put pbzip2 into it either.

Maybe some global variable should be set?

Edit: Strike the part about pbzip2. It's already in community

Last edited by uncholowapo (2010-10-10 23:48:04)

karol · 2010-10-10 23:10:53

An alias?

uncholowapo · 2010-10-10 23:50:27

Where would I put said alias? Would /etc/bashrc work?

Last edited by uncholowapo (2010-10-10 23:56:40)

Allan · 2010-10-11 00:16:58

Or just have the PKGBUILD provide/conflict/replace gzip and add a /usr/bin/gzip etc symlink...

karol · 2010-10-11 10:50:06

uncholowapo wrote:

Where would I put said alias? Would /etc/bashrc work?

Yes, it should work. I have them in /etc/bash.bashrc.local but I don't remember what's the difference ;P

One thing you may want to check: if you compress a file w/ pbzip2 and send it over to me, will I be able to uncompress it with bzip2?

nicedream · 2010-10-11 14:48:09

karol wrote:

One thing you may want to check: if you compress a file w/ pbzip2 and send it over to me, will I be able to uncompress it with bzip2?

I just did some testing by downloading the Linux kernel source, and then using bzip2 and pbzip2 to decompress and recompress it several times. In another terminal window I monitored the speed of each CPU core by running:

watch 'cat /proc/cpuinfo | grep MHz'

What I found out was that yes, the files created by either tool will be able to be decompressed with the other. However, if you use pbzip2 to decompress a file compressed with the regular bzip2, it will only use one core. Only when decompressing a file that was compressed with pbzip2 did pbzip2 use both CPU cores.

karol · 2010-10-12 01:33:29

You may also want to have a look at:
http://aur.archlinux.org/packages.php?ID=24718
http://aur.archlinux.org/packages.php?ID=36362
http://aur.archlinux.org/packages.php?ID=23809

santodelaespada · 2011-03-20 17:56:04

I wonder, if pbzip2 compressed files are decompressable by regular bzip2, and there's the same with pxz and xz, are arch packages parallel compressed so we can take advantage with either tool?.
It should not affect those with single core processors or single threaded (de)compressors. But it certainly improves the package installing experience to use multi-threads.

dolby · 2011-03-21 01:13:49

Might also be worth checking out lzip which is supposed to be better than all of those. Its even said to be better than xz.

travnick · 2011-10-11 21:19:22

adding an alias works great for me. But there is a problem:
It won't work for scripts ... so makepkg (which is the bash script) ans another system scripts are using gzip, xz, bzip2 instead of commands from aliases

Is there solution for this problem?

karol · 2011-10-11 21:59:34

Those scripts likely use the full path e.g. /bin/gzip instead of just 'gzip'. Maybe you can remove the regular binaries and symlink /bin/gzip to its parallel version - not sure how safe it is.
See also https://bugs.archlinux.org/task/23968

travnick · 2011-10-11 22:20:46

I hope no one call in his script /bin/gzip instead of just gzip.
I solve my problem:

[]$ cat /etc/bash.bashrc.local 
alias grep='grep --color=auto'

alias ll='ls -l --color=auto'
alias llh='ls -lh --color=auto'
alias l.='ls -d .* --color=auto'
alias ls='ls --color=auto'


#Do compression with all cores !!
#alias gzip='pigz'
#alias gunzip='unpigz'
#alias bzip2='pbzip2'
#alias bunzip2='pbunzip2'
#alias bzcat='pbzcat'
#alias xz='pxz'

function gzip(){
 pigz $@
}
export -f gzip

function gunzip(){
 unpigz $@
}
export -f gunzip

function bzip2(){
 pbzip2 $@
}
export -f bzip2

function bunzip2(){
 pbunzip2 $@
}
export -f bunzip2

function bzcat(){
 pbzcat $@
}
export -f bzcat

function xz(){
 pxz $@
}
export -f xz

karol · 2011-10-11 22:23:38

That works as longs as the options used are "backwards-compatible".

travnick · 2011-10-11 22:51:44

karol wrote:

That works as longs as the options used are "backwards-compatible".

what do you mean?

karol · 2011-10-11 22:55:02

travnick wrote:

karol wrote:
That works as longs as the options used are "backwards-compatible".
what do you mean?

You're using pigz as a drop-in replacement of gzip. If pigz behaves same as gzip i.e. 'pigz --foo' does the same as 'gzip --foo', everything's fine, but when the apps interpret options in a different way or the new app has some mandatory options you could skip with the previous one - things will break.

travnick · 2011-10-14 13:33:45

as far as I can see it works excellent for me

jimduchek · 2012-02-19 03:40:07

I would very much like to see the packages for pbzip2 and pigz marked with provide/conflict/replace bzip2/gzip, and provide symlinks to where bzip2/gzip live. Is there any reason not to do this? I would think that most (if not all) people installing pbzip2/pigz would want to always use the parallel versions, so replacing the 'original' bzip2 and gzip would not be an issue. It would require some effort to do (for example, all the shell scripts, /usr/bin/bzdiff, etc), and a bit of testing (it appears that /bin/bzcat, and /bin/zcat, which are symbolic links, still work properly with the parallel versions).

Any reasons not to?

Arch Linux

#1 2010-10-10 22:22:18

Replace bzip2 and gzip with pbzip2 and pigz system wide?

#2 2010-10-10 23:10:53

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#3 2010-10-10 23:50:27

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#4 2010-10-11 00:16:58

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#5 2010-10-11 10:50:06

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#6 2010-10-11 14:48:09

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#7 2010-10-12 01:33:29

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#8 2011-03-20 17:56:04

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#9 2011-03-21 01:13:49

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#10 2011-10-11 21:19:22

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#11 2011-10-11 21:59:34

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#12 2011-10-11 22:20:46

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#13 2011-10-11 22:23:38

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#14 2011-10-11 22:51:44

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#15 2011-10-11 22:55:02

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#16 2011-10-14 13:33:45

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

#17 2012-02-19 03:40:07

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Board footer