You are not logged in.

#1 2010-10-10 22:22:18

uncholowapo
Member
From: US
Registered: 2009-03-29
Posts: 238

Replace bzip2 and gzip with pbzip2 and pigz system wide?

I just now got word of these multithreaded ports and would like to know if theres anyway to replace their single threaded brethren? pigz is already in AUR and it wouldn't take much to put pbzip2 into it either.

Maybe some global variable should be set?

Edit: Strike the part about pbzip2. It's already in community tongue

Last edited by uncholowapo (2010-10-10 23:48:04)

Offline

#2 2010-10-10 23:10:53

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

An alias?

Offline

#3 2010-10-10 23:50:27

uncholowapo
Member
From: US
Registered: 2009-03-29
Posts: 238

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Where would I put said alias? Would /etc/bashrc work?

Last edited by uncholowapo (2010-10-10 23:56:40)

Offline

#4 2010-10-11 00:16:58

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,476
Website

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Or just have the PKGBUILD provide/conflict/replace gzip and add a /usr/bin/gzip etc symlink...

Offline

#5 2010-10-11 10:50:06

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

uncholowapo wrote:

Where would I put said alias? Would /etc/bashrc work?

Yes, it should work. I have them in /etc/bash.bashrc.local but I don't remember what's the difference ;P

One thing you may want to check: if you compress a file w/ pbzip2 and send it over to me, will I be able to uncompress it with bzip2?

Offline

#6 2010-10-11 14:48:09

nicedream
Member
From: Pennsylvania, USA
Registered: 2009-07-24
Posts: 14
Website

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

karol wrote:

One thing you may want to check: if you compress a file w/ pbzip2 and send it over to me, will I be able to uncompress it with bzip2?

I just did some testing by downloading the Linux kernel source, and then using bzip2 and pbzip2 to decompress and recompress it several times.  In another terminal window I monitored the speed of each CPU core by running:

watch 'cat /proc/cpuinfo | grep MHz'

What I found out was that yes, the files created by either tool will be able to be decompressed with the other.  However, if you use pbzip2 to decompress a file compressed with the regular bzip2, it will only use one core.  Only when decompressing a file that was compressed with pbzip2 did pbzip2 use both CPU cores.

Offline

#7 2010-10-12 01:33:29

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Offline

#8 2011-03-20 17:56:04

santodelaespada
Member
Registered: 2009-04-15
Posts: 50

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

I wonder, if pbzip2 compressed files are decompressable by regular bzip2, and there's the same with pxz and xz, are arch packages parallel compressed so we can take advantage with either tool?.
It should not affect those with single core processors or single threaded (de)compressors. But it certainly improves the package installing experience to use multi-threads.

Offline

#9 2011-03-21 01:13:49

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Might also be worth checking out lzip which is supposed to be better than all of those. Its even said to be better than xz.


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#10 2011-10-11 21:19:22

travnick
Member
From: Poland
Registered: 2011-04-24
Posts: 75

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

adding an alias works great for me. But there is a problem:
It won't work for scripts ... so makepkg (which is the bash script) ans another system scripts are using gzip, xz, bzip2 instead of commands from aliases

Is there solution for this problem?

Offline

#11 2011-10-11 21:59:34

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

Those scripts likely use the full path e.g. /bin/gzip instead of just 'gzip'. Maybe you can remove the regular binaries and symlink /bin/gzip to its parallel version - not sure how safe it is.
See also https://bugs.archlinux.org/task/23968

Offline

#12 2011-10-11 22:20:46

travnick
Member
From: Poland
Registered: 2011-04-24
Posts: 75

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

I hope no one call in his script /bin/gzip instead of just gzip.
I solve my problem:

[]$ cat /etc/bash.bashrc.local 
alias grep='grep --color=auto'

alias ll='ls -l --color=auto'
alias llh='ls -lh --color=auto'
alias l.='ls -d .* --color=auto'
alias ls='ls --color=auto'


#Do compression with all cores !!
#alias gzip='pigz'
#alias gunzip='unpigz'
#alias bzip2='pbzip2'
#alias bunzip2='pbunzip2'
#alias bzcat='pbzcat'
#alias xz='pxz'

function gzip(){
 pigz $@
}
export -f gzip

function gunzip(){
 unpigz $@
}
export -f gunzip

function bzip2(){
 pbzip2 $@
}
export -f bzip2

function bunzip2(){
 pbunzip2 $@
}
export -f bunzip2

function bzcat(){
 pbzcat $@
}
export -f bzcat

function xz(){
 pxz $@
}
export -f xz

Offline

#13 2011-10-11 22:23:38

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

That works as longs as the options used are "backwards-compatible".

Offline

#14 2011-10-11 22:51:44

travnick
Member
From: Poland
Registered: 2011-04-24
Posts: 75

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

karol wrote:

That works as longs as the options used are "backwards-compatible".

what do you mean?

Offline

#15 2011-10-11 22:55:02

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

travnick wrote:
karol wrote:

That works as longs as the options used are "backwards-compatible".

what do you mean?

You're using pigz as a drop-in replacement of gzip. If pigz behaves same as gzip i.e. 'pigz --foo' does the same as 'gzip --foo', everything's fine, but when the apps interpret options in a different way or the new app has some mandatory options you could skip with the previous one - things will break.

Offline

#16 2011-10-14 13:33:45

travnick
Member
From: Poland
Registered: 2011-04-24
Posts: 75

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

as far as I can see it works excellent for me

Offline

#17 2012-02-19 03:40:07

jimduchek
Member
Registered: 2008-01-06
Posts: 10

Re: Replace bzip2 and gzip with pbzip2 and pigz system wide?

I would very much like to see the packages for pbzip2 and pigz marked with provide/conflict/replace bzip2/gzip, and provide symlinks to where bzip2/gzip live.  Is there any reason not to do this?  I would think that most (if not all) people installing pbzip2/pigz would want to always use the parallel versions, so replacing the 'original' bzip2 and gzip would not be an issue.  It would require some effort to do (for example, all the shell scripts, /usr/bin/bzdiff, etc), and a bit of testing (it appears that /bin/bzcat, and /bin/zcat, which are symbolic links, still work properly with the parallel versions).

Any reasons not to?

Offline

Board footer

Powered by FluxBB