lrzip: faster, better compression or both

superstoned · 2006-10-30 11:18:38

hi guys,

some of you might know Dr. Con Kolivas, author of the -ck kernel patch (with his staircase cpu scheduler for better interactive behaviour).

well, he's got another project: file compression. he added lzma or lzo compression to a rzip file analyse algorithm, and created lrzip.

so, what this actually does is: uses rzip to analyse the file, and bring similair blocks of data together, and then use lzma to compress these parts. or lzo, to speed things up. rzip is even smart enough to take the available memory in account and use it all, but not more, to prevent hitting swap.
the difference between rzip and lrzip is that lrzip uses lzma or lzo instead of bzip2. rzip+lzma is slower than bzip2, but but has a much higher compression; rzip+lzo is faster than gzip, and still has a slightly better compression.
because all this code is actually well-tested and stable, the resulting lrzip should be stable as well.

in my own testing, i got great results. generally, it seems if lrzip is 5% slower, you get 5% better compression than bzip2. it's not a rule, but i've seen it several times. and compression rations of 30% or more OVER bzip2 isn't rare at all.
ow, and lrzip with lzo compression is up to 4 times faster as gzip (!) and still has a slightly better compression!!!

check con's README for more info, benchmarks and FAQ.

http://ck.kolivas.org/apps/lrzip/README

of course i post this here, because i have a rather slow pipe to the World Wide Web and i'd rather see the pacman software compressed with lrzip, as it'll speed things up... esp on big stuff.

btw I got 20% better compression on a big (700 mb) tar file with mostly documents, pictures etc., and 50% better compression on a mix of sourcecode + compiled code. (compared to bzip2). con reports 50% better compression on a fully compiled kernel tree. yes, 50% smaller than the resulting bzip2 file!!!

noriko · 2006-11-12 20:12:10

i've tested this on a bunch of random files.. videos, images... and i am pleased to say, it is both faster and returns higher compression ratios than anything i've ever seen... (zip,bz,gz,rar,..) ... well that's my opinion anyways...

brain0 · 2006-11-12 22:23:22

$ time gzip -9 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar

real    0m23.361s
user    0m21.203s
sys     0m0.590s

$ time bzip2 -9 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar

real    0m35.045s
user    0m32.147s
sys     0m0.675s

$ time lrzip -M -l kernel26mm-2.6.19.rc5.mm1-1.pkg.tar
kernel26mm-2.6.19.rc5.mm1-1.pkg.tar - compression ratio 2.285

real    0m4.567s
user    0m3.649s
sys     0m0.480s

$ time lrzip -M kernel26mm-2.6.19.rc5.mm1-1.pkg.tar
kernel26mm-2.6.19.rc5.mm1-1.pkg.tar - compression ratio 3.966

real    2m6.472s
user    1m57.755s
sys     0m2.561s

-rw-r--r-- 1 thomas users 58M Nov 12 23:18 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar
-rw-r--r-- 1 thomas users 18M Nov 12 23:15 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar.bz2
-rw-r--r-- 1 thomas users 20M Nov 12 23:14 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar.gz
-rw-r--r-- 1 thomas users 15M Nov 12 23:21 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar.lrz
-rw-r--r-- 1 thomas users 26M Nov 12 23:18 kernel26mm-2.6.19.rc5.mm1-1.pkg.tar.lrz-l

lrzip is VERY slow though without -l and has much worse compression than gzip with -l, but is very fast in this mode.

superstoned · 2006-11-12 23:23:36

brain0 wrote:

lrzip is VERY slow though without -l and has much worse compression than gzip with -l, but is very fast in this mode.

well, yes, there is a clear tradeoff. it can be faster than gzip (incredibly fast, actually) or give better compression compared to bzip2 (at longer compression times and memory usage). if space is most important, lrzip is usefull, if you want faster, it's better as well...
you can use it to prepare files for bzip2 as well, speeding up bzip2 compression and increasing compression ratio. this is wat rzip does as well. Con is still working on it, btw, and it'll most likely improve ;-)

Arch Linux

#1 2006-10-30 11:18:38

lrzip: faster, better compression or both

#2 2006-11-12 20:12:10

Re: lrzip: faster, better compression or both

#3 2006-11-12 22:23:22

Re: lrzip: faster, better compression or both

#4 2006-11-12 23:23:36

Re: lrzip: faster, better compression or both

Board footer