lrzip, good compression contender?

Fackamato · 2011-04-09 11:16:40

Source: http://ck-hack.blogspot.com/2011/04/qui … rison.html

Old thread: lrzip: faster, better compression or both

Quoting Con's blog post:

Con Kolivas wrote:

Thursday, 7 April 2011
Quick lrzip comparison
So I was building a kernel package for myself and when I was done saw a large directory and thought, what the heck, I'll compress it with lrzip for grins.
bzip2 size: 1831647009 time: 12m50s (command "tar cjf")
7z size: 945054166 time: 36m23s (command "7za a")
lrzip size: 586087630 time: 17m09s (command "lrztar")

These were done on a quad core 3GHz with 8GB ram using nothing but default options. Note that lrzip was done using the lrztar wrapper to make the comparison fair since it's just a single command without temporary files, so it took 3 passes to compress. If I compressed it with temporary files it would be smaller again (i.e. tar cf directory && lrzip)
Of course there is parallel bzip2 which speeds up bzip2's compression but has no effect on compression. And then there is xz which slows down 7z's compression and has no effect on compression. So why is xz becoming the new defacto standard? Probably because it's not aimed squarely at the windows market the way 7z is and feels more politically correct to the linux crowd. Maybe it's because it's called lzma2 that it must be better than lzma since it has a higher version number. I've never seen any performance or compression advantage of any significance with lzma2 versus lzma. Personally I'm relatively unimpressed with xz so I don't really understand it. Even less understandable is why the kernel has both lzma AND xz support in it now.
I wish I could get people more excited about lrzip

What do you guys think? I'll do a few benchmarks on my own.

karol · 2011-04-09 11:46:48

In case somebody wants to try it out http://aur.archlinux.org/packages.php?O … _Search=Go + pretty old lrzip 0.45-1 in [archlinuxfr].

For people who don't have a lot of RAM / don't compress huge files / many similar files, lrzip may not offer any benefits over e.g. 7z.
Maybe one of reasons lrzip isn't as popular as 7z is that it wasn't actively developed for some time but things started to get better around version 0.3.

Last edited by karol (2011-04-09 11:55:02)

Fackamato · 2011-04-09 11:57:43

karol wrote:

For people who don't have a lot of RAM / don't compress huge files / many similar files, lrzip may not offer any benefits over e.g. 7z.

Is that still true? If you have 4GB of RAM or more (which many people have) isn't it a better option then. It's apparently faster, and compresses better.

karol · 2011-04-09 12:18:35

With 4 GB you can indeed see a difference.

As Yann said in the comments to that blog post, lzma2 is mostly about multithreading.

Another thing you might want to try is binary deltas, like xdelta3 or bsdiff. I use xdelta3 for my daily backups as the data changes very little.

graysky · 2011-04-28 07:01:40

See the graphs CK put up on his blog.

For compressing the kernel26 source tree (v2.6.38) on a quad core Intel, lrzip did it approx 3 times faster than tar/gzip and was more or less the same filesize.

Fackamato · 2011-04-28 10:49:03

graysky wrote:

See the graphs CK put up on his blog.
For compressing the kernel26 source tree (v2.6.38) on a quad core Intel, lrzip did it approx 3 times faster than tar/gzip and was more or less the same filesize.

Impressive

Does it have the same RAM usage during (de)compression though?

karol · 2011-04-28 11:53:04

Fackamato wrote:

Impressive
Does it have the same RAM usage during (de)compression though?

gzip uses around 1 MB, lrzip more like 1 GB - during compression.

broken pipe · 2011-04-28 12:12:14

wow! but what's to do, when the directory has a space in the name!?

lrztar VirtualBox\ VMs/
Failed to open VMs.tar.lrz
No such file or directory
Fatal error - exiting

karol · 2011-04-28 12:14:40

broken pipe wrote:

wow! but what's to do, when the directory has a space in the name!?
lrztar VirtualBox\ VMs/
Failed to open VMs.tar.lrz
No such file or directory
Fatal error - exiting

Have you tried quoting?

Edit: http://ck-hack.blogspot.com/2011/04/lrzip-0603.html claims that "lrztar now properly supports -o, -O, and -S." The version in AUR is out of date, but there's a patch (see the comments) http://aur.archlinux.org/packages.php?ID=7418

I can confirm that version 0.602-1 is not working properly. 'lrztar -o foo2.lrz foo\ bar/' gives you foo bar.tar.lrz that has executable permissions set on: '-rwxr-xr-x'.
I'll try the new version and report back.

Edit 2: lrzip 0.604-1 works almost as expected: 'lrztar -o foo2.lrz foo\ bar/' produces a file named 'foo2.lrz', but it still has the executable permissions set:

-rw-r--r-- 1 1,2M 04-28 15:16 foo1.tar.lrz
-rwxr-xr-x 1 1,2M 04-28 15:15 foo2.lrz

'foo1.tar.lrz' was created by first using tar and then lrzip and 'foo2.lrz' was created using lrztar.

Is is a bug, is it a plane or is it me missing something?

@ broken pipe
Update lrzip and use '-o' switch.

Last edited by karol (2011-04-28 13:15:59)

Arch Linux

#1 2011-04-09 11:16:40

lrzip, good compression contender?

#2 2011-04-09 11:46:48

Re: lrzip, good compression contender?

#3 2011-04-09 11:57:43

Re: lrzip, good compression contender?

#4 2011-04-09 12:18:35

Re: lrzip, good compression contender?

#5 2011-04-28 07:01:40

Re: lrzip, good compression contender?

#6 2011-04-28 10:49:03

Re: lrzip, good compression contender?

#7 2011-04-28 11:53:04

Re: lrzip, good compression contender?

#8 2011-04-28 12:12:14

Re: lrzip, good compression contender?

#9 2011-04-28 12:14:40

Re: lrzip, good compression contender?

Board footer