Arch Linux

tantalum

Member

Registered: 2008-05-21

Posts: 4

More efficient package compression?

Hi,

I find the LZMA algorithm (also used by 7zip) interesting.
( http://en.wikipedia.org/wiki/Lempel-Ziv … _algorithm )
- licensed under the LGPL
- an easy to use LGPL implementation exists (lzma utils)

It's compression ratio is exceptionally good, the compression takes
a relatively long time, but the decompression is only twice as slow as gzip.

For a software repository I'd think "pack once, serve often".

And I made tests. I took the biggest package I have in the cache, unpacked
94MB.

34M     mono-1.9.1-1-x86_64.pkg.tar.gz
32M     mono-1.9.1-1-x86_64.pkg.tar.bz2
18M     mono-1.9.1-1-x86_64.pkg.tar.lzma

So there's significant bandwidth to spare. And I at least, with my max 176Kb/s internet connection,
would be happy to have a "faster" pacman at hand.
But more test results (each test ran only once):

=time=
lzma:
compression \ decompression
real    4m1.071s \ 0m6.885s
user    3m59.784s \ 0m6.086s
sys     0m0.493s \ 0m0.730s

gzip:
co \ dec
real    0m25.152s \ 0m3.353s
user    0m24.452s \ 0m2.483s
sys     0m0.523s \ 0m0.780s

bzip2:
co \ dec
real    0m47.253s \ 0m16.939s
user    0m46.590s \ 0m16.012s
sys     0m0.483s  \ 0m0.867s

I also tested a smaller package:
8,9M    xorg-server-1.4.0.90-13-x86_64.pkg.tar.gz
4,2M    xorg-server-1.4.0.90-13-x86_64.pkg.tar.lzma

and openoffice-base-2.4.0-1-i686:
tar  |  gzip  |  bz2  |  lzma (took 9 minutes compressing, 36 seconds uncompressing)
272   112      106     77    MB

(all with a 1 GHz CPU)

What do you think about that? Is it worth further consideration?

iBertus

Member

From: Greenville, NC

Registered: 2004-11-04

Posts: 2,228

Re: More efficient package compression?

I think so. You could How well does this compression scale to faster/more CPUs? You've tested on a pretty standard/average machine and if decompression time is only increased by 2x over gzip it might be worth a test on a more powerful machine. Of course, I can't speak for the devs and how they feel about having to wait 4x longer to compress packages.

bender02

Member

From: Germany

Registered: 2007-02-04

Posts: 1,328

Re: More efficient package compression?

Seems to come up every now and then: http://archlinux.org/pipermail/arch-gen … 17751.html
Also:

lzma seems to be superior to bz2 in almost every aspect :
http://www.archlinux.org/pipermail/pacm … 11496.html
However, it apparently can't be supported by libarchive for licensing problems :
http://www.archlinux.org/pipermail/pacm … 11498.html

from http://bbs.archlinux.org/viewtopic.php?id=46405

shining

Pacman Developer

Registered: 2006-05-10

Posts: 2,043

Re: More efficient package compression?

http://archlinux.org/pipermail/arch-gen … 17751.html

---

pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

tantalum

Member

Registered: 2008-05-21

Posts: 4

Re: More efficient package compression?

bender02
shining

Thank you for pointing that out.