You are not logged in.

#26 2012-10-03 16:33:38

tobias_
Member
Registered: 2012-09-01
Posts: 25

Re: Proposal: switch Arch packaging standard from xz to lrz

Benefits are probably the save in time when you have a multicore processor. A quick test with a large package on my 6 core system with 6GB memory:

1,2G 22. Sep 23:09 bastion-20120621-6-x86_64.pkg.tar

% time lrzip bastion-20120621-6-x86_64.pkg.tar
Output filename is: bastion-20120621-6-x86_64.pkg.tar.lrz
bastion-20120621-6-x86_64.pkg.tar - Compression Ratio: 1.272. Average Compression Speed: 12.698MB/s. Total time: 00:01:36.60
lrzip bastion-20120621-6-x86_64.pkg.tar  398,46s user 7,45s system 420% cpu 1:36,61 total
959M  3. Okt 18:08 bastion-20120621-6-x86_64.pkg.tar.lrz

% time xz bastion-20120621-6-x86_64.pkg.tar
xz bastion-20120621-6-x86_64.pkg.tar  423,66s user 1,62s system 99% cpu 7:06,30 total
1004M 22. Sep 23:09 bastion-20120621-6-x86_64.pkg.tar.xz

To conclude: lrzip uses less CPU time (uses 3GB memory peaked), compresses a bit better than xz, and is multithreading capable.

Decompression test:

% time xz -d bastion-20120621-6-x86_64.pkg.tar.xz
xz -d bastion-20120621-6-x86_64.pkg.tar.xz  16,90s user 0,66s system 43% cpu 40,003 total


% time lrunzip bastion-20120621-6-x86_64.pkg.tar.lrz
Output filename is: bastion-20120621-6-x86_64.pkg.tar
Decompressing...
100%    1219.17 /   1219.17 MB
Average DeCompression Speed: 24.878MB/s
Output filename is: bastion-20120621-6-x86_64.pkg.tar: [OK] - 1278392320 bytes
Total time: 00:00:48.89
lrunzip bastion-20120621-6-x86_64.pkg.tar.lrz  68,27s user 0,96s system 141% cpu 48,907 total

Decompression uses considerably more cpu time in the lrzip case and is slower than xz, because it seems to begin writing just after it has decompressed everything in memory , whereas xz immediately writes.

Offline

#27 2012-10-03 16:40:21

karol
Archivist
Registered: 2009-05-06
Posts: 25,428

Re: Proposal: switch Arch packaging standard from xz to lrz

tobias_ wrote:

Benefits are probably the save in time when you have a multicore processor.

Save time compressing the package? If I'm building them locally, I don't have to compress at all.
I have a fast enough Internet so I don't care if the downloads are a bit bigger. Slower decompression doesn't sound good either.

Offline

#28 2012-10-03 17:39:13

Awebb
Member
Registered: 2010-05-06
Posts: 4,321

Re: Proposal: switch Arch packaging standard from xz to lrz

So, any further discussion will probably be answered with this quote:

Allan wrote:

Adding lrz compression to makepkg is easy - a one line patch I think.    I have not done it yet as there is little point until a released libarchive supports reading these packages.

Also, I doubt I will have any time to look into the relative benefits/disadvantages of using this compression format until an actual libarchive release with lrzip support is made.

That means we should bring this issue upstream to the libarchive devs.

Offline

#29 2012-10-03 19:10:41

graysky
Member
From: /run/user/1000
Registered: 2008-12-01
Posts: 8,524
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

Awebb wrote:

That means we should bring this issue upstream to the libarchive devs.

Already done; it is merged and we await v3.0.5.
Reference: https://bbs.archlinux.org/viewtopic.php … 3#p1169963

Last edited by graysky (2012-10-03 19:11:05)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#30 2012-10-03 20:54:24

Allan
Developer
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,408
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

You are awaiting libarchive 3.1...  That will not be added in a minor release.

Also, testing a plain decompression is not enough to justify use in pacman.  There are other operations performed on the package file than just extraction (e.g. file lists are obtained for conflict checking, the install script is extracted separately if pre_intsall needs to be run...)

Offline

#31 2012-10-03 21:17:57

graysky
Member
From: /run/user/1000
Registered: 2008-12-01
Posts: 8,524
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

Thanks for the info, Allan.  I think that your are the most qualified person to assess the cost/benefits of switching formats.  Guess we're in a holding pattern until the next major release of libarchive is released.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#32 2012-10-03 22:42:25

Grinch
Member
Registered: 2010-11-07
Posts: 265

Re: Proposal: switch Arch packaging standard from xz to lrz

I tried lrzip a while back, I first tried various large data files and compared it's compression against 7z which generally yielded small gains for lrzip, however I then tried comparing them on a large tar archive filled with tons of smaller somewhat similar files and here there was a HUGE difference, the long range algorithm was much more effective in finding repeated data amongst those files which resulted in a very substantial gain in compression compared to the best settings I could achieve with 7z.

I don't know how much of an impact that has on the packages Arch is distributing, but since it always seem to perform better than xz (speed, compression) I think it's certainly a good candidate.

Offline

#33 2012-10-03 23:44:39

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 6,832

Re: Proposal: switch Arch packaging standard from xz to lrz

Grinch wrote:

since it always seem to perform better than xz (speed, compression)

I'd think decompression is significant. If that's slower, that hits ALL users.

But we'd need to wait for it to actually get tested of course.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#34 2012-10-04 01:55:44

Grinch
Member
Registered: 2010-11-07
Posts: 265

Re: Proposal: switch Arch packaging standard from xz to lrz

Ah yes, you are right. Better compression and compression speed has much less practical benefit than fast decompression in a scenario like this, atleast unless the compression advantage is so substantal that it leads to heavily decreased bandwidth usage.

Even so, with today's connections I doubt alot of users will see a great benefit in a slightly faster download with longer decompression times as a result .

Offline

#35 2012-10-04 08:13:23

graysky
Member
From: /run/user/1000
Registered: 2008-12-01
Posts: 8,524
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

I would point folks to the PDF I referenced in my first post where several aspects of these issues are attempted to be addressed.  What I didn't think about at the time, are some of the extra functions pacman does on packages like Allan mentioned.  You can see, on a multicore machine, decompression is faster.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#36 2012-10-04 11:47:16

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: Proposal: switch Arch packaging standard from xz to lrz

For this comparison to be more fair you need to compare with the alpha version of xz which has smp support.


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#37 2012-10-04 18:59:56

graysky
Member
From: /run/user/1000
Registered: 2008-12-01
Posts: 8,524
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

@dolby - I'll leave future testing to Allan.  Also, I don't think comparing stable code (lrzip) to alpha code (xz smp) is a fair comparison at all.  Once libarchive releases the next stable major release, the lrzip code therein will be considered stable and available.  I highly doubt that Arch will run xz alpha in [core] for example!


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#38 2012-10-04 19:41:26

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: Proposal: switch Arch packaging standard from xz to lrz

graysky wrote:

@dolby - I'll leave future testing to Allan.  Also, I don't think comparing stable code (lrzip) to alpha code (xz smp) is a fair comparison at all.  Once libarchive releases the next stable major release, the lrzip code therein will be considered stable and available.  I highly doubt that Arch will run xz alpha in [core] for example!

Fedora have switched to the 5.1 xz version for a long while so it should be ok. I dont know if they add any other patches from scm to the alpha release though. Its more fair than comparing a program which has such an important feature implemented and one that does not, imho. tongue


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#39 2012-10-04 20:17:52

Allan
Developer
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,408
Website

Re: Proposal: switch Arch packaging standard from xz to lrz

I do not think there are any additional xz patches in git after the alpha release apart from documentation fixes.

Offline

#40 2012-10-04 22:21:09

brain0
Developer
From: Aachen - Germany
Registered: 2005-01-03
Posts: 1,382

Re: Proposal: switch Arch packaging standard from xz to lrz

Fun how such an old thread gets revived.

The major disadvantage of lrz is that it can only use its big advantages over xz when it can do multiple passes over the data, but not when used with stdin/stdout. This may be a showstopper for using it with packages.

Offline

Board footer

Powered by FluxBB