You are not logged in.
Benefits are probably the save in time when you have a multicore processor. A quick test with a large package on my 6 core system with 6GB memory:
1,2G 22. Sep 23:09 bastion-20120621-6-x86_64.pkg.tar
% time lrzip bastion-20120621-6-x86_64.pkg.tar
Output filename is: bastion-20120621-6-x86_64.pkg.tar.lrz
bastion-20120621-6-x86_64.pkg.tar - Compression Ratio: 1.272. Average Compression Speed: 12.698MB/s. Total time: 00:01:36.60
lrzip bastion-20120621-6-x86_64.pkg.tar 398,46s user 7,45s system 420% cpu 1:36,61 total
959M 3. Okt 18:08 bastion-20120621-6-x86_64.pkg.tar.lrz
% time xz bastion-20120621-6-x86_64.pkg.tar
xz bastion-20120621-6-x86_64.pkg.tar 423,66s user 1,62s system 99% cpu 7:06,30 total
1004M 22. Sep 23:09 bastion-20120621-6-x86_64.pkg.tar.xz
To conclude: lrzip uses less CPU time (uses 3GB memory peaked), compresses a bit better than xz, and is multithreading capable.
Decompression test:
% time xz -d bastion-20120621-6-x86_64.pkg.tar.xz
xz -d bastion-20120621-6-x86_64.pkg.tar.xz 16,90s user 0,66s system 43% cpu 40,003 total
% time lrunzip bastion-20120621-6-x86_64.pkg.tar.lrz
Output filename is: bastion-20120621-6-x86_64.pkg.tar
Decompressing...
100% 1219.17 / 1219.17 MB
Average DeCompression Speed: 24.878MB/s
Output filename is: bastion-20120621-6-x86_64.pkg.tar: [OK] - 1278392320 bytes
Total time: 00:00:48.89
lrunzip bastion-20120621-6-x86_64.pkg.tar.lrz 68,27s user 0,96s system 141% cpu 48,907 total
Decompression uses considerably more cpu time in the lrzip case and is slower than xz, because it seems to begin writing just after it has decompressed everything in memory , whereas xz immediately writes.
Offline
Benefits are probably the save in time when you have a multicore processor.
Save time compressing the package? If I'm building them locally, I don't have to compress at all.
I have a fast enough Internet so I don't care if the downloads are a bit bigger. Slower decompression doesn't sound good either.
Offline
So, any further discussion will probably be answered with this quote:
Adding lrz compression to makepkg is easy - a one line patch I think. I have not done it yet as there is little point until a released libarchive supports reading these packages.
Also, I doubt I will have any time to look into the relative benefits/disadvantages of using this compression format until an actual libarchive release with lrzip support is made.
That means we should bring this issue upstream to the libarchive devs.
Offline
That means we should bring this issue upstream to the libarchive devs.
Already done; it is merged and we await v3.0.5.
Reference: https://bbs.archlinux.org/viewtopic.php … 3#p1169963
Last edited by graysky (2012-10-03 19:11:05)
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
You are awaiting libarchive 3.1... That will not be added in a minor release.
Also, testing a plain decompression is not enough to justify use in pacman. There are other operations performed on the package file than just extraction (e.g. file lists are obtained for conflict checking, the install script is extracted separately if pre_intsall needs to be run...)
Offline
Thanks for the info, Allan. I think that your are the most qualified person to assess the cost/benefits of switching formats. Guess we're in a holding pattern until the next major release of libarchive is released.
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
I tried lrzip a while back, I first tried various large data files and compared it's compression against 7z which generally yielded small gains for lrzip, however I then tried comparing them on a large tar archive filled with tons of smaller somewhat similar files and here there was a HUGE difference, the long range algorithm was much more effective in finding repeated data amongst those files which resulted in a very substantial gain in compression compared to the best settings I could achieve with 7z.
I don't know how much of an impact that has on the packages Arch is distributing, but since it always seem to perform better than xz (speed, compression) I think it's certainly a good candidate.
Offline
since it always seem to perform better than xz (speed, compression)
I'd think decompression is significant. If that's slower, that hits ALL users.
But we'd need to wait for it to actually get tested of course.
Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.
Offline
Ah yes, you are right. Better compression and compression speed has much less practical benefit than fast decompression in a scenario like this, atleast unless the compression advantage is so substantal that it leads to heavily decreased bandwidth usage.
Even so, with today's connections I doubt alot of users will see a great benefit in a slightly faster download with longer decompression times as a result .
Offline
I would point folks to the PDF I referenced in my first post where several aspects of these issues are attempted to be addressed. What I didn't think about at the time, are some of the extra functions pacman does on packages like Allan mentioned. You can see, on a multicore machine, decompression is faster.
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
For this comparison to be more fair you need to compare with the alpha version of xz which has smp support.
There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums. That is why we avoid it. -- ewaller (arch linux forum moderator)
Offline
@dolby - I'll leave future testing to Allan. Also, I don't think comparing stable code (lrzip) to alpha code (xz smp) is a fair comparison at all. Once libarchive releases the next stable major release, the lrzip code therein will be considered stable and available. I highly doubt that Arch will run xz alpha in [core] for example!
CPU-optimized Linux-ck packages @ Repo-ck • AUR packages • Zsh and other configs
Offline
@dolby - I'll leave future testing to Allan. Also, I don't think comparing stable code (lrzip) to alpha code (xz smp) is a fair comparison at all. Once libarchive releases the next stable major release, the lrzip code therein will be considered stable and available. I highly doubt that Arch will run xz alpha in [core] for example!
Fedora have switched to the 5.1 xz version for a long while so it should be ok. I dont know if they add any other patches from scm to the alpha release though. Its more fair than comparing a program which has such an important feature implemented and one that does not, imho.
There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums. That is why we avoid it. -- ewaller (arch linux forum moderator)
Offline
I do not think there are any additional xz patches in git after the alpha release apart from documentation fixes.
Offline
Fun how such an old thread gets revived.
The major disadvantage of lrz is that it can only use its big advantages over xz when it can do multiple passes over the data, but not when used with stdin/stdout. This may be a showstopper for using it with packages.
Offline