xz (LZMA2) packages

cleanrock · 2010-03-06 12:06:56

I read that arch is moving to xz packages and i am a bit skeptical.
Is it really worth ~30% smaller archives when decompressing takes 3 times longer and compressing take about 10 times longer ?

lymphatik · 2010-03-06 12:27:31

I'm not sure about the ten times slower. But if they move to it, it's mostly for saving bandwidth.

http://stephane.lesimple.fr/blog/archiv … hmark.html

Last edited by lymphatik (2010-03-06 12:27:49)

Allan · 2010-03-06 12:41:44

I'm prepared to take the 10x longer compression as it will save me a heap of time uploading. You can keep PKGEXT='.pkg.tar.gz' in your makpekg.conf and not worry about compression time.

As far as decompression goes, it is about 2 to 2.5 times as slow and if you have to download the package, you will end up saving time. Of course, reinstalls from the cache would be slower.

n0dix · 2010-03-06 13:22:40

Is there a announce or an article relate to it?

Allan · 2010-03-06 13:30:54

No... there was some discussion on arch-dev-public but I do not think there has been a formal "everyone switch" directive.

n0dix · 2010-03-06 13:35:05

Thanks for the quickly answer Allan.

.:B:. · 2010-03-06 16:05:59

cleanrock wrote:

I read that arch is moving to xz packages and i am a bit skeptical.
Is it really worth ~30% smaller archives when decompressing takes 3 times longer and compressing take about 10 times longer ?

Sounds like a lot of FUD and an utter lack of personal experience.

I've been using XZ for my own package repo ever since libarchive introduced support for it. That was already a while ago. My system is a 1,2 GHz Core 2 Duo. Not really something you want to throw powerful compression at, but it does a good job. Way faster than bzip2, slower than gzip, but better compression than both. 99% of the time though, you'll only decompress. You'll just install packages from the repos, and the few ones you build yourself (if any) won't really make the difference.

Oh, and did I mention it doesn't even blink when it has to install xz packages?

fflarex · 2010-03-06 16:10:33

I think it's the right decision, because (de)compression does not cost anyone very much money, whereas bandwidth definitely does...

anonymous_user · 2010-03-06 16:40:35

So when will this change be starting/taking place?

some-guy94 · 2010-03-06 16:52:45

anonymous_user wrote:

So when will this change be starting/taking place?

It already has, new/updated packages are using xz now.

Unless you have your own repo, you should keep your own PKGEXT to tar.gz.

shinythings · 2010-03-06 17:53:21

some-guy94 wrote:

Unless you have your own repo, you should keep your own PKGEXT to tar.gz.

Why do you suggest that? For the time saved compressing?

some-guy94 · 2010-03-06 18:15:57

shinythings wrote:

some-guy94 wrote:
Unless you have your own repo, you should keep your own PKGEXT to tar.gz.
Why do you suggest that? For the time saved compressing?

Disk space is cheap, and compressing/decompressing takes longer.

JGC · 2010-03-06 18:47:23

I've seen packages like libgweather dropping from 8MB to 2.2MB. That's a 72.5% decrease in size. Compressing packages with .xz takes more time, but decompression is only 2 times slower than gzip. Decompression is only a small part of the process of installing packages though:
(- download)
- read file from disk
- decompress
- write files to disk

By decreasing the package size, you have to download less and you have to read less from disk. If you don't have the file cached, you'll gain speed, if you have the file cached, you will see a slight delay compared to gzip.

One funny finding is this, taken from the OpenOffice package. Times are from uncompressing the compressed tarball itself, the tarball isn't extracted here:
gzip: 0m 5.772s
bzip2: 0m29.433s
xz -1: 0m13.983s
xz -2: 0m12.949s
xz -3: 0m12.706s
xz -6: 0m11.462s

You'll see that decompression times decrease when the compression rate increases. This is caused by the smaller input file. The difference between xz -1 and xz -6 in file size was 25.8MB. The gzip file was 41.3MB larger than the xz -6 compressed file. By using xz -6 compression on the OpenOffice package we saved 25% on package size.

As for cheap diskspace: disk space is cheap yes, but bandwidth isn't. When a large amount of packages is updated on our servers, it can take a day or more to get everything pushed to all mirrors. If we can decrease the needed bandwidth with 25%, we can transfer more packages to our mirrors in less time. Another thing you should keep in mind is that though disk space on your home machine is cheap, mirrors could be coping with expensive SAS or SAN setups that are way more expensive.

godane · 2010-03-06 20:15:43

My thought on xz packages is the should only be used on extra and community packages. This will make it less confusing when building for core packages. You don't have to think if its a depend of pacman etc.

Pierre · 2010-03-06 20:19:51

The upgrade path from older installation which don't know about xz is alreaady taken care of. Ther are only a few pacakaes which need to be kept in gz format; not the entire core repo. (Sure, updating from pacman 2.x might be a problem, but I just don't care)

cleanrock · 2010-03-06 20:22:51

.:B:. wrote:

cleanrock wrote:
I read that arch is moving to xz packages and i am a bit skeptical.
Is it really worth ~30% smaller archives when decompressing takes 3 times longer and compressing take about 10 times longer ?
Sounds like a lot of FUD and an utter lack of personal experience.
I've been using XZ for my own package repo ever since libarchive introduced support for it. That was already a while ago. My system is a 1,2 GHz Core 2 Duo. Not really something you want to throw powerful compression at, but it does a good job. Way faster than bzip2, slower than gzip, but better compression than both. 99% of the time though, you'll only decompress. You'll just install packages from the repos, and the few ones you build yourself (if any) won't really make the difference.
Oh, and did I mention it doesn't even blink when it has to install xz packages?

not really FUD, i understand that xz has more pros than cons, i just wanted to hear some comments/discussion about this since i could not find anything on the forum .. i am happy now (and probably some more people) after seeing some comments from the devs.
PS. you sound a bit angry in ur reply, i dont think thats the arch way

combuster · 2010-03-06 20:58:10

Well bandwith is everything so I support the switch to xz. This will be good for syncing mirrors also

.:B:. · 2010-03-08 03:31:50

cleanrock wrote:

not really FUD, i understand that xz has more pros than cons, i just wanted to hear some comments/discussion about this since i could not find anything on the forum .. i am happy now (and probably some more people) after seeing some comments from the devs.
PS. you sound a bit angry in ur reply, i dont think thats the arch way

Not really angry... I'm just apalled people are being this skeptical. I mean, even Slackware - which is one of the most conservative distros - already switched to xz. You can hardly call that a rash decision.

You can still compress your own packages with gzip or whatever (a choice the user has always had), and since the difference in decompression time is minimal, it's a win-win. Most devs really won't care for the increased compression time; the users even less.

MP2E · 2010-03-08 03:42:40

.tar.xz is amazing! I just took the git package(unfortunately still gzip at the moment) which is a 6MB download, decompressed, then compressed to .tar.xz. 2MB! I was so shocked. Fantastic

And really, compression time is not much larger, and decompression time is even less!
(By the way I used the -9 and -e options for maximum compression and maximum compression time)

SiC · 2010-03-08 10:31:26

Can honestly say that I had not noticed the change over. Just checked my package cache, and sure enough, there are some .xz files there. Just goes to show how much attention I pay and how good the pacman devs are. Nice work guys

sand_man · 2010-03-08 10:54:38

SiC wrote:

Can honestly say that I had not noticed the change over. Just checked my package cache, and sure enough, there are some .xz files there. Just goes to show how much attention I pay and how good the pacman devs are. Nice work guys

Same here. I'm all for it but I have to admit that I didn't even notice either. I would much rather be downloading less and decompressing (slightly) longer. Seems like a no brainer to me.

mangus · 2010-03-10 12:16:15

All I have to say is...good job guys! Thank you!

bharani · 2010-03-10 13:04:43

Having a connection with top speed of about 40kbps i am very happy to see this feature.
Thank you devs.

mathieui · 2010-03-11 17:00:31

Just one issue with those .xz packages: pacman.static does not handle them yet, afaik, so a chroot install is a bit harsh.
Without this problem, I found those great, even if I don't have a really bad connection (around 700kBps), it will save me some disk space when making a local repo .
My friend with a slooow netbook won't be so pleased with this, but I guess that it isn't a real issue.

wonder · 2010-03-11 17:02:47

mathieui wrote:

Just one issue with those .xz packages: pacman.static does not handle them yet, afaik, so a chroot install is a bit harsh.

i remember that the guy who maintain pacman.static submitted a feature requesting us to ship xz-utils with static lib. we did long time ago so i believe he intent to add this support.

Arch Linux

#1 2010-03-06 12:06:56

xz (LZMA2) packages

#2 2010-03-06 12:27:31

Re: xz (LZMA2) packages

#3 2010-03-06 12:41:44

Re: xz (LZMA2) packages

#4 2010-03-06 13:22:40

Re: xz (LZMA2) packages

#5 2010-03-06 13:30:54

Re: xz (LZMA2) packages

#6 2010-03-06 13:35:05

Re: xz (LZMA2) packages

#7 2010-03-06 16:05:59

Re: xz (LZMA2) packages

#8 2010-03-06 16:10:33

Re: xz (LZMA2) packages

#9 2010-03-06 16:40:35

Re: xz (LZMA2) packages

#10 2010-03-06 16:52:45

Re: xz (LZMA2) packages

#11 2010-03-06 17:53:21

Re: xz (LZMA2) packages

#12 2010-03-06 18:15:57

Re: xz (LZMA2) packages

#13 2010-03-06 18:47:23

Re: xz (LZMA2) packages

#14 2010-03-06 20:15:43

Re: xz (LZMA2) packages

#15 2010-03-06 20:19:51

Re: xz (LZMA2) packages

#16 2010-03-06 20:22:51

Re: xz (LZMA2) packages

#17 2010-03-06 20:58:10

Re: xz (LZMA2) packages

#18 2010-03-08 03:31:50

Re: xz (LZMA2) packages

#19 2010-03-08 03:42:40

Re: xz (LZMA2) packages

#20 2010-03-08 10:31:26

Re: xz (LZMA2) packages

#21 2010-03-08 10:54:38

Re: xz (LZMA2) packages

#22 2010-03-10 12:16:15

Re: xz (LZMA2) packages

#23 2010-03-10 13:04:43

Re: xz (LZMA2) packages

#24 2010-03-11 17:00:31

Re: xz (LZMA2) packages

#25 2010-03-11 17:02:47

Re: xz (LZMA2) packages

Board footer