slackware started using xz compression for packages

lord_rel · 2009-05-10 14:41:39

from the -current changelog
Fri May 8 18:49:03 CDT 2009
Hello folks! This batch of updates includes the newly released KDE 4.2.3,
but more noticeably it marks the first departure from the use of gzip for
compressing Slackware packages. Instead, we will be using xz, based on
the LZMA compression algorithm. xz offers better compression than even
bzip2, but still offers good extraction performance (about 3 times better
than bzip2 and not much slower than gzip in our testing). Since support
for bzip2 has long been requested, support for bzip2 and the original lzma
format has also been added (why not?), but this is purely in the interest
of completeness -- we think most people will probably want to use either
the original .tgz or the new .txz compression wrappers. The actual
Slackware package format (which consists of the layout within the package
envelope) has not changed, but this is the first support within Slackware's
package tools for using alternate compression algorithms.
Some people have asked why we don't pick a single extension, such as
.slk. While there's certainly a case to be made for that idea, the tools
would still need to support .tgz to handle older packages. Sticking with
".tgz" for everything makes no sense. Using extensions that reflect the
compression format used by the package envelope seems to be the most
transparent approach, and the one that best follows tradition.
As an example of the compression improvement with .txz, have a look
at the kernel-source package:
Before: kernel-source-2.6.29.2_smp-noarch-1.tgz (73808508 bytes)
After: kernel-source-2.6.29.2_smp-noarch-1.txz (49150104 bytes)
The size of the main package tree in /slackware has been reduced from
1.9GB to 1.4GB by converting most packages to .txz.
Most of the packages have been converted from .tgz to .txz, but we
will continue to make the gzip, pkgtools, slackpkg, tar, and xz packages
in .tgz format for the foreseeable future.
Enjoy! And thanks to Lasse Collin for the great work on xz. :-)

libarchive supports xz since 2.7.0 so maybe its worth testing for archlinux packages

camphor · 2009-05-10 15:29:49

Fingers crossed for Slack 13 to support x86_64..

lord_rel · 2009-05-10 16:02:16

camphor wrote:

Fingers crossed for Slack 13 to support x86_64..

for now slamd64 is all there is for that

thoffmeyer · 2009-05-10 18:02:21

This looks interesting, I really think arch should do the same as far as the xz packages.

.:B:. · 2009-05-10 19:19:22

For those interesting in doing it themselves:

Shameless plug

The file utility already has xz support built in by now (from 5.03 on). It's a piece of cake, and it's nice if you have your own online repo (shaves off some KB - or MB sometimes ).

test1000 · 2009-05-10 22:43:14

Would be good for not taxing the main repo/repos i guess

Last edited by test1000 (2009-05-10 22:43:54)

.:B:. · 2009-05-10 23:24:44

Main repo is throttled already . People are strongly advised not to use it. But since it can bring down the package size considerably, yes, it would be nice .

sand_man · 2009-05-10 23:29:14

I can't see a reason why Arch shouldn't take this path.

dolby · 2009-05-11 03:57:53

Well Slackware has an issue Arch doesnt have.
It ships the whole distribution, inluding the full source code of all distributed applications in a single DVD.
And since every application keeps adding dependencies, eg GCC with ppl, GIMP gegl & babl etc. size was becoming a big problem.
For Arch package size can be an issue only when it comes to downloading them.
Personally i dont consider that such big though.

I would also like to note that xz is still beta software, while gzip is a package format being around for ages, and is pretty much the default.
There are of course conserns about using xz. for example when it comes to file managers, which cant deal with the archives.
And the gain is only in size, as gzip can still do better in speed.

Last edited by dolby (2009-05-11 03:58:29)

sand_man · 2009-05-11 10:52:20

Like I said, I can't see a reason why Arch shouldn't take this path. It would be nice to have something else other distros don't.

test1000 · 2009-05-11 11:21:01

"It would be nice to have something else other distros don't."

why? to be "cool"?

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.

Allan · 2009-05-11 11:32:36

And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.

bender02 · 2009-05-11 11:33:00

test1000 wrote:

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.

xz is supported by the current tar...

phrakture · 2009-05-11 19:54:22

This is already possible for the most part. Try gunzip-ing a package and recompressing with bzip2. pacman -U still works fine.

sand_man · 2009-05-11 22:34:26

test1000 wrote:

"It would be nice to have something else other distros don't."
why? to be "cool"?

Exactly.

Hide · 2009-05-11 22:56:16

Allan wrote:

And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.

Seriously? Awesome!

.:B:. · 2009-05-12 00:43:24

test1000 wrote:

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.

Xz (formerly known as LZMA) is hardly beta. The spec is final, and like someone else said tar (and libarchive, in the 2.6.9x version) already support it. File just got an update to identify the headers correctly, too.

Hardly what I'd call 'beta'. Bluez 4 is what I'd call 'beta' compared to xz .

ks1 · 2009-05-12 03:32:40

bring on XZ!

zenlord · 2009-05-12 05:51:11

Hide wrote:

Allan wrote:
And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.
Seriously? Awesome!

+1

Big thumbs up for this - I think pacman-delta will do much more shaving off than a few MB per package...

dolby · 2009-05-12 10:04:49

B wrote:

Hardly what I'd call 'beta'. Bluez 4 is what I'd call 'beta' compared to xz .

Hahahaha! That is indeed very funny

tomd123 · 2009-05-12 13:14:58

Why doesn't pacman also do this? I'm sure its dead simple to support and I bet it would save a ton of bandwidth.

oopsicrappe · 2009-05-12 14:37:16

Pacman already works with xz if you have xz-utils and libarchive 2.7 installed.

z.s.tar.gz · 2009-05-12 22:35:29

It seems to be not a question of whether pacman can do it or not, but when the repos are going to start with xz. (Or so is my understanding of the situation)

tomd123 · 2009-05-12 23:30:06

z.s.tar.gz wrote:

It seems to be not a question of whether pacman can do it or not, but when the repos are going to start with xz. (Or so is my understanding of the situation)

Ya, I would rather spend less time downloading and more time utilizing my cpu. Only downside I can see from this is that installing will take longer.
I want to see the repo sizes before and after the xz compression replaces gz. I want to lmao.

lucke · 2009-05-13 00:16:25

I did some very basic tests, for whoever is interested: with default settings of both, xz compresses much longer than gzip, decompresses slightly slower and shaves 40MB off openoffice 3.1 (104MB vs 144MB).

Last edited by lucke (2009-05-13 00:16:42)

Arch Linux

#1 2009-05-10 14:41:39

slackware started using xz compression for packages

#2 2009-05-10 15:29:49

Re: slackware started using xz compression for packages

#3 2009-05-10 16:02:16

Re: slackware started using xz compression for packages

#4 2009-05-10 18:02:21

Re: slackware started using xz compression for packages

#5 2009-05-10 19:19:22

Re: slackware started using xz compression for packages

#6 2009-05-10 22:43:14

Re: slackware started using xz compression for packages

#7 2009-05-10 23:24:44

Re: slackware started using xz compression for packages

#8 2009-05-10 23:29:14

Re: slackware started using xz compression for packages

#9 2009-05-11 03:57:53

Re: slackware started using xz compression for packages

#10 2009-05-11 10:52:20

Re: slackware started using xz compression for packages

#11 2009-05-11 11:21:01

Re: slackware started using xz compression for packages

#12 2009-05-11 11:32:36

Re: slackware started using xz compression for packages

#13 2009-05-11 11:33:00

Re: slackware started using xz compression for packages

#14 2009-05-11 19:54:22

Re: slackware started using xz compression for packages

#15 2009-05-11 22:34:26

Re: slackware started using xz compression for packages

#16 2009-05-11 22:56:16

Re: slackware started using xz compression for packages

#17 2009-05-12 00:43:24

Re: slackware started using xz compression for packages

#18 2009-05-12 03:32:40

Re: slackware started using xz compression for packages

#19 2009-05-12 05:51:11

Re: slackware started using xz compression for packages

#20 2009-05-12 10:04:49

Re: slackware started using xz compression for packages

#21 2009-05-12 13:14:58

Re: slackware started using xz compression for packages

#22 2009-05-12 14:37:16

Re: slackware started using xz compression for packages

#23 2009-05-12 22:35:29

Re: slackware started using xz compression for packages

#24 2009-05-12 23:30:06

Re: slackware started using xz compression for packages

#25 2009-05-13 00:16:25

Re: slackware started using xz compression for packages

Board footer