You are not logged in.

#1 2009-05-10 14:41:39

lord_rel
Member
Registered: 2007-04-04
Posts: 34

slackware started using xz compression for packages

from the -current changelog
Fri May 8 18:49:03 CDT 2009
       Hello folks! This batch of updates includes the newly released KDE 4.2.3,
       but more noticeably it marks the first departure from the use of gzip for
       compressing Slackware packages. Instead, we will be using xz, based on
       the LZMA compression algorithm. xz offers better compression than even
       bzip2, but still offers good extraction performance (about 3 times better
       than bzip2 and not much slower than gzip in our testing). Since support
       for bzip2 has long been requested, support for bzip2 and the original lzma
       format has also been added (why not?), but this is purely in the interest
       of completeness -- we think most people will probably want to use either
       the original .tgz or the new .txz compression wrappers. The actual
       Slackware package format (which consists of the layout within the package
       envelope) has not changed, but this is the first support within Slackware's
       package tools for using alternate compression algorithms.
       Some people have asked why we don't pick a single extension, such as
       .slk. While there's certainly a case to be made for that idea, the tools
       would still need to support .tgz to handle older packages. Sticking with
       ".tgz" for everything makes no sense. Using extensions that reflect the
       compression format used by the package envelope seems to be the most
       transparent approach, and the one that best follows tradition.
       As an example of the compression improvement with .txz, have a look
       at the kernel-source package:
       Before: kernel-source-2.6.29.2_smp-noarch-1.tgz (73808508 bytes)
       After: kernel-source-2.6.29.2_smp-noarch-1.txz (49150104 bytes)
       The size of the main package tree in /slackware has been reduced from
       1.9GB to 1.4GB by converting most packages to .txz.
       Most of the packages have been converted from .tgz to .txz, but we
       will continue to make the gzip, pkgtools, slackpkg, tar, and xz packages
       in .tgz format for the foreseeable future.
       Enjoy! And thanks to Lasse Collin for the great work on xz. :-)

libarchive supports xz since 2.7.0 so maybe its worth testing for archlinux packages

Offline

#2 2009-05-10 15:29:49

camphor
Member
Registered: 2009-03-01
Posts: 32

Re: slackware started using xz compression for packages

Fingers crossed for Slack 13 to support x86_64..

Offline

#3 2009-05-10 16:02:16

lord_rel
Member
Registered: 2007-04-04
Posts: 34

Re: slackware started using xz compression for packages

camphor wrote:

Fingers crossed for Slack 13 to support x86_64..

for now slamd64 is all there is for that

Offline

#4 2009-05-10 18:02:21

thoffmeyer
Member
From: /dev/zero
Registered: 2006-07-27
Posts: 90
Website

Re: slackware started using xz compression for packages

This looks interesting, I really think arch should do the same as far as the xz packages.

Offline

#5 2009-05-10 19:19:22

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819

Re: slackware started using xz compression for packages

For those interesting in doing it themselves:

Shameless plug tongue

The file utility already has xz support built in by now (from 5.03 on). It's a piece of cake, and it's nice if you have your own online repo (shaves off some KB - or MB sometimes smile).


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#6 2009-05-10 22:43:14

test1000
Member
Registered: 2005-04-03
Posts: 834

Re: slackware started using xz compression for packages

Would be good for not taxing the main repo/repos i guess

Last edited by test1000 (2009-05-10 22:43:54)


KISS = "It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." - Albert Einstein

Offline

#7 2009-05-10 23:24:44

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819

Re: slackware started using xz compression for packages

Main repo is throttled already wink. People are strongly advised not to use it. But since it can bring down the package size considerably, yes, it would be nice smile.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#8 2009-05-10 23:29:14

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: slackware started using xz compression for packages

I can't see a reason why Arch shouldn't take this path.


neutral

Offline

#9 2009-05-11 03:57:53

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: slackware started using xz compression for packages

Well Slackware has an issue Arch doesnt have.
It ships the whole distribution, inluding the full source code of all distributed applications in a single DVD.
And since every application keeps adding dependencies, eg GCC with ppl, GIMP gegl & babl etc. size was becoming a big problem.
For Arch package size can be an issue only when it comes to downloading them.
Personally i dont consider that such big though.

I would also like to note that xz is still beta software, while gzip is a package format being around for ages, and is pretty much the default.
There are of course conserns about using xz. for example when it comes to file managers, which cant deal with the archives.
And the gain is only in size, as gzip can still do better in speed.

Last edited by dolby (2009-05-11 03:58:29)


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#10 2009-05-11 10:52:20

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: slackware started using xz compression for packages

Like I said, I can't see a reason why Arch shouldn't take this path. It would be nice to have something else other distros don't.


neutral

Offline

#11 2009-05-11 11:21:01

test1000
Member
Registered: 2005-04-03
Posts: 834

Re: slackware started using xz compression for packages

"It would be nice to have something else other distros don't."

why? to be "cool"?

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.


KISS = "It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." - Albert Einstein

Offline

#12 2009-05-11 11:32:36

Allan
Supreme Leader
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,750
Website

Re: slackware started using xz compression for packages

And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.

Offline

#13 2009-05-11 11:33:00

bender02
Member
From: Germany
Registered: 2007-02-04
Posts: 1,328

Re: slackware started using xz compression for packages

test1000 wrote:

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.

xz is supported by the current tar...

Offline

#14 2009-05-11 19:54:22

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: slackware started using xz compression for packages

This is already possible for the most part. Try gunzip-ing a package and recompressing with bzip2. pacman -U still works fine.

Offline

#15 2009-05-11 22:34:26

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: slackware started using xz compression for packages

test1000 wrote:

"It would be nice to have something else other distros don't."

why? to be "cool"?

Exactly.


neutral

Offline

#16 2009-05-11 22:56:16

Hide
Member
From: Castalia
Registered: 2007-02-02
Posts: 368

Re: slackware started using xz compression for packages

Allan wrote:

And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.

Seriously? Awesome!

Offline

#17 2009-05-12 00:43:24

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819

Re: slackware started using xz compression for packages

test1000 wrote:

If this is beta software we so shouldn't use it. Also, yes, i would like to be able to open archlinux packages with unpatched archive managers(console and nonconsole) before anysuch thing is implemented.

Xz (formerly known as LZMA) is hardly beta. The spec is final, and like someone else said tar (and libarchive, in the 2.6.9x version) already support it. File just got an update to identify the headers correctly, too.

Hardly what I'd call 'beta'. Bluez 4 is what I'd call 'beta' compared to xz wink.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#18 2009-05-12 03:32:40

ks1
Member
From: Atlanta, Georgia
Registered: 2006-03-26
Posts: 62
Website

Re: slackware started using xz compression for packages

bring on XZ!

Offline

#19 2009-05-12 05:51:11

zenlord
Member
From: Belgium
Registered: 2006-05-24
Posts: 1,213
Website

Re: slackware started using xz compression for packages

Hide wrote:
Allan wrote:

And the delta implementation is complete(ish...) now so we might get that working once pacman-3.3 comes out.

Seriously? Awesome!

+1

Big thumbs up for this - I think pacman-delta will do much more shaving off than a few MB per package...

Offline

#20 2009-05-12 10:04:49

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: slackware started using xz compression for packages

B wrote:

Hardly what I'd call 'beta'. Bluez 4 is what I'd call 'beta' compared to xz wink.

Hahahaha! That is indeed very funny smile


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#21 2009-05-12 13:14:58

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: slackware started using xz compression for packages

Why doesn't pacman also do this? I'm sure its dead simple to support and I bet it would save a ton of bandwidth.

Offline

#22 2009-05-12 14:37:16

oopsicrappe
Member
Registered: 2009-01-18
Posts: 17

Re: slackware started using xz compression for packages

Pacman already works with xz if you have xz-utils and libarchive 2.7 installed.

Offline

#23 2009-05-12 22:35:29

z.s.tar.gz
Member
From: Rural Florida
Registered: 2008-12-12
Posts: 298

Re: slackware started using xz compression for packages

It seems to be not a question of whether pacman can do it or not, but when the repos are going to start with xz. (Or so is my understanding of the situation)


I need to find a way out so everyone can find their way out.
Resregietd Lunix Uesr: 485581

Offline

#24 2009-05-12 23:30:06

tomd123
Developer
Registered: 2008-08-12
Posts: 565

Re: slackware started using xz compression for packages

z.s.tar.gz wrote:

It seems to be not a question of whether pacman can do it or not, but when the repos are going to start with xz. (Or so is my understanding of the situation)

Ya, I would rather spend less time downloading and more time utilizing my cpu. Only downside I can see from this is that installing will take longer.
I want to see the repo sizes before and after the xz compression replaces gz. I want to lmao.

Offline

#25 2009-05-13 00:16:25

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: slackware started using xz compression for packages

I did some very basic tests, for whoever is interested: with default settings of both, xz compresses much longer than gzip, decompresses slightly slower and shaves 40MB off openoffice 3.1 (104MB vs 144MB).

Last edited by lucke (2009-05-13 00:16:42)

Offline

Board footer

Powered by FluxBB