slackware started using xz compression for packages

heleos · 2009-05-13 00:56:51

test1000 wrote:

If this is beta software we so shouldn't use it.

I thought this was a bleeding-edge distro?

tomd123 · 2009-05-13 01:31:54

I have looked into this issue at hand and have done some tests of my own. I decided to recompress the core repo as xz and find out the difference.
These are the steps I followed:

[tom@archlinux os]$ ~/mirror/scripts/mirrorsync.sh
[tom@archlinux os]$ du -h i686/
176M i686/
[tom@archlinux os]$ gunzip `find ./ -name *pkg*`
[tom@archlinux os]$ du -h i686/
488M i686/
[tom@archlinux os]$ xz `find ./ -name *pkg*`
[tom@archlinux os]$ du -h i686/
119M i686/
[tom@archlinux os]$ unxz `find ./ -name *pkg*`
[tom@archlinux os]$ xz -9 `find ./ -name *pkg*`
[tom@archlinux os]$ du -h i686/
110M i686/

as you can see, switching to xz wouldn't be such a bad idea in terms of space and bandwidth savings. It reduced the core repository from 176M with gz to 110M with xz -9. The xz repo is 62.5% (110/176) the size of the gz repo That's a big difference imo.

Last edited by tomd123 (2009-05-13 01:37:17)

zenlord · 2009-05-13 07:32:45

That is indeed a nice bonus, but this is only relevant the first time you install a new package. With pacman 3.3, which will arrive soon(ish), only the diff between the old package and the new will have to be downloaded, probably reducing your oo.o-package from 176MB to something like 30MB.

the xz-hype (maybe not the correct word since it has advantages) is more useful for distro's like slackware and ubuntu tha have scheduled releases. For a rolling release, bleeding edge distro the delta-patch is thus much more important.

bender02 · 2009-05-13 08:44:31

zenlord wrote:

That is indeed a nice bonus, but this is only relevant the first time you install a new package.

This doesn't make too much sense since using xz also for deltas can over time still reduce the overall traffic by 20-30% perhaps.

Yes, deltas are important, but why "reject" xz just because it's a hype atm, while it has measurable improvement with no work at all (in a way pacman already supports xz compressed packages).

Allan · 2009-05-13 09:01:48

bender02 wrote:

zenlord wrote:
That is indeed a nice bonus, but this is only relevant the first time you install a new package.
This doesn't make too much sense since using xz also for deltas can over time still reduce the overall traffic by 20-30% perhaps.
Yes, deltas are important, but why "reject" xz just because it's a hype atm, while it has measurable improvement with no work at all (in a way pacman already supports xz compressed packages).

According to shining on the pacman-dev list, deltas probably do not work that well with xv as xdelta only supports gzip and bzip2.

Edit: here is the quote

shining on pacman-dev wrote:

To sum up the problems I expect :
* xdelta will be inefficient with xz packages since it does not recognize it

So it appears there is a choice to be made for the time being...

bender02 · 2009-05-13 11:30:36

Ah, then it's a different matter. In this situation I'd go with the deltas.
For some reason I thought that a "delta" would be applied to the uncompresed stuff, and then the differences would be compressed. [EDIT: I've checked the website of xdelta and I understand now that it is in a way done like that but the whole task is passed down onto xdelta.]

Last edited by bender02 (2009-05-13 11:34:53)

xduugu · 2009-05-13 12:40:05

I tested the delta compression on kernel26 and wesnoth packages and the results are really interesting. While computing deltas on gzip'ed tarballs, the usage of external decompression (always?/often?) results in smaller deltas, whereas the deltas of xz compressed tarballs can be (sometimes?/often?) smaller if external decompression is disabled. Probably, it's only applicable to wesnoth, but I think it would be worth to investigate this further.

For the tests I patched xdelta3 to support xz:

diff -ruNa a/xdelta3-main.h b/xdelta3-main.h
--- a/xdelta3-main.h    2009-01-30 05:59:02.000000000 +0100
+++ b/xdelta3-main.h    2009-05-13 12:43:00.000000000 +0200
@@ -355,6 +355,7 @@
     RD_NONEXTERNAL },
   { "bzip2",    "-cf",   "bzip2",      "-dcf",   "B", "BZh",          3, 0 },
   { "gzip",     "-cf",   "gzip",       "-dcf",   "G", "\037\213",     2, 0 },
+  { "xz",       "-cf",   "xz",         "-dcf",   "Y", "\xfd\x37\x7a\x58\x5a\x00",     2, 0 },
   { "compress", "-cf",   "uncompress", "-cf",    "Z", "\037\235",     2, 0 },
 
   /* TODO: add commandline support for magic-less formats */

filename legend:
$pkgname.delta
Delta between uncompressed tarballs; basically for reference; size should be roughly the same like the ones of deltas with external decompression enabled

xdelta3 -e -s $pkgname*.pkg.tar $pkgname.delta

$pkgname.{gz,xz}.delta
Delta between compressed tarballs with external decompression

xdelta3 -e -s $pkgname*.pkg.tar.(gz|xz) $pkgname.(gz|xz).delta

$pkgname.{gz,xz}_without_decompression.delta
Delta between compressed tarballs WITHOUT external decompression

xdelta3 -D -e -s $pkgname*.pkg.tar.(gz|xz) $pkgname.(gz|xz)_without_decompression.delta

Results:
kernel26

86M kernel26-2.6.29.2-1-i686.pkg.tar
30M kernel26-2.6.29.2-1-i686.pkg.tar.gz
22M kernel26-2.6.29.2-1-i686.pkg.tar.xz
86M kernel26-2.6.29.3-1-i686.pkg.tar
30M kernel26-2.6.29.3-1-i686.pkg.tar.gz
22M kernel26-2.6.29.3-1-i686.pkg.tar.xz
22M kernel26.delta
19M kernel26.delta.gz
18M kernel26.delta.xz
22M kernel26.gz.delta
19M kernel26.gz.delta.gz
30M kernel26.gz_without_decompression.delta
30M kernel26.gz_without_decompression.delta.gz
22M kernel26.xz.delta
18M kernel26.xz.delta.xz
22M kernel26.xz_without_decompression.delta
22M kernel26.xz_without_decompression.delta.xz

more detailed sizes:

90152960 kernel26-2.6.29.2-1-i686.pkg.tar
31148268 kernel26-2.6.29.2-1-i686.pkg.tar.gz
22940388 kernel26-2.6.29.2-1-i686.pkg.tar.xz
89927680 kernel26-2.6.29.3-1-i686.pkg.tar
31107904 kernel26-2.6.29.3-1-i686.pkg.tar.gz
22953052 kernel26-2.6.29.3-1-i686.pkg.tar.xz
22282527 kernel26.delta
19708199 kernel26.delta.gz
18868220 kernel26.delta.xz
22282535 kernel26.gz.delta
19708197 kernel26.gz.delta.gz
30932079 kernel26.gz_without_decompression.delta
30911883 kernel26.gz_without_decompression.delta.gz
22282537 kernel26.xz.delta
18868144 kernel26.xz.delta.xz
22724845 kernel26.xz_without_decompression.delta
22726032 kernel26.xz_without_decompression.delta.xz

wesnoth

212M wesnoth-1.4.7-1-i686.pkg.tar
152M wesnoth-1.4.7-1-i686.pkg.tar.gz
134M wesnoth-1.4.7-1-i686.pkg.tar.xz
287M wesnoth-1.6.1-1-i686.pkg.tar
220M wesnoth-1.6.1-1-i686.pkg.tar.gz
202M wesnoth-1.6.1-1-i686.pkg.tar.xz
216M wesnoth.delta
211M wesnoth.delta.gz
208M wesnoth.delta.xz
216M wesnoth.gz.delta
211M wesnoth.gz.delta.gz
219M wesnoth.gz_without_decompression.delta
219M wesnoth.gz_without_decompression.delta.gz
216M wesnoth.xz.delta
208M wesnoth.xz.delta.xz
197M wesnoth.xz_without_decompression.delta
197M wesnoth.xz_without_decompression.delta.xz

more detailed sizes:

221470720 wesnoth-1.4.7-1-i686.pkg.tar
158795896 wesnoth-1.4.7-1-i686.pkg.tar.gz
140260936 wesnoth-1.4.7-1-i686.pkg.tar.xz
300247040 wesnoth-1.6.1-1-i686.pkg.tar
230476356 wesnoth-1.6.1-1-i686.pkg.tar.gz
211382664 wesnoth-1.6.1-1-i686.pkg.tar.xz
226461560 wesnoth.delta
220719423 wesnoth.delta.gz
217241596 wesnoth.delta.xz
226461568 wesnoth.gz.delta
220719416 wesnoth.gz.delta.gz
228946912 wesnoth.gz_without_decompression.delta
228944320 wesnoth.gz_without_decompression.delta.gz
226461570 wesnoth.xz.delta
217242644 wesnoth.xz.delta.xz
206067682 wesnoth.xz_without_decompression.delta
205862860 wesnoth.xz_without_decompression.delta.xz

test1000 · 2009-05-13 17:17:36

heleos wrote:

test1000 wrote:
If this is beta software we so shouldn't use it.
I thought this was a bleeding-edge distro?

I thought everyone knew this, but Arch tries as far as possible to only bundle the latest stable software.

z.s.tar.gz · 2009-05-13 19:56:59

I'm go!

edit: also, xz is just a new name. The actual software is not "beta".

double edit: nevermind if pacman 3.3 is as advertised

Last edited by z.s.tar.gz (2009-05-13 20:00:34)

tomd123 · 2009-05-13 22:07:56

Just out of curiousity, how exactly will this xdelta work? Won't you need to have a different xdelta for every 2 binary package versions? Or are you just going to use xdelta to create a chain of xdeltas leading up to the current version?

Can you maybe provide a link explaining how pacman will use xdelta and how the process will work?

sand_man · 2009-05-13 23:28:03

Is there a planned release for the new pacman with delta support?

Hide · 2009-05-14 05:38:00

http://bbs.archlinux.org/viewtopic.php? … 40#p551340

brain0 · 2009-05-17 22:50:57

This has been discussed already on the development list (you can search for it yourself). The consensus is that most think it is a good idea, there were just some concerns about compatibility with xdelta (which some people also want to see used soon).

Pacman already handles xz-compressed packages if you build libarchive with xz support. I think makepkg has to be adjusted slightly and our dev/repository tools have some gzip filenames hardcoded. Also pacman doesn't care if the repository is a mix of gzip, bzip2, xz and whatever, so a transition could be made smoothly.

sand_man · 2009-05-17 23:21:42

Hide wrote:

http://bbs.archlinux.org/viewtopic.php? … 40#p551340

Sorry I meant was there an estimated time frame?

Allan · 2009-05-17 23:48:18

Timeframe for pacman-3.3 release is.... unknown. But lets say a month.

I need to fix one thing that I know is broken in makepkg. Then Dan needs to spends some time dealing with a few patches floating around and deciding which to pull just before release. Then a string freeze will occur and translators need to do their work while a few of us will be running pacman from git looking for show stoppers.

.:B:. · 2009-05-18 05:26:23

ks1 wrote:

bring on XZ!

Lol . Look who's waking up .

shining · 2009-05-18 15:51:58

xduugu wrote:

I tested the delta compression on kernel26 and wesnoth packages and the results are really interesting. While computing deltas on gzip'ed tarballs, the usage of external decompression (always?/often?) results in smaller deltas, whereas the deltas of xz compressed tarballs can be (sometimes?/often?) smaller if external decompression is disabled. Probably, it's only applicable to wesnoth, but I think it would be worth to investigate this further.

ooooh, this is very interesting indeed.
I just assumed xz compression would have the same effect than gz compression on deltas, but this was completely wrong according to your experiments, thanks for them.
I am very curious to know the explanation of this behavior now.

So, that should mean that the xdelta + xz combination is already fine after all

EDIT : wait wait, I just noticed the size of the deltas in your test are very bad, they are as big as the packages, which makes delta totally useless.

EDIT2 : ok, so doing a test on a bigger number of packages (more than 100), I see what I expected :
for gz packages, the delta size goes from 1% to 100% of the package size (full distribution of the ratio )
for xz packages, it goes from 81% to 100%.

And we actually configured pacman to not use delta which have a ratio bigger than 70% because it is not worth it (more time to apply the delta, more error prone compared to just downloading the whole package).
So for xz packages, delta would never be used.

Now if we patch xdelta to support xz, the delta size will be very similar to gz, because the packages will be automatically uncompressed and re-compressed anyway.
However, the ratio will be less interesting, because the compressed packages are smaller with xz compression than gz compression.

Last edited by shining (2009-05-19 13:35:33)

djgera · 2009-05-20 17:16:50

camphor wrote:

Fingers crossed for Slack 13 to support x86_64..

Slackware64 -current made public!

esters · 2009-05-20 18:52:59

camphor wrote:

Fingers crossed for Slack 13 to support x86_64..

Christmas came early this year

Slackware64 -current made public!
[tap tap tap]... Is this thing on? ;-)
Ready or not, Slackware has now gone 64-bit with an official x86_64 port being maintained in-sync with the regular x86 -current branch. DVDs will be available for purchase from the Slackware store when Slackware 13.0 is released. Many thanks go out to the Slackware team for their help with this branch and a special thank you to Eric Hameleers who did the real heavy lifting re-compiling everything for this architecture, testing, re-testing, and staying in-sync with -current.
We've been developing and testing Slackware64 for quite a while. Most of the team is already using Slackware64 on their personal machines, and things are working well enough that it is time to let the community check our work.
We'd like to thank the unofficial 64 bit projects for taking up the slack for us for so long so that we could take our time getting everything just right. Without those alternatives, we would have been pressured to get things out before they were really ready.
As always -- have fun!
Pat and the Slackware crew

- http://slackware.com/

Zoiks - didn't notice the post above

Last edited by esters (2009-05-20 18:53:37)

tomd123 · 2009-05-20 23:28:26

What's going to happen to those distros that were just slackware but recompiled for the x86_64? Notable mentions are slamd64 and bluewhite64.

Arch Linux

#26 2009-05-13 00:56:51

Re: slackware started using xz compression for packages

#27 2009-05-13 01:31:54

Re: slackware started using xz compression for packages

#28 2009-05-13 07:32:45

Re: slackware started using xz compression for packages

#29 2009-05-13 08:44:31

Re: slackware started using xz compression for packages

#30 2009-05-13 09:01:48

Re: slackware started using xz compression for packages

#31 2009-05-13 11:30:36

Re: slackware started using xz compression for packages

#32 2009-05-13 12:40:05

Re: slackware started using xz compression for packages

#33 2009-05-13 17:17:36

Re: slackware started using xz compression for packages

#34 2009-05-13 19:56:59

Re: slackware started using xz compression for packages

#35 2009-05-13 22:07:56

Re: slackware started using xz compression for packages

#36 2009-05-13 23:28:03

Re: slackware started using xz compression for packages

#37 2009-05-14 05:38:00

Re: slackware started using xz compression for packages

#38 2009-05-17 22:50:57

Re: slackware started using xz compression for packages

#39 2009-05-17 23:21:42

Re: slackware started using xz compression for packages

#40 2009-05-17 23:48:18

Re: slackware started using xz compression for packages

#41 2009-05-18 05:26:23

Re: slackware started using xz compression for packages

#42 2009-05-18 15:51:58

Re: slackware started using xz compression for packages

#43 2009-05-20 17:16:50

Re: slackware started using xz compression for packages

#44 2009-05-20 18:52:59

Re: slackware started using xz compression for packages

#45 2009-05-20 23:28:26

Re: slackware started using xz compression for packages

Board footer