compress files in bz2

Moritz · 2003-07-10 15:31:43

Hey guys!

Just a question:
Why are Arch-ISOs not compressed? That would save a lot of bandwith! And I think everyone knows how to un-bzip2 a file. (even winrar can do that!)
Just have a look at the amlug-live-cd-is: 580MB -> 190MB
(I'm not sure about the real sizes, because amlug.org seems to be down. But that's what I remember!)

Nearly the same question:
Why are packages in gzip and not in bzip2? Is there a reason? I could imagine that bzip2 needs more cpu while uncompressing, but on a i686-distro, this shouldn't be a problem...

Regards,
Moritz

apeiro · 2003-07-10 23:30:33

ISO images are pretty tight as it is. Compression doesn't buy us a lot.

[judd@dragon judd]$ ls -l arch-0.5beta.iso arch-0.5beta.iso.bz2 
-rw-r--r--    1 judd     judd     652869632 Jul  8 12:58 arch-0.5beta.iso
-rw-r--r--    1 judd     judd     651487313 Jul 10 19:00 arch-0.5beta.iso.bz2

As for package compression, we could look at moving to bz2 some day.

Moritz · 2003-07-11 07:22:10

okay, I see...

how do AMLUG get their life-cd that small? :?:

hApy · 2003-07-11 09:36:12

my guess is the difference is that the live cd has all the packages extracted on it (ready for use) whereas the arch cd is mostly archived packages already so bzipping it will not do as well.

Hapy.

Moritz · 2003-07-11 10:30:09

argh... that's true... :oops:

Mork II · 2003-07-12 12:04:04

How much better is bz2 than gz compresserion? I'm sitting at a windows box so I can't try it out. What I do remeber is bz2 taking a lot more time and resources to uncompress (even on my p4)..

If the problem is bandwidth on ftp.archlinux.org other solutions should be considered:
- Pacman could be made to to randomly choose which dowload mirror to use (I think it currently works it way down the list and uses the first mirror that works, and that is ftp.archlinux.org. Correct me if I'm wrong).
- Someone suggested making pacman download simultaniosly from several mirrors, I don't know how hard that is to accomplish.
- For iso downloads bittorrent could help.

dariball · 2003-07-12 12:28:30

ok, i made some tests:

going to compress my /usr/abs directory :

du -k /usr/abs tells me that it is 121.296 kbytes big.
compressing it with tar czf it outputs a file with the size : 32.035 kbytes
compressing it with tar cjf it outputs a file with the size : 27.034 kbytes

while bz2 took way longer than gz on my athlon 800 with 256MBs ....
decompressing those file with the following times :

TAR_GZ
real 14.610s
user 3.250s
sys 3.420s

TAR_BZ2
real 47.668s
user 29.320s
sys 3.260s

this shows to me that bz2 needs much longer than gz for a minimum effectivity win ....
greetz
dariball

Gyroplast · 2003-07-12 12:30:25

[pacman choosing random mirror]

Basically a good idea, but then you'd need to implement a priority mechanism to allow people to define which servers rather to get packages from than others. A simple scheme as in MX DNS entries for example would serve this purpose (decimal number, lowest tried first, if values are identical, choose a random one among those who are equal) well enough. For backwards compatibility one may handle a missing priority value as "0", and this value could be appended the mirror URL, seperated by whitespace.

Wrong forum, huh? Move to pacman?

Greets,
Dennis

xirus · 2003-07-12 17:18:21

please don't move to bz2, the difference with gz isn't that much, and it takes ages longer to complete!

dariball · 2003-07-12 19:43:14

yes this is my opinion

Moritz · 2003-07-13 02:39:25

Mork II wrote:

- Someone suggested making pacman download simultaniosly from several mirrors, I don't know how hard that is to accomplish.

I wrote this some tome in another thread, but i did not get any answer. But I would prefer this way!

Mork II · 2003-07-13 12:00:56

Sorry for the thread drift.

Moritz: I like your suggestion of simultaenous downloads. Both apt-get and urpmi do it that way so there should be some merit to it (speed for one). If this can be done I too prefer it over the random mirror stuff..

gyroplast wrote:

Basically a good idea, but then you'd need to implement a priority mechanism to allow people to define which servers rather to get packages from than others. A simple scheme as in MX DNS entries for example would serve this purpose (decimal number, lowest tried first, if values are identical, choose a random one among those who are equal) well enough. For backwards compatibility one may handle a missing priority value as "0", and this value could be appended the mirror URL, seperated by whitespace.

I didn't think about that. I can't program but the user aspect of the thing is simple enough to understand. A newbie wouldn't even have to think about it (unless debian-style mirror selection based on geografical location is implemented in the installer, but that is hardly a concern yet).

ubermartian · 2004-02-19 03:08:20

please don't move to bz2, the difference with gz isn't that much, and it takes ages longer to complete!

For small packages the size difference isn't a lot, but when you start getting into MB's then the difference becomes more obvious. For dial-up users bz2 would provide more efficiency in the long run.

Bobonov · 2004-02-19 08:11:52

This discussion was done some time ago.
More or less it is going in the same direction, the gain in size is not enogth compared to the increase in the decompression.

Daribal wrote,

du -k /usr/abs tells me that it is 121.296 kbytes big.
compressing it with tar czf it outputs a file with the size : 32.035 kbytes
compressing it with tar cjf it outputs a file with the size : 27.034 kbytes
TAR_GZ
real 14.610s
user 3.250s
sys 3.420s
TAR_BZ2
real 47.668s
user 29.320s
sys 3.260s

Yes on a little file the gain is little and it seem that it is not an interesting solution, especially if you consider the diffrence in time terms.
But you have to consider the percentage and not the difference.
In the past I made some extensive test, sorry I do not have the result anymore but if you search on the web for sure there are other comparision test from other people.
From my test done on file of various size from 500KB up 100MB I saw that the compression difference was from a 5% up to 15% with a medium compresion more or less around 11.5%.
This mean that if you download openoffice pkg you can save around 8/9 MB probably. If you have openoffice for sure you install java (2/3 MB, may this is not the best example since java has several jar file that should be already compressed) and Xfree (5/6 MB).
So hypothesise you save around 15MB. If you ave a good connecton, you can download at 100Mbit (and I belive that most of the time you can't download at this speed) this mean that is going to be more or less 20 minutes less than gz package.
About decompression time:
Yes it is true that bz2 take ages compared to gz, but from my test emerged that difference in % became more litte as the size of the file grown. Another think against bz2 is that it take much more memory for decompression.
Now considering all the above we have to decide if decompressing xfree openoffice and java as bz2 take more than 20 minute + gz decompression time or not.

Bobonov · 2004-02-19 22:11:56

In my previous post I meant 100Kbit (little less than the double of the speed of a 56K modem) and not 100Mbit that is a network connection :oops:

zeppelin · 2004-03-09 18:22:25

IMO arch is a modern distro and should leave GZIP behind. Bzip2 does a better job, so Arch should use it

m33sb3w · 2004-03-09 18:59:00

Mork II wrote:

- For iso downloads bittorrent could help.

I for one would be willing to seed.

terrapin · 2004-03-09 19:37:10

The linux mirror project looks interesting but they are still VERY new. I wrote the admins an email to find out how distros are added but haven't receieved a reply yet. any woo here's the link http://www.tlm-project.org/

Arch Linux

#1 2003-07-10 15:31:43

compress files in bz2

#2 2003-07-10 23:30:33

Re: compress files in bz2

#3 2003-07-11 07:22:10

Re: compress files in bz2

#4 2003-07-11 09:36:12

Re: compress files in bz2

#5 2003-07-11 10:30:09

Re: compress files in bz2

#6 2003-07-12 12:04:04

Re: compress files in bz2

#7 2003-07-12 12:28:30

Re: compress files in bz2

#8 2003-07-12 12:30:25

Re: compress files in bz2

#9 2003-07-12 17:18:21

Re: compress files in bz2

#10 2003-07-12 19:43:14

Re: compress files in bz2

#11 2003-07-13 02:39:25

Re: compress files in bz2

#12 2003-07-13 12:00:56

Re: compress files in bz2

#13 2004-02-19 03:08:20

Re: compress files in bz2

#14 2004-02-19 08:11:52

Re: compress files in bz2

#15 2004-02-19 22:11:56

Re: compress files in bz2

#16 2004-03-09 18:22:25

Re: compress files in bz2

#17 2004-03-09 18:59:00

Re: compress files in bz2

#18 2004-03-09 19:37:10

Re: compress files in bz2

Board footer