tar gzip vs. xz compression

graysky · 2010-06-17 01:12:58

What are folks using these days? I know the official package format got switched over from tar.gz to .tar.xz but unless I'm not using tar correctly, my system backups using xz are taking way longer.

$ tar zcvfp home.tar.gz /home

vs

$ tar cvpJf home.tar.xz /home

Am I omitting an option on the xz format or is it really just that much slower?

falconindy · 2010-06-17 01:35:10

Speed is the tradeoff for higher compression. Apparently, newer releases are making headway in a more efficient algorithm.

sand_man · 2010-06-17 01:41:04

I prefer a smaller download rather than faster compression time.

graysky · 2010-06-17 01:43:48

For smaller dirtrees, it's no biggie but when I run my backup script that does (in part) tar cvpJf arch-system.tar.xz /etc /boot /root /var --exclude "/var/cache/pacman/pkg" and it takes 10x longer than the gz compression only to save about 9 % file size, it doesn't payback for me. I don't want to start a debate of the two formats, I just wanted to make sure that I had the switches right for the xz compression.

Last edited by graysky (2010-06-17 01:44:03)

some-guy94 · 2010-06-17 01:44:42

xz decompression is not supposed to be as 'extreme' as the compression.

IgnorantGuru · 2010-06-17 01:49:00

Another option is --bzip2 in tar, which IIRC is somewhere between gz and xz in compression speed and size.

Also... xz uses multiple threads when run on a multi-core CPU (by default). You can also specify the compression level, which will affect the speed. To specify options to xz you'll probably need to use

tar -cf - files | xz -2 > archive.txz

Personally, I have my system do automated backups in the wee hours of the night, so I don't care if it grinds away for awhile with xz.

Last edited by IgnorantGuru (2010-06-17 01:58:33)

andresp · 2010-06-17 01:54:44

I use gz because just about every system has gzip installed, and bzip is just not useful.

I recently went through the pain of converting an ext4 1tb drive someone else formatted to ext3 because the main netbsd server box couldn't mount it...

Most the time it makes sense to just stay portable.

wonder · 2010-06-17 01:56:16

you are using in the right way. is knew that xz compression take bit longer than gzip.

some-guy94 · 2010-06-17 01:57:08

IgnorantGuru wrote:

Another option is --bzip2 in tar, which IIRC is somewhere between gz and xz in compression speed and size.
Personally, I have my system do automated backups in the wee hours of the night, so I don't care if it grinds away for awhile with xz.

AFAIK, bzip2 is between them in size, but xz actually beats bzip2 is speed.
http://tukaani.org/lzma/benchmarks.html

graysky · 2010-06-17 01:59:51

@wonder - thanks for the feedback. I noticed that neither tar nor xz is multithreaded. By contrast winrar for windows is reported to be multithreaded. I wonder if i/o is the bottle neck or if it's CPU cycles...

EDIT: watching tar go with xz compression, I can speculate that i/o is not the limiting factor. Some times, the system (quad core here) just waits while the CPU compresses a large file using only one core. Perhaps if xz or tar with xz can use all physical cores, we would see a speed increase?

wonder · 2010-06-17 02:01:18

xz is not multicore. if i remember well, the multicore support is on the way

IgnorantGuru · 2010-06-17 02:10:29

some-guy94 wrote:

AFAIK, bzip2 is between them in size, but xz actually beats bzip2 is speed.
http://tukaani.org/lzma/benchmarks.html

Those benchmarks are comparing bzip2 to lzma, but xz is different (aka lzma2). According to the benchmarks below, with default settings xz is quite a bit slower than bzip2.
http://stephane.lesimple.fr/wiki/blog/l … k_reloaded

But I can't say I've tested it. bzip2 is handy because the compression is usually fairly close to xz (compared to gz), but more people have bzip2 installed. XArchiver, for example, doesn't yet support tar.xz.

I recently changed a 1.6G tar.gz tarball to tar.bz2 with a savings of 600MB! I didn't use xz because I'm not sure how widespread it is yet.

IgnorantGuru · 2010-06-17 02:12:13

wonder wrote:

xz is not multicore. if i remember well, the multicore support is on the way

Oh, you're right - I read the man page but missed the comment at the end that says it's not yet implemented.

IgnorantGuru · 2010-06-17 02:40:07

Well now I had to try it. xz is way slower than bzip2, at least in my test.

$ time tar --bzip2 -cf xxx.tar.bz2 xxx

real    0m13.735s
user    0m13.542s
sys    0m0.237s
$ time tar --xz -cf xxx.tar.xz xxx

real    0m55.535s
user    0m55.106s
sys    0m0.387s

Where xxx is a 125MB folder containing a mixture of text file and executables. For the speed difference, bzip2 usually comes pretty close to xz:

bzip2: 125MB -> 75MB
xz: 125MB -> 72MB

Anikom15 · 2010-06-17 04:24:13

I dislike xz, I feel the tradeoff isn't worth it. Faster compression is better than smaller size in my opinion. I have lots of space and a healthy connection, xz seems uneccessary.

But you really shouldn't worry too much about it. Just know that gz and bzip2 have been the de facto and xz hasn't, in fact the repos switched to that format just recently.

Last edited by Anikom15 (2010-06-17 04:24:32)

fredre · 2010-06-17 19:23:59

Although xz compression is way slower than gzip compression, the decompression times are about the same. This makes xz ideal for things like package distribution, as the file is only compressed once but is downloaded and decompressed thousands of times.

However, for backups xz is probably not worth using, as the file is compressed once and usually never decompressed. gzip is imo still the best choice for backing up stuff.

As for bzip, it's slow at compression and decompression, and usually produces file sizes somewhere in-between what you would get from gzip and xz.

Cdh · 2010-06-17 21:26:34

Try building the sage-mathematics-bin packet from AUR on an intel atom. It takes ages to compress the 3.4 gb into the .xz

Last edited by Cdh (2010-06-17 21:26:53)

fredre · 2010-06-19 05:33:13

Cdh wrote:

Try building the sage-mathematics-bin packet from AUR on an intel atom. It takes ages to compress the 3.4 gb into the .xz

makepkg can use gzip by editing PKGEXT in /etc/makepkg.conf, which might be useful for building large packages from AUR for example.

Last edited by fredre (2010-06-19 05:34:56)

Mardoct · 2010-06-19 05:43:46

My favourite thing is not compressing it. When I get somewhere even resembling close to my maximum storage I might care more.

graysky · 2010-06-19 10:36:56

fredre wrote:

makepkg can use gzip by editing PKGEXT in /etc/makepkg.conf, which might be useful for building large packages from AUR for example.

Nice, thanks.

Arch Linux

#1 2010-06-17 01:12:58

tar gzip vs. xz compression

#2 2010-06-17 01:35:10

Re: tar gzip vs. xz compression

#3 2010-06-17 01:41:04

Re: tar gzip vs. xz compression

#4 2010-06-17 01:43:48

Re: tar gzip vs. xz compression

#5 2010-06-17 01:44:42

Re: tar gzip vs. xz compression

#6 2010-06-17 01:49:00

Re: tar gzip vs. xz compression

#7 2010-06-17 01:54:44

Re: tar gzip vs. xz compression

#8 2010-06-17 01:56:16

Re: tar gzip vs. xz compression

#9 2010-06-17 01:57:08

Re: tar gzip vs. xz compression

#10 2010-06-17 01:59:51

Re: tar gzip vs. xz compression

#11 2010-06-17 02:01:18

Re: tar gzip vs. xz compression

#12 2010-06-17 02:10:29

Re: tar gzip vs. xz compression

#13 2010-06-17 02:12:13

Re: tar gzip vs. xz compression

#14 2010-06-17 02:40:07

Re: tar gzip vs. xz compression

#15 2010-06-17 04:24:13

Re: tar gzip vs. xz compression

#16 2010-06-17 19:23:59

Re: tar gzip vs. xz compression

#17 2010-06-17 21:26:34

Re: tar gzip vs. xz compression

#18 2010-06-19 05:33:13

Re: tar gzip vs. xz compression

#19 2010-06-19 05:43:46

Re: tar gzip vs. xz compression

#20 2010-06-19 10:36:56

Re: tar gzip vs. xz compression

Board footer