You are not logged in.

#1 2009-06-23 11:33:54

Rorschach
Member
From: Ankh-Morpork
Registered: 2008-11-07
Posts: 143

Why did kernel-compression switch to lzma?

Hi,
I've read in the arch-news a few minutes ago that arch took lzma for kernel-compression. But what for? Afaik lzma is just for embedded systems interesting because it can compress smaller than gzip.

But for a normal pc this shouldn't be interesting. gzip is !!much!! faster in compressing and decompressing according to benchmarks (http://tukaani.org/lzma/benchmarks)  and this should be what's interesting for us?

I compiled the 2.6.30 the day it came out and also thought several minutes about what algo to choose, and googled and so on,.. and finally decided that gzip is supirior for a PC.

Do I think false? What is the reason why arch-devs choose lzma?

greetings

Offline

#2 2009-06-23 11:54:46

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: Why did kernel-compression switch to lzma?

You can also ask : why not?
http://www.archlinux.org/pipermail/arch … 11961.html

Some arch devs were not completely in favor of lzma either, but none of them were strongly against it.

The decompression is made only once, at boot, and the difference is probably less than one second. But benchmarks are welcome.
If you can prove a noticeable loss, I am sure this change could be reverted.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#3 2009-06-23 13:07:56

broch
Banned
From: L.A. California
Registered: 2006-11-13
Posts: 975

Re: Why did kernel-compression switch to lzma?

I use gzip also. I believe that whole point of lzma is to use with embedded systems when space is limited (better compression), gzip is fastest option available, so not seeing absolutely any advantage of lzma for me (even though my boot partition with usually 4-5 kernels is 70MB) plus crappy IO (some positive changes are comming to 2.6.31) plus general tendency of kernel to be slower and slower (enough published on the web about this) I don't personally see any reason to add one more factor that will slow down boot process.

Offline

#4 2009-06-23 15:24:25

SyXbiT
Member
From: Seattle, WA
Registered: 2008-06-28
Posts: 177
Website

Re: Why did kernel-compression switch to lzma?

My benchmarks have shown that the difference is less than one second, but I agree. Who cares about saving 5MB of space, at the slight cost of speed. On the desktop, and even on a small SSD, 5MB is nothing.
thanks for the benchmark link. I didn't realize bzip2 was so useless. It's slower and offers worse compression than LZMA.

Last edited by SyXbiT (2009-06-23 15:26:13)

Offline

#5 2009-06-23 15:37:34

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: Why did kernel-compression switch to lzma?

Bzip2 isn't useless - it has been deprecated so to say by LZMA both in terms of speed and compression ratio (most of the time, that is). I even think pacman should switch to LZMA from gzip. I already do it for the packages I build myself and it works like a treat.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#6 2009-06-23 16:29:33

combuster
Member
From: Serbia
Registered: 2008-09-30
Posts: 711
Website

Re: Why did kernel-compression switch to lzma?

Yep, smaller oo-base updates smile

Offline

#7 2009-06-23 17:54:06

broch
Banned
From: L.A. California
Registered: 2006-11-13
Posts: 975

Re: Why did kernel-compression switch to lzma?

well this is limited to kernel only. For pacman using deltas only (I don't know if this is possible) would make sense not slower compression engine that requires more resources.
It seems that changing to lzma is one of these moves that makes not much sense, but because these neither harm (very much) or help, then these can stay. Eventually in a few months users will ask again why this is so slow? One can answer that there was a lot of changes, each slowing down OS a little bit but not much, however the cumulative effect is that eventually system is considerably slower.

Offline

#8 2009-06-23 18:03:32

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: Why did kernel-compression switch to lzma?

LZMA takes more time on the side of the compressor than of the decompressor. With bzip2, both times increased. As someone mentioned here the decompression penalty on LZMA vs gzip is negligible.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#9 2009-06-23 19:36:39

broch
Banned
From: L.A. California
Registered: 2006-11-13
Posts: 975

Re: Why did kernel-compression switch to lzma?

B wrote:

LZMA takes more time on the side of the compressor than of the decompressor. With bzip2, both times increased. As someone mentioned here the decompression penalty on LZMA vs gzip is negligible.

not according to this link (mentioned above already)
http://tukaani.org/lzma/benchmarks

showing that gzip is on average 3-4x times faster:
from the link above:
The Linux kernel 2.6.11.0 source tarball
Compression time
    gzip        bzip2        lzmash       
1     8.3s        1m  9s         0m 45s
Decompression time
    gzip        bzip2        lzmash       
1    2.8s        12.8s        7.7s
Compression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
    gzip        bzip2        lzmash       
1    24        2.9        4.4
Decompression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
    gzip        bzip2        lzmash       
1    71        16        26


by the way bootchard will not show differences between kernels.

at the end it does not matter, default kernel compression is lzma, I use custom kernels anyway.

Offline

#10 2009-07-20 17:10:41

lardon
Member
Registered: 2008-05-31
Posts: 264
Website

Re: Why did kernel-compression switch to lzma?

Using LZMA for kernel compression on Arch is really against KISS principles: it has absolutely no use (who cares about a few MB of disk nowadays), takes noticeably more time on boot and complexifies the boot process.

It's a small matter, but I would hate to see "improvements" like this one multiply and tarnish the wonderful KISS distribution Arch is.


Autojump, the fastest way to navigate your filesystem from the command line!

Offline

#11 2009-07-20 17:19:40

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: Why did kernel-compression switch to lzma?

lardon wrote:

Using LZMA for kernel compression on Arch is really against KISS principles: it has absolutely no use (who cares about a few MB of disk nowadays), takes noticeably more time on boot and complexifies the boot process.

It's a small matter, but I would hate to see "improvements" like this one multiply and tarnish the wonderful KISS distribution Arch is.

This has NOTHING to do with KISS... It's just one kernel option.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#12 2009-07-20 17:24:04

Wintervenom
Member
Registered: 2008-08-20
Posts: 1,011

Re: Why did kernel-compression switch to lzma?

If you really think that it is bad, you can always re-compile...

Last edited by Wintervenom (2009-07-23 03:53:27)

Offline

#13 2009-07-21 01:51:42

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Why did kernel-compression switch to lzma?

I, like most of you, don't care a bout a few MB of disk space, but I DO care about the size of downloads. Not all of us are on always-on, high-speed, unlimited-bandwidth connections, some in countries like mine make do with slower and more congested links, others pay for bandwidth used. Smaller makes a difference.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#14 2009-07-21 01:55:04

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: Why did kernel-compression switch to lzma?

Umm....  look in mkinitcpio.conf

# COMPRESSION
# Use this to compress the initramfs image. With kernels earlier than
# 2.6.30, only gzip is supported, which is also the default. Newer kernels
# support gzip, bzip2 and lzma.
COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"

Default looks like gzip to me.  I am missing something or are people bringing up an old complaint without checking the facts?

Offline

#15 2009-07-21 06:24:46

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: Why did kernel-compression switch to lzma?

Looks like fud:
/boot/kernel26.img: gzip compressed data, from Unix, last modified: Mon Jul 20 17:56:07 2009, max compression
/boot/vmlinuz26: Linux kernel x86 boot executable bzImage, version 2.6.30-ARCH (root@T-POWA-LX) #1, RO-rootFS, root_dev 0x904, swap_dev 0x1, Normal VGA

So the kernel is compressed with bzip2, while the initrd is using gzip. At least bzip2 should be slower than lzma; but I don't think anybody would notice the difference.

Offline

#16 2009-07-21 06:56:31

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: Why did kernel-compression switch to lzma?

bzImage is not bzip...

Offline

#17 2009-07-21 16:21:45

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Why did kernel-compression switch to lzma?

Allan wrote:

Default looks like gzip to me.  I am missing something or are people bringing up an old complaint without checking the facts?

*touches the end of his nose and points at Allan*


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#18 2009-07-21 20:06:37

broch
Banned
From: L.A. California
Registered: 2006-11-13
Posts: 975

Re: Why did kernel-compression switch to lzma?

Default looks like gzip to me.  I am missing something or are people bringing up an old complaint without checking the facts?

hmm:
cat /usr/src/linux-2.6.30-ARCH/.config | grep -i lzma
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_LZMA=y <------ kernel compression mode
CONFIG_RD_LZMA=y <---- RAM Disk compression mode
CONFIG_DECOMPRESS_LZMA=y

and
cat /usr/src/linux-2.6.30-ARCH/.config | grep -i gz
CONFIG_HAVE_KERNEL_GZIP=y
# CONFIG_KERNEL_GZIP is not set
CONFIG_RD_GZIP=y
CONFIG_DECOMPRESS_GZIP=y
and

cat /usr/src/linux-2.6.30-ARCH/.config | grep -i bz
CONFIG_HAVE_KERNEL_BZIP2=y
# CONFIG_KERNEL_BZIP2 is not set
CONFIG_RD_BZIP2=y <---- RAM Disk compression mode
CONFIG_DECOMPRESS_BZIP2=y


so I think that users ask about this line:
CONFIG_KERNEL_LZMA=y <------ kernel compression mode

Last edited by broch (2009-07-21 20:14:51)

Offline

#19 2009-07-23 03:45:23

beroal
Member
From: Ukraine
Registered: 2009-06-07
Posts: 325
Website

Re: Why did kernel-compression switch to lzma?

Also an algorithm of LZMA is not specified. Compare to bzip2 (Burrows-Wheeler transform, Huffman coding...).

Last edited by beroal (2009-07-23 03:47:38)


we are not condemned to write ugly code

Offline

#20 2009-07-23 09:07:00

alecmg
Member
Registered: 2008-12-21
Posts: 86

Re: Why did kernel-compression switch to lzma?

let me coin 2 cents

In modern computers hard drive high latency and low throughput is often a bottleneck. If you have to read 5MB less from harddrive it is a welcome improvement.


Xyne wrote:
"We've got Pacman. Wacka wacka, bitches!"

Offline

#21 2009-07-23 09:15:20

Cdh
Member
Registered: 2009-02-03
Posts: 1,098

Re: Why did kernel-compression switch to lzma?

In the near future you will have SSDs with read rates of 200 mb/s and more. This will be no problem.


฿ 18PRsqbZCrwPUrVnJe1BZvza7bwSDbpxZz

Offline

#22 2009-07-23 10:44:11

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Why did kernel-compression switch to lzma?

That is, assuming the SSDs survive for longer than 2 years of hard usage. The sectors holding .gconf files would die very very fast with their wear characteristics.

We've got to deal with what we have now, and SSDs, except for space-starved netbooks, still don't quite cut it.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#23 2009-07-23 12:09:00

Cdh
Member
Registered: 2009-02-03
Posts: 1,098

Re: Why did kernel-compression switch to lzma?

There is no sector "holding" anything because there is an algorithm to spread the write all over the disk.
http://www.storagesearch.com/ssdmyths-endurance.html


฿ 18PRsqbZCrwPUrVnJe1BZvza7bwSDbpxZz

Offline

#24 2009-07-23 12:25:51

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: Why did kernel-compression switch to lzma?

broch wrote:

Default looks like gzip to me.  I am missing something or are people bringing up an old complaint without checking the facts?

hmm:
cat /usr/src/linux-2.6.30-ARCH/.config | grep -i lzma
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_LZMA=y <------ kernel compression mode
CONFIG_RD_LZMA=y <---- RAM Disk compression mode
CONFIG_DECOMPRESS_LZMA=y
<snip>

so I think that users ask about this line:
CONFIG_KERNEL_LZMA=y <------ kernel compression mode

Seems I was wrong and it does use lzma compression...  sorry.

Offline

#25 2009-07-23 12:31:07

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: Why did kernel-compression switch to lzma?

So kernel is lzma and initrd still gzip?

Offline

Board footer

Powered by FluxBB