You are not logged in.

#1 2010-09-26 15:36:27

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Compressing initpcio image with LZMA fails on system with 256 MB RAM

I am running Arch on my AppleTV, and the little monster only has 256 MB RAM (soldered on). I just upgraded to the latest LTS kernel and got this happy message:

(1/1) upgrading kernel26-lts                                                                           [############################################################] 100%
>>> Updating module dependencies. Please wait ...
>>> MKINITCPIO SETUP
>>> ----------------
>>> If you use LVM2, Encrypted root or software RAID,
>>> Ensure you enable support in /etc/mkinitcpio.conf .
>>> More information about mkinitcpio setup can be found here:
>>> http://wiki.archlinux.org/index.php/Mkinitcpio

>>> Generating initial ramdisk, using mkinitcpio.  Please wait...
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.32-lts -c /etc/mkinitcpio.conf -g /boot/kernel26-lts.img
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [autodetect]
:: Parsing hook [keymap]
:: Parsing hook [pata]
:: Parsing hook [filesystems]
:: Generating module dependencies
lzma: (stdin): Onvoldoende geheugen beschikbaar
:: Generating image '/boot/kernel26-lts.img'...FAILED
==> FAIL
==> Building image "fallback"
==> Running command: /sbin/mkinitcpio -k 2.6.32-lts -c /etc/mkinitcpio.conf -g /boot/kernel26-lts-fallback.img -S autodetect
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [keymap]
:: Parsing hook [pata]
:: Parsing hook [filesystems]
:: Generating module dependencies
lzma: (stdin): Onvoldoende geheugen beschikbaar
:: Generating image '/boot/kernel26-lts-fallback.img'...FAILED
==> FAIL

Which translates to 'not enough memory available'.

After investigation it appears that /sbin/mkinitcpio is hardcoded to use the maximum compression level:

# grep COMPRESSION /sbin/mkinitcpio 
COMPRESSION="gzip"
    if ! /sbin/gen_init_cpio ${FILELIST} | ${COMPRESSION} -9 > "${GENIMG}"; then

After changing the compression level to -6 it builds fine. 256 MB of RAM isn't plenty but I can imagine plenty of headless setups will be running that, so isn't -9 a bit of an overly optimistic value? You clearly can't use that with 256 MB of RAM - even in init 3 with only 33 MB RAM in use here, xz will still complain about not having enough RAM available.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#2 2010-09-26 16:04:09

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

[karol@black ~]$ tail -3 /etc/mkinitcpio.conf 
COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"

I'm confused, are you talking about gzip or lzma? IIRC 'gzip -9' doesn't really buy you much, and -6 or -7 is the default value, but many distros use -9 <shrugs>

Offline

#3 2010-09-26 16:20:11

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

I'm talking about Lzma. The confusing thing is, in /sbin/mkinitcpio, you'll see COMPRESSION = gzip, but it seems to be overridden by the COMPRESSION setting in /etc/mkinitpcio.conf. As you can see from the mkinitpcio output, it calls lzma, so it's lzma (I have lzma enabled).

I checked with the xz author, he said using -9 is 'plain madness' for lzma, for an initrd at least.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#4 2010-09-26 16:29:46

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

Open a bug report. mkinitcpio shouldn't set a compression level.

Offline

#5 2010-09-26 16:33:28

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

I've just upgraded and rebuild the kernel, the memory usage was a couple MB with gzip compression.

Offline

#6 2010-09-26 16:34:11

djgera
Developer
From: Buenos Aires - Argentina
Registered: 2008-12-24
Posts: 723
Website

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

I sent an email to the author of xz utils about this memory issue, the answer was:

On 2010-09-24 Gerardo Exequiel Pozzi wrote:
> > I recently encountered an issue with this beta (also with latest
> > 180-ge23e fails). Compressing any file, also null data, with -9 on
> > systems with below ~1G of RAM fails.
Short answer: Older xz versions have a memory usage limiter that is
enabled by default. Newer version still have it, but it is disabled by
default. You can get close to the old behavior by putting

    XZ_DEFAULTS=--memlimit=40%
    export XZ_DEFAULTS

into your shell's initialization script (e.g. .bashrc).

Long answer: With older xz versions, the default memory usage limit is
calculated from the total about of installed RAM, so it is lower on
systems with less RAM. See the section "Memory usage" on xz 143-g3e49
man page for details.

For example, this message suggest that you ran it on a system with 256
MiB RAM (40 % of slightly less than 256 MiB is 99 MiB):

> > # cat /dev/null | lzma -9 > nulo
> > lzma: Adjusted LZMA1 dictionary size from 64 MiB to 8 MiB to not
> > exceed the memory usage limit of 99 MiB
The 64 MiB dictionary from the preset -9 would need around 674 MiB of
memory, so it is scaled down by adjusting the dictionary size. With 8
MiB dictionary, it needs around 94 MiB of memory, and thus is below the
limit of 99 MiB. 8 MiB dictionary happens to match the preset -6, but it
could have been adjusted to something that doesn't match any preset. You
can pass -vv (--verbose --verbose) to xz to see the memory usage info
when compressing.

Reducing the dictionary size means that if you are compressing files
bigger than the dictionary size, the compression ratio is likely to
become worse. Personally I think it's OK, because using swap would be
too slow or, if there's not enough swap, can fail completely like in
your case with a newer xz. But some people have different opinions.

Newer versions have the limiter disabled by default, because I got
several complaints about the default memory usage limit. It caused
problems e.g. for Gentoo and FreeBSD ports. On older systems, the
default memory usage limit become too low to decompress files created
with "xz -9" (65 MiB). When decompressing, there's no way to scale
things down; the memory usage of the decompressor is determined by the
settings used to compress the file.

So xz refused to decompress some source packages unless the memory usage
limiter was disabled with a command line option. Refusing to decompress
prevented compiling new packages, so it was obviously a bad thing. Many
people thought that in general xz should try decompression as long as
memory allocation succeeds, no matter how slow it might be due to heavy
swapping. In other words, disabling the limiter only in package
management scripts wasn't seen good enough; it should be disabled by
default at least for decompression.

I knew that I cannot make everyone happy at the same time, because some
people like the old behavior. I added the XZ_DEFAULTS environment
variable so that people who want a limiter can enable it relatively
easily. However, I'm not sure if this is good enough.

Some people, who were against having a default limit, said that it is
important that xz is predictable. That is, it should do what the user
tells instead of e.g. scaling down the settings to avoid swapping. This
is why the default limit was removed also from compression. I'm not sure
if that was the best decision. Maybe it would have been better to make
the limit for compression e.g. 95 % of RAM, but that too can cause
trouble in some cases.

Do you think setting a limit via XZ_DEFAULTS is good enough for you and
other users in similar situations? I think it's not as good as the old
way, but for me it is still good enough (I have only only one computer
with over 512 MiB RAM; this one has 8 GiB). Note that it is nowadays
possible to set the limit separately for compression and decompression.
See the options --memlimit-compress and --memlimit-decompress on recent
xz man page.

It's not too hard to make xz scale down the settings if it runs out of
memory before no data has been compressed. A harder thing is to decide
how much it should scale it down and if that is a good thing to do by
default. E.g. if it scales it down only so much that memory allocation
succeeds, it may end up using a lot of swap and be far too slow for most
uses.

Finally, it should be noted that too many people tend to use -9 blindly
with xz and also lzma, because they are so used to -9 with gzip and
bzip2. With xz, especially the presets -7, -8, and -9 need a lot of
memory and are useful only for files bigger than 8, 16, and 32 MiB,
respectively. xz can be usable even on systems with only 16 MiB RAM, but
it requires avoiding the high settings.

The --help text in a recent xz warns about using -7 ... -9 without a
need. I have a major man page update coming, and that will include more
memory usage information and description why the highest settings aren't
necessarily what people want in a typical situation. So I hope this
issue will become smaller in the future.

-- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode

Offline

#7 2010-09-26 16:39:49

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

So the issue is present only in the new version of xz, right?

Offline

#8 2010-09-26 16:42:49

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: Compressing initpcio image with LZMA fails on system with 256 MB RAM

In newer versions, yes. Lasse said he could put the limiter back in, but then it would decompress archives that had been compressed with -9 on low memory systems (which is another problem). Chicken and egg, really.

Pierre: I'll do so. You can find the report here.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

Board footer

Powered by FluxBB