mkinitcpio 16 + LZ4 working?

Pse · 2013-12-19 19:47:52

Hi, I just noticed that mkinitcpio 16 supports LZ4. I enabled it and figured it would work. Unfortunately I was left with an unbooting system. I rebuilt the images using a live CD and went back to lzop. I checked the kernel config and it seems LZ4 is enabled. Has anyone else run into problems with this?

graysky · 2013-12-19 20:04:55

Is this needed first?
https://bugs.archlinux.org/task/38128

EDIT: no, I don't think so. Not sure why it's not working... unrelated: if you have the space on /boot, I suggest you use 'cat' compression which is none and the fastest option.

Last edited by graysky (2013-12-19 20:12:59)

Pse · 2013-12-19 20:47:08

graysky wrote:

Is this needed first?
https://bugs.archlinux.org/task/38128
EDIT: no, I don't think so. Not sure why it's not working... unrelated: if you have the space on /boot, I suggest you use 'cat' compression which is none and the fastest option.

Everything needed appears to be set (stock stable kernel: 3.12.5-1-ARCH):

CONFIG_HAVE_KERNEL_LZ4=y
# CONFIG_KERNEL_LZ4 is not set
CONFIG_RD_LZ4=y
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m
CONFIG_LZ4_COMPRESS=m
CONFIG_LZ4HC_COMPRESS=m
CONFIG_LZ4_DECOMPRESS=y
CONFIG_DECOMPRESS_LZ4=y

karol · 2013-12-19 22:28:54

I got a nice juicy kernel panic too.

https://mailman.archlinux.org/pipermail … 01622.html
It would be cool to get a similar comparison with the addition of 'cat' and 'lz4' on a more modern hardware. graysky? :-)

graysky · 2013-12-19 23:53:52

@k - I have been using cat for a while now; honestly, I can't notice a difference vs gzip or lzo on my system (4.4 GHz i7-3770k). Again, since disk space is not an issue, why compress the images at all?

% df -h | grep boot
/dev/sda2       190M   32M  145M  18% /boot

kit · 2013-12-20 07:05:46

Sorry I don't know how I missed this before posting to the mailing list. (https://mailman.archlinux.org/pipermail … 04044.html)
Dave Reisner (who commited lz4 to mkinitcpio) just replied to the mailing list. (https://mailman.archlinux.org/pipermail … 04045.html)

I was comparing the filenames mentioned in a thread on the linux kernel mailing list where the lz4 decompression patches were submitted (https://lkml.org/lkml/2013/2/26/31) to the file list for the linux kernel on the arch package database (https://www.archlinux.org/packages/core … nux/files/) and noticed there was no lz4-decompress.ko when I fell asleep. Not sure if it should be there, or if it will help. From what Pse says I guess not, I hadn't checked my kernel config. Someone since commented on a post I made to the Arch Facebook group stating the kernel handles lz4 fine with zswap too (https://www.facebook.com/groups/2204843 … up_comment). I was also reading a thread on the linux kernel mailing list where patches for support of lz4-compressed kernels were submitted (http://www.gossamer-threads.com/lists/l … el/1667680)

Just had 3 hours sleep though and hope to have another hour before a busy day today so sorry I'm not much help. Dave Reisner suggests reporting upstream to LZ4 at the end of his message on the mailing list, so if anyone does that or finds an upstream bug before I do, please link to it here and maybe in response to the thread I started on the Arch mailing list too. Thank you.

P.s. The reason I compress the ramdisk image is that it is potentially quicker to read, extract and execute a 7.1MB file than it is to read and execute a 14MB file, i.e. it reduces the blocking factor of the disk when you have a fast processor. LZ4 decompresses potentially 66%-150% faster than LZO according to the thread on lkml re: lz4 decompression patches. At the moment my ramdisk image is saved on a HDD with platters so not as fast as an SSD, but I thought even if I upgraded to an SSD I'd still be reducing a write each kernel upgrade by about 50% so then it would help prolong the life of the drive. Not by much but I like to optimise.

Last edited by kit (2013-12-20 08:15:55)

kit · 2013-12-20 10:21:05

Some more pointers...

Could be a related bug that's already been reported to LZ4: http://code.google.com/p/lz4/issues/detail?id=66 - Closed by the LZ4 developers as it was deemed a kernel issue.

The most recent changes to lz4 in linux kernel: https://github.com/torvalds/linux/tree/master/lib (all recent LZ4 changes appear to already be in current Arch kernel)

Last edited by kit (2013-12-20 10:21:33)

falconindy · 2013-12-20 13:55:38

kit wrote:

Some more pointers...
Could be a related bug that's already been reported to LZ4: http://code.google.com/p/lz4/issues/detail?id=66 - Closed by the LZ4 developers as it was deemed a kernel issue.

Sure, but upstream also mentions that they changed their on disk format, and that the legacy format (a mythical -l flag which no longer exists in lz4c and never existed in lz4) is needed to be used in order for the kernel to be able to decompress the archive.

There's 2 possible fixes:

1) upstream lz4 readds some way to generate the original format (how the ?@#$ do you just get rid of this?)
2) the kernel adopts the newer lz4 stream format.

Pse · 2013-12-20 18:01:17

falconindy wrote:

1) upstream lz4 readds some way to generate the original format (how the ?@#$ do you just get rid of this?)
2) the kernel adopts the newer lz4 stream format.

I figured it would be something of the sort. If this serves as any indication, I'd say it's not worth it to switch to a compression format that can change any other day and leave you with an unbootable system after a kernel update.

Thanks everybody.

Last edited by Pse (2013-12-20 18:04:14)

farwayer · 2013-12-21 04:57:48

It will be nice to remove lz4 compression options from mkinitcpio.conf. This is misleading users that it's work.

WonderWoofy · 2013-12-21 06:08:15

farwayer wrote:

It will be nice to remove lz4 compression options from mkinitcpio.conf. This is misleading users that it's work.

Now this is something that should definitely be reported as a bug in Arch's bugtracker.

karol · 2013-12-21 12:24:35

There already is https://bugs.archlinux.org/task/38205
Maybe you just reopen it or ask on the ML if Dave considers it a valid temporary workaround.

vorbote · 2013-12-21 13:58:27

And someone has already opened a bug report upstream[1]. Star the issue if you want to vote for a fix.

(I believe that I don't have to remind people that ranting in a bug tracker makes the developers angry and the possibility of a fix will surely be reduced dramatically or disappear altogether. Same goes for brainless "me too" comments.)

[1] https://code.google.com/p/lz4/issues/detail?id=102

Last edited by vorbote (2013-12-21 14:00:48)

kit · 2013-12-22 12:23:12

The version of lz4 (v109-2) I installed from the arch repo has an undocumented legacy option and the resulting initramfs produces a warning but doesn't cause a kernel panic.

$ lz4 -l initramfs-linux-uncompressed.img.raw initramfs-linux-lz4l.img
! Generating compressed LZ4 using Legacy format (deprecated !) !

Last edited by kit (2013-12-22 12:25:17)

Thorsten Reinbold · 2013-12-22 14:22:50

kit wrote:

The version of lz4 (v109-2) I installed from the arch repo has an undocumented legacy option and the resulting initramfs produces a warning but doesn't cause a kernel panic.
$ lz4 -l initramfs-linux-uncompressed.img.raw initramfs-linux-lz4l.img
! Generating compressed LZ4 using Legacy format (deprecated !) !

You're right! When adding -l to the additional compression options in mkinitcpio.conf, it works. Nice to know.

serdotlinecho · 2013-12-22 14:58:46

@kit and @Thorsten Reinbold, thanks for the awesome info! But I got this when i try to run:

~ » uname -a
Linux gnusmas 3.13.0-1-mainline #1 SMP PREEMPT Fri Dec 20 13:12:35 MYT 2013 i686 GNU/Linux
~ » lsinitcpio -a /boot/initramfs-linux-mainline.img
==> ERROR: Unexpected file type. Are you sure /boot/initramfs-linux-mainline.img is an initramfs?

Anyone have similar lsinitcpio error when using lz4?

vorbote · 2013-12-22 15:06:45

serdotlinecho wrote:

@kit and @Thorsten Reinbold, thanks for the awesome info! But I got this when i try to run:
~ » uname -a
Linux gnusmas 3.13.0-1-mainline #1 SMP PREEMPT Fri Dec 20 13:12:35 MYT 2013 i686 GNU/Linux
~ » lsinitcpio -a /boot/initramfs-linux-mainline.img
==> ERROR: Unexpected file type. Are you sure /boot/initramfs-linux-mainline.img is an initramfs?
Anyone have similar lsinitcpio error when using lz4?

lsinitcpio is a shell script. Patch it and it will work.

vorbote · 2013-12-22 15:17:04

OK. I have added a new task in Flyspray so that this new information is not lost. https://bugs.archlinux.org/task/38229

serdotlinecho · 2013-12-23 03:02:50

vorbote wrote:

serdotlinecho wrote:
@kit and @Thorsten Reinbold, thanks for the awesome info! But I got this when i try to run:
~ » uname -a
Linux gnusmas 3.13.0-1-mainline #1 SMP PREEMPT Fri Dec 20 13:12:35 MYT 2013 i686 GNU/Linux
~ » lsinitcpio -a /boot/initramfs-linux-mainline.img
==> ERROR: Unexpected file type. Are you sure /boot/initramfs-linux-mainline.img is an initramfs?
Anyone have similar lsinitcpio error when using lz4?
lsinitcpio is a shell script. Patch it and it will work.

I just installed mkinitcpio 16-2 from testing. All good now.

honza · 2013-12-31 04:17:29

serdotlinecho wrote:

I just installed mkinitcpio 16-2 from testing. All good now.

I just tried using lz4 compression with the new mkinicpio 16-2 and lz4 and it appears to have worked --just in case anyone was still following this thread or issue.

$ lsinicpio -a /boot/initramfs-linux.img
==> Image: /boot/initramfs-linux.img 
==> Created with mkinitcpio 16
==> Kernel: 3.12.6-1-ARCH
==> Size: 7.06 MiB
==> Compressed with: lz4 -l
  -> Uncompressed size: 13.32 MiB (.530 ratio)
  -> Estimated extraction time: 0.017s

==> Included modules:
  ahci [explicit]	  ehci-hcd		  libcrc32c		  usb-common
  btrfs			  ehci-pci		  mxm-wmi		  usbcore
  button		  hid			  nouveau [explicit]	  usbhid
  cdrom			  hid-logitech-dj	  raid6_pq		  usb-storage
  crc32c		  i2c-algo-bit		  scsi_mod		  video
  crc32c-intel		  i2c-core		  sd_mod		  wmi
  drm			  libahci		  sr_mod		  xor
  drm_kms_helper	  libata		  ttm			  zram [explicit]

==> Included binaries:
  btrfs			  mkdir			  systemctl
  btrfsck		  modprobe		  systemd-tmpfiles
  kmod			  mount			  udevadm

==> Hook run order:
  btrfs

WonderWoofy · 2013-12-31 05:06:40

Just so you know (and this has nothing to do with the topic), the btrfs hook is intended for an initramfs that does not have either udev or systemd. If you use either of those in your initramfs, you don't need the btrfs hook. Additionally, it is intended for use with multi-device filesystems, so if your btrfs is only on one disk or partition, then it is really really not needed.

honza · 2014-01-01 08:21:38

WonderWoofy wrote:

...if your btrfs is only on one disk or partition, then it is really really not needed.

If I have multiple BTRFS partitions, however, I'll need this hook?

falconindy · 2014-01-01 17:20:11

honza wrote:

WonderWoofy wrote:
...if your btrfs is only on one disk or partition, then it is really really not needed.
If I have multiple BTRFS partitions, however, I'll need this hook?

If you use udev/systemd in your initramfs, you do not need the btrfs hook. udev takes care of the ioctl to assemble the devices on its own.

WonderWoofy · 2014-01-01 20:58:03

In any case, it depends on what those multiple btrfs partitions are on. The btrfs hook is meant for btrfs partitions that span more than one block device. It basically just calls 'btrfs device scan' which assembles the filesystem. But as falconindy has already pointed out, as long as you have udev or systemd in your initramfs, it is not necessary in any case with mkinitcpio.

honza · 2014-01-02 13:37:41

@falconindy and @WonderWoofy, thanks guys.

Arch Linux

#1 2013-12-19 19:47:52

mkinitcpio 16 + LZ4 working?

#2 2013-12-19 20:04:55

Re: mkinitcpio 16 + LZ4 working?

#3 2013-12-19 20:47:08

Re: mkinitcpio 16 + LZ4 working?

#4 2013-12-19 22:28:54

Re: mkinitcpio 16 + LZ4 working?

#5 2013-12-19 23:53:52

Re: mkinitcpio 16 + LZ4 working?

#6 2013-12-20 07:05:46

Re: mkinitcpio 16 + LZ4 working?

#7 2013-12-20 10:21:05

Re: mkinitcpio 16 + LZ4 working?

#8 2013-12-20 13:55:38

Re: mkinitcpio 16 + LZ4 working?

#9 2013-12-20 18:01:17

Re: mkinitcpio 16 + LZ4 working?

#10 2013-12-21 04:57:48

Re: mkinitcpio 16 + LZ4 working?

#11 2013-12-21 06:08:15

Re: mkinitcpio 16 + LZ4 working?

#12 2013-12-21 12:24:35

Re: mkinitcpio 16 + LZ4 working?

#13 2013-12-21 13:58:27

Re: mkinitcpio 16 + LZ4 working?

#14 2013-12-22 12:23:12

Re: mkinitcpio 16 + LZ4 working?

#15 2013-12-22 14:22:50

Re: mkinitcpio 16 + LZ4 working?

#16 2013-12-22 14:58:46

Re: mkinitcpio 16 + LZ4 working?

#17 2013-12-22 15:06:45

Re: mkinitcpio 16 + LZ4 working?

#18 2013-12-22 15:17:04

Re: mkinitcpio 16 + LZ4 working?

#19 2013-12-23 03:02:50

Re: mkinitcpio 16 + LZ4 working?

#20 2013-12-31 04:17:29

Re: mkinitcpio 16 + LZ4 working?

#21 2013-12-31 05:06:40

Re: mkinitcpio 16 + LZ4 working?

#22 2014-01-01 08:21:38

Re: mkinitcpio 16 + LZ4 working?

#23 2014-01-01 17:20:11

Re: mkinitcpio 16 + LZ4 working?

#24 2014-01-01 20:58:03

Re: mkinitcpio 16 + LZ4 working?

#25 2014-01-02 13:37:41

Re: mkinitcpio 16 + LZ4 working?

Board footer