geninit - fast and familiar initramfs creation

falconindy · 2011-05-23 02:10:23

geninit has has a similar look and feel to mkinitcpio, but its anywhere between 4 and 10 times faster on image creation. I wanted to keep it similar to make any possible transition smoother, but I've, of course, taken a few liberties...

the "install" files are called builders. I thought this was an important distinction to make as mkinitcpio blurs the line between what a hook and an install is. Hooks are added in the config, but installs are parsed, and hooks are actually side effects of installs. Builders are added in geninit, builders are parsed, and you might have hooks added.
I absolutely can't stand that our initramfs has the word "kernel" in it. A lot of new folks are overly confused by this. The default images created by geninit are 'initramfs-ARCH' and 'initramfs-ARCH-failsafe'.
kernel identification is done, preferrably, via filename instead of version (though the latter is supported). This prevents hacky regeneration of a preset file in the kernel package. You point to a kernel filename such as /boot/vmlinuz26, and geninit determines the kernel version for you.
better support for building from a liveCD. mkinitcpio is somewhat poor on this front, and you're often better off just doing the chroot. Creating an imageset with mkinitcpio's -b flag will leave the images on the real root, rather than placing them in /boot relative to the basedir. This is never mentioned to the user.
support for LZO compression via extra/lzop. LZO, in my testing, is ideal for initramfs's as it provides comparable (if not better) compression than any other supported method, and it's leaps and bounds faster on decompression.
support for virtio. This happened intrinsicly, for perhaps obvious reasons.
a clearly defined API for builders and hooks. This is all documented in the man page and in code.
Use. Proper. Bash. This isn't where geninit derives it real speed from, but it certainly doesn't hurt to write safe and readable code that follows Bash best practices.

And I suppose equally as important, what is geninit (currently) missing to make it feature complete with mkinitcpio?

non-udev based raid assembly. For most people, I suspect this is a non-issue.
verified support for keyfile based root encryption. Simple passphrase unlocking works.
support for custom mount handlers. This means things like the btrfs-advanced hook in the AUR isn't possible just yet. This also precludes NFS roots.

I'm definitely interested in adding the custom mount handler support ASAP. It's going to take me a little longer to get to the crypt stuff, as I'm less familiar with it.

So, how fast is it?

# time geninit -p kernel26
==> Building image from preset: kernel26-default
  -> -k /boot/vmlinuz26 -c /etc/geninit.conf -g /boot/initramfs-ARCH 
==> Starting build: 2.6.39-ARCH
  -> Building: [base]
  -> Building: [udev]
  -> Building: [autodetect]
  -> Building: [ata]
  -> Building: [usbinput]
  -> Building: [filesystems]
==> Generating module dependencies
==> Creating lzop initramfs: /boot/initramfs-ARCH
6856 blocks
==> Image creation completed successfully

==> Building image from preset: kernel26-failsafe
  -> -k /boot/vmlinuz26 -c /etc/geninit.conf -g /boot/initramfs-ARCH-failsafe -S autodetect
==> Starting build: 2.6.39-ARCH
  -> Building: [base]
  -> Building: [udev]
  -> Building: [ata]
  -> Building: [usbinput]
  -> Building: [filesystems]
==> Generating module dependencies
==> Creating lzop initramfs: /boot/initramfs-ARCH-failsafe
15619 blocks
==> Image creation completed successfully


real    0m3.882s
user    0m2.670s
sys     0m0.399s

This is on an SSD, but my QEMU VMs get similar times with their disks sitting on rotational drives (~6s for an lvm+raid image pair). For a straight comparison, 4 seconds is roughly how long it takes mkinitcpio on my SSD to create only the autodetect image.

geninit also comes with a fun little tool called lsinitramfs which can list or extract the contents of an initramfs, as well as analyze the contents and quickly give you an overview of the important pieces inside. For example, here's an image that one of my VMs (mdadm + lvm root) uses:

$ lsinitramfs -a /boot/initramfs-ARCH
==> Image: /boot/initramfs-ARCH
==> Kernel: 2.6.39-ARCH
==> Compressed with: lzop
  -> Compression ratio: .520
  -> Estimated decompression time: 0.022s

==> Included modules:
  dm-log
  raid0
  dm-mirror
  dm-snapshot
  dm-region-hash
  dm-mod
  md-mod
  virtio
  virtio_ring
  virtio_pci
  virtio_blk
  jbd2
  mbcache
  ext4 [explicit]
  crc16

==> Included binaries:
  /sbin/udevadm
  /sbin/blkid
  /sbin/lvm
  /sbin/mdadm
  /sbin/udevd
  /sbin/modprobe
  /sbin/dmsetup
  /bin/busybox

==> Hook run order:
  lvm2

If you're still reading this, you might want to know how to try this out. Download, build, install (see below links), and...

# cp /etc/geninit.d/example.preset /etc/geninit.d/kernel26.preset
# $EDITOR /etc/geninit.conf
# geninit -p kernel26

Update your bootloader and you're set.

Where to get it?
On the AUR: http://aur.archlinux.org/packages.php?ID=48542
On Github: https://github.com/falconindy/geninit

Tech Notes
Two major differences between geninit and mkinitcpio:
1) geninit does away with using gen_init_cpio as mkinitcpio does and creates the cpio archive on its own from a temporary directory hierarchy. While mkinitcpio needs to make a semi-sketchy call to grep before adding every file to the input list (to ensure no duplicates), geninit only needs to perform a simple shell test to check for existance. Using install(1) to place the files in the temporary directory tree also ensures that parent directories are seamlessly created. In the end, geninit calls find and pipes to bsdcpio, ensuring that ordering is correct for extraction. This gets around some of the issues mkinitcpio encountered in attempts to streamline the process (FS #18347).
2) Early userspace is compiled C. Although there's a fair bit of forking involved (mostly for udev), and the need for busybox is almost inevitable, I'm pretty happy with how this has turned out. We still retain the ability for children to speak with init and set variables through clever placement of a pipe. Therefore, the fact that the hooks are executed and not sourced is moot. On the other extreme, if your root setup is simple enough, you can explicitly mention your modules in the config, skip udev, skip busybox, and the resulting image is under 1mb.

As always: comments, criticisms and tomatos are welcome.

Last edited by falconindy (2011-05-23 15:42:59)

chris-kun · 2011-05-23 06:49:52

wow, pretty awesome stuff there mate

litemotiv · 2011-05-23 07:01:53

Great stuff, very promising improvements over mkinitcpio. Thanks falconindy.

hellnest · 2011-05-23 08:36:22

Very fast and promising I really admire all your work and script lol ^^, thanks

Guff · 2011-05-23 15:15:50

Hoo boy, nice. Cuts the time down from ~45s with gzip to ~21s on my netbook with an old-fashioned hard drive.

Very interested in seeing where you go with this.

Haptic · 2011-05-24 01:15:52

What do you mean by 'Update your bootloader?'

sand_man · 2011-05-24 01:38:18

Very nice

sand_man · 2011-05-24 01:40:25

Haptic wrote:

What do you mean by 'Update your bootloader?'

Change the grub config to boot using the initramfs generated by geninit rather than the one generated by mkinitcpio.
I guess grub2 will detect this automatically if you run grub-mkconfig

hellnest · 2011-05-24 06:24:48

if you use burg, you must "modified" a little the arch-folding.patch before it can be detected.

@falconindy
v86d doesn't work on mine, is there any miss steps i take?

==> Included modules:
  ext4
  jbd2
  mbcache
  crc16
  crc-t10dif
  sr_mod
  scsi_mod
  sd_mod
  pata_acpi
  ata_generic
  pata_sis
  libata
  sata_sis
  cdrom

==> Included binaries:
  /sbin/udevadm
  /sbin/modprobe
  /sbin/udevd
  /sbin/v86d
  /sbin/blkid
  /bin/busybox

litemotiv · 2011-05-24 08:36:21

sand_man wrote:

I guess grub2 will detect this automatically if you run grub-mkconfig

Not with the default naming scheme that geninit uses, so you need to change this or edit the /etc/grub.d/10_linux file.

falconindy · 2011-05-24 10:02:27

hellnest wrote:

if you use burg, you must "modified" a little the arch-folding.patch before it can be detected.
@falconindy
v86d doesn't work on mine, is there any miss steps i take?
==> Included modules:
  ext4
  jbd2
  mbcache
  crc16
  crc-t10dif
  sr_mod
  scsi_mod
  sd_mod
  pata_acpi
  ata_generic
  pata_sis
  libata
  sata_sis
  cdrom

==> Included binaries:
  /sbin/udevadm
  /sbin/modprobe
  /sbin/udevd
  /sbin/v86d
  /sbin/blkid
  /bin/busybox

Oops, that's my fault. You can work around it for now by just adding uvesafb to the modules array. It'll be fixed in git soon-ish.

Last edited by falconindy (2011-05-24 10:04:32)

hellnest · 2011-05-24 10:12:02

LoL, ok thanks

freestyler7 · 2011-05-25 22:59:06

Worked with no problems and the speed is really sweet.

el mariachi · 2011-05-26 09:35:56

Great one falconindy! Kernel26 with BFS and this gave me a huge boost in startup time

litemotiv · 2011-05-26 09:39:10

el mariachi wrote:

Great one falconindy! Kernel26 with BFS and this gave me a huge boost in startup time

It did what??

ngoonee · 2011-05-26 09:42:45

litemotiv wrote:

el mariachi wrote:
Great one falconindy! Kernel26 with BFS and this gave me a huge boost in startup time
It did what??

Proof required, if the original quote was accurate.

falconindy · 2011-05-26 12:38:34

Call me a skeptic. You might shave a half second to a second off your actual bootstrap with the compiled init.

Been working on custom mount handlers... Back end is in place -- just need to wire up the bash side.

freestyler7 · 2011-05-26 13:44:00

Is there another way of calling the "geninit -p" command after a new kernel is installed/updated except for downloading the ABS tree and manually editing the install file?

falconindy · 2011-05-26 13:45:07

You can install new kernels with --noscriptlet, but then you need to remeber to call 'depmod' for the new kernel as well (prior to running geninit). I can't really recommend this, but it's a possible workaround.

Last edited by falconindy (2011-05-26 13:50:15)

BasT · 2011-05-27 08:57:20

Looks great! Is it possible to get Plymouth to work with this?

falconindy · 2011-05-27 12:10:29

Sure, it's possible. Feel free to port over the mkinitcpio scripts and give it a try.

BasT · 2011-05-27 19:27:29

Ok, I ported the scripts however I'm not sure how to handle this add_device call:

add_device /dev/fb0 c 29 0

I just commented it out hoping it might just work but of course it didn't. Any advice?

falconindy · 2011-05-27 19:28:28

add it to the hook as a mknod call

Last edited by falconindy (2011-05-27 19:49:18)

BasT · 2011-05-27 19:53:15

There already is a mknod call in the hook:

#!/bin/sh

echo -n ":: Loading plymouth..."

mknod /dev/fb c 29 0 &>/dev/null
#mount devpts /dev/pts -t devpts

# plymouth daemon
/bin/plymouthd

# plymouth frontend
/bin/plymouth --show-splash
echo "done."

falconindy · 2011-05-27 20:01:09

Then the add_device call isn't needed. I'm not familiar with plymouth so I don't really have any advice. The builder is likely more important than the hook itself here.

Arch Linux

#1 2011-05-23 02:10:23

geninit - fast and familiar initramfs creation

#2 2011-05-23 06:49:52

Re: geninit - fast and familiar initramfs creation

#3 2011-05-23 07:01:53

Re: geninit - fast and familiar initramfs creation

#4 2011-05-23 08:36:22

Re: geninit - fast and familiar initramfs creation

#5 2011-05-23 15:15:50

Re: geninit - fast and familiar initramfs creation

#6 2011-05-24 01:15:52

Re: geninit - fast and familiar initramfs creation

#7 2011-05-24 01:38:18

Re: geninit - fast and familiar initramfs creation

#8 2011-05-24 01:40:25

Re: geninit - fast and familiar initramfs creation

#9 2011-05-24 06:24:48

Re: geninit - fast and familiar initramfs creation

#10 2011-05-24 08:36:21

Re: geninit - fast and familiar initramfs creation

#11 2011-05-24 10:02:27

Re: geninit - fast and familiar initramfs creation

#12 2011-05-24 10:12:02

Re: geninit - fast and familiar initramfs creation

#13 2011-05-25 22:59:06

Re: geninit - fast and familiar initramfs creation

#14 2011-05-26 09:35:56

Re: geninit - fast and familiar initramfs creation

#15 2011-05-26 09:39:10

Re: geninit - fast and familiar initramfs creation

#16 2011-05-26 09:42:45

Re: geninit - fast and familiar initramfs creation

#17 2011-05-26 12:38:34

Re: geninit - fast and familiar initramfs creation

#18 2011-05-26 13:44:00

Re: geninit - fast and familiar initramfs creation

#19 2011-05-26 13:45:07

Re: geninit - fast and familiar initramfs creation

#20 2011-05-27 08:57:20

Re: geninit - fast and familiar initramfs creation

#21 2011-05-27 12:10:29

Re: geninit - fast and familiar initramfs creation

#22 2011-05-27 19:27:29

Re: geninit - fast and familiar initramfs creation

#23 2011-05-27 19:28:28

Re: geninit - fast and familiar initramfs creation

#24 2011-05-27 19:53:15

Re: geninit - fast and familiar initramfs creation

#25 2011-05-27 20:01:09

Re: geninit - fast and familiar initramfs creation

Board footer