You are not logged in.

#1 2011-03-30 17:55:30

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

e4rat - reduce boot time (into X) by some 50% for ext4

Please move if this thread if necessary.

I recently stumbled across a rather cool way of radically reducing boot time. Project page: http://e4rat.sourceforge.net/

Quick summary:

During the boot process some 2500 very small files are read which are all over the hard disk. Basically, some 80% of boot time is spent accessing these files. In practice I watched a Debian machine boot into X with firefox in 45 seconds. After doing the e4rat trick it did the same in 15 seconds.

Andreas Rid has written a set of tools called e4rat. They
a) collect data of files accessed during the boot process
b) reallocate them
c) preload them

Note that this is totally unlike upstart or similar projects. Here is the README file:

INTRODUCTION
------------

e4rat reduces disk access times through physical file reallocation. It is based
on the online defragmentation ioctl EXT4_IOC_MOVE_EXT from the ext4 filesystem,
which was introduced in Linux Kernel 2.6.31. Therefore, other filesystem types
or earlier versions of extended filesystems are not supported.

e4rat consists of three binaries. The first is e4rat-collect. Its purpose is to
gather relevant files by monitoring file accesses during an application startup.
The generated file list is the fundament of the second step. With the second
step, e4rat-realloc, files are placed physically in a row on disk.
The reallocation of the files' content yields a higher disk transfer rate which
accelerates program start processes.

Third, you can also read-ahead files to gain a higher cache hit rate.
e4rat-preload transfers files into memory in parallel to program startup.

Because a file consists of file content and its I-Node information the
preloading process is divided into two steps. First, it reads the I-Nodes'
information which are still spread over the entire filesystem. In the second
step, the files' content is read without causing any disk seeks.

For more information see: e4rat-collect(8), e4rat-realloc(8), e4rat-preload(8)
                          and e4rat.conf(5).


SAMPLE USAGE: Accelerate the boot process
-------------

Run e4rat-collect as init process through adding following line to Kernel
parameters:
    init=/usr/sbin/e4rat-collect

After e4rat-collect has terminated the generated file list is stored at:
    /var/lib/e4rat/startup.log

Before reallocating boot files it is recommaned to switch to runlevel 1 to
stop most of all running proceses to gain write access to the process binary
file:
    init 1
    
Reallocating boot files:
    e4rat-realloc  /var/lib/e4rat/startup.log

To start e4rat-preload as initial process append following to kernel your
parameters:
    init=/usr/sbin/e4rat-preload

DEPENDENIES
-----------
The e4rat toolset has the following external dependencies:
 - Linux Kernel (>= 2.6.31)
 - CMake (>= 2.6)
 - pod2man
 - Boost Library (>=1.41): You need the following components installed:
       system, filesytem, regex, signals2
 - Linux Audit Library (libaudit >=0.1.7)
 - Block Device Identification Library (libblkid)
 - Ext2 File System Utilities (e2fsprogs)

BUILDING
--------

The build system is based on CMake, which will generate a Makefile.
To build the release version of e4rat run the following command:
    cmake . -DCMAKE_BUILD_TYPE=release
    make

The default behaviour of cmake is to install binaries and libraries to
/usr/local. To use a different install prefix add the command line parameter to
cmake as follow:
    cmake . -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=release

Once you have successfully built the source code install e4rat as root:
    make install

If you are running a Debian based Linux you can also install e4rat by
generating a Debian package:
    make package
    dpkg -i e4rat-<version>-<arch>-release.deb


AUTHORS
-------

e4rat has been developed by Andreas Rid <conso@hs-augsburg.de> under the
guidance of Gundolf Kiefer <gundolf.kiefer@hs-augsburg.de> at
the University of Applied Sciences, Augsburg.

Unfortunately the tool is written for Debian and that is where my questions start:

It has libaudit as a dependency - unfortunately I cannot install it even though it is in the AUR. Had I been able to I would have tried to get a packagebuild going but then realised that it uses Debian's init stuff and adapting it to Arch's boot process would have been a LITTLE difficult for me.

Anyway, would anybody care about getting into this for Arch? Is it possible? It really works for Debian, I've seen it with my own eyes and IT IS AWESOME!

Cheers...

Last edited by toad (2011-04-06 22:31:11)


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#2 2011-03-30 18:08:53

litemotiv
Forum Fellow
Registered: 2008-08-01
Posts: 5,026

Re: e4rat - reduce boot time (into X) by some 50% for ext4

toad, if you rename your thread to something less generic there will be no problem in keeping it around.

Also check out systemd which is a really fast init system with working Arch support.

Moving to GNU/Linux discussion.


ᶘ ᵒᴥᵒᶅ

Offline

#3 2011-03-30 19:29:09

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: e4rat - reduce boot time (into X) by some 50% for ext4

So... I spend ~20 sec to boot on a P4 machine with

DAEMONS=(syslog-ng irqbalance dbus microcode alsa iptables fcrond network sshd postfix ntpd cups smartd)

and even less on a core2duo. What is the point? Isn't it easier to have /usr on reiserfs?


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#4 2011-03-30 22:01:23

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Leonid.I wrote:

So... I spend ~20 sec to boot on a P4 machine with

DAEMONS=(syslog-ng irqbalance dbus microcode alsa iptables fcrond network sshd postfix ntpd cups smartd)

and even less on a core2duo. What is the point? Isn't it easier to have /usr on reiserfs?

Sorry, but this is beside the point.

This machine booted into X and firefox in 15 seconds what used to take 45 seconds. The interesting thing is optimising CPU and hard disk access.


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#5 2011-03-31 07:30:00

Cdh
Member
Registered: 2009-02-03
Posts: 1,098

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Leonid.I wrote:

So... I spend ~20 sec to boot on a P4 machine with
...
and even less on a core2duo. What is the point? Isn't it easier to have /usr on reiserfs?

Firstly your CPU speed doesn't matter that much in boot speed.
Secondly not everyone has fast hard disk. My netbook's disk is extremely slow (daemons + xfce start sometimes > 1 minute). So if I would use ext4 instead of btrfs I would definitely try it.


฿ 18PRsqbZCrwPUrVnJe1BZvza7bwSDbpxZz

Offline

#6 2011-03-31 14:56:51

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Cdh wrote:

Firstly your CPU speed doesn't matter that much in boot speed.
Secondly not everyone has fast hard disk. My netbook's disk is extremely slow (daemons + xfce start sometimes > 1 minute). So if I would use ext4 instead of btrfs I would definitely try it.

I agree about the CPU being not that important, but my hdds are usual: 5400 and 7200 rpms, the newest one is circa early 2008... And I also have xfce4.

I think we need to differentiate between starting up the system and loading the DE, because all those systemd/upstart/e4rat/... are supposed to only speedup the former. The timeframe for loading the desktop depends on a graphics driver and whether KMS is being used.


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#7 2011-03-31 15:06:23

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Leonid.I wrote:

I think we need to differentiate between starting up the system and loading the DE, because all those systemd/upstart/e4rat/... are supposed to only speedup the former. The timeframe for loading the desktop depends on a graphics driver and whether KMS is being used.

Well, I don't really know to tell you the truth. Until somebody has managed to get this going on Arch (or Debian for that matter) to get the list of files there is no telling...


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#8 2011-04-04 17:00:30

taugenix
Member
Registered: 2011-04-04
Posts: 3

Re: e4rat - reduce boot time (into X) by some 50% for ext4

toad wrote:

Well, I don't really know to tell you the truth. Until somebody has managed to get this going on Arch (or Debian for that matter) to get the list of files there is no telling...

Hi toad

the package name of the audit library is aur/audit.
Do not use aur/libaudit which is out of date.

Offline

#9 2011-04-04 17:33:38

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Coolio Taugenix!

Installed audit. e4rat then complained about lacking boost files, so installed that as well. e4rat compiled properly and the make install went without a hitch.

Still apprehensive to try it because of differences between Arch and Debian - or Canterbury project, where art thou? wink

Unfortunately I cannot read code - is somebody, anybody able to tell me whether I can use this tool on Arch without hosing my install?


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#10 2011-04-05 07:16:07

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Okay, got it working smile

And wow, switch machine on and there it is. Booting into KDM has always been fast, but now logging into KDE is just a blinder. Fantastic tool and I will be writing up a wiki page...


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#11 2011-04-05 08:20:08

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: e4rat - reduce boot time (into X) by some 50% for ext4

toad wrote:

Okay, got it working smile

And wow, switch machine on and there it is. Booting into KDM has always been fast, but now logging into KDE is just a blinder. Fantastic tool and I will be writing up a wiki page...

That's fantastic news smile.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#12 2011-04-05 09:19:09

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

https://wiki.archlinux.org/index.php/E4rat

Now for the packagebuild - but please, if somebody else feels like it... smile

Last edited by toad (2011-04-05 09:20:30)


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#13 2011-04-05 11:30:23

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Nice find, Toad.  I guess this has limited to no utility for SSD-based systems?

Last edited by graysky (2011-04-05 11:30:40)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#14 2011-04-05 11:34:00

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Hm, I have no idea how SSD based hard disks access files. They may still benefit from the reallocation phase though.


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#15 2011-04-05 11:58:47

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

litemotiv wrote:

Also check out systemd which is a really fast init system with working Arch support.

I think this thing can be combined with systemd, both projects shorten boot time (in the case of systemd indirectly, speed is not their main concern), but by using different tricks.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#16 2011-04-05 12:06:44

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

I must confess I haven't even bothered with systemd - the 15 page thread was beyond me. Sorry, litemotiv, no offence.


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#17 2011-04-05 12:15:42

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

I took a stab at a PKG: http://aur.archlinux.org/packages.php?ID=47944

Odd thing is that it doesn't build with makepkg, but it does if you do it long hand.  Help?

https://bbs.archlinux.org/viewtopic.php?pid=913526

Last edited by graysky (2011-04-05 12:22:43)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#18 2011-04-05 12:25:16

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

No idea but am trying it right now. As for the readme - it is called README.

Oh, just got this:

==> Starting package()...
Scanning dependencies of target e4rat-core
[  5%] Building CXX object src/CMakeFiles/e4rat-core.dir/config.cc.o                                                                                 
[ 11%] Building CXX object src/CMakeFiles/e4rat-core.dir/logging.cc.o                                                                               
[ 17%] Building CXX object src/CMakeFiles/e4rat-core.dir/common.cc.o                                                                                 
[ 23%] Building CXX object src/CMakeFiles/e4rat-core.dir/fiemap.cc.o                                                                                 
[ 29%] Building CXX object src/CMakeFiles/e4rat-core.dir/device.cc.o                                                                                 
Linking CXX shared library ../libe4rat-core.so                                                                                                       
[ 29%] Built target e4rat-core                                                                                                                       
Scanning dependencies of target e4rat-collect
[ 35%] Building CXX object src/CMakeFiles/e4rat-collect.dir/e4rat-collect.cc.o                                                                       
[ 41%] Building CXX object src/CMakeFiles/e4rat-collect.dir/fileptr.cc.o                                                                             
[ 47%] Building CXX object src/CMakeFiles/e4rat-collect.dir/listener.cc.o                                                                           
[ 52%] Building CXX object src/CMakeFiles/e4rat-collect.dir/eventcatcher.cc.o                                                                       
Linking CXX executable ../e4rat-collect                                                                                                             
../libe4rat-core.so.0: undefined reference to `boost::filesystem2::detail::not_found_error()'                                                       
../libe4rat-core.so.0: undefined reference to `boost::filesystem2::detail::status_api(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::system::error_code&)'
../libe4rat-core.so.0: undefined reference to `boost::filesystem2::detail::dir_itr_close(void*&, void*&)'
../libe4rat-core.so.0: undefined reference to `boost::filesystem2::detail::get_current_path_api(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
../libe4rat-core.so.0: undefined reference to `unix_io_manager'
../libe4rat-core.

Ahem, suppose I should really have posted this in the AUR section...


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#19 2011-04-05 20:05:58

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

@toad - updated.  Please build and report back.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#20 2011-04-05 20:22:43

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Works a treat, many thanks. Still not quite happy with the readme.install though as the most informative file is the README file in the untarred archive. Is it possible to fix that?

EDIT:

Wow, you were quick on the wiki smile Was just going to do it myself.

Last edited by toad (2011-04-05 20:26:12)


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#21 2011-04-05 20:34:17

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

sure.. I can cat the README if you want.. how about this: edit the readme.install to your liking and put it on pastebin - I will update.  Just note the syntax of the current one.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#22 2011-04-05 20:35:23

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Hm, seems a bit "manual" to me. No way of automating it?

EDIT:

In the meantime, hope this is correct: http://pastebin.com/VRtzKbsg

Last edited by toad (2011-04-05 21:03:17)


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

#23 2011-04-05 20:45:37

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

toad wrote:

Hm, seems a bit "manual" to me. No way of automating it?

Just distill out the main points:

--add kernel line
init=/usr/sbin/e4rat-collect

--log data for 2 min then
sudo init 1
e4rat-realloc  /var/lib/e4rat/startup.log

--remove the kernel line and use
init=/usr/sbin/e4rat-preload


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#24 2011-04-05 20:51:24

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,592
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Ack - 21 s before and 18 s after one of my servers tongue

C2D @ 2.53 GHz/7200 RPM SATA2 HDD.

Last edited by graysky (2011-04-05 20:53:22)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#25 2011-04-05 21:16:48

toad
Member
From: if only I knew
Registered: 2008-12-22
Posts: 1,775
Website

Re: e4rat - reduce boot time (into X) by some 50% for ext4

Here you go: http://pastebin.com/isA0W9S5

As for your result - granted, with a server you most likely do not boot into X and have a number of "heavy" programmes like ff starting automatically. How many lines does your startup.log have?

EDIT:

BTW, how do you keep up-to-date with releases?

Last edited by toad (2011-04-05 21:18:48)


never trust a toad...
::Grateful ArchDonor::
::Grateful Wikipedia Donor::

Offline

Board footer

Powered by FluxBB