mkinitcpio optimization

josh_ · 2009-09-10 17:46:40

Hello!

I have significantly improved the performance of mkinitcpio.
On my system it now completes 3-4 times faster.

Here you can find the whole stuff (report, download):
http://students.mimuw.edu.pl/~kk219459/mkinitcpio/

Before my summer holiday I sent this to the maintainers of mkinitcpio, but I haven't got any response.
A week ago I tried to register on this forum (using the same email) and... haven't received the activation mail.
So finally I switched to another email and I'm posting it

I hope this gets committed, so everyone can enjoy!

The work was done on version 0.5.23, so it might be a little bit outdated.
If any developer is willing to commit it, I can merge this to the latest version.

Pierre · 2009-09-10 17:55:01

The best way to get this included (I did not look at it yet) is to file a feature request in our bug tracker. Mailing the maintainer is not the best idea; we get lots of mails and some of them have the wrong recipient.

nTia89 · 2009-09-10 18:59:33

INTERESTING WORK !!!
yes use a "feature request" also it is not the best but is the most effcient way .....

genisis300 · 2009-09-10 19:18:59

has anyone ever tried a pearl / python port of the mkinicpio scripts?

Matthew

smartboyathome · 2009-09-10 23:19:05

genisis300 wrote:

has anyone ever tried a pearl / python port of the mkinicpio scripts?
Matthew

If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.

Heller_Barde · 2009-09-11 00:01:03

smartboyathome wrote:

If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.

well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient

cheers
Barde

smartboyathome · 2009-09-11 00:06:30

Heller_Barde wrote:

smartboyathome wrote:
If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.
well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient
cheers
Barde

Ah, if so, my bad. I was under the assumption it was written in C(++?) or some other compiled language.

phrakture · 2009-09-11 00:31:56

Heller_Barde wrote:

well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient

It is bash, but it's HEAVILY bound to external apps and file access, meaning the language of choice will change very little here.

To josh_, I apologize for not responding to you sooner. The email IS starred in my inbox, I just haven't rushed into this because mkinitcpio runs so rarely, it's not a major deal to me.... still, this is some great work.

If you *really* want this included, I would need to get this in smaller concise patches. The way to do this would be to checkout the git repo, and begin making individual changes to get to your version. They should be done in smaller concrete steps. This allows for easier review and understanding of the changes.

I know that's probably a lot of work, and I'd be willing to help, if you could outline the steps from point A (the original) to point B (your version).

As for the parallel version, that seems like overkill to me

phrakture · 2009-09-11 00:42:15

After a brief glance at the changes, I've identified the following individual changes:

mkinitcpio
==========
* Add precache function (and call it)

functions
=========
* Use precache output in all_modules and checked_modules
* Remove use of grep to check for existing entries
* Use precache output in add_module
* Remove mv/sort of FILELIST

autodetect
==========
* Remove mv/sort of module file

Please tell me if I missed anything. If you can provide these changes as distinct git patches, I will gladly apply them

josh_ · 2009-09-11 11:51:27

Hi! Thanks for your interest.

To make this list complete, it should be:

mkinitcpio
==========
A Add precache function (and call it)
B Add mv/sort of FILELIST
C Use /bin/install when generating module deps
D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

functions
=========
E Use precache output in all_modules and checked_modules
F Remove use of grep to check for existing entries
G Use precache output in add_module
H Don't use sed in add_module

autodetect
==========
I Add mv/sort of module file

Do you want me to set up a local git repository, importing from
ftp://ftp.archlinux.org/other/mkinitcpi … .25.tar.gz
and produce patches with "git diff" ?

These things don't make sense when done separately.
However, if you insist on having smaller patches, how about this scheme:
original -> P1 -> P2 -> P3 -> P4 = optimized
P1 = only fixes (D)
P2 = no grepping (BF)
P3 = precache (AEGI)
P4 = low level stuff (CH)

Is it okay to do the thing with 4 patches?

Allan · 2009-09-11 12:00:11

You can checkout the current mkinitcpio git repo instead of creating one from the tarball. See the links on this page: http://projects.archlinux.org/?p=mkinit … ;a=summary

And I see not issue with reducing it to four patches, as long as they are fairly small and each change quite self contained then it is a lot easier for us to review.

louipc · 2009-09-11 13:42:49

josh_ wrote:

D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

Hah! The last patches to mkinicpio were to make it dash compatible.
Is is possible that your changes retain that compatibility?

Last edited by louipc (2009-09-11 13:43:13)

phrakture · 2009-09-11 14:51:54

louipc wrote:

josh_ wrote:
D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)
Hah! The last patches to mkinicpio were to make it dash compatible.
Is is possible that your changes retain that compatibility?

Yeah, I was going to say this. Loui made changes so we can ignore the bash/sh change. It is fully sh compatible now (which also adds a speedup if you're using dash as /bin/sh

phrakture · 2009-09-11 14:58:53

josh_ wrote:

Hi! Thanks for your interest.
To make this list complete, it should be:
mkinitcpio
==========
A Add precache function (and call it)
B Add mv/sort of FILELIST
C Use /bin/install when generating module deps
D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

functions
=========
E Use precache output in all_modules and checked_modules
F Remove use of grep to check for existing entries
G Use precache output in add_module
H Don't use sed in add_module

autodetect
==========
I Add mv/sort of module file
Do you want me to set up a local git repository, importing from
ftp://ftp.archlinux.org/other/mkinitcpi … .25.tar.gz
and produce patches with "git diff" ?
These things don't make sense when done separately.
However, if you insist on having smaller patches, how about this scheme:
original -> P1 -> P2 -> P3 -> P4 = optimized
P1 = only fixes (D)
P2 = no grepping (BF)
P3 = precache (AEGI)
P4 = low level stuff (CH)
Is it okay to do the thing with 4 patches?

Please use a git checkout ass Allan suggested.

As for the patches, I would be fine with:
P1 = fixes D
P2 = no grep F
P3 = precache AEG
P4 = mv/sort BI
P5 = /bin/install C
P6 = no sed H

josh_ · 2009-09-14 12:32:42

All right.

D - no longer necessary
H - will not work with dash

The rest of the patches were reordered, so dependencies are not broken:
P1 = mv/sort BI
P2 = no grep F
P3 = precache AEG
P4 = /bin/install C

I have uploaded the patches here: http://students.mimuw.edu.pl/~kk219459/ … pio/files/

Note that I only reapplied them, without any testing. They worked perfectly with v0.5.23.

shining · 2009-09-15 16:51:25

It seems you made these patches using git diff.
What about making real git patches ? see http://wiki.archlinux.org/index.php/Sup … ng_a_patch
This will allow you to write a comment describing your changes, and put the authorship.
That will also make it even easier for the dev to pick your patch, otherwise they will have to do this work themselves.

If additionally you put your repo somewhere (http://git.or.cz/gitwiki/GitHosting), it would make it even easier for everyone to review your changes, to download just one patch, to download the final files, etc.

This is all overkill if you are not going to do anything more than these 4 patches, but otherwise, you could consider it

shining · 2009-09-15 17:13:37

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.

mkinitcpio -p kernel26 took 2m20s without your patches, and 37s with your patches, so I guess that's an improvement

josh_ · 2009-09-16 15:17:53

It seems you made these patches using git diff.
What about making real git patches ? see http://wiki.archlinux.org/index.php/Sup … ng_a_patch
This will allow you to write a comment describing your changes, and put the authorship.
That will also make it even easier for the dev to pick your patch, otherwise they will have to do this work themselves.

I used git-format-patch on my local repository and put the patches in the same HTTP directory.
But I hadn't been writing any descriptions when committing. Maybe this helps...

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.

There's nothing unusual about it. This is how operating systems work to improve performance.
When you run a program next time, disk cache is already filled with it's data, so there's no need to do the disk IO, etc.

mkinitcpio -p kernel26 took 2m20s without your patches, and 37s with your patches, so I guess that's an improvement

Thanks for the measurement.
If mkinitcpio was written in a more powerful language, the borderline on my system is less than 4 sec, so on your system it should be 12s or so.
But rewriting everything from scratch is likely too time-consuming for the developers.

shining · 2009-09-16 15:23:59

josh_ wrote:

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.
There's nothing unusual about it. This is how operating systems work to improve performance.
When you run a program next time, disk cache is already filled with it's data, so there's no need to do the disk IO, etc.

I realized what I said led to confusion.
What I meant is that it was the first day I benchmarked mkinitcpio.
And it was much much slower than the previous times (on previous days).

shining · 2009-10-01 07:04:57

Hm I just broke it

I have drm.ko both in kernel/drivers/gpu/drm/ (normal path) and updates/ (newer drm needed for nouveau), and I added drm to MODULES, and it didn't like it.

==> Using /etc/mkinitcpio.d/kernel26.preset
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.31-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory

josh_ · 2009-10-01 15:31:04

This was written under the assumption that module names are unique.
What does "modprobe drm" do on your setup?

In case you really don't know how to fix it:

1. revert this patch (lib/initcpio/functions):

@@ -135,12 +127,14 @@ HAS_MODULES="n"
 #modules are handled specially in order to enable autodetection
 add_module ()
 {
-    local fil path fw mod deps
+    local fil path fw mod
     #cleanup - remove .ko, replace - and _ with [-_] to match either
     fil=$(basename "${1}" | sed -e "s|[-_]|\[-_\]|g" -e "s|\.ko$||g")
 
-    found=0
-    for path in $(find "${MODULEDIR}" -type f -name "${fil}.ko"); do
+    path=$(grep "^${fil} " ${TMPDIR}/map)
+    path=${path#* }
+
+    if [ -n "$path" ]; then
         #get needed firmware files
         for fw in $(/sbin/modinfo -F firmware "${path}"); do
             [ -f "/lib/firmware/$fw" ] && add_file "/lib/firmware/$fw"
@@ -152,9 +146,8 @@ add_module ()
             fi
         done
         HAS_MODULES="y"
-        add_file "${path}" && found=1
-    done
-    if [ ${found} -eq 0 ]; then
+        add_file "${path}"
+    else
         err "module '$fil' not found"
     fi
 }

2. apply this

-    for path in $(find "${MODULEDIR}" -type f -name "${fil}.ko"); do
+    for path in $(grep "^${fil} " ${TMPDIR}/map | cut -d' ' -f2); do

shining · 2009-10-01 15:37:25

josh_ wrote:

This was written under the assumption that module names are unique.
What does "modprobe drm" do on your setup?

I guess I was just informing you that this assumption is wrong
modprobe drm takes the one in updates/
modules in updates/ always takes preference on the rest. It is a nice feature to provide newer version of kernel modules, and it is exactly what nouveau-drm package needs.

In case you really don't know how to fix it:

The fix was simple, I just reverted to arch mkinitcpio and I saw it worked fine.

But thanks for patching this issue right away. You could also update the patches at
http://students.mimuw.edu.pl/~kk219459/ … pio/files/

josh_ · 2009-10-03 14:17:49

But thanks for patching this issue right away. You could also update the patches at

Here you are http://students.mimuw.edu.pl/~kk219459/ … pio/files/

shining · 2010-01-12 17:48:36

Do you still intend to get this upstream ?
You should open a bug tracker, that's easier to track than a forum post.
And your patches could still be better split, or at least better explained/commented, for example in the git commit log.
And dont use p1/p2/pn, just write a sentence saying what the patch does.
The numbering is done automatically by git, not manually by you
And your last patch (5th) actually modifies/reverts part of the previous ones. So you should re-create the 4 patches from scratch instead.
It's very easy to merge two patches using git rebase -i + squash.

And again, if you had a public git tree, it would be easier to get your patches, review them, rebase them, etc.

Dieter@be · 2010-02-14 11:31:28

So this never got merged? josh_ can you make some nice patches that apply to the latest mkinitcpio? If it's still needed that is, with the big busybox change and all.

Arch Linux

#1 2009-09-10 17:46:40

mkinitcpio optimization

#2 2009-09-10 17:55:01

Re: mkinitcpio optimization

#3 2009-09-10 18:59:33

Re: mkinitcpio optimization

#4 2009-09-10 19:18:59

Re: mkinitcpio optimization

#5 2009-09-10 23:19:05

Re: mkinitcpio optimization

#6 2009-09-11 00:01:03

Re: mkinitcpio optimization

#7 2009-09-11 00:06:30

Re: mkinitcpio optimization

#8 2009-09-11 00:31:56

Re: mkinitcpio optimization

#9 2009-09-11 00:42:15

Re: mkinitcpio optimization

#10 2009-09-11 11:51:27

Re: mkinitcpio optimization

#11 2009-09-11 12:00:11

Re: mkinitcpio optimization

#12 2009-09-11 13:42:49

Re: mkinitcpio optimization

#13 2009-09-11 14:51:54

Re: mkinitcpio optimization

#14 2009-09-11 14:58:53

Re: mkinitcpio optimization

#15 2009-09-14 12:32:42

Re: mkinitcpio optimization

#16 2009-09-15 16:51:25

Re: mkinitcpio optimization

#17 2009-09-15 17:13:37

Re: mkinitcpio optimization

#18 2009-09-16 15:17:53

Re: mkinitcpio optimization

#19 2009-09-16 15:23:59

Re: mkinitcpio optimization

#20 2009-10-01 07:04:57

Re: mkinitcpio optimization

#21 2009-10-01 15:31:04

Re: mkinitcpio optimization

#22 2009-10-01 15:37:25

Re: mkinitcpio optimization

#23 2009-10-03 14:17:49

Re: mkinitcpio optimization

#24 2010-01-12 17:48:36

Re: mkinitcpio optimization

#25 2010-02-14 11:31:28

Re: mkinitcpio optimization

Board footer