You are not logged in.

#1 2009-09-10 17:46:40

josh_
Member
Registered: 2009-09-10
Posts: 6

mkinitcpio optimization

Hello!

I have significantly improved the performance of mkinitcpio.
On my system it now completes 3-4 times faster.

Here you can find the whole stuff (report, download):
http://students.mimuw.edu.pl/~kk219459/mkinitcpio/

Before my summer holiday I sent this to the maintainers of mkinitcpio, but I haven't got any response.
A week ago I tried to register on this forum (using the same email) and... haven't received the activation mail.
So finally I switched to another email and I'm posting it smile

I hope this gets committed, so everyone can enjoy!

The work was done on version 0.5.23, so it might be a little bit outdated.
If any developer is willing to commit it, I can merge this to the latest version.

Offline

#2 2009-09-10 17:55:01

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: mkinitcpio optimization

The best way to get this included (I did not look at it yet) is to file a feature request in our bug tracker. Mailing the maintainer is not the best idea; we get lots of mails and some of them have the wrong recipient.

Offline

#3 2009-09-10 18:59:33

nTia89
Banned
From: varese, italy
Registered: 2008-12-22
Posts: 1,230

Re: mkinitcpio optimization

INTERESTING WORK !!!
yes use a "feature request" also it is not the best but is the most effcient way .....


+pc: custom | AMD Opteron 175 | nForce4 Ultra | 2GB ram DDR400 | nVidia 9800GT 1GB | ArchLinux x86_64 w/ openbox
+laptop: Apple | MacBook (2,1) | 2GB ram | Mac OS X 10.4 -> DIED
+ultrabook: Dell | XPS 13 (9343) | 8GB ram | 256GB ssd | FullHD display | Windows 8.1 64bit ArchLinux x86_64 w/ Gnome

Offline

#4 2009-09-10 19:18:59

genisis300
Member
From: Uk
Registered: 2008-01-15
Posts: 284

Re: mkinitcpio optimization

has anyone ever tried a pearl / python port of the mkinicpio scripts?

Matthew


"is adult entertainment killing our children or is killing our children entertaining adults?" Marilyn Manson

Offline

#5 2009-09-10 23:19:05

smartboyathome
Member
From: $HOME
Registered: 2007-12-23
Posts: 334
Website

Re: mkinitcpio optimization

genisis300 wrote:

has anyone ever tried a pearl / python port of the mkinicpio scripts?

Matthew

If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.

Offline

#6 2009-09-11 00:01:03

Heller_Barde
Member
Registered: 2008-04-01
Posts: 245

Re: mkinitcpio optimization

smartboyathome wrote:

If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.

well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient

cheers
Barde

Offline

#7 2009-09-11 00:06:30

smartboyathome
Member
From: $HOME
Registered: 2007-12-23
Posts: 334
Website

Re: mkinitcpio optimization

Heller_Barde wrote:
smartboyathome wrote:

If anything, that would probably make it less efficient, due to it having to be interpreted twice (by the python/pearl interpreter, then again as bytecode). With mkinitcpio by itself, it is already a binary, so it doesn't need to be interpreted. I am not going to say I am an expert on this subject, though, as I just know enough to say what I did.

well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient

cheers
Barde

Ah, if so, my bad. I was under the assumption it was written in C(++?) or some other compiled language. neutral

Offline

#8 2009-09-11 00:31:56

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: mkinitcpio optimization

Heller_Barde wrote:

well, no... as far as i understood it (i am not at my arch machine so i dont want to check), mkinitcpio is a bash script, so perl/python might be faster/more efficient

It is bash, but it's HEAVILY bound to external apps and file access, meaning the language of choice will change very little here.

To josh_, I apologize for not responding to you sooner. The email IS starred in my inbox, I just haven't rushed into this because mkinitcpio runs so rarely, it's not a major deal to me.... still, this is some great work.

If you *really* want this included, I would need to get this in smaller concise patches. The way to do this would be to checkout the git repo, and begin making individual changes to get to your version. They should be done in smaller concrete steps. This allows for easier review and understanding of the changes.

I know that's probably a lot of work, and I'd be willing to help, if you could outline the steps from point A (the original) to point B (your version).

As for the parallel version, that seems like overkill to me smile

Offline

#9 2009-09-11 00:42:15

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: mkinitcpio optimization

After a brief glance at the changes, I've identified the following individual changes:

mkinitcpio
==========
* Add precache function (and call it)

functions
=========
* Use precache output in all_modules and checked_modules
* Remove use of grep to check for existing entries
* Use precache output in add_module
* Remove mv/sort of FILELIST

autodetect
==========
* Remove mv/sort of module file

Please tell me if I missed anything. If you can provide these changes as distinct git patches, I will gladly apply them

Offline

#10 2009-09-11 11:51:27

josh_
Member
Registered: 2009-09-10
Posts: 6

Re: mkinitcpio optimization

Hi! Thanks for your interest.

To make this list complete, it should be:

mkinitcpio
==========
A Add precache function (and call it)
B Add mv/sort of FILELIST
C Use /bin/install when generating module deps
D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

functions
=========
E Use precache output in all_modules and checked_modules
F Remove use of grep to check for existing entries
G Use precache output in add_module
H Don't use sed in add_module

autodetect
==========
I Add mv/sort of module file

Do you want me to set up a local git repository, importing from
ftp://ftp.archlinux.org/other/mkinitcpi … .25.tar.gz
and produce patches with "git diff" ?

These things don't make sense when done separately.
However, if you insist on having smaller patches, how about this scheme:
original -> P1 -> P2 -> P3 -> P4 = optimized
P1 = only fixes (D)
P2 = no grepping (BF)
P3 = precache (AEGI)
P4 = low level stuff (CH)

Is it okay to do the thing with 4 patches?

Offline

#11 2009-09-11 12:00:11

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,357
Website

Re: mkinitcpio optimization

You can checkout the current mkinitcpio git repo instead of creating one from the tarball.  See the links on this page: http://projects.archlinux.org/?p=mkinit … ;a=summary

And I see not issue with reducing it to four patches, as long as they are fairly small and each change quite self contained then it is a lot easier for us to review.

Offline

#12 2009-09-11 13:42:49

louipc
Member
Registered: 2006-10-09
Posts: 85

Re: mkinitcpio optimization

josh_ wrote:

D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

Hah! The last patches to mkinicpio were to make it dash compatible.
Is is possible that your changes retain that compatibility?

Last edited by louipc (2009-09-11 13:43:13)

Offline

#13 2009-09-11 14:51:54

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: mkinitcpio optimization

louipc wrote:
josh_ wrote:

D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

Hah! The last patches to mkinicpio were to make it dash compatible.
Is is possible that your changes retain that compatibility?

Yeah, I was going to say this. Loui made changes so we can ignore the bash/sh change. It is fully sh compatible now (which also adds a speedup if you're using dash as /bin/sh

Offline

#14 2009-09-11 14:58:53

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: mkinitcpio optimization

josh_ wrote:

Hi! Thanks for your interest.

To make this list complete, it should be:

mkinitcpio
==========
A Add precache function (and call it)
B Add mv/sort of FILELIST
C Use /bin/install when generating module deps
D Minor bugfixes (-n => -z, #!/bin/sh => #!/bin/bash)

functions
=========
E Use precache output in all_modules and checked_modules
F Remove use of grep to check for existing entries
G Use precache output in add_module
H Don't use sed in add_module

autodetect
==========
I Add mv/sort of module file

Do you want me to set up a local git repository, importing from
ftp://ftp.archlinux.org/other/mkinitcpi … .25.tar.gz
and produce patches with "git diff" ?

These things don't make sense when done separately.
However, if you insist on having smaller patches, how about this scheme:
original -> P1 -> P2 -> P3 -> P4 = optimized
P1 = only fixes (D)
P2 = no grepping (BF)
P3 = precache (AEGI)
P4 = low level stuff (CH)

Is it okay to do the thing with 4 patches?

Please use a git checkout ass Allan suggested.

As for the patches, I would be fine with:
P1 = fixes D
P2 = no grep F
P3 = precache AEG
P4 = mv/sort BI
P5 = /bin/install C
P6 = no sed H

Offline

#15 2009-09-14 12:32:42

josh_
Member
Registered: 2009-09-10
Posts: 6

Re: mkinitcpio optimization

All right.

D - no longer necessary
H - will not work with dash

The rest of the patches were reordered, so dependencies are not broken:
P1 = mv/sort BI
P2 = no grep F
P3 = precache AEG
P4 = /bin/install C

I have uploaded the patches here: http://students.mimuw.edu.pl/~kk219459/ … pio/files/

Note that I only reapplied them, without any testing. They worked perfectly with v0.5.23.

Offline

#16 2009-09-15 16:51:25

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

It seems you made these patches using git diff.
What about making real git patches ? see http://wiki.archlinux.org/index.php/Sup … ng_a_patch
This will allow you to write a comment describing your changes, and put the authorship.
That will also make it even easier for the dev to pick your patch, otherwise they will have to do this work themselves.

If additionally you put your repo somewhere (http://git.or.cz/gitwiki/GitHosting), it would make it even easier for everyone to review your changes, to download just one patch, to download the final files, etc.

This is all overkill if you are not going to do anything more than these 4 patches, but otherwise, you could consider it smile


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#17 2009-09-15 17:13:37

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.

mkinitcpio -p kernel26 took 2m20s without your patches, and 37s with your patches, so I guess that's an improvement smile


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#18 2009-09-16 15:17:53

josh_
Member
Registered: 2009-09-10
Posts: 6

Re: mkinitcpio optimization

It seems you made these patches using git diff.
What about making real git patches ? see http://wiki.archlinux.org/index.php/Sup … ng_a_patch
This will allow you to write a comment describing your changes, and put the authorship.
That will also make it even easier for the dev to pick your patch, otherwise they will have to do this work themselves.

I used git-format-patch on my local repository and put the patches in the same HTTP directory.
But I hadn't been writing any descriptions when committing. Maybe this helps...

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.

There's nothing unusual about it. This is how operating systems work to improve performance.
When you run a program next time, disk cache is already filled with it's data, so there's no need to do the disk IO, etc.

mkinitcpio -p kernel26 took 2m20s without your patches, and 37s with your patches, so I guess that's an improvement

Thanks for the measurement.
If mkinitcpio was written in a more powerful language, the borderline on my system is less than 4 sec, so on your system it should be 12s or so.
But rewriting everything from scratch is likely too time-consuming for the developers.

Offline

#19 2009-09-16 15:23:59

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

josh_ wrote:

I have always found mkinitcpio quite slow, but for some reason, the first time I measure its execution, it seems to take much longer than usual.
Maybe because I am running a kernel with some debug stuff enabled.

There's nothing unusual about it. This is how operating systems work to improve performance.
When you run a program next time, disk cache is already filled with it's data, so there's no need to do the disk IO, etc.

I realized what I said led to confusion.
What I meant is that it was the first day I benchmarked mkinitcpio.
And it was much much slower than the previous times (on previous days).


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#20 2009-10-01 07:04:57

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

Hm I just broke it smile

I have drm.ko both in kernel/drivers/gpu/drm/ (normal path) and updates/ (newer drm needed for nouveau), and I added drm to MODULES, and it didn't like it.

==> Using /etc/mkinitcpio.d/kernel26.preset
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.31-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: file '/lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko' does not exist
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory
ERROR: modinfo: could not open /lib/modules/2.6.31-ARCH/kernel/drivers/gpu/drm/drm.ko
drm /lib/modules/2.6.31-ARCH/updates/drm.ko: No such file or directory

pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#21 2009-10-01 15:31:04

josh_
Member
Registered: 2009-09-10
Posts: 6

Re: mkinitcpio optimization

This was written under the assumption that module names are unique.
What does "modprobe drm" do on your setup?

In case you really don't know how to fix it:

1. revert this patch (lib/initcpio/functions):

@@ -135,12 +127,14 @@ HAS_MODULES="n"
 #modules are handled specially in order to enable autodetection
 add_module ()
 {
-    local fil path fw mod deps
+    local fil path fw mod
     #cleanup - remove .ko, replace - and _ with [-_] to match either
     fil=$(basename "${1}" | sed -e "s|[-_]|\[-_\]|g" -e "s|\.ko$||g")
 
-    found=0
-    for path in $(find "${MODULEDIR}" -type f -name "${fil}.ko"); do
+    path=$(grep "^${fil} " ${TMPDIR}/map)
+    path=${path#* }
+
+    if [ -n "$path" ]; then
         #get needed firmware files
         for fw in $(/sbin/modinfo -F firmware "${path}"); do
             [ -f "/lib/firmware/$fw" ] && add_file "/lib/firmware/$fw"
@@ -152,9 +146,8 @@ add_module ()
             fi
         done
         HAS_MODULES="y"
-        add_file "${path}" && found=1
-    done
-    if [ ${found} -eq 0 ]; then
+        add_file "${path}"
+    else
         err "module '$fil' not found"
     fi
 }

2. apply this

-    for path in $(find "${MODULEDIR}" -type f -name "${fil}.ko"); do
+    for path in $(grep "^${fil} " ${TMPDIR}/map | cut -d' ' -f2); do

Offline

#22 2009-10-01 15:37:25

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

josh_ wrote:

This was written under the assumption that module names are unique.
What does "modprobe drm" do on your setup?

I guess I was just informing you that this assumption is wrong smile
modprobe drm takes the one in updates/
modules in updates/ always takes preference on the rest. It is a nice feature to provide newer version of kernel modules, and it is exactly what nouveau-drm package needs.

In case you really don't know how to fix it:

The fix was simple, I just reverted to arch mkinitcpio and I saw it worked fine.

But thanks for patching this issue right away. You could also update the patches at
http://students.mimuw.edu.pl/~kk219459/ … pio/files/


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#23 2009-10-03 14:17:49

josh_
Member
Registered: 2009-09-10
Posts: 6

Re: mkinitcpio optimization

But thanks for patching this issue right away. You could also update the patches at

Here you are http://students.mimuw.edu.pl/~kk219459/ … pio/files/

Offline

#24 2010-01-12 17:48:36

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: mkinitcpio optimization

Do you still intend to get this upstream ?
You should open a bug tracker, that's easier to track than a forum post.
And your patches could still be better split, or at least better explained/commented, for example in the git commit log.
And dont use p1/p2/pn, just write a sentence saying what the patch does.
The numbering is done automatically by git, not manually by you smile
And your last patch (5th) actually modifies/reverts part of the previous ones. So you should re-create the 4 patches from scratch instead.
It's very easy to merge two patches using git rebase -i + squash.

And again, if you had a public git tree, it would be easier to get your patches, review them, rebase them, etc.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#25 2010-02-14 11:31:28

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 2,000
Website

Re: mkinitcpio optimization

So this never got merged? josh_ can you make some nice patches that apply to the latest mkinitcpio? If it's still needed that is, with the big busybox change and all.


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

Board footer

Powered by FluxBB