in need of some help understanding load-modules.sh and the boot proces

kumico · 2008-03-02 16:41:48

umm i was bored a little whiel ago, then i rememberd recent posts about how slow arch boots,,
and also relating the an article in the wiki, about disabling load-modules.sh in variosu ways.
then i got thinking, why remove functionality for just 10 seconds? why not make it faster;;
i know, let's write it in C and see what we can get out of it.
anyway, a couple questions.

i looked in /lib/udev/load-modules.sh; where i encounter these lines;;

depmods="$(/sbin/modprobe -i --show-depends $1)"
...
depmods="$(echo $depmods \
                    | sed "s#^insmod /lib.*/\(.*\)\.ko.*#\1#g" \
                    | sed 's|-|_|g')"

i'm sure i must be missing something. but is $depmods really supposed to be return 1 element, i.e $1. ?

and where does /lib/initcpio/udev/load-modules.sh come into play.

klixon · 2008-03-17 08:18:01

My guess (i can't check right now as i'm at work) is that the first depmods-line is part of a function. If so, $1 is the first parameter to that function, which, in this case, would be the name of a module. If not, it is the first parameter of the batch-file.

"modprobe -i --show-depends $1" returns a list of al dependencies of the module $1 (the module-name you pass as a parameter)
This list gets stored in depmods...

The second depmods strips "insmod /lib/modules/<kernel-name/" and ".ko" from every item, to just keep the module-name (check what the command returns in a shell... you'll see what i mean) in the list and then converts every '-' to a '_' for consitent naming...

Later on this probably gets used like so:

for dep in ${depmods}; do
    ... usefull stuff here ...
done

kumico · 2008-03-17 15:33:08

that's what i thought,
today i decided to give it one more shot.
i assume, after sanitization, $depmods whould be populated by all the moduule deps followed by module itself
i.e '''
soundcore
snd
snd_hwdep
snd_page_alloc
snd_timer
snd_pcm
snd_hda_intel
'''

as i was figuring out, how to make it match all and not just the last line out the modprobe *--show* command
i realised, the problem was that the newlines were being swallowed.
a simple fix was to just quote the echo

 depmods="$(echo $depmods \...
===
depmods="$(echo "$depmods" \

thus newlines are preserved and all mod and deps are returned.
maybe i should submit a patch or something.

phrakture · 2008-03-17 16:23:11

Newlines are not the issue. The issue is sed being greedy. It is matching from the first "insmod" to the last ".ko"

kumico · 2008-03-17 16:43:17

i figured, but i though that was normal behavior of sed,,,
i'm beyond the suck when it comes to regex.
the implementation of the command that led me to this, was to match until a space was encountered
obviously that's as bad as a hack goes, since it doesn't take into account, filenames with spaces.

i dunno if this a solution, or if there is actually a problem
but doing it this way seems to work, in either case. it was a valuable learning experience for me so time well spent.

shining · 2008-03-17 19:26:15

kumico wrote:

i figured, but i though that was normal behavior of sed,,,
i'm beyond the suck when it comes to regex.
the implementation of the command that led me to this, was to match until a space was encountered
obviously that's as bad as a hack goes, since it doesn't take into account, filenames with spaces.
i dunno if this a solution, or if there is actually a problem
but doing it this way seems to work, in either case. it was a valuable learning experience for me so time well spent.

Yes, fixing sed is probably trickier, especially when taking spaces into account indeed.
Your solution of just adding quote seems nicer.

I find it easier to do the same using grep + sed, but well that's probably slower because of the two different invocations

And just because I like reporting bugs :
http://bugs.archlinux.org/task/9857

Bebo · 2008-03-17 19:56:51

I'm not sure I understand what the problem is here, but it is not hard to adapt the regexp to make it accept several insmod statements. For example:

sed -r 's|\<insmod /lib/[^ ]+/([^ ]+).ko|\1|g'

or even (to only have one sed invocation)

sed -r 's|\<insmod /lib/[^ ]+/([^ ]+).ko|\1|g; s|-|_|g'

I'm sure it can be done in several other ways*. Also, can't sed be made non-greedy by appending ? after * or +, or isn't that part of the "dialect"?

______________________
* For instance, I like running sed with extended regexps (-r), but that's just a matter of taste, of course.

shining · 2008-03-17 20:56:00

Well, kumico's fix has already been applied and there is a new udev package in testing.

But well, I guess we can still discuss about it
We were wondering what could happen with spaces in the path or in the module. I am not even sure it's possible to have that in the first space, but the current sed command should support it.
/lib/modules/2.6.24 ARCH/kernel/sound/core/snd-pcm.ko

Though, maybe it would be escaped in that case? Anyway, it works either way.

echo -e "insmod /lib/modules/2.6.24 ARCH/kernel/sound/core/snd-pcm.ko\ninsmod /lib/modules/2.6.24 ARCH/kernel/sound/pci/hda/snd-hda-intel.ko model=hp" | sed -e "s#^insmod /lib.*/\(.*\)\.ko.*#\1#g" -e 's|-|_|g'
snd_pcm
snd_hda_intel

We have a simple way to preserve the newlines, so we might as well do that and stop worrying

Last edited by shining (2008-03-17 20:57:27)

kumico · 2008-03-17 20:57:32

that's pretty nice and all, but can you please read #5?

wtf, double postign bastid i am

depmods=$(echo "$depmods" | sed \
            -e "s#^insmod /lib.*/\(.*\)\.ko.*#\1#g" \
            -e 's|-|_|g')
        BLACKLIST=" $BLACKLIST "
        for mod in $depmods; do
            if [ "${BLACKLIST/ $mod /}" != "$BLACKLIST" ]; then

here's the bit i worked on
lol i used a string replacement hack to test, gets rid of the grep call
and it's now a whopping 10ms+ faster lol

Last edited by kumico (2008-03-17 21:10:02)

phrakture · 2008-03-17 21:25:21

kumico wrote:

lol i used a string replacement hack to test, gets rid of the grep call
and it's now a whopping 10ms+ faster lol

That's actually kinda cool, BUT if we could actually get rid of bash itself (that string replacement is bash specific) and use dash (pacman -S dash, switch the sh symlink, switch to !/bin/sh) it should be even faster

Bebo · 2008-03-17 21:35:11

(Okay, I'm not actually posting this - I don't want to disturb the discussion (more) - but I definitely agree that if the problem is solved by quoting, then use that )

klixon · 2008-03-17 23:24:06

maybe

depmods=$(echo "$depmods" | sed \
            -e "s#^insmod /lib.*/\(.*\)\.ko.*#\1#g" \
            -e 's|-|_|g')
        BLACKLIST=" $BLACKLIST "
        for mod in $depmods; do
            if [ "x$(echo ${BLACKLIST} | grep -v ${mod})" == "x" ]; then # <-- fix is here

Or am i missing the picture completely?
Oh i'm still a bit linux-rusty, as i wasn't able to use my computer for a few months, but i think i got the false/true check right

PS: Aaaah, it's good to be back here... Brings back the fun in computing lol

edit: I think this doesntt bode well for snd modules? module "snd" would blanket out all snd_* modules... might need a hell of a lot refining

Last edited by klixon (2008-03-17 23:26:42)

kumico · 2008-03-17 23:52:30

@phrakture:
hmm, i wanna boot the system in dash now... *itch*itch* should i scratch it?

@klixon:
the script works perfectly as-is,
there was an update about an hour ago or something.
the line you refer to, was broken. i was just messing around with it
to see ho much faster i could make it.
...

i scratched it,
it booted ok,
except when it came boot start the gdm daemon at the end of the daemons array.
i couldn't test if it booted any faster as bootchartd had syntax errors, same as the gdm issue

===========================================================================================
update::

    if [ -n "$BLACKLIST" ]; then 
        depmods="$(/sbin/modprobe -i -q --show-depends $1 | sed -e 's|-|_|g')"
        if [ -z "$depmods" ]; then
            /usr/bin/logger -p info "cannot find module $1"
            #echo "MISSING[ $1 ]"
            exit 1
        fi

        for blackmod in $BLACKLIST; do
            if [[ $depmods == *"/$blackmod.ko"* ]]; then
                /usr/bin/logger -p info "udev load-modules: $blackmod is blacklisted"
                #echo "BLACKLISTED[ $blackmod ]"
                exit 1
            fi
        done
    fi

it appears that by doing the search the other way round, i.e
check if there is a blacklisted mod as dependency as opposed to a mod is blacklisted
doing so we can eliminate most of the sensitization
and chain the remaining san. to the modprobe,
...
long story short
the code above appears to be about 1.6 times faster than the current implementation

Last edited by kumico (2008-03-18 10:53:47)

shining · 2008-03-18 10:41:54

What about module names that are contained in others?
For example, cn appears in ipt_ecn pcnet_cs pcnet32 and icn

kumico · 2008-03-18 10:47:51

that's what the .ko is for
---
woops, bad copy posted
will update to add *"$blackmod.ko"*
----
actually
*"/$blackmod.ko"*

Last edited by kumico (2008-03-18 10:53:27)

shining · 2008-03-18 11:04:19

Ah, of course. That's interesting indeed

klixon · 2008-03-18 12:39:10

Great job!

edit: Cool, this also gets rid of the (bash-specific) character replacements. Does this mean we can now run this script in dash?

Last edited by klixon (2008-03-18 12:40:27)

kumico · 2008-03-18 13:01:19

thanks,
no not yet, there is still a lot of bashism going on in there lol
when i get some time i'll read up on posix-shells
and see if i can get a decent bash-free implementation going

shining · 2008-03-18 13:04:23

kumico wrote:

@phrakture:
hmm, i wanna boot the system in dash now... *itch*itch* should i scratch it?

On this subject, you might want to read this thread if you didn't already :
http://bbs.archlinux.org/viewtopic.php?id=40479

kumico · 2008-03-24 21:49:31

i haven't reasd anything on posix* but thinking about it, it seems to be more work than it's worth
to try and make this bash-free, as we'd then have to do something about rc.conf, etc

shining · 2008-03-24 21:55:28

kumico wrote:

i haven't reasd anything on posix* but thinking about it, it seems to be more work than it's worth
to try and make this bash-free, as we'd then have to do something about rc.conf, etc

That's probably why it hasn't been done

Misfit138 · 2008-03-26 20:17:51

This was discussed here: http://bbs.archlinux.org/viewtopic.php?id=40479
Haven't heard anything else about it.

Arch Linux

#1 2008-03-02 16:41:48

in need of some help understanding load-modules.sh and the boot proces

#2 2008-03-17 08:18:01

Re: in need of some help understanding load-modules.sh and the boot proces

#3 2008-03-17 15:33:08

Re: in need of some help understanding load-modules.sh and the boot proces

#4 2008-03-17 16:23:11

Re: in need of some help understanding load-modules.sh and the boot proces

#5 2008-03-17 16:43:17

Re: in need of some help understanding load-modules.sh and the boot proces

#6 2008-03-17 19:26:15

Re: in need of some help understanding load-modules.sh and the boot proces

#7 2008-03-17 19:56:51

Re: in need of some help understanding load-modules.sh and the boot proces

#8 2008-03-17 20:56:00

Re: in need of some help understanding load-modules.sh and the boot proces

#9 2008-03-17 20:57:32

Re: in need of some help understanding load-modules.sh and the boot proces

#10 2008-03-17 21:25:21

Re: in need of some help understanding load-modules.sh and the boot proces

#11 2008-03-17 21:35:11

Re: in need of some help understanding load-modules.sh and the boot proces

#12 2008-03-17 23:24:06

Re: in need of some help understanding load-modules.sh and the boot proces

#13 2008-03-17 23:52:30

Re: in need of some help understanding load-modules.sh and the boot proces

#14 2008-03-18 10:41:54

Re: in need of some help understanding load-modules.sh and the boot proces

#15 2008-03-18 10:47:51

Re: in need of some help understanding load-modules.sh and the boot proces

#16 2008-03-18 11:04:19

Re: in need of some help understanding load-modules.sh and the boot proces

#17 2008-03-18 12:39:10

Re: in need of some help understanding load-modules.sh and the boot proces

#18 2008-03-18 13:01:19

Re: in need of some help understanding load-modules.sh and the boot proces

#19 2008-03-18 13:04:23

Re: in need of some help understanding load-modules.sh and the boot proces

#20 2008-03-24 21:49:31

Re: in need of some help understanding load-modules.sh and the boot proces

#21 2008-03-24 21:55:28

Re: in need of some help understanding load-modules.sh and the boot proces

#22 2008-03-26 20:17:51

Re: in need of some help understanding load-modules.sh and the boot proces

Board footer