log EVERY module your kernel used for efficient kernel builds

graysky · 2010-10-12 00:30:06

Here's a trivial bash script that will keep track of EVERY kernel module in uses every. Why? For those of us who are obsessed with a minimal kernel - compile in the right kernel modules via a make localmodconfig

My advice is to cron up this script and use your system for a few weeks which should hopefully catch all needed modules. You can then cat the log into a modprobe prior to your kernel compilation like so:

$ sudo modprobe -a $(cat /var/log/modprobe.long)

http://aur.archlinux.org/packages.php?ID=41689

Comments/suggestions are welcomed. Enjoy!

Last edited by graysky (2010-10-12 00:38:14)

fsckd · 2010-10-12 02:08:50

Ok, question. Why are you using md5sum? Why not something like,

...

sort ${db} <(awk '{print $1}' /proc/modules) | uniq > /tmp/.new

nnew="`comm -13 ${db} /tmp/.new | wc -l`"
if [[ ${nnew} -gt 0 ]]
   echo ${nnew} new modules detected, updating database
   mv /tmp/.new ${db}

...

edit: removed redundant sort
edit2: removed redundant cat

Last edited by fsckd (2010-10-12 16:03:11)

karol · 2010-10-12 02:35:51

What's the difference between

cat /proc/modules | awk '{print $1}' | sort -k 1,1

and

cat /proc/modules | cut -d " " -f 1 | sort

?

dbsize=`wc -l ${db}`
echo "The db currently contains this many records: "${dbsize}

dbsize=$(cat $db| wc -l)
echo "The db currently contains $dbsize records."

Same thing for $newdbsize.

Printing md5sums to a log is OK, to stdout ...? Or you can do as fsckd suggested.

Allan · 2010-10-12 02:48:54

karol wrote:

What's the difference between

dbsize=`wc -l ${db}`
echo "The db currently contains this many records: "${dbsize}

dbsize=$(cat $db| wc -l)
echo "The db currently contains $dbsize records."

One has a bad cat... Never cat a file just to pipe it to another command.

karol · 2010-10-12 02:52:07

Allan wrote:

One has a bad cat... Never cat a file just to pipe it to another command.

But you get just the number of lines:

[karol@black blah]$ wc -l somefile
61 somefile
[karol@black blah]$ cat somefile | wc -l
61

so it looks pretty / saves you a 'cut -d " " -f 1'.

Edit: Don't like the cat - fine: 'grep -Ec [[:alpha:]] somefile' ;P

Last edited by karol (2010-10-12 03:00:23)

lolilolicon · 2010-10-12 03:05:50

Good idea graysky. Here's my approach, supposedly cleaner (I think).

#!/bin/bash

db=/tmp/modules.db # change this

if [ -f "${db}" ]; then
  mod_list=$(LC_ALL=C sort -u "${db}" <(cut -d\  -f1 /proc/modules))
  if [ "${mod_list}" != "$(cat "${db}")" ]; then
    echo "Found new modules" >&2
    echo "${mod_list}" > "${db}"
  fi
else
  mod_list=$(LC_ALL=C sort -u <(cut -d\  -f1 /proc/modules))
  echo "${mod_list}" > "${db}"
fi

Edit: rid of "one bad cat" (x2 )

Last edited by lolilolicon (2010-10-12 03:10:23)

skanky · 2010-10-12 10:29:06

Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?
Just in case, I knocked this up:

db=${1:-"/var/log/modprobe.db"}
fmt=${2:-"%a %b %d %H:%M:%S"}

gawk -v fmt="${fmt}" -v db="${db}" '
      BEGIN {
        while ((getline < db) > 0)
          mods[$1] = $1
      }

      $1 ~ /^.+$/ {mods[$1] = $1}

      END {
        n = asorti(mods)
        for (i = 1; i <= n; i++)
          printf "%s ",mods[i] > db
        printf "%s : There are %d mods recorded\n",strftime(fmt),n
     }' /proc/modules

I've not tested it much, so it's only run with the same number of mods loaded.
It should work though as the array is associative and indexed by the mod name, so new modules will generate new array items. That's the theory, anyway.

NOTE: It uses gawk for the sort functionality. I have seen sorts in basic awk but would need to look them up.

Last edited by skanky (2010-10-12 10:31:03)

lolilolicon · 2010-10-12 10:46:32

karol wrote:

Allan wrote:
One has a bad cat... Never cat a file just to pipe it to another command.
But you get just the number of lines:
[karol@black blah]$ wc -l somefile
61 somefile
[karol@black blah]$ cat somefile | wc -l
61
so it looks pretty / saves you a 'cut -d " " -f 1'.
Edit: Don't like the cat - fine: 'grep -Ec [[:alpha:]] somefile' ;P

How about:

$ wc -l <somefile
61

karol · 2010-10-12 10:54:08

@ lolilolicon
That's nice, no more useless cats. Why haven't I thought of that? ;P

/sneaks out to rewrite some personal scripts

fsckd · 2010-10-12 15:02:41

skanky wrote:

Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?

It's supposed to record all of the possible modules you may use. So the list will never shrink. Only grow, like a blob eating everything in its path until it consumes the world! 0_0

skanky · 2010-10-12 15:54:30

fsckd wrote:

skanky wrote:
Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?
It's supposed to record all of the possible modules you may use. So the list will never shrink. Only grow, like a blob eating everything in its path until it consumes the world! 0_0

Yes, that's what's supposed to happen. The way I read the code (which was this morning, and I may have misread) was that if the list changes in anyway, the new version overrides the old. If the new version has fewer modules, then you'd lose some. I think my code avoids that.

fsckd · 2010-10-12 16:01:01

skanky wrote:

The way I read the code (which was this morning, and I may have misread) was that if the list changes in anyway, the new version overrides the old. If the new version has fewer modules, then you'd lose some. I think my code avoids that.

You did misread. See the line

sort -k 1,1 ${db} /tmp/.test | uniq > /tmp/.new

skanky · 2010-10-12 16:06:34

Ah yeah. Whoops.
Saw the uniq, then promptly forgot about it.

Anyway, fwiw, that's what putting the module name as the index in the assoc array should do, too.

graysky · 2010-10-13 21:24:17

Thanks for the great discussion, all. I modded the script taking into consideration the topics raised.

falconindy · 2010-10-14 03:03:54

Another possible solution. I've been finding infinite use for comm lately. The redirect back to the $DBFILE works because of process substitution. Hurray.

#!/bin/bash

DBFILE=$HOME/mp.log

[[ -f $DBFILE ]] || touch "$DBFILE" || exit 1

lcbefore=$(wc -l < "$DBFILE")
comm -23 <(awk '{print $1}' /proc/modules | sort) <(sort "$DBFILE") >> "$DBFILE"
lcafter=$(wc -l < "$DBFILE")

new=$(( lcafter - lcbefore ))
(( new )) && echo "Found $new new modules"

exit 0

MutantTurkey · 2010-10-14 15:35:23

I'm glad to see someone cares about having a minimal kernel

graysky · 2010-10-15 21:57:18

MutantTurkey wrote:

I'm glad to see someone cares about having a minimal kernel

Isn't the minimalist's approach at the foundation of the Arch Way?

falconindy · 2010-10-15 23:46:15

Simplicity is at the foundation of the "arch way", which doesn't necessarily entail minimalism. One might argue that simplicity means leaving the defaults alone and using the binary packages provided to us graciously by upstream. The time/effort expended to streamline your kernel is almost guaranteed to be far greater than the time recouped by any recognizable increases in performance.

graysky · 2011-11-18 23:49:48

Finally got around to writing a wiki page for these two little scripts:

https://wiki.archlinux.org/index.php/Modprobed_db

Arch Linux

#1 2010-10-12 00:30:06

log EVERY module your kernel used for efficient kernel builds

#2 2010-10-12 02:08:50

Re: log EVERY module your kernel used for efficient kernel builds

#3 2010-10-12 02:35:51

Re: log EVERY module your kernel used for efficient kernel builds

#4 2010-10-12 02:48:54

Re: log EVERY module your kernel used for efficient kernel builds

#5 2010-10-12 02:52:07

Re: log EVERY module your kernel used for efficient kernel builds

#6 2010-10-12 03:05:50

Re: log EVERY module your kernel used for efficient kernel builds

#7 2010-10-12 10:29:06

Re: log EVERY module your kernel used for efficient kernel builds

#8 2010-10-12 10:46:32

Re: log EVERY module your kernel used for efficient kernel builds

#9 2010-10-12 10:54:08

Re: log EVERY module your kernel used for efficient kernel builds

#10 2010-10-12 15:02:41

Re: log EVERY module your kernel used for efficient kernel builds

#11 2010-10-12 15:54:30

Re: log EVERY module your kernel used for efficient kernel builds

#12 2010-10-12 16:01:01

Re: log EVERY module your kernel used for efficient kernel builds

#13 2010-10-12 16:06:34

Re: log EVERY module your kernel used for efficient kernel builds

#14 2010-10-13 21:24:17

Re: log EVERY module your kernel used for efficient kernel builds

#15 2010-10-14 03:03:54

Re: log EVERY module your kernel used for efficient kernel builds

#16 2010-10-14 15:35:23

Re: log EVERY module your kernel used for efficient kernel builds

#17 2010-10-15 21:57:18

Re: log EVERY module your kernel used for efficient kernel builds

#18 2010-10-15 23:46:15

Re: log EVERY module your kernel used for efficient kernel builds

#19 2011-11-18 23:49:48

Re: log EVERY module your kernel used for efficient kernel builds

Board footer