You are not logged in.

#1 2010-10-12 00:30:06

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

log EVERY module your kernel used for efficient kernel builds

Here's a trivial bash script that will keep track of EVERY kernel module in uses every.  Why?  For those of us who are obsessed with a minimal kernel - compile in the right kernel modules via a make localmodconfig tongue

My advice is to cron up this script and use your system for a few weeks which should hopefully catch all needed modules.  You can then cat the log into a modprobe prior to your kernel compilation like so:

$ sudo modprobe -a $(cat /var/log/modprobe.long)

http://aur.archlinux.org/packages.php?ID=41689

Comments/suggestions are welcomed.  Enjoy!

Last edited by graysky (2010-10-12 00:38:14)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#2 2010-10-12 02:08:50

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: log EVERY module your kernel used for efficient kernel builds

Ok, question. Why are you using md5sum? Why not something like,

...

sort ${db} <(awk '{print $1}' /proc/modules) | uniq > /tmp/.new

nnew="`comm -13 ${db} /tmp/.new | wc -l`"
if [[ ${nnew} -gt 0 ]]
   echo ${nnew} new modules detected, updating database
   mv /tmp/.new ${db}

...

edit: removed redundant sort
edit2: removed redundant cat

Last edited by fsckd (2010-10-12 16:03:11)


aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#3 2010-10-12 02:35:51

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: log EVERY module your kernel used for efficient kernel builds

What's the difference between

cat /proc/modules | awk '{print $1}' | sort -k 1,1

and

cat /proc/modules | cut -d " " -f 1 | sort

?

dbsize=`wc -l ${db}`
echo "The db currently contains this many records: "${dbsize}
dbsize=$(cat $db| wc -l)
echo "The db currently contains $dbsize records."

Same thing for $newdbsize.

Printing md5sums to a log is OK, to stdout ...? Or you can do as fsckd suggested.

Offline

#4 2010-10-12 02:48:54

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: log EVERY module your kernel used for efficient kernel builds

karol wrote:

What's the difference between

dbsize=`wc -l ${db}`
echo "The db currently contains this many records: "${dbsize}
dbsize=$(cat $db| wc -l)
echo "The db currently contains $dbsize records."

One has a bad cat... Never cat a file just to pipe it to another command.

Offline

#5 2010-10-12 02:52:07

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: log EVERY module your kernel used for efficient kernel builds

Allan wrote:

One has a bad cat... Never cat a file just to pipe it to another command.

But you get just the number of lines:

[karol@black blah]$ wc -l somefile
61 somefile
[karol@black blah]$ cat somefile | wc -l
61

so it looks pretty / saves you a 'cut -d " " -f 1'.

Edit: Don't like the cat - fine: 'grep -Ec [[:alpha:]] somefile' ;P

Last edited by karol (2010-10-12 03:00:23)

Offline

#6 2010-10-12 03:05:50

lolilolicon
Member
Registered: 2009-03-05
Posts: 1,722

Re: log EVERY module your kernel used for efficient kernel builds

Good idea graysky. Here's my approach, supposedly cleaner (I think).

#!/bin/bash

db=/tmp/modules.db # change this

if [ -f "${db}" ]; then
  mod_list=$(LC_ALL=C sort -u "${db}" <(cut -d\  -f1 /proc/modules))
  if [ "${mod_list}" != "$(cat "${db}")" ]; then
    echo "Found new modules" >&2
    echo "${mod_list}" > "${db}"
  fi
else
  mod_list=$(LC_ALL=C sort -u <(cut -d\  -f1 /proc/modules))
  echo "${mod_list}" > "${db}"
fi

Edit: rid of "one bad cat" (x2 tongue)

Last edited by lolilolicon (2010-10-12 03:10:23)


This silver ladybug at line 28...

Offline

#7 2010-10-12 10:29:06

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: log EVERY module your kernel used for efficient kernel builds

Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?
Just in case, I knocked this up:

db=${1:-"/var/log/modprobe.db"}
fmt=${2:-"%a %b %d %H:%M:%S"}

gawk -v fmt="${fmt}" -v db="${db}" '
      BEGIN {
        while ((getline < db) > 0)
          mods[$1] = $1
      }

      $1 ~ /^.+$/ {mods[$1] = $1}

      END {
        n = asorti(mods)
        for (i = 1; i <= n; i++)
          printf "%s ",mods[i] > db
        printf "%s : There are %d mods recorded\n",strftime(fmt),n
     }' /proc/modules

I've not tested it much, so it's only run with the same number of mods loaded.
It should work though as the array is associative and indexed by the mod name, so new modules will generate new array items. That's the theory, anyway. smile

NOTE: It uses gawk for the sort functionality. I have seen sorts in basic awk but would need to look them up.

Last edited by skanky (2010-10-12 10:31:03)


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#8 2010-10-12 10:46:32

lolilolicon
Member
Registered: 2009-03-05
Posts: 1,722

Re: log EVERY module your kernel used for efficient kernel builds

karol wrote:
Allan wrote:

One has a bad cat... Never cat a file just to pipe it to another command.

But you get just the number of lines:

[karol@black blah]$ wc -l somefile
61 somefile
[karol@black blah]$ cat somefile | wc -l
61

so it looks pretty / saves you a 'cut -d " " -f 1'.

Edit: Don't like the cat - fine: 'grep -Ec [[:alpha:]] somefile' ;P

How about:

$ wc -l <somefile
61

This silver ladybug at line 28...

Offline

#9 2010-10-12 10:54:08

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: log EVERY module your kernel used for efficient kernel builds

@ lolilolicon
That's nice, no more useless cats. Why haven't I thought of that? ;P

/sneaks out to rewrite some personal scripts

Offline

#10 2010-10-12 15:02:41

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: log EVERY module your kernel used for efficient kernel builds

skanky wrote:

Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?

It's supposed to record all of the possible modules you may use. So the list will never shrink. Only grow, like a blob eating everything in its path until it consumes the world! 0_0


aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#11 2010-10-12 15:54:30

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: log EVERY module your kernel used for efficient kernel builds

fsckd wrote:
skanky wrote:

Nice idea. One potential issue I *think* I see with the code is that it doesn't distinguish between mod lists that shrink --- however is that likely?

It's supposed to record all of the possible modules you may use. So the list will never shrink. Only grow, like a blob eating everything in its path until it consumes the world! 0_0

Yes, that's what's supposed to happen. The way I read the code (which was this morning, and I may have misread) was that if the list changes in anyway, the new version overrides the old. If the new version has fewer modules, then you'd lose some. I think my code avoids that.


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#12 2010-10-12 16:01:01

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: log EVERY module your kernel used for efficient kernel builds

skanky wrote:

The way I read the code (which was this morning, and I may have misread) was that if the list changes in anyway, the new version overrides the old. If the new version has fewer modules, then you'd lose some. I think my code avoids that.

You did misread. wink See the line

sort -k 1,1 ${db} /tmp/.test | uniq > /tmp/.new

aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#13 2010-10-12 16:06:34

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: log EVERY module your kernel used for efficient kernel builds

Ah yeah. Whoops.
Saw the uniq, then promptly forgot about it. hmm

Anyway, fwiw, that's what putting the module name as the index in the assoc array should do, too.


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#14 2010-10-13 21:24:17

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: log EVERY module your kernel used for efficient kernel builds

Thanks for the great discussion, all.  I modded the script taking into consideration the topics raised.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#15 2010-10-14 03:03:54

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: log EVERY module your kernel used for efficient kernel builds

Another possible solution. I've been finding infinite use for comm lately. The redirect back to the $DBFILE works because of process substitution. Hurray.

#!/bin/bash

DBFILE=$HOME/mp.log

[[ -f $DBFILE ]] || touch "$DBFILE" || exit 1

lcbefore=$(wc -l < "$DBFILE")
comm -23 <(awk '{print $1}' /proc/modules | sort) <(sort "$DBFILE") >> "$DBFILE"
lcafter=$(wc -l < "$DBFILE")

new=$(( lcafter - lcbefore ))
(( new )) && echo "Found $new new modules"

exit 0

Offline

#16 2010-10-14 15:35:23

MutantTurkey
Member
Registered: 2010-02-07
Posts: 17

Re: log EVERY module your kernel used for efficient kernel builds

I'm glad to see someone cares about having a minimal kernel smile

Offline

#17 2010-10-15 21:57:18

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: log EVERY module your kernel used for efficient kernel builds

MutantTurkey wrote:

I'm glad to see someone cares about having a minimal kernel smile

Isn't the minimalist's approach at the foundation of the Arch Way?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#18 2010-10-15 23:46:15

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: log EVERY module your kernel used for efficient kernel builds

Simplicity is at the foundation of the "arch way", which doesn't necessarily entail minimalism. One might argue that simplicity means leaving the defaults alone and using the binary packages provided to us graciously by upstream. The time/effort expended to streamline your kernel is almost guaranteed to be far greater than the time recouped by any recognizable increases in performance.

Offline

#19 2011-11-18 23:49:48

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: log EVERY module your kernel used for efficient kernel builds

Finally got around to writing a wiki page for these two little scripts:

https://wiki.archlinux.org/index.php/Modprobed_db


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

Board footer

Powered by FluxBB