make localmodconfig failing to function properly?

graysky · 2010-08-23 08:41:53

The announcement of several new make scripts in the 2.6.32 release notes is very exciting.

1.8. Easy local kernel configuration
Most people uses the kernel shipped by distros - and that's good. But some people like to compile their own kernels from kernel.org, or maybe they like following the Linux development and want to try it. Configuring your own kernel, however, has become a very difficult and tedious task - there're too many options, and some times userspace software will stop working if you don't enable some key option. You can use a standard distro .config file, but it takes too many time to compile all the options it enables.
To make easier the process of configuration, a new build target has been added: make localmodconfig. It runs "lsmod" to find all the modules loaded on the current running system. It will read all the Makefiles to map which CONFIG enables a module. It will read the Kconfig files to find the dependencies and selects that may be needed to support a CONFIG. Finally, it reads the .config file and removes any module "=m" that is not needed to enable the currently loaded modules. With this tool, you can strip a distro .config of all the unuseful drivers that are not needed in our machine, and it will take much less time to build the kernel. There's an additional "make localyesconfig" target, in case you don't want to use modules and/or initrds.

localmodconfig - Update current config disabling modules not loaded
localyesconfig - Update current config converting local mods to core

So if I run them in that order, first the 'make localmodconfig' should only enable modules that I have currently loaded then the 'make localyesconfig' should switch them from modules to compiled into the kernel when I build it -- this is isn't the case for me.

$ extract linux-2.6.35.tar.bz2 && cd linux-2.6.35
$ make mrproper
$ zcat /proc/config.gz > .config
$ make localmodconfig
$ make menuconfig

After running the the "make localmodconfig" script, have TONS of options that don't apply to my system flagged as modules when I inspect it via a "make menuconfig." For example:

Device Drivers>Graphics support>Direct Rendering Manager
<M> ATI Radeon
[*] Enable modesetting on radeon by default
<M> Intel 830M, 845G, 852M, 855GM, 865G
<M> i915 driver

My system does not have ATI anything on board, nor does it have that those Intel chips.

Here is another representation that there are many extra module options:

The stock ARCH kernel config:

$ cat .config | grep =m | wc -l
2434

The kernel config after I run the make localmodconfig:

cat .config-make | grep =m | wc -l
341

According to /proc/modules I only have 76 modules currently loaded:

$ cat /proc/modules | wc -l
76

What am I doing wrong? BTW, I do have /bin/lsmod linked to /sbin/lsmod as per the note on the wiki page wonder mentioned. BTW, I do have /bin/lsmod linked to /sbin/lsmod as per the note on the wiki page wonder mentioned.

archman-cro · 2010-08-23 09:28:17

I'd like to know a solution to this, too, since localmodconfig works the same as described above for me.
I always end up doing a makemenuconfig after localmodconfig in the end, cause localmodeconfig doesn't seem to do its job. Maybe we're missing something, though...

falconindy · 2010-08-23 11:25:55

Counting the number of config lines isn't a fair comparison. There's plenty of options markable as a module that don't actually produce a module: CONFIG_ATA, CONFIG_DRM, CONFIG_SND ... I'm pretty sure what you're experiencing happens to everyone, but your beef is more likely with udev and not localmodconfig. I've noticed udev trying to load intel_agp on my box (which only has an nvidia card).

I've never believed that localmodconfig is meant to build a fully trimmed, "lightweight" build. Consider it an educated suggestion, but do not rely on it 100%. Example from the other side: If you use an iPod and it hasn't been plugged in recently, localmodconfig will fail to add support for msdos filesystems, vfat, and codepage 437.

graysky · 2010-08-23 18:43:27

Agreed, but I ultimately want to run a make localyesconfig and roll only the needed modules up into the kernel itself. When I do that will all those graphics options as modules, they become hard-coded. The result is that I cannot compile the nvidia driver package because that ATI crap is part of my kernel. Plus, if I'm reading the release notes correctly, that "lightweight" build isn't the intent of the script.

rostedt · 2010-09-15 00:14:16

Note, make localmodconfig will run lsmod to find out what modules are loaded, and then it does the best to disable all module configs that are not needed to compile those modules. If you have modules loaded that are not used, it will still keep them enabled. Do a lsmod, and try to remove all modules you do not want before running make localmodconfig.

Note the original intent of the script was to make build time faster. Most people use a distro kernel which has practically every module enabled. For those that want to build there own kernel without building the world, make localmodconfig was created to help. It takes a conservative approach on disabling modules. It is better to miss a disable than to disable something you needed.

Some modules are dependent on other modules with an OR statement. Mod X depends on Mod Y || Mod Z. If this happens, it will keep both Mod Y and Mod Z if Mod X is loaded. You may only need Y and not Z, but the script may not be able to tell that.

Last edited by rostedt (2010-09-15 00:15:33)

fphillips · 2010-09-16 05:52:32

It's been a long while since I've built kernels, but don't you have to ask menuconfig to load a particular .config, otherwise it just loads default values? It seems like that's what its doing, anyway.

tomk · 2010-09-16 06:08:40

I had the same problem with localmodconfig, but I got it to work as intended by running the script - scripts/kconfig/streamline_config.pl - as recommended in the script's own comments.

Calling it from the kernel's Makefile seems to be the problem - someone should post a bug.

codycarey · 2010-09-16 06:11:15

fphillips wrote:

It's been a long while since I've built kernels, but don't you have to ask menuconfig to load a particular .config, otherwise it just loads default values? It seems like that's what its doing, anyway.

make oldconfig

graysky · 2010-09-18 09:45:34

tomk wrote:

I had the same problem with localmodconfig, but I got it to work as intended by running the script - scripts/kconfig/streamline_config.pl - as recommended in the script's own comments.
Calling it from the kernel's Makefile seems to be the problem - someone should post a bug.

Nice, tomk. Thank you. It finally works!

I added a few lines to the PKGBUILD for kernel26-ck to use this. Thanks again!

## uncomment the below to use the functional equivalent of make locamodconfig
## make SURE that you have modprobed any modules you want to have included BEFORE makepkg -s

#chmod +x ./scripts/kconfig/streamline_config.pl && ./scripts/kconfig/streamline_config.pl > config_strip
#mv config_strip .config
#make oldconfig

##

Make a difference in compile time and package size:

/lib/modules
114.1MiB [##########] /2.6.35-ARCH                                                                     
21.8MiB [#         ] /2.6.35-ck

/var/cache/pacman/pkg
23.5MiB [####      ]  kernel26-2.6.35.4-2-x86_64.pkg.tar.xz
9.3MiB [##        ]  kernel26-ck-2.6.35.4-2-x86_64.pkg.tar.xz

Last edited by graysky (2010-09-18 10:16:53)

broch · 2010-09-20 14:33:12

I think that this is only demo tool for users learning how to set first custom kernel. Not for real use. any module not loaded will be skipped and this may cause a lot of problems. Not to mention that output quality depends on original .config file /matrix

tomk · 2010-09-21 15:51:05

Well..sure, but all of that is covered in the comments at the start of the script. It's just another tool, and like any tool, it does the job it was designed for if it is used as the designer intended.

Works fine for me in "real use" - and no, this is not my first custom kernel.

broch · 2010-09-21 17:32:27

I have no doubt that this is not your first custom kernel

I draw my conclusion about localmodconfig after running following experiment:
1) create basic .config without any modifications and run localmodconfig
2) create optimized .config and run localmodconfig
3) run diff

results are really bad:
I have iwlwifi but at the moment I was connected through wired so module not loaded and missing from config file created by localmodconfig, also missing: fireware, missing some IDE stuff, but enabled ATA SFF which I don't have, enabled Tulip and 3Com and nvidia forcedeth (none of this present on my laptop)
while fireware was disabled, macintosh drivers were enabled (why?), also whole bunch of video/graphic drivers enabled (with potential issues for proper work of video card).

In conclusion I would have problems with my HDD, wireless network, video and few other things. In fact I would not be able to boot system at all: localmodconfig disabled XFS and enabled Ext3. Now I have /boot ext2 and the rest of partitions on xfs.
While I understand that Ext3 is preferable, this error kills any real use for localmodconfig. Whoever wrote this script should (instead of enabling all useless network drivers) make sure that at least system can boot so if there is no proven way of finding what fs system uses, then enable them all. Definitely more important than network.

In contrast when I was using as a "original" .config a file that I optimized, localmodconfig script left it almost unchanged, or rather changes made were not a big deal.

Now if I have to go and customize after localmodconfig customization, then this is useless tool not for any advanced user but in general for any real use as kernel configuration tool.

I do understand limitations of pre-defined configs, but this means that I do not have any use for something that will generate in effect poor quality product. I could leave .config as it is created by first run of old and tried menuconfig.

Last edited by broch (2010-09-21 17:43:44)

tomk · 2010-09-21 20:14:34

OK - you've just proved my point.

You didn't use it as the writer intended, so it didn't work as intended. Start with a typical distro config, like the Arch one. Make sure all your devices are attached, and all the modules that you want to use with your slimmed-down kernel are loaded. Run the script - and I mean run the script, not 'make localmodconfig' - this thread exists because that make target appears to be broken.

Alternatively, just don't use it - you seem to be happy with your own way of doing things.

broch · 2010-09-21 23:49:41

tomk wrote:

OK - you've just proved my point.
You didn't use it as the writer intended, so it didn't work as intended. Start with a typical distro config, like the Arch one. Make sure all your devices are attached, and all the modules that you want to use with your slimmed-down kernel are loaded. Run the script - and I mean run the script, not 'make localmodconfig' - this thread exists because that make target appears to be broken.
Alternatively, just don't use it - you seem to be happy with your own way of doing things.

I would not know how to make fs more visible that it is now and missing xfs (this would render my system not bootable). I believe that this is most severe problem.

While I have no doubts that someone with experience will not have any problems fixing such issues, I think that for novice this might be too much. Still good learning tool.

I am not trashing the script/trolling, my intention is only to caution before localmodconfig will be added to any PKGBUILD. That is all.

Last edited by broch (2010-09-21 23:52:18)

tomk · 2010-09-22 06:01:36

Agreed - this is not something to be used in any automated context. The results have to be checked.

graysky · 2010-09-23 20:12:45

tomk wrote:

Agreed - this is not something to be used in any automated context. The results have to be checked.

Agreed. It took me several iterations before I got all the modules that I need (I think)! Here are a few more statistics about running the compile with this option.

Processor: X3360 @ 3.40 GHz (Xeon version of the Q9550)
Memory: 8 GB of DDR2 @ 1,066.
Compiling in /dev/shm as we all should!

Stats for the kernel26-ck package without this option:

real compile time: 12m2.616s
# of modules: 2,572
package size: 28.54 MB
installed size: 141.24 MB

Stats for the kernel26-ck package with this option after first having modprobed a total of 95 modules:

real compile time: 2m30.607s
# of modules: 131
package size: 10.09 MB
installed size: 46.42 MB

Totals:

Compile time: 6x faster
# of modules: 20x less
Footprint on FS: 3x less

Pretty significant differences

FYI: here is the module set:

modprobe -a usb_storage loop udf nls_utf8 isofs twofish cifs vfat ext2 cryptd nls_cp437 aes-x86_64 serpent xts gf128mul dm_crypt fat aes_generic twofish_common dm_mod

Also see:

Last edited by graysky (2010-10-10 10:29:57)

tvale · 2010-09-24 00:17:29

Is there any helpful guide for a kernel modules trim? I'm not sure how can I know which modules I can safely disable!

karol · 2010-09-24 00:19:37

tvale wrote:

Is there any helpful guide for a kernel modules trim? I'm not sure how can I know which modules I can safely disable!

This seems like a good advice: https://bbs.archlinux.org/viewtopic.php … 28#p829428

graysky · 2010-09-24 08:07:15

tvale wrote:

Is there any helpful guide for a kernel modules trim? I'm not sure how can I know which modules I can safely disable!

EDIT: I have a script in the AUR that keeps track of your modules for easy recall.

http://aur.archlinux.org/packages.php?ID=41689

Enjoy!

Last edited by graysky (2010-10-25 20:32:17)

graysky · 2010-10-25 20:33:35

tomk wrote:

I had the same problem with localmodconfig, but I got it to work as intended by running the script - scripts/kconfig/streamline_config.pl - as recommended in the script's own comments.
Calling it from the kernel's Makefile seems to be the problem - someone should post a bug.

I finally got around to emailing the script's author, pointing him to this thread

rostedt · 2010-10-26 03:44:44

I'll reply with email too. But I do not see "make localmodconfig" as broken. You say you run the script directly. The difference that makes is that the script will not examine the Kconfig files. Yes, this means that it will not select as many things, but it is more likely not to boot.

What the Kconfig files give you is the select and dependencies of configs. There are options that have a <M> state that do not translate into an actual file. But you may have a module that depends on that option being selected. If you run the script as you have, it will disable all module configs that do not have a direct association with a module, and that may disable the module you need when running menuconfig. For example: CONFIG_USB. If that is a module, you may be quite surprised at what you get when you run the script directly.

What you are seeing with the radeon and intel, is that you probably have one of those modules loaded, or a module that happened to have a dependency on one of those modules.

To keep the code less complex, any dependency is used. If module A depends on B or C, it will keep B and C even though only B is loaded. I may change this in the future, but so far I have not seen the point.

BTW, you do not need a .config in your source to run "make menuconfig", if no .config is around, it will look for one to use. The order it looks is: ".config", "/proc/config.gz", "/boot/config-<uname -r>", "/boot/vmlinuz-<uname -r>" (embedded configs), "vmlinux", "/lib/modules/<uname -r>/kernel/kernel/configs.ko, "kernel/configs.ko", "kernel/configs.o".

If you want to use a different lsmod (say from another machine). You can capture the output of the lsmod from another box "lsmod > mymods", and use that version in localmodconfig:

make LSMODE=mymods localmodconfig

tomk · 2010-10-26 07:32:22

OK - you're the author, so I guess you know what you're talking about.

All I can tell you is that when I run "make localmodconfig", it does not "turn off all the modules that are not loaded on your system", to quote from the "What it does?" comments at the top of the script.

When I run the script directly, as described in the "Howto" comments, it works as described.

rostedt · 2010-10-26 12:43:36

Yeah, the original way (without reading the Kconfigs) was a bit more aggressive in disabling modules.But because it ignored the dependency structure of the build system, it would also remove dependencies of required modules. Then I got bug reports from people telling me that they plugged stuff in and it still did not keep those modules around. Thus the script was changed to remove everything that is not related to your modules. Unfortunately, the way the kconfig system works, if a module is set via a dependency on several other modules, all of them are considered related to your system. It may look like these modules should not be there, but the build system says otherwise.

I may revisit the script and add more logic to filter more out. If mod A depends on B or C and C has a direct relation to a module that is already loaded, then I can disable B. But developers got pretty fancy in their dependencies, so this logic will need to cope with that.

I wrote streamline_config.pl years ago (2005) as I was developing on several machines and I needed to cut down the time of the build. I did not have the time to figure out the modules for each of these machines (I had a new box every week), so I wrote this script to remove the modules not needed to boot the box, and speed up my productivity in compiling kernels.

I think it was 2 years ago at Kernel Summit, the topic came up that when someone reports a bug, and the developer asks the user to perform a git bisect to find the change that cause the bug, the user is reluctant because they only have a distro config and do not know enough to thin it down. A git bisect takes usually 13 full kernel compiles. This could take a week for someone with an old machine and a distro config. Linus Torvalds started yelling at the developers, asking why we do not have something that can trim down the config to speed up compile times for the users. One of the developers in attendance, that uses my script, answered him saying "We do, it's Steve Rostedt's streamline_config". Linus then asked why its not in the kernel already. The rest is history ;-)

So there you have it. The rational for why make localmodconfig is in the kernel.

My take is to make sure everything is enabled that will keep all your modules enabled too, even if it means keeping modules enabled that may not bee needed.

tomk · 2010-10-27 01:22:18

Understood, thanks for the explanation.

doomguard88 · 2011-06-04 18:10:36

One question. Could we use the output of

hwdetect --modules
hwdetect --modules-not-loaded

to load all the necesserary modules before running

$ make localmodconfig

thx in advance

Arch Linux

#1 2010-08-23 08:41:53

make localmodconfig failing to function properly?

#2 2010-08-23 09:28:17

Re: make localmodconfig failing to function properly?

#3 2010-08-23 11:25:55

Re: make localmodconfig failing to function properly?

#4 2010-08-23 18:43:27

Re: make localmodconfig failing to function properly?

#5 2010-09-15 00:14:16

Re: make localmodconfig failing to function properly?

#6 2010-09-16 05:52:32

Re: make localmodconfig failing to function properly?

#7 2010-09-16 06:08:40

Re: make localmodconfig failing to function properly?

#8 2010-09-16 06:11:15

Re: make localmodconfig failing to function properly?

#9 2010-09-18 09:45:34

Re: make localmodconfig failing to function properly?

#10 2010-09-20 14:33:12

Re: make localmodconfig failing to function properly?

#11 2010-09-21 15:51:05

Re: make localmodconfig failing to function properly?

#12 2010-09-21 17:32:27

Re: make localmodconfig failing to function properly?

#13 2010-09-21 20:14:34

Re: make localmodconfig failing to function properly?

#14 2010-09-21 23:49:41

Re: make localmodconfig failing to function properly?

#15 2010-09-22 06:01:36

Re: make localmodconfig failing to function properly?

#16 2010-09-23 20:12:45

Re: make localmodconfig failing to function properly?

#17 2010-09-24 00:17:29

Re: make localmodconfig failing to function properly?

#18 2010-09-24 00:19:37

Re: make localmodconfig failing to function properly?

#19 2010-09-24 08:07:15

Re: make localmodconfig failing to function properly?

#20 2010-10-25 20:33:35

Re: make localmodconfig failing to function properly?

#21 2010-10-26 03:44:44

Re: make localmodconfig failing to function properly?

#22 2010-10-26 07:32:22

Re: make localmodconfig failing to function properly?

#23 2010-10-26 12:43:36

Re: make localmodconfig failing to function properly?

#24 2010-10-27 01:22:18

Re: make localmodconfig failing to function properly?

#25 2011-06-04 18:10:36

Re: make localmodconfig failing to function properly?

Board footer