Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

iamdanhenry · 2012-01-01 01:17:06

I plan on compiling my kernel for arch from scratch. I have backed everything up on my computer and I am going to format the partitions and install a fresh copy of Arch.

I intend on following this guide to do it.

Doing some Googling and general research on the subject of compiling the kernel I mixed messages about what it can actually do for you. I am going to do it either way, for the learning experience (and bragging rights) and so I can say I have done it at least once. I also wan't to know more about the kernel, it seems like a good way to find out.

Anyways, Any advice is appreciated.

Thank you,
Daniel

headkase · 2012-01-01 01:22:48

The biggest piece of advice I can give is be familiar with the "make localmodconfig" command. When I compiled my own kernel on Debian that reduced compilation time from 2½ hours down to 20 minutes. The caveat with that one is make sure everything you need for modules are loaded at the time you issue that command. Like having a USB thumbdrive plugged with an NTFS file-system on it if you need that kind of thing!

iamdanhenry · 2012-01-01 01:27:24

headkase wrote:

The biggest piece of advice I can give is be familiar with the "make localmodconfig" command. When I compiled my own kernel on Debian that reduced compilation time from 2½ hours down to 20 minutes. The caveat with that one is make sure everything you need for modules are loaded at the time you issue that command. Like having a USB thumbdrive plugged with an NTFS file-system on it if you need that kind of thing!

To be honest, I'm not sure I know everything that I need. What happens if I forget something? I need to recompile the kernel again with that module that I forgot loaded?

/dev/zero · 2012-01-01 01:33:48

I haven't compiled my own kernel since I went through this guide from LFS.

The main trick I've learned since then was you can find out what modules you need by running,

lspci -mvk | awk '/^Module/ {print $2}' | sort -u

This will greatly simplify the decisions you need to make about what to include in your configuration.

Also, be prepared for it to take a long time to compile, and for it to not work on the first few tries.

headkase · 2012-01-01 01:40:58

As /dev/zero said, that will give you the modules you need for your hardware - but those should be loaded so they should be picked up by localmodconfig. Also as he said: be prepared to compile more than once. I specifically mentioned a USB thumbdrive and an NTFS filesystem because those are also built into the kernel and if they are not active at the time you issue make localmodconfig then they won't be compiled into your resulting kernel. Then you plug one in and... Nothing. Recompile. Ouch.

There is another compilation option that can significantly speed up the process: "-j N" where N is the number of cores you have available on your machine. The "-j" means "jobs" and it is the number of concurrent threads that will be used to compile the kernel pieces. If you have 2 cores then 2 different pieces will compile at the same time with each piece on a core. This can also really speed things up just make sure you avoid using a higher value for "N" than you have cores available.

karol · 2012-01-01 01:58:12

You may want to have a look at https://wiki.archlinux.org/index.php/Modprobed_db too.

ewaller · 2012-01-01 02:09:11

You do realize, you don't have to replace the stock kernel when you compile a new one. You can install multiple kernels and their support files are in /lib/modules in directories defined by the kernel name.

My system has 3 kernels installed. The standard Arch kernel, the -ck kernel from AUR, and (Because you can take the boy away from Gentoo, but you can't take Gentoo out of the boy) a kernel that has been downloaded from kernel.org and custom built for this laptop. All three kernels are on my boot partition and are selected by name in my grub entries.

When you screw up and build a kernel that won't boot your hardware, just reboot and pick one known to work. Fix the broken kernel, and try again. repeat.

iamdanhenry · 2012-01-01 22:30:01

/dev/zero wrote:

I haven't compiled my own kernel since I went through this guide from LFS.
The main trick I've learned since then was you can find out what modules you need by running,
lspci -mvk | awk '/^Module/ {print $2}' | sort -u
This will greatly simplify the decisions you need to make about what to include in your configuration.
Also, be prepared for it to take a long time to compile, and for it to not work on the first few tries.

I ran that and all I got was

ahci
amd64_edac_mod
ata_generic
ehci-hcd
i2c-piix4
k10temp
nouveau
nvidia
nvidiafb
ohci-hcd
pata_acpi
pata_atiixp
r8169
shpchp
snd-hda-intel
sp5100_tco
xhci-hcd

That's all the modules that I need? I guess I expected more.

/dev/zero · 2012-01-01 22:39:33

iamdanhenry wrote:

ahci
amd64_edac_mod
ata_generic
ehci-hcd
i2c-piix4
k10temp
nouveau
nvidia
nvidiafb
ohci-hcd
pata_acpi
pata_atiixp
r8169
shpchp
snd-hda-intel
sp5100_tco
xhci-hcd

That's all the modules that I need? I guess I expected more.

I guess I would consider it a minimum to get all your on-board hardware working. You can include more, if some of the other kernel options seem interesting or useful, but maybe treat all that as optional until you come across evidence to the contrary. When I've compiled kernels in the past, I just did a depth-first search on all the available options looking for interesting and useful things to include :-)

irrlicht · 2012-01-01 22:44:13

this is just essintials hardware modules for hardware that are hooked up to the pci bus.

/dev/zero · 2012-01-01 22:46:05

irrlicht wrote:

this is just essintials hardware modules for hardware that are hooked up to the pci bus.

So, I could use some clarification on this, myself. Do you mean that OP does need more, or is the given list sufficient?

headkase · 2012-01-01 22:47:53

@iamdanhenry,

The very first thing you should do is follow ewaller's suggestion and learn how to have multiple kernels with Arch. I don't know how to do that yet as I'm an Arch newbie: my experience with compiling kernel's is on Debian. The reason you want multiple kernels is exactly as ewaller said: what if your shiny new kernel won't even boot the system.

As for the modules being listed: that's just the hardware in your machine. Not all modules you may need may be loaded until you actually need them. Remember my USB/NTFS example? Well, there's also the "character encoding" needed to mount an NTFS volume. You can have both USB and NTFS support compiled into your kernel and still not be able to plug a thumbdrive successfully with an NTFS filesystem because your kernel doesn't have another piece: the character encoding used on the filesystem.

If you are unsure of which modules you need then there's perhaps do not use localmodconfig and instead just use the jobs number. More jobs will still compile the kernel faster and if you don't use localmodconfig you'll just have a resulting "and the kitchen sink" kernel.

Now the link karol gave might be useful too. From my gross understanding of it is that it has an option to keep a record of "all modules ever modprobed." This could - if I'm understanding it right - help greatly in building a successful localmodconfig kernel that has everything you need. You'd just have to build up a history of using your system day-to-day and have that package build up a list of everything you happen to need. Once you have that history then compile a minimal kernel.

Another thing to explore - and I'm not certain of the compiler flags: perhaps someone else can help - is to enable specific compilation options for your machine. For example, if your CPU flags include SSE2 support then compile your kernel with that flag set. The benefit of setting compilation flags based on your actual processor's abilities is that in the end you may end up with a slightly faster overall machine.

irrlicht · 2012-01-01 23:19:12

/dev/zero wrote:

irrlicht wrote:
this is just essintials hardware modules for hardware that are hooked up to the pci bus.
So, I could use some clarification on this, myself. Do you mean that OP does need more, or is the given list sufficient?

the OP does need more: usb devices, file system, probably encryption and pseudo device modules. Depends on the requirements. Invoke lsmod and you'll see much more loaded modules.

Last edited by irrlicht (2012-01-01 23:19:33)

iamdanhenry · 2012-01-01 23:53:53

If you are unsure of which modules you need then there's perhaps do not use localmodconfig and instead just use the jobs number. More jobs will still compile the kernel faster and if you don't use localmodconfig you'll just have a resulting "and the kitchen sink" kernel.

Is there anything essentialy wrong with this aside from that fact that it isn't streamlined for my machine?

I am not too worried about compile times, I am patient plus I have 6 cores so the jobs flag should speed it up. I more want the learning experience of doing it. Then maybe do it again a little better. On another note though, My computer isn't a supercomputer so I could always use anther couple percent free resources.

falconindy · 2012-01-01 23:54:46

/dev/zero wrote:

I guess I would consider it a minimum to get all your on-board hardware working.

It's not enough. Rather, it's missing quite a lot (some of them more important than others). I would recommend looking at what mkinitcpio in the auto_modules() function (/lib/initcpio/functions)

headkase · 2012-01-01 23:59:27

Nope there's nothing wrong with a "and the kitchen sink" kernel. That's what you're running right now! The default Arch kernel is some odd 40MB in size. If you compiled a minimal kernel you could get that size down to less than 10MB. It would be slightly faster, especially with compilation flags set properly for your CPU, however: given the abundance of resources nowadays - CPU speed, RAM, HD Space - a minimal kernel isn't so much a requirement than it is personal taste.

Since you have 6 available cores then compilation time should be fairly speedy. I'm running a Pentium Dual Core machine @ 1.6Ghz and my full time like said is 2½ hours and minimal 20 minutes. Just because you have 6 cores available I wouldn't be shocked if with 6 jobs your full compile time was 20 minutes.

iamdanhenry · 2012-01-02 02:43:10

Thanks a ton for the advice guys. I must say, as a linux user for about 4-5 years now. I have learned more about how Linux works in the couple weeks of Arch use than I did in the years of using Ubuntu/Debian.

Arch Linux

#1 2012-01-01 01:17:06

Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#2 2012-01-01 01:22:48

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#3 2012-01-01 01:27:24

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#4 2012-01-01 01:33:48

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#5 2012-01-01 01:40:58

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#6 2012-01-01 01:58:12

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#7 2012-01-01 02:09:11

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#8 2012-01-01 22:30:01

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#9 2012-01-01 22:39:33

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#10 2012-01-01 22:44:13

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#11 2012-01-01 22:46:05

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#12 2012-01-01 22:47:53

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#13 2012-01-01 23:19:12

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#14 2012-01-01 23:53:53

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#15 2012-01-01 23:54:46

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#16 2012-01-01 23:59:27

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

#17 2012-01-02 02:43:10

Re: Compiling a Kernel. Benefits / Risks / Tips & Tricks ?

Board footer