[SOLVED] Compile and Install a Parallel Kernel

akhilbehl · 2010-04-16 01:09:28

Hey,

I believe the topic explains my aim. I would like to compile a kernel ( mainly for the learning experience and also the benefits if any are to be realized ) and install it alongside my stock *-ARCH kernel, so I can choose off grub when I boot in.

I would like to say that I have actually spent more than 5 hours searching the wiki ( and not just on Arch ) but I am still a little confused as to the method. I found these posts in the wiki / elsewhere:

http://wiki.archlinux.org/index.php/Ice … l_PKGBUILD

http://wiki.archlinux.org/index.php/Cus … n_with_ABS

http://wiki.archlinux.org/index.php/Ker … rom_Source

http://bugs.archlinux.org/task/12384

http://ubuntuforums.org/showthread.php?t=311158

However I am still quite confused. That is because most of these methods / PKGBUILDS seem to be written to replace the *-ARCH kernel. Or at least it appears like that to me. Please note that I find myself a good learner but I am nowhere near an expert on ARCH/Linux, therefore I might be wrong here and would be happy to be told so and directed in the right direction. I have only been using Arch for around 6 months (coming from Ubuntu where I started with Linux for a few months ) and am absolutely in love with Arch ( and the wiki, so much so that this is only the third time ever I have had to post on the forum ).

Another issue that seems to keep cropping up in these posts is that of lib/firmware being replaced by the new custom kernel and the modules of each kernel conflicting with those of the other. I read that this problem apparently has been solved by packaging the firmware seperately, but again my reading of these posts may not be very accurate. If one of you could choose the right PKGBUILD from one of these links or provide one of your own, it would be great help. I am keen to learn, so partial solutions / guidelines are also appreciated.

P.S.: Btw, are there any substantial benefits to be had from compiling your own kernel in terms of performance, besides the learning?
P.P.S.: I read the wiki on how to post on the forum and hope that I am doing well by the guidelines!

Last edited by akhilbehl (2010-04-16 21:15:59)

lilsirecho · 2010-04-16 01:50:49

There are many posts regarding compiling a custom kernel in arch.

Try "search" for such to identify the performance projected for custom kernels.

Many posts discuss the pros and cons.........

Ber · 2010-04-16 08:33:56

IMHO (I insist on that), if you want a first experience with kernel compilation (you wouldn't gain anything in terms of performance, so this is no valid reason, a valid reason would be to have one more point in "geekitude" tests), I woudl first do it "the old way" (it's a condensed resume, please look for tutorials on the internet) :

- download the kernel sources from kernel.org
- untar to any place and cd to it
- zcat /proc/config.gz > .config (configure your kernel with the current kernel's config file)
- make oldconfig (answer some questions) then make menuconfig and take a look at the options. This may take time but its instructive
- don't forget to change "General Setup > local Version" to a custom one
- save config
- make as user
- as root : make modules_install (you will then find a new folder with your modules in /lib/modules/$(kernel-version-with-local-version) )
- as root : cp arch/x86/boot/bzImage /boot/vmlinuz-my-custom-kernel && cp System.map /boot/System.map-my-custom-kernel && cp .config /boot/config-my-custom-kernel
- as root : generate cpio for this kernel (search wiki, I think it's mkinitcpio -k $(kernel-version-with-local-version) )
- as root : add the corresponding entry to grub/menu.lst

I've never had any problems with firmwares, a new folder is also created in /lib/firmware, named after your kernel.

One you have done and understood that, I'd start playing with PKGBUILD to automate the task and integrate it with pacman.

To uninstall this kernel, simply remove : the sources, the /lib/firmware/my-custom-kernel folder, the /lib/modules/my-custom-kernel folder, all the files you copied to boot, and the menu.lst entry. If I forget anything please correct me.

I repeat, all this is IMHO : this is how I do it and how I find it the more instructive.

Have fun

Ramses de Norre · 2010-04-16 09:17:42

As a reference, I used this PKGBUILD to build a kernel that installed alongside the one from core, every file/directory had a suffix "iwl_debug". It is for 2.6.32 but I guess it should not be too hard to adapt it. As you can see, the only changes in the PKGBUILD are in the naming section, so if you take the 2.6.33 PKGBUILD from abs and change the naming as I did you should be fine. The only thing you'll probably want to change then is the config (don't forget to update the md5sums, makepkg -g >> PKGBUILD).

Also, you might want to check out ccache when compiling kernels.

# $Id: PKGBUILD 62387 2010-01-07 21:55:35Z tpowa $
# Maintainer: Tobias Powalowski <tpowa@archlinux.org>
# Maintainer: Thomas Baechler <thomas@archlinux.org>
pkgbase="kernel26iwl_debug"
pkgname=('kernel26iwl_debug' 'kernel26-firmware' 'kernel26-headers') # Build stock -ARCH kernel
_kernelname=${pkgname#kernel26}
_basekernel=2.6.32
pkgver=${_basekernel}.3
pkgrel=1
_patchname="patch-${pkgver}-${pkgrel}-ARCH"
arch=(i686 x86_64)
license=('GPL2')
url="http://www.kernel.org"
source=(ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-$_basekernel.tar.bz2
        ftp://ftp.archlinux.org/other/kernel26/${_patchname}.bz2
        # the main kernel config files
        config config.x86_64
        # standard config files for mkinitcpio ramdisk
        kernel26.preset)

build() {
  cd ${srcdir}/linux-$_basekernel
  # Add -ARCH patches
  # See http://projects.archlinux.org/linux-2.6-ARCH.git/
  patch -Np1 -i ${srcdir}/${_patchname} || return 1

  if [ "$CARCH" = "x86_64" ]; then
    cat ../config.x86_64 >./.config
  else
    cat ../config >./.config
  fi
  if [ "${_kernelname}" != "" ]; then
    sed -i "s|CONFIG_LOCALVERSION=.*|CONFIG_LOCALVERSION=\"${_kernelname}\"|g" ./.config
  fi
  # get kernel version  
  make prepare
  # load configuration
  # Configure the kernel. Replace the line below with one of your choice.
  #make menuconfig # CLI menu for configuration
  #make xconfig # X-based configuration
  #make oldconfig # using old config from previous kernel version
  # ... or manually edit .config
  ####################
  # stop here
  # this is useful to configure the kernel
  #msg "Stopping build"
  #return 1
  ####################
  yes "" | make config
  # build!
  make bzImage modules || return 1
}

package_kernel26iwl_debug() {
  pkgdesc="The Linux Kernel and modules"
  groups=('base')
  backup=(etc/mkinitcpio.d/${pkgname}.preset)
  depends=('coreutils' 'kernel26-firmware>=2.6.32' 'module-init-tools' 'mkinitcpio>=0.5.20')
  # pwc, ieee80211 and hostap-driver26 modules are included in kernel26 now
  # nforce package support was abandoned by nvidia, kernel modules should cover everything now.
  # kernel24 support is dropped since glibc24
  replaces=('kernel24' 'kernel24-scsi' 'kernel26-scsi'
            'alsa-driver' 'ieee80211' 'hostap-driver26'
            'pwc' 'nforce' 'squashfs' 'unionfs' 'ivtv'
            'zd1211' 'kvm-modules' 'iwlwifi' 'rt2x00-cvs'
            'gspcav1' 'atl2' 'wlan-ng26' 'rt2500')
  install=kernel26.install
  optdepends=('crda: to set the correct wireless channels of your country')

  KARCH=x86
  cd ${srcdir}/linux-$_basekernel
  # get kernel version
  _kernver="$(make kernelrelease)"
  mkdir -p ${pkgdir}/{lib/modules,boot}
  make INSTALL_MOD_PATH=${pkgdir} modules_install || return 1
  cp System.map ${pkgdir}/boot/System.map26${_kernelname}
  cp arch/$KARCH/boot/bzImage ${pkgdir}/boot/vmlinuz26${_kernelname}
  #  # add vmlinux
  install -m644 -D vmlinux ${pkgdir}/usr/src/linux-${_kernver}/vmlinux

  # install fallback mkinitcpio.conf file and preset file for kernel
  install -m644 -D ${srcdir}/kernel26.preset ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset || return 1
  # set correct depmod command for install
  sed \
    -e  "s/KERNEL_NAME=.*/KERNEL_NAME=${_kernelname}/g" \
    -e  "s/KERNEL_VERSION=.*/KERNEL_VERSION=${_kernver}/g" \
    -i $startdir/kernel26.install
  sed \
    -e "s|source .*|source /etc/mkinitcpio.d/kernel26${_kernelname}.kver|g" \
    -e "s|default_image=.*|default_image=\"/boot/${pkgname}.img\"|g" \
    -e "s|fallback_image=.*|fallback_image=\"/boot/${pkgname}-fallback.img\"|g" \
    -i ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset

  echo -e "# DO NOT EDIT THIS FILE\nALL_kver='${_kernver}'" > ${pkgdir}/etc/mkinitcpio.d/${pkgname}.kver
  # remove build and source links
  rm -f ${pkgdir}/lib/modules/${_kernver}/{source,build}
  # remove the firmware
  rm -rf ${pkgdir}/lib/firmware
}

package_kernel26-headers() {
  pkgdesc="Header files and scripts for building modules for kernel26"
  mkdir -p ${pkgdir}/lib/modules/${_kernver}
  cd ${pkgdir}/lib/modules/${_kernver}
  ln -sf ../../../usr/src/linux-${_kernver} build
  cd ${srcdir}/linux-$_basekernel
  install -D -m644 Makefile \
    ${pkgdir}/usr/src/linux-${_kernver}/Makefile
  install -D -m644 kernel/Makefile \
    ${pkgdir}/usr/src/linux-${_kernver}/kernel/Makefile
  install -D -m644 .config \
    ${pkgdir}/usr/src/linux-${_kernver}/.config
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/include

  for i in acpi asm-{generic,x86} config linux math-emu media net pcmcia scsi sound trace video; do
    cp -a include/$i ${pkgdir}/usr/src/linux-${_kernver}/include/
  done

  # copy arch includes for external modules
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/arch/x86
  cp -a arch/x86/include ${pkgdir}/usr/src/linux-${_kernver}/arch/x86/

  # copy files necessary for later builds, like nvidia and vmware
  cp Module.symvers ${pkgdir}/usr/src/linux-${_kernver}
  cp -a scripts ${pkgdir}/usr/src/linux-${_kernver}
  # fix permissions on scripts dir
  chmod og-w -R ${pkgdir}/usr/src/linux-${_kernver}/scripts
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/.tmp_versions

  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/arch/$KARCH/kernel

  cp arch/$KARCH/Makefile ${pkgdir}/usr/src/linux-${_kernver}/arch/$KARCH/
  if [ "$CARCH" = "i686" ]; then
    cp arch/$KARCH/Makefile_32.cpu ${pkgdir}/usr/src/linux-${_kernver}/arch/$KARCH/
  fi
  cp arch/$KARCH/kernel/asm-offsets.s ${pkgdir}/usr/src/linux-${_kernver}/arch/$KARCH/kernel/

  # add headers for lirc package
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/video
  cp drivers/media/video/*.h  ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/video/
  for i in bt8xx cpia2 cx25840 cx88 em28xx et61x251 pwc saa7134 sn9c102 usbvideo zc0301; do
   mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/video/$i
   cp -a drivers/media/video/$i/*.h ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/video/$i
  done
  # add docbook makefile
  install -D -m644 Documentation/DocBook/Makefile \
    ${pkgdir}/usr/src/linux-${_kernver}/Documentation/DocBook/Makefile
  # add dm headers
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/drivers/md
  cp drivers/md/*.h  ${pkgdir}/usr/src/linux-${_kernver}/drivers/md
  # add inotify.h
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/include/linux
  cp include/linux/inotify.h ${pkgdir}/usr/src/linux-${_kernver}/include/linux/
  # add wireless headers
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/net/mac80211/
  cp net/mac80211/*.h ${pkgdir}/usr/src/linux-${_kernver}/net/mac80211/
  # add dvb headers for external modules
  # in reference to:
  # http://bugs.archlinux.org/task/9912
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/dvb/dvb-core
  cp drivers/media/dvb/dvb-core/*.h ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/dvb/dvb-core/
  # add dvb headers for external modules
  # in reference to:
  # http://bugs.archlinux.org/task/11194
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/include/config/dvb/
  cp include/config/dvb/*.h ${pkgdir}/usr/src/linux-${_kernver}/include/config/dvb/
  # add dvb headers for http://mcentral.de/hg/~mrec/em28xx-new
  # in reference to:
  # http://bugs.archlinux.org/task/13146
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/dvb/frontends/
  cp drivers/media/dvb/frontends/lgdt330x.h ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/dvb/frontends/
  cp drivers/media/video/msp3400-driver.h ${pkgdir}/usr/src/linux-${_kernver}/drivers/media/dvb/frontends/
  # add xfs and shmem for aufs building
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/fs/xfs
  mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/mm
  cp fs/xfs/xfs_sb.h ${pkgdir}/usr/src/linux-${_kernver}/fs/xfs/xfs_sb.h
  # add headers vor virtualbox
  # in reference to:
  # http://bugs.archlinux.org/task/14568
  cp -a include/drm $pkgdir/usr/src/linux-${_kernver}/include/
  # add headers for broadcom wl
  # in reference to:
  # http://bugs.archlinux.org/task/14568
  cp -a include/trace $pkgdir/usr/src/linux-${_kernver}/include/
  # copy in Kconfig files
  for i in `find . -name "Kconfig*"`; do 
    mkdir -p ${pkgdir}/usr/src/linux-${_kernver}/`echo $i | sed 's|/Kconfig.*||'`
    cp $i ${pkgdir}/usr/src/linux-${_kernver}/$i
  done

  cd ${pkgdir}/usr/src/linux-${_kernver}/include && ln -s asm-$KARCH asm
  # add header for aufs2-util
  cp -a ${srcdir}/linux-$_basekernel/include/asm-generic/bitsperlong.h ${pkgdir}/usr/src/linux-${_kernver}/include/asm/

  chown -R root.root ${pkgdir}/usr/src/linux-${_kernver}
  find ${pkgdir}/usr/src/linux-${_kernver} -type d -exec chmod 755 {} \;
  # remove unneeded architectures
  rm -rf ${pkgdir}/usr/src/linux-${_kernver}/arch/{alpha,arm,arm26,avr32,blackfin,cris,frv,h8300,ia64,m32r,m68k,m68knommu,mips,microblaze,mn10300,parisc,powerpc,ppc,s390,sh,sh64,sparc,sparc64,um,v850,xtensa}
}

package_kernel26-firmware() {
  pkgdesc="The included firmware files of the Linux Kernel"
  groups=('base')

  cd ${srcdir}/linux-$_basekernel
  make firmware || return 1
  make INSTALL_MOD_PATH=${pkgdir} firmware_install || return 1
}
md5sums=('260551284ac224c3a43c4adac7df4879'
         '7fba47d9b1d87de87a612a265fbfb4b1'
         '5bd3d560a8bdc9e5d336309535d6d60f'
         '5c91374d56f115ba4324978d5b002711'
         '25584700a0a679542929c4bed31433b6')

Last edited by Ramses de Norre (2010-04-16 09:18:29)

akhilbehl · 2010-04-16 09:50:11

Thanks all of you for the replies, lilsirecho, Ber and de Norre.

Actually Ber that is how I started to do when reading the wiki. But I think the wiki says that the makepkg method is the recommended method. And when I started reading up on all the PKGBUILDs, it just confused me regarding the firmware etc.

I went till make menuconfig in Ber's guide and then just retracted my steps because I got scared about the new kernel conflicting with my old one. So now that Ber has confirmed that this leads to no conflicts, I am all ready to go.

Thanks everybody for the hlep.

P.S.: I shall mark this post as [SOLVED] once I am done with the compilation. Am all excited for my next point in the "geekitude" test!

Spacenick · 2010-04-16 10:13:18

There can also be other reasons for example I compiled 2.6.34-rc4 a few days back (from git that's imho the best method especially if it breaks something you can start bisecting right away and help the kernel community and all of us) and then one can make sure there are no big showstopper bugs.

Ber · 2010-04-16 11:21:10

I agree with you on both points, I also compiled it yesterday to get a better kgdb support (and it worked, happily). But for a start taking a stable kernel is probably simpler (you don't take much risk, but still)

akhilbehl · 2010-04-16 16:28:12

@Spacenik: Thanks. You are right, I had not thought of those reasons. And I too agree with you on both. But to reiterate Ber, I think since this is my first time of compiling a kernel, I would just like to see if I can at all compile at least a stable kernel well. You know, with a person of my experience, I might not be able to differentiate a bug from it being in the kernel to it being due my stupidity in the compile process. Once I think I am fairly experienced with the process I shall be very happy to test new kernels and give some contribution back to the Linux community which has really taught me a lot and improved my computing experience by at least a Gazillion times. I cringe now, if I have to sit in front of a Windows computer, not because I hate M$ but simply because everything on my ArchLinux is so much better.

akhilbehl · 2010-04-16 16:29:04

P.S. I have only now found the time to actually sit down and pour over all the configuration options. I will soon post the results. Thanks all for help.

akhilbehl · 2010-04-16 20:04:03

completed the configuration by manually going over all the configuration options and reading the help files. seems like I have been able to include all my modules. confirmed the same by using make localmodconfig. I think I took a very conservative approach of not letting anything not really in my hardware ( at least to the best of my knowledge ) enter my kernel. have used the new LZO compression method because I read somewhere that it is fast. the make process is going on. I have no idea how long it will take. post back soon with the results.

akhilbehl · 2010-04-16 21:15:31

The kernel did not work the first time! So I rebooted into my stock kernel and there was no .config file in the folder. I had spent nearly 4 hours configuring that kernel. Sorry for posting so frequently like a kid. I will keep at it. Anyway, I will mark this thread as solved, since my original question has been answered. Thanks everybody.

Ber · 2010-04-19 08:02:18

Unless you deleted or moved the .config file it has to be there. Didn't you forget to copy the config from /proc/config.gz, and to rebuild the initcpio? If you haven't removed support for you hard drive/partition type, your kernel should at least start. What error do you get when trying to start it from Grub?

sultanoswing · 2010-04-19 09:46:27

Having tried roll-your-own methods and also compiling using the Arch Build System (abs), I heartily recommend the abs method, unless you're interested in problem solving. Your newly compiled kernel is then installed using pacman, and plays nicely alongside a stock kernel, which can be used as a backup if needed.

As of kernel26-2.6.32, you need to make some minor alterations to the PKGBUILD so three separate packages are made and the mkinitcpio happens properly. Links explaining this are here:

http://bbs.archlinux.org/viewtopic.php?id=87953
http://bbs.archlinux.org/viewtopic.php?id=87795

Last edited by sultanoswing (2010-04-19 09:47:31)

jurkan · 2010-04-19 11:57:11

This Tutorial might be helpful for getting the right config:

http://zen-kernel.org/tutorials/creatin … figuration

Its for zen kernel, but you can also apply it on the normal vanilla kernel. Following this guide, you will get a minimal kernel config that supports your hardware.
(Enabling support for large files (see Blockdevices) seems to be a good idea ;-) )

akhilbehl · 2010-04-19 18:32:58

Hey all thanks for writing in. I thought it was bad on my part to be imposing on the time of you all.

Actually Ber, the first time I compiled the kernel, it did not work at all. I came up to a prompt on the ramfs. I had compiled this kernel using the /proc/config.gz file with a lot of un-necessary ( in my un-educated opinion ) modules removed. Since I had no idea what went wrong, I rebooted into my stock kernel to rebuild using the old config. But could not find the config file. I later realized that this was because I used the command "make mrproper" which cleans the config file too.

Then I started all over again. I had read up on a Debian wiki that a config file is not necessary for the compilation process. So this time I tried without a config file and tried to include at least all those modules that were then running on my machine. Compiled, mkinitcpio'ed and all that and the kernel booted well this time. The first thing I do with every new kernel is to build the wl module for my Broadcom driver. However it did not compile this time. Some variables in the source were not found! I also realized that my webcam was not being detected. So dejected I deleted that kernel again.

Right now I have exams at my univ. going on. So can't find the time since configuration takes a lot of time. But would surely try again soon.

@sultan: Thanks for the links. Would try the ABS method too the next time. Though I have to say I am actually quite happy with this method too. It is _so_ simple that it makes compiling a kernel seem like only so much work.

@ Jurkan: Thanks a lot for that one. It is the configuration only which I seem to be not getting right. Perhaps because I am not very knowledgable about hardware and all. I have been make-doing with using lspci and lsmod to assess what modules and hardware support I need. Would take a _good_ look at this one. Thanks again.

Cheers!

Last edited by akhilbehl (2010-04-19 18:35:44)

Arch Linux

#1 2010-04-16 01:09:28

[SOLVED] Compile and Install a Parallel Kernel

#2 2010-04-16 01:50:49

Re: [SOLVED] Compile and Install a Parallel Kernel

#3 2010-04-16 08:33:56

Re: [SOLVED] Compile and Install a Parallel Kernel

#4 2010-04-16 09:17:42

Re: [SOLVED] Compile and Install a Parallel Kernel

#5 2010-04-16 09:50:11

Re: [SOLVED] Compile and Install a Parallel Kernel

#6 2010-04-16 10:13:18

Re: [SOLVED] Compile and Install a Parallel Kernel

#7 2010-04-16 11:21:10

Re: [SOLVED] Compile and Install a Parallel Kernel

#8 2010-04-16 16:28:12

Re: [SOLVED] Compile and Install a Parallel Kernel

#9 2010-04-16 16:29:04

Re: [SOLVED] Compile and Install a Parallel Kernel

#10 2010-04-16 20:04:03

Re: [SOLVED] Compile and Install a Parallel Kernel

#11 2010-04-16 21:15:31

Re: [SOLVED] Compile and Install a Parallel Kernel

#12 2010-04-19 08:02:18

Re: [SOLVED] Compile and Install a Parallel Kernel

#13 2010-04-19 09:46:27

Re: [SOLVED] Compile and Install a Parallel Kernel

#14 2010-04-19 11:57:11

Re: [SOLVED] Compile and Install a Parallel Kernel

#15 2010-04-19 18:32:58

Re: [SOLVED] Compile and Install a Parallel Kernel

Board footer