module directory not found because wrong kernel version

Dave Cohen · 2010-05-09 02:27:51

After a pacman -Syu earlier today, my machine is crippled.

During boot, I notice some modules are not loaded, with errors like "FATAL: Could not load /lib/modules/2.6.32-ARCH/..."

The problem seems to be that /lib/modules/2.6.33 exists, but not /lib/modules/2.6.32. Where when I perform a `uname -a`, the kernel running is 2.6.32-ARCH.

I'd paste the exact messages, but networking does not work on the crippled machine. I need help!

toffyrn · 2010-05-09 07:32:02

Did you by any chance install a network driver module from aur or compile by source?

If so you may need to do it once more with most recent kernel.

hokasch · 2010-05-09 07:48:06

Some ideas: Check menu.lst if you are loading the correct kernel version, reinstall kernel26 (cache), regenerate initramfs.

Dave Cohen · 2010-05-09 08:07:40

I tried to pacman -U ... to rebuild the most recent kernel. Rebooted to the same problem.

Then I used pacman -U ... to downgrade to the earlier kernel. That worked around the problem, but I still don't have a real fix.

toffyrn · 2010-05-09 08:28:00

Did you by any chance install a network driver module from aur or compile by source?

tomk · 2010-05-09 09:33:27

If you use a separate /boot partition, make sure it is mounted when you do your kernel upgrade. An unmounted /boot will cause the symptoms you describe i.e. updated modules in /lib/modules, previous kernel remaining in /boot.

wonder · 2010-05-09 09:41:57

Dave Cohen wrote:

After a pacman -Syu earlier today, my machine is crippled.
During boot, I notice some modules are not loaded, with errors like "FATAL: Could not load /lib/modules/2.6.32-ARCH/..."
The problem seems to be that /lib/modules/2.6.33 exists, but not /lib/modules/2.6.32. Where when I perform a `uname -a`, the kernel running is 2.6.32-ARCH.
I'd paste the exact messages, but networking does not work on the crippled machine. I need help!

next time reboot after a kernel upgrade.

Dave Cohen · 2010-05-09 17:39:08

Did you by any chance install a network driver module from aur or compile by source?

While I'm not sure what this question means, I'm 99.9% sure the answer is no and no.

If you use a separate /boot partition, make sure it is mounted when you do your kernel upgrade.

/boot was mounted, as it always is.

next time reboot after a kernel upgrade.

I didn't have the problem until after reboot. Thanks for asking.

Dave Cohen · 2010-05-09 17:44:04

So after I ran into this problem, I tried to rebuild the updated kernel with

pacman -U /var/cache/pacman/pkg/kernel26-2.6.33.3-1-x86_64.pkg.tar.xz

While that reported success, I had the exact same problem after a ... wait for it ... REBOOT.

So I downgraded the kernel with

pacman -U /var/cache/pacman/pkg/kernel26-2.6.32.10-1-x86_64.pkg.tar.xz

That's where things stand now, and the machine works with kernel 2.6.32.

toffyrn · 2010-05-09 19:18:44

Hmmm. that was strange... I have no clue what it is then.

wonder · 2010-05-09 19:42:09

if you say that you rebooted and still says that is 2.6.32 then the initrd images wasn't generated correctly and i guess it failed and you didn't see it.

hokasch · 2010-05-09 19:59:28

Could you post /etc/fstab and grub's menu.lst?

Dave Cohen · 2010-05-10 17:10:45

sure, here are those files...

[dave@starbuck ~]$ cat /etc/fstab                                                                      
#                                                                                                      
# /etc/fstab: static file system information                                                           
#
# <file system>        <dir>         <type>    <options>          <dump> <pass>
none                   /dev/pts      devpts    defaults            0      0
none                   /dev/shm      tmpfs     defaults            0      0


/dev/cdrom /media/cdrom   auto    ro,user,noauto,unhide   0      0
/dev/dvd /media/dvd   auto    ro,user,noauto,unhide   0      0
/dev/mapper/grande-root / ext3 defaults 0 1
/dev/mapper/venti-media /mnt/media ext3 defaults 0 1
/dev/md1 /boot ext3 defaults 0 1
/dev/md2 swap swap defaults 0 0

https://dav.messagingengine.com /mnt/dav.messagingengine.com davfs user,noauto 0 0

# /proc/bus/usb/devices should be there automatically, but somehow broken.
# http://www.linux-usb.org/FAQ.html
#none                    /proc/bus/usb           usbfs noauto         0 0

# auto detect raid array? http://wiki.archlinux.org/index.php/Installing_with_Software_RAID_or_LVM
# (0) Arch Linux
title  Arch Linux root=/dev/grande/root
root   (hd0,0)
kernel /vmlinuz26 root=/dev/grande/root ro
initrd /kernel26.img

title  Arch Linux root=/dev/md1
root   (hd0,0)
kernel /vmlinuz26 root=/dev/md1 ro
initrd /kernel26.img


# (0) Arch Linux
title  Arch Linux (old raid info provided)
root   (hd0,0)
kernel /vmlinuz26 root=/dev/grande/root ro md=1,/dev/sdb1,/dev/sdc1 md=0,/dev/sdb4,/dev/sdc4
initrd /kernel26.img

# (1) Arch Linux
title  Arch Linux (if old drive is physically removed)
root   (hd0,0)
kernel /vmlinuz26 root=/dev/grande/root ro md=1,/dev/sda1,/dev/sdb1 md=0,/dev/sda4,/dev/sdb4
initrd /kernel26.img

# (2) Arch Linux
title  Arch Linux Fallback
root   hd(0,0)
kernel /vmlinuz26 root=/dev/grande/root ro md=1,/dev/sda1,/dev/sda1 md=0,/dev/sda4,/dev/sdb4
initrd /kernel26-fallback.img

# (3) Just guessing
title  Arch Linux EXPERIMENT
root   (hd0,0)
kernel /vmlinuz26 root=/dev/grande/root-root ro md=1,/dev/sdb1,/dev/sdc1 md=0,/dev/sdb4,/dev/sdc4
initrd /kernel26.img

# (1) Windows
#title Windows
#rootnoverify (hd0,0)
#makeactive
#chainloader +1

hokasch · 2010-05-10 17:30:51

Sorry, no clue about software raids (or whatever it is) here, but if hd(0,0) = /dev/md1 that part should be alright (I guess??).

Maybe the raid thing is worth looking into? Basically the only way this would make any sense to me is if the /boot mounted while updating the kernel is somehow different to the one you are booting from. Maybe someone else has a better idea.

Dave Cohen · 2010-05-10 17:57:07

I went through the process again. Here's what it looks like...

First, while running the old kernel....

[root@starbuck ~]# uname -a
Linux starbuck 2.6.32-ARCH #1 SMP PREEMPT Mon Mar 15 20:44:03 CET 2010 x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
[root@starbuck ~]#

Before building the new kernel...

dave@starbuck ~]$ ls -l /boot/
total 14646
drwxr-xr-x 2 root root    1024 May  8 11:19 grub
-rw-r--r-- 1 root root 8842796 May  8 13:19 kernel26-fallback.img
-rw-r--r-- 1 root root 2799478 May  8 13:18 kernel26.img
drwx------ 2 root root   12288 Sep 26  2008 lost+found
-rw-r--r-- 1 root root 1304317 Mar 15 12:46 System.map26
-rw-r--r-- 1 root root 1972768 Mar 15 12:46 vmlinuz26
[dave@starbuck ~]$

Then the build process....

[root@starbuck ~]# pacman -U /var/cache/pacman/pkg/kernel26-2.6.33.3-1-x86_64.pkg.tar.xz 
loading package data...
checking dependencies...
(1/1) checking for file conflicts                   [############################################] 100%
(1/1) upgrading kernel26                            [############################################] 100%
>>> Updating module dependencies. Please wait ...
>>> MKINITCPIO SETUP
>>> ----------------
>>> If you use LVM2, Encrypted root or software RAID,
>>> Ensure you enable support in /etc/mkinitcpio.conf .
>>> More information about mkinitcpio setup can be found here:
>>> http://wiki.archlinux.org/index.php/Mkinitcpio

>>> Generating initial ramdisk, using mkinitcpio.  Please wait...
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.33-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [autodetect]
error: /dev/sde: No medium found
error: /dev/sdd: No medium found
error: /dev/sr0: No medium found
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [mdadm]
Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays.
:: Parsing hook [usbinput]
:: Parsing hook [keymap]
:: Parsing hook [lvm2]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26.img'...SUCCESS
==> SUCCESS
==> Building image "fallback"
==> Running command: /sbin/mkinitcpio -k 2.6.33-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26-fallback.img -S autodetect
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [mdadm]
Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays.
:: Parsing hook [usbinput]
:: Parsing hook [keymap]
:: Parsing hook [lvm2]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26-fallback.img'...SUCCESS
==> SUCCESS
[root@starbuck ~]#

Yes there are some errors, but the big SUCCESS at the end makes me think they were not fatal errors.

After the build, before the reboot...

[root@starbuck ~]# ls -l /boot/
total 15265
drwxr-xr-x 2 root root    1024 May  8 11:19 grub
-rw-r--r-- 1 root root 9290733 May 10 10:15 kernel26-fallback.img
-rw-r--r-- 1 root root 2814498 May 10 10:13 kernel26.img
drwx------ 2 root root   12288 Sep 26  2008 lost+found
-rw-r--r-- 1 root root 1363160 Apr 26 10:33 System.map26
-rw-r--r-- 1 root root 2081728 Apr 26 10:33 vmlinuz26
[root@starbuck ~]#

(kernel.26.img timestamp has been updated.)

After a reboot, I have the same problems as before. Modules don't load, with this error:

FATAL: Could not load /lib/modules/2.6.32-ARCH/modules.dep: No such file or directory

And the running kernel still believes it to be 2.6.32. Uname -a returns

Linux starbuck 2.6.32-ARCH #1 SMP PREEMPT Mon Mar 15 20:44:03 CET 2010 x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux

I appreciate the help with this. If anyone sees the problem please let me know. Until then I plan to use the older kernel.

wonder · 2010-05-10 18:04:27

the timestamp of vmlinuz26 and System.map26 is changing backwards. so clearly is something that needs checking as this contains the kernel.

Dave Cohen · 2010-05-10 18:11:13

Huh. Well, they change from March 15 to April 26. So the change is forward in time. But I don't know why they are not the same date as kernel26.img. Are those files downloaded rather than built?

BTW, after downgrading to the earlier kernel, /boot looks like this:

[dave@starbuck ~]$ ls -l /boot/
total 14646
drwxr-xr-x 2 root root    1024 May  8 11:19 grub
-rw-r--r-- 1 root root 8842801 May 10 03:59 kernel26-fallback.img
-rw-r--r-- 1 root root 2799214 May 10 03:57 kernel26.img
drwx------ 2 root root   12288 Sep 26  2008 lost+found
-rw-r--r-- 1 root root 1304317 Mar 15 12:46 System.map26
-rw-r--r-- 1 root root 1972768 Mar 15 12:46 vmlinuz26
[dave@starbuck ~]$

(kernel6.img is dated today, while System.map26 and vmlinuz26 are again Mar 15)

Last edited by Dave Cohen (2010-05-10 18:13:50)

hokasch · 2010-05-10 18:13:17

it's not really going backwards, but again: if you are still booting into 2.6.32, the kernel/img you have in the second ls -l /boot/ somehow is not loaded at boot. I am still suspecting you have two different /boot - one that it boots from, and one that is mounted in your system. However that would work...

edit: that's alright, the initramfs image is build on your system, vmlinuz has the official build time-stamp since it is just installed.

Last edited by hokasch (2010-05-10 18:17:26)

wonder · 2010-05-10 18:18:26

heh, i was sure that i've seen there May and not Mar. the date is the build date.

hokasch · 2010-05-10 18:24:23

To test my theory above, rename kernel26.img to something else and try to boot - if it won't work, you can still edit grub to point it to the renamed .img

joaca_rj · 2010-05-10 22:00:24

I had the same symptons and it was related to this bug : http://bugs.archlinux.org/task/19403
also explained in this post : http://bbs.archlinux.org/viewtopic.php?id=96841
removing intel.modeset = 1 from grub solved it for me.

.:B:. · 2010-05-11 00:16:14

To me it sounds like a problem with your RAID setup Dave.

Would you care to post the output of

cat /proc/mdstat

What type of RAID are we looking at?

I had a similar problem back in the days when my server ran on RAID 1 for /. Array got marked as degraded, system would still boot though but no changes would be written (although everything seemed to function normally so you would never get the impression something was wrong, until you rebooted).

Dave Cohen · 2010-05-11 00:55:32

[dave@starbuck ~]$ cat /proc/mdstat 
Personalities : [raid1] 
md3 : active raid1 sdc3[1]
      499998912 blocks [2/1] [_U]
      
md2 : active raid1 sdc2[1]
      4000064 blocks [2/1] [_U]
      
md1 : active raid1 sdc1[1]
      96256 blocks [2/1] [_U]
      
md0 : active raid1 sdc4[1]
      472664320 blocks [2/1] [_U]
      
unused devices: <none>
[dave@starbuck ~]$

I did encounter an unrelated RAID problem some months back. Didn't upgrade the kernel for a while because of it. But was hoping I was past all that.

In case its not obvious, I'm no expert with RAID or any other aspect of sys admin. Just followed instructions on arch wiki to accomplish most of this.

.:B:. · 2010-05-11 11:01:12

No worries - we all learn along the way. It's by running into problems or making mistakes that you pick up stuff (and actually remember it).

Guess what mate. Every underscore you see is a missing device in your RAID array - which means all four (?) RAID arrays are degraded. I assume you have three HDs, seems sdb has gone AWOL...

Since it's RAID 1 the solution should be rather easy - get mdadm to pick up the faulty partitions again and force it to sync. That should get your RAID setups in a sane state again. You can follow the sync process by reading the /proc/mdstat output. You might need to boot from a Live CD though to fix all arrays though, not sure about that, it's been a while.

A sudden shutdown could have caused this, don't know if you had any power outages?

Moving this to another forum, we now know pacman is not the culprit .

Dave Cohen · 2010-05-18 00:40:33

I can confirm this is solved.

My raid array did have a drive missing. In building the kernel, /etc/mdadm.conf is consulted and used in some way that I don't pretend to understand. Clearly, if the information in that file is not in sync with the actual raid array, problems like the one I reported can arise.

I repaired my raid arrays by adding the missing drive. Then, rebuilt the newer kernel with the same `pacman -U ...` command I mentioned earlier. Rebooted and now all appears to be working.

Thanks to everyone for helping me out.

[dave@starbuck ~]$ cat /proc/mdstat 
Personalities : [raid1] 
md3 : active raid1 sdb3[0] sdc3[1]
      499998912 blocks [2/2] [UU]
      
md2 : active raid1 sdb2[0] sdc2[1]
      4000064 blocks [2/2] [UU]
      
md1 : active raid1 sdb1[0] sdc1[1]
      96256 blocks [2/2] [UU]
      
md0 : active raid1 sdb4[0] sdc4[1]
      472664320 blocks [2/2] [UU]
      
unused devices: <none>
[dave@starbuck ~]$ uname -a
Linux starbuck 2.6.33-ARCH #1 SMP PREEMPT Mon Apr 26 19:31:00 CEST 2010 x86_64 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
[dave@starbuck ~]$

Arch Linux

#1 2010-05-09 02:27:51

module directory not found because wrong kernel version

#2 2010-05-09 07:32:02

Re: module directory not found because wrong kernel version

#3 2010-05-09 07:48:06

Re: module directory not found because wrong kernel version

#4 2010-05-09 08:07:40

Re: module directory not found because wrong kernel version

#5 2010-05-09 08:28:00

Re: module directory not found because wrong kernel version

#6 2010-05-09 09:33:27

Re: module directory not found because wrong kernel version

#7 2010-05-09 09:41:57

Re: module directory not found because wrong kernel version

#8 2010-05-09 17:39:08

Re: module directory not found because wrong kernel version

#9 2010-05-09 17:44:04

Re: module directory not found because wrong kernel version

#10 2010-05-09 19:18:44

Re: module directory not found because wrong kernel version

#11 2010-05-09 19:42:09

Re: module directory not found because wrong kernel version

#12 2010-05-09 19:59:28

Re: module directory not found because wrong kernel version

#13 2010-05-10 17:10:45

Re: module directory not found because wrong kernel version

#14 2010-05-10 17:30:51

Re: module directory not found because wrong kernel version

#15 2010-05-10 17:57:07

Re: module directory not found because wrong kernel version

#16 2010-05-10 18:04:27

Re: module directory not found because wrong kernel version

#17 2010-05-10 18:11:13

Re: module directory not found because wrong kernel version

#18 2010-05-10 18:13:17

Re: module directory not found because wrong kernel version

#19 2010-05-10 18:18:26

Re: module directory not found because wrong kernel version

#20 2010-05-10 18:24:23

Re: module directory not found because wrong kernel version

#21 2010-05-10 22:00:24

Re: module directory not found because wrong kernel version

#22 2010-05-11 00:16:14

Re: module directory not found because wrong kernel version

#23 2010-05-11 00:55:32

Re: module directory not found because wrong kernel version

#24 2010-05-11 11:01:12

Re: module directory not found because wrong kernel version

#25 2010-05-18 00:40:33

Re: module directory not found because wrong kernel version

Board footer