[SOLVED]Problematic upgrade to newest kernel 3.10.5-1

Onyros · 2013-08-07 01:26:09

Aloha, guys. I've just had a miserably failed upgrade to the latest kernel, probably due to something I had misconfigured and hadn't noticed before.

I was running 3.10 already perfectly before this last update to 3.10.5-1 (x86_64), but as soon as I rebooted after upgrading to the brand spanking new kernel, I got the fsck warning other people mentioned around this forum - corrected already in my syslinux.cfg by replacing ro with rw in the append line.

Both with ro and rw in the append line the boot process hangs (afterall, that fsck warning wouldn't stop the laptop from booting). The problem I'm having is that udev is not starting, meaning that boot hangs with lots of failed udev warnings.

The first one that comes up is "Failed to start udev Coldplug all Devices", and "Failed to start udev Kernel Device Manager" from then on until the boot process just hangs.

I've chrooted and tried rebuilding the initramfs, but it's giving me an "invalid kernel specified: 'boot/vmlinuz-linux'" error.

Right now I'm totally at a loss, as I cannot even begin to troubleshoot it any further. I hadn't kept the older kernel pkg in order to downgrade, but would like to try correcting this before eventually downgrading - as that just temporarily solves my problem.

My mkinitcpio hooks were "base udev autodetect modconf block filesystems keyboard fsck" and I've never had any previous problems with my initramfs. I know I may be missing something basic, or some further information needed to troubleshoot it, so please, please, please, just let me know and I will provide it here.

Any help will be greatly appreciated! Thanks in advance!

Last edited by Onyros (2013-08-07 21:03:54)

WonderWoofy · 2013-08-07 02:30:58

You tried rebuilding the initramfs, but did you try reinstalling the kernel? Maybe something went awry when writing the kernel to disk?

von · 2013-08-07 04:35:37

Same here. Initially thought that I have some issues with my raid.

UPD: Fixed my problem by reinstalling mkinitcpio and restoring my raid configuration. Still have a strange issue with ldconfig complaining about some libs in /usr/lib that are 'empty, not checked'.

Last edited by von (2013-08-07 05:41:25)

Onyros · 2013-08-07 11:30:26

I did try reinstalling the kernel, but that it gives me the same error when rebuilding the initramfs - apart from the same complaints regarding ldconfig von up here mentioned:

==> Building image from preset /etc/mkinitcpio.d/linux.preset 'default'
 -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> ERROR: invalid kernel specified: '/boot/vmlinuz-linux'

and then repeats the error when it tries building the fallback initramfs image.

The ldconfig errors are the following:

ldconfig: File /usr/lib/libblkid.so.1 is empty, not checked.
ldconfig: File /usr/lib/libuuid.so. is empty, not checked. 
ldconfig: File /usr/lib/libuuid.so.1.3.0 is empty, not checked. 
ldconfig: File /usr/lib/libuuid.so.1 is empty, not checked. 
ldconfig: File /usr/lib/libblkid.so is empty, not checked. 
ldconfig: File /usr/lib/libmount.so is empty, not checked.
ldconfig: File /usr/lib/libmount.so.1.1.0 is empty, not checked.
ldconfig: File /usr/lib/libblkid.so.1.1.0 is empty, not checked.

So, I tried reinstalling the linux kernel and mkinitcpio and it's the initramfs generation that fails with that error...

Boot partition is mounted and I can access it, I'm at risk of hair loss from head scratching.

MAGNIFICENT UPDATE: there's beer involved for whoever helps me solve this one!

Last edited by Onyros (2013-08-07 13:15:17)

mcloaked · 2013-08-07 13:25:27

This must be something specific about your system - I have three machines running without any problem with kernel 3.10.5-1 - one uses the latest grub 2.00.5086-1, and the other two boot with rEFInd. All three boot fine and are stable.

Did you have any customisation in your mikinitcpio.conf or kernel parameters during boot?

In case it is of interest here is the output in my pacman log for the kernel update section:

[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Starting build: 3.10.5-1-ARCH
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Creating gzip initcpio image: /boot/initramfs-linux.img
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Image generation successful
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Starting build: 3.10.5-1-ARCH
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: smsmdtv
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2013-08-07 13:23] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
[2013-08-07 13:23] [ALPM-SCRIPTLET] ==> Image generation successful

Last edited by mcloaked (2013-08-07 13:29:16)

Onyros · 2013-08-07 13:35:50

No customisation whatsoever, running a very vanilla mkinitcpio.conf and the only kernel parameter I had running during boot was the disabling of the nmi watchdog. What troubles me the most here is that all previous updates on the kernel were flawless, and I introduced nothing new in the meantime. I remember that after the upgrade, when I rebooted, that shutdown took much longer than usual.

I have fsck'ed the partitions, and there are no errors. Somewhere, somehow there must an error in recognising the vmlinuz-linux image as the appropriate one, at least from the message when I try regenerating the initramfs. I tried reinstalling file-5.14 as well - and I had no error information when I upgraded the kernel the first time. It was after reboot that I got these errors.

Reinstalling the linux kernel should clear a supposedly corrupted vmlinuz-linux image, I have cleared the cache and reinstalled anew, also to no avail. mkinitcpio fails every time while generating the initramfs.

I suppose this is what keeps udev from starting, too, bringing all chaos and mayhem to the boot process.

BTW, mcloaked, that was the exact output from the update to 3.5.10-1, up to the missing modules (bfa, aic94xx and smsmdtv - running on a Thinkpad X220). You also seem to have the exact same hooks I have on my mkinitcpio.conf.

Last edited by Onyros (2013-08-07 13:39:38)

mcloaked · 2013-08-07 13:59:06

Onyros wrote:

BTW, mcloaked, that was the exact output from the update to 3.5.10-1, up to the missing modules (bfa, aic94xx and smsmdtv - running on a Thinkpad X220). You also seem to have the exact same hooks I have on my mkinitcpio.conf.

I presume that when you reinstalled mkinitcpio that you have version mkinitcpio 0.15.0-1

Also presuming that if you run mkinitcpio manually via mkinitcpio -p linux that it fails with the same error that you quoted - though I was slightly puzzled by the reference to /boot/vmlinuz-linux being an invalid kernel reference!

You might compare your file for mkinitcpio.d/linux-preset with mine:

$ cat /etc/mkinitcpio.d/linux.preset 
# mkinitcpio preset file for the 'linux' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-linux.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-linux-fallback.img"
fallback_options="-S autodetect"

Last edited by mcloaked (2013-08-07 14:04:03)

Onyros · 2013-08-07 14:14:54

Yes, running mkinitcpio -p linux by itself renders the exact same error. It somehow detects the vmlinuz-linux as an invalid kernel.

My linux-preset file, which I haven't touched, ever, is exactly the same as yours...

mcloaked · 2013-08-07 14:49:24

Onyros wrote:

Yes, running mkinitcpio -p linux by itself renders the exact same error. It somehow detects the vmlinuz-linux as an invalid kernel.
My linux-preset file, which I haven't touched, ever, is exactly the same as yours...

OK one other thing to try - my mkinitcpio package file in /var/cache/pacman/pkg/ is

-rw-r--r-- 1 root root 36064 Jul 19 20:58 /var/cache/pacman/pkg/mkinitcpio-0.15.0-1-any.pkg.tar.xz

When you re-installed did you remove that file first? Is it possible that this package file is corrupt? If you remove it and then pacman -S mkinitcpio so that it pulls the file from the repo does it make any difference?

Onyros · 2013-08-07 15:08:05

I did reinstall mkinitcpio (and the kernel) again, pulling the file from the repo. Same result, I'm not even running testing, too.

Thank you for your help, anyway, but my problem persists...

This one is turning out one very hard one to solve for me. Reinstall was my last option here, really wanted to know what happened here.

I'm upping the challenge to several beers (promise!)

Last edited by Onyros (2013-08-07 20:02:21)

Onyros · 2013-08-07 20:35:30

I'm sorry to bump this, but just to introduce a new detail into this... Trying to solve those ldconfig problems posted above, I tried reinstalling util-linux, and that is giving me some trouble, as well.

There was an update to lib32-util-linux today, too. When I try to reinstall util-linux, it fails, with lots of file conflicts (file exists in filesystem). I don't dare to force it, but could this all be stemming from here?

Something definitely went wrong with this machine, as I have several others running Arch flawlessly. I'd really like to learn from this one, in order never to repeat whatever happened.

file -k /boot/vmlinuz-linux output, just in case it's helpful:

/boot/vmlinuz-linux: x86 boot sector\012 - Linux kernel x86 boot executable bzImage, version 3.10.5-1-ARCH (tobias@T-POWA-LX) #1 SMP PREEMPT Mon Aug 5 00:04, RO-rootFS, swap_dev 0x3, Normal VGA PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows

LAST UPDATE: marking this one as solved. Force reinstalled util-linux, there must have been some file corruption there through that shutdown after upgrading to 3.5.10-1 as there are a few other configs that seem affected by this. Now for the clean up!

Thanks to those who gave me a hand in this - or made me think of other possible stuff

Last edited by Onyros (2013-08-07 21:12:58)

mcloaked · 2013-08-07 21:09:54

Great that you finally found the corrupted file.

Last edited by mcloaked (2013-08-07 21:11:30)

Onyros · 2013-08-07 21:14:45

Thank you, mate. And thank you for your help, too. Should you ever come by these parts, there are a couple of beers with your name on it

Arch Linux

#1 2013-08-07 01:26:09

[SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#2 2013-08-07 02:30:58

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#3 2013-08-07 04:35:37

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#4 2013-08-07 11:30:26

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#5 2013-08-07 13:25:27

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#6 2013-08-07 13:35:50

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#7 2013-08-07 13:59:06

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#8 2013-08-07 14:14:54

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#9 2013-08-07 14:49:24

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#10 2013-08-07 15:08:05

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#11 2013-08-07 20:35:30

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#12 2013-08-07 21:09:54

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

#13 2013-08-07 21:14:45

Re: [SOLVED]Problematic upgrade to newest kernel 3.10.5-1

Board footer