Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

smakovits · 2014-06-23 17:47:16

I have a sort of an issue, which is made worse because of a previously existing one...

Long story short, for whatever reason, in recent weeks/months, my Dom0 Arch system would boot fine, but upon loading the kernel, the screen would go out of range. As a result, I did not have access locally with mouse and keyboard. Instead I just relied on SSH/putty, which was perfectly fine. I did some reading on the issue, but never actually enough to resolve the problem. I sort of pushed it off because the system was perfectly functional and because it is in the basement, this never really bothered me.

That is until last night any way. I ran a full system update via pacman -Syu. I then issues

 mkinitcpio -p linux

and rebooted. Only issue being that the system never came back up. The system does not ping on its static IP so I can only assume the system is not fully up and instead hung somewhere. This is where the first issues causes my issue. I am not able to see what is displayed on the screen because it is out of range.

I do briefly see grub when the system first boots, so I am not sure if I can get access from there or if this is a case I need to boot the CD and work things from there. As a result, I am hoping to get some direction as opposed to doing the wrong thing and instead making the situation worse.

Thanks.

Last edited by smakovits (2014-06-28 18:12:54)

fungle · 2014-06-23 21:40:58

Try booting with a liveCD and chrooting into your system and check the logs

smakovits · 2014-06-24 00:41:07

OK.

Before seeing this, I did manage to boot with fall-back. I ran mkinit again and re-did grub. Here is the out from that.

arch_1 ~ # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found Xen hypervisor image: /boot/xen-4.4.0.gz
        Found linux image: /boot/vmlinuz-linux
        Found initrd image: /boot/initramfs-linux.img
Found Xen hypervisor image: /boot/xen-4.4.0.gz
        Found linux image: /boot/vmlinuz-linux
        Found initrd image: /boot/initramfs-linux.img
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/initramfs-linux.img
Found fallback initramfs image: /boot/initramfs-linux-fallback.img
done
arch_1 ~ #

As far as the logs, which ones am I most interested in?

Edit:
Rebooted again and still nothing. I once again rebooted to fallback and it comes up. The good thing in this mode is that the video is fine too and does not go out of range like when it booted to xen.

Last edited by smakovits (2014-06-24 02:33:21)

WonderWoofy · 2014-06-24 02:26:41

If you were able to boot with the fallback image, it is likely an issue with mkinitcpio and not grub. Something that the initramfs-linux.img should have is not being picked up automatically.

You should start your search there and post any relevant information.

smakovits · 2014-06-24 02:52:34

I'm not going to lie to you, I do not even know where to start my search. Like how do I search initramfs-linux.img not being picked up right?

When executing mkinitcpio -p linux, I get:

arch_1 ~ # mkinitcpio -p linux
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 3.15.1-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 3.15.1-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: smsmdtv
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful

Which looks ok to me, but I do not know.

WonderWoofy · 2014-06-24 03:17:24

I guess you would try and boot it and see if you get any indication on where it might be stopping.

Basically, the initramfs just needs to be able to find the partition and filesystem so that in can do a switch_root and then launch /sbin/init on the real system. So if you have 'quiet' on the kernel command line, you should remove that. To get the initramfs to print to the console, you should use 'rd.debug'. You might be able to get away with just using 'rd.log=console' though, as the default log level might give you enough info to nail down the issue.

Also, sometimes lvm2 can be flimsy when it is used for the rootfs. You might want to try using the systemd hook in place of base and udev. Then replace the lvm2 hook with sd-lvm. Since systemd is udev these days, it does a pretty good job at ordering things right to get things going. Using the lvm2 hook with base and udev basically just scripts out the lvm activation.

smakovits · 2014-06-24 04:10:58

Does it mean anything if when booting to fallback i see the following on the screen?

failed to start load kernel modules

Another thing I found interesting is that if I try to boot the xen fallback, it bombs out right away as well. It seems that about the time that stuff should appear on the screen like it does with the working fallback is when the other options bomb off. Again, just something I observed and thought might be relevant.

As for the second part of you message. Are you suggesting to try the following?

HOOKS="base udev autodetect modconf block lvm2 filesystems keyboard fsck"

and changing it to

HOOKS="systemd autodetect modconf block sd-lvm filesystems keyboard fsck"

And then execute

mkinitcpio -p linux

Thanks

WonderWoofy · 2014-06-24 04:38:31

Yeah, but you may want to keep a copy of your fallback initramfs that works. Since rebuilding will overwrite both /boot/initramfs-linux.img as well as /boot/initramfs-linux-fallback.img.

Edit: Also, I know next to nothing about using xen. But it doesn't sound as though this is a xen problem at all.

Last edited by WonderWoofy (2014-06-24 04:39:00)

smakovits · 2014-06-24 13:51:50

As much as I do not want it to be a Xen issue, this morning I tested the non fallback arch boot option from grub and it loaded Ok. Not sure what that means for troubleshooting, if it is still an initramfs issue or if I have to focus on the Xen side of things.

smakovits · 2014-06-25 18:10:46

So I have some logs from a good boot (4:27) and a bad boot (4:16). Not sure if they are of use or not.

http://pastebin.com/rxhV3nLX
http://pastebin.com/bjq0L915

Beyond the logs, I was looking at my grub config to see if anything stuck out. I noticed ( module /vmlinuz-linux /vmlinuz-linux root=UUID=9d49c1a1-daad-4eed-a03f-760df8bd0ec2 rw quiet console=tty0)

menuentry "Xen 4.4.0 / Arch Linux pkg-linux kernel" --class xen --class arch-linux --class arch --class gnu-linux --class gnu --class os {

    if [ x$feature_all_video_module = xy ]; then
        insmod all_video
    fi
    set gfxpayload=keep
    insmod ext2
    if [ x$feature_platform_search_hint = xy ]; then
        search --no-floppy --fs-uuid  --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  3800d8a7-f91f-4cc8-a0f8-259cfd$
    else
        search --no-floppy --fs-uuid  --set=root 3800d8a7-f91f-4cc8-a0f8-259cfd0c48b8
    fi
    echo 'Loading Xen 4.4.0 ...'
    multiboot /xen-4.4.0.gz /xen-4.4.0.gz xsave=1
    echo 'Loading Arch Linux pkg-linux kernel ...'
    module /vmlinuz-linux /vmlinuz-linux root=UUID=9d49c1a1-daad-4eed-a03f-760df8bd0ec2 rw  quiet console=tty0
    echo 'Loading Arch Linux pkg-linux kernel initramfs ...'
    module /initramfs-linux.img

Not sure the double /vmlinuz-linux /vmlinuz-linux is normal or not. I would lean to the not, but I am not sure so I thought it was worth the mention.

edit:

Maybe not, from the wiki:

menuentry 'Arch Linux Stock Kernel' {
  set root='(hd0,1)'; set legacy_hdbias='0'
  legacy_kernel   '/vmlinuz-linux' '/vmlinuz-linux' 'root=/dev/sda2' 'ro'
  legacy_initrd '/initramfs-linux.img' '/initramfs-linux.img'

Last edited by smakovits (2014-06-25 18:56:12)

smakovits · 2014-06-28 02:58:42

So I ended up going ahead and blowing away my system. I re-installed Arch, went through the beginners guide and installed Xen. The same thing happens. Therefore, it would seem that my issue is the result of kernel 3.15 and Xen 4.4. For whatever reason, the 2 do not seem to play nice. At this time I need to ask where to go for help. Being in the newbie thread is obviously not the best place as I instead need to get support for xen. Just wanted to come back and ask for guidance as to the best place to seek help regarding why my system crashes when booting to xen.

Thanks

jasonwryan · 2014-06-28 04:51:33

Edit your first post and change the title to more accurately reflect your issue...

Moving to Kernel & Hardware.

smakovits · 2014-06-30 15:19:34

As a follow up, I was able to install debian with xen and everything works, so from purely a system standpoint, I know I am OK.

Some things I would like to try if possible. I want to try deploying arch again, but want to use the 3.14 kernel. However, I am not sure how this can be done. Since it upgrades at the time of installation. If it is somethign that is easy, I am up for it. Otherwise I will wait for the nextkernel upgrade since that is when my issue seems to have started.

smakovits · 2014-07-17 12:01:53

Appears the issue is a result of the lvm2 hook.

Re-installed to a new drive using type 83 partition, loaded xen and everything was fine. Added the lvm2 hook because I thought I needed it when my vms werent booting and upon reboot the system did not start. Removed the hook and again everything was fine.

When the issue first started, I had the lvm2 hook in place because my arch dom0 was installed to an lv.

smakovits · 2014-07-17 12:03:54

WonderWoofy wrote:

I guess you would try and boot it and see if you get any indication on where it might be stopping.
Basically, the initramfs just needs to be able to find the partition and filesystem so that in can do a switch_root and then launch /sbin/init on the real system. So if you have 'quiet' on the kernel command line, you should remove that. To get the initramfs to print to the console, you should use 'rd.debug'. You might be able to get away with just using 'rd.log=console' though, as the default log level might give you enough info to nail down the issue.
Also, sometimes lvm2 can be flimsy when it is used for the rootfs. You might want to try using the systemd hook in place of base and udev. Then replace the lvm2 hook with sd-lvm. Since systemd is udev these days, it does a pretty good job at ordering things right to get things going. Using the lvm2 hook with base and udev basically just scripts out the lvm activation.

Seems you were on to something. Once I completely left lvm for my rootfs things seem OK. Using just a single linux (83) volume now for Arch Dom0.

Arch Linux

#1 2014-06-23 17:47:16

Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#2 2014-06-23 21:40:58

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#3 2014-06-24 00:41:07

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#4 2014-06-24 02:26:41

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#5 2014-06-24 02:52:34

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#6 2014-06-24 03:17:24

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#7 2014-06-24 04:10:58

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#8 2014-06-24 04:38:31

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#9 2014-06-24 13:51:50

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#10 2014-06-25 18:10:46

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#11 2014-06-28 02:58:42

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#12 2014-06-28 04:51:33

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#13 2014-06-30 15:19:34

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#14 2014-07-17 12:01:53

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

#15 2014-07-17 12:03:54

Re: Xen 4.4 kernel not booting after upgrading to 3.15 Linux kernel

Board footer