[Solved again] Updated to linux-3.10.8; Can no longer boot

HalosGhost · 2013-07-22 18:50:18

Hey everyone.

So, linux-3.10.2-1 just dropped to the [testing] repos, and I got it through a standard -Syu. Along with it were upgraded libxfont, linux-headers, lirc-utils (and I installed extundelete during the same command).
Here's the relevant bit of /var/log/pacman.log:

[2013-07-22 11:03] [PACMAN] Running 'pacman --color=always -Syu extundelete'
[2013-07-22 11:03] [PACMAN] synchronizing package lists
[2013-07-22 11:03] [PACMAN] starting full system upgrade
[2013-07-22 11:06] [PACMAN] installed extundelete (0.2.4-1)
[2013-07-22 11:06] [PACMAN] upgraded libxfont (1.4.5-1 -> 1.4.6-1)
[2013-07-22 11:07] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2013-07-22 11:07] [ALPM-SCRIPTLET] >>> Generating initial ramdisk, using mkinitcpio.  Please wait...
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Starting build: 3.10.2-1-ARCH
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Creating gzip initcpio image: /boot/initramfs-linux.img
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Image generation successful
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Starting build: 3.10.2-1-ARCH
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: bfa
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: smsmdtv
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2013-07-22 11:07] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
[2013-07-22 11:07] [ALPM-SCRIPTLET] ==> Image generation successful
[2013-07-22 11:07] [PACMAN] upgraded linux (3.10.1-1 -> 3.10.2-1)
[2013-07-22 11:07] [PACMAN] upgraded linux-headers (3.10.1-1 -> 3.10.2-1)
[2013-07-22 11:07] [PACMAN] upgraded lirc-utils (1:0.9.0-51 -> 1:0.9.0-52)

Then, because I am on a UEFI system, I copied the initrd images and the vmlinuz image over to my ESP (just like I've done many a time before), and now the system won't boot. It doesn't throw any errors; rather, as soon as UEFI selects the entry to run, the computer just sits there (there is nothing in the logs so far as I can find).

I've booted with a live medium and I've tried regenerating the images and recopying them (the checksums of the images changed, so I figured that might do the trick), but it's still a no-go. Is there something obvious that I'm missing? Has anyone else run into this problem?

[Edit]: Though this appeared to be solved with 3.10.3, the problem appears to have returned with 3.10.8. No actual solution has been found. As with before, a new kernel has been released (3.10.9) with which this issue does not exist…

All the best,

-HG

Last edited by HalosGhost (2013-08-21 17:01:46)

cfr · 2013-07-22 22:17:51

Are you booting directly from the EFI boot menu built into the firmware or are you using a boot manager (gummiboot/rEFInd/?) or booting the EFI stub loader as default (i.e. without any menu at all)?

If you have 'quiet' on your kernel command line, remove it.

What is the last text written to screen?

HalosGhost · 2013-07-22 22:43:04

cfr wrote:

Are you booting directly from the EFI boot menu built into the firmware or are you using a boot manager (gummiboot/rEFInd/?) or booting the EFI stub loader as default (i.e. without any menu at all)?

Well, I have gummiboot installed, but I'm actually using the EFI boot menu built into the firmware (I've been biding my time before switching to syslinux-6.x).

If you have 'quiet' on your kernel command line, remove it.

I don't have 'quiet' in my entries (I prefer to know what's happening).

What is the last text written to screen?

The firmware EFI boot menu comes up with Arch selected as the default and a 2 second timeout (my choice), then either after having jumped ahead and selected Arch or having let the timeout run down, the menu disappears, and nothing else. So, depending on how you define it, the last text printed to the screen was either the boot menu or the "Lenovo" logo. But the logo was printed before the menu (it just stays there after the menu disappears).

Also might be worth mentioning, I gave the fallback initramfs a shot, and it has the same issue. I'm honestly not sure what's happening here...

All the best,

-HG

jasonwryan · 2013-07-22 22:51:16

I am seeing the same behaviour.

Looking in /boot, I can see that the initrd has been updated to 3.10.2, but the new kernel has not been copied across: hence the boot failure.

Like HalosGhost, I am using gummiboot and mount my ESP on /boot.

Downgrading or booting from a custom kernel works fine.

HalosGhost · 2013-07-22 22:59:05

jasonwryan wrote:

I am seeing the same behaviour.
Looking in /boot, I can see that the initrd has been updated to 3.10.2, but the new kernel has not been copied across: hence the boot failure.
Like HalosGhost, I am using gummiboot and mount my ESP on /boot.

Downgrading or booting from a custom kernel works fine.

I'm glad to hear that it wasn't personal incompetence, though I'm sorry to hear you had to deal with it too. Sadly, I no longer have the old kernel package in my cache, so I'll likely end up finally taking the plunge on the -ck kernel (or I might just wait until an update drops; we'll see.

All the best,

-HG

jasonwryan · 2013-07-23 20:59:08

I had hoped it was related to this (undocumented) behaviour between systemd and gummiboot, but sadly not.

HalosGhost · 2013-07-23 21:02:52

jasonwryan wrote:

I had hoped it was related to this (undocumented) behaviour between systemd and gummiboot, but sadly not.

Yeah, I'm not doing anything special with systemd auto-copying things over to my ESP either. Pretty vanilla setup. As far as I can tell, this seems to be a problem with the kernel images themselves.

All the best,

-HG

jasonwryan · 2013-07-23 21:10:17

Can you confirm that, after the install, the new kernel is not copied to /boot (and silently fails)? That is what happens on my machine.

65kid · 2013-07-23 21:11:15

jasonwryan wrote:

I had hoped it was related to this (undocumented) behaviour between systemd and gummiboot, but sadly not.

actually it may be. From what I can see the undocumented EFI fstab generator seems to be broken in 206, the automount unit is not generated anymore. So /boot was never mounted on my system and the new kernel was written directly on my root partition instead of the ESP. Not sure what's going on there yet, I'm gonna report back.

Last edited by 65kid (2013-07-23 21:12:06)

HalosGhost · 2013-07-23 21:18:51

65kid wrote:

actually it may be. From what I can see the undocumented EFI fstab generator seems to be broken in 206, the automount unit is not generated anymore. So /boot was never mounted on my system and the new kernel was written directly on my root partition instead of the ESP. Not sure what's going on there yet, I'm gonna report back.

I'm still on systemd-204 (I use `systemd --user`, so I'm waiting to upgrade), and this problem still exists for me.

jasonwryan wrote:

Can you confirm that, after the install, the new kernel is not copied to /boot (and silently fails)? That is what happens on my machine.

I'm not chomping at the bit to go through the install again, but that is what appeared to happen for me the first time around, yes.

All the best,

-HG

Last edited by HalosGhost (2013-07-23 21:21:04)

jasonwryan · 2013-07-23 21:22:58

65kid wrote:

jasonwryan wrote:
I had hoped it was related to this (undocumented) behaviour between systemd and gummiboot, but sadly not.
actually it may be. From what I can see the undocumented EFI fstab generator seems to be broken in 206, the automount unit is not generated anymore. So /boot was never mounted on my system and the new kernel was written directly on my root partition instead of the ESP. Not sure what's going on there yet, I'm gonna report back.

To test this, I masked systemd's boot.automout as WW suggested in the other thread and rebooted. I checked that /boot was mounted only by fstab and then upgraded to 3.10.2 - no errors, but vmlinuz-linux on /boot was the same file originally installed on 14/7 (ie., 3.10.1) - I didn't think to check if it was written directly to / though... Happy to test again if you need confirmation.

# edit: and I only just upgraded to systemd 206-1 this morning (30 mins ago).

# edit 2: tidied up the thread to remove the OT stuff...

65kid · 2013-07-23 21:25:44

So I added /boot to my fstab with option "auto" to not rely on the systemd-efi-generator and /boot was mounted just fine. Then I set noauto,x-systemd.automount and the autofs mount point was created as expected. Then I removed the /boot entry in fstab again and the systemd-efi-generator seems to work fine again...

I'm seriously confused right now...

btw, I did all of this mit kernel 3.9.9 booted. 3.10.2 doesn't boot on my system, it's the famous ""EFI machine randomly hangs while loading kernel" bug again. But this shouldn't have anything to do with the kernel install problem...

edit: I just realized that my "btw" is actually what OP describes. I should have read the thread completely, sorry

Either way, as far as I can see these are two separate issues. The EFI bug with the long thread that I mentioned above and the fact that /boot seems to not mount in some cases. I did some more testing regarding this, but I still have no idea what is happening. I could have sworn there was no automount point set up by systemd at some point, which would have made sense. But I can't reproduce it anymore...

Last edited by 65kid (2013-07-23 21:54:55)

65kid · 2013-07-23 22:07:30

ok, I may have figured out what I missunderstood. I didn't actually check whether the automount point was set up but whether the automount unit existed under /run/systemd/. The systemd install script does a "systemctl daemon-reexec". This generally should simply recreate these files (beyond other things). However, it seems like systemd doesn't do this if the device has already been mounted at this point (meaning really mounted, not just automount point set up). The fact that the automount unit was missing in /run somehow made me think that the automount units were never even created by the systemd-efi-generator. So I though that the ESP was not actually mounted because the automount point was missing.

Long story short, /boot was in fact mounted while I though it wasn't, so what I wrote before about the broken systemd-efi-generator is not true.

sorry for the confusion... I should really get some sleep...

Last edited by 65kid (2013-07-23 22:07:53)

ngoonee · 2013-07-24 03:22:21

Anyone filed a bug report yet? I don't use gummiboot myself, and sounds like refind isn't affected.

jasonwryan · 2013-07-24 03:37:31

ngoonee wrote:

Anyone filed a bug report yet?

The challenge is trying to work out where the bug is... 3.10.2, gummiboot or somewhere else.

All we have so far is two of us with ESPs mounted at /boot, using gummiboot and 3.10.2 not copying vmlinuz to /boot.
Unlike 65kid, when I install 3.10.2 it doesn't write the new kernel anywhere...

the.ridikulus.rat · 2013-07-24 20:32:14

I am able to boot fine with 3.10.2 and gummiboot (ESP mounted at /efisys, my own mountpoint). I use systemd path units to sync the kernel and initramfs files to /efisys/EFI/arch .

Last edited by the.ridikulus.rat (2013-07-25 18:26:55)

jasonwryan · 2013-07-25 08:13:58

It's not a gummiboot bug. I just installed syslinux and can successfully boot a custom kernel and 3.10.1, but .2 fails silently...

HalosGhost · 2013-07-25 16:35:31

Just to clarify, my ESP is mounted at /boot/efi (though I may eventually repartition and just have it mounted at boot for simplicity's sake.

@jasonwryan, was that with the [testing] edition, or the newly-dropped-to-[core] edition? If it is in the [core] edition, then perhaps this thread should be moved over to the kernel issues sub-forum since it may no longer be a [testing]-only issue.

All the best,

-HG

Last edited by HalosGhost (2013-07-25 17:02:34)

nomorewindows · 2013-07-25 19:06:49

Pardon me for the interruption, but I had different machines making different updates earlier today. But the machine I put the new kernel on from core, had a booting error. After recognizing the main disk, it gave an error about not being able to mount another partition (not root). It went to the emergency shell...and beyond that without a keyboard attached I went back to the immediate past kernel. It is not an UEFI machine; it is BIOS.

graysky · 2013-07-25 19:49:09

I use grub with no issue (EFI) but am wondering if any of you affected peeps have tried the 3.10.3-rc1 patch? I didn't actually read through the diffstat but there are >100 individual patches in this set.

EDIT: https://lkml.org/lkml/2013/7/23/801

Last edited by graysky (2013-07-25 19:52:23)

jasonwryan · 2013-07-25 20:01:00

HalosGhost wrote:

Just to clarify, my ESP is mounted at /boot/efi (though I may eventually repartition and just have it mounted at boot for simplicity's sake.
@jasonwryan, was that with the [testing] edition, or the newly-dropped-to-[core] edition? If it is in the [core] edition, then perhaps this thread should be moved over to the kernel issues sub-forum since it may no longer be a [testing]-only issue.

Still with [testing]...

HalosGhost · 2013-07-25 20:05:12

jasonwryan wrote:

Still with [testing]...

Alright, I'm about to test the [core] version and see if it goes any better.

All the best,

-HG

graysky · 2013-07-25 20:12:17

I'm pretty sure they're the same packages, just moved to a different repo.

https://projects.archlinux.org/svntogit … ages/linux

HalosGhost · 2013-07-25 20:14:13

graysky wrote:

I'm pretty sure they're the same packages, just moved to a different repo.
https://projects.archlinux.org/svntogit … ages/linux

Yeah, you're right, they're identical, and the install succeeds, but the boot still fails silently.

All the best,

-HG

Last edited by HalosGhost (2013-07-25 20:14:49)

nomorewindows · 2013-07-26 22:37:51

When it does boot it reports 3.9.9-1. Then the X windows doesn't start up.

Arch Linux

#1 2013-07-22 18:50:18

[Solved again] Updated to linux-3.10.8; Can no longer boot

#2 2013-07-22 22:17:51

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#3 2013-07-22 22:43:04

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#4 2013-07-22 22:51:16

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#5 2013-07-22 22:59:05

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#6 2013-07-23 20:59:08

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#7 2013-07-23 21:02:52

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#8 2013-07-23 21:10:17

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#9 2013-07-23 21:11:15

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#10 2013-07-23 21:18:51

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#11 2013-07-23 21:22:58

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#12 2013-07-23 21:25:44

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#13 2013-07-23 22:07:30

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#14 2013-07-24 03:22:21

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#15 2013-07-24 03:37:31

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#16 2013-07-24 20:32:14

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#17 2013-07-25 08:13:58

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#18 2013-07-25 16:35:31

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#19 2013-07-25 19:06:49

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#20 2013-07-25 19:49:09

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#21 2013-07-25 20:01:00

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#22 2013-07-25 20:05:12

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#23 2013-07-25 20:12:17

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#24 2013-07-25 20:14:13

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

#25 2013-07-26 22:37:51

Re: [Solved again] Updated to linux-3.10.8; Can no longer boot

Board footer