Unbootable system after update today - rEFInd hangs loading kernel

mcloaked · 2014-03-25 14:08:00

Having updated two systems without any problem earlier today I updated a laptop a short time ago, and when I rebooted to the new kernel (I use rEFInd as the boot manager), then rEFInd loads, goes to the normal arch boot icon, gives the familiar console message initially and then immediately says "Loading with options...." but then hangs. This system did not have lz4 installed, but knowing about the problematic symlink on my other two machines, which do have lz4 installed, after removing the offending symlink those two systems updated just fine, and they then booted without issue.

However the laptop after updating with pacman -Syu as usual and then doing

pacman -S lz4

refuses to boot. This boot issue may be unrelated to lz4 - and I booted into an arch install usbkey and removed lz4 via an arch-chroot but this makes no difference.

The system is a Lenovo S540 with hybrid Intel/ATI graphics.

Has anyone had a similar problem and if so can you suggest a way of fixing the issue?

Thanks

Last edited by mcloaked (2014-03-25 14:58:09)

karol · 2014-03-25 14:14:22

Was lz4 installed correctly? Any errors during 'mkinitcpio' or anything interesting in the logs or systemd's journal?

Edit: What other packages did you install?

Last edited by karol (2014-03-25 14:19:16)

mcloaked · 2014-03-25 14:19:08

At the time I ran pacman -Syu the package was not installed. I installed lz4 after updating but before rebooting. That install of lz4 gave no errors. It then wouldn't boot. So I then removed lz4 by booting to a rescue usbkey and then rebooted - and again the system would not boot.

I can boot the usbkey again and chroot into the system to make repairs - perhaps I should leave lz4 not installed and re-install the kernel?

if lz4 is installed is it necessary to remake the symlink /usr/lib/liblz4.so.1 -> liblz4.so ?

I presume that installing lz4 and removing it should not have caused a problem but I don't know if this is an issue or not?

Last edited by mcloaked (2014-03-25 14:19:52)

karol · 2014-03-25 14:20:49

What other packages did you install?
The symlink will be made automatically if you reinstall.

Are you using lz4 for kernel image compression?

mcloaked · 2014-03-25 14:23:59

I did not install any additional packages than lz4 today - so nothing new over and above what was working fine until today.

I had not changed mkinitcpio.conf to use lz4 and it is left as the default (gzip) as far as I know. I will double check by booting into the rescue usbkey again. I will then check the logs - but I did not notice any error messages or anything unusual at the time I ran the update.

karol · 2014-03-25 14:28:47

You said you run 'pacman -Syu' before rebooting. Post the list of packages you updated since the last successful reboot. You can use this nice script if you want: https://bbs.archlinux.org/viewtopic.php … 87#p626487

$ whatdidido
[2014-03-25 13:45] [PACMAN] starting full system upgrade
[2014-03-25 13:47] [PACMAN] upgraded ca-certificates (20140223-1 -> 20140223-2)
[2014-03-25 13:47] [PACMAN] upgraded fontconfig-infinality-ultimate (2.11.0-23 -> 2.11.1-2)
[2014-03-25 13:47] [PACMAN] upgraded x264 (20131030-2 -> 20140311-1)
[2014-03-25 13:47] [PACMAN] upgraded ffmpeg (1:2.1.4-1 -> 1:2.2-2)
[2014-03-25 13:47] [PACMAN] upgraded gstreamer0.10-ugly (0.10.19-9 -> 0.10.19-10)
[2014-03-25 13:47] [PACMAN] upgraded gstreamer0.10-ugly-plugins (0.10.19-9 -> 0.10.19-10)
[2014-03-25 13:48] [PACMAN] upgraded linux (3.13.6-2 -> 3.13.7-1)
[2014-03-25 13:48] [PACMAN] upgraded lz4 (113-1 -> 116-1)
[2014-03-25 13:48] [PACMAN] upgraded mpv (0.3.6-1 -> 0.3.6-2)
[2014-03-25 13:48] [PACMAN] upgraded nss (3.15.5-1 -> 3.15.5-2)
[2014-03-25 13:48] [PACMAN] upgraded vim-runtime (7.4.135-2 -> 7.4.214-1)
[2014-03-25 13:48] [PACMAN] upgraded vim (7.4.135-2 -> 7.4.214-1)

mcloaked · 2014-03-25 14:35:56

The first system update after its previous normal successful boot upgraded the following packages:

bash
chrony
cups-filters
foomatic-db
foomatic-db-engine
foomatic-db-nonfree
libva
libva-intel-driver
linux-firmware
linux

I checked the pacman log for problems during these updates - there were no errors reported, and the linux update proceeded with the usual messages including "Image generation successful" for the main and fallback images. I ran the linux package update again inside the rescue chroot and tried to reboot again - the same result as before (lz4 is no longer installed so presumably this issue is not related to lz4 but some other issue)

I don't know if this is a rEFInd issue or something to do with the kernel or initial ramdisk - refind does start to boot the vmlinuz image, and the last message on the screen at the point it hangs is "Using load options" and quoting the linux kernel line from the refind_linux.conf file as usual. But that is where nothing further happens.

P.S. yes that looks like a nice script in the link you posted - which I may well adopt once I sort out this boot issue.

Edit: This is a dual boot system with Windows 8.1 - I checked that it boots Windows from the rEFInd graphical screen and it does so without problems. So it is a change in the kernel or initial ramdisk during today's kernel update that seems to have initiated this problem.

Last edited by mcloaked (2014-03-25 15:00:29)

mcloaked · 2014-03-25 15:03:47

I downgraded both the kernel and kernel-headers to 3.13.6 and the system boots normally... so clearly something in the kernel 3.13.7 package caused this problem. Clearly it only affects some systems - I have now updated another laptop (different hardware) and it boots 3.13.7 just fine with an otherwise identical arch setup.

Once the system was booted back to 3.13.6 I then installed lz4 and the system rebooted normally, so this is not related to lz4, but seems to be a bug in the 3.13.7 kernel package.

Last edited by mcloaked (2014-03-25 15:19:26)

WonderWoofy · 2014-03-25 15:23:01

@mcloaked, maybe you should see if this thread gets any traction. It sounds the same as yours.

mcloaked · 2014-03-25 15:28:03

Thanks WonderWoofy: yes perhaps that thread could also be cross linked to this one? I will monitor and see if other posts provide some clues. I will report back if I find a workaround to getting my system booted with 3.13.7. My S540 is indeed also a Thinkpad!

Last edited by mcloaked (2014-03-25 15:29:40)

WonderWoofy · 2014-03-25 15:31:34

Some people have found that compiling the linux package on their own makes it work. Others (in the original thread) seemed to indicate that somehow making a direct EFISTUB firmware entry made things work, whether or not it was actually used (honestly, I don't see how this would make a difference, but whatever floats their boat).

I just keep grub-efi installed on my machine. I am not a huge grub fan, but syslinux doesn't do very well with btrfs subvolumes still, and syslinux's memdisk functionality doesn't work with EFI, so I use grub's loopback support. But keeping a traditional bootloader setup can quickly tell you whether it is EFISTUB issues you are running into, as well as serve as a backup in case the EFISTUB stuff does go wonky.

mcloaked · 2014-03-25 18:18:38

It looks like another possible instance of the bug at https://bugs.archlinux.org/task/33745

My machine was only recently installed with arch so I don't have a long history of which kernels may in the past have had this issue on my S540.

liticovjesac · 2014-03-25 19:13:10

Similar problem here. Apple laptop. Successfully start boot, start listing boot processes, it comes to virtual console line and after few more lines (too fast to read) suddenly screen goes dark. I can hear that computer is still running. I tried fallback but same thing. Any advice?

srs5694 · 2014-03-25 23:06:54

mcloaked wrote:

It looks like another possible instance of the bug at https://bugs.archlinux.org/task/33745

Yup. This issue is discussed quite heavily in this thread. The closest thing to a sure-fire solution is to switch to GRUB, ELILO, or SYSLINUX -- in other words, a boot loader that does not use the EFI stub loader. Sometimes switching from rEFInd to gummiboot, or from a rEFInd built with Tianocore to one built with GNU-EFI, fixes the problem, at least for a specific kernel. (rEFInd packages built in both ways are available on the rEFInd downloads page.)

mcloaked · 2014-03-26 09:42:17

Thanks Rod. A question - once the GNU-EFI zip file for the build is downloaded I presume that I can simply replace the existing refind_x64.efi in my system with the one from the refind-bin-gnuefi-0.7.8.zip and simply reboot to try it - without changing any of the other rEFInd files?

This bug seems to have been running for quite a while, and I would really like to know if there is any way to generate some directly useful diagnostics that could point to where the problem originates. In the past I have run a serial console output to another machine during boot to get at what is happening with line by line console output, so that when the boot fails in the early stages I already had some diagnostics, but I have not done that for around 8 years or so, and that was originally with Fedora, and I have no idea if it is even possible with my arch linux system!

Either way I am sure that all those people who have been impacted by this issue would really like to see some way to get it resolved!

Edit: There is some guidance on getting more diagnostics during boot at the two wiki pages at:

https://wiki.archlinux.org/index.php/Boot_Debugging
https://wiki.archlinux.org/index.php/wo … al_console

Maybe this approach could generate some diagnostics?

Last edited by mcloaked (2014-03-26 11:03:35)

dieelt · 2014-03-27 12:25:25

Hi!

I've got the same problem. Every kernel from 3.12.19-2 until 3.13.6-1 works perfectly fine. However, 3.13.7-1 broke booting from refind on my MacBook Pro, downgrading solved this issue. I'm thinkig: perhaps it is possible to have refind boot, e.g., grub which in turn boot my arch install w/o similar issues in the future..

mcloaked · 2014-03-27 17:19:40

I don't know if this is relevant or not but I do know that there was a significant number of radeon patches in 3.13.7, and my system has hybrid Intel/ATI graphics. Is there a possibility that one or other of the radeon related patches may have given an initial ramdisk that may be a problem right at the start of the boot process when loading modules? Is it worth comparing the various users' systems to see if there is no correlation between the current problematic kernel and the kind of hardware? If I am barking up the wrong tree here just say so.

dice · 2014-03-28 10:18:10

I have Intel/Nvidia hybrid graphics and have run into the exact same problem. So I guess it is not radeon related.

HellSt0rm · 2014-03-28 17:21:47

I have the same combo as dice: Intel/Nvidia hybrid. And also running UEFI boot with gummiboot. A downgrade of the kernel to 3.13.6-1 worked for me.
I thought it was a problem with systemd since it got upgraded also or btrfs since the kernel log ends just after mounting my btrfs root partition.

Last edited by HellSt0rm (2014-03-28 17:22:24)

mrunion · 2014-03-28 17:35:48

I assume dice and HellSt0rm, y'all have tried booting with grub or SysLinux? If it's the same UEFI thing that's going around where some Kernels boot and other's don't, try using grub or SysLinux to boot. Check out the (long) bug FS 33745 in the bug tracker. It has things to try and test out.

dieelt · 2014-03-28 20:55:45

I managed to solve it by installing a grub standalone which is booted by refind, which in turn boot arch.

sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo grub-mkstandalone -o /boot/boot.efi -d /usr/lib/grub/x86_64-efi -O x86_64-efi /boot/grub/grub.cfg

This solution is not perfect but will have to do until the next EFI-bootable kernel is released.

srs5694 · 2014-03-29 13:46:58

mcloaked wrote:

Thanks Rod. A question - once the GNU-EFI zip file for the build is downloaded I presume that I can simply replace the existing refind_x64.efi in my system with the one from the refind-bin-gnuefi-0.7.8.zip and simply reboot to try it - without changing any of the other rEFInd files?

Yes.

This bug seems to have been running for quite a while, and I would really like to know if there is any way to generate some directly useful diagnostics that could point to where the problem originates.

There probably is, but that's a topic about which I know relatively little.

To those discussing changes in recent kernels (like 3.13.6->3.13.7), forget it. This problem goes all the way back to the 3.7 kernel series. That said, it's conceivable that there are actually two or more bugs with similar symptoms, but the problem comes and goes -- that is, kernel A is buggy, kernel B is fine, kernel C is fine, kernel D is buggy, and so on. Another computer will experience a different pattern (say, A is fine, B is fine, C is buggy, and D is fine). This makes it very difficult to discern patterns, and it means that a person might have a run of several working kernels that actually contain the bug, making that person focus on a much more recent change that's actually irrelevant to the bug.

My own hypothesis is that it has something to do with alignment of the code in memory, or perhaps some other memory access issue. This could explain the near-randomness of the symptoms because this detail would vary depending on tiny differences between kernel builds and even between computers. Unfortunately, I have yet to encounter this problem myself, so I can't dig in and track it down.

mcloaked · 2014-03-30 12:37:26

I downloaded the GNU-EFI zip file for refind, and copied in the refind_x64.efi file as a simple replacement for the standard arch package version. Using this the 3.13.7 kernel boots without a problem, although this leads to the dreaded blank screen with a static underline cursor at the top left with graphical boot ( my system has Intel/ATI hybrid graphics and there are quite a few radeon patches in this new kernel) - so that is an unrelated problem that I must now look at to fix. However at least the gnu-efi version of refind does boot the previously unbootable kernel, although given Rod's comments about code alignment possibilities in memory it is entirely possible that a future kernel may not boot with this same refind binary, so time will tell.

Last edited by mcloaked (2014-03-30 17:13:26)

dice · 2014-04-03 17:49:04

Recent kernel update to 3.13.8-1 works with refind. At least for me. Anyway I installed Grub and will keep it there as a fallback when booting with refind possibly stops working again with future kernels.

dieelt · 2014-04-03 19:09:59

It does however not work for me arbitrary and strange issue.

Last edited by dieelt (2014-04-03 19:10:13)

Arch Linux

#1 2014-03-25 14:08:00

Unbootable system after update today - rEFInd hangs loading kernel

#2 2014-03-25 14:14:22

Re: Unbootable system after update today - rEFInd hangs loading kernel

#3 2014-03-25 14:19:08

Re: Unbootable system after update today - rEFInd hangs loading kernel

#4 2014-03-25 14:20:49

Re: Unbootable system after update today - rEFInd hangs loading kernel

#5 2014-03-25 14:23:59

Re: Unbootable system after update today - rEFInd hangs loading kernel

#6 2014-03-25 14:28:47

Re: Unbootable system after update today - rEFInd hangs loading kernel

#7 2014-03-25 14:35:56

Re: Unbootable system after update today - rEFInd hangs loading kernel

#8 2014-03-25 15:03:47

Re: Unbootable system after update today - rEFInd hangs loading kernel

#9 2014-03-25 15:23:01

Re: Unbootable system after update today - rEFInd hangs loading kernel

#10 2014-03-25 15:28:03

Re: Unbootable system after update today - rEFInd hangs loading kernel

#11 2014-03-25 15:31:34

Re: Unbootable system after update today - rEFInd hangs loading kernel

#12 2014-03-25 18:18:38

Re: Unbootable system after update today - rEFInd hangs loading kernel

#13 2014-03-25 19:13:10

Re: Unbootable system after update today - rEFInd hangs loading kernel

#14 2014-03-25 23:06:54

Re: Unbootable system after update today - rEFInd hangs loading kernel

#15 2014-03-26 09:42:17

Re: Unbootable system after update today - rEFInd hangs loading kernel

#16 2014-03-27 12:25:25

Re: Unbootable system after update today - rEFInd hangs loading kernel

#17 2014-03-27 17:19:40

Re: Unbootable system after update today - rEFInd hangs loading kernel

#18 2014-03-28 10:18:10

Re: Unbootable system after update today - rEFInd hangs loading kernel

#19 2014-03-28 17:21:47

Re: Unbootable system after update today - rEFInd hangs loading kernel

#20 2014-03-28 17:35:48

Re: Unbootable system after update today - rEFInd hangs loading kernel

#21 2014-03-28 20:55:45

Re: Unbootable system after update today - rEFInd hangs loading kernel

#22 2014-03-29 13:46:58

Re: Unbootable system after update today - rEFInd hangs loading kernel

#23 2014-03-30 12:37:26

Re: Unbootable system after update today - rEFInd hangs loading kernel

#24 2014-04-03 17:49:04

Re: Unbootable system after update today - rEFInd hangs loading kernel

#25 2014-04-03 19:09:59

Re: Unbootable system after update today - rEFInd hangs loading kernel

Board footer