[SOLVED] Above 4G Decoding prevents the kernel from loading

Omar007 · 2015-12-14 21:22:21

If above 4G Decoding is enabled in the system BIOS (ASUS X99-A) Arch Linux will not boot for me. The systemd boot loader (previously gummiboot) does show but as soon as the loader attempts to boot the kernel, the screen just stays black.
I've also attempted to boot the latest Arch ISO but the ISO is showing the exact same behaviour. I have no problem booting Windows.

Mobo: ASUS X99-A
BIOS Version: 2001
Running in pure UEFI mode
Kernel: 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux
Alt. OS: Windows 10

I'm sorry I can't provide much more information than this but I just don't have any log files for this.
Does anyone have a clue as to what is causing this behaviour?

Last edited by Omar007 (2016-01-01 22:29:33)

Lone_Wolf · 2015-12-15 11:29:21

Maybe this helps a bit : https://www.asus.com/support/FAQ/1004170/

Do you have 64-bit PCIe devices that NEED to be above the 4G boundary?

Omar007 · 2015-12-16 15:46:16

Thanks for the link but I had already stumbled across that FAQ. However, it literally contains no information that anyone could possibly use imho. Furthermore, if I were to take it literally, it shouldn't even be disabled unless you were using GRID cards (and sadly there is no date on the FAQ page because iirc I read on the nVidia GRID forums that this hasn't been needed since at least somewhere in the first half of this year).
And I'm aware that Arch boots when this option is disabled or non-existent. The question is why doesn't it boot when this option is enabled?

I have some more references relating to this setting as well but none help me in the explanation of why Arch has problems booting with this setting, especially since the whole system consists completely of 64bit capable hardware and software:
https://en.wikipedia.org/wiki/PCI_hole# … emory_hole
https://en.wikipedia.org/wiki/Memory-mapped_I/O
http://resources.infosecinstitute.com/s … d-systems/
http://resources.infosecinstitute.com/s … d-systems/
And an older one but was still an interesting read:
https://blogs.technet.microsoft.com/mar … al-memory/

Last edited by Omar007 (2015-12-16 16:24:26)

Lone_Wolf · 2015-12-17 10:31:57

Ok, sounds like Asus "Above 4G Decoding" is similar to "allow relocation of PCI memory hole" in other firmware.

Does the asus firmware list options that mention IOMMU ?
(practically all x86_64 OSes need IOMMU, but many manufacturers disable that by default because it can give problems for 32-bit OSes ).

Are you using latest firmware for your motherboard ?

mich41 · 2015-12-17 13:22:55

Short solution: disable it unless you really use 8 GPUs or some HPC card with 8 gigs of CPU-mapped RAM.
Long solution: start with removing "quiet" from kernel command line.

Omar007 · 2015-12-18 01:06:30

Lone_Wolf wrote:

Does the asus firmware list options that mention IOMMU ?

Yes it has (Intel VT-d) and this is enabled.

Lone_Wolf wrote:

Are you using latest firmware for your motherboard ?

Yes, version 2001 (released on 2015-11-19).

mich41 wrote:

Long solution: start with removing "quiet" from kernel command line.

I would remove it if I had ever added it. I'm never booting 'quiet' ;P.
My boot options are as follows:

root=UUID=7c754022-0b85-4c36-bc48-fd084ff49888 rw intremap=no_x2apic_optout init=/lib/systemd/systemd

When I need to do some PCIe device forwarding I also have 'intel_iommu=on' in there as well. I don't have this in there permanently due DMAR errors when using the nVidia driver + Xorg config files so when I need to do forwarding I switch to nouveau first.

I get the impression it's already halting at the point where it should be loading up the initial ram disk (mkinitpcio).

Last edited by Omar007 (2015-12-18 01:08:14)

mich41 · 2015-12-18 08:51:33

You can remove initramfs temporarily. The kernel should still boot and print "panic - unable to mount root fs".

And lets get the other thing straight - something clears the screen and nothing is printed even though you always boot without quiet?

Omar007 · 2015-12-18 18:23:49

Disabled the initramfs but that did not result in different behaviour if above 4G decoding is enabled. It still stays stuck on a pure black screen after the boot loader.

And to get it clear, all that happens is: boot loader -> black screen. Without the above 4G decoding it'd be: boot loader -> initramfs output -> kernel/systemd output -> GDM

(for the record, if 4G decoding is disabled removing the initramfs does result in a kernel panic)

mich41 · 2015-12-18 19:12:34

I think we can agree now that it has nothing to do with loading initramfs.

Omar007 wrote:

boot loader -> initramfs output -> kernel/systemd output -> GDM

Oh, it seems that Arch devs thought it would be very funny to change the default loglevel so that quiet becomes a no-op.

Add loglevel=7 to enable full kernel spam. Maybe it will say something interesting before dying.

Omar007 · 2015-12-18 21:07:35

mich41 wrote:

I think we can agree now that it has nothing to do with loading initramfs.

Yup. This is basically confirming my feeling that it was halting before it even loaded the initial ram disk.

mich41 wrote:

Add loglevel=7 to enable full kernel spam. Maybe it will say something interesting before dying.

Tried this, as well as 'debug' and 'ignore_loglevel'. In all of the cases the screen stayed black though..

In the meantime I also attempted a boot of Fedora Workstation 23 to see what would happen with a different boot loader. Result was exactly the same behaviour; instant black screen after the boot loader.

mich41 · 2015-12-18 22:19:27

Omar007 wrote:

Yup. This is basically confirming my feeling that it was halting before it even loaded the initial ram disk.

I think we aren't quite agreeing on something The initramfs is "loaded" (as in "read from disk and put in RAM") by the bootloader before kernel execution. Then the kernel is started (and prints several hundred lines of spam to console if you set loglevel=7). Once initialized, the kernel executes software contained on initramfs which prints some more spam, loads disk drivers, mounts the root partition and launches some more software from there.

So it seems to hang during early kernel execution, which is after loading initramfs

But yeah, initramfs is irrelevant if it hangs even without one.

Omar007 wrote:

Tried this, as well as 'debug' and 'ignore_loglevel'. In all of the cases the screen stayed black though..

Did you verify that these options work (i.e. cause the kernel to print lots of garbage) with 4G decoding disabled? Maybe I got something wrong or maybe there is something weird about booting in UEFI mode - frankly I've never used this thing so far.

Omar007 · 2015-12-19 01:31:54

mich41 wrote:

I think we aren't quite agreeing on something The initramfs is "loaded" (as in "read from disk and put in RAM") by the bootloader before kernel execution. Then the kernel is started (and prints several hundred lines of spam to console if you set loglevel=7). Once initialized, the kernel executes software contained on initramfs which prints some more spam, loads disk drivers, mounts the root partition and launches some more software from there.
So it seems to hang during early kernel execution, which is after loading initramfs

Ah ok thanks for explaining how the initramfs is used in the boot process in a bit more detail. Then loaded is indeed the incorrect term to use in this case. Executed would have been more appropriate I guess
Then I guess the next question would be; how can one debug this early kernel execution if the use of loglevel=7, debug and ignore_loglevel do not help?

mich41 wrote:

Did you verify that these options work (i.e. cause the kernel to print lots of garbage) with 4G decoding disabled?

Always do
No sense in reporting results if the parameters have no impact/make no change anyway right?

Last edited by Omar007 (2015-12-19 01:41:35)

mich41 · 2015-12-19 11:10:13

Omar007 wrote:

Then I guess the next question would be; how can one debug this early kernel execution if the use of loglevel=7, debug and ignore_loglevel do not help?

It's possible to use serial port, if the machine has one. Or hack the code to communicated using beeps generated by direct programming of the PIT

Normally Linux is supposed to print quite a lot of debugging output before it starts to play with PCI, so your hangs may be caused by some bad interaction with UEFI. Maybe even the UEFI facilities used to print on the screen.

I'd check if there is some way to disable using UEFI framebuffer (and don't display anything until native GPU drivers kick in).

And, of course, first of all - I wouldn't be using UEFI mode Actually, you may even check if it boots in BIOS mode. To avoid messing with UEFI boot chain on the HDD, make a USB stick with syslinux or grub 0.97 and your vmlinuz image (and nothing more, just see if it boots an panics). One advantage of BIOS mode is that it uses very simple 80x25 text mode which only requires accessing GPU memory at 0xb8000 because it has to work with 16-bit DOS.

Omar007 wrote:

No sense in reporting results if the parameters have no impact/make no change anyway right?

There certainly are people who don't think like that

Omar007 · 2015-12-20 16:57:37

I've tried booting with nomodeset, nofb, vga=normal and even tried vga=ask but even without 4G decoding these do not seem to make any change in the boot process. I'm not aware if any other kernel parameters that play with the framebuffer.

Next up I configured my system to boot in BIOS/CMS mode with 4G decoding enabled. Linux did boot but without UEFI, not all of my PCIe cards are working properly (e.g. At most 1 of the GPUs will be active at a given time)

Last edited by Omar007 (2015-12-20 16:59:34)

Omar007 · 2016-01-01 22:28:42

I've updated my kernel today from 4.2 to 4.3 and this now seems to be working perfectly fine

The system booted just fine and below an example of one of the GPU's being decoded in above 4G address space:

02:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd Device 366a
Physical Slot: 6
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 45
NUMA node: 0
Region 0: Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 383fe0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at 383ff0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at d000 [size=128]
[virtual] Expansion ROM at f9000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia

Marking this topic as solved.

Last edited by Omar007 (2016-01-01 22:29:04)

Apneal · 2017-04-27 17:30:40

I just wanted to add to this because I spent days trying to resolve this same issue and this is one of the topics I found.

The fix for me was to have a full UEFI/GPT installation, and to boot with the kernel's EFI stub and circumvent grub completely. This fixed the issue for me!

WorMzy · 2017-04-27 23:12:07

Welcome to the forums, Apneal. Please take the time to read our Code of Conduct, and in the future, please do not necrobump.

https://wiki.archlinux.org/index.php/Co … bumping.22

Closing.

Arch Linux

#1 2015-12-14 21:22:21

[SOLVED] Above 4G Decoding prevents the kernel from loading

#2 2015-12-15 11:29:21

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#3 2015-12-16 15:46:16

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#4 2015-12-17 10:31:57

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#5 2015-12-17 13:22:55

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#6 2015-12-18 01:06:30

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#7 2015-12-18 08:51:33

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#8 2015-12-18 18:23:49

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#9 2015-12-18 19:12:34

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#10 2015-12-18 21:07:35

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#11 2015-12-18 22:19:27

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#12 2015-12-19 01:31:54

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#13 2015-12-19 11:10:13

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#14 2015-12-20 16:57:37

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#15 2016-01-01 22:28:42

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#16 2017-04-27 17:30:40

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

#17 2017-04-27 23:12:07

Re: [SOLVED] Above 4G Decoding prevents the kernel from loading

Board footer