[SOLVED] Unable to boot after upgrade to kernel 3.2.12

krigun · 2012-03-21 21:41:07

Totally clueless at this point. Ran pacman -Syu and after the upgrade to kernel 3.2.12 system does not boot anymore. After the Grub screen, the following appears almost instantly (and system freezes):

Kernel 3.2.12 boot problems

Mounted the root partition using the livecd and couldn't find any more info in any logs. All I have to show is that screenshot.

Any clues as to what this could be? Apart from a full reinstall, that is.

The motherboard is a Gigabyte Z68XP-UD3, no exotic RAID setup or anything. Disk (OCZ Vertex 3 Series 60 GB) seems fine as I was able to mount it using the livecd.

Thanks!

Last edited by krigun (2012-04-04 08:22:19)

krigun · 2012-03-21 22:25:17

~~UPDATE: Solved the problem temporarily by downgrading to kernel 2.3.11-1~~

UPDATE:
was able to boot 3.2.12 with the following kernel parameter:

pcie_aspm=force

~~Still no clues as to what might have caused the kernel panic.~~

~~Downloaded linux-3.2.11-1-x86_64.pkg.tar.xz and followed the downgrade using livecd procedure found~~ here

Note to self - don't clear pacman pkg cache (pacman -Scc) inbetween kernel updates

Last edited by krigun (2012-03-22 23:57:23)

Lampus · 2012-03-22 18:41:19

I have absolutely the same problem.
And I'm using the same motherboard: Gigabyte Z68X-UD3-B3
krigun, what release of BIOS is used in your motherboard? My Release Date: 04/15/2011
Please, show output of command:
sudo dmidecode |head -n 13

krigun · 2012-03-22 19:37:51

Interesting!

Output of dmidecode:

BIOS Information
        Vendor: Award Software International, Inc.
        Version: F3
        Release Date: 06/15/2011
        Address: 0xE0000
        Runtime Size: 128 kB
        ROM Size: 4096 kB

I'm running F3, and there are newer BIOS versions (F9 is the latest I believe). But I looked though the changelogs and didn't find anything interesting - just support for new CPU types and similar.

Perhaps it could be worthwhile to disable some of the motherboard features in BIOS and see if that helps. ~~I know this has that onboard msata thing, although I am not using it currently.~~

Oh, noticed that you have the "Z68X" and not "Z68XP", which means that you don't have the msata connector, but you have the same problem.

Last edited by krigun (2012-03-22 19:44:50)

gr2mx · 2012-03-22 19:38:21

Same problem here.
I have a Gigabyte z68xp-ud3p motherboard with bios version F3 released on 06/22/2011.

Last edited by gr2mx (2012-03-22 19:44:47)

Lampus · 2012-03-22 20:00:05

I have created bug report for this problem: https://bugs.archlinux.org/task/29063?project=1
Please, vote for it and add any additional information

krigun · 2012-03-22 20:09:15

Lampus wrote:

I have created bug report for this problem: https://bugs.archlinux.org/task/29063?project=1
Please, vote for it and add any additional information

Good initiative, but I'm not sure this is an ArchLinux bug. More likely a problem with upstream - more specifically the Linux kernel itself. I'm thinking that there is a problem in one of the device drivers in the linux kernel that this motherboard apparently triggers. But I have no idea what, and google doesn't have much to say about it either. Not sure where to go from here. If other people have the similar issue, but on a different MB, then we could possibly narrow it down to specific components, but at this point it's all up in the air.

fire_dragon · 2012-03-22 20:55:14

Same issue here on an Asus p8h67-v.

I also suspect that the kernel is the problem.
The reason this seems to be an Arch only problem, is probably that other distroes hasn't upgraded this far.
But a little disturbing that this is the only thread I can find on the issue...

krigun · 2012-03-22 21:06:08

fire_dragon wrote:

Same issue here on an Asus p8h67-v.

Interesting. Could you post the output of lspci? Given you have a running system, that is

Here is mine:

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
00:1f.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 4 port SATA IDE Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
00:1f.5 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 2 port SATA IDE Controller (rev 05)
01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 560 Ti] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)
03:00.0 PCI bridge: Integrated Technology Express, Inc. Device 8892 (rev 30)
04:02.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
06:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
08:00.0 IDE interface: Marvell Technology Group Ltd. Device 917a (rev 11)

Last edited by krigun (2012-03-22 21:06:40)

krigun · 2012-03-22 21:09:15

fire_dragon wrote:

The reason this seems to be an Arch only problem, is probably that other distroes hasn't upgraded this far.

I guess one way to find out is to compile the vanilla kernel from source and try to boot it.

fire_dragon · 2012-03-22 21:26:59

Had to downgrade as well...

We seem to share the same pci-bridge.

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation H67 Express Chipset Family LPC Controller (rev 05)
00:1f.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 4 port SATA IDE Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
00:1f.5 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 2 port SATA IDE Controller (rev 05)
01:00.0 VGA compatible controller: NVIDIA Corporation GF104 [GeForce GTX 460] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF104 High Definition Audio Controller (rev a1)
03:00.0 IDE interface: VIA Technologies, Inc. VT6415 PATA IDE Host Controller
04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
05:00.0 Ethernet controller: Atheros Communications Inc. AR8151 v2.0 Gigabit Ethernet (rev c0)
06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
07:00.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI Bridge Controller [Ver 1.0] (rev 01)
07:01.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI Bridge Controller [Ver 1.0] (rev 01)
07:02.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)

krigun · 2012-03-22 21:34:55

fire_dragon wrote:

Had to downgrade as well...
We seem to share the same pci-bridge.
...

Great - thanks. I'm currently compiling the vanilla kernel from source, and then I will try to boot that, and see if the problem persists. It probably will, and then I will have a go at trying to disable stuff in BIOS, but that might not lead anywhere.

Any idea how to capture the full output of that kernel panic message somehow?

fire_dragon · 2012-03-22 21:38:59

krigun wrote:

fire_dragon wrote:
Had to downgrade as well...
We seem to share the same pci-bridge.
...
Great - thanks. I'm currently compiling the vanilla kernel from source, and then I will try to boot that, and see if the problem persists. It probably will, and then I will have a go at trying to disable stuff in BIOS, but that might not lead anywhere.
Any idea how to capture the full output of that kernel panic message somehow?

Perhaps some strange kernel param will produce more output...
I'am really not into that stuff, but i will have a look around.

fire_dragon · 2012-03-22 21:47:24

fire_dragon wrote:

krigun wrote:
fire_dragon wrote:
Had to downgrade as well...
We seem to share the same pci-bridge.
...
Great - thanks. I'm currently compiling the vanilla kernel from source, and then I will try to boot that, and see if the problem persists. It probably will, and then I will have a go at trying to disable stuff in BIOS, but that might not lead anywhere.
Any idea how to capture the full output of that kernel panic message somehow?

Do you have a test-server running ?

Found this kernel parameter.
Setting this to 7(don't know the default) should print everything.

loglevel= All Kernel Messages with a loglevel smaller than the
console loglevel will be printed to the console. It can
also be changed with klogd or other programs. The
loglevels are defined as follows:

0 (KERN_EMERG) system is unusable
1 (KERN_ALERT) action must be taken immediately
2 (KERN_CRIT) critical conditions
3 (KERN_ERR) error conditions
4 (KERN_WARNING) warning conditions
5 (KERN_NOTICE) normal but significant condition
6 (KERN_INFO) informational
7 (KERN_DEBUG) debug-level messages

fire_dragon · 2012-03-22 21:52:39

This will probably come usefull as well...

pci=option[,option...] [PCI] various PCI subsystem options:
off [IA-32] don't probe for the PCI bus
bios [IA-32] force use of PCI BIOS, don't access
the hardware directly. Use this if your machine
has a non-standard PCI host bridge.
nobios [IA-32] disallow use of PCI BIOS, only direct
hardware access methods are allowed. Use this
if you experience crashes upon bootup and you
suspect they are caused by the BIOS.
conf1 [IA-32] Force use of PCI Configuration
Mechanism 1.
conf2 [IA-32] Force use of PCI Configuration
Mechanism 2.
nosort [IA-32] Don't sort PCI devices according to
order given by the PCI BIOS. This sorting is
done to get a device order compatible with
older kernels.
biosirq [IA-32] Use PCI BIOS calls to get the interrupt
routing table. These calls are known to be buggy
on several machines and they hang the machine
when used, but on other computers it's the only
way to get the interrupt routing table. Try
this option if the kernel is unable to allocate
IRQs or discover secondary PCI buses on your
motherboard.
rom [IA-32] Assign address space to expansion ROMs.
Use with caution as certain devices share
address decoders between ROMs and other
resources.
irqmask=0xMMMM [IA-32] Set a bit mask of IRQs allowed to be
assigned automatically to PCI devices. You can
make the kernel exclude IRQs of your ISA cards
this way.
pirqaddr=0xAAAAA [IA-32] Specify the physical address
of the PIRQ table (normally generated
by the BIOS) if it is outside the
F0000h-100000h range.
lastbus=N [IA-32] Scan all buses thru bus #N. Can be
useful if the kernel is unable to find your
secondary buses and you want to tell it
explicitly which ones they are.
assign-busses [IA-32] Always assign all PCI bus
numbers ourselves, overriding
whatever the firmware may have done.
usepirqmask [IA-32] Honor the possible IRQ mask stored
in the BIOS $PIR table. This is needed on
some systems with broken BIOSes, notably
some HP Pavilion N5400 and Omnibook XE3
notebooks. This will have no effect if ACPI
IRQ routing is enabled.
noacpi [IA-32] Do not use ACPI for IRQ routing
or for PCI scanning.
routeirq Do IRQ routing for all PCI devices.
This is normally done in pci_enable_device(),
so this option is a temporary workaround
for broken drivers that don't call it.
firmware [ARM] Do not re-enumerate the bus but instead
just use the configuration from the
bootloader. This is currently used on
IXP2000 systems where the bus has to be
configured a certain way for adjunct CPUs.

krigun · 2012-03-22 21:56:10

fire_dragon wrote:

This will probably come usefull as well...

Excellent - thank you! Almost done compiling the vanilla kernel now, will then have a go at booting it up and playing around with those parameters. Will post back in a few moments.

Picoli · 2012-03-22 22:12:13

Hi!

I'm having the same problem.

Motherboard: Gigabyte GA-P67X-UD3-B3

By the way: downgrading is not a solution...

fire_dragon · 2012-03-22 22:17:49

Picoli wrote:

By the way: downgrading is not a solution...

Have you followed this guide ?
https://wiki.archlinux.org/index.php/Kernel_panic

fire_dragon · 2012-03-22 22:25:55

fire_dragon wrote:

Picoli wrote:
By the way: downgrading is not a solution...
Have you followed this guide ?
https://wiki.archlinux.org/index.php/Kernel_panic

Or did you just state that downgrading is a workaround and not a solution ?

Could you post the output of lspci ?

Picoli · 2012-03-22 22:28:53

fire_dragon wrote:

Picoli wrote:
By the way: downgrading is not a solution...
Have you followed this guide ?
https://wiki.archlinux.org/index.php/Kernel_panic

I meant it in a general sense. But never mind. I hope this gets solved soon.

mefistofeles · 2012-03-22 22:59:28

Same happened with a Gigabyte-H67A-UD3H-B3 with Award BIOS. Solved by downgrading to 3.2.11 (with arch's cd and chrooting to my installed system).

krigun · 2012-03-22 23:50:18

Progress.

After a lot of mocking around with various kernel versions, I ended up testing the 3.3. Same problem, but the 3.3 kernel gave me a bit more info to work with.

Apparently the kernel panic was in "pcie_aspm_init_link_state" if I interpret it correctly, and apparently appending the following to your boot line will boot the system on 3.2.12:

pcie_aspm=force

This worked on both 3.3 and 3.2.12. So it seems that the problem lies within ASPM

Seems to be a kernel bug though. More info here

Last edited by krigun (2012-03-22 23:59:47)

fire_dragon · 2012-03-23 12:33:50

krigun wrote:

Progress.
After a lot of mocking around with various kernel versions, I ended up testing the 3.3. Same problem, but the 3.3 kernel gave me a bit more info to work with.
Apparently the kernel panic was in "pcie_aspm_init_link_state" if I interpret it correctly, and apparently appending the following to your boot line will boot the system on 3.2.12:
pcie_aspm=force
This worked on both 3.3 and 3.2.12. So it seems that the problem lies within ASPM
Seems to be a kernel bug though. More info here

Thanks...
That more or less bring us back on track.
I haven't tried it out though...

Found this at Bugzilla
Lets see what they come up with...

krigun · 2012-03-23 12:57:47

fire_dragon wrote:

Found this at Bugzilla
Lets see what they come up with...

Great find - this certainly looks like the same problem that we have been experiencing here aswell. Good to know that the problem is getting attention.

Picoli · 2012-03-23 13:16:54

Works for my MB as well. Thanks a lot!

Arch Linux

#1 2012-03-21 21:41:07

[SOLVED] Unable to boot after upgrade to kernel 3.2.12

#2 2012-03-21 22:25:17

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#3 2012-03-22 18:41:19

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#4 2012-03-22 19:37:51

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#5 2012-03-22 19:38:21

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#6 2012-03-22 20:00:05

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#7 2012-03-22 20:09:15

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#8 2012-03-22 20:55:14

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#9 2012-03-22 21:06:08

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#10 2012-03-22 21:09:15

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#11 2012-03-22 21:26:59

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#12 2012-03-22 21:34:55

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#13 2012-03-22 21:38:59

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#14 2012-03-22 21:47:24

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#15 2012-03-22 21:52:39

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#16 2012-03-22 21:56:10

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#17 2012-03-22 22:12:13

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#18 2012-03-22 22:17:49

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#19 2012-03-22 22:25:55

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#20 2012-03-22 22:28:53

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#21 2012-03-22 22:59:28

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#22 2012-03-22 23:50:18

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#23 2012-03-23 12:33:50

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#24 2012-03-23 12:57:47

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

#25 2012-03-23 13:16:54

Re: [SOLVED] Unable to boot after upgrade to kernel 3.2.12

Board footer