Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

andreagi · 2012-03-08 13:42:07

Hello, I'm trying to record a kernel panic on boot for the kernels 3.2.9-1-ck & 3.2.6-1-pf (no problem with the previous release and the Archlinux official kernel).

The kernel panic is "due" to udevd, with a lot of patience I manually copied what displayed on screen.
BTW does anybody know how to record it? I tried with netconsole but no luck. Is there any way to reliable track a kernel panic on boot?

And most of all, does anybody know if there is a workaround for the report below?

Thank you,
Andrew.

Edit: Few hours after my post I upgraded to 3.2.9-1-ARCH and discovered it has the same issue.

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [< (null)>] (null)
PGD 250e50067 PUD 250e51067 PMD 0
Oops: 0010 [#1 PREEMPT SMP
CPU 0
Modules linked in: ite_cir(+) rc_core thermal tun fuse ext4 crc16 jbd2 mbcache usbhid hid sr_mod sd_mod cdrom ahci libahci libata ohci_hcd scsi_mod ehci_hcd usbcore forcedeth usb_common

Pid: 156, comm: udevd Not tainted 3.2.9-1-ck #1 Alienware M17X /
RIP: 0010[<00000000000000000>] [< (null)>] (null)
RSP: 0018:ffff88025fc03df0 EFLAGS: 00010046
RAX: 0000000000000097 RBX: ffff88025171e21c RCX: 0000000000000000
RDX: 0000000000000097 RSI: ffff88025171e200 RDI: ffff88025171e200
RBP: ffff88025fc03e88 R08: ffff880257005c00 R09: ffff880256c00068
R10: 0000000000000000 R11: 0000000000000001 R12: ffff88025171e200
R13: 0000000000000097 R14: ffff8802506abbb8 R15: 000000000000000a
FS: 00007ff07746d740(0000) GS:ffff88025fc00000(0000) knIGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000002507ab000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
Process udevd (pid: 156, threadinfo ffff8802506aa000, task ffff8802504b5cc0)
Stack:
ffffffffa012c2ae ffff88025fc13600 ffff88025fc03e00 ffff88025fc03e00
0000000000000000 ffffffff81a28ac0 0000000000000001 0000000000000001
ffff88025fc03ec0 ffffffff810691b0 000000015b190612 ffff8802506abfd8
Call Trace:
<IRQ>
[<ffffffffa012c2ae>] ? ite_circ?isr+0x5e/0x290 [ite_cir]
[<ffffffff810691b0>] ? run_timer_softirq+0x1a0/0x440
[<ffffffff810be3b5>] handle_irq_event_percpu+0x75/0x2a0
[<ffffffff8108c0c4>] ? clockevents_program_event+0x75/0x2a0
[<ffffffff810be628>] handle_irq_event+0x74/0x100
[<ffffffff810c1637>] handle_edge_irq+0x77/0x120
[<ffffffff81017792>] handle_irq+0x22/0x40
[<ffffffff8145b35a>] do_IRQ+0x5a/0xe0
[<ffffffff81457fae>] common_interrupt+0x6e/0x6e
<EOI>
[<ffffffff81037840>] ? ioapic_set_affinity+0x50/0xf0
[<ffffffff814579f0>] ? _raw_spin_unlock_irqrestore+0x10/0x40
[<ffffffff810bfd8e>] __setup_irq+0x3de/0x540
[<ffffffff81145ad3>] ? kmem_cache_alloc_trace+0x133/0x160
[<ffffffffa012c250>] ? ite_tx_ir+0x480/0x480 [ite_cir]
[<ffffffff810bffda>] request_threaded_irq+0xea/0x1a0
[<ffffffff81053190>] ? try_to_wake_up+0x1f0/0x1f0
[<ffffffffa012bb52>] ite_probe+0x2b2/0x530 [ite_cir]
[<ffffffffa012b8a0>] ? it8709_enable_rx+0x90/0x90 [ite_cir]
[<ffffffff812b5420>] pnp_device_probe+0x70/0xf0
[<ffffffff81302b0a>] ? driver_sysfs_add+0x7a/0xb0
[<ffffffff81302d6e>] driver_probe_device+0x7e/0x1b0
[<ffffffff81302f4b>] __driver_attach+0xab<70xb0
[<ffffffff81302ea0>] ? driver_probe_device+0x1b0/0x1b0
[<ffffffff81302ea0>] ? driver_probe_device+0x1b0/0x1b0
[<ffffffff81301d2c>] bus_for_each_dev+0x5c/0x90
[<ffffffff813029fe>] driver_attach+0x1e/0x20
[<ffffffff81302650>] bus_add_driver+0x1a9/0x270
[<ffffffffa012c4e0>] ? ite_cir_isr+0x290/0x290 [ite_cir]
[<ffffffff813034b6>] driver_register+0x76/0x140

Last edited by andreagi (2012-03-08 19:49:01)

falconindy · 2012-03-08 13:53:13

ite_cir looks like an obvious candidate for blame. Given that this is a kernel you compiled yourself with third party patches, the possibilities are endless. Common cause, though, would be setting your interrupt frequency too high (anything above 1000Hz is prone to breaking drivers).

andreagi · 2012-03-08 19:37:55

falconindy wrote:

ite_cir looks like an obvious candidate for blame. Given that this is a kernel you compiled yourself with third party patches, the possibilities are endless. Common cause, though, would be setting your interrupt frequency too high (anything above 1000Hz is prone to breaking drivers).

Hi, when I wrote the message I was running the kernel 3.2.8-1-ARCH. Few hours later the 3.2.9-1-ARCH reached the mirrors and I upgraded to it.
Unfortunately with exactly the same issue.

Now I downgraded to 3.2.8-1-ck and I'm back, but I can confirm that also the "stock" Archlinux kernel has this issue.

I have a lot of difficult in reporting because the oops is happening so early in the boot sequence. I tried to configure netconsole, with no luck and I don't have a serial port directly connected to the laptop. Is there any way to capture the report?

Bye,
Andrew

graysky · 2012-03-08 20:26:32

andreagi wrote:

Hi, when I wrote the message I was running the kernel 3.2.8-1-ARCH. Few hours later the 3.2.9-1-ARCH reached the mirrors and I upgraded to it.
Unfortunately with exactly the same issue.

OK, so it's not the fault of the linux-ck package. Something in upstream is likely.

llg179 · 2012-03-09 00:17:39

I tried to compile 3.2.9 with traditional way. It did not boot either. After some days googling I can summarize the troubleshoot methods:
~ during boot use e to edit grub's config. Append init=/bin/bash to end of line where the bzImage is specified then press b to boot. You can check some things with the bash shell
~ use live cd and mount the root partition

In my case /dev of the root partition was totally empty. After creating /dev/console (mknod -m 660 console c 5 1) and /dev/null (mknod -m 660 null c 1 3) I could see that boot goes on. After knowing that dev is empty searching went better : devtmpfs must be configured in kernel in order to boot archlinux. See : http://www.archlinux.org/news/users-of- … s-support/
After adding the proper config items (see README there) and after a make && install process it booted properly.

For me the pacman-installable 3-2-9-1 boots well.
During a custom kernel build you can specify the processor type which I guess helps to increase system performance. How can I check this setting in the pacman installed kernel 3-2-9 ?

Thanks

Last edited by llg179 (2012-03-09 00:23:50)

graysky · 2012-03-09 00:49:27

The ARCH kernel uses generic x86_64. I forget what it's called for i686... pentiumpro maybe? What happens if you compile linux-ck without altering the config at all? Switch off the "nconfig" option.

Johnyjohn · 2012-03-11 11:54:38

Hello,

I have the same problem (i think) since i'm running 3.2.9-1-ARCH. I get a kernel panic at boot time during udev events processing but the log is too long to be visible on the screen and i didn't found how to log it.

I'm not using testing repo or custom kernel or other stuff. Just a regular user.

Here is the begining :

Waiting for udev events to be processed
BUG: unable to handle kernel
ite_cir: TX-capable: 1
ite_cir: Sample period (ns): 8680
ite_cir: TX carrier frequency (Hz): 38000
ite_cir: IX duty_cycle (%): 33
NULL pointer dereference at (null)
IP: [< (null)>] (null)
PGD 1501f7067 PUD 1581fc067 PMD 0
Oops: 0010 [#1] PREEMPT SMP
CPU 0

Then it's about linked modules, udevd tainted and bad rip values.

The kernel panic message is not always the same, but ite_cir appears in each (not with the same content).

I tried to chroot and run mkinitcpio thinking that an error may have occured in kernel upgrade but it's not solving the problem.

Any idea ?

Thant you.

Miche · 2012-03-11 18:25:11

i think i have the same problem too. In my system boot process stops while "Loading udev events" and then a kernel trace is printed. Tha trace is a bit different though. I made a picture of it.
In the last days (just before all the problem started) linux and nvidia was updated respectively to 3.2.9-1 e 295.20-3.
I tried disabling (via chroot) nvidia in favor of nouveau but that didn't solve.
would you suggest to downgrade as andreagi did?

uastasi · 2012-03-11 23:15:52

I have the same issue too but I can't downgrade since I have no fallback mode working. Could you explain me how to do that? Thank you in advance.
[Stock Kernel and Nvidia drivers updated today]

Last edited by uastasi (2012-03-11 23:23:50)

uastasi · 2012-03-12 09:13:34

uastasi wrote:

I have the same issue too but I can't downgrade since I have no fallback mode working. Could you explain me how to do that? Thank you in advance.
[Stock Kernel and Nvidia drivers updated today]

Sorry, I had to mount -rw root partition then I did it.

Miche · 2012-03-12 13:09:40

uastasi wrote:

uastasi wrote:
I have the same issue too but I can't downgrade since I have no fallback mode working. Could you explain me how to do that? Thank you in advance.
[Stock Kernel and Nvidia drivers updated today]
Sorry, I had to mount -rw root partition then I did it.

Can you explain better? you downgraded? or the problem was root was mounted rw during boot?

Miche · 2012-03-12 14:03:51

I downgraded (for more info look up Chroot and Downgrading) linux to 3.2.8-1 and now the system is booting again.

uastasi · 2012-03-12 15:38:11

Miche wrote:

uastasi wrote:
uastasi wrote:
I have the same issue too but I can't downgrade since I have no fallback mode working. Could you explain me how to do that? Thank you in advance.
[Stock Kernel and Nvidia drivers updated today]
Sorry, I had to mount -rw root partition then I did it.
Can you explain better? you downgraded? or the problem was root was mounted rw during boot?

Yes I downgraded to kernel 3.2.8 and nvidia 295.20-2.
Before I wasn't able to do that since I had to mount rw the root partition.

bryogenic · 2012-03-13 03:24:33

I have this issue as well. Almost identical. I'm relatively new to arch so I just re-installed my root partition back to 3.0-ARCH. My sophomoric questions follow below...

Any idea how long I will need to treat pacman -S linux-headers like a death wish?
Would installing linux-lts be a amicable workaround for someone like me who doesn't need the bleeding edge kernels?
Would adding linux to the ignore list in pacman.conf allow me to do regular updates without any adverse side affects?

Edit - Has this been submitted as a bug? I didn't see any in the official kernel packages bug tracker.

Update - installing linux-3.2.8-1 and nvidia-295.20 got my machine back to a working state, thanks!

Last edited by bryogenic (2012-03-14 22:49:46)

pacheco017 · 2012-03-14 00:12:35

I think I´m also having this same problem.

Similar to what uastasi said, I upgraded packages 'linux' from 3.2.8-1 to 3.2.9-1 and 'nvidia' from 295.20-2 to 295.20-3 (among others) which made the system unbooteable. As Miche said, downgrading just 'linux' to 3.2.8-1 made it work again.

Is there anything I can do to help solving this problem? Linux newbie here.

Edit: Maybe this is unrelated, but I also had to downgrade 'nvidia' to 295.20-2, otherwise I get this error when trying to start SLiM (in /var/log/slim.log): '
ERROR: could not insert 'nvidia': Exec format error

Last edited by pacheco017 (2012-03-14 00:28:25)

mgzpac · 2012-03-15 10:03:46

Has anyone managed to get 3.2.9-1 working / identified the source of the issue?

wanderingmathy · 2012-03-15 23:49:30

This may or may not be related: I can boot the 3.2.9-1-ARCH kernel just fine, but I've had intermittent kernel panics after my system has been up and running. It could be a hardware issue - I built this desktop last week, but I'm suspicious as I had no issues during my ill-fated sojourn to Debian. Possibly of note(?) is that I'm using a Radeon card, while most of you all seem to have nVidias.

OBLiQUE · 2012-03-16 20:05:09

in my case i had to alias my sound card

/etc/modprobe.d/sound.conf:

alias snd-card-0 snd-hda-intel
alias snd-slot-0 snd-hda-intel

i also cp it at /lib/modprobe.d

now i can boot..

Last edited by OBLiQUE (2012-03-16 20:05:33)

uastasi · 2012-03-17 08:44:42

Upgrading to Kernel 3.2.11 the issue happens again. Sooo boring.

Wilco · 2012-03-17 09:37:00

I had a similar issue that was caused by blacklisted modules that shouldn't be blacklisted. Could be worth investigating, in my case the broadcom-wl driver was loaded together with the brcmsmac driver causing a kernel panic.

dualbus · 2012-03-18 17:37:25

This is the reason that made me run away after upgrading Debian from Squeeze to Sid. I couldn't even pass the boot-loading. Then I tried Arch Linux, and after an upgrade it also broke. I thought it was the kernel, so I tried doing upgrades, and all kinds of weird things.

After all, the solution was to blacklist the "ite_cir" module. So 'echo blacklist ite_cir >> /etc/modprobe.d/ite_cir.conf' did the job. (Tip: For newbies like me that don't know how to do that change if you can't even boot, try the live CD - mount the partition where /etc lives, and do the editing from there.).

mgzpac · 2012-03-20 20:40:07

Tried that, got a kernel panic with new messages.

OBLiQUE · 2012-03-21 00:34:29

try this

cp /etc/modprobe.d/* /lib/modprobe.d

OBLiQUE · 2012-03-21 23:16:11

ok i found the main problem.. i put a printk() in load_module() kernel function to see which module causes the kernel panic.. and is the module video.. so i did

echo blacklist video >> /etc/modprobe.d/video.conf

i don't have the time to debug the kernel to see what is actually wrong in the code

PS: everything is working ok.. i have gui etc..

Last edited by OBLiQUE (2012-03-21 23:17:01)

uastasi · 2012-03-22 16:53:06

Is there any bug filled yet or am I not able to find it?

Arch Linux

#1 2012-03-08 13:42:07

Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#2 2012-03-08 13:53:13

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#3 2012-03-08 19:37:55

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#4 2012-03-08 20:26:32

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#5 2012-03-09 00:17:39

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#6 2012-03-09 00:49:27

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#7 2012-03-11 11:54:38

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#8 2012-03-11 18:25:11

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#9 2012-03-11 23:15:52

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#10 2012-03-12 09:13:34

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#11 2012-03-12 13:09:40

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#12 2012-03-12 14:03:51

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#13 2012-03-12 15:38:11

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#14 2012-03-13 03:24:33

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#15 2012-03-14 00:12:35

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#16 2012-03-15 10:03:46

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#17 2012-03-15 23:49:30

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#18 2012-03-16 20:05:09

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#19 2012-03-17 08:44:42

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#20 2012-03-17 09:37:00

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#21 2012-03-18 17:37:25

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#22 2012-03-20 20:40:07

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#23 2012-03-21 00:34:29

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#24 2012-03-21 23:16:11

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

#25 2012-03-22 16:53:06

Re: Kernel (3.2.9-1-ck & 3.2.6-1-pf) panic on boot. Edit: also 3.2.9-1

Board footer