Kernel lockups for about 2 weeks

mpan · 2012-08-01 21:11:02

Note: this thread may be related to similar threads lately posted on the forum. However in my case laptop/WiFi issues can't be the cause, so I'm starting a new thread.

For about 2 weeks I'm getting random soft lockups. Every time history is the same: some GUI application hangs and after about half minute everything freezes. In some cases I can get lockup-related entries in my logs. Here is an example of one (I was working in Nautilus, when this happened, but the lockups are happening during working with other apps too):

Jul 27 02:06:58 localhost kernel: [343112.103331] BUG: soft lockup - CPU#1 stuck for 23s! [nautilus:853]
Jul 27 02:06:58 localhost kernel: [343112.103334] Modules linked in: fuse usb_storage uas nvidia(PO) iTCO_wdt snd_hda_codec_realtek intel_agp intel_gtt serio_raw psmouse pcspkr i2c_i801 r8169 snd_hda_intel iTCO_vendor_support snd_hda_codec microcode evdev processor i2c_core snd_hwdep snd_pcm snd_page_alloc snd_timer snd soundcore mii ppdev parport_pc parport button coretemp ext4 crc16 jbd2 mbcache sr_mod cdrom sd_mod pata_jmicron pata_acpi ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore usb_common
Jul 27 02:06:58 localhost kernel: [343112.103366] CPU 1
Jul 27 02:06:58 localhost kernel: [343112.103368] Modules linked in: fuse usb_storage uas nvidia(PO) iTCO_wdt snd_hda_codec_realtek intel_agp intel_gtt serio_raw psmouse pcspkr i2c_i801 r8169 snd_hda_intel iTCO_vendor_support snd_hda_codec microcode evdev processor i2c_core snd_hwdep snd_pcm snd_page_alloc snd_timer snd soundcore mii ppdev parport_pc parport button coretemp ext4 crc16 jbd2 mbcache sr_mod cdrom sd_mod pata_jmicron pata_acpi ata_generic ata_piix uhci_hcd libata ehci_hcd scsi_mod usbcore usb_common
Jul 27 02:06:58 localhost kernel: [343112.103391]
Jul 27 02:06:58 localhost kernel: [343112.103393] Pid: 853, comm: nautilus Tainted: P O 3.4.6-1-ARCH #1 Gigabyte Technology Co., Ltd. EP43-DS3LR/EP43-DS3LR
Jul 27 02:06:58 localhost kernel: [343112.103397] RIP: 0010:[<ffffffff81469912>] [<ffffffff81469912>] _raw_spin_lock+0x32/0x40
Jul 27 02:06:58 localhost kernel: [343112.103405] RSP: 0018:ffff880127ce3e18 EFLAGS: 00000282
Jul 27 02:06:58 localhost kernel: [343112.103407] RAX: 00000000000000c0 RBX: ffff880127ce3e08 RCX: 0000000000000000
Jul 27 02:06:58 localhost kernel: [343112.103410] RDX: 0000000000000040 RSI: 000000000800afce RDI: ffff880118e73600
Jul 27 02:06:58 localhost kernel: [343112.103412] RBP: ffff880127ce3e18 R08: 0000000000000000 R09: 0000000000000000
Jul 27 02:06:58 localhost kernel: [343112.103415] R10: ffffffff8117a503 R11: ffff880037e576f8 R12: ffffffff8114488d
Jul 27 02:06:58 localhost kernel: [343112.103417] R13: ffff880127ce3db8 R14: ffff880118e736c0 R15: ffff88012816af20
Jul 27 02:06:58 localhost kernel: [343112.103420] FS: 00007f8c3d62f900(0000) GS:ffff88012fc80000(0000) knlGS:0000000000000000
Jul 27 02:06:58 localhost kernel: [343112.103423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 27 02:06:58 localhost kernel: [343112.103425] CR2: 00007f8c3d472000 CR3: 0000000127cee000 CR4: 00000000000407e0
Jul 27 02:06:58 localhost kernel: [343112.103427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 27 02:06:58 localhost kernel: [343112.103430] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 27 02:06:58 localhost kernel: [343112.103433] Process nautilus (pid: 853, threadinfo ffff880127ce2000, task ffff8801275ecf60)
Jul 27 02:06:58 localhost kernel: [343112.103434] Stack:
Jul 27 02:06:58 localhost kernel: [343112.103435] ffff880127ce3e78 ffffffff811ad65d 0000000000000000 ffff8801222bddb0
Jul 27 02:06:58 localhost kernel: [343112.103439] ffff8801222bde48 ffff880118e7371c ffff8801222bdd30 ffff8801222bdd30
Jul 27 02:06:58 localhost kernel: [343112.103442] ffff880037e57708 ffff88011b4ea874 ffff88011b4ea800 ffff880037e576e8
Jul 27 02:06:58 localhost kernel: [343112.103446] Call Trace:
Jul 27 02:06:58 localhost kernel: [343112.103451] [<ffffffff811ad65d>] __fsnotify_update_child_dentry_flags+0xed/0x140
Jul 27 02:06:58 localhost kernel: [343112.103455] [<ffffffff811aecf6>] fsnotify_add_mark+0x1a6/0x220
Jul 27 02:06:58 localhost kernel: [343112.103457] [<ffffffff811b0905>] sys_inotify_add_watch+0x315/0x3b0
Jul 27 02:06:58 localhost kernel: [343112.103461] [<ffffffff8146ab29>] system_call_fastpath+0x16/0x1b
Jul 27 02:06:58 localhost kernel: [343112.103462] Code: 66 90 65 48 8b 04 25 f0 c6 00 00 83 80 44 e0 ff ff 01 b8 00 01 00 00 f0 66 0f c1 07 0f b6 d4 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 <0f> b6 07 38 d0 75 f7 5d c3 0f 1f 44 00 00 55 48 89 e5 66 66 66
Jul 27 02:06:58 localhost kernel: [343112.103488] Call Trace:
Jul 27 02:06:58 localhost kernel: [343112.103491] [<ffffffff811ad65d>] __fsnotify_update_child_dentry_flags+0xed/0x140
Jul 27 02:06:58 localhost kernel: [343112.103493] [<ffffffff811aecf6>] fsnotify_add_mark+0x1a6/0x220
Jul 27 02:06:58 localhost kernel: [343112.103496] [<ffffffff811b0905>] sys_inotify_add_watch+0x315/0x3b0
Jul 27 02:06:58 localhost kernel: [343112.103499] [<ffffffff8146ab29>] system_call_fastpath+0x16/0x1b

Since the lockups started near kernel upgrade to 3.4.6, I've downgraded some packages to see if this helps. Downgrading linux (3.4.6 to 3.4.5), nvidia (302.17 to 295.59) and fuse (2.9.0 -> 2.8.7) didn't helped. I have upgraded kernel to 3.4.7 today, so I don't know yet if it solves anything.

Memtest86 passed normal test (x2) and bit-fade test. mprime running for almost 11 hours caused no problems.

Just few moments ago I have realized that during the last lockup I was unable to switch from xorg to terminal. I'll try to downgrade xorg later.

Hardware:
Mobo: Gigabyte EP43-DS3LR
CPU: Intel E5300
RAM: 2x PV224G6400LLK (voltage is ok)
GPU: nVidia 9500gt

PSU voltages are fine

mpan · 2013-11-15 19:47:49

I'll probably win a Golden Shovel of Gravedigger, but maybe this will help someone else with similar problems.
It seems that the problem was in RAM voltages. After few months of lockups and kernel panics, I've increased the voltages by 100mV and for another few months everything works fine. Either this was the reason (despite tests showed nothing) or some strange correlation has happened.

r08 · 2013-11-21 05:33:33

(Edited out)

How do you increase RAM voltages?

Last edited by r08 (2016-02-21 04:20:20)

mpan · 2013-11-21 09:31:16

There is no need to quote a message that is just above the reply. It's obvious to who you're replying.

My motherboard exposes this setting in BIOS setup. Many others do too. For Gigabyte mobos search in "MB Intelligent Tweaker" section. For others search in something related to "advanced", "performance" or similar.

Be warned, however, that this is not a magic solution for lockups! Keep in mind that increasing voltage will also increase power dissipated in the RAM, and the relation of power to voltage is quadratic, not linear. Higher power means that RAM may get damaged due to excessive heating, if it's not cooled well enough. Also you may actually get decrease in system stability if your motherboard is not able to supply higher power or bit-flips will happen in overheated RAM.

In my case RAM voltages were one of the suspects from the begining, as the memory sticks require higher than normal voltages. I've just considered software problem simply because: 1) the problems started suddenly, as if it was an update issue; 2) bit-fade test and mprime stressing went fine. I've came back to the idea that voltages are the cause after another problem emerged: suddenly programs were crashing and restarting them was imposible until they were either reinstalled or caches were not cleaned manually. This was a clear sign that something is wrong with RAM. However, in your case the cause may be different.

The RAM voltage regulation for ga ep43-ds3lr (on the bottom).

Last edited by mpan (2013-11-21 12:31:56)

Arch Linux

#1 2012-08-01 21:11:02

Kernel lockups for about 2 weeks

#2 2013-11-15 19:47:49

Re: Kernel lockups for about 2 weeks

#3 2013-11-21 05:33:33

Re: Kernel lockups for about 2 weeks

#4 2013-11-21 09:31:16

Re: Kernel lockups for about 2 weeks

Board footer