Hard lockup, no logs, how to debug?

Wilco · 2012-03-09 19:57:51

So I'm having some hard crashes lately. The only way to resolve is to hold the power button to shut down the machine. Nothing gets written to /var/log and the screen freezes. So, is it possible to find out what happens? Crashes don't happen on other OSes on the same machine so I don't think it's a hardware problem.

SimpleKiwi · 2012-03-09 21:28:43

I am having the same issue with an ArchLinux test server of mine. Seems that after 15 minutes or so the machine will lockup and a hard reset is required

archman-cro · 2012-03-09 21:41:20

I am experiencing the same thing with an HP Compaq 6560b laptop. I suspect it might be a problem with the integrated Inter GPU, since I'm also getting some strange screen artifacts at random occasions..

Last edited by archman-cro (2012-03-09 21:42:26)

SimpleKiwi · 2012-03-09 22:08:44

Also when I plugged my keyboard back into the server, the caps/num/scroll lock lights were flashing similar to kernel panic

Lone_Wolf · 2012-03-10 12:33:12

Wilco :

try to narrow down in which circumstances it crashes.

for a start , test if it crashes in the following situations :

-1. console environment only, no one logged in
-2. X started, no user activity

SimpleKiwi · 2012-03-11 01:09:43

For me it crashes in a console environment only and no-one is logged in

Wilco · 2012-03-11 13:34:39

For me the crashes have disappeard since going from xf86-video-ati to the binary catalyst driver (the newest version). Strange, because the open source driver was always better in the past.

eduedix · 2012-03-12 11:41:12

i am having hard lockup when i open camera ( for example throught googltalk or skype), can provide any further info if requested.

Lone_Wolf · 2012-03-12 14:16:59

Simplekiwi, eduedix you probably have different problems then Wilco had, it's better to start your own threads.

mr.MikyMaus · 2012-03-16 07:59:32

I'd recommend checking your RAM by memtest - grab an Arch install iso,boot it off and choose memtest+ in Grub's boot menu. If nothing, it will at least eliminate one of the possibilities.

I also remember having similar almost-random lockups which in the end turned up to be caused by shitty Broadcom driver and fired off by Wicd clients (either of gtk and curses) trying to power up the wifi card. It was really hard to debug as there were no logs pointing in that direction.. I don't remember how I solved it though (an upgrade perhaps?). It was on an HP 6515 (I think) laptop with Ubuntu.

-m.

Wilco · 2012-03-16 11:11:00

Coincidence, I have the same Broadcom card and I use the WL driver because the others don't work.

Wilco · 2012-03-16 20:26:40

Still hard lockups, temperature was 67C at the last crash so that should be ok. But this really sucks, I'm not used to hard crashes like this, especially on Linux

edit: memtest is ok btw

edit 2: now testing a couple days without the broadcom-wl driver

Last edited by Wilco (2012-03-16 20:39:05)

subatomic · 2012-03-18 11:12:06

I've been experiencing hard lockups since late last year. After loosing a significant amount of data I reinstalled Arch64, KDE and upgraded all packages but the freezing is still there. The thing is, Ubuntu runs very stable on the same machine and hardware.

I ran a memtest without errors and at the time of the last crash there was no unusual behavior, high temps or ram usage. And obviously no logs. I just went with the linux-lts kernel because Ubuntu runs a 3.0 kernel, but it froze up again despite the change. It freezes randomly without activities like if I leave it on overnight it will crash at 3AM or during the day when I am away it will crash at 11AM or sometimes 5PM.

It used to freeze while playing audio or video but this seems to be solved. The last time it locked up while I was actually sitting here was when I opened Chromium and although I rarely use it the system has crash maybe 3 times while opening Chromium. At least twice it locked up while loading archlinux.org in Opera and occasionally just clicking the mouse.

One of the threads I've been folowing is here. I falsely narrowed it down to the kernel because many users have Gnome and/or ATI cards whereas I have KDE and nvidia, although I may try the nouveau driver just to be able to eliminate that. Another alternative is to install the 32-bit Arch unless someone can report lockups with it.

Lone_Wolf · 2012-03-18 12:45:15

Another option to help with troubleshooting (or have to chance to do a clean reboot) :

enable Magic SysRq key.

archwiki on sysctl
wikipedia onMagic_SysRq_key

Last edited by Lone_Wolf (2012-03-18 12:45:53)

NerdWorld · 2012-03-21 00:21:57

Having the same problem. Running X86_64. Locks at command prompt, or when X starts. Can't see anything in the logs.

It started after I updated last night (some obscure library updated.. not sure which one) Memtest has been running for an hour without any problem.

<sigh>

Ephexeve · 2012-03-21 12:23:41

Having the same problem here , any idea?!

Wilco · 2012-03-22 08:31:40

There are many threads about this issue. I'm currently testing without vboxdrv and other virtualbox modules to see if that works. Can everybody post their specs here to see if we have something in common?

Arch Linux x64 up-to-date
AMD Fusion E450
Integrated HD6320
Icewm
Xorg 1.11 (because catalyst does not support 1.12 yet)
AMD Catalyst binary drivers
Broadcom-wl

lsmod coming later today...

NerdWorld · 2012-03-23 09:38:21

It turns out my problem was caused by a failed fan on my video card. Memtest would run all night because it didn't heat up the nVidia graphics chip, but the system would lock up hard in normal GUI mode for Linux because the video chip would overheat.

I realized I had a hardware issue when I booted to Debian stable and had lock-ups that immediately took down the display, but still allowed me to ssh into the machine. Once I started looking for hardware issues, I discovered the fan.

megabrain · 2012-03-27 08:15:43

in my case lock up happen when i'm on battery only (64,intel video,testing). all after kernel 3.2. no logs, no clues whatsoever. impossible to work.
sometimes it can last longer then hour but dead anyway before discharge.

cfr · 2012-03-28 00:54:36

Try the lts kernel... It is quite a relief here. As a bonus, the graphics corruption I was experiencing is gone, too.

.:B:. · 2012-04-04 19:27:39

For the record, I have hard lockups (in the GUI) here with the LTS kernel. Might be an issue with my xbmc build though, but this is the first time it's happening, and it always happens within 5 minutes after boot.

Last edited by .:B:. (2012-04-04 19:30:51)

subatomic · 2012-04-04 21:19:25

I've just noticed this: very often the system will freeze at 1 minute after the hour, but not always and I left this for coincidence until it happened again the other day and, believe it or not, just now at 11:01
Usually there would be an Xorg.0.log.old written soon before, maybe 20 minutes, but not at the time of the freeze which I found very strange. Now the log is from 11:01 and the last lines are the same as in the others

[  4925.422] (II) config/udev: Adding input device Streamzap PC Remote Infrared Receiver (0e9c:0000) (/dev/input/event6)
[  4925.422] (**) Streamzap PC Remote Infrared Receiver (0e9c:0000): Ignoring device from InputClass "Ignore Streamzap IR"

This is from an xorg conf file to stop the remote from becoming a keyboard.

EDIT

Alright I think I'm on to it - last night it froze at 3:34. Just checked my crond.log

Apr  5 03:01:01 localhost /USR/SBIN/CROND[4031]: (root) CMD (run-parts /etc/cron.hourly)
Apr  5 03:01:01 localhost anacron[4037]: Anacron started on 2012-04-05
Apr  5 03:01:01 localhost anacron[4037]: Will run job `cron.daily' in 33 min.
Apr  5 03:01:01 localhost anacron[4037]: Will run job `cron.monthly' in 73 min.

I have to admit I have no idea what cron jobs are run.

Last edited by subatomic (2012-04-05 06:07:18)

cfr · 2012-04-05 01:46:11

https://bugs.archlinux.org/task/29276

I've had one lockup with the lts kernel. Lockups with the mainline kernel are fast and almost predictable.

Mine also tends to happen shortly after the hour. A CROND job complaint about sendmail not existing is often one of the last errors. My other theory was something to do with usb devices or (prime suspect) intel video drivers and the integrated gpu...

cfr · 2012-04-09 22:34:14

I've posted some logs I obtained using the sysreq key combo on the bug I filed. If anybody is interested, it would be great to know if they mean anything to somebody. I can tell I got something but I have no idea if it might be useful or informative. I wasn't sure it was working at all as I got no response from the key combo at all - and it did not, unfortunately let me shutdown cleanly or else I did that bit wrong - but I did find an unusual amount of stuff waiting for me in kernel.log and messages.log. I'm really hoping this will give somebody enough information to at least suggest what I might do/try/examine next as it really isn't good that this keeps happening. Every time I get file system corruption - I'm glad I'm using a journalling fs, but still...

cfr · 2012-04-09 22:41:14

Lone_Wolf wrote:

Another option to help with troubleshooting (or have to chance to do a clean reboot) :
enable Magic SysRq key.

Beyond what's in the wiki, is there any trick involved in (1) getting some sort of feedback when you use the key? (I had the impression from the wiki that I would see text messages or something.) (2) rebooting cleanly? (I tried pressing r <pause> e <pause> i <pause> s <pause> u <pause> b <pause> but still had to do a hard shutdown in the end.)

I am pretty sure the key combo is enabled from looking at the logs I posted. I found bits which started:

Apr  9 21:24:18 localhost kernel: [35077.533535] SysRq : Show backtrace of all active CPUs

I assume that means that I successfully enabled and triggered the relevant key combination? I don't usually see that in the logs, at any rate, so I assumed it must be a result of the sysrq thing?

Arch Linux

#1 2012-03-09 19:57:51

Hard lockup, no logs, how to debug?

#2 2012-03-09 21:28:43

Re: Hard lockup, no logs, how to debug?

#3 2012-03-09 21:41:20

Re: Hard lockup, no logs, how to debug?

#4 2012-03-09 22:08:44

Re: Hard lockup, no logs, how to debug?

#5 2012-03-10 12:33:12

Re: Hard lockup, no logs, how to debug?

#6 2012-03-11 01:09:43

Re: Hard lockup, no logs, how to debug?

#7 2012-03-11 13:34:39

Re: Hard lockup, no logs, how to debug?

#8 2012-03-12 11:41:12

Re: Hard lockup, no logs, how to debug?

#9 2012-03-12 14:16:59

Re: Hard lockup, no logs, how to debug?

#10 2012-03-16 07:59:32

Re: Hard lockup, no logs, how to debug?

#11 2012-03-16 11:11:00

Re: Hard lockup, no logs, how to debug?

#12 2012-03-16 20:26:40

Re: Hard lockup, no logs, how to debug?

#13 2012-03-18 11:12:06

Re: Hard lockup, no logs, how to debug?

#14 2012-03-18 12:45:15

Re: Hard lockup, no logs, how to debug?

#15 2012-03-21 00:21:57

Re: Hard lockup, no logs, how to debug?

#16 2012-03-21 12:23:41

Re: Hard lockup, no logs, how to debug?

#17 2012-03-22 08:31:40

Re: Hard lockup, no logs, how to debug?

#18 2012-03-23 09:38:21

Re: Hard lockup, no logs, how to debug?

#19 2012-03-27 08:15:43

Re: Hard lockup, no logs, how to debug?

#20 2012-03-28 00:54:36

Re: Hard lockup, no logs, how to debug?

#21 2012-04-04 19:27:39

Re: Hard lockup, no logs, how to debug?

#22 2012-04-04 21:19:25

Re: Hard lockup, no logs, how to debug?

#23 2012-04-05 01:46:11

Re: Hard lockup, no logs, how to debug?

#24 2012-04-09 22:34:14

Re: Hard lockup, no logs, how to debug?

#25 2012-04-09 22:41:14

Re: Hard lockup, no logs, how to debug?

Board footer