Computer crashes every day - can you figure out why?

hear-me-roar · 2015-12-12 21:44:14

Hello,

I'm a beginner user of Linux and over the months I've solved all my problems using the resources on this forum and on the wiki (Many thanks for that!). But I'm having a problem that I don't even know where to begin. Any help would be appreciated. I'm dumping all information I think might be relevant, please excuse the verbosity, I just couldn't figure out what is important and what is not.

The problem I would like to solve is this: My computer crashes almost daily if I use 'heavy' programs such as Firefox or Spotify. If I live on the console then the crashes are less frequent but still there. The last crash corrupted my harddisk; lesson learned, never ignore crashes. (BTW, by crash I mean of such a caliber that REISUB doesn't even register, so I have to go "postal on the power button" as one user put it.)

I have Lenovo T400, second hand and possibly refurbished. I used `peppermint 6` before and at some point it started crashing every few hours or so. I tried `ubuntu` which crashed often even when running from a live-usb. I could not even make the installed version boot, it would crash during boot time.

Finally I started using Arch. I suspect due to the fractional use of system requirements, the frequency of crashes became tolerable (~every week or so). Until I started using Firefox (as opposed to w3m).

I'm attaching a snippet down below from the `journalctl` referring to the latest crash. There are also a dozen or so errors in the journal for each boot but they are too many to list here. If I should be looking for something in particular please let me know.

## Last bits before a crash (I may have been afk)

I checked a couple of crashes, they are not all similar but the following has two of the most common themes: the perf interrupts taking too long and the last two lines. Although, some crashes have none or only one of these features.

Dec 05 18:26:14 localhost kernel: perf interrupt took too long (2516 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
Dec 05 18:30:25 localhost systemd[1]: Starting Cleanup of Temporary Directories...
Dec 05 18:30:25 localhost systemd[1]: Started Cleanup of Temporary Directories.
Dec 05 19:20:41 localhost kernel: perf interrupt took too long (5011 > 4960), lowering kernel.perf_event_max_sample_rate to 25200
Dec 05 20:38:31 localhost kernel: Monitor-Mwait will be used to enter C-3 state
Dec 05 20:38:31 localhost kernel: thinkpad_acpi: EC reports that Thermal Table has changed

## One other clue for the really curious

Finally, I recently encountered a problem that may be relevant. I religiously followed the wiki guidelines to install `uvesafb` and `v86d`. When I rebooted, I got a black screen. Everything was working, behind the black screen, I could type `echo "\a"` to make the computer beep etc. So I removed the hooks and recompiled the kernel. No black screen but also no frame buffer. So that issue suggests a problem with the graphics card.

## Parting words

Most likely, I'm not looking at the right places. If you ask me for relevant data I will update this post and will consider your request a significant help.

Thanks so much for reading all this!

# Edit - looking back

There has been a lot of feedback so here is a summary of important points. More importantly, some details for other people whose computers are crashing like mine.

## Suggestion 1 - overheating
I logged the temperature until the last crash. No overheating is seen. Also my laptop doesn't feel too warm at any time.

## Suggestion 2 - microcode update for intel
Turns out I already had it installed.

## Suggestion 3 - memtest
I already did 2 sessions of memtest, total of 18 hours, something like 40 iterations (2GB RAM). Though the memtest crashed at some point I could see on the frozen screen that it still had not found anything.

## Suggestion 4 - (zebra) problem with power source, faulty cable or battery
Blinking caps lock seems to rule this out (see below), I don't think that would cause kernel panic. But I never noted if crashes are more frequent when charging vs on battery. I will still keep an eye on it.

## Additional data

When the computer freezes I get blinking capslock light. So there is kernel panic for some reason.

I also dug these two things out of the journal. Do you know if I could fix these things? Would they be the cause?

error1

error2

Last edited by hear-me-roar (2015-12-15 21:51:16)

crmullins · 2015-12-13 17:07:28

I'm using a T410 and had a similar issue, but it was due to the fans not running aggressively enough. You didn't post much of the log, but any chance it's overheating?

Just from googling those log entries I'd say it's some sort of power management problem.

hear-me-roar · 2015-12-13 20:26:25

Thanks a lot for your response! It never occurred to me that it could be the core temperature because whenever I look at it (it's at my status bar) it's lower than 50 degrees Celcius (and usually below 40).

But perhaps there are crazy moments when the heat spikes. I am now trying to set up `sensord` to log temperature, in case of a crash I will see if if temperature was the reason.

Thanks for giving me something to try! I'll post here what happened.

Last edited by hear-me-roar (2015-12-13 20:26:54)

ewaller · 2015-12-14 03:07:37

Is this is an Intel processor? Have you installed and configured the microcode updates?

hear-me-roar · 2015-12-14 09:59:06

ewaller wrote:

Is this is an Intel processor? Have you installed and configured the microcode updates?

Thanks ewaller! I think that's it!

In my newbie(r) days I thought I installed it but apparently did not put the hooks up. So my intel processor is most likely suffering from that.

I will write here in 5 days if no crashes occur and make this topic solved. In anycase, this should improve things. I'm grateful for your help!

R00KIE · 2015-12-14 14:21:23

It wouldn't hurt to run memtest too. From your description it seems something went bad at some point

hear-me-roar wrote:

I used `peppermint 6` before and at some point it started crashing every few hours or so. I tried `ubuntu` which crashed often even when running from a live-usb.

You say that with light usage it takes longer to crash, so I would suspect a ram problem.

graysky · 2015-12-14 14:23:51

Rule out memory using memtest86.

Silkworm205 · 2015-12-14 15:14:19

Adding on to the advice to use memtest, you should let it do a couple of cycles of scanning. I've had two systems where errors only occur after long usage.

katsuki · 2015-12-14 18:38:51

Everyone has given you great advice; crashes under load usually mean CPU issue (generally heat) or RAM (not enough and not enough swap or bad RAM).

Both are pretty easy to diagnose. I know you are looking at logging temperatures, but really, with a laptop, you can use the hand test. When you are under high load (such as funning firefox as you say), does the machine feel unusually warm to the touch. If so, do you hear the fans spooling up (you can't miss it, a laptop under durress is not a quiet thing; even mac books scream when they get hot and they are among the quietest laptops I've ever used)?

A bit of research into the T400 does yield some information about overheating. Some suggest raising the back a bit (to allow airflow to the fans). Others recommend custom fan control (so you can change the thresholds yourself).

Good luck!

hear-me-roar · 2015-12-14 19:23:37

Hey everyone! Thanks for all the great advice. I have been trying to solve this problem by myself for months now, it means so much that you are all helping! Here are a few additional details with regards to the latest recommendations.

I did the memtest a while ago, twice. One session was 8 hours long and the other 10 hours long. In total that is roughly 40 iterations (I have only 2GB RAM). Neither of them found anything, *but* the second memtest crashed around the 10 hour mark (it still had not found anything, the screen had frozen so I could see). Do you think that means the RAM is bad or that it's another hardware failure?

The laptop certainly does not overheat while I'm around, it doesn't even hint at possibly heating too much under load. The log so far has been going pretty much flat around 40 degrees C. I'm logging just to see if there are odd moments when the heat spikes suddenly and computer crashes. If this happens it should happen so quickly that I can not feel it, rather unlikely but most of the crashes are when I'm afk so there is a remote possibility.

The microcode upgrade seems very promising though, I am betting on that to work.

Fun fact: The computer did not crash in the >24 hours or so I have started logging temperature. Which reminds me, the computer tends to crash more often if I'm afk and if these heavy programs are not doing anything but just sitting there (e.g. not playing music). But this is not a hard rule. It can also crash when just the linux terminal is open and I haven't done anything yet.

Again thanks for all the tips, I'll let you know if it survives a 5 days or so with the microcode upgrade. Otherwise I'll suplly additional data!

Dude-man · 2015-12-15 09:15:13

I wanted to chime in to add I've been having similar problems with 2 desktop computers i5 and a Q1900 motherboard cpu combo. Memtest was fine... on the Q1900 i didn't have the kernel intel patches as mentioned above... so have added that and will also keep fingers crossed.

LCvanDinteren · 2015-12-15 10:04:13

My 2 cents:
Does it make a difference if you use adapter or battery? Perhaps one of those is faulty?

ewaller · 2015-12-15 15:00:44

When you hear hoofbeats, think horses, not zebra

hear-me-roar · 2015-12-15 20:44:00

Hello everyone,

I'm trying to figure out if microcode updates had any effect. I suspect the refurbishers of my laptop might have done a BIOS update. Because when I look at `dmesg` I don't see "early updates" that I'm supposed to be seeing. Here is the situation:

microcode update result

Does this mean my cpu does not admit updates or that they are done already? If so, should I be thinking zebras?

BTW: When the computer freezes the capslock light is blinking periodically. Does that have an additional significance? It just seems to be kernel panic, which could mean just about anything.

Last edited by hear-me-roar (2015-12-15 20:55:19)

ewaller · 2015-12-15 20:58:33

What are the output of cat /proc/cmdline and of sudo bash -c "journalctl --boot | grep microcode" ?

hear-me-roar · 2015-12-15 21:08:05

/proc/cmdline reads:
BOOT_IMAGE=/vmlinuz-linux root=UUID=f64ee14f-0e25-4bee-ae3f-33a1568e2151 rw quiet

and the journal has this to say:
Dec 15 20:44:38 localhost kernel: microcode: CPU0 sig=0x10676, pf=0x80, revision=0x610
Dec 15 20:44:38 localhost kernel: microcode: CPU1 sig=0x10676, pf=0x80, revision=0x610
Dec 15 20:44:38 localhost kernel: microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

Does that mean there is an update?

ewaller · 2015-12-15 21:12:07

Yes. It would appear you have a dual core processor. The microcode update is working like it should.

Mine looks like this:

ewaller@turing ~ 1001 %journalctl --boot | grep micro
Dec 11 09:59:08 turing kernel: microcode: CPU0 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU1 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU2 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU3 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU4 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU5 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU6 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: CPU7 sig=0x306c3, pf=0x20, revision=0x1c
Dec 11 09:59:08 turing kernel: microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
ewaller@turing ~ 1002 %

ewaller · 2015-12-15 21:20:12

hear-me-roar wrote:

BTW: When the computer freezes the capslock light is blinking periodically. Does that have an additional significance? It just seems to be kernel panic, which could mean just about anything.

That is sure fire evidence of a kernel panic. When something goes so wrong that the kernel can make no sense of things it will stop rather than risk doing more damage. As there is little in the way of I/O that the kernel can trust, it indicates that it has panicked by entering a loop where all it does is harmlessly blink the keyboard lights.
https://en.wikipedia.org/wiki/Kernel_panic

hear-me-roar · 2015-12-15 21:26:29

Aha! But now that you told me what to look for, I could see that I managed to do the microcode update already when I installed arch. It has been there all along.

Nevertheless I found these two blocks of errors in the journal, does any of these look serious to you?

error1

error2

Thank you for your feedback! It feels wonderful to have a pro comment on these things!

ewaller · 2015-12-15 21:29:17

Hardly a pro, just a reasonably experienced Linux user. $DAYJOB is electrical engineering.

I think you need to go back and revisit your post. We are not seeing the details you think we are.

hear-me-roar · 2015-12-15 22:08:40

Updated my original post with a summary of all the suggestions and my responses. I added two bits of additional data: blinking caps lock and errors from the journal.

Thanks for the heads up ewaller! (Electrical engineering sounds close enough to me! Especially when combined with afternoons as an arch forum moderator. I am a pure mathematician, so the most technological thing I use is a pen. And now arch.)

R00KIE · 2015-12-16 14:07:14

Regarding the first image, the acpi exceptions can most probably be safely ignored, I have those too and everything works fine here. The warning about the mtrr settings might be worth looking into, if nothing else to know what causes it and if any problems can arise from it, but most probably it's just informative.

Regarding the second image, those acpi bios warnings might also be harmless but you should check if there are reports of something like that causing trouble.

What I'd like to point out is that your machine was working properly before and then stopped working, so I'd say some kind of hardware problem is a good probable cause. On top of that you say the ubuntu live media doesn't work and that is something that should just work, unless it is hitting some corner case in your hardware.

I suppose we should also ask if you are using any out of tree modules/drivers either for hardware or virtualization or using binary drivers for the gpu, anything that might be able to make the kernel panic.

hear-me-roar · 2015-12-16 22:42:45

I will look further into mtrr settings then! Thanks for the lead.

Indeed, hardware failure seems very likely. Why else would there be kernel panic during memtest? There is basically no out of the tree software to talk about while I'm running the memtest. Furthermore, I remember the computer crashed even in the second day of my arch installation and by then I had barely anything but the linux terminal on the machine.

Ubuntu live usb by itself works, but it is still prone to crashes. If I install Ubuntu however, it will never boot; crashing everytime I try to start Ubuntu from my computer. That last bit is the only consistent behavior of this crashing behavior.

The only thing 'out of the ordinary' that I have done since my installation of arch is that I have activated virtualization from BiOS. Otherwise, I have intel built in graphics card with its own software instead of vesa.

What I don't understand is this: There must be a connection to the fact that I can not enable frame buffer in this computer. When I try, I just get black screen and have to recompile the kernel (no easy feat!) But then why would memtest crash? It doesn't use the graphics card. Or should I be thinking of Hickam's dictum?

R00KIE · 2015-12-17 11:28:30

If you have more than one dimm of ram try swapping their places and run memtest again. I would also revert that bios setting you changed, just to be sure it's not somehow causing trouble.

One idea that I've seen dismissed before is a bad power supply, I've seen _once_ a desktop computer crash when under load without any apparent reason, after changing the power supply it worked just fine. But yours is a laptop, I have never seen, read or heard that this could happen with laptops.

hear-me-roar · 2015-12-17 11:44:53

OK, I disabled virtualization. Now I'm also using the laptop much more frequently on battery, charging only when needed. I'll see if it crashes w/o being connected to power supply. I don't seem to recall it crashing when running on battery, could be because of the unimpressive battery life simply a matter of probability.

I'll swap the RAM's, or change the position of the one, whatever the case may be.

Thanks for all your tips R00KIE! With this much tinkering the computer just has to improve

Arch Linux

#1 2015-12-12 21:44:14

Computer crashes every day - can you figure out why?

#2 2015-12-13 17:07:28

Re: Computer crashes every day - can you figure out why?

#3 2015-12-13 20:26:25

Re: Computer crashes every day - can you figure out why?

#4 2015-12-14 03:07:37

Re: Computer crashes every day - can you figure out why?

#5 2015-12-14 09:59:06

Re: Computer crashes every day - can you figure out why?

#6 2015-12-14 14:21:23

Re: Computer crashes every day - can you figure out why?

#7 2015-12-14 14:23:51

Re: Computer crashes every day - can you figure out why?

#8 2015-12-14 15:14:19

Re: Computer crashes every day - can you figure out why?

#9 2015-12-14 18:38:51

Re: Computer crashes every day - can you figure out why?

#10 2015-12-14 19:23:37

Re: Computer crashes every day - can you figure out why?

#11 2015-12-15 09:15:13

Re: Computer crashes every day - can you figure out why?

#12 2015-12-15 10:04:13

Re: Computer crashes every day - can you figure out why?

#13 2015-12-15 15:00:44

Re: Computer crashes every day - can you figure out why?

#14 2015-12-15 20:44:00

Re: Computer crashes every day - can you figure out why?

#15 2015-12-15 20:58:33

Re: Computer crashes every day - can you figure out why?

#16 2015-12-15 21:08:05

Re: Computer crashes every day - can you figure out why?

#17 2015-12-15 21:12:07

Re: Computer crashes every day - can you figure out why?

#18 2015-12-15 21:20:12

Re: Computer crashes every day - can you figure out why?

#19 2015-12-15 21:26:29

Re: Computer crashes every day - can you figure out why?

#20 2015-12-15 21:29:17

Re: Computer crashes every day - can you figure out why?

#21 2015-12-15 22:08:40

Re: Computer crashes every day - can you figure out why?

#22 2015-12-16 14:07:14

Re: Computer crashes every day - can you figure out why?

#23 2015-12-16 22:42:45

Re: Computer crashes every day - can you figure out why?

#24 2015-12-17 11:28:30

Re: Computer crashes every day - can you figure out why?

#25 2015-12-17 11:44:53

Re: Computer crashes every day - can you figure out why?

Board footer