You are not logged in.

#1 2019-03-31 15:18:28

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

[SOLVED] Freezes / Crashes on ThinkPad X61s

Unit information:
ThinkPad X61s
4 GB RAM
250 GB SSD
L7700 Intel Processor


What happens:
At unexpected times I will experience a freeze that renders the unit unresponsive. It does not seem to be an xorg crash as all input is ignored during a freeze including trying to switch the tty. The only way to recover from a freeze is to hold the power button to force shut down. The majority of the freezes I've experienced occur when I stop interacting with the unit to write something down and return having no input register. Usually about a minute away from keyboard, although not strictly. I have experienced a freeze during continuous use while using IntelliJ Idea. I have not been able to reproduce the freeze intentionally. I have been logging the last 95% of crashes to try to find a pattern before making a forum post (see Freeze / crash log). The big gap from 3.7.19 - 3.24.19 was due to not using the unit daily during that duration.


Background:
The first freeze occured not long after a fresh install, which was not long after purchasing the X61s itself. When I rebooted after the first freeze I was brought to GRUB Rescue. My initial thought was that the SSD included in the unit failed, causing the crash. I replaced the SSD with my own, new, Samsung Evo to solve the problem. After reinstalling Arch on the new SSD I experienced a crash again. This was before the first log on 2.4.19 as it was on a different install and was before I realized I couldn't figure this out on my own. After this crash I did not enter GRUB Rescue nor have I since. Therefore, I concluded that it wasn't the SSD, but instead I must've crashed in the middle of a write (or trim?), corrupting the SSD. I have confirmed that the original SSD works as it's now in use on another system.


My thoughts:
I thought the crashes might be due to not updating frequently enough. So I have been good about staying up to date as of recently. I have been updating every day or every other day, but still crashed on 3.30.19. I thought I might find some insight by determining if it happens when it's plugged in or not. It has happened both plugged and unplugged. Thought it might be my wifi manager. I have experienced crashes using both netctl and still after switching to connman. I thought it my be do to some scheduled event. The only timer I have set is periodic trim.


Freeze / crash log:
Crashed on 2.4.19 @ 8:04
Crashed on 2.8.19 @ 10:15
Crashed on 2.15.19 @ 20:42 (plugged in; shortly after -Syu)
Crashed on 2.18.19 @ 10:13 (firefox, inbox)
Crashed on 2.25.19 @ 13:46 (IntelliJ; during use)
Crashed on 3.7.19 @ 18:04 (firefox left open)
Crashed on 3.24.19 @ 18:50 (firefox reading pdf)
Crashed on 3.30.19 @ 16:22 (plugged in; firefox)


Outputs:
Times in log (see Freeze / crash log) to be referenced are not exact, but close and get increasingly accurate chronologically. Journal outputs are since (-S) an hour before to until (-U) hour after (journalctl -S "1 hour before" -U "1 hour after").

journalctl -S -U

mkinitcpio -p

pacman -Qe


SOLUTION:
Re-seat the RAM cards.
Replace wpa_supplicant and ConnMan with iwd and dhcpcd. (#22)

Last edited by ThinkPad (2019-08-06 02:01:38)

Offline

#2 2019-03-31 22:17:43

d_fajardo
Member
Registered: 2017-07-28
Posts: 497

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

The links you've provided are non-functional 404 errors.

Just a guess but this might be hardware related - MB, memory or the CPU itself since it seems your storage is not a problem. The only test I can think of that you can readily run is Memtest that to check memory. The L7700 was launched in 2007 so age might start failing it. But as I said, I am just guessing. I would run Memtest nevertheless which would also eliminate memory as a cause if passed.

Last edited by d_fajardo (2019-03-31 22:23:26)

Offline

#3 2019-04-01 15:34:04

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Thank you for bringing this to my attention. Apparently my github account has been flagged. I have contacted support and will respond here when the links are back up and running and I have results from Memtest.

Offline

#4 2019-04-02 03:50:51

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

d_fajardo github has yet to reply. In the meantime I've run memtest. It returned 4 errors, before I stopped it at Test #7. Does this conclude that this is a ram issue?

Offline

#5 2019-04-02 08:23:21

d_fajardo
Member
Registered: 2017-07-28
Posts: 497

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

It is looking like it is. If you have more than one stick of RAM you can test each one separately. Also I believe Memtest test the CPU caches L1, L2., etc as well  Just make sure where the error comes from. If it comes from the CPU cache, then the fault would lie on the CPU. I haven't run Memtest for a while but I think you can isolate which memory to test specifically.

Maybe this will help.

Last edited by d_fajardo (2019-04-02 08:28:21)

Offline

#6 2019-04-03 14:12:44

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Will do. Thanks for getting me on track despite the links not working.

Offline

#7 2019-04-10 03:39:01

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

I experienced another crash today and ran memtest shortly after. I did not get any errors after letting it run a full pass. I will test again in a couple of days and report back.

I have contacted github multiple times about my account being flag. I will work on getting those links up.

Offline

#8 2019-04-10 08:01:51

ghus
Member
Registered: 2016-11-14
Posts: 417

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

d_fajardo wrote:

It is looking like it is. If you have more than one stick of RAM you can test each one separately. Also I believe Memtest test the CPU caches L1, L2., etc as well  Just make sure where the error comes from. If it comes from the CPU cache, then the fault would lie on the CPU. I haven't run Memtest for a while but I think you can isolate which memory to test specifically.

Maybe this will help.

Hi, sorry but that version on memtest is more specific for UEFI Bios (imho), https://www.memtest86.com/features.htm

MemTest86+ is a part of many Linux distributions, including Debian, the derived Ubuntu, and Arch Linux.[15] Ubuntu includes it as part of the default installation if the machine is booting in BIOS mode; it is available from the GRUB OS-select menu.[16] These distributions include many types of "patches" (small modifications) to keep the 2013 code base functional. For example, Debian has patches that update the Makefile for GCC version 5 and makes the program Multiboot-compliant.[17] None of these modifications, however, were enough to convert the real-mode application to UEFI use. The closest one can get to "installing" MemTest86 on a UEFI machine is via a script in the Arch Linux user repository, which unpacks the PassMark version into the EFI system partition and tells GRUB where to find it.

https://en.wikipedia.org/wiki/Memtest86

Offline

#9 2019-04-10 08:13:39

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 7,813

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Without seeing your logs, most of the suggestions here are just speculation, use another paste site if you still continue to have issues with github.

FWIW since you are using a samsung SSD and the are very notorious for issues with SATA  power management, one thing you should try is to switch that to max_performance, see this recent thread on similar symptoms and the same resolution as well: https://bbs.archlinux.org/viewtopic.php?id=245454

Last edited by V1del (2019-04-10 08:13:56)

Online

#10 2019-04-10 22:29:49

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Links are now up. Let me know if there are any problems.

Offline

#11 2019-04-10 22:40:48

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

V1del, I don't think this is the problem for two reasons: One, the system in that thread remains somewhat responsive. In my case it completely freezes. I cannot even change tty's using ctrl-alt-F2 etc. Two, this problem first became apparent when using my ADATA SSD.

Offline

#12 2019-04-11 08:22:14

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 7,813

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Links work now, but there's no clear indication towards a problem. Even if the system in the thread I linked remained responsive enough to dump out a dmesg clearly showing the issue, it can quite well be that your disk locks up and no writes go through anymore. It's also not exclusive to Samsung SSDs, they are however very notorious for this, it won't hurt to try.

How does your RAM usage look like? The utilities you've mentioned having in use are quite heavy and if you start to swap and depending on the type of memory that swapping has to wait on it can look like the system is frozen. 

Other than that from the logs, at each point you mention for having a freeze it seems like there was some sort of wifi activity involved, so it might as well be an issue with the wifi driver. One thing you can try here is disabling hardware encryption e.g. /etc/modprobe.d/ath5k.conf

options ath5k nohwcrypt=1

Also in general, in hopes of gaining more information and "safely" shutting down you could try and enable sysrq and check if you can reboot and get a more complete log using the REISUB method

Last edited by V1del (2019-04-11 08:24:12)

Online

#13 2019-04-11 14:26:51

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Thank you. I will give these all a shot and report back. Given that the crashes are infrequent and unpredictable, it may be some time before I can report anything conclusive. I have re-seated the RAM since and am waiting to see if that has any effect. I want to try one thing at a time so I can pinpoint what fixed it.

The swap usage is a very interesting thought. I have waited it out to see if it would unfreeze. The longest I waited is 5 minutes, without success. My RAM usage is not intense. I rarely have more than two applications open at once. Most of the time it's Firefox with less than 5 tabs and IntelliJ Idea. I've never seen my RAM usage exceed the 4 GB I have installed and use swap.

Again, thank you for the very useful and insightful suggestions. I will report back with information.

Offline

#14 2019-04-14 08:08:29

Ashren
Member
From: Denmark
Registered: 2007-06-13
Posts: 1,226
Website

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Could it be the same problem described here:

https://bugs.launchpad.net/ubuntu/+sour … ug/1798961

I have had similar experiences with a fresh arch install on a T580.

Last edited by Ashren (2019-04-14 08:09:07)

Offline

#15 2019-04-23 15:42:55

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Thanks, Ashren, I will look into this.

Update:
I haven't had a freeze in 14 days of consistent use, after simply re-seating the RAM. That's a PR! It is possible that I swapped RAM card slots as I didn't take note of the initial coupling. Although I don't think the swapping should have an effect unless the RAM is indeed faulty, I think it's important to spare no detail when troubleshooting.

I am not ready to mark this as the solution. If I go a full month I'll reconsider.

Offline

#16 2019-05-01 20:51:02

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Thank you to all who helped. I am willing to mark this as solved as it has been a month now with no crashes whatsoever. It seems the re-seating of the RAM is all it needed. Maybe the cards were slightly dislodged during transit.

Last edited by ThinkPad (2019-05-03 01:01:17)

Offline

#17 2019-05-03 01:06:16

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Hey guys, shortly after posting the update my system froze. It happened after an -Syu, using the system without rebooting. I thought that was the reason and it was a fluke, but today it froze again. The packages currently available for upgrade are

dhcpcd-7.2.1-1  glib-networking-2.60.2-1  parted-3.2-9

if that gives any context.

I will restart troubleshooting from top to bottom of this thread. If anything else comes to mind feel free to add it to my list.

Offline

#18 2019-06-08 16:09:01

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

You're on the money V1del. It has crashed since, albeit less frequently, but both times it involved wifi interaction usually through wpa_supplicant. I do not want to try the software encryption yet because it is posted as a solution to problems I don't experience (web pages randomly loading very slow, or if the device is unable to lease an IP address), and I am wondering if my problem is related to these message outputs:

dmesg | grep ath5k
----
[    8.500348] ath5k 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control
[    8.500633] ath5k 0000:03:00.0: registered as 'phy0'
[    9.150930] ath5k: phy0: Atheros AR5414 chip found (MAC: 0xa3, PHY: 0x61)

Two things to note in this output: The OS not having ASPM control and Atheros AR5414 chip found.

How should I give OS ASPM control and is it necessary? I did not see an option for disabling this in my BIOS. Checking journalctl [1]:

sudo journalctl -b | grep ASPM 
----
Jun 08 08:06:57 mcp kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jun 08 08:06:57 mcp kernel: acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
Jun 08 08:06:57 mcp kernel: acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
Jun 08 08:07:00 mcp kernel: ath5k 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control

Does this mean the kernel tried to disable ASPM, but failed? Should I follow link [1] to force disable it?

The chip found should be AR5212:

lspci -k | grep Atheros
----
03:00.0 Ethernet controller: Qualcomm Atheros AR5212 802.11abg NIC (rev 01

If these problems are not the source, I will put your software encryption suggestion into play.

Also, worth noting: your suggestion made me think of a fix on my T60 running Debian where wifi problems were traced back to the driver. I thought to ensure I had the firmware properly installed and uploaded on boot, but it turns out that
[2]:

ath5k is a Linux kernel driver supporting Atheros 802.11a/bg PCI/PCI-E chips, introduced at Linux 2.6.25. Supported devices are listed at the end of this page.

Unlike the MadWifi driver, ath5k does not require a binary HAL (hardware abstraction layer). No firmware is required to be loaded from userspace.

Does this explain why the 'dmesg | grep ath5k' output above doesn't include something like "loaded firmware..." [3]?

Offline

#19 2019-06-19 13:17:37

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 7,813

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Apart from potentially increased power draw, you shouldn't have any ill-effects from disabling aspm so I'd say try it out, it certainly is a possible avenue.

You might also want to give e.g. the LTS kernel a shot (though I don't think that this driver sees many really active changes anymore, with the age of the supported chips and all).

There have also been some potential quirks with wpa_supplicant recently, one thing you could try here is using iwd instead.

Online

#20 2019-06-19 14:09:08

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

I agree about the driver. After reading the wiki, I will definitely make the switch to iwd, but first I will test disabling aspm alone.

After adding the kernel parameter 'pcie_aspm=off' and rebooting, journalctl now reads:

sudo journalctl -b | grep ASPM
----
Jun 19 06:45:24 mcp kernel: PCIe ASPM is disabled
Jun 19 06:45:24 mcp kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Jun 19 06:45:24 mcp kernel: acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]

I'm assuming the last line means the OS is no longer requesting control to disable ASPM, because the OS requires ASPM in order to do this and it has been disabled.

I still haven't found a way to reproduce the freeze so I will have to just continue to use the machine and wait. Sometimes this could be a week, lately it could be a month. Thank you very much for you continued patience and support.

Offline

#21 2019-06-19 21:55:38

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

Very cool, I crashed about 20 minutes ago so that shortens the wait.

Reverted the ASPM settings to default. Replaced wpa_supplicant with iwd. Will report back.

Offline

#22 2019-08-06 01:56:51

ThinkPad
Member
Registered: 2019-01-29
Posts: 31

Re: [SOLVED] Freezes / Crashes on ThinkPad X61s

It's been over a month, no crashes. I replaced wpa_supplicant and ConnMan with iwd and dhcpcd (using iwd as standalone). Marking as solved. Thank you to all who helped.

Last edited by ThinkPad (2019-08-06 02:01:52)

Offline

Board footer

Powered by FluxBB