Arch Linux

jakub

Member

Registered: 2013-09-14

Posts: 151

How to analyze why system crashed?

My fresh archlinux installation just crashed today making me worry about the hardware. How could I check what caused the crashed? (It crashed just after maven started compiling code - it was "mvn clean install" command executed from IntelliJ IDE).

➜  ~ journalctl --list-boots
...
 -1 db1d8a1cb19549818fa5005907982919 Fri 2020-01-24 19:17:27 CET—Sun 2020-01-26 19:37:31 CET
  0 2ec244bc2b4e42138e527db8e82fd931 Sun 2020-01-26 19:38:30 CET—Sun 2020-01-26 20:01:01 CET

So my last boot crashed around 19:37.

Journalctl is not telling much for me:

➜  ~ sudo journalctl --since="2020-01-26 19:32:00" 
-- Logs begin at Sat 2020-01-18 19:50:04 CET, end at Sun 2020-01-26 20:10:42 CET. --
sty 26 19:33:34 mayday NetworkManager[531]: <info>  [1580063612.8751] manager: NetworkManager state is now CONNECTED_SITE
sty 26 19:34:21 mayday dolphin[104602]: kf5.kio.core: couldn't create slave: "Nie można nawiązać łączności z klauncher: Did not receive a reply. Possible causes inc>
sty 26 19:34:48 mayday dbus-daemon[529]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.serv>
sty 26 19:34:53 mayday dhcpcd[678]: enp10s0: pid 531 deleted default route via 192.168.1.1
sty 26 19:34:59 mayday systemd[1]: Starting Cleanup of Temporary Directories...
sty 26 19:35:14 mayday dbus-daemon[529]: [system] Failed to activate service 'org.freedesktop.nm_dispatcher': timed out (service_start_timeout=25000ms)
sty 26 19:35:48 mayday plasmashell[1318]: plasma-nm: Wireless scan on "wlp6s0" failed: "Did not receive a reply. Possible causes include: the remote application did>
sty 26 19:35:52 mayday systemd[1]: Starting Network Manager Script Dispatcher Service...
sty 26 19:36:21 mayday dolphin[104602]: kf5.kdbusaddons: 'kdeinit5' is taking more than 30 seconds to start.
sty 26 19:36:59 mayday dolphin[104602]: kf5.kdbusaddons: 'kdeinit5' is taking more than 30 seconds to start.
sty 26 19:37:01 mayday dolphin[104602]: kf5.kio.core: couldn't create slave: "Nie można nawiązać łączności z klauncher: The name org.kde.klauncher5 was not provided>
sty 26 19:37:07 mayday systemd[1]: NetworkManager-dispatcher.service: start operation timed out. Terminating.
sty 26 19:37:15 mayday systemd[1]: NetworkManager-dispatcher.service: Failed with result 'timeout'.
sty 26 19:37:18 mayday systemd[1]: Failed to start Network Manager Script Dispatcher Service.
sty 26 19:37:26 mayday kernel: audit: type=1130 audit(1580063846.045:689): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="syst>
sty 26 19:37:26 mayday audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/s>
sty 26 19:37:31 mayday plasmashell[1318]: plasma-nm: Wireless scan on "wlp6s0" failed: "Did not receive a reply. Possible causes include: the remote application did>
-- Reboot --
sty 26 19:38:17 mayday kernel: microcode: microcode updated early to revision 0xd6, date = 2019-10-03
...

➜  ~ sudo smartctl -a /dev/sda                                          
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.14-arch1-1] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 850 PRO 256GB
Serial Number:    S1SUNSAG124437J
LU WWN Device Id: 5 002538 8a08fa456
Firmware Version: EXM02B6Q
User Capacity:    256060514304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Jan 26 20:12:21 2020 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x53) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 136) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always       -       13108
 12 Power_Cycle_Count       0x0032   097   097   000    Old_age   Always       -       2374
177 Wear_Leveling_Count     0x0013   095   095   000    Pre-fail  Always       -       275
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   068   044   000    Old_age   Always       -       32
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       134
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       22672604972

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     13108         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Is it possible that I ran out of disk space and system wasn't able to persist anything more in journal? It would be strange, because I have around 2,4GB free disk space on root partition (and I never experienced system crash when I was running literally even 0MB free).

PS. last thing I saw on screen was mvn output in terminal:

[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ *** ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 465 source files to /mnt/storage/proj/***/target/classes

So PC crashed under some moderate stress.

Last edited by jakub (2020-01-26 19:18:18)

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 15,098

Re: How to analyze why system crashed?

[INFO] Compiling 465 source files to /mnt/storage/proj/***/target/classes

That path suggests you're using some kind of external storage ?

please post journals from the boot where things crashed as well as one from a good boot.
use   SYSTEMD_LESS=FRXMK journalctl -b some-negative-number

also the output of mount and your /etc/fstab

Last edited by Lone_Wolf (2020-01-27 14:51:37)

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

jakub

Member

Registered: 2013-09-14

Posts: 151

Re: How to analyze why system crashed?

[INFO] Compiling 465 source files to /mnt/storage/proj/***/target/classes

That path suggests you're using some kind of external storage ?

please post journals from the boot where things crashed as well as one from a good boot.
use   SYSTEMD_LESS=FRXMK journalctl -b some-negative-number

also the output of mount and your /etc/fstab

It's just a partition on same drive:

➜  ~ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 238,5G  0 disk 
├─sda1   8:1    0   200M  0 part 
├─sda2   8:2    0   500M  0 part 
├─sda3   8:3    0    76G  0 part /
└─sda5   8:5    0 161,8G  0 part /mnt/storage
sdb      8:16   1   7,5G  0 disk 
├─sdb1   8:17   1   635M  0 part 
└─sdb2   8:18   1    64M  0 part 

➜  ~ cat /etc/fstab 
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sda3
UUID=44b3b445-0ccc-4c4a-a664-981ad929cfe2       /               ext4            rw,relatime     0 1
UUID=1fe06d1e-41d0-499b-91b8-a9ad5b6d6180       /mnt/storage    ext4            rw,user         0 2

List of boots looks like following, so I'm attaching boot "-21":

-21 db1d8a1cb19549818fa5005907982919 Fri 2020-01-24 19:17:14 CET—Sun 2020-01-26 19:37:31 CET
-20 2ec244bc2b4e42138e527db8e82fd931 Sun 2020-01-26 19:38:17 CET—Sun 2020-01-26 21:27:50 CET

https://gist.github.com/jmayday/31e33d7 … 850b559f8c

And here, my current boot - I'm not sure if it's a "good boot" 
https://gist.github.com/jmayday/c873e96 … d5476c0870

Last edited by jakub (2020-01-31 10:13:54)

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 15,098

Re: How to analyze why system crashed?

Several things stand out to me, but network connections look rather bad.

There may be an issue with the kernel module for your wireless, post lspci -k .

You're using NetworkManager to manage your connections and there have been reports about troubles with it's internal dhcp client on archlinux.
If you want to stick with NM, try using it with dhclient .

See https://wiki.archlinux.org/index.php/Ne … HCP_client

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

jakub

Member

Registered: 2013-09-14

Posts: 151

Re: How to analyze why system crashed?

Regarding DHCP - I've already figured out, that I misconfigured my installation by using both NM and dhcpcd service on particular interface. dhcpcd has been uninstalled.

➜  ~ lspci -k
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
        Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
        Kernel driver in use: skl_uncore
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
        DeviceName:  Onboard IGD
        Subsystem: ASUSTeK Computer Inc. HD Graphics 530
        Kernel driver in use: i915
        Kernel modules: i915
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family MEI Controller
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
        Subsystem: ASUSTeK Computer Inc. Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1b.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #17 (rev f1)
        Kernel driver in use: pcieport
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
        Kernel driver in use: pcieport
00:1c.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #3 (rev f1)
        Kernel driver in use: pcieport
00:1c.4 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)
        Kernel driver in use: pcieport
00:1c.6 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 (rev f1)
        Kernel driver in use: pcieport
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
        Kernel driver in use: pcieport
00:1f.0 ISA bridge: Intel Corporation Z170 Chipset LPC/eSPI Controller (rev 31)
        Subsystem: ASUSTeK Computer Inc. Z170 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family HD Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family SMBus
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
02:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
        Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:01.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:02.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:03.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:04.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:05.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:06.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
04:07.0 PCI bridge: ASMedia Technology Inc. Device 1187
        Kernel driver in use: pcieport
06:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4360 802.11ac Wireless Network Adapter (rev 03)
        Subsystem: ASUSTeK Computer Inc. BCM4360 802.11ac Wireless Network Adapter
        Kernel driver in use: wl
        Kernel modules: bcma, wl
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
        Subsystem: ASUSTeK Computer Inc. ASM1062 Serial ATA Controller
        Kernel driver in use: ahci
        Kernel modules: ahci
0a:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
        Subsystem: ASUSTeK Computer Inc. I211 Gigabit Network Connection
        Kernel driver in use: igb
        Kernel modules: igb
0c:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
        Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
0d:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
        Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

jakub

Member

Registered: 2013-09-14

Posts: 151

Re: How to analyze why system crashed?

System crashed again. But this time it's quite visible, that it crashed because of OOM: https://gist.github.com/jmayday/6b170b6 … d3fa9b4f61

I tried running system without swap partition, but apparently it's not an option. I'll setup swap.

Last edited by jakub (2020-02-02 10:02:33)

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 15,098

Re: How to analyze why system crashed?

Running without any swap space is a bad idea. You can use a swapfile though if you prefer.

Changing the thread title to reflect your findings may help to attract more attention.

I suggest you install some monitoring programs, like top from procps-ng and iotop from iotop .

* Why iotop ?
I used to have excessive swapping and slow response when compiling large stuff multithreaded.
After several months I figured out the problem was not lack of memory, but all those threads competing for IO time.

I now use the values mentioned at https://wiki.archlinux.org/index.php/Sy … em_freezes and no longer have that issue.

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky