You are not logged in.

#1 2019-10-20 11:34:10

CocaineJohnsson
Member
Registered: 2019-10-20
Posts: 8

system frozen, grub missing from gpt boot manager on reboot

Since system administration appears locked and this isn't specifically a server issue or issue with a graphical environment per se, so I guess this would appear to be the most relevant section.



I was using my system, left it on and went to bed -- when I got back to it I noticed that the system was frozen at 2:38 in the morning, htop showed 4 cores (ryzen 7 2700, 8C/16T) out of 16 at 100%, the remaining all below 5% with 5.4GiB available memory. Exhausting system resources thus seems unlikely, thermal also seems unlikely given that this system has never exhibited thermal issues in the past and booting it into windows to check sensors via hwinfo64 (if only because I wanted to eliminate a thermal issue ASAP, this ranked higher than fixing grub) revealed expected thermals (below 40C @ 100% load 16T, 5 minute prime95 burn in), memory thermals are unknown but should be within spec as this system has had month+ uptimes without issue in the past

usb peripheral devices were unresponsive (i.e quitting out of xorg to a tty was impossible), nor was the system responding when attempting to ssh into it (or, indeed just ping it -- as far as any other machine on the network cared it was offline). Therefore I concluded that the system was hard frozen (presumably crashed) and hard-rebooted, upon rebooting grub was missing from the boot manager (this was quickly resolved by hopping on a live environment and reinstalling grub from an arch-chroot, minor annoyance).

xorg.log does not seem to contain anything of note, nor does lightdm's log files and anything else in /var/log appears irrelevant at a cursory glance.


obligatory journalctl output

Oct 20 02:16:07 Valhalla deepin-screenshot[419927]: "dtkwidget" can not find qm files
Oct 20 02:16:07 Valhalla deepin-screenshot[419927]: "deepin-screenshot" can not find qm files
Oct 20 02:16:12 Valhalla deepin-screenshot[419937]: "dtkwidget" can not find qm files
Oct 20 02:16:12 Valhalla deepin-screenshot[419937]: "deepin-screenshot" can not find qm files
Oct 20 02:16:17 Valhalla dbus-daemon[571]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.14929' (uid=1000 pid=368489 comm="/opt/discord/Discord ")
Oct 20 02:16:17 Valhalla systemd[1]: Starting Hostname Service...
Oct 20 02:16:17 Valhalla dbus-daemon[571]: [system] Successfully activated service 'org.freedesktop.hostname1'
Oct 20 02:16:17 Valhalla systemd[1]: Started Hostname Service.
Oct 20 02:16:17 Valhalla audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:16:17 Valhalla kernel: audit: type=1130 audit(1571530577.954:752): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:16:48 Valhalla systemd[1]: systemd-hostnamed.service: Succeeded.
Oct 20 02:16:48 Valhalla audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:16:48 Valhalla kernel: audit: type=1131 audit(1571530608.001:753): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:18:46 Valhalla krita[369475]: ""

********************
*I went to bed here*
********************

Oct 20 02:25:54 Valhalla dbus-daemon[571]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.14973' (uid=1000 pid=368489 comm="/opt/discord/Discord ")
Oct 20 02:25:54 Valhalla systemd[1]: Starting Hostname Service...
Oct 20 02:25:55 Valhalla dbus-daemon[571]: [system] Successfully activated service 'org.freedesktop.hostname1'
Oct 20 02:25:55 Valhalla systemd[1]: Started Hostname Service.
Oct 20 02:25:55 Valhalla audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:25:55 Valhalla kernel: audit: type=1130 audit(1571531155.084:754): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:26:52 Valhalla systemd[1]: systemd-hostnamed.service: Succeeded.
Oct 20 02:26:52 Valhalla audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:26:52 Valhalla kernel: audit: type=1131 audit(1571531212.905:755): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 20 02:36:22 Valhalla krita[369475]: ""
-- Reboot --
Oct 20 12:35:51 Valhalla kernel: Linux version 5.3.5-arch1-1-ARCH (builduser@heftig-1706204) (gcc version 9.2.0 (GCC)) #1 SMP PREEMPT Mon Oct 7 19:03:08 UTC 2019

Obligatory relevant specs:

CPU: Ryzen 7 2700
RAM: 16GB DDR4-2666, stock clock
GPU: Nvidia GeForce GTX 1070
Motherboard: MSI x470 GAMING PRO CARBON
Drive: Samsung Evo 860 SSD

Kernel: 5.3.5-arch1-1-ARCH
Nvidia driver version: 435.21

and if you care about cpu microcode (shouldn't matter unless there's a known bug listing -- i.e you almost certainly don't care about it, but here it is nonetheless.

$ journalctl --no-hostname -o short-monotonic --boot -0 | sed -n '1,/PM: Preparing system for sleep/p' | grep 'microcode\|smp'
[    0.000000] kernel: smpboot: Allowing 32 CPUs, 16 hotplug CPUs
[    0.115554] kernel: smpboot: CPU0: AMD Ryzen 7 2700 Eight-Core Processor (family: 0x17, model: 0x8, stepping: 0x2)
[    0.170016] kernel: smp: Bringing up secondary CPUs ...
[    0.772183] kernel: smp: Brought up 1 node, 16 CPUs
[    0.772183] kernel: smpboot: Max logical packages: 2
[    0.772183] kernel: smpboot: Total of 16 processors activated (102426.76 BogoMIPS)
[    1.021234] kernel: microcode: CPU0: patch_level=0x0800820b
[    1.021238] kernel: microcode: CPU1: patch_level=0x0800820b
[    1.021244] kernel: microcode: CPU2: patch_level=0x0800820b
[    1.021248] kernel: microcode: CPU3: patch_level=0x0800820b
[    1.021255] kernel: microcode: CPU4: patch_level=0x0800820b
[    1.021263] kernel: microcode: CPU5: patch_level=0x0800820b
[    1.021269] kernel: microcode: CPU6: patch_level=0x0800820b
[    1.021276] kernel: microcode: CPU7: patch_level=0x0800820b
[    1.021281] kernel: microcode: CPU8: patch_level=0x0800820b
[    1.021286] kernel: microcode: CPU9: patch_level=0x0800820b
[    1.021291] kernel: microcode: CPU10: patch_level=0x0800820b
[    1.021295] kernel: microcode: CPU11: patch_level=0x0800820b
[    1.021301] kernel: microcode: CPU12: patch_level=0x0800820b
[    1.021307] kernel: microcode: CPU13: patch_level=0x0800820b
[    1.021312] kernel: microcode: CPU14: patch_level=0x0800820b
[    1.021318] kernel: microcode: CPU15: patch_level=0x0800820b
[    1.021341] kernel: microcode: Microcode Update Driver: v2.2.

I reviewed the immediate 24 hours prior to the crash, there's nothing particularly interesting (attached are the events immediately preceding, though they're uneventful), the system stopped responding at around 2:40 in the morning according to journalctl and the frozen clock under xorg.

The drive's healthy according to smartmontools (as expected, it's a more or less brand new ssd)


I have, at this point, exhausted my initial ideas for debugging this.



So the question is thus twofold.

1) how should I proceed with trying to figure out the cause of the crash?
2) why would grub be de-registered from the GPT boot manager? It is very strange to me that the grub entry would disappear following a system crash, even though all relevant files still appeared to be on the EFI boot partition, especially since reinstalling fixed the issue.

Offline

Board footer

Powered by FluxBB