Random black screen with unresponsive keyboard and looping audio

HotDogEnemy · 2021-08-04 20:23:29

I have this bug where Arch will sometimes display a black screen with very faint grey vertical bars, and the keyboard will become unresponsive, with the audio looping over and over. The PC LEDs stayed on during this, however I didn't check if the disk LED was flashing continuously. It doesn't seem to be related to the system going to sleep automatically, since this issue once occurred while I was attending an online meeting and doing some work. This issue seems to pop up randomly, I haven't found a consistent trigger for it. Whenever this issue pops up I have to force shutdown the PC using the power button. I checked the journalctl, but not much seems to be out of place.

I should mention that I'm using linux zen and KDE with SDDM as display manager.

Here's the ending one of the journalctl logs from 20 minutes before the issue popped up:

-- Journal begins at Fri 2020-12-18 02:00:34 IST, ends at Thu 2021-08-05 01:55:43 IST. --
Aug 03 07:55:39 archlinux kernel: Linux version 5.13.7-zen1-1-zen (linux-zen@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 ZEN SMP PREEMPT Sat, 31 Jul 2021 13:18:56 +0000
Aug 03 07:55:39 archlinux kernel: Command line: root=UUID=0994c97d-207b-4556-876f-1142de972d3d rw loglevel=3 radeon.si_support=0 radeon.cik_support=0 amdgpu.cik_support=1 amdgpu.si_support=1 module_blacklist=mei_me,mei initrd=boot\initramfs-linux-zen.img
...
...
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 11 threads of 6 processes of 1 users.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 11 threads of 6 processes of 1 users.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 11 threads of 6 processes of 1 users.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 11 threads of 6 processes of 1 users.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Successfully made thread 100127 of process 87310 owned by '1001' RT at priority 10.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:41:34 archlinux kernel: usb 1-5: reset high-speed USB device number 4 using xhci_hcd
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:41:34 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:52 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:52 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Supervising 12 threads of 7 processes of 1 users.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Successfully made thread 100860 of process 100129 owned by '1001' RT at priority 10.
Aug 03 12:43:57 archlinux rtkit-daemon[4918]: Supervising 13 threads of 8 processes of 1 users.
Aug 03 12:43:58 archlinux kernel: usb 1-5: reset high-speed USB device number 4 using xhci_hcd
Aug 03 13:00:58 archlinux kernel: usb 1-5: reset high-speed USB device number 4 using xhci_hcd
Aug 03 13:01:02 archlinux kernel: usb 1-5: reset high-speed USB device number 4 using xhci_hcd        #NOTE: the journalctl log for this boot ends here.

And the ouput of sudo jounalctl -p err to filter out error messages:

~ > sudo journalctl -p err -b-5
-- Journal begins at Fri 2020-12-18 02:00:34 IST, ends at Thu 2021-08-05 01:48:14 IST. --
Aug 03 07:55:39 archlinux kernel: [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0xb2 (or later)
Aug 03 07:55:39 archlinux kernel: x86/cpu: SGX disabled by BIOS.
Aug 03 07:55:39 archlinux kernel: kfd kfd: amdgpu: PITCAIRN  not supported in kfd
Aug 03 07:55:46 archlinux kernel: Module mei is blacklisted
Aug 03 07:56:01 archlinux ntpd[391]: bind(23) AF_INET6 fe80::aa2a:a5df:8334:243f%2#123 flags 0x11 failed: Cannot assign requested address
Aug 03 07:56:01 archlinux ntpd[391]: unable to create socket on enp2s0 (4) for fe80::aa2a:a5df:8334:243f%2#123
Aug 03 07:56:35 archlinux sddm-helper[3700]: gkr-pam: unable to locate daemon control file
Aug 03 07:57:12 archlinux pulseaudio[4885]: GetManagedObjects() failed: org.freedesktop.systemd1.NoSuchUnit: Unit dbus-org.bluez.service not found.
Aug 03 12:05:32 archlinux kernel: usb 1-5: Failed to query (GET_DEF) UVC control 2 on unit 2: -110 (exp. 2).

Any help would be appreciated.

Last edited by HotDogEnemy (2021-08-04 20:27:02)

seth · 2021-08-05 08:33:29

The symptoms describe a kernel panic/halt.
Avoid a hard reboot, https://wiki.archlinux.org/title/Keyboa … el_(SysRq) - you won't get a journal otherwise
Last resort: https://wiki.archlinux.org/title/Kdump

Another way to get to better info on the cause is to keep "dmesg -w" running in a visible terminal (and hope that the kernel prints a lot of errors before halting, but if it eg. crashes around the GPU stack, that won't help at all)

Did you try for the behavior w/ the non-zen kernel?

HotDogEnemy · 2021-08-05 12:20:23

Alright, I'll enable and try using Sysrq + REISUB next time and see what happens. I've already enabled the sysrq key and tested it, it seems to work.
As for Kdump, I'll try to set it up , but the setup looks a bit complex, so it may take some time.

Did you try for the behavior w/ the non-zen kernel?

Honestly, I'm unsure whether the zen kernel is to be blamed for this, since my boot manager, rEFInd, sorts kernels according to when they're updated, so sometimes it boots linux-lts instead of zen, but in most cases zen is the one being booted. It's also inconsistent, and I haven't found a consistent trigger for the kernel panic to occur, it's only occurred a couple times in the past week.

seth · 2021-08-05 14:13:40

I'm unsure as well.
Which is why it would be good to acquire some data on this, eg. by controlling which kernel you boot and look for a pattern ;-)

HotDogEnemy · 2021-08-05 15:24:14

Yeah I'll do that.

I just noticed something strange: I was setting up KDump, and I induced a kernel panic to check if it was working. During the panic, the keyboard LEDs were working, ie capslock and numlock were turning off and on when i pressed their respective keys. But when I encounter the bug, the keyboard would become unresponsive, and neither the capslock or numlock key would light up on press. Is this supposed to happen?

seth · 2021-08-05 15:29:06

You mean "echo c > /proc/sysrq-trigger"?
That triggers a(n artificial) kernel crash but not a real panic (that will have an underlying reason) - nothing to see here as long as you got a crash dump from it.

HotDogEnemy · 2021-08-05 16:04:44

To check if I set up KDump correcty, I tried inducing an artificial kernel panic again, this time using Alt+Sysrq+C, but the system stayed frozen and nothing happened, both the keyboard and mouse LEDs weren't responding. It seems that the kdump kernel gets stuck while loading. This is the journalctl I got from the earlier kernel panic I induced using "echo c > /proc/sysrq-trigger" before the keyboard stopped responding:

Aug 05 20:35:22 archlinux systemd[1]: Mounting /efi...
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in First Boot Wizard being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in First Boot Complete being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in Repartition Root Disk being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in First Boot Wizard being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in First Boot Complete being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Aug 05 20:35:22 archlinux systemd[1]: Condition check resulted in Repartition Root Disk being skipped.
Aug 05 20:35:22 archlinux systemd-fsck[284]: /dev/sdb2: recovering journal
Aug 05 20:35:23 archlinux systemd[1]: Mounted /efi.
Aug 05 20:35:26 archlinux kernel: mei_me 0000:00:16.0: wait hw ready failed
Aug 05 20:35:27 archlinux kernel: mei_me 0000:00:16.0: hw_start failed ret = -62
Aug 05 20:35:27 archlinux kernel: mei_me 0000:00:16.0: H_RST is set = 0x80050531
Aug 05 20:35:28 archlinux kernel: mei_me 0000:00:16.0: wait hw ready failed
Aug 05 20:35:28 archlinux kernel: mei_me 0000:00:16.0: hw_start failed ret = -62
Aug 05 20:35:28 archlinux kernel: mei_me 0000:00:16.0: H_RST is set = 0x80050531
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: wait hw ready failed
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: hw_start failed ret = -62
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: reset: reached maximal consecutive resets: disabling>
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: reset failed ret = -19
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: link layer initialization failed.
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: init hw failure.
Aug 05 20:35:31 archlinux kernel: mei_me 0000:00:16.0: initialization failed.
Aug 05 20:35:34 archlinux kernel: sysrq: Keyboard mode set to system default
Aug 05 20:35:35 archlinux systemd-fsck[284]: /dev/sdb2: Clearing orphaned inode 46405529 (uid=1000, gid=100>
Aug 05 20:35:35 archlinux systemd-fsck[284]: /dev/sdb2: clean, 785587/53739520 files, 137601907/214958080 b>
Aug 05 20:35:35 archlinux kernel: sysrq: Terminate All Tasks
Aug 05 20:35:35 archlinux systemd[1]: systemd-fsck@dev-disk-by\x2duuid-6bc55df8\x2d39ab\x2d4fd4\x2d9f89\x2d>
Aug 05 20:35:36 archlinux systemd-journald[180]: Journal stopped

The mei_me process refers to Intel Management Engine Interface,. How can I blacklist it for the kdump kernel? I'm using vanilla linux kernel as a kdump kernel, with crashkernel=256M boot parameter. I've also made and enabled a systemd service to automatically load the KDump kernel, here's the contents of that file:

~ > cat /etc/systemd/system/kdump.service
[Unit]
Description=Load dump capture kernel
After=local-fs.target

[Service]
ExecStart=/usr/bin/kexec -p /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --append="root=UUID="0994c97d-207b-4556-876f-1142de972d3d" single irqpoll maxcpus=1 reset_devices"
Type=oneshot

[Install]
WantedBy=multi-user.target

HotDogEnemy · 2021-08-05 19:01:45

I should mention that kdump kernel is loaded, as given bellow:

~ > cat /sys/kernel/kexec_crash_loaded
1

It's just, when I induce a kernel panic using Alt + Sysrq + C, nothing happens for more than a minute, the screen seems to be frozen just as it was when I issued the sysrq command, and the keyboard and mouse are unresponsive. What am I doing wrong?

Edit: After testing a bit more and following the Archwiki page all the way, the KDump kernel doesn't appear to boot in graphical mode, I waited for more than 10 minutes but the PC didn't reboot itself as it should have. When using console, the KDump kernel seems to have booted, as the system rebooted after 5 minutes. However, there was no crash dump at /var/crash.

Last edited by HotDogEnemy (2021-08-05 19:47:12)

seth · 2021-08-05 22:31:39

How can I blacklist it for the kdump kernel?

https://wiki.archlinux.org/title/Kernel … and_line_2

the KDump kernel doesn't appear to boot in graphical mode

https://wiki.archlinux.org/title/Kexec# … g_(Nvidia)

Not sure whether that's a problem w/ amdgpu, though.
However: do you get the same crashes w/ the radeon driver (given you've a SI chip)?

HotDogEnemy · 2021-08-06 10:35:45

I tried again with the mei_me and mei kernel modules blacklisted, like so:

~ > cat /etc/systemd/system/kdump.service                                                              at 15:18:51
[Unit]
Description=Load dump capture kernel
After=local-fs.target

[Service]
ExecStart=/usr/bin/kexec -p /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --append=root=UUID="0994c97d-207b-4556-876f-1142de972d3d" systemd.unit=kdump-save.service irqpoll module_blacklist=mei,mei_me nr_cpus=1 reset_devices
Type=oneshot

[Install]
WantedBy=multi-user.target

That didn't work, but on force rebooting the pc I saw some unusual error messages, here's the relevant portion from journalctl:

Aug 06 15:04:09 archlinux kernel: Yama: becoming mindful.
Aug 06 15:04:09 archlinux kernel: LSM support for eBPF active
Aug 06 15:04:09 archlinux kernel: Mount-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
Aug 06 15:04:09 archlinux kernel: Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
Aug 06 15:04:09 archlinux kernel: x86/cpu: SGX disabled by BIOS.
Aug 06 15:04:09 archlinux kernel: CPU0: Thermal monitoring enabled (TM1)
Aug 06 15:04:09 archlinux kernel: process: using mwait in idle threads
Aug 06 15:04:09 archlinux kernel: Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
Aug 06 15:04:09 archlinux kernel: Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 3: be00000000800400                   # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: TSC 0 ADDR ffffffffc039573e MISC ffffffffc039573e                  # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: PROCESSOR 0:506e3 TIME 1628242442 SOCKET 0 APIC 0 microcode a6     # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: Machine check events logged                                        # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 4: be00000000800400                   # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: TSC 0 ADDR ffffffffc039573e MISC ffffffffc039573e                  # UNUSUAL
Aug 06 15:04:09 archlinux kernel: mce: [Hardware Error]: PROCESSOR 0:506e3 TIME 1628242442 SOCKET 0 APIC 0 microcode a6     # UNUSUAL

I don't know whether these are from the kdump kernel or from "echo c > /proc/sysrq-trigger", but I didn't see them again yet.

Changing amdgpu to radeon didn't work, but even if the pc does reboot when in console, it didn't leave any crash logs at /var/crash.

I tried using kdump in an ubuntu vm and it rebooted within 10 seconds of the kernel crash, plus there was a crash log in /var/crash.

seth · 2021-08-07 07:24:47

Have you successfully replicated the issue w/ the non-zen kernel so far?

HotDogEnemy · 2021-08-08 08:51:13

Not yet, but I'll need a proper crash dump from the kernel panic to see if its actually the zen kernel or something else, and if it is, the crash dump will help in getting a good idea for what went wrong. Currently I'm trying to get kdump working.

Last edited by HotDogEnemy (2021-08-08 09:07:08)

HotDogEnemy · 2021-08-10 02:47:18

I got KDump working, now I just have to wait for the kernel panic to pop up again. There's one small issue with Kdump though, during panic, the kernel seems to load, but the video display is still frozen, even in console, meaning I have no way to check the on screen messages from the capture kernel to make sure that it's working properly or not.

Arch Linux

#1 2021-08-04 20:23:29

Random black screen with unresponsive keyboard and looping audio

#2 2021-08-05 08:33:29

Re: Random black screen with unresponsive keyboard and looping audio

#3 2021-08-05 12:20:23

Re: Random black screen with unresponsive keyboard and looping audio

#4 2021-08-05 14:13:40

Re: Random black screen with unresponsive keyboard and looping audio

#5 2021-08-05 15:24:14

Re: Random black screen with unresponsive keyboard and looping audio

#6 2021-08-05 15:29:06

Re: Random black screen with unresponsive keyboard and looping audio

#7 2021-08-05 16:04:44

Re: Random black screen with unresponsive keyboard and looping audio

#8 2021-08-05 19:01:45

Re: Random black screen with unresponsive keyboard and looping audio

#9 2021-08-05 22:31:39

Re: Random black screen with unresponsive keyboard and looping audio

#10 2021-08-06 10:35:45

Re: Random black screen with unresponsive keyboard and looping audio

#11 2021-08-07 07:24:47

Re: Random black screen with unresponsive keyboard and looping audio

#12 2021-08-08 08:51:13

Re: Random black screen with unresponsive keyboard and looping audio

#13 2021-08-10 02:47:18

Re: Random black screen with unresponsive keyboard and looping audio

Board footer