Shipping system logs _directly_ over the network without touching disc

gtf21 · 2023-09-13 09:04:12

I have a strange problem causing my machine to periodically lock up, but nothing shows up in the logs. The system becomes totally unresponsive (incl. to trying to switch TTY), so I can't see what's happening. The unresponsiveness comes on slowly, however -- first, say, my browser stops taking input, then after a few seconds the cursor freezes, and then it's kaput.

I want to try shipping logs directly to my home server over the wire, and because I'm not sure if there is a disc issue (nvmctl, smartmontools, and btrfsck all say there isn't one) I don't want to have to rely on the logs being recorded successfully on the disc first.

I've read the documentation for systemd-journal-upload, but that reads from disc and uploads. I'm also not sure if journalctl requires logs to be echoed to disc _first_ before displaying them (so can't just pipe journalctl -f into netcat or something).

How can I solve for this (can I solve for it)?

seth · 2023-09-13 10:38:17

What makes you confident the network stack won't be affected?

Do you have a serial console?
https://freedesktop.org/wiki/Software/s … al_Console

Alternatives:
https://wiki.archlinux.org/title/Rsyslog
https://wiki.archlinux.org/title/Genera … netconsole

From the symptoms: OOM condition?

gtf21 · 2023-09-14 08:06:09

What makes you confident the network stack won't be affected?

Fair point, having thought about it, I'm not confident.

Do you have a serial console?

I do not, but I could into trying to run one when my laptop is at my desk. In the meantime I will try your suggestions for remote logging with either rsyslog or netconsole.

From the symptoms: OOM condition?

Maybe, but I'm not sure how to test this, and I have a reasonable amount of RAM (32GB).

On this, however, the reason I thought this might be disc related is a previous post (https://bbs.archlinux.org/viewtopic.php?id=286456) in which I appeared to have disc errors, but I always suspected it was something else happening.

My current symptoms are different, and I can see things in the logs (e.g. seeing my attempts to use sysrq rejected before I enabled it).

Ok, I actually just caught the error as I was writing this reply. Weirdly, despite having `dmesg -w` set so I could follow any messages to see if something was up, nothing showed up there, BUT I have the following in my journal from the previous (crashed) session (I included up until I used sysrq to REISUB):

Sep 14 09:58:17 anaximander kernel: i915 0000:00:02.0: [drm] *ERROR* [CRTC:131:pipe B] flip_done timed out
Sep 14 09:58:27 anaximander kernel: i915 0000:00:02.0: [drm] *ERROR* flip_done timed out
Sep 14 09:58:27 anaximander kernel: i915 0000:00:02.0: [drm] *ERROR* [CRTC:131:pipe B] commit wait timed out
Sep 14 09:59:20 anaximander kernel: sysrq: Keyboard mode set to system default
Sep 14 09:59:37 anaximander iwd[836]: Terminate
Sep 14 09:59:37 anaximander iwd[836]: Removing scan context for wdev 2
Sep 14 09:59:37 anaximander bluetoothd[832]: Terminating
Sep 14 09:59:37 anaximander dhcpcd[840]: received SIGTERM, stopping
Sep 14 09:59:37 anaximander dhcpcd[840]: wlan0: removing interface
Sep 14 09:59:37 anaximander systemd[977]: Activating special unit Exit the Session...
Sep 14 09:59:37 anaximander kernel: sysrq: Terminate All Tasks

(I can move this to a different post because it's off-topic.)

Last edited by gtf21 (2023-09-14 08:13:28)

gtf21 · 2023-09-14 09:00:30

(Posted about the issue as I have seen it in here: https://bbs.archlinux.org/viewtopic.php … #p2120588)

Arch Linux

#1 2023-09-13 09:04:12

Shipping system logs _directly_ over the network without touching disc

#2 2023-09-13 10:38:17

Re: Shipping system logs _directly_ over the network without touching disc

#3 2023-09-14 08:06:09

Re: Shipping system logs _directly_ over the network without touching disc

#4 2023-09-14 09:00:30

Re: Shipping system logs _directly_ over the network without touching disc

Board footer