How to investigate archlinux crash

buntolo · 2017-07-14 10:43:23

I was running badblocks on a drive for 2 days, this morning archlinux was stuck at gdm login screen showing 9:42 (hour) and I couldn't access, not even using ssh.

I've rebooted and used 'journalctl --since=today' but I couldn't find any reason for the crash, at 9:42 mark there's only a lot of lines like these:
"imarchio dhcpcd[489]: eno1: requesting DHCPv6 information
Router Advertisement from fe80::1"

It seems instead, kernel was still running.

Here's the paste (it's over 9000) of 'journalctl --since=today': https://ptpb.pw/ebQq

Trilby · 2017-07-14 12:56:47

Welcome to later.

I'd start by trying to boot into mutliuser.target.

buntolo · 2017-07-14 15:59:18

Trilby wrote:

Welcome to later.
I'd start by trying to boot into mutliuser.target.

Ok, I'm in multi-user now, what's next? I'm running 2 badblocks scannings at the same time.

ps: the computer having this issue is my "new" desktop, with a fresh archlinux install, like 1 week ago, to test Wayland + GNOME.

Morn · 2017-07-14 16:17:52

buntolo wrote:

to test Wayland + GNOME.

That might be the cause of your stability problems right there. I would try old-fashioned X11, a different window manager or KDE, and no login manager at all. Login managers on Linux are nothing but trouble in my experience so far.

Yay for "startx", and may it never leave my system.

buntolo · 2017-07-14 17:17:13

Morn wrote:

buntolo wrote:
to test Wayland + GNOME.
That might be the cause of your stability problems right there. I would try old-fashioned X11, a different window manager or KDE, and no login manager at all. Login managers on Linux are nothing but trouble in my experience so far.
Yay for "startx", and may it never leave my system.

I agree, personally I'm a Xfce user since the day G3 came out, but I'm testing this "new" computer with new stuff. Sadly I can use only G3 for Wayland, as I've a nVIDIA VGA.

Anyway what bothers me is that I can't find any clue about the crash in journalctl.

JohnBobSmith · 2017-07-14 18:43:11

You have 3 systemd-coredumps in your journal. Ctrl + f on your journal pastebin and search for systemd-coredump and there are your 3 crashes right there. Why systemd is dumping the core is a good question, which I dont have an answer to. Start by investigating these crashes with coredumpctl. From there you might be able to find something of value to troubleshoot further.

buntolo · 2017-07-14 19:30:40

There's something strange: today I've rebooted the computer for the first time at ~10:50, infact journalctl shows a reboot line between 09:42 (the time shown on frozen gdm) and 10:52:

Jul 14 09:42:29 imarchio dhcpcd[489]: eno1: requesting DHCPv6 information
-- Reboot --
Jul 14 10:52:14 imarchio kernel: Linux version 4.11.6-3-ARCH (builduser@anthraxx) (gcc version 7.1.1 20170516 (GCC) ) #1 SMP PREEMPT Thu Jun 22 12:21:46 CEST 2017

The first systemd-coredump is at 10:52 mark, so after reboot, and the first crash happened before, so its reason may be unrelated.

I've followed these instructions: https://wiki.archlinux.org/index.php/Core_dump
I understand nothing. I've understood only that colord-sane is a library provided by package colord and needed by gtk and weston (and a bunch of other packages). It exited with signal 6, abort.

'coredumpctl info colord-sane'
https://ptpb.pw/hkLt

'sudo coredumpctl gdb colord-sane'
https://ptpb.pw/hVfN

Morn · 2017-07-14 19:41:46

buntolo wrote:

Anyway what bothers me is that I can't find any clue about the crash in journalctl.

That's why I'm thinking it might be related to the graphics card, its driver, or Wayland somehow. Graphics and network cards can lock up a Linux system without evidence in the logs.

This is actually an area where Windows is more resilient than Linux; if the graphics driver crashes on Windows the system does not lock up. But on Linux the whole system freezes.

JohnBobSmith · 2017-07-14 19:42:09

According to https://www.freedesktop.org/software/colord/intro.html colord is "... a system service that makes it easy to manage, install and generate color profiles to accurately color manage input and output devices." It seems to be a package in the official repositories:

Fri Jul 14 jbs@dmb-gaming-laptop -> pacman -Ss colord
extra/colord 1.3.5-1
    System daemon for managing color devices

So now you want to run

pacman -Qi colord

To determine why/if you need the package and service. If you decide you dont need it, disable the service and remove the package. See if that fixes your crashing issues. On a related note, you've definitely got issues worthy of a bug report. I'm not sure where upstream you need to go, but include this forum thread and your coredump output in the bug report, should you decide to report these issues upstream as bugs.

loqs · 2017-07-14 20:09:33

The colord-sane issue looks to be https://bugs.archlinux.org/task/54205 see also https://bbs.archlinux.org/viewtopic.php?id=226867

buntolo · 2017-07-14 20:30:46

Ok, I'll try to report a bug, I need to know who was calling colord, reading the lines containing "colord" from journalctl, I don't understand if it's dbus, org.freedesktop,ColorManager (CUPS?), Avahi, GLib, or whatever:

Jul 14 10:52:22 imarchio dbus[503]: [system] Activating via systemd: service name='org.freedesktop.ColorManager' unit='colord.service'
[...]
Jul 14 10:52:24 imarchio dbus[503]: [system] Activating via systemd: service name='org.freedesktop.Avahi' unit='dbus-org.freedesktop.Avahi.service'
Jul 14 10:52:24 imarchio dbus[503]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.Avahi.service': Unit dbus-org.freedesktop.Avahi.service not found.
Jul 14 10:52:27 imarchio dbus[503]: [system] Activating via systemd: service name='org.freedesktop.Avahi' unit='dbus-org.freedesktop.Avahi.service'
Jul 14 10:52:27 imarchio dbus[503]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.Avahi.service': Unit dbus-org.freedesktop.Avahi.service not found.
Jul 14 10:52:27 imarchio colord[1084]: GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.

Can you use advanced search on the bugtracker? I get a blank page:

https://bugs.archlinux.org/index.php?string=+colord-sane+abort&project=1&type[]=1&sev[]=&pri[]=&due[]=&reported[]=&cat[]=2&status[]=&percent[]=&opened=&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=&do=index

Last edited by buntolo (2017-07-14 20:33:08)

buntolo · 2017-07-15 10:13:26

Ok, this morning I've got a kernel panic, I only have a picture as kernel panic has not been logged, can I post it (as it would be against rules)?

I want to make a custom kernel to log kernel panic with Kdump, but I'm stuck at the beginning, I don't understand where the PKGBUILD the wiki is referring to is located: https://wiki.archlinux.org/index.php/Ke … e_PKGBUILD
There are various PKGBUILDs for every kernel, but I don't get why it downloaded testing as well, if I checked out asp for linux and I'm not using testing anywhere.

Should I just navigate to the folder I want (x86_64 core kernel), edit PKGBUILD and then go on?

This is the last journalctl, crash happened before reboot (indicated as "-- Reboot --"): https://ptpb.pw/BVr6

Last edited by buntolo (2017-07-15 10:29:44)

loqs · 2017-07-15 10:41:16

buntolo wrote:

Ok, this morning I've got a kernel panic, I only have a picture as kernel panic has not been logged, can I post it (as it would be against rules)?

If the system is booting under UEFI the panic might be stored in /sys/fs/pstore/ if not Just post a thumbnail link.

buntolo wrote:

Should I just navigate to the folder I want (x86_64 core kernel), edit PKGBUILD and then go on?

Yes

ewaller · 2017-07-15 15:15:51

Post a link to the image, not the image.

buntolo · 2017-07-15 15:54:17

Ok, I've made a custom kernel, now I've 2 problems:
1) Linux rEFInd menu voice has duplicate entries, both for normal kernel and custom one.
2) I can't insert nvidia module in initramfs.

I've searched for hours, I'm behind this stuff since this morning, but for what I've seen informations are scarce. I can boot normal kernel, but not custom one, only its fallback mode works.

I'm trying to see how custom kernels in AUR solve this problem, but I can't understand: https://aur.archlinux.org/cgit/aur.git/ … =nvidia-ck
(its the nvidia driver for linux-ck kernel)

Added picture, its quality is very shitty, as my phone is very crappy: http://i.imgur.com/Z9KTmQ9.jpg

Last edited by buntolo (2017-07-15 16:53:19)

loqs · 2017-07-15 17:29:20

buntolo wrote:

I can boot normal kernel, but not custom one, only its fallback mode works.

What happens when you try to use the custom kernel with its normal initrd?

buntolo wrote:

I'm trying to see how custom kernels in AUR solve this problem, but I can't understand: https://aur.archlinux.org/cgit/aur.git/ … =nvidia-ck
(its the nvidia driver for linux-ck kernel)

What is the current PKGBUILD you are using for your nvidia-custom package? Have you considered using DKMS?

buntolo · 2017-07-15 20:27:53

loqs wrote:

What happens when you try to use the custom kernel with its normal initrd?

Kernel panic: http://i.imgur.com/CLVqsUO.jpg

I've also tried removing changing 'initrd=/boot/initramfs-linux-kdump' to 'initrd=/initramfs-linux-kdump' but the result is the same.

loqs wrote:

What is the current PKGBUILD you are using for your nvidia-custom package?

I don't have one, I don't know how make a custom nvidia package. I've tried to read an example nvidia-custom PKGBUILD but I don't understand how to make a nvidia-custom PKGBUILD.

loqs wrote:

Have you considered using DKMS?

Yes and it doesn't build initramfs.

I boot using "nvidia-drm.modeset=1" as kernel parameter, so as instructed in the wiki (https://wiki.archlinux.org/index.php/NV … de_setting) I've added nvidia, nvidia_modeset, nvidia_uvm and nvidia_drm in /etc/mkinitcpio.conf.
When I use 'sudo mkinitcpio -p linux-kdump' (my custom kernel), it build everything but nvidia (the other nvidia_foo get built for what I understand).

Last edited by buntolo (2017-07-15 20:28:54)

loqs · 2017-07-15 20:58:19

buntolo wrote:

loqs wrote:
What happens when you try to use the custom kernel with its normal initrd?
Kernel panic: http://i.imgur.com/CLVqsUO.jpg
I've also tried removing changing 'initrd=/boot/initramfs-linux-kdump' to 'initrd=/initramfs-linux-kdump' but the result is the same.

Is the initrd initramfs-linux-kdump.img present in /boot? If it is compare the refind entry with that for the fallback initrd that boots.

buntolo wrote:

loqs wrote:
What is the current PKGBUILD you are using for your nvidia-custom package?
I don't have one, I don't know how make a custom nvidia package. I've tried to read an example nvidia-custom PKGBUILD but I don't understand how to make a nvidia-custom PKGBUILD.

You could not find any kernel speciffic lines in the PKGBUILD and install file that would need altering such as
PKGBUILD

pkgname=nvidia-ck
_extramodules=extramodules-4.11-ck
_pkgdesc="NVIDIA drivers for linux-ck."
depends=('linux-ck>=4.11' 'linux-ck<4.12' 'libgl' "nvidia-utils=${pkgver}")
makedepends=('linux-ck-headers>=4.11' 'linux-ck-headers<4.12' 'nvidia-libgl')
conflicts=('nvidia-340xx-ck' 'nvidia-304xx-ck')

readme.install

	EXTRAMODULES='extramodules-4.11-ck'

buntolo wrote:

loqs wrote:
Have you considered using DKMS?
Yes and it doesn't build initramfs.

So why not manually rebuild the initrd after DKMS generated the module?
Edit:
As upstream does not support the nvidia module an upstream bug report made with it present may simply be rejected.
Reporting the issue on the arch bugtracker I would expect you to be directed upstream with the previously mentioned issue.
Have you also considered that the issue is hardware in nature in which case there may be nothing upstream can do.

Last edited by loqs (2017-07-15 22:46:44)

buntolo · 2017-07-16 13:53:52

I've a new kernel panic and a better picture: http://i.imgur.com/N2VzwVm.jpg

loqs wrote:

You could not find any kernel speciffic lines in the PKGBUILD and install file that would need altering such as
PKGBUILD

pkgname=nvidia-ck
_extramodules=extramodules-4.11-ck
_pkgdesc="NVIDIA drivers for linux-ck."
depends=('linux-ck>=4.11' 'linux-ck<4.12' 'libgl' "nvidia-utils=${pkgver}")
makedepends=('linux-ck-headers>=4.11' 'linux-ck-headers<4.12' 'nvidia-libgl')
conflicts=('nvidia-340xx-ck' 'nvidia-304xx-ck')

readme.install

	EXTRAMODULES='extramodules-4.11-ck'

I'll try, but it's like black magic for me and I couldn't find any clue or explanation around the web.

So why not manually rebuild the initrd after DKMS generated the module?

If you mean using 'mkinitcpio -p linux-kdump' after installing nvidia-dkms, I did it.

Edit:
As upstream does not support the nvidia module an upstream bug report made with it present may simply be rejected.
Reporting the issue on the arch bugtracker I would expect you to be directed upstream with the previously mentioned issue.

I'm sorry but I didn't understand, probably my English isn't good enough, could you explain again using different words?

Have you also considered that the issue is hardware in nature in which case there may be nothing upstream can do.

I'd exclude an hardware issue with the VGA, as I've tested it for hours with several steam games (like 30 games launched and kept running for minimium 2 hours each).

EDIT: I've edited nvidia-ck PKGBUILD and readme.install, replacing every -ck with -kdump, but still it doesn't build initramfs:

% sudo mkinitcpio -p linux-kdump
==> Building image from preset: /etc/mkinitcpio.d/linux-kdump.preset: 'default'
  -> -k /boot/vmlinuz-linux-kdump -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-kdump.img
==> Starting build: 4.11.9-1-kdump
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> ERROR: module not found: `nvidia_kdump'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-kdump.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux-kdump.preset: 'fallback'
  -> -k /boot/vmlinuz-linux-kdump -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-kdump-fallback.img -S autodetect
==> Starting build: 4.11.9-1-kdump
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> ERROR: module not found: `nvidia_kdump'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-kdump-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
sudo mkinitcpio -p linux-kdump  62.29s user 5.15s system 102% cpu 1:05.56 total

Last edited by buntolo (2017-07-16 14:29:08)

loqs · 2017-07-16 16:40:42

buntolo wrote:

loqs wrote:
So why not manually rebuild the initrd after DKMS generated the module?
If you mean using 'mkinitcpio -p linux-kdump' after installing nvidia-dkms, I did it.

So what was the result?

buntolo wrote:

loqs wrote:
Edit:
As upstream does not support the nvidia module an upstream bug report made with it present may simply be rejected.
Reporting the issue on the arch bugtracker I would expect you to be directed upstream with the previously mentioned issue.
I'm sorry but I didn't understand, probably my English isn't good enough, could you explain again using different words?

Trying to put it differently what are you going to do with this crash dump? Who do you expect to examine it?

buntolo wrote:

loqs wrote:
Have you also considered that the issue is hardware in nature in which case there may be nothing upstream can do.
I'd exclude an hardware issue with the VGA, as I've tested it for hours with several steam games (like 30 games launched and kept running for minimium 2 hours each).

That would match up with the new screen capture which shows a disk/controller/block subsystem error.

buntolo wrote:

EDIT: I've edited nvidia-ck PKGBUILD and readme.install, replacing every -ck with -kdump, but still it doesn't build initramfs:

% sudo mkinitcpio -p linux-kdump
==> Building image from preset: /etc/mkinitcpio.d/linux-kdump.preset: 'default'
  -> -k /boot/vmlinuz-linux-kdump -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-kdump.img
==> Starting build: 4.11.9-1-kdump
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> ERROR: module not found: `nvidia_kdump'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-kdump.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux-kdump.preset: 'fallback'
  -> -k /boot/vmlinuz-linux-kdump -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-kdump-fallback.img -S autodetect
==> Starting build: 4.11.9-1-kdump
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> ERROR: module not found: `nvidia_kdump'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-kdump-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
sudo mkinitcpio -p linux-kdump  62.29s user 5.15s system 102% cpu 1:05.56 total

The module should still be named nvidia please post the edited PKGBUILD and readme.install.

buntolo · 2017-07-16 18:19:24

Thank you very much for all this help and attention.

loqs wrote:

So what was the result?

Like in the previous post: initramfs building failed for nvidia error, module not found. Previously it was only nvidia as I didn't create a custom nvidia package. I don't remember if when using nvidia-dkms the module not found was nvidia or nvidia_dkms.

Trying to put it differently what are you going to do with this crash dump? Who do you expect to examine it?

Honestly I don't know, my hope is finding more clues about kernel panic using a dump kernel. I'm following this article: https://wiki.archlinux.org/index.php/Kdump

That would match up with the new screen capture which shows a disk/controller/block subsystem error.

Everytime I got a kernel panic, it was while running badblocks on a disk. Is there a way to locate more precisely who's causing issues? May an empty disk scanned for badblocks cause a kernel panic? Shouldn't something this bad happens only if there are issues with the disk where system (or its components, like var or swap) is installed on?

The module should still be named nvidia please post the edited PKGBUILD and readme.install.

Module is still nvidia.ko:
PKGBUILD: https://ptpb.pw/q43p
readme.install:

post_install() {
	EXTRAMODULES='extramodules-4.11-kdump'
	depmod $(cat /usr/lib/modules/$EXTRAMODULES/version)
	echo 'In order to use nvidia module, reboot the system.'
}

post_upgrade() {
    post_install
}

post_remove() {
    post_install
}

loqs · 2017-07-16 18:33:43

==> ERROR: module not found: `nvidia_kdump'

Are you specifying nvidia_kdump in mkinitcpio.conf?

Ideally a device failure should not itself lead to a kernel panic. The issue though would be if you submit such a bug report upstream with results from a tainted kernel.
A tainted kernel means a kernel using any module that is not supplied by the upstream itself such as the nvidia module.

As the system panics appear to be a direct result of trying to repair that disk I would expect continued attempts to use the disk to result in more kernel panics.

buntolo · 2017-07-16 19:46:55

I'm specifying nvidia-kdump, with - and not _, I'll try wth _.

I'll see about reporting bug upstream when I'll know which module is causing kernel panic.

loqs · 2017-07-16 19:57:20

The module name should remain the same ie nvidia.

buntolo · 2017-07-17 21:44:48

Thank you! Now I've successfully built initramfs and runinng custom kernel:

% uname -r
4.11.9-1-kdump

I've spoken too early, kdump.service (created following https://wiki.archlinux.org/index.php/Kd … mp_kernel) fails to start and I can't understand why:

% systemctl status kdump
● kdump.service - Load dump capture kernel
   Loaded: loaded (/etc/systemd/system/kdump.service; enabled; vendor preset: di
   Active: failed (Result: exit-code) since Tue 2017-07-18 11:13:13 CEST; 9min a
  Process: 448 ExecStart=/usr/bin/kexec -p /boot/vmlinuz-linux-kdump --initrd=/b
 Main PID: 448 (code=exited, status=255)

Jul 18 11:13:13 imarchio systemd[1]: Starting Load dump capture kernel...
Jul 18 11:13:13 imarchio kexec[448]: ELF core (kcore) parse failed
Jul 18 11:13:13 imarchio kexec[448]: Cannot load /boot/vmlinuz-linux-kdump
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Main process exited, code=ex
Jul 18 11:13:13 imarchio systemd[1]: Failed to start Load dump capture kernel.
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Unit entered failed state.
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Failed with result 'exit-cod
lines 1-13/13 (END)...skipping...
● kdump.service - Load dump capture kernel
   Loaded: loaded (/etc/systemd/system/kdump.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-07-18 11:13:13 CEST; 9min ago
  Process: 448 ExecStart=/usr/bin/kexec -p /boot/vmlinuz-linux-kdump --initrd=/boot/initramfs-linux-kdump.img --append=root=UUID=ed399856-4756-40f8-9dfd-8a654fb6337e single irqpoll maxcpus=1 reset_devices (code=exited, status=255)
 Main PID: 448 (code=exited, status=255)

Jul 18 11:13:13 imarchio systemd[1]: Starting Load dump capture kernel...
Jul 18 11:13:13 imarchio kexec[448]: ELF core (kcore) parse failed
Jul 18 11:13:13 imarchio kexec[448]: Cannot load /boot/vmlinuz-linux-kdump
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Main process exited, code=exited, status=255/n/a
Jul 18 11:13:13 imarchio systemd[1]: Failed to start Load dump capture kernel.
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Unit entered failed state.
Jul 18 11:13:13 imarchio systemd[1]: kdump.service: Failed with result 'exit-code'.

Last edited by buntolo (2017-07-18 07:42:19)

Arch Linux

#1 2017-07-14 10:43:23

How to investigate archlinux crash

#2 2017-07-14 12:56:47

Re: How to investigate archlinux crash

#3 2017-07-14 15:59:18

Re: How to investigate archlinux crash

#4 2017-07-14 16:17:52

Re: How to investigate archlinux crash

#5 2017-07-14 17:17:13

Re: How to investigate archlinux crash

#6 2017-07-14 18:43:11

Re: How to investigate archlinux crash

#7 2017-07-14 19:30:40

Re: How to investigate archlinux crash

#8 2017-07-14 19:41:46

Re: How to investigate archlinux crash

#9 2017-07-14 19:42:09

Re: How to investigate archlinux crash

#10 2017-07-14 20:09:33

Re: How to investigate archlinux crash

#11 2017-07-14 20:30:46

Re: How to investigate archlinux crash

#12 2017-07-15 10:13:26

Re: How to investigate archlinux crash

#13 2017-07-15 10:41:16

Re: How to investigate archlinux crash

#14 2017-07-15 15:15:51

Re: How to investigate archlinux crash

#15 2017-07-15 15:54:17

Re: How to investigate archlinux crash

#16 2017-07-15 17:29:20

Re: How to investigate archlinux crash

#17 2017-07-15 20:27:53

Re: How to investigate archlinux crash

#18 2017-07-15 20:58:19

Re: How to investigate archlinux crash

#19 2017-07-16 13:53:52

Re: How to investigate archlinux crash

#20 2017-07-16 16:40:42

Re: How to investigate archlinux crash

#21 2017-07-16 18:19:24

Re: How to investigate archlinux crash

#22 2017-07-16 18:33:43

Re: How to investigate archlinux crash

#23 2017-07-16 19:46:55

Re: How to investigate archlinux crash

#24 2017-07-16 19:57:20

Re: How to investigate archlinux crash

#25 2017-07-17 21:44:48

Re: How to investigate archlinux crash

Board footer