Diagnosing hardware problems

Drvanon · 2020-09-06 09:28:01

Hi,
I have been lurking here for a very long time (been using arch since 2015 I think), but I have never really had the need to engage, as usually my problem was "googlable". This time however I am rather quite out of my depth and seeing that I only use arch and feel far from being a proper power user, I think I will ask my question here, as I am very new to the lower levels of software development.

The issue: my pc running Arch freezes sometimes. Today it froze after 4 minutes of uptime, sometimes it freezes after 3 days of uptime, it's rather quite unpredictable when it will happen. I installed the proper micro code package after I read that this could be a source of these types of problems, however it did not resolve the issue for me.

Sometimes instead of freezing and forcing me to force shutdown using my hardware button, it will simply reboot itself. This then goes together with a "Hardware error" message in the login screen (I have the default TTY login that arch comes with), however it does not last long enough for me to read it, let alone deduce its meaning. I also could not find this message in my journalctl. Quite often these errors lead to problems in my filesystem, which then has to be cleaned up. I have lost some files due to this, I think.

All this to say: my expectation is that I am having an hardware issue, but no clue where it could be. Any help trying to figure out where this issue could be would be greatly appreciated.

d_fajardo · 2020-09-06 14:22:14

PC rebooting by itself very much sound like a hardware problem.
You can check the logs at the point when it freezes and the point when it reboots itself.
You can run smartctl to check your disk.
You can run memtest to check your RAM.

Drvanon · 2020-09-06 16:22:03

Ran memtest: no errors found.

Journalctl -b -2 gives the following output: https://termbin.com/jy83. Of interest there is the lines near the end where my graphics card seems to have suffered. (Search for " ------------[ cut here ]------------").

After running "# smartctl -t short /dev/sda", I got as a result for "# smartctl -H /dev/sda": https://termbin.com/wx0h, which looks good to me.

d_fajardo · 2020-09-06 16:41:01

------------[ cut here ]------------
Sep 06 11:04:16 robin-tower kernel: WARNING: CPU: 2 PID: 133 at drivers/gpu/drm/nouveau/nvif/vmm.c:68 nvif_vmm_put+0x73/0x80 [nouveau]
Sep 06 11:04:16 robin-tower kernel: Modules linked in: vxlan ip6_udp_tunnel udp_tunnel xt_policy iptable_mangle xt_mark xt_u32 xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter bridge stp llc ccm input_leds joydev mousedev hid_generic usbhid wmi_bmof hid nouveau rtl8821ae edac_mce_amd snd_hda_codec_realtek btcoexist snd_hda_codec_generic snd_hda_codec_hdmi ledtrig_audio rtl_pci snd_hda_intel rtlwifi overlay mxm_wmi kvm snd_intel_dspcfg mac80211 snd_hda_codec i2c_algo_bit irqbypass ttm snd_hda_core snd_hwdep crct10dif_pclmul crc32_pclmul ghash_clmulni_intel drm_kms_helper cfg80211 snd_pcm cec aesni_intel r8169 rc_core crypto_simd ccp realtek cryptd snd_timer glue_helper syscopyarea sp5100_tco sysfillrect snd rapl rfkill sysimgblt k10temp i2c_piix4 libphy rng_core libarc4 fb_sys_fops soundcore wmi gpio_amdpt evdev pinctrl_amd mac_hid acpi_cpufreq drm
Sep 06 11:04:16 robin-tower kernel:  agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 crc32c_intel xhci_pci xhci_pci_renesas xhci_hcd
Sep 06 11:04:16 robin-tower kernel: CPU: 2 PID: 133 Comm: kworker/2:1 Not tainted 5.8.1-arch1-1 #1
Sep 06 11:04:16 robin-tower kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./AB350M Pro4, BIOS P2.30 04/10/2017
Sep 06 11:04:16 robin-tower kernel: Workqueue: events nouveau_cli_work [nouveau]
Sep 06 11:04:16 robin-tower kernel: RIP: 0010:nvif_vmm_put+0x73/0x80 [nouveau]
Sep 06 11:04:16 robin-tower kernel: Code: 8b 06 48 89 f3 b9 10 00 00 00 48 89 e2 be 02 00 00 00 48 89 44 24 08 e8 5b e3 ff ff 85 c0 75 0a 48 c7 43 08 00 00 00 00 eb bf <0f> 0b eb f2 e8 94 e5 0d c6 0f 1f 40 00 0f 1f 44 00 00 53 48 83 ec
Sep 06 11:04:16 robin-tower kernel: RSP: 0018:ffff9765003d3de0 EFLAGS: 00010282
Sep 06 11:04:16 robin-tower kernel: RAX: 00000000fffffffe RBX: ffff9765003d3e08 RCX: 0000000000000000
Sep 06 11:04:16 robin-tower kernel: RDX: 0000000000000000 RSI: ffff9765003d3d50 RDI: ffff9765003d3df0
Sep 06 11:04:16 robin-tower kernel: RBP: ffff888c67cea880 R08: 00000000fffffffe R09: 0000000000000000
Sep 06 11:04:16 robin-tower kernel: R10: 0000000000000030 R11: 0000000000000018 R12: dead000000000122
Sep 06 11:04:16 robin-tower kernel: R13: dead000000000100 R14: ffff888c591d4480 R15: ffff888c6dd4f6a8
Sep 06 11:04:16 robin-tower kernel: FS:  0000000000000000(0000) GS:ffff888c96e80000(0000) knlGS:0000000000000000
Sep 06 11:04:16 robin-tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 06 11:04:16 robin-tower kernel: CR2: 000004135e94f008 CR3: 00000001b1974000 CR4: 00000000003406e0
Sep 06 11:04:16 robin-tower kernel: Call Trace:
Sep 06 11:04:16 robin-tower kernel:  nouveau_vma_del+0x89/0xc0 [nouveau]
Sep 06 11:04:16 robin-tower kernel:  nouveau_gem_object_delete_work+0x36/0x60 [nouveau]
Sep 06 11:04:16 robin-tower kernel:  nouveau_cli_work+0xbb/0x110 [nouveau]
Sep 06 11:04:16 robin-tower kernel:  process_one_work+0x1da/0x3d0
Sep 06 11:04:16 robin-tower kernel:  worker_thread+0x4d/0x3d0
Sep 06 11:04:16 robin-tower kernel:  ? rescuer_thread+0x410/0x410
Sep 06 11:04:16 robin-tower kernel:  kthread+0x142/0x160
Sep 06 11:04:16 robin-tower kernel:  ? __kthread_bind_mask+0x60/0x60
Sep 06 11:04:16 robin-tower kernel:  ret_from_fork+0x22/0x30
Sep 06 11:04:16 robin-tower kernel: ---[ end trace f282be3ad8ad9d85 ]---

Yes it seems nouveau crashes which will point to your graphics card (or nouveau).
It's not clear for me in the logs - is this integrated or discrete graphics you're using?

Drvanon · 2020-09-06 17:23:52

The graphics card is not integrated. It's external. Should I expect to be buying a new one, or is this something fixable?

d_fajardo · 2020-09-06 17:44:01

Should I expect to be buying a new one, or is this something fixable?

I don't know anyone who has fixed a graphics card and besides it's time like this that presents an opportunity to update or upgrade.
Still I would see if it might just be nouveau crashing by trying a different driver if possible, or further checking logs when another crash happens to see if graphics is really the problem. The symptoms you described though points to hardware.
One thing I'm concerned is how your filesystem gets corrupted and how if the problem with is with graphics, it can corrupt the file system.
What card is it?

zpg443 · 2020-09-06 18:13:40

If your CPU has integrated graphics, you could use that to confirm the issue is your graphics card.

d_fajardo · 2020-09-06 18:49:39

If your CPU has integrated graphics, you could use that to confirm the issue is your graphics card.

Yes, good thinking.

Drvanon · 2020-09-06 19:03:11

@zpg443, That would be a solid idea, if I had an integrated one. Sadly I do not.

@d_fajardo, Concerning the crashing the file system: somehow it manages to freeze my entire computer, so I think that the creates the orphaned inodes.

I tend to keep my distro relatively up to date, not going more than 2 months without an upgrade. How can I deduce if it is the driver or the hardware?

V1del · 2020-09-06 20:18:19

Nouveau generally does not tend to be the most stable driver, try the proprietary nvidia driver.

FWIW based on purely anecdotal evidence without doing much digging, it seems that the 5.8 kernel has a load of low level, and not too stable drm subsystem changes, if you don't want to test the nvidia driver, you might want to give the LTS kernel a shot.

Last edited by V1del (2020-09-06 20:20:05)

Drvanon · 2020-09-07 07:36:45

I just had another crash and this time it left a message:

Sep 07 09:30:01 robin-tower kernel: mce: [Hardware Error]: Machine check events logged
Sep 07 09:30:01 robin-tower kernel: mce: [Hardware Error]: CPU 1: Machine Check: 0 Bank 5: bea0000000000108
Sep 07 09:30:01 robin-tower kernel: mce: [Hardware Error]: TSC 0 ADDR 18c1c523c1180 MISC d012000101000000 SYND 4d000000 IPID 500b000000000 
Sep 07 09:30:01 robin-tower kernel: mce: [Hardware Error]: PROCESSOR 2:800f11 TIME 1599463799 SOCKET 0 APIC 1 microcode 800111c

Is this also from the GPU, or do I have two problems?

V1del · 2020-09-07 07:49:04

That's CPU and going by the posted journal you do not apply your microcode updates yet: https://wiki.archlinux.org/index.php/Microcode that could definitely be a contributing factor (installing the package is often not enough, apply it early as mentioned for the bootloader you've chosen). If that doesn't help already, make sure your UEFI is at the latest update, there have been lots of issues in earlier AMD CPU firmwares.

Last edited by V1del (2020-09-07 07:51:36)

Drvanon · 2020-09-07 13:01:40

Thanks for the hint on microcode not being properly installed. I have fixed that now, so hoping that that would do it! As this is my only graphics card I want to set up SSH properly in case that I am going to get a black screen.

Drvanon · 2020-09-08 08:51:29

@V1del, after installing microcode my computer seems to boot significantly faster. I can't give hard numbers because I never timed it before, but I am getting the impression of a 10s drop. Is this supposed to happen?

V1del · 2020-09-08 08:59:58

All sorts of things can happen, you replace the firmware of your CPU and there could be bugs/race conditions/broken instructions that got fixed/changed runtime (for better or worse). E.g. if you've followed this somewhat the CPU vulnerabilities for spectre and the like necessitated a microcode fix and you only really get the potential performance decreases (and the resulting security fixes...) when your microcode is relevantly up to date.

The kernel often internaly disables/blacklists certain functionality when it detects you are on a microcode version that has known bugs with whatever implementation they use.

What I can see having a notable impact here is that there's a lot of RNG involved during the initial startup phase for populating keys and early service setup. And I have a faint memory of early AMD CPU RNG being slow/not implemented properly that got fixed in subsequent updates so something along those lines is definitely possible

Arch Linux

#1 2020-09-06 09:28:01

Diagnosing hardware problems

#2 2020-09-06 14:22:14

Re: Diagnosing hardware problems

#3 2020-09-06 16:22:03

Re: Diagnosing hardware problems

#4 2020-09-06 16:41:01

Re: Diagnosing hardware problems

#5 2020-09-06 17:23:52

Re: Diagnosing hardware problems

#6 2020-09-06 17:44:01

Re: Diagnosing hardware problems

#7 2020-09-06 18:13:40

Re: Diagnosing hardware problems

#8 2020-09-06 18:49:39

Re: Diagnosing hardware problems

#9 2020-09-06 19:03:11

Re: Diagnosing hardware problems

#10 2020-09-06 20:18:19

Re: Diagnosing hardware problems

#11 2020-09-07 07:36:45

Re: Diagnosing hardware problems

#12 2020-09-07 07:49:04

Re: Diagnosing hardware problems

#13 2020-09-07 13:01:40

Re: Diagnosing hardware problems

#14 2020-09-08 08:51:29

Re: Diagnosing hardware problems

#15 2020-09-08 08:59:58

Re: Diagnosing hardware problems

Board footer