[SOLVED] Machine locks up after 16-48 hours.

SirBlobman · 2021-03-16 18:46:23

I have a custom built computer with the following specs:

Board: ASUS Prime A320M-K
CPU: AMD Ryzen 3 1200
RAM: 8GB DDR4
GPU: AMD Radeon R5 232
Cooler: Asetek 550LC

This is my first successful setup with archlinux (previously I would always forget to setup the network or boot manager correctly), but it seems that the machine cannot stay online for more than 48 hours. It always locks up or freezes, which doesn't allow me to login through SSH or even through a wired keyboard to the device.

Attempts to Fix:
These are the attempts I've had so far at fixing the issue with no luck.

Installing `amd-ucode` and rebooting the system.
Adding "processor.max_cstate=1 intel_idle.max_cstate=0" to the kernel line in grub.cfg
Updating the system with "pacman -Syu" and rebooting.

Here is my command output for "journalctl -b -1", any help is appreciated.
https://pastebin.com/QQVFSQ2q

Last edited by SirBlobman (2021-03-21 02:09:30)

Morn · 2021-03-16 20:29:04

First confirm that

journalctl |grep -i "hardware err"

outputs nothing. That should mean the Ryzen CPU freeze bug has been successfully mitigated.

If the CPU isn't locking the system up, it is usually the GPU in my experience. Is the machine running X or Wayland when it happens? If so, try to run the system in console mode for a while.

SirBlobman · 2021-03-16 20:32:58

Morn wrote:

First confirm that
journalctl |grep -i "hardware err"
outputs nothing. That should mean the Ryzen CPU freeze bug has been successfully mitigated.
If the CPU isn't locking the system up, it is usually the GPU in my experience. Is the machine running X or Wayland when it happens? If so, try to run the system in console mode for a while.

Thanks for replying! Here is a screenshot of the command being run in an SSH window.

The machine is in console mode only, I have not installed X or Wayland as far as I know.
I'm using it to host some services I need (Apache2, Jenkins, and Nexus) which is why I need it to stay online and not lock up.

Morn · 2021-03-16 20:47:12

Good, then it looks like it's not the Ryzen bug nor the GPU causing issues.

Another simple experiment would be to install and boot a different kernel. "linux-lts" is probably a good choice for this.

SirBlobman · 2021-03-16 20:55:22

Morn wrote:

Good, then it looks like it's not the Ryzen bug nor the GPU causing issues.
Another simple experiment would be to install and boot a different kernel. "linux-lts" is probably a good choice for this.

Thanks for the suggestion, I'll try changing the kernel after the system freezes again. Good to know its not my CPU or GPU.

Morn · 2021-03-16 21:05:13

Maybe also unplug your USB devices when not in use. You seem to have some kind of no name gaming keyboard. Badly programmed USB devices can sometimes cause weird problems.

SirBlobman · 2021-03-16 21:08:30

Morn wrote:

Maybe also unplug your USB devices when not in use. You seem to have some kind of no name gaming keyboard. Badly programmed USB devices can sometimes cause weird problems.

I doubt its my keyboard, I've had it for many years and the only reason the keyboard is attached is because the machine locked up the first time and I wanted to check my BIOS settings.

Morn · 2021-03-16 21:31:03

With USB, sometimes it's also the USB hub in the machine misbehaving, not just the device itself. But if that keyboard (or other USB peripherals) was not plugged in when the machine locked up, I guess we can rule that one out.

It's always a Sherlockian process of elimination when investigating these mystery lockups because the log files never seem to have anything helpful in them. And if the machine only locks up after two days, that really slows this process down.

SirBlobman · 2021-03-16 23:16:39

Morn wrote:

With USB, sometimes it's also the USB hub in the machine misbehaving, not just the device itself. But if that keyboard (or other USB peripherals) was not plugged in when the machine locked up, I guess we can rule that one out.
It's always a Sherlockian process of elimination when investigating these mystery lockups because the log files never seem to have anything helpful in them. And if the machine only locks up after two days, that really slows this process down.

The machine did end up freezing again. Just as an experiment I have now installed the lts kernel and disconnected all USB devices. I'll report back in 48 hours or sooner if it freezes again.

Morn · 2021-03-17 00:50:31

By the way, in those 2017 Ryzen CPUs there were actually two major Linux bugs: a segfault when compiling and that freezing bug. So if your CPU is really that old, maybe this is a CPU issue after all, even if there is no hardware error in the logs. AMD had a free replacement program for those early Ryzen chips, I don't know if that still exists.

I have a Ryzen 7 1700 from 2017. Initially I experienced lockups but the kernel parameter (and perhaps messing with the BIOS settings) fixed it. So from my perspective, switching to AMD for both the CPU and GPU has been a big success so far. I hope there will be a solution for your machine too.

drcouzelis · 2021-03-17 01:09:56

Morn wrote:

By the way, in those 2017 Ryzen CPUs there were actually two major Linux bugs: a segfault when compiling and that freezing bug.

Thank you for the reminder... I had a 1st gen 2017 Ryzen 5 that I remember sending to AMD for a free replacement, but I also remember I have to go into BIOS and disable C-States (or restrict them or something?). I couldn't remember what I mailed it in for, but now I remember the compile bug.

I have no regrets with my purchase though.

EDIT: https://en.wikipedia.org/wiki/Ryzen#Known_issues

Last edited by drcouzelis (2021-03-17 01:15:31)

inout · 2021-03-17 01:38:54

SirBlobman wrote:

CPU: AMD Ryzen 3 1200

a BIOS update is required for Ryzen 3 12xx and Ryzen 3 22xx.

euromatlox · 2021-03-17 05:19:21

Since console only, perhaps you could try a passive (no fan) GPU. Something you can find cheap (almost free), and not too many watts power consumption (AMD Radeon R5 232 is 19 watts).
As a non-gamer I have always preferred passive silent GPUs.

Last edited by euromatlox (2021-03-17 05:29:32)

Morn · 2021-03-17 16:35:03

drcouzelis wrote:

Morn wrote:
By the way, in those 2017 Ryzen CPUs there were actually two major Linux bugs: a segfault when compiling and that freezing bug.
Thank you for the reminder... I had a 1st gen 2017 Ryzen 5 that I remember sending to AMD for a free replacement, but I also remember I have to go into BIOS and disable C-States (or restrict them or something?). I couldn't remember what I mailed it in for, but now I remember the compile bug.
I have no regrets with my purchase though.
EDIT: https://en.wikipedia.org/wiki/Ryzen#Known_issues

And it's already the second time I have switched to AMD now. History is repeating itself. I can be such a weather vane sometimes:

1995: Intel Pentium 133 ("Hasta la vista, Amiga! Games will be so much better in SVGA.")
1999: AMD K6-III 450 ("Yay, 3-D graphics card. Games will be even better now.")
2002: AMD Athlon XP 2100+ ("Goodbye Linux, hello FreeBSD. And what is this Windows thing you speak of?")
2012: Intel Pentium G630 ("Do I even need a PC anymore? Maybe I can go cheap this time around...")
2017: AMD Ryzen 7 1700 ("OMFG, 16 threads. I need this! Going cheap was so stupid.")

SirBlobman · 2021-03-17 19:45:33

inout wrote:

a BIOS update is required for Ryzen 3 12xx and Ryzen 3 22xx.

I don't think this board has any more BIOS updates. The Ryzen CPU was working previously without issues on a Windows machine with the same motherboard.

euromatlox wrote:

Since console only, perhaps you could try a passive (no fan) GPU. Something you can find cheap (almost free), and not too many watts power consumption (AMD Radeon R5 232 is 19 watts).
As a non-gamer I have always preferred passive silent GPUs.

This GPU doesn't actually have a fan, and I don't have any lower power GPUs available (The others in my house have full fans and are being used by other computers)

Current Uptime:
I'll post an update to this post if it locks up before 48 hours.

17h 25m 35s

Last edited by SirBlobman (2021-03-17 20:00:28)

Maniaxx · 2021-03-17 20:06:20

If it locks up you could try to setup kdump to get a backtrace.
https://bbs.archlinux.org/viewtopic.php … 2#p1961662

SirBlobman · 2021-03-18 04:59:26

Changing kernel to lts and unplugging the USB keyboard did not seem to fix the issue, I'll look into setting up the kdump backtrace.

Updated Journal:
Output of "journalctl -b -1" created after force reboot from frozen state.
https://pastebin.com/DG48pS1S

Total uptime was around 25 hours.

Last edited by SirBlobman (2021-03-18 05:01:42)

Morn · 2021-03-19 13:42:31

SirBlobman wrote:

Changing kernel to lts and unplugging the USB keyboard did not seem to fix the issue, I'll look into setting up the kdump backtrace.

You know, FreeBSD is actually pretty nice on a server, so you might give that a try. It's good enough for Netflix. If you cannot change the hardware, change the software.

I don't think Arch is such a great choice for a server anyway, too much breakage. Acceptable for a desktop OS, but on a server I prefer Debian/Raspbian/Raspberry Pi OS and FreeBSD.

SirBlobman · 2021-03-21 02:08:20

inout wrote:

a BIOS update is required for Ryzen 3 12xx and Ryzen 3 22xx.

Strange how things work out, I updated my Bios from a 2017 version to a 2020 version and now everything seems to be fine.

Morn · 2021-03-21 15:43:17

As I always say, "BIOS updates are like a box of chocolates. You never know what you're gonna get." Anyway, congrats on finally having a usable and stable system!

Arch Linux

#1 2021-03-16 18:46:23

[SOLVED] Machine locks up after 16-48 hours.

#2 2021-03-16 20:29:04

Re: [SOLVED] Machine locks up after 16-48 hours.

#3 2021-03-16 20:32:58

Re: [SOLVED] Machine locks up after 16-48 hours.

#4 2021-03-16 20:47:12

Re: [SOLVED] Machine locks up after 16-48 hours.

#5 2021-03-16 20:55:22

Re: [SOLVED] Machine locks up after 16-48 hours.

#6 2021-03-16 21:05:13

Re: [SOLVED] Machine locks up after 16-48 hours.

#7 2021-03-16 21:08:30

Re: [SOLVED] Machine locks up after 16-48 hours.

#8 2021-03-16 21:31:03

Re: [SOLVED] Machine locks up after 16-48 hours.

#9 2021-03-16 23:16:39

Re: [SOLVED] Machine locks up after 16-48 hours.

#10 2021-03-17 00:50:31

Re: [SOLVED] Machine locks up after 16-48 hours.

#11 2021-03-17 01:09:56

Re: [SOLVED] Machine locks up after 16-48 hours.

#12 2021-03-17 01:38:54

Re: [SOLVED] Machine locks up after 16-48 hours.

#13 2021-03-17 05:19:21

Re: [SOLVED] Machine locks up after 16-48 hours.

#14 2021-03-17 16:35:03

Re: [SOLVED] Machine locks up after 16-48 hours.

#15 2021-03-17 19:45:33

Re: [SOLVED] Machine locks up after 16-48 hours.

#16 2021-03-17 20:06:20

Re: [SOLVED] Machine locks up after 16-48 hours.

#17 2021-03-18 04:59:26

Re: [SOLVED] Machine locks up after 16-48 hours.

#18 2021-03-19 13:42:31

Re: [SOLVED] Machine locks up after 16-48 hours.

#19 2021-03-21 02:08:20

Re: [SOLVED] Machine locks up after 16-48 hours.

#20 2021-03-21 15:43:17

Re: [SOLVED] Machine locks up after 16-48 hours.

Board footer