weird and consistent system lock-ups & VGA/GPU crashes.

Mrkd1904 · 2023-11-29 15:04:07

The VGA issue happens primarily with an Intel A750 (PCIe slot 1) but has also happened on integrated graphics and Nvidia cards as well.

The setup is:

ASUS Z790 ROG Maximus Hero
x2 (32gb) 16gb GSkill @ 6400 xmp
13900k
Water cooled
Plenty of ventilation
EVGA 800w platinum PSU
x3 Samsung 1tb M.2 NVMe ssd in a very non
-persistent md Sof-RAID5 config
x2 SKHynix 2tb M.2 NVMe in a PCIe add on card md Sof-RAID1 config
GPU's: Intel A750, Nvidia 4060, Nvidia 3060ti, Intel integrated on the 13900k - issue happens with all configs.

The issue is: Seemingly random, but consistently (think every 3-4 hours on average) my system will completely lock up to the point where I have to power cycle the machine which is savaging any volume I have mounted at that given time. I've now had four separate md arrays straight up disappear and almost the same amount of thin-pools become unrecoverable just over the past 72ish hours.

A *couple* times the system actually shook it off and kept going, which leads me to believe it's memory related but poking and prodding around with valgrind, perf and strace. Is proving fruitless. Running Memtest86 the board runs flawlessly. I have literally tried every configuration bootable in bios focusing around SA voltage and XMP. Issue persists regardless of configuration.

It seemingly happens more often than not after initiating a download such as git cloning a repo or right after a download. It also happens more often than not when the system starts to come under load (i.e make'ing a package or Paxman installing a package) but has happened at idle too.

It won't throttle or race like the last ASUS board I had but will lock up, and slowly creep higher in temp until I either hit the reset button; which sometimes will take 10-15 seconds to respond or have to force shutdown by holding the power button when that does not respond at all. Almost every time after that happens I'll notice the work I had been doing directly preceding is no long there, and about 1/15-20 times it'll be destructive; taking either the logical volume, or the md devices with it.

The GPU issue is similar, with the only difference really being that I lose display and the system is usually slightly hotter.

There are conveniently no journalctl or trace files left after rebooting and booting with kernel debug parameters hasn't really yielded anything either.

The build is just about 30 days old with another bad experience with an ASUS ROG STRIX WiFi Gaming E,
which was a dumpster fire through and through with the same issues, but more pronounced with the VGA issues and actual memory issues with one memtest yielding over 3k errors in the first run. Everything I've read about the issue really points back to the Intel ME and microcode/vulnerability mitigations the past 6-12 months particularly with XMP enabled and to a lesser degree hyperthreading and turboboost. This was OS agnostic and affected a large number of consumer SKU'S for ASUS.

With the latest BIOS (1501) those issues seem to be dissipating, at least for most.

I'm currently posting from my phone before going back in and rebuilding and restoring, again. Let me know which logs to pull and anything off the top of everyone's collective head's that would help because I'm at a loss.

Thanks in advance everyone.

Edit: forgot to mention - mce. I have to disable it via commandline at boot as it's so pronounced. But really that's and issue I've been seeing over the course of the past 6 months across different platforms and builds and Linux in general. But, without mce disabled most of these lockups would be preceeded by MCE log entries usually when the system would first come under load or at boot.

Last edited by Mrkd1904 (2023-11-29 15:11:36)

xerxes_ · 2023-11-29 17:57:17

MCE is Machine Check Exception - probably a faulty hardware, to be replaced. You can also try older microcode package from ALA: https://wiki.archlinux.org/title/Arch_Linux_Archive .
There were some instability problems with Samsung nvme. You can get them away from system and see if it helps.
You can also check your nvme devices by nvme-cli (https://wiki.archlinux.org/title/NVMe, man page) and if your system packages are ok: 'pacman -Qk'.

seth · 2023-11-29 20:19:29

There are conveniently no journalctl or trace files left after rebooting

where I have to power cycle the machine

https://wiki.archlinux.org/title/Keyboa … el_(SysRq)

which leads me to believe it's memory related

x2 (32gb) 16gb GSkill @ 6400 xmp

Disable XMP?

Gergo94 · 2023-11-29 20:34:56

Are you getting kernel panics? Even if the output gets disabled due to a kernel panic, it is indicated by a flashing caps lock light.

Kernel panic messages are not getting to the journal because the disks are not writable during then. You can capture them with Kdump (https://wiki.archlinux.org/title/Kdump). If reading them would help you then you may try setting it up.

A *couple* times the system actually shook it off and kept going, which leads me to believe it's memory related but poking and prodding around with valgrind, perf and strace. Is proving fruitless. Running Memtest86 the board runs flawlessly. I have literally tried every configuration bootable in bios focusing around SA voltage and XMP. Issue persists regardless of configuration.

x2 (32gb) 16gb GSkill @ 6400 xmp

A high XMP on DDR5 is a gamble, manufacturers think that they can reliably sell high memory overclock setups for money, but the harsh reality is that even if 95% of the times they work, there will be the rest of 5% of cases where the user is just getting a bad and unstable memory overclock with all the suffering it causes.

And DDR5 is horror to overclock with good stability:
- unlike DDR4, DDR5 RAM stick heat up significantly, and their stability is sensitive to the temperature - so an overclock stable at 40 C sticks may just crash when the sticks reach 50 C.
- even non-ECC DDR5 has a form of error correction - it is very good at hiding a slightly unstable overclock which will crash when the type of the memory workload changes to something untested

Thus a simple Memtest86 is useless testing DDR5 overclock / XMP. You need 12h+ of multiple memory stress tests passed without errors to declare a DDR5 memory clock stable.
Pick some from here: https://wiki.archlinux.org/title/Stress_testing

I would try saving the current BIOS setting, resetting BIOS and trying the system without enabling XMP.
If it works without XMP enabled - then yes is sucks that you were sold a non-working XMP profile with your memory kit - but that memory kit will still do some awesome speeds if you invest time to overclock it manually to speeds and timings comparable to its XMP profile.

Last edited by Gergo94 (2023-11-29 21:19:03)

Mrkd1904 · 2023-11-30 17:55:31

xerxes_ wrote:

MCE is Machine Check Exception - probably a faulty hardware, to be replaced. You can also try older microcode package from ALA: https://wiki.archlinux.org/title/Arch_Linux_Archive .
There were some instability problems with Samsung nvme. You can get them away from system and see if it helps.
You can also check your nvme devices by nvme-cli (https://wiki.archlinux.org/title/NVMe, man page) and if your system packages are ok: 'pacman -Qk'.

Regarding MCE: it's something I keep running into regardless of hardware as I've recently replaced everything aside from the processor which itself is only about six months old.

The frustrating part is that MCE seems to be deprecated and the only tools to help decode it are via the AUR (as far as I remember) and anything I could dredge up online was amorphous or fleeting in reasoning or explanaition. It's just something that's there with very little support or reasoning.

As per the NVMe disks everything looks vanilla. A little while back I had issue with the default features regarding ASPM changing on me at the disk firmware level which was strange. But as far as I can see the firmware is all up to date. I appreciate the reply and references though.

Mrkd1904 · 2023-11-30 17:58:12

seth wrote:

There are conveniently no journalctl or trace files left after rebooting
where I have to power cycle the machine
https://wiki.archlinux.org/title/Keyboa … el_(SysRq)
which leads me to believe it's memory related
x2 (32gb) 16gb GSkill @ 6400 xmp
Disable XMP?

Unfortunately it's not that easy (why would it be?) As it happens with or without XMP enabled. And with or without just about every other setting toggled on or off as well. Appreciate the reply though.

Mrkd1904 · 2023-11-30 18:07:20

Gergo94 wrote:

Are you getting kernel panics? Even if the output gets disabled due to a kernel panic, it is indicated by a flashing caps lock light.
Kernel panic messages are not getting to the journal because the disks are not writable during then. You can capture them with Kdump (https://wiki.archlinux.org/title/Kdump). If reading them would help you then you may try setting it up.
A *couple* times the system actually shook it off and kept going, which leads me to believe it's memory related but poking and prodding around with valgrind, perf and strace. Is proving fruitless. Running Memtest86 the board runs flawlessly. I have literally tried every configuration bootable in bios focusing around SA voltage and XMP. Issue persists regardless of configuration.
x2 (32gb) 16gb GSkill @ 6400 xmp
A high XMP on DDR5 is a gamble, manufacturers think that they can reliably sell high memory overclock setups for money, but the harsh reality is that even if 95% of the times they work, there will be the rest of 5% of cases where the user is just getting a bad and unstable memory overclock with all the suffering it causes.
And DDR5 is horror to overclock with good stability:
- unlike DDR4, DDR5 RAM stick heat up significantly, and their stability is sensitive to the temperature - so an overclock stable at 40 C sticks may just crash when the sticks reach 50 C.
- even non-ECC DDR5 has a form of error correction - it is very good at hiding a slightly unstable overclock which will crash when the type of the memory workload changes to something untested
Thus a simple Memtest86 is useless testing DDR5 overclock / XMP. You need 12h+ of multiple memory stress tests passed without errors to declare a DDR5 memory clock stable.
Pick some from here: https://wiki.archlinux.org/title/Stress_testing
I would try saving the current BIOS setting, resetting BIOS and trying the system without enabling XMP.
If it works without XMP enabled - then yes is sucks that you were sold a non-working XMP profile with your memory kit - but that memory kit will still do some awesome speeds if you invest time to overclock it manually to speeds and timings comparable to its XMP profile.

Good insight.

The DDR5 thing I've really been in denial about but sadly it's starting to become more and more of a reality. I keep telling myself there's a secret sauce somewhere. Just have to find it. Is this a product of the tech being still new? Or bad manufacturing? Or just inherently bad tech?? I'm curious to know.

And yes I am getting panics, as well as soft and RCU lockups. But preceding the hard assumed memory lockups or failures they're not preceeded by anything really. Kernel panics I honestly just take as the cost of doing business on Linux. Same with segfaults.

As per the testing I've literally tried almost all configurations at the bios level available. The trouble is that they're seemingly random in trigger and timing so I will think the systems GTG and then get smacked right when I start to get optimisitc.

Looping back to DDR5 - Is it just recommended to run at 4800 or whatever the bios default is set at?

And again good insight, thanks for the reply.

seth · 2023-11-30 18:27:16

Are you able to preserve the journal by rebooting w/ the magic sysrq?

Gergo94 · 2023-11-30 19:02:32

And yes I am getting panics, as well as soft and RCU lockups. But preceding the hard assumed memory lockups or failures they're not preceeded by anything really. Kernel panics I honestly just take as the cost of doing business on Linux. Same with segfaults.

Is the kernel backtrace of the soft and RCU lockups in dmesg/journal and the backtrace of panics totally inconsistent and different every time?

If data just randomly disappears in the hardware due to a hardware / hardware setup / power management issue, only then the backtraces should be random and changing every time.

Looping back to DDR5 - Is it just recommended to run at 4800 or whatever the bios default is set at?

Yes, until you manage to debug this I would recommend to leave it on BIOS default. A 6400 XMP is a significant overclock, and it is better not to complicate things with it until you resolve the problem.

You may try testing with only one of your two stick of RAM seated in the motherboard at a time if you suspect one of them to be faulty.
Also if the CPU lets it, then you can test a single stick in the RAM socket of both (either A or B) memory channels.

Mrkd1904 · 2023-11-30 19:46:10

Gergo94 wrote:

And yes I am getting panics, as well as soft and RCU lockups. But preceding the hard assumed memory lockups or failures they're not preceeded by anything really. Kernel panics I honestly just take as the cost of doing business on Linux. Same with segfaults.
Is the kernel backtrace of the soft and RCU lockups in dmesg/journal and the backtrace of panics totally inconsistent and different every time?
If data just randomly disappears in the hardware due to a hardware / hardware setup / power management issue, only then the backtraces should be random and changing every time.
Looping back to DDR5 - Is it just recommended to run at 4800 or whatever the bios default is set at?
Yes, until you manage to debug this I would recommend to leave it on BIOS default. A 6400 XMP is a significant overclock, and it is better not to complicate things with it until you resolve the problem.
You may try testing with only one of your two stick of RAM seated in the motherboard at a time if you suspect one of them to be faulty.
Also if the CPU lets it, then you can test a single stick in the RAM socket of both (either A or B) memory channels.

Yea a lot of the time it's whatever process is running is or was tainted. Normally a process that would put the system under load such as make or mkinitcpio. Some of the panics are pretty nasty though. What I would consider a step above catastrophic in terms of failures. A lot of them are usually centered around glibc, libc.so.6, and less often QT* whenever running graphical.

The sticks I've validated on different machines as I learned the hard way earlier this year when I had a one out of a set of GSkill go bad on me. I've already ran all slots with both or one stick. Also, one the higher end ASUS boards you can manipulate which slots have power and which bank to use, etc. Manually i can get up to about 6000-6200 before i start to see any negative reactions. But when using just straight XMP it invites a slew of different seemingly random issues. But again, i want to reiterate that the issue with the GPU, as well as the hard lock up issues are not affected noticeably by memory configuration. As far as i can tell. Contrary the MCE I feel like it's software related at this point.

I had another install/md array delete itself a few hours ago. So am rebuilding. Again. But, next panic I get I'll post here as long as I'm able to.

Mrkd1904 · 2023-11-30 22:09:30

seth wrote:

Are you able to preserve the journal by rebooting w/ the magic sysrq?

I just learned this - but as it turns out the keyboard I have (Logitech MX Mechanical) does not have a sysrq key...

Edit: just tried rebuilding and after the build of the md arrays arrays partitioning, pacstrapping and then git cloning all the items I wanted I reboot to the entire array just not there.

So, I attempted to use the archinstall script with all three drives, which failed. The only errors it threw were after part probe stating that partitions 1-64 had been written, but unable to inform the kernel and then python dumped the install.

Tried again with just one disk and it threw the same error, but still made it through the install.

Debugging an issue with nouveau and dkms signing the Nvidia kernel and then I should be able to post logs.

All drives were cleaned with blkdiscard, and then nvme format, nvme sanitize, and nvme subsystem-reset before the last two attempts.

Edit2: first lockup at 'modprobe nouveau'. Everything is at default settings.

Edit3: apparently kernel headers weren't installed(????). Installed and rebooted to initrd not finding the disk. Unlocked and mounted in emergency mode. Booted into graphical. Ran 'mod probe' nouveau once more, and it locked up once more.

Last edited by Mrkd1904 (2023-11-30 23:55:37)

Gergo94 · 2023-11-30 23:14:15

I just learned this - but as it turns out the keyboard I have (Logitech MX Mechanical) does not have a sysrq key...

Alt+PrintScreen should work

Debugging an issue with nouveau and dkms signing the Nvidia

I don't use nvidia so someone might correct me, but IFAIK one can use either proprietary nvidia or nouveau. Never both. If you use proprietary nvidia then nouveau must be blacklisted so it never gets loaded.
https://wiki.archlinux.org/title/NVIDIA

GPU's: ..., Intel integrated on the 13900k

If I would really struggle with nvidia, then I would just install without the nvidia card in the PC, using the iGPU. And resolve the nvidia issue once I have the system working.

Mrkd1904 · 2023-11-30 23:48:34

Mrkd1904 wrote: