Random reboots with green screen and MCE events

agapito · 2024-04-25 09:55:45

noisypiano wrote:

I don't understand how do you assign some light task to a specific core and reduce 2 points less on a certain curve specifically to make a crash happen identical to mines. All I know is my setup wasn't laggy when I was using regular 6.6 kernel, and it was not laggy when I was using 6.8.7 kernel either.

I repeat again what i wrote some weeks ago:

The only way to solve this problem (your problem) is by individually calibrating each of the cores with the help of the CoreCycler program and Curve Optimize or if you don't want to waste too much time testing just add a +10 to all the cores in the curve, but I don't recommend it if you want to get the maximum performance out of the CPU.

Some people claim that deactivating the C-States has solved the problem, but this is not entirely true. What happens is that when a core does not shut down completely, it does not need as much voltage when it wakes up. For example, a core that with the C-States deactivated is 100% stable with a -20 on the curve, may need a -12 when you activate the C-States. Even a core that passes a lot of hours the Core Cycler test successfully could still fail, so I recommend adding 3 or 4 more points for safety and to anticipate future degradation of the CPU. Taking the above example I would set a -17 or -9 to that core, depending on whether or not the C-States are activated.

There are a lot of tutorials there about how Curve Optimizer works. If you want to make sure that your PC is 100% stable and reliable, my advice is to use CoreCycler in Windows 10 safe mode and make sure that each core individually passes that test for 10 hours without error. Once this is achieved, you add 2 or 3 more points to each core, for safety. All the Zen3 CPU's I've had, restarted automatically; after doing what I said, I haven't seen a spontaneous restart in 2 years.

noisypiano · 2024-04-25 18:07:20

agapito wrote:

noisypiano wrote:
I don't understand how do you assign some light task to a specific core and reduce 2 points less on a certain curve specifically to make a crash happen identical to mines. All I know is my setup wasn't laggy when I was using regular 6.6 kernel, and it was not laggy when I was using 6.8.7 kernel either.
I repeat again what i wrote some weeks ago:
The only way to solve this problem (your problem) is by individually calibrating each of the cores with the help of the CoreCycler program and Curve Optimize or if you don't want to waste too much time testing just add a +10 to all the cores in the curve, but I don't recommend it if you want to get the maximum performance out of the CPU.
Some people claim that deactivating the C-States has solved the problem, but this is not entirely true. What happens is that when a core does not shut down completely, it does not need as much voltage when it wakes up. For example, a core that with the C-States deactivated is 100% stable with a -20 on the curve, may need a -12 when you activate the C-States. Even a core that passes a lot of hours the Core Cycler test successfully could still fail, so I recommend adding 3 or 4 more points for safety and to anticipate future degradation of the CPU. Taking the above example I would set a -17 or -9 to that core, depending on whether or not the C-States are activated.

There are a lot of tutorials there about how Curve Optimizer works. If you want to make sure that your PC is 100% stable and reliable, my advice is to use CoreCycler in Windows 10 safe mode and make sure that each core individually passes that test for 10 hours without error. Once this is achieved, you add 2 or 3 more points to each core, for safety. All the Zen3 CPU's I've had, restarted automatically; after doing what I said, I haven't seen a spontaneous restart in 2 years.

I don't have Curve Optimizer in my BIOS. I don't even have Precision Boost Overdrive.
[1] https://www.reddit.com/r/gigabyte/comme … t/kplabvo/
[2] https://www.reddit.com/r/gigabyte/comme … t/kyo752f/

But you can increase the CPU voltage offset (recommended in ArchWiki) in the BIOS by first changing VCORE SOC > Normal which makes the Dynamic VCORE SOC(DVID) available, and then you can set +0.054V offset.
bios settings
ArchWiki said you should add 4 points to the offset, which I believe +0.04V. I had add 5 points but still got random reboots. Am I doing anything wrong? Or am I not changing the correct settings? Please let me know. If what I am doing is correct then I don't think there is a reason for me to do anything with voltage anymore. Also since I don't have the settings in BIOS, it would be a pain in the ass to setup in userspace. Also seth, if you know/understand anything please reply.

With all that said I ran this machine with the LTS kernel for 8 hours and 46 minutes without a crash and then the electricity was gone. I don't know if anything will happen but it seems good so far. The only problem I had was occasional mouse lag and scrolling lag in websites. But it was way too less.

Last edited by noisypiano (2024-05-02 09:32:31)

agapito · 2024-04-25 19:00:33

seth wrote:

https://wiki.archlinux.org/title/Ryzen#Random_reboots
use the AMD curve optimiser which is accessible via your motherboard's bios. Access it and put a positive offset of 4 points, which will increase the voltage your CPU is getting at higher loads
https://stackoverflow.com/questions/339 … cess-linux
https://man.archlinux.org/man/core/util … skset.1.en
But I cannot quantify "light", maybe just "top"?

Yes, i was using taskset for that purpose. A light load means a load that does not consume much CPU consumption, in my particular case it was 15% of that particular core. Being a light load, the CPU frequency was raised to the maximum possible as the temperature was low, causing a random reboot and showing that MC5_ERROR, because the voltage for that high frequency was wrong. That is the purpose of the Curve Optimizer to correct incorrect factory voltage values.

noisypiano wrote:

I don't have Curve Optimizer in my BIOS. I don't even have Precision Boost Overdrive.

Oh, I didn´t know that. It sucks.

Anyway, according to that screenshot you ARE NOT increasing the CPU voltage, that's the VSOC voltage that you should set on auto or you can even lower the voltage if you want. You only want to increase the VSOC voltage if you plan to overclock the Infinity Fabric. Since this is not the case, you can even subtract voltage from it safely.

You can probably raise the voltage in the CPU/VRM section. If you can set a positive offset, start with 1 step and if it still crashes, try 2.

Another option, if your motherboard allows it, would be to apply a manual voltage (carefully, and not higher than 1,15) and a fixed frequency, which would end the problem once the appropriate values for your CPU have been found.

noisypiano · 2024-04-25 20:09:51

agapito wrote:

You can probably raise the voltage in the CPU/VRM section. If you can set a positive offset, start with 1 step and if it still crashes, try 2.
Another option, if your motherboard allows it, would be to apply a manual voltage (carefully, and not higher than 1,15) and a fixed frequency, which would end the problem once the appropriate values for your CPU have been found.

These are the options I get in the CPU/VRM options. No CPU overclock things.
bios settings

From Tweaker > Advanced CPU Settings.
bios settings

These two images are of Settings > AMD Overclocking. Is it possible to do anything from here?
bios settings

Last edited by noisypiano (2024-04-26 15:34:23)

agapito · 2024-04-26 05:35:02

noisypiano wrote:

These two images are of Settings > AMD Overclocking. Is it possible to do anything from here?

No.

As far as I can see, the only option that can help you is to deactivate Global C-States. Disabling Core Performance Boost could help too, but it is not recommended since the cores would not increase in frequency and performance would not be optimal. Anyway, it seems strange to me that you can't set a manual voltage and a fixed frequency either.

seth · 2024-04-26 07:38:07

There's a lot of stuff on auto in the "tweaker" page what might unlock additional features?
Assuming the LTS kernel is stable and rolling w/ the theory that the origin/trigger of the CPU undervoltage is the GPU, you could try to disable its power saving mechanisms not not cause spikes when it steps up.

amdgpu.sg_display=0 amdgpu.ppfeaturemask=0xffffbffd amdgpu.runpm=0 amdgpu.bapm=0 amdgpu.aspm=0 pcie_aspm=off

For a curveball and assuming (we haven't seen amy journal, have we?) you've an nvme, add to that

iommu=soft nvme_core.default_ps_max_latency_us=0

noisypiano · 2024-04-26 15:31:09

agapito wrote:

Anyway, it seems strange to me that you can't set a manual voltage and a fixed frequency either.

It is not possible from what I can understand by reading other people's discussion on the internet.

seth wrote:

There's a lot of stuff on auto in the "tweaker" page what might unlock additional features?

No they don't unlock anything related to overclocking. Cruve Optimizer and Precision Boost Overdrive should be stated as is in the BIOS. It's not available for my BIOS.

seth wrote:

Assuming the LTS kernel is stable and rolling w/ the theory that the origin/trigger of the CPU undervoltage is the GPU ...For a curveball and assuming (we haven't seen amy journal, have we?) you've an nvme, add to that

Yeah I have a NVME M.2 SSD. But how this could be an issue to the CPU? Also I gotta append those whole big kernel parameters?

However after using it with the LTS kernel, I haven't had any crashes. I was not able to test it for longer period of time (more uptime) for some reason, but unlike before, it is not happening for 2 days. The only problem I have right now is occasional mouse lag which is happening since the day I started using the LTS kernel (6.6.28-1-lts). It happens way too less and not bothersome. The scrolling lag is not present for a long time.

Last edited by noisypiano (Today 00:24:53)

seth · 2024-04-26 16:49:02

Also I gotta append those whole big kernel parameters?

For testing purposes to see whether this has any impact on the situation. Yes. You'd just throw all parameters at the 6.8 kernel and if that actually helps, try to figure which one is critical which should narrow down the trigger.

The LTS kernel is a good stopgap, but eventually some 6.8 or 6.9 kernel will become the next LTS kernel and you probably want to get this under control before that happens.

agapito · 2024-04-26 19:41:22

If the problem were GPU related, disabling C-States would have no relevance, and from what the OP says, the system is more stable with them disabled.

Being stable with LTS kernel is only circumstantial and does not mean that the cause of the reboots is a kernel bug. Linux 6.8 introduced the preferred core method which distributes the tasks in the cores differently from the previous kernels. Even if your PC does not reboot, it is still unstable and prone to errors.

The real root cause of all this is that AMD continues to sell broken CPUs.

noisypiano · 2024-04-26 19:46:39

seth wrote:

For testing purposes to see whether this has any impact on the situation. Yes. You'd just throw all parameters at the 6.8 kernel and if that actually helps, try to figure which one is critical which should narrow down the trigger.
The LTS kernel is a good stopgap, but eventually some 6.8 or 6.9 kernel will become the next LTS kernel and you probably want to get this under control before that happens.

Alright I will append those kernel parameters later. However as I have said I have no problems so I am gonna test with this kernel alone first. I actually haven't cross a big uptime with this kernel yet.

noisypiano · 2024-04-26 20:05:28

agapito wrote:

Linux 6.8 introduced the preferred core method which distributes the tasks in the cores differently from the previous kernels. Even if your PC does not reboot, it is still unstable and prone to errors.

If this is the case,

agapito wrote:

The real root cause of all this is that AMD continues to sell broken CPUs.

I don't understand how this is valid. You can find just as much problems if you search for "intel idle freeze". Internet is flooded with this type of issues and most people recommend disabling C-States and/or increasing voltage.

seth · 2024-04-26 21:01:31

"AMD continues to sell broken CPUs" does not induce that intel doesn't…

https://wiki.archlinux.org/title/Intel_ … Intel_CPUs
https://www.radgametools.com/oodleintel.htm

Due to what seem to be [o]overly optimistic BIOS settings[/o], some small percentage of processors go out of their functional range of clock rate and power draw under high load, and execute instructions incorrectly.

And mind you, agapito's instructions boil down to "configure your BIOS less optimistically" - this is a dynamically interdependent system and even though I very much agree that what breaks away here is the CPU, I'm not ruling in or out what element might be causing this, because everyhing attached to the board influencs the power supply to everyhing else attached the board.

@agapito

Linux 6.8 introduced the preferred core method

https://www.phoronix.com/news/AMD-P-Sta … ed-Core-13 ?
Does amd_pstate=disable/passive/active/guided have discernible impact on your situation?
You had those issues very reliably before, did you?

noisypiano · 2024-04-26 21:44:58

Alright I just had another green screen with the LTS kernel.

[    2.696731] mce: [Hardware Error]: Machine check events logged
[    2.696732] [Hardware Error]: System Fatal error.
[    2.696739] [Hardware Error]: CPU:1 (19:21:2) MC5_STATUS[-|UE|MiscV|AddrV|PCC|TCC|SyndV|-|-|-]: 0xbea0000001000108
[    2.696752] [Hardware Error]: Error Addr: 0x00ffffffc03cc10a
[    2.696757] [Hardware Error]: IPID: 0x000500b000000000, Syndrome: 0x000000004d000000
[    2.696763] [Hardware Error]: Execution Unit Ext. Error Code: 0
[    2.696764] [Hardware Error]: cache level: RESV, tx: GEN, mem-tx: GEN

I'm pretty much sure at this point that the issue is related to the CPU. Since I had used the regular version of 6.6 kernel before without any issue. A lot of other people had this issue as well and they also think this is related to the CPU. The main thing differs my problem from others is that I have a green screen. I highly doubt that my GPU is defect since this build is only half a year old and I had run benchmarks, played games (both on Windows and Linux) without any problems before. I had run benchmark after having this issue without a problem as well (I ran Unigine Superposition 1080p extreme 3 times back to back without any break). I have also tried disabling C-state and enabling Typical current idle which all failed. I cannot increase the CPU voltage because my BIOS unfortunately does not support this. The final and only option remains is installing Windows and waiting for something awkward to happen, if it does then I have to claim warranty. After all I think I should try other options that are recommended by seth but at this point I am really frustrated and tired. This problem is very hard to detect and if I change a setting, I don't know after how long it will appear again. This time with the LTS kernel it took the machine 2 days for the crash to happen. It took 11 hours for the previous crash. Really frustrated. Nonetheless, at this point I really doubt it is the linux kernel because as I keep saying from post to post, I have been using linux the way I want for 3/4 months without any issue and then it appeared suddenly, and after thinking it is maybe the kernel, when I changed back to a LTS kernel, the crash happened again. The LTS kernel I used was 6.6.28-1-lts. I have using linux on this machine since 6.6 this never happened.

noisypiano · 2024-04-27 00:40:20

I have gathered two journalctl logs.

[1] Journalctl after the crash
[2] Journalctl of the boot where the crash happened

Last edited by noisypiano (2024-04-27 00:41:26)

noisypiano · 2024-04-27 03:05:41

I just had another green screen again while using 6.8.7 kernel. I checked the journal of the previous boot (where the crash happened). Everything was normal but like in my previous post if you look at the bottom of the 2nd log, you can see two kernel messages, I got the same type of message this time as well but the last line now is different.

Apr 27 08:57:43 kernel: [drm] Fence fallback timer expired on ring gfx_0.0.0
Apr 27 08:57:46 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 4157600149 wd_nsec: 4157598940

Last edited by noisypiano (2024-04-27 03:06:42)

noisypiano · 2024-04-27 03:27:02

Just got a another. Happens so fast in 6.8.7 kernel. This time, at the end of the journalctl log from the boot where this crash happened returned one kernel message

Apr 27 09:21:22 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1983437717 wd_nsec: 1983436442

agapito · 2024-04-27 05:52:01

agapito wrote:

I don't understand how this is valid. You can find just as much problems if you search for "intel idle freeze". Internet is flooded with this type of issues and most people recommend disabling C-States and/or increasing voltage.

We are talking about AMD CPUs here, what Intel does is not relevant. I've had 5 different Ryzen CPUs on my motherboard, so I know what I'm talking about.

seth wrote:

@agapito
Linux 6.8 introduced the preferred core method
https://www.phoronix.com/news/AMD-P-Sta … ed-Core-13 ?
Does amd_pstate=disable/passive/active/guided have discernible impact on your situation?
You had those issues very reliably before, did you?

I was wrong here. As you can read in the article, this feature will arrive in kernel 6.9 which I am using right now, hence the confusion.

@noisypiano You can try the different amd_pstate modes, but I repeat what I said before: any progress will be circumstantial, your CPU is still broken.

noisypiano wrote:

I'm pretty much sure at this point that the issue is related to the CPU.

That has been clear from the moment I told you that on my own machine I can reproduce that MC5_STATUS error if I modify the voltage to a particular core.

noisypiano wrote:

I highly doubt that my GPU is defect since this build is only half a year old and I had run benchmarks, played games (both on Windows and Linux) without any problems before. I had run benchmark after having this issue without a problem as well (I ran Unigine Superposition 1080p extreme 3 times back to back without any break).

Like CPUs, GPUs can be 100% stable under stress, but crash under idle conditions. Have you overclocked or undervolted your GPU?

noisypiano wrote:

The final and only option remains is installing Windows and waiting for something awkward to happen

Don´t waste your time, it will crash too. Do you remember if the problems started when you flashed the last bios?

noisypiano wrote:

if it does then I have to claim warranty.

It is your only option, although they will probably send you another CPU with the same symptoms.

Have you considered purchasing a higher quality motherboard? This would allow you to fix your current CPU and in the future you can upgrade to the most powerful CPU (5950x) for that socket.

noisypiano · 2024-04-27 06:54:10

agapito wrote:

We are talking about AMD CPUs here, what Intel does is not relevant. I've had 5 different Ryzen CPUs on my motherboard, so I know what I'm talking about.

It is because it indirectly elevates intel's position.

agapito wrote:

@noisypiano You can try the different amd_pstate modes

It's not effective. I had tried disabling amd_pstate, it doesn't help. I have been using the amd_pstate since day 1, had no problems.

agapito wrote:

Like CPUs, GPUs can be 100% stable under stress, but crash under idle conditions. Have you overclocked or undervolted your GPU?

Linux doesn't allow controls over the GPU I have. I couldn't even set a custom fan curve.

agapito wrote:

Have you considered purchasing a higher quality motherboard? This would allow you to fix your current CPU and in the future you can upgrade to the most powerful CPU (5950x) for that socket.

I don't see myself buying anything in the future for some time.

noisypiano · 2024-04-27 07:14:12

I have found a tool called Zenstates that can modify the CPU voltage from userspace. I have added +0.05V to all the P-States. This url explains how. Since I had already done something wrong before, I want anybody who understands to verify that I am doing the correct thing.

The default settings from zenstates.py was,

P0 - Enabled - FID = 8C - DID = 8 - VID = 48 - Ratio = 35.00 - vCore = 1.10000
P1 - Enabled - FID = 8C - DID = A - VID = 58 - Ratio = 28.00 - vCore = 1.00000
P2 - Enabled - FID = 84 - DID = C - VID = 68 - Ratio = 22.00 - vCore = 0.90000

Then I added +0.05V to the vCore and the result right now is

P0 - Enabled - FID = 8C - DID = 8 - VID = 40 - Ratio = 35.00 - vCore = 1.15000
P1 - Enabled - FID = 8C - DID = A - VID = 50 - Ratio = 28.00 - vCore = 1.05000
P2 - Enabled - FID = 84 - DID = C - VID = 60 - Ratio = 22.00 - vCore = 0.95000

How did I do it? Well to do this you need to know about VID hex codes.

sudo ./zenstates.py -v 40 -p 0

Here -v parameter says what hex code you want to put. I put 40. That means 1.15000V. These hex codes represent different vCore values for your processor. Note that the hex codes of processors might vary, but as long as you are using a Ryzen processor, it won't be that much different. For example, if you use 40 as hex code, instead of 1.15000V, you might be 1.05000V, or 1.25000V. At least that is what happened to me. I didn't get a value which is very different from the one that I thought I would get. You can find different VID in this reddit post and the url I specified at the beginning of the post. For testing the situation, I have set C-state to Auto and Power supply idle control to Auto just like before when I had no problems.

Last edited by noisypiano (2024-04-27 08:21:30)

agapito · 2024-04-27 08:14:25

Since the processor seems stable in the max performance state, can you add +0,01v to the P1 and +0,02v to the P2 state?

EDIT: Wiki talks about Curve Optimizer steps that have nothing to do with this topic.

Last edited by agapito (2024-04-27 08:18:58)

noisypiano · 2024-04-27 08:16:24

I have a decent cooler and temps are not going any high (40C watching videos on youtube). But do I really have to? I am totally not an expert in this.

Last edited by noisypiano (2024-04-27 08:36:32)

agapito · 2024-04-27 08:24:24

noisypiano wrote:

I have a decent cooler and temps are not going any high (40C watching videos on youtube). But do I really have to? Because the archwiki says add 4 offset. I am totally not an expert in this. Please let me know if I'm doing the right thing.
agapito wrote:
Since the processor seems stable in other states, you only have to modify the lowest state.
I don't understand? How do you know it's better in other states?

I changed my answer, edit it in your message please.

You said that you still had hangs with the C-States off. C-States off = P2 off. So P1 is not totally stable either.

Try to add +0,01v to the P1 and +0,02v to the P2 state. If after that, it is still not stable, try +0,02v and 0,03v.

noisypiano · 2024-04-27 08:29:20

agapito wrote:

ou said that you still had hangs with the C-States off. C-States off = P2 off. So P1 is not totally stable either.
Try to add +0,01v to the P1 and +0,02v to the P2 state. If after that, it is still not stable, try +0,02v and 0,03v.

But what about P0? Shouldn't the voltage get changed in every place?
EDIT: Oh I get it, P0 is not where the problem happens. It happens when it enters the C-state so P1, P2 might be affected. But shouldn't it just have higher voltage for all the parts? I mean if you change it from the bios, that would happen, right?

agapito wrote:

EDIT: Wiki talks about Curve Optimizer steps that have nothing to do with this topic.

Then does it mean what I am doing here is obsolete? Is it of any good?

Last edited by noisypiano (2024-04-27 08:32:54)

agapito · 2024-04-27 08:54:26

noisypiano wrote:

I mean if you change it from the bios, that would happen, right?

noisypiano wrote:

Then does it mean what I am doing here is obsolete? Is it of any good?

Your motherboard does not allow a fixed voltage and does not allow the use of the Curve Optimizer, just forget that and focus on the way you have to vary the voltages and fix your problem

P0 is the maximum load state and does not usually give problems, you can test it with programs such as mprime.

noisypiano · 2024-04-27 09:14:42

Alright, this is the closest I could get to. It's hard to find exact increment of 0.1 and 0.2.

P0 - Enabled - FID = 8C - DID = 8 - VID = 48 - Ratio = 35.00 - vCore = 1.10000
P1 - Enabled - FID = 8C - DID = A - VID = 56 - Ratio = 28.00 - vCore = 1.01250
P2 - Enabled - FID = 84 - DID = C - VID = 64 - Ratio = 22.00 - vCore = 0.92500

Last edited by noisypiano (2024-04-27 09:15:10)

Arch Linux

#26 2024-04-25 09:55:45

Re: Random reboots with green screen and MCE events

#27 2024-04-25 18:07:20

Re: Random reboots with green screen and MCE events

#28 2024-04-25 19:00:33

Re: Random reboots with green screen and MCE events

#29 2024-04-25 20:09:51

Re: Random reboots with green screen and MCE events

#30 2024-04-26 05:35:02

Re: Random reboots with green screen and MCE events

#31 2024-04-26 07:38:07

Re: Random reboots with green screen and MCE events

#32 2024-04-26 15:31:09

Re: Random reboots with green screen and MCE events

#33 2024-04-26 16:49:02

Re: Random reboots with green screen and MCE events

#34 2024-04-26 19:41:22

Re: Random reboots with green screen and MCE events

#35 2024-04-26 19:46:39

Re: Random reboots with green screen and MCE events

#36 2024-04-26 20:05:28

Re: Random reboots with green screen and MCE events

#37 2024-04-26 21:01:31

Re: Random reboots with green screen and MCE events

#38 2024-04-26 21:44:58

Re: Random reboots with green screen and MCE events

#39 2024-04-27 00:40:20

Re: Random reboots with green screen and MCE events

#40 2024-04-27 03:05:41

Re: Random reboots with green screen and MCE events

#41 2024-04-27 03:27:02

Re: Random reboots with green screen and MCE events

#42 2024-04-27 05:52:01

Re: Random reboots with green screen and MCE events

#43 2024-04-27 06:54:10

Re: Random reboots with green screen and MCE events

#44 2024-04-27 07:14:12

Re: Random reboots with green screen and MCE events

#45 2024-04-27 08:14:25

Re: Random reboots with green screen and MCE events

#46 2024-04-27 08:16:24

Re: Random reboots with green screen and MCE events

#47 2024-04-27 08:24:24

Re: Random reboots with green screen and MCE events

#48 2024-04-27 08:29:20

Re: Random reboots with green screen and MCE events

#49 2024-04-27 08:54:26

Re: Random reboots with green screen and MCE events

#50 2024-04-27 09:14:42

Re: Random reboots with green screen and MCE events

Board footer