AMD Microcode failing with "Speculative Return Stack Overflow"

agapito · 2023-11-21 20:44:08

loqs wrote:

agapito wrote:
If you have a Threadripper CPU you just have to wait for the amd-ucode package containing the updated firmware from AMD.
https://bbs.archlinux.org/viewtopic.php … 6#p2119246
Has AMD not already done the Data Center updates for Speculative Return Stack Overflow? This is based on a comparison of [1] [2]. V1del referenced a 17h part which is not vulnerable to SRSO.
[1]: https://www.amd.com/en/resources/produc … -7005.html
[2]: https://bbs.archlinux.org/viewtopic.php … 5#p2117535

It's not relased yet, but this it is easy to understand: Threadripper and Epyc CPU's can update their firmware via the amd-ucode package. Ryzen desktop CPU's only using "platomav method" or by flashing a new bios containing the updated microcode.

loqs · 2023-11-21 21:10:14

agapito wrote:

loqs wrote:
agapito wrote:
If you have a Threadripper CPU you just have to wait for the amd-ucode package containing the updated firmware from AMD.
https://bbs.archlinux.org/viewtopic.php … 6#p2119246
Has AMD not already done the Data Center updates for Speculative Return Stack Overflow? This is based on a comparison of [1] [2]. V1del referenced a 17h part which is not vulnerable to SRSO.
[1]: https://www.amd.com/en/resources/produc … -7005.html
[2]: https://bbs.archlinux.org/viewtopic.php … 5#p2117535
It's not relased yet, but this it is easy to understand: Threadripper and Epyc CPU's can update their firmware via the amd-ucode package. Ryzen desktop CPU's only using "platomav method" or by flashing a new bios containing the updated microcode.

Do you agree that contradicts AMD's technical bulletin? Which AMD EPYC CPU has AMD not released a ucode update mitigating SRSO for?

agapito · 2023-11-21 22:09:29

loqs wrote:

Do you agree that contradicts AMD's technical bulletin? Which AMD EPYC CPU has AMD not released a ucode update mitigating SRSO for?

What do you mean? The microcode update that fixes Inception on Threadripper CPU's is not released yet, but when it is released it will appear a few days later in linux-firmware.git repo, then a refresh of amd-ucode/linux-firmware Arch's package will be enough for Threadripper CPU´s.

loqs · 2023-11-21 22:29:03

agapito wrote:

loqs wrote:
Do you agree that contradicts AMD's technical bulletin? Which AMD EPYC CPU has AMD not released a ucode update mitigating SRSO for?
What do you mean? The microcode update that fixes Inception on Threadripper CPU's is not released yet, but when it is released it will appear a few days later in linux-firmware.git repo, then a refresh of amd-ucode/linux-firmware Arch's package will be enough for Threadripper CPU´s.

What is the source of your information that there will a ucode update that fixes Inception on Threadripper?
Do you agree [1] only mentions µcode for Datacenter processors. Further that all those processor have already received that update and it is part of amd-ucode? The same document makes no mention of ucode updates for any other type of CPU. Again in that document under Workstation AMD Ryzen™ Threadripper™ PRO 5000WX Processors the AGESA™ Firmware entry is ChagallWSPI-sWRX8 1.0.0.7 (Target Dec 2023)?

[1]: https://www.amd.com/en/resources/produc … -7005.html

Last edited by loqs (2023-11-21 22:29:36)

agapito · 2023-11-21 22:58:05

loqs wrote:

agapito wrote:
loqs wrote:
Do you agree that contradicts AMD's technical bulletin? Which AMD EPYC CPU has AMD not released a ucode update mitigating SRSO for?
What do you mean? The microcode update that fixes Inception on Threadripper CPU's is not released yet, but when it is released it will appear a few days later in linux-firmware.git repo, then a refresh of amd-ucode/linux-firmware Arch's package will be enough for Threadripper CPU´s.
What is the source of your information that there will a ucode update that fixes Inception on Threadripper?
Do you agree [1] only mentions µcode for Datacenter processors. Further that all those processor have already received that update and it is part of amd-ucode? The same document makes no mention of ucode updates for any other type of CPU. Again in that document under Workstation AMD Ryzen™ Threadripper™ PRO 5000WX Processors the AGESA™ Firmware entry is ChagallWSPI-sWRX8 1.0.0.7 (Target Dec 2023)?
[1]: https://www.amd.com/en/resources/produc … -7005.html

Zen 3 and Zen 4 CPU's are affected by Inception and all of them need a microcode to solve it. AMD has only released the microcode update for Zen 3 and Zen 4 DESKTOP CPU's. The rest of Zen 3 and Zen 4 CPU's (Epyc and Threadripper) will receive the microcode in December.

amd-ucode package is useless for Zen 3 and Zen 4 DESKTOP CPU's; not for Threadripper as V1del proved. It doesn't matter what microcode you have in /lib/firmware/amd-ucode because it will only load the version that is in the bios, that's the reason you have to apply the "platomav method" for DESKTOP CPU's. If you can't update the bios because the motherboard vendor hasn't updated it with the latest AGESA that includes the microcode, then you have to use the "platomav method" like i did a month ago.

loqs · 2023-11-21 23:58:42

agapito wrote:

Zen 3 and Zen 4 CPU's are affected by Inception and all of them need a microcode to solve it. AMD has only released the microcode update for Zen 3 and Zen 4 DESKTOP CPU's. The rest of Zen 3 and Zen 4 CPU's (Epyc and Threadripper) will receive the microcode in December.

kernel/x86/microcode/AuthenticAMD.bin extracted from /boot/amd-ucode.img of amducode-20230804.7be2766d-2:

./amd_ucode_info.py kernel/x86/microcode/AuthenticAMD.bin | grep -F 'Family=0x19'
  Family=0x19 Model=0x11 Stepping=0x01: Patch=0x0a10113e Length=5568 bytes # Genoa B1: 0x0A10113E
  Family=0x19 Model=0x11 Stepping=0x02: Patch=0x0a10123e Length=5568 bytes # Genoa-X B2: 0x0A10123E
  Family=0x19 Model=0xa0 Stepping=0x02: Patch=0x0aa00212 Length=5568 bytes # Bergamo A2: 0x0AA00212
  Family=0x19 Model=0x01 Stepping=0x01: Patch=0x0a0011d1 Length=5568 bytes # Milan B1 – 0x0A0011CF or 0x0A0011D1
  Family=0x19 Model=0x01 Stepping=0x00: Patch=0x0a001079 Length=5568 bytes # Milan B0 – 0x0A001079
  Family=0x19 Model=0x01 Stepping=0x02: Patch=0x0a001234 Length=5568 bytes # Milan-X B2 – 0x0A001234
  Family=0x19 Model=0xa0 Stepping=0x01: Patch=0x0aa00116 Length=5568 bytes # Bergamo A1: 0x0AA00116

kernel/x86/microcode/AuthenticAMD.bin extracted from /boot/amd-ucode.img of amd-ucode-20231030.2b304bfe-1 which includes https://git.kernel.org/pub/scm/linux/ke … 6f2aaf77fa

./amd_ucode_info.py kernel/x86/microcode/AuthenticAMD.bin | grep -F 'Family=0x19'
  Family=0x19 Model=0x11 Stepping=0x02: Patch=0x0a101244 Length=5568 bytes # Updated past Genoa-X B2: 0x0A10123E
  Family=0x19 Model=0x01 Stepping=0x01: Patch=0x0a0011d1 Length=5568 bytes # Milan B1 – 0x0A0011CF or 0x0A0011D1
  Family=0x19 Model=0x01 Stepping=0x00: Patch=0x0a001079 Length=5568 bytes # Milan B0 – 0x0A001079
  Family=0x19 Model=0xa0 Stepping=0x02: Patch=0x0aa00213 Length=5568 bytes # Updated past Bergamo A2: 0x0AA00212
  Family=0x19 Model=0x01 Stepping=0x02: Patch=0x0a001234 Length=5568 bytes # Milan-X B2 – 0x0A001234
  Family=0x19 Model=0xa0 Stepping=0x01: Patch=0x0aa00116 Length=5568 bytes # Bergamo A1: 0x0AA00116
  Family=0x19 Model=0x11 Stepping=0x01: Patch=0x0a101144 Length=5568 bytes # Updated past Genoa B1: 0x0A10113E

As you have stated AMD have not released an Inception ucode update for Zen 3 and Zen 4 Epyc parts it can not be in the current linux-firmware and you are disputing all the patch identifiers in AMD's own security bulletin.

agapito wrote:

amd-ucode package is useless for Zen 3 and Zen 4 DESKTOP CPU's; not for Threadripper as V1del proved. It doesn't matter what microcode you have in /lib/firmware/amd-ucode because it will only load the version that is in the bios, that's the reason you have to apply the "platomav method" for DESKTOP CPU's. If you can't update the bios because the motherboard vendor hasn't updated it with the latest AGESA that includes the microcode, then you have to use the "platomav method" like i did a month ago.

As the linux packages in the repositories do not support late loading I agree nothing in /lib/firmware/amd-ucode provided by linux-firmware will be used provided. It should be noted ${pkgbase}/amd-ucode/microcode_amd*.bin are used by the
linux-firmware PKGBUILD in generating amd-ucode.

You also have not provided a source for your statement that there will a ucode update that fixes Inception on Threadripper

Last edited by loqs (2023-11-22 00:16:04)

gen2arch · 2023-11-25 15:13:26

@agapito:

What is meant when you say "microcode update that fixes Inception"? Isn't there already a mitigation in the form of the "spec_rstack_overflow=off/on" kernel parameter; by default it seems to be on. But this setting has a huge impact on the encryption/disk IO performance!

When I boot with spec_rstack_overflow=off, "cryptsetup benchmark" as well as the simple dd disk benchmark show significantly better results (between 35 and 50% higher!).

So, "fixing" Inception in my understanding would mean: this kernel parameter can be set "off" and the computer being nevertheless protected.

Is that, what the ucode update planned for december is expected to do? Frankly, I don't know, so this is an honest question.

In the meantime it still gets weirder. There has been an update to the amd-ucode package:

Version: 20231110.74158e7a-1

But what does this?

journalctl shows microcode patchlevel for our Threadripper Pro to be still the same:

[    2.184085] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    2.184086] Speculative Return Stack Overflow: Vulnerable: Safe RET, no microcode
[    2.760868] microcode: CPU1: patch_level=0x0a008205

Yet the mitigation seems to have vanished!
Before the last update I had

    Spec rstack overflow:  Mitigation; safe RET, no microcode

but after the latest update I get

  Spec rstack overflow:  Vulnerable: Safe RET, no microcode

Isn't that a regression, from "Mitigation" to "Vulnerable".

Can anyone explain whats going on here?

Thanks!

gen2arch

agapito · 2023-11-25 15:43:51

THIS IS FOR ZEN3 AND ZEN 4 DESKTOP CPU'S ONLY, NOT MOBILE, NOT EPYC, NOT THREADRIPPER.

It is your motherboard vendor providing a new BIOS with AMD AM4 AGESA Combo V2 PI 1.2.0.B?

YES: Flash it to be FULLY protected.
NO: Use the platomav method to be FULLY protected.

If you have a Threadripper CPU you will have to wait because the microcode is not released yet. When the microcode is available you will have to flash the new bios containing it, but UNLIKE DESKTOP CPU'S users, the amd-ucode package will also take care of that.

Read this for more info: https://docs.kernel.org/next/admin-guid … /srso.html

This is my last intervention in this thread because I am tired of repeating the same thing all the time.

loqs · 2023-11-25 16:27:33

gen2arch wrote:

@agapito:
What is meant when you say "microcode update that fixes Inception"? Isn't there already a mitigation in the form of the "spec_rstack_overflow=off/on" kernel parameter; by default it seems to be on. But this setting has a huge impact on the encryption/disk IO performance!
When I boot with spec_rstack_overflow=off, "cryptsetup benchmark" as well as the simple dd disk benchmark show significantly better results (between 35 and 50% higher!).
So, "fixing" Inception in my understanding would mean: this kernel parameter can be set "off" and the computer being nevertheless protected.

Correct. Without replacing the CPU there will be no fix as the issue is in the silicon. The kernel and ucode are mitigating the issue.

gen2arch wrote:

Is that, what the ucode update planned for december is expected to do? Frankly, I don't know, so this is an honest question.

AMD has announced there will be a firmware update ChagallWSPI-sWRX8 1.0.0.7 (Target December 2023) [1]. agapito has announced without providing any source that there will be a ucode update containing the fix for ThreadRipper. [2]

gen2arch wrote:

In the meantime it still gets weirder. There has been an update to the amd-ucode package:
Version: 20231110.74158e7a-1
But what does this?
journalctl shows microcode patchlevel for our Threadripper Pro to be still the same:
[    2.184085] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    2.184086] Speculative Return Stack Overflow: Vulnerable: Safe RET, no microcode
[    2.760868] microcode: CPU1: patch_level=0x0a008205
Yet the mitigation seems to have vanished!
Before the last update I had
    Spec rstack overflow:  Mitigation; safe RET, no microcode
but after the latest update I get
  Spec rstack overflow:  Vulnerable: Safe RET, no microcode
Isn't that a regression, from "Mitigation" to "Vulnerable".
Can anyone explain whats going on here?

The vulnerability reporting was changed to only report mitigated if both kernel and user space were protected [3][4]

[1]: https://www.amd.com/en/resources/produc … -7005.html
[2]: https://bbs.archlinux.org/profile.php?id=20636
[3]: https://git.kernel.org/pub/scm/linux/ke … c45f8d1bcf
[4]: https://docs.kernel.org/next/admin-guid … /srso.html

gen2arch · 2023-11-28 10:14:24

Thanks loqs and agapito, that was helpful!
With regard to the mentioned massive performance penalty AMD is expecting its customers to put up with, — a somewhat bleak outlook!
I try to report again, if and when BIOS/ucode updates are made available.

gen2arch

agapito · 2023-12-06 17:39:08

A new microcode has been uploaded for Zen 2 desktop CPU's: https://git.kernel.org/pub/scm/linux/ke … 1c2379bbb2

This is probably the microcode who fixes the Zenbleed vulnerability: https://www.amd.com/en/resources/produc … -7008.html

But as I have already mentioned in this post we will have to wait for a bios update that includes the ComboAM4v2PI_1.2.0.C firmware or use the platomav method when its repository is updated: https://github.com/platomav/CPUMicrocod … its/master

Last edited by agapito (2023-12-06 17:58:29)

Archttila · 2023-12-30 10:40:28

dobo wrote:

I've noticed similar behavior on my Zen 3 CPU. That's why I've done some digging and I have found out that there is a community project collecting microcodes for CPUs. It was even mentioned in the kernel mailing list. I've created amd-zen-ucode-platomav AUR package. You can try installing it with your favorite AUR helper, it will replace amd-ucode package. After reboot on my machine it works:
➜  sudo dmesg | grep -i microcode
[    0.764781] microcode: microcode updated early to new patch_level=0x0a50000f
...

thanks for AUR package! Works on my AMD Ryzen 5700G

journalctl -b
before
Dec 30 10:26:41 desktop kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied!

after
Dec 30 11:28:48 desktop kernel: Speculative Return Stack Overflow: Mitigation: Safe RET

Last edited by Archttila (2023-12-30 10:42:30)

mercysnack · 2024-02-20 10:30:56

Somebody here wouldn't happen to know what to do when the amd-ucodegen tool doesn't like one of the platomav microcode blobs? Like this one:

$ ./amd-ucodegen ../../Downloads/amd-r5-5560/cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin 
Bad processor ID 0x4573n

Sorry if this is too off-topic! This is the latest blob that seems to exist for the AMD R5 5560 CPUs, which seems to be one that AMD doesn't give to update via the official amd-ucode package for some reason. (It does baffle me that they just don't bother.)

Last edited by mercysnack (2024-02-20 10:31:31)

loqs · 2024-02-20 11:29:33

mercysnack wrote:

Somebody here wouldn't happen to know what to do when the amd-ucodegen tool doesn't like one of the platomav microcode blobs? Like this one:
$ ./amd-ucodegen ../../Downloads/amd-r5-5560/cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin 
Bad processor ID 0x4573n
Sorry if this is too off-topic! This is the latest blob that seems to exist for the AMD R5 5560 CPUs

Have you tried extracting from a firmware update provided by your mainboard vendor or another vendor for the same chipset?

mercysnack · 2024-02-20 12:27:38

I checked the code of amd-ucodegen's source code and this abort condition doesn't seem to depend on the machine it runs on. So I don't think it's a compatibility issue with my machine that's causing this error message. Nevertheless, there is currently no mainboard update so I had to rely on the blob in the one location everyone seems to get them from where others extracted them. But amd-ucodegen doesn't like that one, now I'm wondering how to get around that. If anyone has an idea, it would be appreciated!

Last edited by mercysnack (2024-02-20 12:28:13)

loqs · 2024-02-20 13:51:31

mercysnack wrote:

I checked the code of amd-ucodegen's source code and this abort condition doesn't seem to depend on the machine it runs on. So I don't think it's a compatibility issue with my machine that's causing this error message.

You misunderstood me. I was suggesting you obtain a different copy of the ucode. If there is no other source download it again.

$ amd-ucodegen cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin
CPU type 0xa50f00 [0xa500], file AMD/cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin
$ sha256sum AMD/cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin
7f364848ba3d97a719a7a13d188fdcb2b4d2effc1874e30ae4c7244d77d1b128  AMD/cpu00A50F00_ver0A50000F_2023-07-07_72B4B8C6.bin

mercysnack · 2024-02-20 15:20:47

You're right, I just downloaded it again and now it works fine lol. Somehow I never thought to do that. Thank you so much!!

Last edited by mercysnack (2024-02-20 15:22:01)

gcb · 2024-03-13 14:56:45

Not sure if this is better or worse, but recent bios update (available for my machine in fwup but installed via their tool) seems to have ucode for consumer cpus.

Mine changed from

archlinux kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied!
archlinux kernel: Speculative Return Stack Overflow: Mitigation: safe RET, no microcode

to

archlinux kernel: Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
archlinux kernel: Speculative Return Stack Overflow: Mitigation: Safe RET

Arch Linux

#26 2023-11-21 20:44:08

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#27 2023-11-21 21:10:14

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#28 2023-11-21 22:09:29

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#29 2023-11-21 22:29:03

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#30 2023-11-21 22:58:05

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#31 2023-11-21 23:58:42

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#32 2023-11-25 15:13:26

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#33 2023-11-25 15:43:51

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#34 2023-11-25 16:27:33

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#35 2023-11-28 10:14:24

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#36 2023-12-06 17:39:08

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#37 2023-12-30 10:40:28

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#38 2024-02-20 10:30:56

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#39 2024-02-20 11:29:33

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#40 2024-02-20 12:27:38

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#41 2024-02-20 13:51:31

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#42 2024-02-20 15:20:47

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#43 2024-03-13 14:56:45

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

Board footer