AMD Microcode failing with "Speculative Return Stack Overflow"

Jphillips · 2023-08-22 19:45:46

I'm trying to figure out how to get early microcode loading on a Ryzen 9 6950H, Lenovo Laptop.

Here's the journalctl message from boot:

$ journalctl -k --grep=microcode
Aug 21 13:18:11 eastarch kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied!
Aug 21 13:18:11 eastarch kernel: Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
Aug 21 13:18:11 eastarch kernel: microcode: CPU0: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU1: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU4: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU6: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU5: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU7: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU9: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU8: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU2: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU3: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU11: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU10: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU12: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU13: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU14: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: CPU15: patch_level=0x0a404102
Aug 21 13:18:11 eastarch kernel: microcode: Microcode Update Driver: v2.2.

What exactly is this Speculative Return Stack Overflow and/or the IBPB-extending microcode? Any thoughts on how to fix this problem?

loqs · 2023-08-22 19:50:16

https://docs.kernel.org/admin-guide/hw-vuln/srso.html https://www.amd.com/en/resources/produc … -7005.html
The microcode did not fail. The kernel did not fail it selected safe RET which does not need microcode support. Assuming you have the latest version of amd-ucode installed then the update for your CPU has not been released yet.

Last edited by loqs (2023-08-22 19:52:03)

Jphillips · 2023-08-23 09:36:03

Interesting, ok. I'm still not totally sure I get the safe RET but I'll take your word for it. Do you know if there's a way to verify that this is indeed the latest microcode level?

Last edited by Jphillips (2023-08-23 09:36:16)

loqs · 2023-08-23 11:12:08

Jphillips wrote:

Do you know if there's a way to verify that this is indeed the latest microcode level?

Possibly check against https://git.kernel.org/pub/scm/linux/ke … ode/README

Jphillips · 2023-08-23 11:57:52

Since I have the 19h family, it unfortunately appears not to be the latest. This is supported here, which shows my version is a year old, https://github.com/platomav/CPUMicrocod … master/AMD. All of the latest 19h series updates are from a month ago.

So again we're back to my microcode not updating.

Last edited by Jphillips (2023-08-23 11:58:40)

Scimmia · 2023-08-23 12:51:53

19h family, but is your specific model and stepping listed in that readme?

Jphillips · 2023-08-23 14:11:49

Is there a way to get this info from the terminal? cat /proc/cpuinfo doesn't give me hex numbers.

Scimmia · 2023-08-23 14:23:26

so convert decimal to hex

Jphillips · 2023-08-23 14:38:05

It doesn't work that way -- it's not a straight conversion, it's some cryptic internal AMD code. The closest I can find is this page but it's incomplete, https://en.wikichip.org/wiki/amd/cpuid# … _.2819h.29

Scimmia · 2023-08-23 14:40:34

No, it IS a straight conversion.

Jphillips · 2023-08-23 14:59:11

Turns out there's an easier way to get this info anyway for anyone interested:

$ journalctl -b | grep smpboot
Aug 23 13:44:40 eastarch kernel: smpboot: Allowing 16 CPUs, 0 hotplug CPUs
Aug 23 13:44:40 eastarch kernel: smpboot: CPU0: AMD Ryzen 9 PRO 6950H with Radeon Graphics (family: 0x19, model: 0x44, stepping: 0x1)
Aug 23 13:44:40 eastarch kernel: smpboot: Max logical packages: 1
Aug 23 13:44:40 eastarch kernel: smpboot: Total of 16 processors activated (105436.83 BogoMIPS)

So no, my model isn't listed. Thanks for the help!

loqs · 2023-08-23 15:00:51

Note in https://www.amd.com/en/resources/produc … -7005.html ucodes are mentioned only for data center parts with all other sections only listing AGESA firmware likely RembrandtPI-FP7_1.0.0.9a
(Target Aug 2023) for your CPU.

agapito · 2023-08-25 08:52:34

The microcode that solves Inception has been available since August 9: https://git.kernel.org/pub/scm/linux/ke … 11da1eef8f

The problem is AMD does not allow you to update the microcode manually on your desktop/laptop Ryzen CPU. The Arch's amd-ucode package is useless.

The only way to update the microcode is through a new bios provided by the motherboard or laptop manufacturer.

There seems to be an unofficial method to update it manually, but I won't advertise it here

V1del · 2023-09-06 18:39:21

FWIW a microcode update from amd-ucode is being applied for my AMD Ryzen Threadripper 3960X (family: 0x17, model: 0x31, stepping: 0x0), so a blanket statement that it can't be applied doesn't seem to be entirely correct. According to the readme there's at least one special condition for an AGESA version for family 19 and that has to come from the MB vendor

agapito · 2023-09-06 20:31:16

V1del wrote:

FWIW a microcode update from amd-ucode is being applied for my AMD Ryzen Threadripper 3960X (family: 0x17, model: 0x31, stepping: 0x0), so a blanket statement that it can't be applied doesn't seem to be entirely correct. According to the readme there's at least one special condition for an AGESA version for family 19 and that has to come from the MB vendor

To be clear, in my previous post I was talking about the desktop Ryzen processors that almost everyone has and not other server or workstation processors like Threadripper or Epyc.

For example, the firmware of the typical 2700X, 3700X or 5950X processors can NOT be upgraded by the operating system and only through a bios update.

dobo · 2023-09-18 08:08:00

I've noticed similar behavior on my Zen 3 CPU. That's why I've done some digging and I have found out that there is a community project collecting microcodes for CPUs. It was even mentioned in the kernel mailing list. I've created amd-zen-ucode-platomav AUR package. You can try installing it with your favorite AUR helper, it will replace amd-ucode package. After reboot on my machine it works:

➜  sudo dmesg | grep -i microcode
[    0.764781] microcode: microcode updated early to new patch_level=0x0a50000f
...

agapito · 2023-09-18 19:33:25

Thanks for the package dobo it works, what's your CPU?

As I compile my own kernel I preferred to use another method to update the firmware for my 5950X CPU.

I grabbed the firmware from here.
I used the amd-ucodegen utility to convert that firmware to microcode_amd_fam19h.bin
I moved microcode_amd_fam19h.bin (SHA256=1b0ed6b512655f08581ff46b277ad2c09a35f17c7848f18b7a68894900026304) to /lib/firmware/amd-ucode/

As in my kernel configuration I have these parameters:

CONFIG_EXTRA_FIRMWARE="amd-ucode/microcode_amd_fam19h.bin
CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"

I just had to recompile my kernel and reboot.

microcode: microcode updated early to new patch_level=0x0a20120e

It was patch_level=0x0a20120a previously.

I'm fully protected now:

$ lscpu
Vulnerabilities:                        
  Gather data sampling:                  Not affected
  Itlb multihit:                         Not affected
  L1tf:                                  Not affected
  Mds:                                   Not affected
  Meltdown:                              Not affected
  Mmio stale data:                       Not affected
  Retbleed:                              Not affected
  Spec rstack overflow:                  Mitigation; safe RET
  Spec store bypass:                     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                                 Not affected
  Tsx async abort:                       Not affected

dobo · 2023-09-18 21:10:43

@agapito, I've got 5850U:

➜  ~ sudo dmesg | grep 'AMD Ryzen'
[    0.294543] smpboot: CPU0: AMD Ryzen 7 PRO 5850U with Radeon Graphics (family: 0x19, model: 0x50, stepping: 0x0)
➜  ~ sudo dmesg | grep 'microcode updated'
[    0.586776] microcode: microcode updated early to new patch_level=0x0a50000f
➜  ~ sudo cpuid -r | grep '80000001' | head -1
   0x80000001 0x00: eax=0x00a50f00 ebx=0x00000000 ecx=0x75c237ff edx=0x2fd3fbff
➜  ~ lscpu | tail -13
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Yeah, I'm aware of amd-ucodegen. PKGBUILD internally uses a slightly modified version of it to bundle all family 17h microcodes into microcode_amd_fam17h.bin and 19h into microcode_amd_fam19h.bin. The rest is a copy-paste from amd-ucode package (compressing into cpio archive).

Last edited by dobo (2023-09-18 21:11:27)

hxss · 2023-11-05 15:50:00

@dobo thanks for package! Works on my AMD Ryzen 7 5800U

gen2arch · 2023-11-21 13:37:36

Sorry for stepping in so late:

I came here, because I had the same messages concerning AMD microcode as OP.

archlinux kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied!
archlinux kernel: Speculative Return Stack Overflow: Mitigation: safe RET, no microcode

The processor in our server is a Threadripper Pro, Zen3, 19h.

I installed dobo's AUR package for the Platomav alternative AMD firmwares.

Yet upon rebooting nothing seems to have changed:

dmesg | grep microcode | grep update
2.764844] microcode: microcode updated early to new patch_level=0x0a008205

But with the official arch amd-ucode package I already got this:

archlinux kernel: microcode: CPU1: patch_level=0x0a008205

So the "update part of the message is gone, but the microcode version seems to be exactly the same.

I fail to see what this means now: simply that both amd microcodes are the same?

And could anyone elaborate on how the AMD microcode is related to the "spec_rstack_overflow / inception" vulnerability: is the microcode intended to mitigate this vulnerability?

lscpu still shows this:

Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Mitigation; safe RET, no microcode
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

Thanks

gen2arch

Last edited by gen2arch (2023-11-21 13:41:30)

loqs · 2023-11-21 13:46:09

@gen2arch what is the model and stepping of the CPU? This can be obtained from the smpboot entry in dmesg and also what is the mainboard and if it on the latest firmware?

Last edited by loqs (2023-11-21 13:46:34)

gen2arch · 2023-11-21 15:13:38

loqs wrote:

@gen2arch what is the model and stepping of the CPU? This can be obtained from the smpboot entry in dmesg and also what is the mainboard and if it on the latest firmware?

I get this:

journalctl -b | grep smpboot 
family: 0x19, model: 0x8, stepping: 0x2

Mainboard is ASUS Pro WS WRX80E-SAGE SE WIFI on the latest firmware Version 1201.
Thanks
gen2arch

Last edited by gen2arch (2023-11-21 15:13:59)

loqs · 2023-11-21 17:11:46

You probably need to wait for ChagallWSPI-sWRX8 1.0.0.7 (Target December 2023) to either be available for your mainboard or extracted and uploaded to https://github.com/platomav/CPUMicrocodes

agapito · 2023-11-21 18:21:50

gen2arch wrote:

The processor in our server is a Threadripper Pro, Zen3, 19h.

If you have a Threadripper CPU you just have to wait for the amd-ucode package containing the updated firmware from AMD.

https://bbs.archlinux.org/viewtopic.php … 6#p2119246

loqs · 2023-11-21 19:21:16

agapito wrote:

If you have a Threadripper CPU you just have to wait for the amd-ucode package containing the updated firmware from AMD.
https://bbs.archlinux.org/viewtopic.php … 6#p2119246

Has AMD not already done the Data Center updates for Speculative Return Stack Overflow? This is based on a comparison of [1] [2]. V1del referenced a 17h part which is not vulnerable to SRSO.
[1]: https://www.amd.com/en/resources/produc … -7005.html
[2]: https://bbs.archlinux.org/viewtopic.php … 5#p2117535

Arch Linux

#1 2023-08-22 19:45:46

AMD Microcode failing with "Speculative Return Stack Overflow"

#2 2023-08-22 19:50:16

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#3 2023-08-23 09:36:03

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#4 2023-08-23 11:12:08

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#5 2023-08-23 11:57:52

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#6 2023-08-23 12:51:53

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#7 2023-08-23 14:11:49

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#8 2023-08-23 14:23:26

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#9 2023-08-23 14:38:05

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#10 2023-08-23 14:40:34

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#11 2023-08-23 14:59:11

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#12 2023-08-23 15:00:51

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#13 2023-08-25 08:52:34

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#14 2023-09-06 18:39:21

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#15 2023-09-06 20:31:16

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#16 2023-09-18 08:08:00

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#17 2023-09-18 19:33:25

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#18 2023-09-18 21:10:43

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#19 2023-11-05 15:50:00

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#20 2023-11-21 13:37:36

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#21 2023-11-21 13:46:09

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#22 2023-11-21 15:13:38

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#23 2023-11-21 17:11:46

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#24 2023-11-21 18:21:50

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

#25 2023-11-21 19:21:16

Re: AMD Microcode failing with "Speculative Return Stack Overflow"

Board footer