Arch Linux

OpusOne

Member

Registered: 2023-05-31

Posts: 186

[SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

Hi,

a change from kernel 6.5.4 to 6.5.5 has introduced this new issue during boot on my machine:

0.342194] APIC: Switch to symmetric I/O mode setup
[    0.342284] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.342330] Switched APIC routing to physical flat.
[    0.342812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    5.164680] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    5.164681] ...trying to set up timer (IRQ0) through the 8259A ...
[    5.164682] ..... (found apic 0 pin 2) ...
[    5.178460] ....... works.

As you can see, it takes nearly 5 s more than before the change, due to this. Not sure exactly what has changed, the change log lists changes in 'arch/x86/kernel/apic/x2apic_uv_x.c'.
The above error (and related kernel workaround which I'm pretty sure was not triggered before) makes boot 5 s longer, which is annoying. I already had another (probably BIOS issue) that was adding 6 s to my boot time (ref. in another thread.)
I'm willing to accept that the BIOS on this motherboard (ASUS X99 Deluxe) is a pile of crap, but this above is certainly new.

Anyone having run into the same specific "BIOS bug" when updating to 6.5.5?

The journal for the previous boot (which was with kernel 6.5.4) showed this for the same steps:

113698 xxx kernel: APIC: Switch to symmetric I/O mode setup
113708 xxx kernel: x2apic: IRQ remapping doesn't support X2APIC mode
113720 xxx kernel: Switched APIC routing to physical flat.
113731 xxx kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
113741 xxx kernel: clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x25748d0f7f0, max_idle_ns: 440795202126>

Definitely no sign of this "MP-BIOS bug" sequence, although the "TIMER" info line was the same.

Last edited by OpusOne (2023-09-28 05:39:08)

just4arch

Member

Registered: 2023-01-07

Posts: 122

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

[    0.000000] Linux version 6.5.5-arch1-1 (linux@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.41.0) #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000
[    0.000000] DMI: ASUS All Series/X99-A, BIOS 4101 07/10/2019
[    0.054208] Kernel command line: <*snip*> intremap=no_x2apic_optout intel_iommu=on
[    0.141322] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.141762] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.141763] x2apic enabled
[    0.141767] Switched APIC routing to cluster x2apic.
[    0.142249] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1

"works for me"
I don't have the "Deluxe" though.

Last edited by just4arch (2023-09-27 05:04:45)

OpusOne

Member

Registered: 2023-05-31

Posts: 186

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

Interesting, someone with an ASUS X99!

About your kernel options:  intremap=no_x2apic_optout intel_iommu=on
which may have something to do with it, how did you find out to set them and what was the rationale?

Also, I see you have BIOS 4101, which is the latest available for my MB as well (I'm pretty sure it's the same BIOS), but I'm currently on v. 3802, which was the last stable version, the newer ones were all beta versions. Now if v 4101 works fine for you, I may consider updating.

What is your average boot time?
(Could you issue a

sudo dmesg | grep _OSC

and post your result?)

Last edited by OpusOne (2023-09-27 06:29:54)

just4arch

Member

Registered: 2023-01-07

Posts: 122

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

About your kernel options:  intremap=no_x2apic_optout intel_iommu=on
which may have something to do with it, how did you find out to set them and what was the rationale?

Uuh... it's been a while... fiddling around probably.
Booting without:

[    0.141494] APIC: Switch to symmetric I/O mode setup
[    0.141511] DMAR-IR: IOAPIC id 1 under DRHD base  0xfbffc000 IOMMU 1
[    0.141513] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 1
[    0.141514] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.141515] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.141954] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.141955] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.141958] Switched APIC routing to physical flat.
[    0.142436] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.158166] Calibrating delay loop (skipped), value calculated using timer frequency.. 6602.89 BogoMIPS (lpj=10999830)

That's still with the EFI settings enabled.

Also, I see you have BIOS 4101, which is the latest available for my MB as well (I'm pretty sure it's the same BIOS), but I'm currently on v. 3802, which was the last stable version, the newer ones were all beta versions. Now if v 4101 works fine for you, I may consider updating.

It's been a bit wonky since I upgraded my RAM, beyond that, it looks like it's only mitigations added.

What is your average boot time?

Startup finished in 20min 33.556s (firmware) + 32.993s (loader) + 6.744s (kernel) + 2.782s (initrd) + 1.847s (userspace) = 21min 17.923s
graphical.target reached after 1.846s in userspace.

Firmware time is bogus and the loader time includes editing the cmdline.
That's with LXDE on a Samsung SSD 970 EVO Plus with BTRFS.

(Could you issue a

sudo dmesg | grep _OSC

and post your result?)

dmesg | grep _OSC
[    0.304209] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.314911] acpi PNP0A03:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    3.341555] acpi PNP0A03:03: _OSC: OS now controls [PME AER PCIeCapability]
[    3.346011] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.354892] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    6.381552] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]

I also use

acpi_osi=Linux pcie_aspm=force

in my cmdline and don't recall why either.

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,101

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

Drop all of that and retry. Especially setting acpi_osi=Linux will in 99% of the cases be counter-productive since most ACPI implementations don't know or make a wrong distinction in that case.

just4arch

Member

Registered: 2023-01-07

Posts: 122

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

Doesn't seem make a difference for me, I can't reproduce the error on my board.
The boot "felt" longer, but neither systemd-analyze nor dmesg corroborate.

So maybe some EFI settings...

Last edited by just4arch (2023-09-27 15:42:42)

OpusOne

Member

Registered: 2023-05-31

Posts: 186

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

About your kernel options:  intremap=no_x2apic_optout intel_iommu=on
which may have something to do with it, how did you find out to set them and what was the rationale?

Uuh... it's been a while... fiddling around probably.

If you use PCI passthrough for VMs, that would explain it for the 'intel_iommu=on' :

https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF

The 'intremap=no_x2apic_optout' option is suggested as well for the X299 chipset here: https://wiki.archlinux.org/title/PCI_pa … F/Examples
It's likely the same for X99.

I hadn't enabled IOMMU yet as I haven't used PCI passthrough yet (but something I was planning to for later), so I hadn't considered these options yet.

Booting without:

[    0.141494] APIC: Switch to symmetric I/O mode setup
[    0.141511] DMAR-IR: IOAPIC id 1 under DRHD base  0xfbffc000 IOMMU 1
[    0.141513] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 1
[    0.141514] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.141515] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.141954] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.141955] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.141958] Switched APIC routing to physical flat.
[    0.142436] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.158166] Calibrating delay loop (skipped), value calculated using timer frequency.. 6602.89 BogoMIPS (lpj=10999830)

That's still with the EFI settings enabled.

That the BIOS has the x2apic opt out bit set on this MB is not surprising, it's extremely common on many desktop MBs.
From what I've heard, Windows was known not to handle the x2apic properly / had issues with it (maybe that's not true anymore with more recent Windows versions), so MB vendors used to "disable" it by default.

What is weird compared to my MB (which again has the same chipset and most likely the same BIOS except for the version) is that without the 'intremap=no_x2apic_optout' option, you get the expected message "x2apic is disabled because BIOS sets x2apic opt out bit.", while I don't. In my case, it just reports "x2apic: IRQ remapping doesn't support X2APIC mode". I don't know why. Apart from the BIOS version, the other difference is that I didn't enable IOMMU either. Maybe that's required for x2apic to be accessed?

What EFI settings are you talking about here?

Also, I see you have BIOS 4101, which is the latest available for my MB as well (I'm pretty sure it's the same BIOS), but I'm currently on v. 3802, which was the last stable version, the newer ones were all beta versions. Now if v 4101 works fine for you, I may consider updating.

It's been a bit wonky since I upgraded my RAM, beyond that, it looks like it's only mitigations added.

Versions past 3802 were beta and ASUS listed only microcode updates for them, which I don't care about since the microcode is uploaded at boot anyway (with a more recent version), I dont care for the BIOS to do that. So I suppose the BIOS is not what makes a difference there, unless they have also fixed a few bugs that they don't list. Which is also probably not uncommon. Who knows.

dmesg | grep _OSC
[    0.304209] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.314911] acpi PNP0A03:03: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    3.341555] acpi PNP0A03:03: _OSC: OS now controls [PME AER PCIeCapability]
[    3.346011] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.354892] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug LTR DPC]
[    6.381552] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]

Exactly the same as I get and which I have talked about in another thread. We basically get a 6 s delay just for these ACPI _OSC queries, which is not "normal". On my other machines, the same takes just a few ms.
As we can see here, the BIOS seems to have 2 tables for _OSC (while most other machines I've used only have 1), and for each, there is what looks like a 3 s timeout for the _OSC queries. I haven't found any fix for that so far.

Linus once said:

We do not trust BIOS tables, because BIOS writers are invariably totally incompetent crack-addicted monkeys. If they weren't, they wouldn't be BIOS writers. QED.

(ref.: https://lwn.net/Articles/387609/ )

Anyway, I will try to add these two kernel options "intremap=no_x2apic_optout intel_iommu=on" and see if it solves my "MP-BIOS" timer bug that adds 5 s for nothing.
Since there were changes in 'apic/x2apic_uv_x.c' in kernel 6.5.5, that is maybe an hint about how enabling x2apic may solve the issue, or rather not run into it as it would use another method for configuring the timer with x2apic.

I'll report whether this fixes the issue.

Last edited by OpusOne (2023-09-27 20:41:01)

OpusOne

Member

Registered: 2023-05-31

Posts: 186

Re: [SOLVED]Kernel 6.5.5: MP-BIOS bug: 8254 timer not connected to IO-APIC

Alright, it did fix the issue. Yay.

To sum it up:
- kernel 6.5.5 seems to have added a change which raised this particular "MP-BIOS bug" on this series of ASUS X99 motherboards. While it was non-blocking and without any visible side-effect once booted, it added a 5 s delay during boot.
- Switching to X2APIC mode does bypass this issue altogether, and has the added benefit of now using X2APIC, which is supposed to yield better performance for higher-core count CPUs (my CPU is a Xeon with 14 cores, so that may be a plus here.)
- X2APIC apparently requires IOMMU.
- For IOMMU to work properly with Intel CPUs, looks like we need to enable the "intel_iommu=on" option on the kernel command line.
- For some MBs (which is the case with ASUS X99 MBs and probably many others), the option "intremap=no_x2apic_optout" is required to enable X2APIC.
- Lastly, the reason I was only getting the "x2apic: IRQ remapping doesn't support X2APIC mode" message and not the expected "x2apic is disabled because BIOS sets x2apic opt out bit" (since I didn't have the "intremap=no_x2apic_optout" option in the kernel command line) was that, while I had enabled Intel VT in UEFI settings, VT-d (directed I/O's) was not enabled. (On this MB, both options are in separate sections, which is a bit confusing.) So I had to enable Intel VT-d as well.

With all that, the above issue is gone, the kernel is using X2APIC and I can now use PCI passthrough, so that'll be handy as well.
Thanks to just4arch for the pointers that helped me solve this.

(Note: this is now what it looks like in dmesg:

(...)
[    0.342760] APIC: Switch to symmetric I/O mode setup
[    0.342762] DMAR: Host address width 46
[    0.342763] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0
[    0.342768] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap 8d2008c10ef0466 ecap f0205b
[    0.342771] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.342775] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.342776] DMAR: RMRR base: 0x000000bdb74000 end: 0x000000bdb82fff
[    0.342778] DMAR: ATSR flags: 0x0
[    0.342779] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.342782] DMAR-IR: IOAPIC id 1 under DRHD base  0xfbffc000 IOMMU 1
[    0.342783] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 1
[    0.342784] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.342785] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.343382] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.343383] x2apic enabled
[    0.343389] Switched APIC routing to cluster x2apic.
[    0.343902] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.359430] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2573ab5ef49, max_idle_ns: 440795249114 ns
(...)

Last edited by OpusOne (2023-09-28 05:37:19)