[SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

evilsnoo · 2023-04-25 16:03:01

I'm installing Arch Linux on a new HP Pavilion Aero 13 (AMD Ryzen 7 7735U with Radeon 680M GPU). Most things, including the wifi, work perfectly even right from the install medium, if we exclude the slowness of the keyboard.

Typing is extremely slow which makes following through the wiki very painful as I have to wait after each keystroke; otherwise SHIFT might get mixed up with other keys and capitalize the wrong letters.

I have this issue in the TTY of both the install live USB (archlinux-2023.04.01-x86_64 iso) and the installed system (followed the wiki exactly up to including the GRUB install). I wiped out Windows 11 which didn't have this issue (so this clearly is not a hardware issue).

Most input lag problems in this forum or elsewhere relate to X11, while my issue is with the input in the TTY itself before even installing a desktop environment.

Anyone know what might possibly be wrong? Thanks!

Here are pastebins of the outputs of :

EDIT: typing is smooth inside the Grub menu, so probably something to do with the kernel

Last edited by evilsnoo (2023-04-28 21:29:58)

seth · 2023-04-25 19:30:28

"i8042.nopnp=1 i8042.dumbkbd=1" - already a mitigational effort?
Is it the input or the output, ie. does "ls -R /etc" render immediately or take several hours?
Does it happen if you boot "nomodeset" (nb. that this will prevent you from starting any GUI)

evilsnoo · 2023-04-25 20:41:00

seth wrote:

"i8042.nopnp=1 i8042.dumbkbd=1" - already a mitigational effort?

forgot to mention that yes I played around with the options in https://lightrush.ndoytchev.com/random- … irkoptions since many claim it fixes their keyboard issues but no success on my side, that line has no effect (except disabling the caps lock led)

Is it the input or the output, ie. does "ls -R /etc" render immediately or take several hours?

It is the input that is extremely slow. "ls -R /etc" is almost instantaneous.

Does it happen if you boot "nomodeset" (nb. that this will prevent you from starting any GUI)

sadly "nomodeset" has no effect on these keyboard lags

seth · 2023-04-25 21:26:15

Can you quantify "extremely slow"?
¼s, 1s, 5s, 25s, …?

lsmod | grep atkbd # whether the next isn't pointless
atkbd.softrepeat=1 # kernel parameter

You can also try

# kbdrate -r 0 # disables autorepeat

If you install X11 and some WM and xterm, how's the behavior there?

evilsnoo · 2023-04-25 21:53:57

small (no pun intended) update: Puppy Linux (fossapup64, based on Ubuntu Focal) works fine with its 5.4.53 kernel, no typing lags either in the tty or inside the desktop environment. Going to do a diff between their kernel config and arch's to see if it's just the kernel version or some changes they or arch made in the config files.

seth wrote:

Can you quantify "extremely slow"?
¼s, 1s, 5s, 25s, …?

You have to wait approximately 1 or even 2 seconds for the corresponding character to show up on the screen after you pressed the key. You can also keep pressing keys in advance, lift your hands and watch the shell slowly print your characters one by one for ~5 seconds (but you can't type too much in advance either, looks like its keyboard buffer is limited to around 5 characters). I think we can agree that's extremely slow even by 90s standards.

seth wrote:

lsmod | grep atkbd # whether the next isn't pointless

I get:

atkbd                  40960  0
libps2                 20480  1 atkbd
vivaldi_fmap           16384  1 atkbd
serio                  28672  5 amd_pmc,serio_raw,atkbd,i8042

seth wrote:

atkbd.softrepeat=1 # kernel parameter

didn't solve the issue

seth wrote:

You can also try
# kbdrate -r 0 # disables autorepeat

the issue is still there (and that didn't change anything since the manpage says the value next to -r cannot be less than 2, but even with that it didn't solve the problem)

seth wrote:

If you install X11 and some WM and xterm, how's the behavior there?

I haven't gotten there yet since it's painfully slow to write anything on the shell, so I figured I fix this while the install is still minimal before attempting anything else

Last edited by evilsnoo (2023-04-25 21:57:59)

evilsnoo · 2023-04-25 22:18:29

another update: the issue is also there with linux-lts and linux-zen

dimich · 2023-04-25 22:36:03

lsusb -vt

?
Can you try external keyboard?
Does BIOS have any USB compatibility options?
Can you try to disable USB 3.0 hubs?

evilsnoo · 2023-04-25 22:52:20

dimich wrote:

lsusb -vt

?

Sure!

/:  Bus 10.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 09.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/0p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/3p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
        ID 13fe:4300 Phison Electronics Corp. 
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=btusb, 480M
        ID 0489:e0f2 Foxconn / Hon Hai 
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=btusb, 480M
        ID 0489:e0f2 Foxconn / Hon Hai 
    |__ Port 3: Dev 3, If 2, Class=Wireless, Driver=, 480M
        ID 0489:e0f2 Foxconn / Hon Hai 
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 3: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
        ID 30c9:0065 Luxvisions Innotech Limited 
    |__ Port 3: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
        ID 30c9:0065 Luxvisions Innotech Limited 
    |__ Port 4: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 12M
        ID 04f3:0c00 Elan Microelectronics Corp.

Can you try external keyboard?

This works! Typing is fast and smooth when using an external keyboard.

So right now what's working is:
* PuppyLinux with their 5.4.53 kernel, no external keyboard needed
* Current Arch kernel + necessary external keyboard

Does BIOS have any USB compatibility options?
Can you try to disable USB 3.0 hubs?

The only USB-related options I have in the BIOS are "USB Charging" and "USB Boot" (both enabled).

Last edited by evilsnoo (2023-04-25 22:54:09)

evilsnoo · 2023-04-26 00:08:50

Got the following diff between Puppy's and Arch's kernel configs using the diffconfig script from the kernel git repo: https://termbin.com/sf5to

- sign means the line is present in the Arch config but not in the Puppy config, + sign means the opposite
the arrow "y -> n" means it's set to "y" in the Arch config and to "n" in the Puppy config

Last edited by evilsnoo (2023-04-26 00:49:05)

seth · 2023-04-26 06:08:23

Puppy has atkbd and i8042 compiled into the kernel, you could try to add them to the MODULES in your mkinitcpio.conf (and rebuild the initramfs), but I suspect the far more relevant condition is that the puppy linux kernel is ~3 years old.

evilsnoo · 2023-04-26 21:43:41

just compiled a fresh kernel based off the linux package with atkbd and i8042 compiled with it but that sadly didn't solve the keyboard slowness

which leaves us with the possibility that something went wrong between the versions 5.4.53 and 6.2.12

Last edited by evilsnoo (2023-04-26 21:47:40)

evilsnoo · 2023-04-27 00:04:02

Narrowed the issue down to the linux kernel, indeed manually installing the 5.19.9 version from https://archive.archlinux.org/packages/l/linux/ solved the problem entirely.
Don't know if this merits a [solved] tag since having to downgrade the kernel a few months back to use new hardware and not being able to normally use the keyboard with the official version are still big issues IMO, and we still don't know what exactly caused the regression.

Last edited by evilsnoo (2023-04-27 01:01:43)

dimich · 2023-04-27 02:24:11

evilsnoo wrote:

Narrowed the issue down to the linux kernel, indeed manually installing the 5.19.9 version from https://archive.archlinux.org/packages/l/linux/ solved the problem entirely.

Can't you bisect to find exact regressed version? 46 releases should take 6 tries in worst case.

evilsnoo · 2023-04-28 12:58:29

Found the culprit: it's with the kernel version 5.19.10 that the keyboard problem emerges, 5.19.9 is indeed the last version where the keyboard works flawlessly and without the slowness I described above. I guess I got really lucky with my guess for the 5.19.9 above.

Looking at the changelog of the 5.19.10 I found this very curious part:

commit 37c81d9f1d1b1458894454efcb857f6a769b6bc4
Author: Chuanhong Guo <gch981213@gmail.com>
Date: Tue Jul 12 10:00:58 2022 +0800
ACPI: resource: skip IRQ override on AMD Zen platforms

commit 9946e39fe8d0a5da9eb947d8e40a7ef204ba016e upstream.

IRQ override isn't needed on modern AMD Zen systems.
There's an active low keyboard IRQ on AMD Ryzen 6000 and it will stay
this way on newer platforms. This IRQ override breaks keyboards for
almost all Ryzen 6000 laptops currently on the market.

Skip this IRQ override for all AMD Zen platforms because this IRQ
override is supposed to be a workaround for buggy ACPI DSDT and we can't
have a long list of all future AMD CPUs/Laptops in the kernel code.
If a device with buggy ACPI DSDT shows up, a separated list containing
just them should be created.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=216118
Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Chuanhong Guo <gch981213@gmail.com>
Acked-by: Mario Limonciello <mario.limonciello@amd.com>
Tested-by: XiaoYan Li <lxy.lixiaoyan@gmail.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

I think for these reasons and in order to not scare potential users of Arch Linux away with a horrible keyboard experience and since we're starting to have many modern AMD Ryzen laptops (my experience perfectly contradicts the "IRQ override isn't needed on modern AMD Zen systems" from the quoted dev), the 5.19.9 version should be offered in the official repo without having to go and manually download it from the archive.

Last edited by evilsnoo (2023-04-28 13:02:12)

seth · 2023-04-28 13:37:35

5.19.9 version should be offered in the official repo

Keeping kernels around anytime some kernel dev breaks fringe HW is no sustainable strategy. Nor is it helpful.
What if there there's a massive CVE and you *have* to update the kernel?
Also what about all other distros?

This needs to be fixed properly.

The patch wrote:

If a device with buggy ACPI DSDT shows up, a separated list containing just them should be created.

1. See whether there's a firmware/BIOS update for your device
2. Try to revert the patch in isolation and then see https://bugzilla.kernel.org/show_bug.cgi?id=216118#c10

evilsnoo · 2023-04-28 21:28:39

My BIOS is up-to-date.

I made this simple patch which "reverts" the above commit (not exactly a revert since you get conflicts and other changes have been made there since, but it turns out it's just the #ifdef that needs to be removed):

--- a/drivers/acpi/resource.c
+++ b/drivers/acpi/resource.c
@@ -536,16 +536,6 @@
 			return entry->override;
 	}
 
-#ifdef CONFIG_X86
-	/*
-	 * IRQ override isn't needed on modern AMD Zen systems and
-	 * this override breaks active low IRQs on AMD Ryzen 6000 and
-	 * newer systems. Skip it.
-	 */
-	if (boot_cpu_has(X86_FEATURE_ZEN))
-		return false;
-#endif
-
 	return true;
 }

Applying it to the linux (or any flavor like lts, zen...) source package and building with makepkg yields a functioning 6.2.13 kernel without this keyboard issue.

I'll mark this as solved and will file a bug report soon.

Thank you everyone!

Last edited by evilsnoo (2023-04-28 21:31:29)

liewkj · 2023-06-15 02:18:49

Having to custom compile the mainline kernel shouldn't have been considered as viable solution. This is a HUGE PR disaster for AMD Ryzen 6000/7000 series, HP and Lenovo in particular. The ACPI APIC/MADT "Interrupt Source Override" should be the highest priority for any modern system designs using APIC interrupts The AMD engineer who suggested the commit was embarrassingly inexperienced. Ironically, an Intel folk signed off the commit since the X86_FEATURE_ZEN effectively made the commit not applicable for Intel. The commit could have used a kernel cmdline parameter instead of using X86_FEATURE_ZEN.

Wouldn't Arch consider this a critical flaw in user experience that the patch should be cherry-picked ASAP?

Yes, I reproduced the exact issue as the OP described on a 2-day new HP Pavilion Aero 13z-be200. Kudo to the OP that the root cause had been identified. But current linux 6.3.8 still suffer the same issue. This is a disaster for rolling distros.

seth · 2023-06-15 06:28:23

And ranting into soem random distro forum won't solve anything for sure. See the bug linked in #15 and file your HW details there.
It's not gonna fix upstream by magic.

liewkj · 2023-06-16 23:48:16

I wouldn't perceive the post as ranting.

The bug linked in #15 was incorrect, closed and that was the cause resulting in the commit to skipping the IRQ override. Putting the issue into right context, the OP seemed to have filed another bug exactly for this issue.
https://bugzilla.kernel.org/show_bug.cgi?id=217406

It remains Opened as of writing despite revised patch is available with on-going discussion. I hope Arch would cherry-pick the patch ASAP.

An importance of "P3 Normal" -- Are you nuts AMD??! I thought HP was one of AMD most important customers!!

seth · 2023-06-17 07:36:21

Whining? Rambling?
Call it whatever you want, it's all the same.

The link in #15 is actually to a comment that asks for details on the system and the OP has followed that advice and opened 217406 (but not linke it here
So all you've to do is wait, what makes your <insert preferred description here> even more pointless.

Arch will unlikely add a patch that has not (yet) been upstreamed at all and the kernel packager will not have read this thread either.

Arch Linux

#1 2023-04-25 16:03:01

[SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#2 2023-04-25 19:30:28

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#3 2023-04-25 20:41:00

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#4 2023-04-25 21:26:15

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#5 2023-04-25 21:53:57

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#6 2023-04-25 22:18:29

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#7 2023-04-25 22:36:03

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#8 2023-04-25 22:52:20

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#9 2023-04-26 00:08:50

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#10 2023-04-26 06:08:23

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#11 2023-04-26 21:43:41

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#12 2023-04-27 00:04:02

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#13 2023-04-27 02:24:11

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#14 2023-04-28 12:58:29

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#15 2023-04-28 13:37:35

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#16 2023-04-28 21:28:39

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#17 2023-06-15 02:18:49

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#18 2023-06-15 06:28:23

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#19 2023-06-16 23:48:16

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

#20 2023-06-17 07:36:21

Re: [SOLVED] keyboard input extremely slow in TTY (6.2.12 kernel)

Board footer