Windows driver updates thrash linux input on laptop

hasnoidea · 2022-06-01 08:56:22

Hello everyone

My laptop (Thinkpad E580) has dualbooted Windows 11 with Arch Linux.

2 days ago Arch still worked fine on my laptop, having installed all updates to-date and having rebooted.

1 day ago I had to do smth. on Windows, which I normally never use at all.
I never bothered to install some needed Windows drivers (for special buttons, screen brightness regulation, power profile, bluetooth, sound etc.).
However, this time Windows felt so sluggish that I opted to install the drivers (source) which greatly improved the experience on Windows.

After I rebooted to Arch, I noticed that the keyboard input on keys 'h,j,u,k,...' (side of the right hand) produced garbage output, e.g.
- j = a7ds2f
- h = ESC + ghä<
A longer keypress resulted in a longer sequence of garbage with max. length. This output is deterministic and not random at all.

It seems to me, that the installation of Windows drivers messed with some firmware such that Arch now has weird problems.
I booted into a live Arch environment to reinstall the packages `linux`, `linux-api-headers` and `linux-firmware`. This did not help.
I also then installed any other update to packages that occured over the last 2 days. This did not help either.
Interestingly the live Arch environment has no problem with the input.

I've tried searching for similar errors, but couldn't find anything.

I am at a loss now what to do since I need my laptop for university, I'd appreciate any help very much.

V1del · 2022-06-01 09:31:44

Make sure and double and triple check that Windows fast boot is disabled: https://wiki.archlinux.org/title/Dual_b … ibernation

hasnoidea · 2022-06-01 11:18:22

V1del wrote:

check that Windows fast boot is disabled

Seems like some random Windows update enabled it again.
I disabled it (and triple checked it ;-) and tried running Arch again, same effect: input remains like before.

I was planning on re-installing Arch on a new small partition to see whether the new system would work.
I'll report back what I find.

If possible I'd rather not reinstall my main system though.

Last edited by hasnoidea (2022-06-01 11:19:08)

hasnoidea · 2022-06-01 12:00:12

I installed Arch anew on a small side partition.
It has the same effect of right-hand keys producing garbage input :-(

I really have no clue what else to do.
I think I need to somehow roll back to the previous firmware before the Windows "drivers" "updated" them.
But I don't think there's any possibility of finding out which version was installed previously and where to find it.

d.ALT · 2022-06-01 12:02:16

Just to say... Does these drivers changed something about the keyboard layout? Did they set something weird inside UEFI Setup Settings? Some Fn selectors have been enabled?

Can you try entering Windows, performing a full shotdown and remove the laptop's battery?
Wait 2 minutes, re-insert the battery and start your computer booting directly into Arch.

Last edited by d.ALT (2022-06-01 12:02:37)

hasnoidea · 2022-06-01 12:29:18

d.ALT wrote:

Just to say... Does these drivers changed something about the keyboard layout? Did they set something weird inside UEFI Setup Settings? Some Fn selectors have been enabled?
Can you try entering Windows, performing a full shotdown and remove the laptop's battery?
Wait 2 minutes, re-insert the battery and start your computer booting directly into Arch.

I don't know whether they changed the keyboard layout internally, it doesn't seem that way, however, as using a live disk has no problems. Unless my logic is flawed here.
I looked into the BIOS and there seems to be no change to how Fn influences buttons. These never had any effect on the normal characters anyway (only F1-F12 and such).

I don't really know what you mean by "full shutdown of Windows". I already tried `shutdown /s /f /t 0` inside the Windows terminal.
I can't remove the battery of the laptop without opening the chassis and remove internals. I can try to let the battery run completely out of juice, however. This will take a while :-)

By the way, I am using systemd-boot with standard booting into Arch.

seth · 2022-06-01 13:19:32

however, as using a live disk has no problems

The windows driver might be coincidence.
What output do the affected keys produce on

sudo libinput debug-events
sudo evtest # libinput told you which device is the relevant one
xev -event keyboard

On a guess and because the affected keys, this is probably some numlock situation.

hasnoidea · 2022-06-01 13:41:57

seth wrote:

however, as using a live disk has no problems
The windows driver might be coincidence.
What output do the affected keys produce on
sudo libinput debug-events
sudo evtest # libinput told you which device is the relevant one
xev -event keyboard
On a guess and because the affected keys, this is probably some numlock situation.

How do I test this?
I can't input this to the native OS, does it work when in a chroot?
I can try plugging in another keyboard later (currently commuting).

seth · 2022-06-01 13:49:30

I can't input this to the native OS, does it work when in a chroot?

The broken condition would be relevant, you could copypaste the commands into a terminal (though the sudo password could be a problem)

Does the keyboard work as expected in the mulit-user.target (2nd link below)?

According to https://smallbusiness.chron.com/disable … 50583.html Fn+NumLock toggles the keyboard numpad.

hasnoidea · 2022-06-01 17:48:04

In all logs I simply tried to type julius with some delay of roughly 1s inbetween each character.

sudo libinput debug-events

log: https://0x0.st/oBUb.log

sudo evtest

log: https://0x0.st/oBUc.log

xev -event keyboard

As I am on wayland this one wouldn't work. I've omitted it.

The NumLock didn't do anything either when in the faulty behaviour.

I plugged in the keyboard from my PC to log in and then noticed the followings:
- laptop keyboard works flawless after login to my `sway` environment
- after logging out (into the previous issue-taken login screen) the keyboard continues working correctly.
- rebooting results in the same faulty behaviour until logged in using the USB keyboard.

Last edited by hasnoidea (2022-06-01 17:52:19)

seth · 2022-06-01 18:34:07

Event: time 1654105410.819202, type 4 (EV_MSC), code 4 (MSC_SCAN), value 1e
Event: time 1654105410.819202, type 1 (EV_KEY), code 30 (KEY_A), value 1
Event: time 1654105410.819202, -------------- SYN_REPORT ------------
Event: time 1654105410.836933, type 4 (EV_MSC), code 4 (MSC_SCAN), value 47
Event: time 1654105410.836933, type 1 (EV_KEY), code 71 (KEY_KP7), value 1
Event: time 1654105410.836933, -------------- SYN_REPORT ------------
Event: time 1654105410.850327, type 4 (EV_MSC), code 4 (MSC_SCAN), value 20
Event: time 1654105410.850327, type 1 (EV_KEY), code 32 (KEY_D), value 1
Event: time 1654105410.850327, -------------- SYN_REPORT ------------
Event: time 1654105410.853206, type 4 (EV_MSC), code 4 (MSC_SCAN), value 1f
Event: time 1654105410.853206, type 1 (EV_KEY), code 31 (KEY_S), value 1
Event: time 1654105410.853206, -------------- SYN_REPORT ------------
Event: time 1654105410.869214, type 4 (EV_MSC), code 4 (MSC_SCAN), value 50
Event: time 1654105410.869214, type 1 (EV_KEY), code 80 (KEY_KP2), value 1
Event: time 1654105410.869214, -------------- SYN_REPORT ------------

Event: time 1654105410.836933, type 1 (EV_KEY), code 71 (KEY_KP7), value 1
Event: time 1654105410.869214, type 1 (EV_KEY), code 80 (KEY_KP2), value 1
Event: time 1654105410.979745, type 1 (EV_KEY), code 71 (KEY_KP7), value 0
Event: time 1654105411.002031, type 1 (EV_KEY), code 80 (KEY_KP2), value 0
Event: time 1654105411.982758, type 1 (EV_KEY), code 55 (KEY_KPASTERISK), value 1
Event: time 1654105412.014101, type 1 (EV_KEY), code 77 (KEY_KP6), value 1
Event: time 1654105412.125479, type 1 (EV_KEY), code 55 (KEY_KPASTERISK), value 0
Event: time 1654105412.150314, type 1 (EV_KEY), code 77 (KEY_KP6), value 0
Event: time 1654105415.133179, type 1 (EV_KEY), code 55 (KEY_KPASTERISK), value 1
Event: time 1654105415.164477, type 1 (EV_KEY), code 77 (KEY_KP6), value 1
Event: time 1654105415.297328, type 1 (EV_KEY), code 55 (KEY_KPASTERISK), value 0
Event: time 1654105415.319554, type 1 (EV_KEY), code 77 (KEY_KP6), value 0

The keypad certainly shows up a lot in the evtest …

- rebooting results in the same faulty behaviour until logged in using the USB keyboard.

Despite the external keyboard being attached during the boot?
How do you log in? From the console or using some DM?

Which kernel are you using? Did you recently update to 5.18?
Does the problem remain w/ the LTS kernel?

hasnoidea · 2022-06-01 19:09:35

seth wrote:

The keypad certainly shows up a lot in the evtest

Toggling NumLock only results in numbers dis-/appearing when typing after reboot.

Despite the external keyboard being attached during the boot?

Yes, that had no impact at all.

How do you log in? From the console or using some DM?

I am using greetd with tuigreet normally.
Both [greetd] and the console show the same behaviour.

Which kernel are you using? Did you recently update to 5.18?

Yes, I am using

$ uname -a
Linux arch 5.18.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 30 May 2022 17:53:11 +0000 x86_64 GNU/Linux

Before installing the Windows drivers I had no problem with this specific kernel.

Does the problem remain w/ the LTS kernel?

Yes, the problems remains with

$ uname -a
Linux arch 5.15.44-1-lts #1 SMP Mon, 30 May 2022 13:45:47 +0000 x86_64 GNU/Linux

Thank you very much for trying to help me!

Last edited by hasnoidea (2022-06-01 19:48:54)

seth · 2022-06-01 19:18:35

Toggling NumLock only results in numbers dis-/appearing when typing after reboot.

It's not about toggling numlock - the right-hand keyboard operates as numpad for keyboards w/o a physical one and apparently you can toggle that numpad (that's a normal/required feature) w/ Fn+numlock on lenovo notebooks as well (despite them having a physical numpad)

Yes, the problems remains with
$ uname -a
Linux arch 5.18.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 30 May 2022 17:53:11 +0000 x86_64 GNU/Linux

That's not the LTS kernel…

hasnoidea · 2022-06-01 19:43:31

seth wrote:

It's not about toggling numlock - the right-hand keyboard operates as numpad for keyboards w/o a physical one and apparently you can toggle that numpad (that's a normal/required feature) w/ Fn+numlock on lenovo notebooks as well (despite them having a physical numpad)

Pressing the Fn + NumLock (also as mentioned in the article you referenced) only switches between
- numbers, and
- special keys (Home, End) and arrow keys
From what I can see, only effects whether numbers occur in the "random" input.

That's not the LTS kernel…

Whops, sorry, that was a wrong copy. It was this version:

$ uname -a
Linux arch 5.15.44-1-lts #1 SMP Mon, 30 May 2022 13:45:47 +0000 x86_64 GNU/Linux

seth · 2022-06-01 20:30:44

Pressing the Fn + NumLock (also as mentioned in the article you referenced) only switches between
- numbers, and
- special keys (Home, End) and arrow keys

Sounds as if that simply toggle numlock, not the numeric block in the alphanumeric block

however, as using a live disk has no problems

Does that include a recent archlinux installation iso?

I assume just using the external keyboard "somehow™" (not necessarily logging in) fixes the situation?

* Windows can only carry over through the firmware
* a different udev match could not explain the "external keyboard fixes it" situation
* it's not some userspace since evtest picks up the bogus events
* external boot devices likely get recognized by the FW

After disabling windows fast start, did you reboot windows (twice, because voodoo) or just shut it down?

hasnoidea · 2022-06-01 20:59:32

seth wrote:

Does that include a recent archlinux installation iso?

I assume just using the external keyboard "somehow™" (not necessarily logging in) fixes the situation?

It's only fixed after logging in. The laptop keyboard keeps misbehaving until after logging in.
SSH does not fix it however.

After disabling windows fast start, did you reboot windows (twice, because voodoo) or just shut it down?

I restarted various times from when I disabled it and kept trying things, yes.

seth · 2022-06-02 06:02:49

The laptop keyboard keeps misbehaving until after logging in.

Did you ever try the behavior when just booting the multi-user.target (not starting greetd)?

hasnoidea · 2022-06-02 10:01:28

seth wrote:

Did you ever try the behavior when just booting the multi-user.target (not starting greetd)?

I just did, yields the same effect :-/

I am running the following services then:

  UNIT                                                  LOAD      ACTIVE   SUB     DESCRIPTION
  auditd.service                                        loaded    inactive dead    Security Auditing Service
  dbus.service                                          loaded    active   running D-Bus System Message Bus
● display-manager.service                               not-found inactive dead    display-manager.service
  dm-event.service                                      loaded    inactive dead    Device-mapper event daemon
  emergency.service                                     loaded    inactive dead    Emergency Shell
  getty@tty1.service                                    loaded    active   running Getty on tty1
  getty@tty2.service                                    loaded    active   running Getty on tty2
  initrd-cleanup.service                                loaded    inactive dead    Cleaning Up and Shutting Down Daemons
  initrd-parse-etc.service                              loaded    inactive dead    Reload Configuration from the Real Root
  initrd-switch-root.service                            loaded    inactive dead    Switch Root
  initrd-udevadm-cleanup-db.service                     loaded    inactive dead    Cleanup udev Database
  iwd.service                                           loaded    active   running Wireless service
  kmod-static-nodes.service                             loaded    active   exited  Create List of Static Device Nodes
  ldconfig.service                                      loaded    inactive dead    Rebuild Dynamic Linker Cache
  lm_sensors.service                                    loaded    active   exited  Initialize hardware monitoring sensors
  lvm2-lvmpolld.service                                 loaded    inactive dead    LVM2 poll daemon
  lvm2-monitor.service                                  loaded    active   exited  Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
  man-db.service                                        loaded    inactive dead    Daily man-db regeneration
  mkinitcpio-generate-shutdown-ramfs.service            loaded    inactive dead    Generate shutdown-ramfs
  modprobe@configfs.service                             loaded    inactive dead    Load Kernel Module configfs
  modprobe@drm.service                                  loaded    inactive dead    Load Kernel Module drm
  modprobe@fuse.service                                 loaded    inactive dead    Load Kernel Module fuse
● plymouth-quit-wait.service                            not-found inactive dead    plymouth-quit-wait.service
● plymouth-start.service                                not-found inactive dead    plymouth-start.service
  rescue.service                                        loaded    inactive dead    Rescue Shell
  shadow.service                                        loaded    inactive dead    Verify integrity of password and group files
  sshd.service                                          loaded    active   running OpenSSH Daemon
  sshdgenkeys.service                                   loaded    inactive dead    SSH Key Generation
● syslog.service                                        not-found inactive dead    syslog.service
  systemd-ask-password-console.service                  loaded    inactive dead    Dispatch Password Requests to Console
  systemd-ask-password-wall.service                     loaded    inactive dead    Forward Password Requests to Wall
  systemd-backlight@backlight:intel_backlight.service   loaded    active   exited  Load/Save Screen Backlight Brightness of backlight:intel_backlight
  systemd-backlight@leds:tpacpi::kbd_backlight.service  loaded    active   exited  Load/Save Screen Backlight Brightness of leds:tpacpi::kbd_backlight
  systemd-binfmt.service                                loaded    inactive dead    Set Up Additional Binary Formats
  systemd-boot-system-token.service                     loaded    inactive dead    Store a System Token in an EFI Variable
  systemd-boot-update.service                           loaded    active   exited  Automatic Boot Loader Update
  systemd-firstboot.service                             loaded    inactive dead    First Boot Wizard
  systemd-fsck-root.service                             loaded    inactive dead    File System Check on Root Device
  systemd-fsck@dev-disk-by\x2duuid-CEDC\x2dFFB9.service loaded    active   exited  File System Check on /dev/disk/by-uuid/CEDC-FFB9
  systemd-hwdb-update.service                           loaded    inactive dead    Rebuild Hardware Database
  systemd-journal-catalog-update.service                loaded    inactive dead    Rebuild Journal Catalog
  systemd-journal-flush.service                         loaded    active   exited  Flush Journal to Persistent Storage
  systemd-journald.service                              loaded    active   running Journal Service
  systemd-logind.service                                loaded    active   running User Login Management
  systemd-machine-id-commit.service                     loaded    inactive dead    Commit a transient machine-id on disk
  systemd-modules-load.service                          loaded    active   exited  Load Kernel Modules
  systemd-network-generator.service                     loaded    active   exited  Generate network units from Kernel command line
  systemd-networkd.service                              loaded    active   running Network Configuration
  systemd-quotacheck.service                            loaded    inactive dead    File System Quota Check
  systemd-random-seed.service                           loaded    active   exited  Load/Save Random Seed
  systemd-remount-fs.service                            loaded    active   exited  Remount Root and Kernel File Systems
  systemd-repart.service                                loaded    inactive dead    Repartition Root Disk
  systemd-resolved.service                              loaded    active   running Network Name Resolution
  systemd-rfkill.service                                loaded    inactive dead    Load/Save RF Kill Switch Status
  systemd-sysctl.service                                loaded    active   exited  Apply Kernel Variables
  systemd-sysusers.service                              loaded    inactive dead    Create System Users
  systemd-timesyncd.service                             loaded    active   running Network Time Synchronization
  systemd-tmpfiles-clean.service                        loaded    inactive dead    Cleanup of Temporary Directories
  systemd-tmpfiles-setup-dev.service                    loaded    active   exited  Create Static Device Nodes in /dev
  systemd-tmpfiles-setup.service                        loaded    active   exited  Create Volatile Files and Directories
  systemd-udev-settle.service                           loaded    inactive dead    Wait for udev To Complete Device Initialization
  systemd-udev-trigger.service                          loaded    active   exited  Coldplug All udev Devices
  systemd-udevd.service                                 loaded    active   running Rule-based Manager for Device Events and Files
  systemd-update-done.service                           loaded    inactive dead    Update is Completed
  systemd-update-utmp.service                           loaded    active   exited  Record System Boot/Shutdown in UTMP
  systemd-user-sessions.service                         loaded    active   exited  Permit User Sessions
  systemd-vconsole-setup.service                        loaded    inactive dead    Setup Virtual Console
  user-runtime-dir@1000.service                         loaded    active   exited  User Runtime Directory /run/user/1000
  user@1000.service                                     loaded    active   running User Manager for UID 1000

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
69 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

Last edited by hasnoidea (2022-06-02 10:03:07)

hasnoidea · 2022-06-02 10:13:00

I explicitly set

$ su
# systemctl set-default multi-user

as well, this doesn't fix the input :')

hasnoidea · 2022-06-02 14:27:16

By the way, as Windows drivers from here I had installed the following (ordered by install):
1) Thinkpad Monitor (display) 6.13.0.0
2) Hotkey Features Integration 9.2.0.5
3) Conexant Audio 8.66.88.62
4) Lenovo Power Manager 10.0.145.0
5) Synaptics UltraNav 19.3.4.229
6) AMD Video 30.0.14030.0

The only thing related to input (as I understand it) would be (2) and (5).

hasnoidea · 2022-06-03 01:38:40

I uninstalled the keyboard driver in Windows to force it to reinstall.

The same behaviour of bad input was then observed on the next Windows reboot as well until after login.

Somehow Windows can cope with it the following reboots as well. Just not Arch :-/

I'll try to install some other Linux distros tomorrow to check if some of them work.
Might help pinpoint the issue perhaps if it was only software-side from Arch.

Arch Linux

#1 2022-06-01 08:56:22

Windows driver updates thrash linux input on laptop

#2 2022-06-01 09:31:44

Re: Windows driver updates thrash linux input on laptop

#3 2022-06-01 11:18:22

Re: Windows driver updates thrash linux input on laptop

#4 2022-06-01 12:00:12

Re: Windows driver updates thrash linux input on laptop

#5 2022-06-01 12:02:16

Re: Windows driver updates thrash linux input on laptop

#6 2022-06-01 12:29:18

Re: Windows driver updates thrash linux input on laptop

#7 2022-06-01 13:19:32

Re: Windows driver updates thrash linux input on laptop

#8 2022-06-01 13:41:57

Re: Windows driver updates thrash linux input on laptop

#9 2022-06-01 13:49:30

Re: Windows driver updates thrash linux input on laptop

#10 2022-06-01 17:48:04

Re: Windows driver updates thrash linux input on laptop

#11 2022-06-01 18:34:07

Re: Windows driver updates thrash linux input on laptop

#12 2022-06-01 19:09:35

Re: Windows driver updates thrash linux input on laptop

#13 2022-06-01 19:18:35

Re: Windows driver updates thrash linux input on laptop

#14 2022-06-01 19:43:31

Re: Windows driver updates thrash linux input on laptop

#15 2022-06-01 20:30:44

Re: Windows driver updates thrash linux input on laptop

#16 2022-06-01 20:59:32

Re: Windows driver updates thrash linux input on laptop

#17 2022-06-02 06:02:49

Re: Windows driver updates thrash linux input on laptop

#18 2022-06-02 10:01:28

Re: Windows driver updates thrash linux input on laptop

#19 2022-06-02 10:13:00

Re: Windows driver updates thrash linux input on laptop

#20 2022-06-02 14:27:16

Re: Windows driver updates thrash linux input on laptop

#21 2022-06-03 01:38:40

Re: Windows driver updates thrash linux input on laptop

Board footer