xinput and XTest playing nicely

jneth · 2018-12-30 04:55:08

I've installed my first Arch Linux system and have really enjoyed it! The forums and wiki are awesome!

My next question is one I've done a lot of searching on. I don't think anybody else is trying to do it and I just don't have the internal knowledge of xlib and xtest yet.

I'm using Java to make an automated mouse movement scrip to interact with another window. The Java part shouldn't be important, as I'm using JNA to call the XTest methods to move the mouse.

To be able to keep using the computer while I'm running the code I've added a second mouse/kb in the form of a Belkin device. My goal was to get the Belkin to be the primary mouse/keyboard and my normal mouse an keyboard to be secondary so I could still interact with the computer normally.

It took a while to figure out the scripts to automate this, but I successfully generate the xinput I want as follows:

xinput
 Virtual core pointer                    	id=2	[master pointer  (3)]
    Virtual core XTEST pointer              	id=4	[slave  pointer  (2)]
    Belkin Belkin n52te Mouse               	id=15	[slave  pointer  (2)]
    Belkin Belkin n52te Consumer Control    	id=16	[slave  pointer  (2)]
 Virtual core keyboard                   	id=3	[master keyboard (2)]
     Virtual core XTEST keyboard             	id=5	[slave  keyboard (3)]
     Power Button                            	id=6	[slave  keyboard (3)]
     Power Button                            	id=7	[slave  keyboard (3)]
     Sleep Button                            	id=8	[slave  keyboard (3)]
     Belkin Belkin n52te                     	id=14	[slave  keyboard (3)]
     Belkin Belkin n52te System Control      	id=17	[slave  keyboard (3)]
     Belkin Belkin n52te Consumer Control    	id=19	[slave  keyboard (3)]
 Auxiliary pointer                       	id=20	[master pointer  (21)]
    USB OPTICAL MOUSE                       	id=9	[slave  pointer  (20)]
    USB USB Keyboard Mouse                  	id=11	[slave  pointer  (20)]
    USB USB Keyboard Consumer Control       	id=13	[slave  pointer  (20)]
    Auxiliary XTEST pointer                 	id=22	[slave  pointer  (20)]
 Auxiliary keyboard                      	id=21	[master keyboard (20)]
     USB USB Keyboard                        	id=10	[slave  keyboard (21)]
     USB USB Keyboard System Control         	id=12	[slave  keyboard (21)]
     USB USB Keyboard Consumer Control       	id=18	[slave  keyboard (21)]
     Auxiliary XTEST keyboard

I would like to continue improving on this, but have hit a pretty hard block. The secondary mouse/keyboard isn't the greatest. It fights the primary causing me to lose focus a lot and sometimes won't interact with menus (particularly in Eclipse) and system apps (like task manager). I've tried tackling this two ways:

1) I tried to force xlib to use the secondary pointer instead of the primary pointer. After several hours of xinput -set-cp, xdotool, wmctrl and combinations, I was unable to get the Java program to use the secondary mouse my when interacting with XTestFakeButtonPush

2) Look for settings in xinput to make the mice not interfere with each other

I've have absolutely zero success here. I can't seem to find anything that would let multi-pointer be more independent

Would be very appreciative if anyone had any thoughts on improving this. While I'm in a usable state, I usually strive for better than just usable

For understanding, I was doing this on windows by sending commands directly to the window using Java/JNA and USER32 dll. This made the mouse interactions invisible since I was sending the events directly to the window. My attempts to duplicate this on linux all failed miserably, and after reading a ton of forums, the XTest seemed like a better route since windows don't like doing actions when they don't have focus. If there is a better way to handle window events directly, I'd certainly be interested.

I'm using xfce4 as my window manager is that make a difference interacting with xlib...

-- John

seth · 2018-12-30 10:41:42

man xdotool

…
SENDEVENT NOTES
       If you are trying to send key input to a specific window, and it does not appear to be working, then it's likely your
       application is ignoring the events xdotool is generating. This is fairly common.

       Sending keystrokes to a specific window uses a different API than simply typing to the active window. If you specify
       'xdotool type --window 12345 hello' xdotool will generate key events and send them directly to window 12345.
       However, X11 servers will set a special flag on all events generated in this way (see XEvent.xany.send_event in X11's
       manual). Many programs observe this flag and reject these events.

       It is important to note that for key and mouse events, we only use XSendEvent when a specific window is targeted.
       Otherwise, we use XTEST.

       Some programs can be configured to accept events even if they are generated by xdotool. Seek the documentation of
       your application for help.

       Specific application notes (from the author's testing): * Firefox 3 seems to ignore all input when it does not have
       focus.  * xterm can be configured while running with ctrl+leftclick, 'Allow SendEvents' * gnome-terminal appears to
       accept generated input by default.
…

No matter how many pointers you create, there's only one active/focused window at a time and my personal experience is that sent events are vastly ignored for security concerns.
You could test your client in a nested server (xephyr) where it can be the only (and active) client and receive all xdotool pointer events via xtest (eg. "export DISPLAY=:1" to xdotool)

jneth · 2018-12-30 19:33:38

You could test your client in a nested server (xephyr) where it can be the only (and active) client and receive all xdotool pointer events via xtest (eg. "export DISPLAY=:1" to xdotool)

I'm learning vision processing, so I'm using this code to play a game with opencv for java. As soon as I switch display or run in a VNC server, it eats up my CPU like crazy since my GPU isn't helping with the rendering anymore. There is a lot of screen capture and image parsing going on (which is super fun by the way)

I'm pretty functional now, so my goal from here will be to play with combinations of xinput, xdotools and JNA to see if I can come up with a better combo. I think my next try will be to get focus with the mouse and XTestFakeButtonPush and then try to use JNA to send events to a focused window. I hadn't tried to use xdotool directly to send the input, that looks like something to try as well.

Thanks for the suggestions!

-- john

seth · 2018-12-30 20:44:06

Try the CPU load on "Xephyr -glamor :1", it's not a design like VNC.

jneth · 2018-12-31 18:13:25

No go on Xephyr. Still pegs out my CPU when I redirect the display there for the game

Arch Linux

#1 2018-12-30 04:55:08

xinput and XTest playing nicely

#2 2018-12-30 10:41:42

Re: xinput and XTest playing nicely

#3 2018-12-30 19:33:38

Re: xinput and XTest playing nicely

#4 2018-12-30 20:44:06

Re: xinput and XTest playing nicely

#5 2018-12-31 18:13:25

Re: xinput and XTest playing nicely

Board footer