Running multiple X sessions and capturing the output

MrNeuneu · 2021-06-13 11:00:06

Hi,

I am trying to capture the output of a program and send commands to it automatically, in the background.

I can run a different X session by executing "startx -- :1", but I have trouble capturing what happens on this display. I can theoretically capture the output of DISPLAY :1 with obs, but obs says the display is always black while there are graphical applications on it (and I can see them if I activate this display). This does not happen when I'm using Xephyr, for instance on DISPLAY :13, but I don't want to use Xephyr because of the high resource usage. Even with vglrun, it's a lot higher than in a simple X server (and simply unmanageable without). The program "scrot" has the same behaviour than obs.

Is there a way to do what I want?

Last edited by MrNeuneu (2021-06-13 11:02:52)

Trilby · 2021-06-13 13:21:15

By "capture output" it sounds like you mean an image of the X11 window created by the program, right? If so, why would that ever be needed (as it sounds like an X-Y problem)? What is the program?

There are a lot of programs that will do some processing with data and create a visual representation and then display it in an X11 window. But if you goal is to get this output in an image file, there are likely better ways than scrot-ing in the background, just have it rendered to a file in the first place rather than to an X11 display.

seth · 2021-06-13 14:21:00

and I can see them if I activate this display

And you can record them when you activate the display.
You cannot grab an inactive framebuffer, resp. you can, but you'll only get nothing (aka "black") because it has no valid data because it's controlled by a different VT
You'd have to run the X11 server on something that's not your GPU … thus slow for OpenGL.

Unless you've a second GPU, you need to look for a different approach to whatever you're actually trying to do.

MrNeuneu · 2021-06-13 16:37:32

By "capture output" it sounds like you mean an image of the X11 window created by the program, right?

That's right.

If so, why would that ever be needed (as it sounds like an X-Y problem)? What is the program?

It's a game and I've built a bot to play it. But I would prefer not to monopolize my screen to play the game. It would be better if it runs in the background and if I can review what the bot has done afterwards. The bot plays as a human would: it analyses the image produced by the game and sends keystrokes and clicks to the program. (Another problem is that the bot can't even play the game if it is not displayed on the screen ; with Xephyr it works though.)

Unless you've a second GPU, you need to look for a different approach to whatever you're actually trying to do.

If I understand correctly what you are saying, my GPU can only deal with a single screen? But what about external screens? I can plug an external screen to my laptop, and I believe they can both use GPU acceleration. Can this be false? So in theory, it should be possible to do "as if" there was an external monitor, but record the output instead of sending it to this monitor (this will consume some cpu).

seth · 2021-06-13 16:40:58

Multiple outputs do not mean multiple framebuffers.
Try to make one monitor run :0 and one :1…

Edit: you can run the game on one output and use the other for your regular tasks, though.

Last edited by seth (2021-06-13 16:50:59)

MrNeuneu · 2021-06-13 17:01:08

Ok. But I only have one monitor with me... i serached for "virtual" displays and found this: https://bbs.archlinux.org/viewtopic.php?id=180904
I will say if it does what I want when I can.

seth · 2021-06-13 17:05:04

Probably not.
How is this different from running the game in a window on your regular desktop - the bot will likely even work when it's minimized (more likely w/ an active compositor because the window isn't necessarily withdrawn)

MrNeuneu · 2021-06-15 04:24:22

I cannot send X events to minimized windows and I cannot record them or at least I don't know how to. It seems the window needs to be displayed in some way in order to do that.

Here is the workaround I made. First, I create a "false" screen with xrandr as in the link I gave above. I configure my window manager (xmonad) so that each time this screen becomes active, the focus instantly comes back to the "real" screen. This way, one can send events to the other screen without grabbing the focus (if I'm typing something, it should not be impacted). Secondly, I use Multiple X pointers: one pointer that I control with my mouse, and one pointer being controlled by the program. The false screen can be recorder with obs (so that I can actually see what happens on it).

I'm not really confident in the stability of this system because it is very hacky... But so far it works (5 minutes... edit: now 30mn, and the cpu usage is low enough by a big margin).

Last edited by MrNeuneu (2021-06-15 04:47:34)

MrNeuneu · 2021-06-15 04:29:52

I would have prefered to do as you said (one monitor for :0 and one monitor for :1) but I only know how to add a virtual monitor when X is already running... There is also this "multiseat" thing but it seems a bit complex to look into it just for my problem.

Last edited by MrNeuneu (2021-06-15 04:49:53)

Arch Linux

#1 2021-06-13 11:00:06

Running multiple X sessions and capturing the output

#2 2021-06-13 13:21:15

Re: Running multiple X sessions and capturing the output

#3 2021-06-13 14:21:00

Re: Running multiple X sessions and capturing the output

#4 2021-06-13 16:37:32

Re: Running multiple X sessions and capturing the output

#5 2021-06-13 16:40:58

Re: Running multiple X sessions and capturing the output

#6 2021-06-13 17:01:08

Re: Running multiple X sessions and capturing the output

#7 2021-06-13 17:05:04

Re: Running multiple X sessions and capturing the output

#8 2021-06-15 04:24:22

Re: Running multiple X sessions and capturing the output

#9 2021-06-15 04:29:52

Re: Running multiple X sessions and capturing the output

Board footer