You are not logged in.

#1 2018-12-10 12:04:56

Registered: 2008-11-14
Posts: 1,777

ssh-rdp: zero latency remote desktop suitable for gaming.

Hi all,
i'd like to share with you my work of the last week,
Basically, it is a bash script that allows you to connect to a remote desktop, but has some nice features:

* Everything is done through ssh
* Good video quality thanks to h264 codec
* Latency is zero on localhost, it entirely depends on the network and cpu power.
* Hardware encoding (nvidia only, couldn't test amd or intel)
* Sound support
* Suitable for remote gaming (depends on your network)

From my tests, i've found that to achieve the lowest latencies, the best tool to use on the client is ffplay with some nice flags; when using the script to connect to the local desktop, (dual head setup), i can't really notice any delay!

So ffmpeg on the remote machine grabs the screen, encodes it and sends everything through an ssh pipe.
The same is done for audio, with the help of pulseaudio on the remote side. I've found that sending audio and video on two separate streams, helps with latency.
Keyboard and mouse forwarding is handled by synergy; even here data passes through ssh via port forwarding.

Software needed:
* Local and Remote: ffmpeg,openssh,netevent-git
* Local:  wmctrl
* Remote: xdpyinfo,pulseaudio

It is also mandatory to:
* set-up an ssh key based authentication
* have local and remote users in the input group

How to make it work:
* setup an ssh based authentication with the remote host
* Open the script and edit it to suit your needs
* Start the script from a terminal
* Select which devices you want to share
* Wait some seconds for the remote desktop to appear
* Use ScrollLock key to give keyboard and mouse to remote machine and back
* Use Pause key to switch fullscreen

* Due to some pulseaudio shortcomings, sometimes the audio get desyncronized, it will recover by itself by speeding up, and you can hear high-pitched sound.
* Just clicking in the remote window does not pass the control to it, because that is the ffplay window; so hitting the hotkey is mandatory.
* For remote gaming, turning on cpu encoder may give better results than gpu encoder, if the gpu is already at its limit.

I'm open to suggestions to make this software better!
Notably, it lacks a way to use amd and intel gpu encoders.
If anybody could write and post a VIDEO_ENC string, it would be really nice.


    #Local+Remote: ffmpeg,openssh,netevent-git
    #Local: wmctrl
    #Remote: xdpyinfo,pulseaudio
    #read/write access to input devices on local and remote system (input group) (sudo gpasswd --add username input)

#Restrictions: only one keyboard supported.

#Remote host (you can pass the following via command line in the format:  john@server:22:0.0)
    RHOST=  # Remote ip or hostname
    RPORT=22               # Remote ssh port to connect to
    RUSER=koko             # The user on the remote side running the real X server
    RDISPLAY="0.0"        # The remote display (ex: 0.0)
    ICFILE="$HOME/.config/ssh-rdp.input.config" #where the device names to be forwarded are stored

    #Netevent code, to show a code, use the following:
      #sleep 0.5 ; echo Press a key, twice ; K=$(sudo netevent show /dev/input/by-id/usb-LITEON_Technology_USB_Multimedia_Keyboard-event-kbd |grep KEY|cut -d ":" -f 2|head -n 1) ; echo key is $K

    GRAB_HOTKEY="70" # Grab/Ungrab devices 70=scroll_lock
    FULLSCREENSWITCH_HOTKEY="119" # Switch fullscreen
    AUDIO_CAPTURE_SOURCE="guess" # "pulseaudio name like alsa_output.pci-0000_00_1b.0.analog-stereo.monitor" or "guess"
    FPS=60         # frames per second of the stream
    RES="auto"     # "ex: RES="1280x1024" or RES="auto". 
                   # If wrong, video grab will not work.
    OFFSET=""      # ex: OFFSET="" or OFFSET="+10,+40".
                   # If wrong, video grab will not work.

    AUDIO_BITRATE=128 #kbps
    AUDIO_ENC="-acodec libopus -vbr off -application lowdelay"
    AUDIO_DELAY_COMPENSATION="4500" #The higher the value, the lower the audio delay.
                                    #Setting this too high will likely produce crackling sound.
                                    #Try in range 0-9000
    #nvidia gpu encoder
    #VIDEO_ENC="-threads 1 -c:v h264_nvenc -preset llhq -delay 0 -zerolatency 1"
    #ati gpu encoder
    #intel gpu encoder
    #cpu encoder
    VIDEO_ENC="-threads 1 -vcodec libx264 -thread_type slice -slices 1 -level 32 -preset ultrafast -tune zerolatency -intra-refresh 1 -x264opts vbv-bufsize=1:slice-max-size=1500:keyint=$FPS:sliced_threads=1"

# ### User config ends here ### #

create_ic_file() {
	cd /dev/input/by-id/
	for d in $devices ; do 
		list="$list $d - off"
	dialog --checklist "Choose devices to forward:" 10 80 3 $list  2>$ICFILE

if [ $1 = "inputconfig" ] ; then

if [ ! $1 = "" ] ; then
	read RUSER RHOST RPORT RDISPLAY <<< $(echo "$1" | awk -F [@:] '{print $1" "$2" "$3" "$4}')

#Sanity check	
	me=$(basename "$0")
	if [ -z $RUSER ] || [ -z $RHOST ] || [ -z $RPORT ] || [ -z $RDISPLAY  ] ; then
		echo Missing parameters.
		if [ $1 = "" ] ; then 
			echo Please edit "$me" to suid your needs.
			echo Format: "$me" "user@host:port:DISPLAY"
			echo "    Ex: "$me" john@server:22:0.0"
			echo "   Use: "$me" inputconfig to create or change the input config file"
	if [ ! -f "$ICFILE" ] ; then
		echo "ERROR: Input configuration file "$ICFILE" not found!"
		ecgi "Please, Select which devices to share."
		sleep 2
		echo "Using input configuration file "$ICFILE""
echo Trying to connect to $RHOST:$RPORT as user $RUSER
echo and stream display $DISPLAY

#Shortcut to start remote commands:

#Remote window title

#netevent script file
#We need to kill some processes on exit, do it by name.
    $SSH_EXEC "ln -s \$(which ffmpeg) $FFMPEGEXE"
    $SSH_EXEC "ln -s \$(which ffplay) $FFPLAYEXE"

list_descendants() {
	local children=$(ps -o pid= --ppid "$1")
	for pid in $children ; do
		list_descendants "$pid"
	echo "$children"
#Clean function
finish() {
	echo ; echo TRAP: finish.
	kill $(list_descendants $$)
        rm $NESCRIPT
trap finish INT TERM EXIT

#Test and report net download speed
benchmark_net() {
	$SSH_EXEC sh -c '"timeout 1 dd if=/dev/zero  bs=1b "' | cat - > /tmp/zero
	#KBPS=$(( $(wc -c < /tmp/zero) *8/1000   ))  # 100%
	#KBPS=$(( $(wc -c < /tmp/zero) *8/1200   ))  # 80%
	KBPS=$(( $(wc -c < /tmp/zero) *8/2000   ))  # 50%
	#KBPS=$(( $(wc -c < /tmp/zero) *8/3000   ))  # 33%
	#KBPS=$(( $(wc -c < /tmp/zero) *8/10000   )) # 10%
	echo $KBPS

#Parse remote hotkeys and perform local actions (eg: Fullscreen switching)
setup_input_loop() {    
	echo "Setting up input loop and forwarding devices"
	#Prepare netevent script
	touch $NESCRIPT
	for DEVICE in $(<$ICFILE) ; do
		echo forward input from device $DEVICE...
		if [[ $DEVICE == *"event-kbd"* ]] ; then
			echo "device add mykbd $DEVICE"  >>$NESCRIPT
			echo "device add dev$i $DEVICE"  >>$NESCRIPT
		let i=i+1
	echo "hotkey add mykbd key:$GRAB_HOTKEY:1 grab toggle" >>$NESCRIPT
	echo "hotkey add mykbd key:$GRAB_HOTKEY:0 nop" >>$NESCRIPT
	echo "hotkey add mykbd key:$FULLSCREENSWITCH_HOTKEY:1 exec \"/usr/bin/echo FULLSCREENSWITCH_HOTKEY\"" >>$NESCRIPT
	echo "hotkey add mykbd key:$FULLSCREENSWITCH_HOTKEY:0 nop" >>$NESCRIPT
	echo "output add myremote exec:$SSH_EXEC netevent create" >>$NESCRIPT
	echo "use myremote" >>$NESCRIPT

	netevent daemon -s $NESCRIPT netevent-command.sock | while read -r hotkey; do
	echo "read hotkey: " $hotkey
	if [ "$hotkey" = "FULLSCREENSWITCH_HOTKEY" ] ; then
		if [ "$FS" = "F" ] ; then
		wmctrl -b add,fullscreen -r "$WTITLE"
		wmctrl -b add,above -r "$WTITLE"
		wmctrl -b remove,fullscreen -r "$WTITLE"
		wmctrl -b remove,above -r "$WTITLE"
    setup_input_loop &

#Play a test tone to open the pulseaudio sinc prior to recording it to (avoid audio delays at start!?)
    #$SSH_EXEC sh -c 'export SDL_AUDIODRIVER=pulse ; ffplay -nostats -nodisp -f lavfi -i "sine=220:4" -af volume=0.001 -autoexit' &

#Measure network download speed?
if [ "$VIDEO_BITRATE_MAX" = "AUTO" ] ; then
	echo "Measuring network throughput"
	if [ $VIDEO_BITRATE_MAX -gt 294987 ] ; then
		echo $VIDEO_BITRATE_MAX too high!
	echo Using "$VIDEO_BITRATE_MAX"Kbps

#Guess audio capture device?
    if [ "$AUDIO_CAPTURE_SOURCE" = "guess" ] ; then
        AUDIO_CAPTURE_SOURCE=$($SSH_EXEC echo '$(pacmd list | grep "<.*monitor>" |awk -F "[<>]" "{print \$2}" | tail -n 1)')
        echo "Guessed audio capture source:" $AUDIO_CAPTURE_SOURCE
#Auto video grab size?
    if [ "$RES" = "auto" ] ; then
        RES=$($SSH_EXEC "export DISPLAY=$RDISPLAY ; xdpyinfo | awk '/dimensions:/ { print \$2; exit }'")
        echo "Auto grab resolution: $RES"

#Grab Audio
    $SSH_EXEC sh -c "\
        export DISPLAY=$RDISPLAY ;\
        $FFMPEGEXE -v quiet -nostdin -y -f pulse -ac 2 -i "$AUDIO_CAPTURE_SOURCE"  -b:a "$AUDIO_BITRATE"k "$AUDIO_ENC" -f nut -\
    " | \
    ffplay - -nostats -flags low_delay -nodisp -probesize 32 -fflags nobuffer+fastseek+flush_packets -analyzeduration 0 -sync ext -af aresample=async=1:min_comp=0.1:first_pts=$AUDIO_DELAY_COMPENSATION &

#Grab Video
    $SSH_EXEC sh -c "\
        export DISPLAY=$RDISPLAY ;\
        $FFMPEGEXE -nostdin -y -f x11grab -r $FPS -framerate $FPS -video_size $RES -i "$RDISPLAY""$OFFSET" -maxrate "$VIDEO_BITRATE_MAX"k \
        "$VIDEO_ENC" -f_strict experimental -syncpoints none -f nut -\
    " | \
    ffplay - -nostats -window_title "$WTITLE" -probesize 32 -flags low_delay -framedrop  -fflags nobuffer+fastseek+flush_packets -analyzeduration 0 -sync ext

VIDEO_BITRATE_MAX is set to 10000 (about 8Mbps), but you can choose "AUTO" instead.
This will cause the script to estimate the network throughput, and by default it will use 50% of it.
(search for the lines starting with  "KBPS=$(( $(wc -c < /tmp/zero) " and uncomment/comment according to your needs)

...actually i'm having success in remotely playing:
- "The turing test" via steamplay
- nvidia gpu encoder
- stream resolution is 1280x720@30fps
- cpu on both sides is Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
- gpu on remote side is an nvidia GTX 750ti
- remote fps is about 60fps
- maximum bandwidth used: 10Mbps

Last edited by kokoko3k (2019-01-02 15:17:18)


#2 2019-01-02 15:12:50

Registered: 2008-11-14
Posts: 1,777

Re: ssh-rdp: zero latency remote desktop suitable for gaming.

Just edited and partially rewrote the script, since i discovered the great netevent and made an aur package too.
With netevent i removed synergy,xbindkeys and xdotool dependancies, so the script is much simpler now.
As a bonus, you can now share your local gamepad too!
Note that the local user and the remote one have to be in the input group (a logout may needed after modifying the group membership)

Last edited by kokoko3k (2019-01-02 15:14:03)


Board footer

Powered by FluxBB