Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

kokoko3k · 2012-01-23 16:14:31

D you know Onlive? Here is a DIY howto, this is what i'm using to play games remotely on a poor netbook

Required packages:
Server: virtualgl,ffmpeg,socat,sshd
Client: virtualgl,mplayer,ssh,socat

Server: in /etc/ssh/sshd_config:
"AcceptEnv ALSA_CARD"

The user who wants to play remotely has to execute the following script on (say) the poor netbook:
(configure $user,$server and $sshport first), and as soon as it gains access to the remote shell, it has to execute:
vglrun (parameters) /path/to/3dgame

Main Script

#!/bin/bash
user=remote_username
server=remote_server
sshport=22

ServerSetup="\
killall socat ffmpeg ; sleep 1 ; killall -9 socat ffmpeg; \
echo inserting loopback module, ask for sudo pass:; \
sudo modprobe snd-aloop; \
socat UDP4-LISTEN:6000,fork,reuseaddr TCP:127.0.0.1:5000 &>/dev/null & \
 ffmpeg -f alsa -ac 2 -i hw:Loopback,1,0 -acodec libmp3lame -b 128k -f rtp rtp://127.0.0.1:6000 &>/dev/null\
"

  echo "Executing ServerSetup on " $Server:$sshport ...
  ssh -f $user@$server -oPort=$sshport "$(echo $ServerSetup)"

  killall mplayer socat ; sleep 1 ; killall -9 mplayer socat
  echo "o=- 0 0 IN IP4 127.0.0.1" >/tmp/stream.sdp
  echo "c=IN IP4 127.0.0.1" >>/tmp/stream.sdp
  echo "m=audio 6000 RTP/AVP 14" >>/tmp/stream.sdp
  socat TCP4-LISTEN:5000,fork,reuseaddr UDP4:localhost:6000 &>/dev/null &
  sh -c "sleep 20 ; mplayer /tmp/stream.sdp -really-quiet </dev/null" &
  ALSA_CARD=Loopback vglconnect -s $user@$server -o SendEnv=ALSA_CARD -p $sshport -R localhost:5000:localhost:5000

Explaination
Graphic streaming is very easy and totally managed by virtualgl as video data is taken from the server by using a seamless VirtualGL window,
vglconnect (part of virtualgl package) is an ssh wrapper that set-up port forwarding to tunnel frames,
no problem here for nat environments.

When comes to audio, the 3d application on the server will output all audio data to a virtual loopback device,
this is done by modprobing snd-aloop and then set the environment variable ALSA_CARD to "Loopback"
Sound is then encoded by ffmpeg into an mp3 streamed via rtp udp transport for lower delay.
Finally, that audio will be played back on the netbook via a background mplayer.
It will take as argument a text file (compiled by a bunch of echo runtime) describing the stream.
(ffplay would do it as well, but how to disable the spectrum visualization?)

Making audio work for client behind a nat is somehow tricky, because ssh only forwards TCP connections and our stream works over UDP,
but with socat we will 'convert' UDP to TCP (ssh here!) to UDP again so that we can use ssh tunnelling facilities while in the TCP domain:
server: ffmpeg will stream to localhost, udp port 6000
server: socat will forward from localhost, udp port 6000 to localhost tcp port 5000
server: vglconnect (ssh wrapper) will forward all tcp traffic directed to localhost:5000(tcp) to the remote endpoint (client), port 5000(tcp)
client: socat will forward from localhost, tcp port 5000 to localhost udp port 6000
client: finally mplayer will play from localhost port 6000

Note that mplayer will run on the client, but the last step we'll do in the script is to open an ssh connection into the server,
and that connection is necessary for mplayer to work as we'll setup port redirection there.
So, to keep things easy from the user point of view, mplayer is started early in background with a delay (20secs)
(read: your audio will start after a while)

If you are not behind a nat or prefer to use a vpn, you can (but it is not necessary at all)
* comment all of the socat commands,
* point ffmpeg streaming to the client ip,
* change the line that contains "c=IN IP4 127.0.0.1" to the client ip too.

Drawbacks of the script:
.It will kill any socat,ffmpeg and mplayer instances at startup (any idea?)
.No process has to listen on port 5000 and 6000 (tcp,udp), change the script or use some vars if you need it.
.Some programs (eg: braid) don't like alsa loopback device and outputs some garbage sound
.ALSA_CARD trick isn't going to work if you forced a !default output device in your .asoundrc (server side)

What is missing:
.A way to terminate leftover processes when we're done with playing

Highly suggested:
.remove "&>/dev/null" occurrences from the script if you run into problems
.Setup ssh key authentication
.Avoid sound skip by running ffmpeg with realtime priority (schedtool -n -19 -F -p 10 -e ffmpeg...)

Side notes:
.This thing doesn't work very well with virtualized environments, virtualbox client on the same server machine performs poorly, think that a real n280 netbook over a real 10Mbps link is smoother.
.Audio latency is about 300ms (i know it is not THAT low, but acceptable)

example code for a 10Mbit network which take about 600KBps at 1024x600

vglrun -np 2 -c jpeg -q 40 -samp 1 -fps 20 ./aquaria

-np 2 = use 2 threads to encode frames
-q 40 = jpeg quality=40
-samp 1 = Chrominance subsampling factor
-fps 20 = limit the framerate to 20fps

Last edited by kokoko3k (2012-01-24 14:11:44)

Ken · 2012-02-26 07:53:06

Hi kokoko3k thanks for the script. I had some troubles to start with, but hacked around and got it going.
I am using AMD64 Ubuntu based distributions for both client and server.
Anyways I thought I would post the (slightly) modified script in case it helps others out there.

#!/bin/bash

#Prereqs:

#Server: virtualgl,ffmpeg,socat,sshd,libavcodec-extra-53
#Client: virtualgl,mplayer,ssh,socat

#Server: in /etc/ssh/sshd_config:
#"AcceptEnv ALSA_CARD"

#Server: add to /etc/sudoers file:
#<user> ALL=(ALL) NOPASSWD: /sbin/modprobe

user=<user>
server=<Server Hostname>
sshport=22

ServerSetup="\
killall socat ffmpeg ; sleep 1 ; killall -9 socat ffmpeg; \
sudo modprobe snd-aloop; \
socat UDP4-LISTEN:6000,fork,reuseaddr TCP:127.0.0.1:5000 &>/dev/null & \
 ffmpeg -f alsa -ac 2 -i hw:Loopback,1,0 -acodec libmp3lame -b 128k -f rtp rtp://127.0.0.1:6000 &>/dev/null\
"

  echo "Executing ServerSetup on " $server:$sshport ...
  ssh -f $user@$server -oPort=$sshport "$(echo $ServerSetup)"

  killall vglclient mplayer socat ; sleep 1 ; killall -9 vglclient mplayer socat
  echo "o=- 0 0 IN IP4 127.0.0.1" >/tmp/stream.sdp
  echo "c=IN IP4 127.0.0.1" >>/tmp/stream.sdp
  echo "m=audio 6000 RTP/AVP 14" >>/tmp/stream.sdp
  socat TCP4-LISTEN:5000,fork,reuseaddr UDP4:localhost:6000 &>/dev/null &
  sh -c "sleep 60 ; mplayer /tmp/stream.sdp -really-quiet </dev/null" &
  ALSA_CARD=Loopback /opt/VirtualGL/bin/vglconnect -x -s $user@$server -o SendEnv=ALSA_CARD -p $sshport -R localhost:5000:localhost:5000

Last edited by Ken (2012-02-27 18:18:33)

kokoko3k · 2012-02-26 09:29:11

Seems your changes just touches the mplayer start delay and vgl bin path.
Good point in terminating vglconnect too!

theGunslinger · 2012-02-26 12:25:44

So what do you think what the optimal upload bandwidth on the server should be to make this possible?

kokoko3k · 2012-02-26 13:17:35

"optimal" ? as much as you can!

you can tweak the video quality and/or fps depending on how much bandwidth you have.
At the end of the first post, there is an example with jpeg quality=40;
compress a jpeg at quality=40 (1024x600) with a graphic tool to have an idea.

Earnestly · 2012-02-26 15:00:01

Holy balls batman, add this to the wiki

Last edited by Earnestly (2012-02-26 15:01:53)

Ken · 2012-02-26 18:35:54

kokoko3k wrote:

Seems your changes just touches the mplayer start delay and vgl bin path.
Good point in terminating vglconnect too!

Yep the bulk of the script is unchanged.
Here is a summary of the problems I encountered:

- ffmpeg could not find the libmp3lame codec, I have to install the libavcodec-extra-53 package (Ubuntu specific problem I suspect)
- The ServerSetup subscript could not handle the password for sudo modprobe, so I had to add the entry into /etc/sudoers (Not very secure, but could not find a better way)
- I added the -x switch to the vglconnect command for a more efficient (unencrypted) direct X11 connection to the client. The client may need to be configured to accept direct X11 connections.

After the changes, it worked pretty well, but occasionally cut out for a few seconds. I think this is just the video eating up to much bandwidth though.

My Virtualgl server is my media box, so one unexpected feature of your script is that it allows simultaneous audio from the server's speakers and my client.
i.e. I can watch a movie and play a game the same time.

Anyways good stuff and thanks again.

Last edited by Ken (2012-02-27 18:20:07)

jar00n · 2012-07-19 05:29:43

I'm pretty new to socat and I can get everything but the socat connections to work. I think it's an iptables issue but I allowed both 5000 and 6000 on tcp and udp on both machines and I get:

Server side: 2012/07/19 01:26:24 socat[3486] E write(4, 0x827feb0, 1168): Broken pipe

Client side: 2012/07/19 01:26:25 socat[2455] E read(3, 0x8ce9878, 8192): Connection refused

No idea how to approach this, been reading, researching, and pounding away at this problem for a few days...

Googling doesn't provide much if anything and the man page for socat isn't pointing me in any direction, at this point I'm sure it's a simple fix but I can't see it.

Any advice?

Last edited by jar00n (2012-07-19 05:32:00)

kokoko3k · 2012-07-19 11:57:09

It is hard to 'debug' your issue.
You may want to try to workaround all of the network issues by set up a vpn p2p connection by using openvpn.
As soon as i have time, i'll update the main post with an alternate version that uses that approach.

The drawback will be a 5% network bandwidth overhead, but fortunately the audio latency will be the same.

Last edited by kokoko3k (2012-07-19 12:00:30)

jar00n · 2012-07-19 16:01:18

I believe I narrowed down the socat issue to the final step where you pull the udp out of the tcp, this stage is where I am getting the connection refused, I think this is where my limited expertise is becoming a fault. From what I can tell, I cannot redirect the incoming traffic from tcp port 5000 to udp port 6000.

I haven't opened this machine up on my router yet so I can't debug it from work, I have been looking into the vpn option and it looks like a pptp via ssl would be the ideal way to handle it. But I would prefer the socat option if possible.

kokoko3k · 2012-07-20 13:43:27

Well, just in case someone need it, as already covered the unfirewalled/unNatted part in the main post,
here's how to quiclky setup a p2p vpn connection between "client" and "server" (change every occurrence of client and server to the respective ip address)

client:

pacman -Sy openvpn #just the first time
openvpn --genkey --secret ~/secret.key #just the first time
scp ~/secret.key server:/tmp/secret.key #just the first time
sudo openvpn --dev tun --ifconfig 192.168.10.201 192.168.10.200 --remote server 65530  --proto udp --ping 10 --ping-restart 60 --float --secret  ~/secret.key --log /tmp/vpnclient.txt --daemon

Server (first Open the udp port 65530 for incoming connections):

pacman -Sy openvpn #just the first time
mv /tmp/secret.key ~/secret.key #just the first time
sudo openvpn --dev tun --ifconfig 192.168.10.200 192.168.10.201 --port 65530  --proto udp --ping 10 --ping-restart 60 --float --secret  ~/secret.key --log /tmp/vpnserver.txt --daemon

Do that prior of the main script and you should be able to ping client and server each other via 192.168.10.201 and 192.168.10.200 addresses.

finn · 2012-11-08 13:35:25

hi all!

kokoko3k, could you please recommend a how-to to get virtualgl works?

kokoko3k · 2012-11-08 14:06:15

What do you mean? After you installed it from aur, just type vglrun followed by the opengl executable you want to run.
type vglrun -h for options.
example:

vglrun -np 2 -c jpeg -q 40 -samp 1 -fps 20 ./aquaria

finn · 2012-11-08 14:51:13

ok, thanks. just for best understand - i have debian wheezy, and when i search possible solution to make own onlive using my home computers - i found given topic.

Reded · 2012-11-10 22:12:28

I know this is a bit of a stretch, but is there a way to do this using an Android device as a client? I considered using the ol' Ubuntu installer and doing it off that but generally the framerates on that are a bit too low thanks to the use of VNC.

Maybe I'll do some messing using the X server and connectbot

Really interesting idea though, I like OnLive as a service, just wish they had more games

EDIT: Something tells me we probably can't get virtualgl on Android, shame

Last edited by Reded (2012-11-10 22:13:52)

kokoko3k · 2012-11-11 12:01:13

Check if android can run turbovnc.
Nothing stops you from wunning turbovnc on the server and connect to it.
virtualgl provides gl extensions to vnc session too; that's all...

kokoko3k · 2014-09-04 10:39:28

I'm experimenting with xpra instead of vglconnect.
It seems to provide much lower latencies, bandwidth auto-adjustments and it is easy to setup.
It also forwards audio (via pulse, that i don't use so i didn't tested).
Unfortunately there is a critical bug that won't allow more than one key presses at once, but it is being investigated and hopefully fixed in the next xpra revision.
I'll update this thread as soon as i'll get a working and stable setup.

For those who want to experiment, that's what i'm using to test.
From the client:

kokonicki@netbook ~ $ cat scripts/xpra.gozer.sh 
#!/bin/bash
xpra start ssh:root@Gozer:1010 --ssh "ssh -c arcfour -oPort=41122" --exit-with-children \
--no-speaker --no-microphone --no-pulseaudio \
--encoding=h264 \
--quality=0 \
--min-quality=0 \
--speed=100 \
--auto-refresh-delay=5000 \
--start-child="$@"

Change Gozer to the hostname hosting your xpra server and the ssh port from 4122 to 22 or whatever your ssh server runs at.
Then install xpra on the server and from the client start the script that way:

scripts/xpra.gozer.sh "/path/to/my/game -and -arguments"

Last edited by kokoko3k (2014-09-04 10:40:29)

Arch Linux

#1 2012-01-23 16:14:31

Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#2 2012-02-26 07:53:06

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#3 2012-02-26 09:29:11

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#4 2012-02-26 12:25:44

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#5 2012-02-26 13:17:35

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#6 2012-02-26 15:00:01

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#7 2012-02-26 18:35:54

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#8 2012-07-19 05:29:43

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#9 2012-07-19 11:57:09

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#10 2012-07-19 16:01:18

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#11 2012-07-20 13:43:27

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#12 2012-11-08 13:35:25

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#13 2012-11-08 14:06:15

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#14 2012-11-08 14:51:13

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#15 2012-11-10 22:12:28

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#16 2012-11-11 12:01:13

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

#17 2014-09-04 10:39:28

Re: Run 3d games remotely (even behind a nat) with sound (Onlive DiY)

Board footer