Arch Linux

kero

Member

Registered: 2012-08-26

Posts: 49

[solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

Hi everybody,

In the school where I work, I connect my laptop to a public wifi network on which I have to identify through a webpage (a product of StormShield Network Security IIRC).

On this laptop, on booting, I set up an autossh tunnel to one of my home's devices (an Intel NUC that I use for several purposes). Autossh starts immediately after boot through a systemd (--user) service, when my user logs into the laptop. Basicly it works without problems (if I'm already identified to the school network) but I experience a strange behaviour when the laptop is still not identified (typically, boot time). I'd like to understand.

1. autossh logs are telling me that I'm connected to the remote (home)server. How is that possible considering that the laptop has still not access to the internet though is already connected to the local network ?

myuser@mylaptop ~ $ autossh -M 0 -N -vvv -oServerAliveInterval=60 -oServerAliveCountMax=3 > -o ExitOnForwardFailure=yes -L localhost:22000:localhost:22000 -R localhost:22003:localhost:22003 kdomserv.mynetwork
[...]
debug3: ssh_connect_direct: entering debug1: Connecting to 82.65.XXX.XXX [82.65.XXX.XXX] port 2202.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.

2. in the same time, autossh is failing to establish an ssh tunnel (which is logical since I'm still not connected):

myuser@mylaptop ~ $ sudo lsof -Pi -n | grep -E '<ssh>'
ssh 594 myuser 3u IPv4 7456 0t0 TCP 172.20.XXX.XXX:56762->82.65.XXX.XXX:2202 (ESTABLISHED)

3. BUT: in my understanding, the "-o ExitOnForwardFailure=yes" part of my autossh command (see point 1) should force autossh to exit and try to connect again until it builds a successful tunnel. Which it doesn't (there's nothing in the logs). It also doesn't when I finally identify my laptop on the school's network.

So, my question is:
- how can autossh say it is connected to a remote server with which it can't actually communicate ?
- why is the ExitOnForwardFailure not forcing autossh to stop/restart ?

Last edited by kero (2025-01-15 21:42:57)

-thc

Member

Registered: 2017-03-15

Posts: 775

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

In the school where I work, I connect my laptop to a public wifi network on which I have to identify through a webpage (a product of StormShield Network Security IIRC).

What kind of basic access (DNS, IPv4/IPv6 to certain hosts/services) a firewall allows while you are still unidentified, depends on the product.

For a while I used "iodine" which allowed me low-bandwith free access in blocked WiFis by tunneling the complete IP conversation via DNS packets.

On this laptop, on booting, I set up an autossh tunnel to one of my home's devices (an Intel NUC that I use for several purposes). Autossh starts immediately after boot through a systemd (--user) service, when my user logs into the laptop. Basicly it works without problems (if I'm already identified to the school network) but I experience a strange behaviour when the laptop is still not identified (typically, boot time). I'd like to understand.

I fail to understand the difference in the two scenarios "working" and "not working": If autossh is started after logon - how are you already identified to the WiFi? Is this a "Suspend-to-RAM" wake-up case?

1. autossh logs are telling me that I'm connected to the remote (home)server. How is that possible considering that the laptop has still not access to the internet though is already connected to the local network ?

myuser@mylaptop ~ $ autossh -M 0 -N -vvv -oServerAliveInterval=60 -oServerAliveCountMax=3 > -o ExitOnForwardFailure=yes -L localhost:22000:localhost:22000 -R localhost:22003:localhost:22003 kdomserv.mynetwork
[...]
debug3: ssh_connect_direct: entering debug1: Connecting to 82.65.XXX.XXX [82.65.XXX.XXX] port 2202.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.

This clearly shows an outgoing connection to an IPv4 host in the ProXad/Free SAS IP range in France. Is that IPv4 address your home address? If yes, the laptop has internet access despite being unidentified...

2. in the same time, autossh is failing to establish an ssh tunnel (which is logical since I'm still not connected):

myuser@mylaptop ~ $ sudo lsof -Pi -n | grep -E '<ssh>'
ssh 594 myuser 3u IPv4 7456 0t0 TCP 172.20.XXX.XXX:56762->82.65.XXX.XXX:2202 (ESTABLISHED)

What are you showing us here? There clearly is an established connection - what should we see instead?

3. BUT: in my understanding, the "-o ExitOnForwardFailure=yes" part of my autossh command (see point 1) should force autossh to exit and try to connect again until it builds a successful tunnel. Which it doesn't (there's nothing in the logs). It also doesn't when I finally identify my laptop on the school's network.

The firewall may block or filter out all kinds of traffic - that's hard to predict.

Last edited by -thc (2025-01-11 17:52:18)

kero

Member

Registered: 2012-08-26

Posts: 49

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

Thank you for your answer.

I fail to understand the difference in the two scenarios "working" and "not working": If autossh is started after logon - how are you already identified to the WiFi? Is this a "Suspend-to-RAM" wake-up case?

Sorry if I wasn't clear.
When I boot the laptop, I log then my user in an XFCE session. At that moment, my laptop is already connected (by Networkmanager.service) to the school's local network (wifi, via DHCP configuration), but has no internet access. The login into XFCE also launches autossh (via a systemd user service), which is unable to establish an SSH tunnel (see infra). That's the "not working" scenario.
After that, I open the browser, and the home page is automatically redirected to a local identification service. There I have to give my local network's credentials and after that, I can access the internet. Now if I restart the autossh systemd service, it is now able to build the SSH tunnel (I see immediately that the tunnel is open, since that allows syncthing to connect to the other devices).

This clearly shows an outgoing connection to an IPv4 host in the ProXad/Free SAS IP range in France. Is that IPv4 address your home address? If yes, the laptop has internet access despite being unidentified...

As soon as I'm back in the school, I'll try to launch a basic SSH session and see what happens. Still, SSH is unable to launch a tunnel, which leads me to your last comment:

What are you showing us here? There clearly is an established connection - what should we see instead?

While working on my tunnel, I searched on the internet the best way to monitor the effectiveness of an SSH tunnel, and it turned out that "lsof... | grep ssh" was the way to go (example: https://superuser.com/questions/248389/ … h-tunnels).

Theorically a working tunnel should give that kind of result:

myuser@mydesktop ~ $ sudo lsof -Pi -n | grep -E '\<ssh\>'
ssh       82609 myuser   3u  IPv4 950584      0t0  TCP 192.168.1.10:39358->82.65.XXX.XXX:2202 (ESTABLISHED)
ssh       82609 myuser   4u  IPv6 950587      0t0  TCP [::1]:22000 (LISTEN)
ssh       82609 myuser   5u  IPv4 950588      0t0  TCP 127.0.0.1:22000 (LISTEN)

The first line states that there is an SSH connection, the second/third indicate the port redirection, which means the tunnel is active (here localhost:22000 is redirected to remote:22000). That example comes from my desktop, where i just launched "autossh -M 0 -N -vvv -oServerAliveInterval=60 -oServerAliveCountMax=3 -o ExitOnForwardFailure=yes -L localhost:22000:localhost:22000 kdomserv.crevette".

Last edited by kero (2025-01-11 21:28:01)

-thc

Member

Registered: 2017-03-15

Posts: 775

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

Thank you for clarifying things - it wasn't clear (to me at least ) that you meant the absence of the local SSH redirections.

Be aware that without a technical manual from your school's firewall my analysis is somewhat predictive .

When you establish a SSH tunnel you have to pass two handshakes:

- The outer TCP protocol handshake consists of a fixed 3-way SYN/SYN-ACK/ACK handshake that establishes the TCP connection.
- The inner handshake consists of 5 steps of cyptographically secured data exchange inside the TCP connection for establishing the working SSH tunnel.

Based on your information the school's firewall accepts a complete TCP handshake from a "non-identified" PC and your machine clearly states that the outer handshake was successful (state: ESTABLISHED). But the firewall blocks the actual data transport inside this TCP connection in such a way that openssh "hangs" at a specific point of the inner handshake and waits (possibly forever) for an answer.

This specific hang is not covered by the "ExitOnForwardFailure=yes" option. It's possible that SSH can not recover from it on it's own.

kero

Member

Registered: 2012-08-26

Posts: 49

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

Thank you -thc, that's very interesting. It isn't possible to have more detailed informations about the school's network configuration. Our public authorities doesn't communicate about that of things.

Basing on your explanations, I should find a way to force a periodic autossh reload in case the connection to internet is impossible. I just need to find the right way. (If somebody has an idea...)

Edit: I'll try to set up a ping test in the bash script launching autossh.

Last edited by kero (2025-01-12 13:34:54)

kero

Member

Registered: 2012-08-26

Posts: 49

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

To close this thing...

I tried a lot of solutions, based on autossh at first, then switching to ssh but adding a "restart" directive to my systemd service file. Nothing worked.

The solution that, in the end, works, is to test if I have a valid route with ping (and then, test regularly if the route is still valid, since the firewall only gives a temporary connection).

Thank you again to @-thc for ist help !

bulletmark

Member

From: Brisbane, Australia

Registered: 2013-10-22

Posts: 681

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

FYI, four years ago I wrote a post quite related to what you are doing: https://dev.to/bulletmark/create-a-reve … chine-1ma0

kero

Member

Registered: 2012-08-26

Posts: 49

Re: [solved]autossh/ssh failing ... tunnel despite -o ExitOnForwardFailure

FYI, four years ago I wrote a post quite related to what you are doing: https://dev.to/bulletmark/create-a-reve … chine-1ma0

This is nice. Gives me ideas to improve my tunnel.

Thankyou.