You are not logged in.

#1 2019-11-26 05:00:25

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

[SOLVED] ssh works locally, times out over internet

I've been working on setting up an ssh server on my desktop, and I want to be able to ssh into it from my laptop over the internet.

Everything is working when I do the following from my laptop on my local network:

ssh -p <ssh-port> -i <private-key> user@<local-network-ip-of-desktop>

To get it exposed on the internet, I went into my router's settings to obtain its current public IP address, and have set up Dynamic DNS to assign a domain name to this router's dynamic IP. I have also set up port forwarding for <ssh-port>, and have ensured that my desktop (my ssh server) has a fixed IP on the local network, so that the port forwarding should always be pushed to the correct machine (desktop/ssh server).

If I try to run the ssh command to either the hard-coded public IP of the router or the dynamic DNS name for the router, I get a connection timeout, though.

Does anyone have any idea why this might be happening and/or any possible solutions? Thanks smile

Last edited by hoWlExat (2019-12-02 06:52:42)

Offline

#2 2019-11-26 05:24:01

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

Run `tcpdump -lnnieth0 port 22` on the server while you try to connect from the outside (replace eth0 with your actual network interface).

If you can see the packets there, then your problem is on the box. If the packets don't reach there, then the problem upstream (your router/port forwarding/ISP).

Offline

#3 2019-11-26 05:35:42

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

@fukawi2, I am seeing lines pop up of the following form:

me@laptop $ tcpdump -lnni<network-interface> port <ssh-port>
<time> IP <some-ip> > <local-network-ip-of-desktop>.<ssh-port> Flags [S], seq ..., win 29200, options [mss 1104,sackOK,TS val ... ecr 0,nop,wscale 7], length 0

I don't know what would be going wrong, though, if it reaches my machine, since I can ssh into it over the local network just fine.

Last edited by hoWlExat (2019-11-26 05:37:09)

Offline

#4 2019-11-26 05:37:55

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

That's the only output?

What is the output of `iptables-save` and/or `nft list ruleset`?

Offline

#5 2019-11-26 05:53:27

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

There are some chains showing up there that I'd defined before, but I don't know how to get rid of them (LOG_AND_DROP, STATE0, STATE1, STATE2, STATE3). Those were from when I was trying to set up port knocking, but that's going to be my next step. For the purposes of this current question, we can assume it's fine to blow those away (but I don't know how, currently).

Remember that I can ssh in successfully over the local network (I don't know if that applies here). Thank you for the help!

me@laptop $ iptables-save
# Generated by iptables-save v1.8.3 on <date>
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:LOG_AND_DROP - [0:0]
:STATE0 - [0:0]
:STATE1 - [0:0]
:STATE2 - [0:0]
:STATE3 - [0:0]
:xvpn - [0:0]
:xvpn_dns - [0:0]
:xvpn_dns_iface_exceptions - [0:0]
:xvpn_dns_ip_exceptions - [0:0]
:xvpn_ks - [0:0]
:xvpn_ks_iface_exceptions - [0:0]
:xvpn_ks_ip_exceptions - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name DEFAULT --mask 255.255.255.255 --rsource -j DROP
-A INPUT -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --set --name DEFAULT --mask 255.255.255.255 --rsource
-A INPUT -p tcp -m tcp --dport <ssh-port> -m state --state NEW -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on <date>

Last edited by hoWlExat (2019-11-26 05:54:03)

Offline

#6 2019-11-26 06:03:11

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

I don't see anything there at a quick glance that looks like it would interfere - but just to eliminate it, can you run these commands and try again?

iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F

Offline

#7 2019-11-26 06:07:59

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

If I do that, then I get the same output as before with `tcpdump` when trying to ssh over the internet.

My `iptables-save` output is updated to:

# Generated by iptables-save v1.8.3 on <date>
*filter
:INPUT ACCEPT [5:2385]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [5:2385]
:LOG_AND_DROP - [0:0]
:STATE0 - [0:0]
:STATE1 - [0:0]
:STATE2 - [0:0]
:STATE3 - [0:0]
:xvpn - [0:0]
:xvpn_dns - [0:0]
:xvpn_dns_iface_exceptions - [0:0]
:xvpn_dns_ip_exceptions - [0:0]
:xvpn_ks - [0:0]
:xvpn_ks_iface_exceptions - [0:0]
:xvpn_ks_ip_exceptions - [0:0]
COMMIT
# Completed on <date>

Last edited by hoWlExat (2019-11-26 06:08:42)

Offline

#8 2019-11-26 06:12:57

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

What's the output of `ss -tunlp`?
And it would be helpful to see the uncensored output of tcpdump along with that if you can.

Offline

#9 2019-11-26 06:12:58

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 441

Re: [SOLVED] ssh works locally, times out over internet

Are you running the ssh test from a client from within the same local LAN you are trying to forward back to? Some home routers allow this, some don't, and yet some others require a special setting on the port forward to allow it. My router calls the setting "Enable LAN loopback".

Offline

#10 2019-11-26 06:26:16

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

@fukawi2

me@laptop $ ss -tunlp
Netid                        State                         Recv-Q                        Send-Q                                                     Local Address:Port                                                Peer Address:Port                                                                                        
udp                          UNCONN                        0                             0                                                           192.168.1.12:68                                                       0.0.0.0:*                           users:(("dhcpcd",pid=872,fd=11))                                
udp                          UNCONN                        0                             0                                                    192.168.1.12%enp7s0:68                                                       0.0.0.0:*                           users:(("NetworkManager",pid=788,fd=18))                        
udp                          UNCONN                        0                             0                                                            10.127.0.62:123                                                      0.0.0.0:*                           users:(("ntpd",pid=878,fd=20))                                  
udp                          UNCONN                        0                             0                                                           192.168.1.12:123                                                      0.0.0.0:*                           users:(("ntpd",pid=878,fd=19))                                  
udp                          UNCONN                        0                             0                                                              127.0.0.1:123                                                      0.0.0.0:*                           users:(("ntpd",pid=878,fd=18))                                  
udp                          UNCONN                        0                             0                                                                0.0.0.0:123                                                      0.0.0.0:*                           users:(("ntpd",pid=878,fd=17))                                  
udp                          UNCONN                        0                             0                                                                0.0.0.0:40610                                                    0.0.0.0:*                           users:(("openvpn",pid=1173,fd=4))                               
udp                          UNCONN                        0                             0                                                                   [::]:123                                                         [::]:*                           users:(("ntpd",pid=878,fd=16))                                  
tcp                          LISTEN                        0                             128                                                            127.0.0.1:9150                                                     0.0.0.0:*                           users:(("tor",pid=3497,fd=6))                                   
tcp                          LISTEN                        0                             128                                                            127.0.0.1:9151                                                     0.0.0.0:*                           users:(("tor",pid=3497,fd=7))                                   
tcp                          LISTEN                        0                             128                                                            127.0.0.1:34463                                                    0.0.0.0:*                           users:(("expressvpn-agen",pid=1441,fd=7))                       
tcp                          LISTEN                        0                             128                                                            127.0.0.1:2015                                                     0.0.0.0:*                           users:(("expressvpnd",pid=907,fd=7))                            
tcp                          LISTEN                        0                             128                                                              0.0.0.0:<ssh-port>                                                    0.0.0.0:*                           users:(("sshd",pid=875,fd=3))                                   
tcp                          LISTEN                        0                             1                                                              127.0.0.1:44391                                                    0.0.0.0:*                           users:(("openvpn",pid=1173,fd=3))                               
tcp                          LISTEN                        0                             128                                                                 [::]:<ssh-port>                                                       [::]:*                           users:(("sshd",pid=875,fd=4))                                   

In the below block, 192.168.1.12 is equivalent to <local-network-ip-of-desktop>:

me@laptop $ tcpdump -lnnienp7s0 port <ssh-port>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp7s0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:19:43.611542 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353520933 ecr 0,nop,wscale 7], length 0
06:19:44.617505 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353521938 ecr 0,nop,wscale 7], length 0
06:19:46.629511 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353523950 ecr 0,nop,wscale 7], length 0
06:19:50.885805 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353528206 ecr 0,nop,wscale 7], length 0
06:19:59.078590 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353536398 ecr 0,nop,wscale 7], length 0
06:20:15.206988 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353552526 ecr 0,nop,wscale 7], length 0
06:20:47.462208 IP 173.239.199.183.34340 > 192.168.1.12.<ssh-port>: Flags [S], seq 511406441, win 29200, options [mss 1104,sackOK,TS val 2353584782 ecr 0,nop,wscale 7], length 0
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel

@bulletmark
Yes, I am running the ssh test from my laptop on the same local LAN that my desktop resides on. My router is a Netgear R6220, and I'm not seeing an "Enable LAN loopback" setting, initally. I don't know if NAT loopback is the same, but this page says my router doesn't have a NAT loopback feature.

Offline

#11 2019-11-26 06:32:31

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 441

Re: [SOLVED] ssh works locally, times out over internet

Try running your ssh test from an external site. The forward may actually be working. It only costs a few cents and a minute to spin up a VPS (e.g. Ubuntu or Debian) to try this, e.g. on DigitalOcean or Vultr.

Last edited by bulletmark (2019-11-26 06:33:05)

Offline

#12 2019-11-26 06:34:31

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

What is the output of `ip r s`?

bulletmark wrote:

My router calls the setting "Enable LAN loopback".

Good thought - I didn't think of that. The tcpdump confirms that the packet is coming to the desktop from the public IP address, and the desktop just isn't responding with the expected SYN ACK packet though.

I'm running out of ideas here - the desktop is seeing the incoming SYN packet, but isn't responding for whatever reason. Usually it's because of iptables. Are you positive you don't have any nftable rules?

Offline

#13 2019-11-26 06:37:53

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

@fukawi2

me@laptop $ ip r s
0.0.0.0/1 via 10.127.0.61 dev tun0 
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
10.0.0.0/8 via 192.168.1.1 dev enp7s0 
10.127.0.1 via 10.127.0.61 dev tun0 
10.127.0.61 dev tun0 proto kernel scope link src 10.127.0.62 
45.135.184.11 via 192.168.1.1 dev enp7s0 
128.0.0.0/1 via 10.127.0.61 dev tun0 
172.16.0.0/12 via 192.168.1.1 dev enp7s0 
192.168.0.0/16 via 192.168.1.1 dev enp7s0 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

@bulletmark
I'll try a VPS!

Offline

#14 2019-11-26 07:07:40

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

You have a VPN setup - it's routing the return traffic via the VPN which then gets dropped by their firewall probably.

Turn your VPN off and try again.

Offline

#15 2019-11-26 07:11:44

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

@fukawi2 @bulletmark I get the same `tcpdump` results on my ssh server when trying to ssh in from a VPS...

And I've tried with and without my VPN enabled on my laptop. Same `tcpdump` results.

Last edited by hoWlExat (2019-11-26 07:13:59)

Offline

#16 2019-11-26 23:48:52

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

What is the output of `ip r s` after disabling the VPN?

Offline

#17 2019-11-28 08:02:15

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

@fukawi2, not sure if you were asking for output on desktop (server) or laptop (client), so I've included before and afters for both.

me@desktop $ ip r s
0.0.0.0/1 via 10.127.0.21 dev tun0 
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
10.0.0.0/8 via 192.168.1.1 dev enp7s0 
10.127.0.1 via 10.127.0.21 dev tun0 
10.127.0.21 dev tun0 proto kernel scope link src 10.127.0.22 
45.135.184.11 via 192.168.1.1 dev enp7s0 
128.0.0.0/1 via 10.127.0.21 dev tun0 
172.16.0.0/12 via 192.168.1.1 dev enp7s0 
192.168.0.0/16 via 192.168.1.1 dev enp7s0 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

me@desktop $ expressvpn disconnect
Disconnecting...
Disconnected

me@desktop $ ip r s
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

#########################################

me@laptop $ ip r s
0.0.0.0/1 via 10.22.0.201 dev tun0 
default via 192.168.1.1 dev wlp110s0 proto dhcp metric 600 
10.0.0.0/8 via 192.168.1.1 dev wlp110s0 
10.22.0.1 via 10.22.0.201 dev tun0 
10.22.0.201 dev tun0 proto kernel scope link src 10.22.0.202 
128.0.0.0/1 via 10.22.0.201 dev tun0 
169.254.0.0/16 dev docker0 scope link metric 1000 linkdown 
172.16.0.0/12 via 192.168.1.1 dev wlp110s0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
173.214.168.253 via 192.168.1.1 dev wlp110s0 
192.168.0.0/16 via 192.168.1.1 dev wlp110s0 
192.168.1.0/24 dev wlp110s0 proto kernel scope link src 192.168.1.16 metric 600 

me@laptop $ expressvpn disconnect
Disconnecting...
Disconnected.

me@laptop $ ip r s
default via 192.168.1.1 dev wlp110s0 proto dhcp metric 600 
169.254.0.0/16 dev docker0 scope link metric 1000 linkdown 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.1.0/24 dev wlp110s0 proto kernel scope link src 192.168.1.42 metric 600

Last edited by hoWlExat (2019-11-30 02:38:57)

Offline

#18 2019-11-28 08:12:42

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

EDIT: sorry, on mobile; misread your last post.

Your laptop appears to have no networking at all after turning off the VPN?

Last edited by fukawi2 (2019-11-28 08:14:44)

Offline

#19 2019-11-30 02:40:21

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

Ah, that was because my VPN had a network lock on when it was disabled. I tried it with the network lock disabled, and I got the same results. That was expected, given it didn't work via the VPS, either.

I updated my previous message with the `ip r s` results on my laptop with the network lock disabled.

Last edited by hoWlExat (2019-11-30 02:40:39)

Offline

#20 2019-11-30 06:46:31

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

Alright, @fukawi2, I have some exciting news! I figured out what was causing the issue, but I don't yet understand why. I'm wondering if you might have some insight, here.

## Solution
I had a VPN enabled on *both* my laptop and my desktop. I had forgotten that the VPN was enabled on my desktop. As soon as I disconnected from the VPN on my desktop, the ssh from the laptop over the internet to the desktop worked! And it worked both when my laptop was connected and disconnected from its VPN.

## Follow up
To wrap this up, would you be able to help me troubleshoot why the VPN on the desktop was preventing ssh from working? Below are the results of executing `iptables -L` on the desktop after having completely opened up my firewall and enabling the VPN:

me@desktop $ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
xvpn       all  --  anywhere             anywhere            

Chain xvpn (1 references)
target     prot opt source               destination         
xvpn_dns   all  --  anywhere             anywhere            
xvpn_ks    all  --  anywhere             anywhere            

Chain xvpn_dns (1 references)
target     prot opt source               destination         
xvpn_dns_iface_exceptions  all  --  anywhere             anywhere            
xvpn_dns_ip_exceptions  all  --  anywhere             anywhere            
DROP       udp  --  anywhere             anywhere             udp dpt:domain

Chain xvpn_dns_iface_exceptions (1 references)
target     prot opt source               destination         

Chain xvpn_dns_ip_exceptions (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             10.127.0.1           udp dpt:domain

Chain xvpn_ks (1 references)
target     prot opt source               destination         
xvpn_ks_iface_exceptions  all  --  anywhere             anywhere            
xvpn_ks_ip_exceptions  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp dpts:bootps:bootpc
DROP       all  --  anywhere             anywhere            

Chain xvpn_ks_iface_exceptions (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain xvpn_ks_ip_exceptions (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.0.0.0/8          
ACCEPT     all  --  anywhere             172.16.0.0/12       
ACCEPT     all  --  anywhere             192.168.0.0/16      
ACCEPT     all  --  anywhere             169.254.0.0/16      
ACCEPT     all  --  anywhere             base-address.mcast.net/24 
ACCEPT     all  --  anywhere             45.135.184.11

Last edited by hoWlExat (2019-11-30 06:53:55)

Offline

#21 2019-12-01 23:12:28

fukawi2
Administrator
From: .vic.au
Registered: 2007-09-28
Posts: 5,777
Website

Re: [SOLVED] ssh works locally, times out over internet

With the VPN enabled on your desktop, the return packets (from desktop to laptop) are being routed via the VPN. This causes asynchronous routing. It's kind of like someone ringing your doorbell on the front door of your house, but you open the back door. You're both equally confused because neither of you are getting the result you expect from the doorbell.

You'll need to setup policy-based routing to make return packets be sent via the interface they were received on. This article should be able to help you: https://blog.scottlowe.org/2013/05/29/a … y-routing/

Offline

#22 2019-12-02 06:52:08

hoWlExat
Member
Registered: 2019-11-15
Posts: 28

Re: [SOLVED] ssh works locally, times out over internet

That makes sense. Since my networking skills/knowledge is lackluster, and I don't understand half of the jargon/concepts, I'll ask a separate question in this room about that, while I try to learn what I can in my free time. I appreciate all the help! Thanks!

Offline

Board footer

Powered by FluxBB