Arch Linux

hoWlExat

Member

Registered: 2019-11-15

Posts: 35

[SOLVED] SSH into a machine that has VPN enabled

I have an ssh server that I would like to ssh into while having its VPN enabled. When I have the VPN disabled, I observe the following on the ssh server, and I can successfully ssh into it:

me@ssh_server $ ip route list table main
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

192.168.1.12 is the ssh server, and 192.168.1.1 is my router.

When the VPN is enabled, I observe the following, and any ssh connection attempts to the ssh server fail:

me@ssh_server $ ip route list table main
0.0.0.0/1 via 10.77.0.57 dev tun0
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
10.0.0.0/8 via 192.168.1.1 dev enp7s0 
10.77.0.1 via 10.77.0.57 dev tun0 
10.77.0.57 dev tun0 proto kernel scope link src 10.77.0.58 
45.135.184.5 via 192.168.1.1 dev enp7s0 
128.0.0.0/1 via 10.77.0.57 dev tun0 
172.16.0.0/12 via 192.168.1.1 dev enp7s0 
192.168.0.0/16 via 192.168.1.1 dev enp7s0 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

I am under the impression that the above output means that all packets will go via the VPN (tun0), because all packets are using that `main` table (used in the terminal commands, above), as observed below:

me@ssh_server $ ip rule list
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

I ran a little experiment to validate this and added another table, `ssh_custom`, so I get the following output:

me@ssh_server $ ip rule list
0:      from all lookup local
1:      from all lookup ssh_custom
32766:  from all lookup main
32767:  from all lookup default

me@ssh_server $ ip route list table ssh_custom
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

If I have that set up, then I can ssh into the ssh_server when the VPN is enabled, but this isn't the solution, because now nothing is being tunneled via the VPN because everything is going to the `ssh_custom` table. I only want to make the exception to use the `ssh_custom` table (avoiding the VPN tunnel) for outbound traffic of an ssh connection. So, I followed this link and added the following to my pre-existing list of iptables rules (which were otherwise working when all traffic is sent to `ssh_custom` table or VPN is disabled):

sudo iptables -t mangle -A PREROUTING -p tcp -m tcp --dport <ssh-port> -m conntrack --ctstate NEW -j CONNMARK --set-xmark 1
sudo iptables -t mangle -A OUTPUT -m connmark --mark 0x1 -j MARK --set-xmark 2

My intent with the above code is to mark all packets related to an ssh connection as soon as they come in, and then any output packets related to that connection will set a separate mark -- "2". I then update my ip rules to look for that mark:

me@ssh_server $ ip rule list
0:      from all lookup local
1:      from all fwmark 0x2 lookup ssh_custom
32766:  from all lookup main
32767:  from all lookup default

The intent of that above code is to have any packets marked with the special "2" mark to use the `ssh_custom` table, which avoids the VPN tunnel (tun0).

This isn't working, however. I may just have one small thing wrong, or I may be wrong in what I'm trying to do, completely. I don't know. This is all very new to me. Does anyone have any idea where I'm going wrong or what I need to do/fix?

Thank you so much!!

Last edited by hoWlExat (2023-12-19 03:29:00)

Stellarator

Member

Registered: 2019-01-12

Posts: 32

Re: [SOLVED] SSH into a machine that has VPN enabled

I would try to simplify things.

me@ssh_server $ ip route list table ssh_custom
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

This quote clearly states that you don't completely understand how things works. This is no offense, just a hint that the problem requires a slightly more thoughtful approach than just substituting solutions from third-party sources .

There will be enough:

me@ssh_server $ ip route list table ssh_custom
default via 192.168.1.1 dev enp7s0

Second, please check one of the following:

cat /proc/sys/net/ipv4/conf/enp7s0/rp_filter
sysctl net.ipv4.conf.enp7s0.rp_filter

The value returned should be 0 or 2. If not - try to setup it to some value (2 is preferred for mentioned cases), sysctl or echo can do the trick.

Then you should understand the following. You need just two things:
- allow tcp packets with dst port 22 to pass in on any (or particular?) interface
- allow tcp packets with src port 22 to pass out on specific interface following particular routing rules

You don't need something special for the first point. You didn't post complete rule set but some general rule in the INPUT chain should be sufficient here. Something like:

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

The main trick should be done in OUTPUT chain. But I would start with something like this:

iptables -t mangle -I OUTPUT -p tcp --sport 22 -j MARK --set-mark 2

And after everything works, I would start playing with -m conntrack etc.

hoWlExat

Member

Registered: 2019-11-15

Posts: 35

Re: [SOLVED] SSH into a machine that has VPN enabled

Thank you for your answer, @Stellarator!

1)
The output of `ip route list table ssh_custom` was set to match what my computer had by default when initially running the command on the `main` table without the VPN enabled. I have tried with and without your simpler set of routes for the `ssh_custom` table, and in both cases, I'm still having no luck.

2)
I get this output:

me@ssh_server $ cat /proc/sys/net/ipv4/conf/enp7s0/rp_filter
0

3)
I do have iptables rules set up to allow ssh connections, and I can make it work in two cases: 1) if my VPN is off, 2) if I route all traffic to the `ssh_custom` table (avoiding the VPN), but I only want to avoid the VPN for ssh connections. So, my INPUT chain is set up, correctly. However, when I enable my VPN, it adds rules to the main table, as given in my original post:

me@ssh_server $ ip route list table main
0.0.0.0/1 via 10.77.0.57 dev tun0
default via 192.168.1.1 dev enp7s0 proto dhcp src 192.168.1.12 metric 202 
10.0.0.0/8 via 192.168.1.1 dev enp7s0 
10.77.0.1 via 10.77.0.57 dev tun0 
10.77.0.57 dev tun0 proto kernel scope link src 10.77.0.58 
45.135.184.5 via 192.168.1.1 dev enp7s0 
128.0.0.0/1 via 10.77.0.57 dev tun0 
172.16.0.0/12 via 192.168.1.1 dev enp7s0 
192.168.0.0/16 via 192.168.1.1 dev enp7s0 
192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.12 metric 100 
192.168.1.0/24 dev enp7s0 proto dhcp scope link src 192.168.1.12 metric 202

I believe this sends every OUTPUT packet via the VPN interface (tun0), which causes the ssh connection attempt to hang on my client.

I've tried using the specific rule for the OUTPUT chain that you mentioned:

me@ssh_server $ ./set_up_iptables.sh
me@ssh_server $ iptables -t mangle -I OUTPUT -p tcp --sport <ssh-port> -j MARK --set-mark 2
me@ssh_server $ ip rule list
0:      from all lookup local
1:      from all fwmark 0x2 lookup ssh_custom
32766:  from all lookup main
32767:  from all lookup default
me@ssh_server $ ip route list table ssh_custom
default via 192.168.1.1 dev enp7s0

But it still isn't working if I try to ssh to the ssh server. :'( My suspicion is that the ssh connection isn't actually being established on the ssh port (22, generally, but I'm using a different port). I'm guessing that the ssh server listens on the ssh port, but establishes ssh connections on a different, open port, so simply adding a rule to mark OUTPUT packets from the ssh port probably isn't listening on the right port. This assumption could be wrong, but that's what the `iptables` rule you suggested does to my knowledge, and it isn't currently working for me. Any other/further ideas?

4)
For full disclosure, here are my iptables rules. Anything `xvpn` is added by my VPN. Everything else is added by me, and all the rules (including port knocking) do work when either the VPN is off or I force all packets to use the `ssh_custom` table (avoiding the enabled VPN), as described above:

me@ssh_server $ iptables-save
# Generated by iptables-save v1.8.3 on Tue Dec 24 09:22:56 2019
*mangle
:PREROUTING ACCEPT [1354:583337]
:INPUT ACCEPT [1229:576023]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1238:521545]
:POSTROUTING ACCEPT [1237:521493]
-A OUTPUT -p tcp -m tcp --sport <ssh-port> -j MARK --set-xmark 0x2/0xffffffff
COMMIT
# Completed on Tue Dec 24 09:22:56 2019
# Generated by iptables-save v1.8.3 on Tue Dec 24 09:22:56 2019
*nat
:PREROUTING ACCEPT [54:4951]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [307:23477]
:POSTROUTING ACCEPT [307:23477]
COMMIT
# Completed on Tue Dec 24 09:22:56 2019
# Generated by iptables-save v1.8.3 on Tue Dec 24 09:22:56 2019
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:LOG_AND_DROP - [0:0]
:STATE0 - [0:0]
:STATE1 - [0:0]
:STATE2 - [0:0]
:STATE3 - [0:0]
:xvpn - [0:0]
:xvpn_dns - [0:0]
:xvpn_dns_iface_exceptions - [0:0]
:xvpn_dns_ip_exceptions - [0:0]
:xvpn_ks - [0:0]
:xvpn_ks_iface_exceptions - [0:0]
:xvpn_ks_ip_exceptions - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m recent --rcheck --name KNOCK3 --mask 255.255.255.255 --rsource -j STATE3
-A INPUT -m recent --rcheck --name KNOCK2 --mask 255.255.255.255 --rsource -j STATE2
-A INPUT -m recent --rcheck --name KNOCK1 --mask 255.255.255.255 --rsource -j STATE1
-A INPUT -j STATE0
-A INPUT -j DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -j xvpn
-A LOG_AND_DROP -j LOG --log-prefix "iptables deny: " --log-level 7
-A LOG_AND_DROP -j DROP
-A STATE0 -p udp -m udp --dport <first-knock-port> -m recent --set --name KNOCK1 --mask 255.255.255.255 --rsource -j DROP
-A STATE0 -j DROP
-A STATE1 -m recent --remove --name KNOCK1 --mask 255.255.255.255 --rsource
-A STATE1 -p udp -m udp --dport <second-knock-port> -m recent --set --name KNOCK2 --mask 255.255.255.255 --rsource -j DROP
-A STATE1 -j STATE0
-A STATE2 -m recent --remove --name KNOCK2 --mask 255.255.255.255 --rsource
-A STATE2 -p udp -m udp --dport <third-knock-port> -m recent --set --name KNOCK3 --mask 255.255.255.255 --rsource -j DROP
-A STATE2 -j STATE0
-A STATE3 -m recent --remove --name KNOCK3 --mask 255.255.255.255 --rsource
-A STATE3 -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --set --name SSH --mask 255.255.255.255 --rsource
-A STATE3 -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSH --mask 255.255.255.255 --rsource -j LOG_AND_DROP
-A STATE3 -p tcp -m tcp --dport <ssh-port> -j ACCEPT
-A STATE3 -j STATE0
-A xvpn -j xvpn_dns
-A xvpn -j xvpn_ks
-A xvpn_dns -j xvpn_dns_iface_exceptions
-A xvpn_dns -j xvpn_dns_ip_exceptions
-A xvpn_dns ! -o lo -p udp -m udp --dport 53 -j DROP
-A xvpn_dns_ip_exceptions -d 10.77.0.1/32 -p udp -m udp --dport 53 -j ACCEPT
-A xvpn_ks -j xvpn_ks_iface_exceptions
-A xvpn_ks -j xvpn_ks_ip_exceptions
-A xvpn_ks -p udp -m udp --dport 67:68 -j ACCEPT
-A xvpn_ks ! -o lo -j DROP
-A xvpn_ks_iface_exceptions -o tun0 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 10.0.0.0/8 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 172.16.0.0/12 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 192.168.0.0/16 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 169.254.0.0/16 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 224.0.0.0/24 -j ACCEPT
-A xvpn_ks_ip_exceptions -d 45.135.184.5/32 -j ACCEPT
COMMIT
# Completed on Tue Dec 24 09:22:56 2019

Last edited by hoWlExat (2019-12-24 09:47:52)

Stellarator

Member

Registered: 2019-01-12

Posts: 32

Re: [SOLVED] SSH into a machine that has VPN enabled

I didn't see

-A INPUT -p tcp --dport SSH_PORT -j ACCEPT

Your default INPUT chain policy for filter table is to DROP packets. You definitely need some rule to enable incoming SSH traffic.
Did you try (slightly modified) my example?

iptables -I INPUT -p tcp --dport SSH_PORT -j ACCEPT

Update: was in a hurry and did not notice rules related recent module. Still, I would consider to simplify things and start from bare config(s).

Last edited by Stellarator (2019-12-24 10:53:54)

hoWlExat

Member

Registered: 2019-11-15

Posts: 35

Re: [SOLVED] SSH into a machine that has VPN enabled

I believe these lines are doing what you said:

...
-A STATE3 -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --set --name SSH --mask 255.255.255.255 --rsource
-A STATE3 -p tcp -m tcp --dport <ssh-port> -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSH --mask 255.255.255.255 --rsource -j LOG_AND_DROP
-A STATE3 -p tcp -m tcp --dport <ssh-port> -j ACCEPT
...

But long story short, we can assume that the INPUT chain (which STATE3) is part of, works for ssh without further modification, since it in fact does work. We can probably assume the simpler approach is used for all intensive purposes (and I have tested it with the simple rule set, a.k.a. without port knocking). It does not work with or without port knocking, so there is most likely still something wrong with the OUTPUT chain, and telling the packets to use the `ssh_custom` table, as described in previous comments.

hoWlExat

Member

Registered: 2019-11-15

Posts: 35

Re: [SOLVED] SSH into a machine that has VPN enabled

This was solved in https://bbs.archlinux.org/viewtopic.php?pid=1875610