[SOLVED] OpenVPN Routing problem

monzer4819 · 2009-04-08 14:07:16

Greetings,

So I've finally migrated my last Ubuntu Server to Archlinux. (Almost) Everything is working better than before, not necessarily due to Arch but also do to increased experience since the Ubuntu install.

Anyways, Arch is like the best distro ever, I can't praise it enough. That being said, I've lost one simple functionality that I had with Ubuntu.

This is the setup:

The Openvpn server is located at home, and the client is at work. The idea is, the vpn client punches a hole through the cooperate firewall, establishes a connection at home, and the home network gains access to the cooperate network.

Before I go any further, there's nothing illegal going on, I'm a network admin at work, this is for testing purposes.

Anyways, everything was working with Ubuntu server <-- Archlinux client. The Arch client was running nat so the packets originating from the server could find their way back across the vpn. My home computers were configured with a static route to the vpn server and everything was bliss.

Since migrating the Ubuntu server to Archlinux, the vpn server can no longer see hosts beyond the client. I have successful connections between the server and the client, and the client has access to all computers behind the server. The server cannot see anything beyond the client. I can't figure out why.

I highly doubt the problem is on the client side, because this same configuration was working when the server was running Ubuntu. But I'll post configs regardless...

Client is configured for ipv4_forwading = 1 with nat:

iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE (eth1 being the cooperate interface) (eth0 is a second, seperate dsl connection on which the vpn link is established).

client has a file named after it's certificate common name, sitting in etc/openvpn/ccd containing "iroute 10.0.0.0 255.0.0.0"

Also if anyone can suggest improvements to the whole setup, it would be greatly appreciated.

cheers!

Server.conf:

port blah
proto udp
dev tun
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key  # This file should be kept secret
dh /etc/openvpn/keys/dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.8.0 255.255.255.0"
client-config-dir /etc/openvpn/ccd
route 10.0.0.0 255.0.0.0
client-to-client
keepalive 10 120
cipher AES-128-CBC   # AES
comp-lzo
max-clients 2
persist-key
persist-tun
status openvpn-status.log
verb 3

client.conf

client
dev tun
proto udp
remote blah
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client1.crt
key /etc/openvpn/client1.key
cipher AES-128-CBC   # AES
comp-lzo
verb 3

EDIT: The solution, aside from the fact that I'm an idiot, was not realizing that "client-config-dir" means putting the client config files on the server. I had them on the client. The whole client-config-dir option wasn't needed on Ubuntu at all, which I guess is why it worked on Ubuntu.

Last edited by monzer4819 (2009-04-14 13:39:36)

cactus · 2009-04-08 16:10:26

since you only have two endpoints, it would probably be better to set it up in site-to-site (p2p) mode, instead of in client mode.

As to your problem.. what do the route tables look like on each box, after the tunnel is up and running?
And what are the internal subnets and ips involved? (note. you dont have to give the real ones if you dont want, just make sure you consistently replace real values with fake values so things can be deduced)

monzer4819 · 2009-04-08 16:25:13

Thanks for the reply, holy crap I didn't even know there was a p2p mode. How to I enable that? Remove 'client' from config file?

Routing tables:

Note the client has 2 physical interfaces on 2 isp's. eth0 is a dsl connection on which the vpn link is established and eth1 is the the cooperate class A network.

Server from typing 'sudo route'
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.8.0.2 * 255.255.255.255 UH 0 0 0 tun0
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
192.168.8.0 * 255.255.255.0 U 0 0 0 eth0
10.0.0.0 10.8.0.2 255.0.0.0 UG 0 0 0 tun0
default 192.168.8.1 0.0.0.0 UG 0 0 0 eth0

Client:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.8.0.5 * 255.255.255.255 UH 0 0 0 tun0
10.10.10.0 * 255.255.255.0 U 0 0 0 eth1
192.168.2.0 * 255.255.255.0 U 0 0 0 eth0
10.8.0.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0
192.168.8.0 10.8.0.5 255.255.255.0 UG 0 0 0 tun0
10.0.0.0 10.10.10.1 255.0.0.0 UG 0 0 0 eth1
default 192.168.2.1 0.0.0.0 UG 0 0 0 eth0

cactus · 2009-04-08 23:09:12

So you have:

And on a machine behind the 'server' machine, you are trying to ping a machine on the 10.0.0.0/8 network behind 10.10.10.1 on the 'client' side (which is probably an internal router or something)?

If that is the case, I would say do a traffic capture on the client on eth1, and see if the traffic is leaving the client box nat'd or not, to whatever destination you are attempting.

Do a similar capture on the server side, on the eth0 interface, to see if traffic is trying to enter the box, and one on the same interface, capturing only the openvpn udp traffic (to make sure an encrypted packet is heading out too).

things to check:
make sure you have the 'enable ip forwarding' set on both client and server boxes (via sysctl).

net.ipv4.ip_forward = 1

EDIT: oh. forgot your first question..
As to p2p, that is just site-to-site mode. um.. you use a static key instead of ssl certs (just as secure, but you two known endpoints instead of clients who may wander around), and you configure the ips in the openvpn config file.
and you just specify the key, instead of client.
example you can probably jump start with:
http://openvpn.net/index.php/documentat … howto.html

might be able to use ssl-certs instead of keys, but still use the p2p mode. not sure. never tried it.

Last edited by cactus (2009-04-08 23:21:07)

monzer4819 · 2009-04-09 00:52:09

And on a machine behind the 'server' machine, you are trying to ping a machine on the 10.0.0.0/8 network behind 10.10.10.1 on the 'client' side (which is probably an internal router or something)?

Yes this is exactly what I'm trying to accomplish. It should be noted that not even the vpn server can ping the 10.0.0.0/8 network nor the eth1 interface of the client.
10.10.10.1 is indeed an internal router which leads to the rest of the 10.0.0.0/8 MAN network. (Metropolitan Area Network).

If that is the case, I would say do a traffic capture on the client on eth1, and see if the traffic is leaving the client box nat'd or not, to whatever destination you are attempting.

I have done this already on the server side. I ran a few tcpdumps on the server and the ping packets appear on the local tun0 interface, but they don't seem to arrive on the tun0 interface of the client. There's is some udp traffic back and forth between the client and server on the vpn port, which I'm assuming are keep-alives and persist packets. The volume doesn't seem to change when I try to ping unsuccessfully 10.10.10.1 from the server. Btw there is no error message when I ping, the packets just seem to get lost somewhere.

Anyways to summerize, I believe the packets are not leaving the server. I'm wondering if this is a Archlinux specific issue. Perhaps I need to load a kernel module for tunnels or bridging or something? Either that or maybe the TTL isn't long enough and the packets die before they reach the client?

EDIT: Forgot to say that yes, sysctl.conf is edited for ip_forward = 1 on both client and server

I'm grasping at straws here

Also thanks for the link to the p2p. I'll look into it once I get this fixed.

Cheers!

Last edited by monzer4819 (2009-04-09 01:07:13)

cactus · 2009-04-09 01:31:18

monzer4819 wrote:

It should be noted that not even the vpn server can ping the 10.0.0.0/8 network nor the eth1 interface of the client.

that is definitely a problem, and likely when you solve that, everything else will 'just work'.
since you said nothing changed on the client side... do you have any iptables rules on the server side?

iptables -nvL
and
iptables -nvL -t nat

also check the output of /var/log/messages (and/or openvpn logs), to see if there are any messages about your tun interface. I suppose it could be possible that the permissions on your /dev/net/tun aren't letting things access the module properly.

monzer4819 · 2009-04-09 13:01:49

Quite the bugger of a problem. iptables hadn't yet been installed when I noticed the problem.

Would tcpwrappers affect this? I tried putting ALL: ALL: ALLOW in hosts.allow, still no luck.

Permissions on /dev/net/tun are 0666. The same as on the client.

Well, today I'm going to scrap all the keys and start from scratch. I'll post my results....

monzer4819 · 2009-04-09 15:02:53

Just an update,

Issue still not resolved...

I've delete and regenerated new certificates for the server and client. I've tried a new cipher scheme. I've added ta.key and ns-cert-type server to their respective places. Basically I've taken care of every warning comming out of the log files. Still no joy.

This is the log output after a fresh reboot, right after the client connects.

Thu Apr  9 10:53:37 2009 OpenVPN 2.0.9 i686-pc-linux [SSL] [LZO] [EPOLL] built on Jul 11 2008
Thu Apr  9 10:53:37 2009 Diffie-Hellman initialized with 1024 bit key
Thu Apr  9 10:53:37 2009 Control Channel Authentication: using '/etc/openvpn/keys/ta.key' as a OpenVPN static key file
Thu Apr  9 10:53:37 2009 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr  9 10:53:37 2009 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr  9 10:53:37 2009 TLS-Auth MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Thu Apr  9 10:53:37 2009 TUN/TAP device tun0 opened
Thu Apr  9 10:53:37 2009 /sbin/ifconfig tun0 10.8.0.1 pointopoint 10.8.0.2 mtu 1500
Thu Apr  9 10:53:37 2009 /sbin/route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.8.0.2
Thu Apr  9 10:53:37 2009 /sbin/route add -net 10.8.0.0 netmask 255.255.255.0 gw 10.8.0.2
Thu Apr  9 10:53:37 2009 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Apr  9 10:53:37 2009 GID set to nobody
Thu Apr  9 10:53:37 2009 UID set to nobody
Thu Apr  9 10:53:37 2009 UDPv4 link local (bound): [undef]:****
Thu Apr  9 10:53:37 2009 UDPv4 link remote: [undef]
Thu Apr  9 10:53:37 2009 MULTI: multi_init called, r=256 v=256
Thu Apr  9 10:53:37 2009 IFCONFIG POOL: base=10.8.0.4 size=62
Thu Apr  9 10:53:37 2009 IFCONFIG POOL LIST
Thu Apr  9 10:53:37 2009 ****,10.8.0.4
Thu Apr  9 10:53:37 2009 ****,10.8.0.8
Thu Apr  9 10:53:37 2009 Initialization Sequence Completed

After the client connects:

Thu Apr  9 10:55:23 2009 MULTI: multi_create_instance called
Thu Apr  9 10:55:23 2009 **** Re-using SSL/TLS context
Thu Apr  9 10:55:23 2009 **** LZO compression initialized
Thu Apr  9 10:55:23 2009 **** Control Channel MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Thu Apr  9 10:55:23 2009 **** Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Apr  9 10:55:23 2009 **** Local Options hash (VER=V4): '14168603'
Thu Apr  9 10:55:23 2009 **** Expected Remote Options hash (VER=V4): '504e774e'
Thu Apr  9 10:55:23 2009 **** TLS: Initial packet from ****, sid=a14c0e31 f803d04c
Thu Apr  9 10:55:24 2009 **** VERIFY OK: *******
Thu Apr  9 10:55:24 2009 **** VERIFY OK: *******
Thu Apr  9 10:55:24 2009 **** Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Thu Apr  9 10:55:24 2009 **** Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr  9 10:55:24 2009 **** Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Thu Apr  9 10:55:24 2009 **** Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr  9 10:55:24 2009 **** Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Thu Apr  9 10:55:24 2009 **** [****] Peer Connection Initiated with ****
Thu Apr  9 10:55:24 2009 ****/**** MULTI: Learn: 10.8.0.10 -> ****/****
Thu Apr  9 10:55:24 2009 ****/**** MULTI: primary virtual IP for ****/****: 10.8.0.10
Thu Apr  9 10:55:25 2009 ****/**** PUSH: Received control message: 'PUSH_REQUEST'
Thu Apr  9 10:55:25 2009 ****/**** SENT CONTROL [****]: 'PUSH_REPLY,route 192.168.8.0 255.255.255.0,route 10.8.0.0 255.255.255.0,ping 10,ping-restart 120,ifconfig 10.8.0.10 10.8.0.9' (status=1)

monzer4819 · 2009-04-09 18:24:24

More updates:

I've narrowed the problem down even further.

When I run "tcpdump -i eth0 udp" on the server, I can see traffic (albeit encrypted) flowing back and forth between itself and the client. This is normal I suppose.

Here is a sample from "tcpdump -i eth0 udp" Note the names and ports have been edited...

13:50:28.610325 IP client**host > server.local.1000: UDP, length 53
13:50:29.642421 IP server.local.1000 > client**host: UDP, length 53
13:50:38.824209 IP client**host > server.local.1000: UDP, length 53
13:50:39.907781 IP server.local.1000 > client**host: UDP, length 53
13:50:49.043562 IP client**host > server.local.1000: UDP, length 53
13:50:49.044153 IP server.local.1000 > client**host: UDP, length 53
13:50:58.524903 IP client**host > server.local.1000: UDP, length 53
13:50:59.572556 IP server.local.1000 > client**host: UDP, length 53
13:51:09.022144 IP client**host > server.local.1000: UDP, length 53
13:51:09.022722 IP server.local.1000 > client**host: UDP, length 53
13:51:19.163371 IP client**host > server.local.1000: UDP, length 53
13:51:19.163978 IP server.local.1000 > client**host: UDP, length 53

A packet every 10 seconds. Probably a keep-alive since there's no traffic atm. This is good!

When I try to ping 10.8.0.10 I see increased traffic, going back and forth. This is good!
Sample:

13:55:18.407416 IP server.local.1000 > client**host: UDP, length 125
13:55:18.485878 IP client**host > server.local.1000: UDP, length 125
13:55:19.408732 IP server.local.1000 > client**host: UDP, length 125
13:55:19.437745 IP client**host > server.local.1000: UDP, length 125
13:55:20.410581 IP server.local.1000 > client**host: UDP, length 125
13:55:20.431631 IP client**host > server.local.1000: UDP, length 125
13:55:21.412424 IP server.local.1000 > client**host: UDP, length 125
13:55:21.431044 IP client**host > server.local.1000: UDP, length 125
13:55:22.413764 IP server.local.1000 > client**host: UDP, length 125
13:55:22.431448 IP client**host > server.local.1000: UDP, length 125
13:55:23.415275 IP server.local.1000 > client**host: UDP, length 125
13:55:23.438352 IP client**host > server.local.1000: UDP, length 125

A packet every second! Note the length 125. I can see you icmp traffic! Despite being encrypted! Excellent!

Ok ping 10.8.0.6, but now I'm dumping tun0 "tcpdump -i tun0"

14:01:39.795355 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 1, length 64
14:01:39.823392 IP 10.8.0.6 > 10.8.0.1: ICMP echo reply, id 1806, seq 1, length 64
14:01:40.796569 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 2, length 64
14:01:40.859330 IP 10.8.0.6 > 10.8.0.1: ICMP echo reply, id 1806, seq 2, length 64
14:01:41.798404 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 3, length 64
14:01:41.858766 IP 10.8.0.6 > 10.8.0.1: ICMP echo reply, id 1806, seq 3, length 64
14:01:42.799851 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 4, length 64
14:01:42.819583 IP 10.8.0.6 > 10.8.0.1: ICMP echo reply, id 1806, seq 4, length 64
14:01:43.801685 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 5, length 64
14:01:43.835999 IP 10.8.0.6 > 10.8.0.1: ICMP echo reply, id 1806, seq 5, length 64
14:01:44.803138 IP 10.8.0.1 > 10.8.0.6: ICMP echo request, id 1806, seq 6, length 64

Everything is working, and I can see the decrypted traffic. Perfect!

Now, I ping the eth1 interface on the client. Which happens to be 10.10.10.20
ping 10.10.10.20
"tcpdump -i tun0"

14:04:41.520743 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 1, length 64
14:04:42.520480 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 2, length 64
14:04:43.520471 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 3, length 64
14:04:44.520468 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 4, length 64
14:04:45.520461 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 5, length 64
14:04:46.520460 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 6, length 64
14:04:47.520477 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 7, length 64
14:04:48.520470 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 8, length 64
14:04:49.520473 IP 10.8.0.1 > 10.10.10.20: ICMP echo request, id 2318, seq 9, length 64

Interesting! Only echo requests! No replys! What gives?

Still pinging, now to check eth0 on the server. Back to the first command!
"tcpdump -i eth0 udp"

14:15:15.065878 IP client**host > server.local.1000: UDP, length 53
14:15:23.096241 IP server.local.1000 > client**host: UDP, length 53
14:15:24.368572 IP client**host > server.local.1000: UDP, length 53
14:15:33.413931 IP server.local.1000 > client**host: UDP, length 53
14:15:34.665194 IP client**host > server.local.1000: UDP, length 53

Ah! back to a packet every 10 seconds or so. These are keep-alives! My icmp traffic is not here!

So to conclude. My packets are getting lost between tun0 and eth0. tun0 receives the packets, but does not forward them to eth0.

Again, sysctl.conf is edited. ip_forward = 1 is indeed the case. I' keep checking it over and over like it's gonna change....

Something is wrong with my server. Do I need to load kernel modules or something?

Also, sorry for the long post. I'm hoping this will help someone else some day.

Arch Linux

#1 2009-04-08 14:07:16

[SOLVED] OpenVPN Routing problem

#2 2009-04-08 16:10:26

Re: [SOLVED] OpenVPN Routing problem

#3 2009-04-08 16:25:13

Re: [SOLVED] OpenVPN Routing problem

#4 2009-04-08 23:09:12

Re: [SOLVED] OpenVPN Routing problem

#5 2009-04-09 00:52:09

Re: [SOLVED] OpenVPN Routing problem

#6 2009-04-09 01:31:18

Re: [SOLVED] OpenVPN Routing problem

#7 2009-04-09 13:01:49

Re: [SOLVED] OpenVPN Routing problem

#8 2009-04-09 15:02:53

Re: [SOLVED] OpenVPN Routing problem

#9 2009-04-09 18:24:24

Re: [SOLVED] OpenVPN Routing problem

Board footer