Arch Linux

squalou

Member

Registered: 2018-06-04

Posts: 112

[SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Hi,

first of all, this post is probably out of place BUT I'm at a complete loss, and I know from past readings that very competent people are around in arch community.

Sorry for the long story, this issue requires context. Here it is :

------ long story's prologue -----

Once upon a time in a small company ...

I'm the guy supposed to manage network there, and guess what : it's only because others know even less than I in that domain.

My situation is that I have a setup that *does* work with any archlinux client (whatever the 'flavour' I tried : manjaro, pure arch, antergos, you name it). So, what's wrong you may ask ?
Well, other coworkers can use the distro they want. And ... it used to work with ubuntu 16.04, works with debian stretch ... but not with ubuntu 18.04 or Mint or Fedora.

SO :
- I'm looking for a way to fix them
- I'm afraid it will cease to work one day on Arch and I won't be able to fix it

All in all I have to understand why and how it 'works' on arch.

---- end of prologue ------

What's the setups I'm bragging about :

- from Home to Office, I've setup a L2TP IPSEC VPN to a Ubiquiti gateway. Let's call it VPNH
- from Office to a datacenter, I've setup a Site 2 Site VPN on this same gateway : this connexion works like a charm. Let's call it VPNS2S.
- on this very same gateway, there's a static route added to provide a 'next hop' to the datacenter end of VPN2S2 tunnel for any IP in the range given to guys connectin to VPNH (that's where sentences start to be tricky to follow )
- let's say I have a small debian server in the Office, and call it office-server

Now, what works from Arch machines
- From home, establish VPNH : OK
- access office-server : OK
- access any server in datacenter : OK (direct ssh from arch machine to end server, for instance)

And here's the broken behaviour I see on other distros :
- From home, establish VPNH : OK
- access office-server : OK
- ping any server on datacenter :  .... 80 % loss, end up with VPNH disconnecting (sometimes it 'almost works' for 2 minutes, no more)

Additional 'funny' behaviour :
- ping office-server, leave the ping running in a terminal
- ping datacenter : then the ping on office-server instantly start loosing packets !

I made sure on every linux I tried that the 'user end' of the setup was the same :
- network-manager
- networkmanager-l2tp
- same version of xl2tpd (even tried head of master direct from repo, just to try)
- approcimately same kernels (4.15.something)
- libreswan (also tried strongswan, only changes the VPNH 'connexion' step somtimes, no difference after)

I tried to follow this :
https://wiki.archlinux.org/index.php/Op … ient_setup

but no better result.

I made sure the output of 'route -n' is the same of any machine. It is, but still not working.

I 'looks like' a routing problem, as if for whatever reason, 'something' didn't know where to send packets.

I didn't find interesting things in logs, except from this in NewtorkManager logs before VPNH dies

xl2tpd[2667]: check_control: Received out of order control packet on tunnel 42880 (got 3, expected 4)

Thank anyone who had the guts to read until this line,
now if anyone has any idea in what direction I could look in porder to understand what's happening ... I'd be more than happy !

Best regards

Squalou.

Last edited by squalou (2018-08-24 08:56:59)

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Check for accept_source_route differences - that can be the cause of such seeming weirdness, since ping is *ICMP* traffic, and the sending of new routing is also ICMP traffic.

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

I'm comparing an Arch and an Ubuntu 18.04 right now : same config on both side :

net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.docker0.accept_source_route = 1
net.ipv4.conf.ip_vti0.accept_source_route = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.ppp0.accept_source_route = 1
net.ipv4.conf.wlp58s0.accept_source_route = 1

no idea what these may imply though

Quickly checked other values mentioned in the article : forwarding and redirects ... same thing there too

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Try changing those ones to zeros. The purpose is to (hopefully) prevent the weirdness that you mentioned:

- ping office-server, leave the ping running in a terminal
- ping datacenter : then the ping on office-server instantly start loosing packets !

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Just tried that, still the same behaviour.

Also, note that I mentioned the 'ping', but I see the same thing with http access to web servers.
On distros when things are broken : the website hosted on datacenter will sometimes not be reachable at all, and sometimes show some pieces, then nothing again, and finally VPN dies..

brebs

Member

Registered: 2007-04-03

Posts: 3,742

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

I should have clarified - disable accept_source_routing on the server, also. The purpose is to prevent routes from being unintentionally changed.

As always, tcpdump and wireshark are very useful network debugging tools

Try with Devuan, then hopefully you can just blame systemd

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Ok, I just checked our ubiquity gateway (the one dealing with office network and VPNs). Fir the records it's a vyatta based system.
- accept_source_route is set to 0 there, for any ipv4 entry.
- for ipv6 it is set to -1 (but anyway I disabled ip6 everywhere to narrow the potential problems)

On target servers inside datacenter, it's a bit more tricky :
- for web access and ping they are behind a loadbalancer I cannot acces directly. (saas offer).
- still, i can check on the the end-servers : there, accept_source_route is set to 0 too

(Thanks for the info anyway, still learning everyday

I'll have a look at tcpdump output,

and give Devuan a try  (didn't know of this one either ! thx)
sorry but no time to loose to learn yet another debian flavored distro. Tried 30 minutes, felt like a time machine, back 10 years ago. I like xfce, but wicd ? seriously ? Anyway I've seen things work with systemd on Debian stretch and on Arch, so it's probably not the issue.

regards

Last edited by squalou (2018-08-16 11:48:02)

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Only good news so far : I've tested with a Debian Testing (buster alpha1) : it works fine.

dkosovic

Member

Registered: 2017-12-16

Posts: 21

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Although you mentioned you are using the same version of xl2tpd across the various linux distros, xl2tpd's pppd child process is very likely different.

Upstream PPP maintainer hasn't been accepting patches, so for example Fedora has around 26 patches :

  https://src.fedoraproject.org/rpms/ppp/tree/master

Debian and by extension Ubuntu also has heaps of patches, for example the following 16 pacthes :

  https://sources.debian.org/patches/ppp/2.4.7-2+3/

Regarding the `Received out of order control packet` issue, you could try setting `DFLAGS=-DCONFIG_WATCHDOG_FIREWALL` (for xl2tpd github master branch's Makefile) which is now used for the following issue:
  https://github.com/xelerance/xl2tpd/issues/136
  https://github.com/xelerance/xl2tpd/issues/136
but not sure if it will help in your case.

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Thanks for pointing out all these patches,
that probably explains the differences between distros. I'll dig a bit more in this direction. (knowing that Debian testing works is a good hint)

On the other hand, just tried the xl2tpd suggestion : it didn't change anything.

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Also tried ppp update : took the one from debian testing, installed it on top of ubuntu 18 (yeah, mixing distro time )
... still the exact same behaviour.

I'd rather think something is playing with net packets routing, but no idea what.

Last edited by squalou (2018-08-18 12:56:58)

squalou

Member

Registered: 2018-06-04

Posts: 112

Re: [SOLVED] Need help with a weird VPN setup (ipsec/l2tp)

Maybe solved. ?  ... but no idea why.

I changed  'IPSec Settings ...' 

Phase 1 : aes256-sha1-modp1024
Phase2 : aes256-sha1

... and it works with this on Ubuntu 18.04. And also on NixOS

On the other hand :
- on a plain debian stretch : I have to set 3des-sha1-modp1024
- on a debian buster alpha : I leave all empty

Still issues on Mint though. In this case it's DNS related, and I *think* (=not tested yet) that replacing systemd-resolved by the good old resolv.conf managed by network manager may solve thing. But that's another story entirely.

Marking as solved ... even if I don't understand why.