You are not logged in.

#1 2019-06-05 17:23:04

mvasi90
Member
Registered: 2017-02-02
Posts: 41

[Solved] VPNC Route Gateway disappears

Hello everyone,

I'm using vpnc to route all my traffic to few vpn servers. (The DNS has several Address Mapping Records, each time returns distinct IP address)

When a new connection is established, vpnc-script adds the following route (it's the gateway route):

SERVER-IP via ROUTER-IP dev eth0 src ETH0-IP

All works well several seconds or minutes. After that time the gateway route disappears.

vpnc-script debug:

18:11:23 run_hooks() 
18:11:23 do_pre_init() 
18:11:24 run_hooks() 
18:11:24 do_connect() 
18:11:24 set_vpngateway_route() 
18:11:24 fix_ip_get_output () 
18:11:24 fix_ip_get_output () 
/sbin/ip route add SERVER-IP via ROUTER-IP dev eth0 src ETH0-IP
18:11:24 do_ifconfig() 
18:11:24 set_network_route() 
/sbin/ip route replace ROUTER-NETWORK/24 dev tun0
18:11:24 set_default_route() 
18:11:24 fix_ip_get_output () 
/sbin/ip route replace default dev tun0
18:11:24 modify_resolved_manager() 
18:11:24 get_if_index() 
18:11:24 busctl_set_nameservers() 
18:11:24 busctl_call() 
18:11:24 busctl_set_search() 
18:11:24 busctl_call() 
18:11:24 run_hooks() 

As you can see, the vpnc-script does not delete the gateway route.
Two minutes later, the debug stays the same but the gateway route disappears.

Which service are modifying the route table?

Update
Monitoring the route

$ ip monitor route

# Connecting to VPN server
SERVER-IP via ROUTER-IP dev eth0 src ETH0-IP
IPV6/8 dev tun0 table local metric 256 pref medium
IPV6/64 dev tun0 proto kernel metric 256 pref medium
local IPV6 dev tun0 table local proto kernel metric 0 pref medium
local TUN0-IP dev tun0 table local proto kernel scope host src TUN0-IP
SERVER-NETWORK/24 dev tun0 scope link
default dev tun0 scope link
# Connection VPN established. All works well.

# Two minutes later:
Deleted ROUTER-IP dev eth0 proto dhcp scope link src ETH0-IP metric 1024 
Deleted default via ROUTER-IP dev eth0 proto dhcp src ETH0-IP metric 1024 
Deleted LAN-IP/24 dev eth0 proto kernel scope link src ETH0-IP
Deleted broadcast BROADCAST-IP dev eth0 table local proto kernel scope link src ETH0-IP
Deleted broadcast NETWORK-IP dev eth0 table local proto kernel scope link src ETH0-IP
Deleted local ETH0-IP dev eth0 table local proto kernel scope host src ETH0-IP
local ETH0-IP dev eth0 table local proto kernel scope host src ETH0-IP
broadcast BROADCAST-IP dev eth0 table local proto kernel scope link src ETH0-IP
LAN-IP/24 dev eth0 proto kernel scope link src ETH0-IP
broadcast NETWORK-IP dev eth0 table local proto kernel scope link src ETH0-IP
ROUER-IP dev eth0 proto dhcp scope link src ETH0-IP metric 1024 
default via ROUTER-IP dev eth0 proto dhcp src ETH0-IP metric 1024 

I will temporarily use a script that monitors the route table and every time create the route when it's deleted.

PostData: This behavior exists for several days, when I updated the system. I had not updated it for a few weeks or a month.

Last edited by mvasi90 (2019-06-10 12:59:29)

Offline

#2 2019-06-06 11:17:31

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] VPNC Route Gateway disappears

Those deleted messages suggest the changed routing table is a symptom, not the cause.
Try monitoring addresses to see if the ip-adress of eth0 changes.

The output of

systemctl list-unit-files --state=enabled

\
would also be helpful.

Last edited by Lone_Wolf (2019-06-06 11:17:47)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#3 2019-06-06 15:25:50

mvasi90
Member
Registered: 2017-02-02
Posts: 41

Re: [Solved] VPNC Route Gateway disappears

monitoring ip address (the IP address never changes)

$ ip mon all
# every few seconds (30 seconds, 40, etc) are showing: STABLE, REACHABLE, PROBE...
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz PROBE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz REACHABLE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz STALE 
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz PROBE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz REACHABLE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz STALE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz PROBE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz REACHABLE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz STALE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz PROBE
[NEIGH]ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz REACHABLE

# after a minute or two minutes
[ROUTE]Deleted ROUTER-IP dev eth0 proto dhcp scope link src ETH0-IP metric 1024
[ROUTE]Deleted default via ROUTER-IP dev eth0 proto dhcp src ETH0-IP metric 1024
[ADDR]Deleted 2: eth0    inet ETH0-IP/24 brd BROADCAST-IP scope global dynamic eth0
       valid_lft 326sec preferred_lft 326sec
[ROUTE]Deleted LAN/24 dev eth0 proto kernel scope link src ETH0-IP
[ROUTE]Deleted broadcast BROADCAST-IP dev eth0 table local proto kernel scope link src ETH0-IP
[ROUTE]Deleted broadcast LAN dev eth0 table local proto kernel scope link src ETH0-IP
[ROUTE]Deleted local ETH0-IP dev eth0 table local proto kernel scope host src ETH0-IP
[NEIGH]Deleted 224.0.0.252 dev eth0 lladdr 01:00:5e:00:00:fc NOARP
[NEIGH]Deleted ROUTER-IP dev eth0 lladdr zz:zz:zz:zz:zz:zz REACHABLE
[ADDR]2: eth0    inet ETH0-IP/24 brd BROADCAST-IP scope global dynamic eth0
       valid_lft 600sec preferred_lft 600sec
[ROUTE]local ETH0-IP dev eth0 table local proto kernel scope host src ETH0-IP
[ROUTE]broadcast BROADCAST-IP dev eth0 table local proto kernel scope link src ETH0-IP
[ROUTE]LAN/24 dev eth0 proto kernel scope link src ETH0-IP
[ROUTE]broadcast LAN dev eth0 table local proto kernel scope link src ETH0-IP
[ROUTE]ROUTER-IP dev eth0 proto dhcp scope link src ETH0-IP metric 1024
[ROUTE]default via ROUTER-IP dev eth0 proto dhcp src ETH0-IP metric 1024
[ROUTE]SERVER-VPN-IP via ROUTER-IP dev eth0 src ETH0-IP
[NEIGH]ROUTER-IP dev eth0 lladdr c:ec:da:08:19:f7 REACHABLE



dmesg output

$ dmesg -H
[Jun 6 12:41] audit: type=1130 audit(1559817717.912:1351): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:42] audit: type=1131 audit(1559817747.982:1352): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:46] audit: type=1130 audit(1559817995.571:1353): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:47] audit: type=1131 audit(1559818025.661:1354): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:51] audit: type=1130 audit(1559818271.320:1355): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.122714] audit: type=1131 audit(1559818301.440:1356): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:55] audit: type=1130 audit(1559818538.510:1357): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 12:56] audit: type=1131 audit(1559818568.590:1358): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:00] audit: type=1130 audit(1559818801.469:1359): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.094728] audit: type=1131 audit(1559818831.559:1360): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:04] audit: type=1130 audit(1559819074.378:1361): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:05] audit: type=1131 audit(1559819104.458:1362): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:09] audit: type=1130 audit(1559819351.957:1363): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.070639] audit: type=1131 audit(1559819382.026:1364): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:13] audit: type=1130 audit(1559819620.096:1365): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:14] audit: type=1131 audit(1559819650.205:1366): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:18] audit: type=1130 audit(1559819882.555:1367): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.046271] audit: type=1131 audit(1559819912.595:1368): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:22] audit: type=1130 audit(1559820157.274:1369): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:23] audit: type=1131 audit(1559820187.344:1370): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:27] audit: type=1130 audit(1559820435.763:1371): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.099386] audit: type=1131 audit(1559820465.863:1372): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:31] audit: type=1130 audit(1559820697.112:1373): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:32] audit: type=1131 audit(1559820727.222:1374): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:36] audit: type=1130 audit(1559820964.401:1375): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.095875] audit: type=1131 audit(1559820994.492:1376): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:40] audit: type=1130 audit(1559821228.492:1377): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.057239] audit: type=1131 audit(1559821258.542:1378): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:45] audit: type=1130 audit(1559821506.412:1379): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.124963] audit: type=1131 audit(1559821536.532:1380): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:49] audit: type=1130 audit(1559821772.722:1381): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:50] audit: type=1131 audit(1559821802.832:1382): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:54] audit: type=1130 audit(1559822047.802:1383): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.121251] audit: type=1131 audit(1559822077.922:1384): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:58] audit: type=1130 audit(1559822316.152:1385): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 13:59] audit: type=1131 audit(1559822346.252:1386): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:03] audit: type=1130 audit(1559822592.521:1387): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.072039] audit: type=1131 audit(1559822622.601:1388): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:07] audit: type=1130 audit(1559822863.461:1389): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:08] audit: type=1006 audit(1559822886.001:1390): pid=21587 uid=0 old-auid=4294967295 auid=1001 tty=(none) old-ses=4294967295 ses=8 res=1
[  +7.541891] audit: type=1131 audit(1559822893.541:1391): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:12] audit: type=1130 audit(1559823136.690:1392): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.109174] audit: type=1131 audit(1559823166.800:1393): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:16] audit: type=1130 audit(1559823407.129:1394): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:17] audit: type=1131 audit(1559823437.178:1395): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:21] audit: type=1130 audit(1559823676.508:1396): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ +30.099907] audit: type=1131 audit(1559823706.608:1397): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:25] audit: type=1130 audit(1559823937.487:1398): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:26] audit: type=1131 audit(1559823967.577:1399): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[Jun 6 14:30] audit: type=1130 audit(1559824211.976:1400): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

journalctl output

$ journalctl -f
Jun 06 14:34:44 SYSTEM2 systemd-networkd[269]: eth0: DHCP lease lost
Jun 06 14:34:44 SYSTEM2 systemd-networkd[269]: eth0: DHCPv4 address ETH0-IP/24 via ROUTER-IP
Jun 06 14:34:44 SYSTEM2 dbus-daemon[258]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.14' (uid=981 pid=269 comm="/usr/lib/systemd/systemd-networkd ")
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: reading /etc/resolv.conf
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver A-LOCAL-DNS-ONLY-WORKS-WITHOUT-VPN#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver 8.8.8.8#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver SERVER-VPN-DNS#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: reading /etc/resolv.conf
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver A-LOCAL-DNS-ONLY-WORKS-WITHOUT-VPN#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver 8.8.8.8#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver SERVER-VPN-DNS#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: reading /etc/resolv.conf
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver A-LOCAL-DNS-ONLY-WORKS-WITHOUT-VPN#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver 8.8.8.8#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver SERVER-VPN-DNS#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: reading /etc/resolv.conf
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver A-LOCAL-DNS-ONLY-WORKS-WITHOUT-VPN#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver 8.8.8.8#53
Jun 06 14:34:44 SYSTEM2 dnsmasq[1818]: using nameserver SERVER-VPN-DNS#53
Jun 06 14:34:44 SYSTEM2 systemd[1]: Starting Hostname Service...
Jun 06 14:34:45 SYSTEM2 dbus-daemon[258]: [system] Successfully activated service 'org.freedesktop.hostname1'
Jun 06 14:34:45 SYSTEM2 systemd[1]: Started Hostname Service.
Jun 06 14:34:45 SYSTEM2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 06 14:34:45 SYSTEM2 kernel: audit: type=1130 audit(1559824485.235:1402): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

When this problem arose I created a script to solve it temporarily. The script does check the connection (with ping) every 10 seconds.
If there was no internet connection, it killed the vpnc process and reopened it. The maximum delay was up to 20 seconds.
The connection stopped working every minute or two minutes. The connection was very unstable. (20 seconds disconnected every minute or two minutes).

Upon discovering that vpnc was still maintaining the active connection, I started looking for the problem.
Then I discovered that the gateway route disappears, and it is not vpnc who removes it.

I decided to update the script to make it keep the route.
Here I publish the new version in case someone has the similar problem.

Note: The connection is stable. I've had VOIP call for more than two hours without interruptions. The maximum delay is one second to recreate the route.

vpnc.sh

#!/bin/bash
[[ `id -u` -ne 0 ]] && echo "run as root" && exit

ctrlc() {
        echo "Killing background services"
        echo "vpn: $vpn_pid - mrd: $mrd_pid - mra: $mra_pid"
        [[ -z "$vpn_pid" ]] && kill $vpn_pid
        [[ -z "$mrd_pid" ]] && kill $mrd_pid
        [[ -z "$mra_pid" ]] && kill $mra_pid
        exit 0
}

trap ctrlc SIGINT

run_vpnc() {
        vpnc --no-detach >/dev/null 2>&1 &
        vpn_pid=$!
        echo "vpn connected: $vpn_pid"
}

monitor_route_add() {
        echo "Monitor route add"
        (ip monitor route | grep -m 1 -P "^\d.*via.*dev.*src.*" > route) &
        mra_pid=$!
}

monitor_route_del() {
        while :; do
                echo "Monitor route del started"
                ip monitor route | grep -qi "default via" && sleep 1 && route_add
        done
}

route_add() {
        echo "Route add: $r"
        ip r a $r
}

main() {
        while :; do
                monitor_route_add
                run_vpnc
                echo "Waiting for monitor_route_add: $mra_pid"
                wait $mra_pid
                r=`cat route && rm route`
                monitor_route_del &
                mrd_pid=$!
                echo "Waiting for vpnc: $vpn_pid"
                wait $vpn_pid
                "Killing monitor_route_del: $mrd_pid"
                kill $mrd_pid
        done
}

main

How does it work?

  1. Start ip monitor for route add (in background)

  2. Run vpnc --no-detach (in background to control its pid) *

  3. Waits for monitor route add (to get the route added)

  4. Save the route in $r

  5. Start ip monitor for route del (a loop in background)
    Every time that route disappears, monitor route dell recreate it in a second (from $r)

  6. Waits for vpnc. When vpnc ends kill the monitor route del and starts again from step 1

* It is also possible to run without '--no-detach' and get the PID, but I prefer to have a command in the foreground and execute it as a job

vpnc-stable-route-keep.png

Last edited by mvasi90 (2019-06-07 15:12:57)

Offline

#4 2019-06-06 16:40:34

mvasi90
Member
Registered: 2017-02-02
Posts: 41

Re: [Solved] VPNC Route Gateway disappears

list enabled services

$ systemctl list-unit-files --state=enabled
UNIT FILE                              STATE  
autovt@.service                        enabled
dbus-org.freedesktop.network1.service  enabled
dbus-org.freedesktop.resolve1.service  enabled
dbus-org.freedesktop.timesync1.service enabled
dnsmasq.service                        enabled
getty@.service                         enabled
haveged.service                        enabled
hostapd.service                        enabled
iptables.service                       enabled
sshd.service                           enabled
systemd-networkd-wait-online.service   enabled
systemd-networkd.service               enabled
systemd-resolved.service               enabled
systemd-timesyncd.service              enabled
systemd-networkd.socket                enabled
remote-fs.target                       enabled

Offline

#5 2019-06-07 12:25:07

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] VPNC Route Gateway disappears

This looks like there may be  a conflict between ipv4 and ipv6 .

I'd like to see

$ ip address show
$ip route show

several times :

- before connecting to the vpn
- just after you made connection to the vpn
- after routes have been changed.

N.B. I have to be able to see ipv4 / ipv6 network designations and addresses.
Please don't use things like ROUTER-IP unless you also post their value.
If there are parts you need to obfuscate, replace some decimal/hex digits with xx , yy , zz etc .


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#6 2019-06-07 14:43:01

mvasi90
Member
Registered: 2017-02-02
Posts: 41

Re: [Solved] VPNC Route Gateway disappears

x.x.x.x -> vpn tun0
y.y.y.y -> vpn server
zz:zz:zz:zz:zz:zz -> mac address

Before starting vpnc
a.png

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group d
efault qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
 group default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.151/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 460sec preferred_lft 460sec
    inet6 fe80::ba27:ebff:fe66:zzzz/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP grou
       link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
       inet 192.168.1.1/24 brd 192.168.1.255 scope global wlan0
          valid_lft forever preferred_lft forever
       inet6 fe80::c66e:1fff:fe26:xxxx/64 scope link
          valid_lft forever preferred_lft forever


$ ip r
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.151 metric 1024
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.151        
192.168.0.1 dev eth0 proto dhcp scope link src 192.168.0.151 metric 1024 
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.1         

After starting vpnc
b.png

$ sudo vpnc --no-detach                                     
[sudo] password for user:                                                   
VPNC started in foreground...                                               

--------------------------------------

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group d
efault qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
 group default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.151/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 449sec preferred_lft 449sec
    inet6 fe80::ba27:ebff:fe66:zzzz/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP grou
p default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:fe26:xxxx/64 scope link
       valid_lft forever preferred_lft forever
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1412 qdisc fq_codel
state UNKNOWN group default qlen 500
    link/none
    inet xxx.xxx.xxx.116/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::2651:decd:30ca:zzzz/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

-------------------------------------

$ ip r                                                                            
default dev tun0 scope link                                                 
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.151 metric 1024   
xxx.xxx.xxx.0/24 dev tun0 scope link                                        
yyy.yyy.yyy.yy via 192.168.0.1 dev eth0 src 192.168.0.151                   
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.151           
192.168.0.1 dev eth0 proto dhcp scope link src 192.168.0.151 metric 1024    
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.1            

120 seconds after starting vpnc (eth0 valid_lft starts again in 600 seconds)
c.png

$ sudo vpnc --no-detach                                     
[sudo] password for user:                                                   
VPNC started in foreground...                                               

--------------------------------------

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group d
efault qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
 group default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.151/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 594sec preferred_lft 594sec
    inet6 fe80::ba27:ebff:fe66:zzzz/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP grou
p default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:fe26:xxxx/64 scope link
       valid_lft forever preferred_lft forever
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1412 qdisc fq_codel
state UNKNOWN group default qlen 500
    link/none
    inet xxx.xxx.xxx.116/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::2651:decd:30ca:zzzz/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

------------------------------------
                                                                       
$ ip r
default dev tun0 scope link                                                 
default via 192.168.0.1 dev eth0 proto dhcp src 192.168.0.151 metric 1024   
xxx.xxx.xxx.0/24 dev tun0 scope link                                        
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.151           
192.168.0.1 dev eth0 proto dhcp scope link src 192.168.0.151 metric 1024    
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.1

Play on asciinema
asciinema vpnc gateway

Offline

#7 2019-06-08 11:01:27

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] VPNC Route Gateway disappears

Information conflicts eachother.

In post #1 ip monitor route clearly shows IPV6 and IPV4 routes being used, in #6 it looks like only ipv4 is used.
Also there's no route disappearing in #6, maybe because your script restored it already ?


Time to try a different approach .
What we do know suggests the dhcp lease renewal for eth0 gives a different result depending on whether the vpnc is active or not.

Disable your workaround script and reboot inbetween tries.

A Increase the "valid_lft 460sec" for eth0 to a longer period, does that result in the default gateway route being removed later ?
(It's probably set on your router , changing will probably require admin access to it and maybe a reboot of the router)


B disable dnsmasq

C keep dnsmasq disabled
disable  systemd-networkd & systemd-resolved . use dhcpcd or dhclient instead.
Not sure if systemd-timesyncd works without systemd-networkd running, best to disable it also

Post results of each step and doin't forget to restore your normal network setup .


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#8 2019-06-08 13:13:26

mvasi90
Member
Registered: 2017-02-02
Posts: 41

Re: [Solved] VPNC Route Gateway disappears

Also there's no route disappearing in #6

After starting vpnc

yyy.yyy.yyy.yy via 192.168.0.1 dev eth0 src 192.168.0.151

120 seconds after vpnc started

deleted

maybe because your script restored it already ?

You can see that I'm running vpnc manually without any script. The tunX interface does not exist before starting vpnc. Watch the asciinema video.

Good news:
A. Changing the lease from 600s to 6000s works. But it does not seem to be definitive, it's just an increase.
I can use a static IP, but I would like to know exactly what happens. Why the route is lost?

B. I do not need dnsmasq. Only to give IPs through hostapd. But I always use static IPs.

In fact, dnsmasq depends on the wlan0 interface that is "started" by the hostapd service and therefore dnsmasq does not start with the system. It fails.

I tried to change the dnsmasq service and make it depend on the interface, also of hostapd service but it does not work well.

$ systemctl status dnsmasq
 dnsmasq.service - A lightweight DHCP and caching DNS server
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-06-07 14:46:07 CEST; 23h ago
     Docs: man:dnsmasq(8)
  Process: 273 ExecStartPre=/usr/bin/dnsmasq --test (code=exited, status=0/SUCCESS)
  Process: 278 ExecStart=/usr/bin/dnsmasq -k --enable-dbus --user=dnsmasq --pid-file (code=exited, status=2)
 Main PID: 278 (code=exited, status=2)

jun 07 14:46:06 SYSTEM2 systemd[1]: Starting A lightweight DHCP and caching DNS server...
jun 07 14:46:06 SYSTEM2 dnsmasq[273]: dnsmasq: revisión de sintaxis OK.
jun 07 14:46:07 SYSTEM2 dnsmasq[278]: dnsmasq: interfase desconocida wlan0
jun 07 14:46:07 SYSTEM2 systemd[1]: dnsmasq.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
jun 07 14:46:07 SYSTEM2 dnsmasq[278]: interface disconnected wlan0
jun 07 14:46:07 SYSTEM2 systemd[1]: dnsmasq.service: Failed with result 'exit-code'.
jun 07 14:46:07 SYSTEM2 dnsmasq[278]: el inicio ha FALLADO
jun 07 14:46:07 SYSTEM2 systemd[1]: Failed to start A lightweight DHCP and caching DNS server.

C. Using dhcpcd instead of systemd-networkd seems to solve the problem because dhcpcd does not use the lease.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.152/24 brd 192.168.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::171:c009:23zz:zzzz/64 scope link 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether zz:zz:zz:zz:zz:zz brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global noprefixroute wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::22cf:5910:c7zz:zzzz/64 scope link 
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1412 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet yyy.yyy.yyy.yyy/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::56bc:bc2d:82zz:zzzz/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
$ sudo systemctl list-unit-files --state enabled
UNIT FILE                              STATE  
autovt@.service                        enabled
dbus-org.freedesktop.resolve1.service  enabled
dbus-org.freedesktop.timesync1.service enabled
dhcpcd.service                         enabled
dnsmasq.service                        enabled
getty@.service                         enabled
haveged.service                        enabled
hostapd.service                        enabled
iptables.service                       enabled
sshd.service                           enabled
systemd-resolved.service               enabled
systemd-timesyncd.service              enabled
remote-fs.target                       enabled

Last edited by mvasi90 (2019-06-08 13:22:09)

Offline

#9 2019-06-10 12:43:33

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [Solved] VPNC Route Gateway disappears

You're right, I overlooked the missing route.

What I THINK happens is that when the dhcp lease expires, a new lease is requested and routes are reset.

The weird thing here is that the route added by vpnc is supposed to be static and should be kept when a dhcp lease is renewed.
This suggests one of your network services does something it's not supposed to do.

If you want to troubleshoot further, simplify the setup step by step. You want to be sure as little as possible is running.

I'd start with disabling sshd , hostapd and dnsmasq, then test again.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#10 2019-06-10 12:59:05

mvasi90
Member
Registered: 2017-02-02
Posts: 41

Re: [Solved] VPNC Route Gateway disappears

For now I will not do anything because I'm very busy.

Thank you for your help.

Offline

Board footer

Powered by FluxBB