[SOLVED] Networking not working after update, except through a VPN

LSS · 2021-06-27 14:53:34

Hi. I upgraded my Arch install yesterday, and ever since the reboot, networking only works through NordVPN. The behavior is really weird, and none of the resources I could find online was of much help.

My gateway's local IP is 192.168.1.1, and my WiFi interface is wlp3s0. Running `ip link show wlp3s0` shows the interface is up and runnning:

3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
    link/ether 5c:e0:c5:59:a4:48 brd ff:ff:ff:ff:ff:ff

Running `ip neighbour show` shows my gateway, sometimes as "REACHABLE" or "DELAY," but most of the time "STALE."

192.168.1.1 dev wlp3s0 lladdr 30:7c:b2:3a:dd:1f REACHABLE

In the wpa-supplicant CLI, running `status` shows `wpa_state=COMPLETED`, meaning the connection is established. Even better, running `ip address show wlp3s0` shows I have a local IP address, suggesting DHCP works, though this one is a bit weird since the IPv6 address seems to disappear later (this might be a byproduct of enabling the VPN, not sure):

3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:e0:c5:59:a4:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.14/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 67946sec preferred_lft 57146sec

In the interface of my router, I could see that DHCP is configured to always assign that IP address to my laptop. My routing table also appears correct as shown by `ip route show`:

default via 192.168.1.1 dev wlp3s0 proto dhcp src 192.168.1.14 metric 3003 
192.168.1.0/24 dev wlp3s0 proto dhcp scope link src 192.168.1.14 metric 3003

However, I have no actual connection as I cannot even ping my own router (`ping -c 1 -t 5 192.168.1.1`):

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

If I try to ping a domain name, I get different results:

If `systemd-resolved.service` is not running, "Temporary failure in name resolution"
If it is running, "Name or service not known"

The content of `/etc/resolv.conf` is as follows, though rebooting the machine or enabling the VPN temporarily changes it:

# Generated by resolvconf
nameserver 103.86.96.100
nameserver 103.86.99.100

Changing those DNS servers to Google's (8.8.4.4 and 8.8.8.8) does not help. Neither does restarting dhcpcd (which is run for all interfaces by the way) or rebooting the machine.

With my VPN on, I have a working Internet connection, though I am still unable to access my router by its IP. I have another laptop with an old (and extremely outdated) Gentoo install. Many of its settings, including those of dhcpcd, are very similar, and that laptop has a working connection. My other non-Linux devices too (they're also able to access the router). The VPN does meddle with `/etc/resolv.conf` and the routing table and creates another network interface called tun0.

Are there other tests I could run to pinpoint the problem better or stuff worth trying to fix it?

Thank you for your consideration.

Last edited by LSS (2021-06-28 21:21:03)

ewaller · 2021-06-27 15:28:14

Welcome to Arch. Great first post with lots of detail.
Do other systems work with that router without issue? Have you restarted the router? I am wondering if its DNS server is having issues.

LSS · 2021-06-27 18:33:46

Hi. Other systems work just fine. I also tried restarting my router with no success. It has its own DNS servers, but they don't seem to work (I can't even telnet into them on port 53). Though I'm not sure this is where the issue comes from since even with IP addresses I can't access anything without my VPN.

tucuxi · 2021-06-27 19:00:14

It is unclear how a VPN connection would be established if you cannot even reach the router.

Are you sure the VPN was disabled when you tried to ping the router?

seth · 2021-06-27 21:12:23

wlp3s0 is NIC #3, #1 is the loopback, what's #2?
Please post the complete output of "ip a; ip r" w/ and w/o VPN active.
Also

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

You can forget about the DNS situation for the moment - if you can't ping the gateway, DNS will predictably fail.

LSS · 2021-06-28 12:42:58

tucuxi wrote:

Are you sure the VPN was disabled when you tried to ping the router?

Just to be sure, I disabled the NordVPN service in systemd, and it works normally that way. But now I have to choose between disabling the entire service and using the VPN. Is it worth a shot to reinstall it?

seth wrote:

wlp3s0 is NIC #3, #1 is the loopback, what's #2?

#2 is an ethernet interface I never use.

seth wrote:

Please post the complete output of "ip a; ip r" w/ and w/o VPN active.

Without VPN

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 68:f7:28:f7:64:6d brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:e0:c5:59:a4:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.14/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 86219sec preferred_lft 75419sec

default via 192.168.1.1 dev wlp3s0 proto dhcp src 192.168.1.14 metric 3003 
192.168.1.0/24 dev wlp3s0 proto dhcp scope link src 192.168.1.14 metric 3003

With VPN

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 68:f7:28:f7:64:6d brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:e0:c5:59:a4:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.14/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 86092sec preferred_lft 75292sec
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.3.6/24 brd 10.8.3.255 scope global tun0
       valid_lft forever preferred_lft forever

tun0 is created/destroyed every time I connect/disconnect the VPN hence the gap.

0.0.0.0/1 via 10.8.3.6 dev tun0 
default via 192.168.1.1 dev wlp3s0 proto dhcp src 192.168.1.14 metric 3003 
10.8.3.0/24 dev tun0 proto kernel scope link src 10.8.3.6 
45.152.181.179 via 192.168.1.1 dev wlp3s0 
128.0.0.0/1 via 10.8.3.6 dev tun0 
192.168.1.0/24 dev wlp3s0 proto dhcp scope link src 192.168.1.14 metric 3003

seth wrote:

Also

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

dbus-fi.w1.wpa_supplicant1.service       | system
dhcpcd.service                           | multi-user.target.wants
dirmngr.socket                           | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gpg-agent-browser.socket                 | sockets.target.wants
gpg-agent-extra.socket                   | sockets.target.wants
gpg-agent.socket                         | sockets.target.wants
gpg-agent-ssh.socket                     | sockets.target.wants
nordvpnd.service                         | default.target.wants
p11-kit-server.socket                    | sockets.target.wants
postgresql.service                       | multi-user.target.wants
pulseaudio.socket                        | sockets.target.wants
remote-fs.target                         | multi-user.target.wants
wpa_supplicant.service                   | multi-user.target.wants
wpa_supplicant@wlp3s0.service            | multi-user.target.wants

seth · 2021-06-28 12:49:26

You've two wpa_supplicant services, a special and a general one.
Since you're using dhcpcd, you should disable both and instead add https://wiki.archlinux.org/title/Dhcpcd … supplicant

Sanitize that and check the impact.

LSS · 2021-06-28 15:21:04

I've done that and rebooted. This is now the output of the command you gave me:

dhcpcd.service                           | multi-user.target.wants
dirmngr.socket                           | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gpg-agent-browser.socket                 | sockets.target.wants
gpg-agent-extra.socket                   | sockets.target.wants
gpg-agent.socket                         | sockets.target.wants
gpg-agent-ssh.socket                     | sockets.target.wants
nordvpnd.service                         | default.target.wants
p11-kit-server.socket                    | sockets.target.wants
postgresql.service                       | multi-user.target.wants
pulseaudio.socket                        | sockets.target.wants
remote-fs.target                         | multi-user.target.wants

Apart from that, I'm still in the same situation. Either I connect to the VPN to access everything except my router, or I disable its service altogether.

seth · 2021-06-28 15:36:09

At least we know it's not an instable connection because of conflicting services.
And since enp0s25 has no carrier, the VPN isn't using that either.

Let's assume the router just doesn't respond to ICMP requests - can you "ping 8.8.8.8" w/o the VPN?
"nmap -Pn 192.168.1.1"?
Are there other systems on the LAN that you could try to ping?

LSS · 2021-06-28 17:13:15

seth wrote:

Let's assume the router just doesn't respond to ICMP requests - can you "ping 8.8.8.8" w/o the VPN?

The router does respond to ICMP requests when the NordVPN service is stopped. But with the service on, not even 8.8.8.8 responds:

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

seth wrote:

"nmap -Pn 192.168.1.1"?

Host discovery disabled (-Pn). All addresses will be marked 'up' and scan times will be slower.
Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-28 18:47 CEST
Verbosity Increased to 1.
Verbosity Increased to 2.
Verbosity Increased to 3.
Verbosity Increased to 4.
Verbosity Increased to 5.
Verbosity Increased to 6.
Verbosity Increased to 7.
Verbosity Increased to 8.
Verbosity Increased to 9.
Verbosity Increased to 10.
Verbosity Increased to 10.
Verbosity Increased to 10.
Verbosity Increased to 10.
Completed Parallel DNS resolution of 1 host. at 18:48, 13.00s elapsed
DNS resolution of 1 IPs took 13.00s. Mode: Async [#: 2, OK: 0, NX: 0, DR: 1, SF: 0, TR: 4, CN: 0]
Initiating Connect Scan at 18:48
Scanning 192.168.1.1 [1000 ports]
Connect Scan Timing: About 15.50% done; ETC: 18:51 (0:02:49 remaining)
Connect Scan Timing: About 30.50% done; ETC: 18:51 (0:02:19 remaining)
Connect Scan Timing: About 45.50% done; ETC: 18:51 (0:01:49 remaining)
Connect Scan Timing: About 60.50% done; ETC: 18:51 (0:01:19 remaining)
Connect Scan Timing: About 75.50% done; ETC: 18:51 (0:00:49 remaining)
Completed Connect Scan at 18:51, 201.23s elapsed (1000 total ports)
Nmap scan report for 192.168.1.1
Host is up.
All 1000 scanned ports on 192.168.1.1 are filtered

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 214.30 seconds

seth wrote:

Are there other systems on the LAN that you could try to ping?

I tried pinging other machines on the LAN, with no result.

seth · 2021-06-28 18:43:41

The router does respond to ICMP requests when the NordVPN service is stopped.

However, I have no actual connection as I cannot even ping my own router

Afaiu the inaccessible gateway was unrelated to the VPN and VPN (weirdly) would however (only) allow you to access the WAN.
For clarification: Cna you ping the router and does the internet work w/ NVPN stopped, ie. is the connection issue actually because of the VPN or the other way round?

progandy · 2021-06-28 18:48:07

seth wrote:

For clarification: Cna you ping the router and does the internet work w/ NVPN stopped, ie. is the connection issue actually because of the VPN or the other way round?

If ping and internet works with the nordvpn service stopped, but not with nordvpn running unconnected, then you might have enabled the kill switch in nordvpn.
https://nordvpn.com/en/features/kill-switch-technique/

LSS · 2021-06-28 21:20:45

progandy wrote:

If ping and internet works with the nordvpn service stopped, but not with nordvpn running unconnected, then you might have enabled the kill switch in nordvpn.
https://nordvpn.com/en/features/kill-switch-technique/

Holy cow that was it! I've had kill switch permanently on for over a year, and it never actually worked, so I didn't even suspect it...
Now I feel really dumb. Still, thanks a lot!

Arch Linux

#1 2021-06-27 14:53:34