Intermittent Internet Connection Issue, New Connections Fail Randomly

spaceduck · 2023-08-04 19:07:47

Hello Everyone,
I've been having the weirdest networking/internet problem the last couple weeks, I couldn't even think of a good way to title this post.

Here are the symptoms I'm experiencing:

On boot, my system takes a very long time to establish an internet connection (maybe 2 or 3 minutes? I haven't timed it.) I'm connected to my network, I have an IP address, I can access other devices on my local network, however nothing from the internet will load. Network monitor will typically show a few bytes of traffic
Once connected, everything works as expected for a while. Seemingly at random - sometimes several times in an hour, sometimes only once in 2 hours or so - this issue with opening new connections crops up again
Things that I am already connected to continue to work fine. Zoom calls proceed uninterrupted, streaming videos continue to stream, online game matches will continue to the end of the match. It is only starting a new connection that fails
When I am unable to connect I can ping devices on my local network, but I cannot ping internet pages. I do not believe it is a DNS issue, because ping correctly lists an IP address if I ping, for example, "google.com". Ping reports 100% packet loss. Directly pinging an IP like 1.1.1.1 or 8.8.8.8 makes no difference.
After about the same time as when my system first boots, whatever issue there is has resolved, and everything works as expected

Relevant information:

I am on a wired connection and use KDE.
I've always used Network Manager and whatever its defaults are to handle my network connections. I tried doing a force reinstall of Network Manager, and when that didn't work even removing it and then installing, but neither helped.
It's not an internet issue as other devices do not have these interruptions.
I don't believe it's a hardware issue, as booting from the install media doesn't seem to experience the same problems (instant connectivity).
I had multiple adapters (a built in Realtek 1G adapter, and an add-in Realtek 2 port 2.5G adapter), but as part of the troubleshooting process have removed the add-in adapter, to no avail.
I do some virtualization with QEMU and had my physical connection and VirtManager's default virtual bridge connection bridged, to expose one of my VMs to my physical network. As part of the troubleshooting process I removed the bridge, but there was no change in symptoms.
At some point (long before this issue) I installed OpenVPN, intending to use this computer as a VPN server, however, I never finished doing all the necessary setup.
I also use this machine to run an SMB server.

I'm sure everybody says this, but I didn't make any configuration changes when the issue started. I powered my system down to install a water block on my host GPU, and when I booted back up this issue started. No, I didn't spill any water in the system, and the loop isn't leaking. It's not a new loop, it's been water cooled for almost 3 years now, I was just adding the host GPU to the loop.

Any help would be greatly appreciated. This issue is driving me nuts and I really don't want to have to do the nuclear option of just starting fresh. Thank you in advance!

seth · 2023-08-04 19:29:04

Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Please post your complete system journal for an affected boot, eg.

sudo journalctl -b -1 | curl -F 'file=@-' 0x0.st

for the previous one.

ip a; ip r

Do you configure any kind of netfilter (itables, firewalld, ufw, …)?

spaceduck · 2023-08-04 20:29:37

Output of the find command:

avahi-daemon.service                     | multi-user.target.wants
avahi-daemon.socket                      | sockets.target.wants
bluetooth.service                        | bluetooth.target.wants
dbus-fi.w1.wpa_supplicant1.service       | system
dbus-org.bluez.service                   | system
dbus-org.freedesktop.Avahi.service       | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.resolve1.service    | system
dbus-org.freedesktop.timesync1.service   | system
display-manager.service                  | system
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
iptables-openvpn.service                 | multi-user.target.wants
libvirtd-ro.socket                       | sockets.target.wants
libvirtd.service                         | multi-user.target.wants
libvirtd.socket                          | sockets.target.wants
lm_sensors.service                       | multi-user.target.wants
mysqld.service                           | multi-user.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
nmb.service                              | multi-user.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
plexmediaserver.service                  | multi-user.target.wants
pump_watchdog.service                    | multi-user.target.wants
remote-fs.target                         | multi-user.target.wants
smb.service                              | multi-user.target.wants
sshd.service                             | multi-user.target.wants
systemd-resolved.service                 | sysinit.target.wants
systemd-timesyncd.service                | sysinit.target.wants
virtlockd.socket                         | sockets.target.wants
virtlogd.socket                          | sockets.target.wants
winbind.service                          | multi-user.target.wants
wireplumber.service                      | pipewire.service.wants
wpa_supplicant.service                   | multi-user.target.wants
wsdd2.service                            | multi-user.target.wants
xdg-user-dirs-update.service             | default.target.wants

Here is the link to my system journal: http://0x0.st/H27H.txt

and here is the output of ip a; ip r:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether f0:2f:74:ac:9e:37 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.19/24 brd 192.168.0.255 scope global dynamic noprefixroute enp7s0
       valid_lft 8538622sec preferred_lft 8538622sec
    inet6 fe80::ee45:f13c:30d7:d375/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:bf:2e:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
default via 192.168.0.1 dev enp7s0 proto dhcp src 192.168.0.19 metric 100 
192.168.0.0/24 dev enp7s0 proto kernel scope link src 192.168.0.19 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Edit: forgot to answer: I haven't configured any ip tables or anything like that. I don't use firewalld - looks like it is installed, but is currently disabled. I was originally planning to run OPNSense in a VM (hence the add-in two port ethernet card, for passthrough), so I haven't configured any other sort of filtering or firewalls. I never finished setting that up, so the VM is not running, my ethernet cable is connected directly to the modem/router.

Last edited by spaceduck (2023-08-04 20:36:16)

seth · 2023-08-04 20:48:45

Warning: The journal is 10MB

Disable wpa_supplicant.service - NM starts that itself.

Then there

iptables-openvpn.service                 | multi-user.target.wants

Disable that as well.

The next obvious contender would be libvirtd.

Was every NIC you tried some RTL8125B/r8169 ? Did you try the LTS kernel?

spaceduck · 2023-08-04 21:18:29

seth wrote:

Warning: The journal is 10MB

I tend to have a lot of uptime, sorry! Here's the journal from my current boot:
http://0x0.st/H27L.txt

I disabled iptables-openvpn and wpa-supplicant. The system hasn't been running long enough for the intermittent aspect to show up, but it took 5 minutes before I could connect to anything, so I think the issue is still here.

Up until now, every NIC I've tried has been one of the realtek cards you listed. 8169 is my onboard NIC and 8125B is the add-in. Just now, I plugged in a USB wi-fi adapter that I had lying around and tested that. The issue is actually even worse, it took around 5 minutes to get a usable connection, and after only about 5 minutes it once again dropped out.

I haven't tried a different kernel, but that's a great idea, it would explain the behavior change after power cycling with no config changes. I'll figure out how to do that and give an LTS kernel a shot next.

Thanks for all of your help so far!

seth · 2023-08-04 21:23:34

https://bbs.archlinux.org/viewtopic.php … 0#p2111730

https://bugzilla.kernel.org/show_bug.cgi?id=217596#c53

Edit: you're currently on

kernel: r8169 0000:07:00.0 eth0: RTL8168h/8111h, f0:2f:74:ac:9e:37, XID 541, IRQ 114

so I'd say there's a huuuuge chance you're hitting that regression.

Last edited by seth (2023-08-04 21:24:51)

Arch Linux

#1 2023-08-04 19:07:47

Intermittent Internet Connection Issue, New Connections Fail Randomly

#2 2023-08-04 19:29:04

Re: Intermittent Internet Connection Issue, New Connections Fail Randomly

#3 2023-08-04 20:29:37

Re: Intermittent Internet Connection Issue, New Connections Fail Randomly

#4 2023-08-04 20:48:45

Re: Intermittent Internet Connection Issue, New Connections Fail Randomly

#5 2023-08-04 21:18:29

Re: Intermittent Internet Connection Issue, New Connections Fail Randomly

#6 2023-08-04 21:23:34

Re: Intermittent Internet Connection Issue, New Connections Fail Randomly

Board footer