[SOLVED]Network interface connection bouncing up and down on home serv

hersi · 2023-07-16 21:36:54

Hello!

I recently built a server PC and got a multi-port network card for it. I plan to use it as a network gateway. But, after running for a while (after a day or two), it starts to behave erratically.
The connection to other devices on the network interrupts randomly. I thought it was some ASPM problem with PCIe, but after disabling it in the BIOS it still does this.

I'm at my wits' end on this one. Maybe someone here ran into this one as well?

# journalctl -keb
Jul 16 22:59:44 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Down
Jul 16 22:59:44 NAS-GW kernel: br0: port 2(int0) entered disabled state
Jul 16 22:59:47 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Up - 2.5Gbps/Full - flow control rx/tx
Jul 16 22:59:47 NAS-GW kernel: br0: port 2(int0) entered blocking state
Jul 16 22:59:47 NAS-GW kernel: br0: port 2(int0) entered forwarding state
Jul 16 22:59:47 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Down
Jul 16 22:59:48 NAS-GW kernel: br0: port 2(int0) entered disabled state
Jul 16 22:59:51 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Up - 2.5Gbps/Full - flow control rx/tx
Jul 16 22:59:51 NAS-GW kernel: br0: port 2(int0) entered blocking state
Jul 16 22:59:51 NAS-GW kernel: br0: port 2(int0) entered forwarding state
Jul 16 22:59:52 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Down
Jul 16 22:59:52 NAS-GW kernel: br0: port 2(int0) entered disabled state
Jul 16 22:59:59 NAS-GW kernel: r8169 0000:09:00.0 int0: Link is Up - 2.5Gbps/Full - flow control rx/tx
Jul 16 22:59:59 NAS-GW kernel: br0: port 2(int0) entered blocking state
Jul 16 22:59:59 NAS-GW kernel: br0: port 2(int0) entered forwarding state

#  uname -a
Linux NAS-GW 6.4.1-arch2-1 #1 SMP PREEMPT_DYNAMIC Tue, 04 Jul 2023 08:39:40 +0000 x86_64 GNU/Linux

#  lspci -v | grep -A20 RTL
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Flags: bus master, fast devsel, latency 0, IRQ 17, NUMA node 0
        I/O ports at d000 [size=256]
        Memory at fb410000 (64-bit, non-prefetchable) [size=64K]
        Memory at fb420000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fb400000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178] Transaction Processing Hints
        Capabilities: [204] Latency Tolerance Reporting
        Capabilities: [20c] L1 PM Substates
        Capabilities: [21c] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125

0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 0
        I/O ports at c000 [size=256]
        Memory at fb310000 (64-bit, non-prefetchable) [size=64K]
        Memory at fb320000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fb300000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178] Transaction Processing Hints
        Capabilities: [204] Latency Tolerance Reporting
        Capabilities: [20c] L1 PM Substates
        Capabilities: [21c] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125

0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Flags: bus master, fast devsel, latency 0, IRQ 17, NUMA node 0
        I/O ports at b000 [size=256]
        Memory at fb210000 (64-bit, non-prefetchable) [size=64K]
        Memory at fb220000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fb200000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178] Transaction Processing Hints
        Capabilities: [204] Latency Tolerance Reporting
        Capabilities: [20c] L1 PM Substates
        Capabilities: [21c] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125

0d:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 0
        I/O ports at a000 [size=256]
        Memory at fb110000 (64-bit, non-prefetchable) [size=64K]
        Memory at fb120000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fb100000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178] Transaction Processing Hints
        Capabilities: [204] Latency Tolerance Reporting
        Capabilities: [20c] L1 PM Substates
        Capabilities: [21c] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125

Here's my networkd config:

#  cat /etc/systemd/network/*

# /etc/systemd/network/10-br0.netdev
[NetDev]
Name=br0
Kind=bridge
MACAddress=a0:36:9f:50:cf:ef

# /etc/systemd/network/10-ext0.link
[Match]
MACAddress=ac:9e:17:b5:50:bd

[Link]
Description=External network interface (WAN)
Name=ext0

# /etc/systemd/network/10-int0.link
[Match]
MACAddress=a0:36:9f:50:cf:e8
Type=ether

[Link]
Name=int0

# /etc/systemd/network/10-int1.link
[Match]
MACAddress=a0:36:9f:50:cf:e9
Type=ether

[Link]
Name=int1

# /etc/systemd/network/10-int2.link
[Match]
MACAddress=a0:36:9f:50:cf:ea
Type=ether

[Link]
Name=int2

# /etc/systemd/network/10-int3.link
[Match]
MACAddress=a0:36:9f:50:cf:eb
Type=ether

[Link]
Name=int3

# /etc/systemd/network/20-bridge.network
[Match]
Name=br0

[Network]
Address=192.168.1.1/24
IPMasquerade=ipv4
IPForward=ipv4
DHCPServer=true

[DHCPServer]
PoolOffset=100
PoolSize=150
EmitDNS=yes
DNS=192.168.1.1

# /etc/systemd/network/20-external.network
[Match]
Name=ext0

[Network]
DHCP=ipv4
IPForward=ipv4

# /etc/systemd/network/21-int0.network
[Match]
Name=int0

[Network]
Bridge=br0
LinkLocalAddressing=no

# /etc/systemd/network/21-int1.network
[Match]
Name=int1

[Network]
Bridge=br0
LinkLocalAddressing=no

# /etc/systemd/network/21-int2.network
[Match]
Name=int2

[Network]
Bridge=br0
LinkLocalAddressing=no

# /etc/systemd/network/21-int3.network
[Match]
Name=int3

[Network]
Bridge=br0
LinkLocalAddressing=no

Last edited by hersi (2023-07-18 13:27:02)

seth · 2023-07-16 21:58:09

You apparently have https://aur.archlinux.org/packages/r8125-dkms (I guess it's the dkms package) installed but it's not in use?
Try adding "module_blacklist=r8169" to the kernel parametes.

hersi · 2023-07-16 22:18:12

Ah right, I was meaning to test the realtek driver. I put your suggestion into /etc/modprobe.d/rtl8125.conf.
I'll try it with that driver for the time being.

hersi · 2023-07-16 23:43:49

Well... no, the realtek driver didn't help.
The message is slightly different but it's doing the same.

[ 5519.345602] r8125: int0: link down
[ 5519.350742] br0: port 1(int0) entered disabled state
[ 5522.245933] r8125: int0: link up
[ 5522.245965] br0: port 1(int0) entered blocking state
[ 5522.245969] br0: port 1(int0) entered forwarding state

seth · 2023-07-17 06:09:09

The problem in the OP is that the link goes from up to down (immediately) again, please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'f:1=<-' ix.io

and on a limb the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

hersi · 2023-07-17 11:53:13

seth wrote:

The problem in the OP is that the link goes from up to down (immediately) again, please post your complete system journal for the boot:
sudo journalctl -b | curl -F 'f:1=<-' ix.io

This command outputs nothing for me.

The second command outputs:

#  find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

certbot.timer                            | timers.target.wants
cockpit.socket                           | sockets.target.wants
container-nextcloud-db.service           | default.target.wants
container-nextcloud-elasticsearch.service | default.target.wants
container-nextcloud-web.service          | default.target.wants
dbus-org.freedesktop.network1.service    | system
dbus-org.freedesktop.nm-dispatcher.service | system
ddclient.service                         | multi-user.target.wants
getty@tty1.service                       | getty.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
nextcloudcron.timer                      | timers.target.wants
nginx.service                            | multi-user.target.wants
ntpd.service                             | multi-user.target.wants
openvpn-server@server.service            | multi-user.target.wants
p11-kit-server.socket                    | sockets.target.wants
pihole-FTL.service                       | multi-user.target.wants
pmcd.service                             | multi-user.target.wants
pmlogger.service                         | multi-user.target.wants
pod-gitea.service                        | default.target.wants
pod-gitea.service                        | multi-user.target.wants
podman.socket                            | sockets.target.wants
pod-nextcloud.service                    | default.target.wants
remote-fs.target                         | multi-user.target.wants
shorewall.service                        | basic.target.wants
smb.service                              | multi-user.target.wants
sshd.service                             | multi-user.target.wants
systemd-networkd.service                 | multi-user.target.wants
systemd-networkd.socket                  | sockets.target.wants
systemd-networkd-wait-online.service     | network-online.target.wants
unbound.service                          | multi-user.target.wants

seth · 2023-07-17 12:27:07

You've systemd-networkd and NetworkManager enabled, pick one, disable the other (unless you're positively sure that the controlled NICs are disjunct)
Ftr, you're also running unbound and idk how systemd-networkd responds to not be paired w/ resolved.

This command outputs nothing for me.

It either responds a url or an error.

hersi · 2023-07-17 12:46:56

NetworkManager is only running because of cockpit network moditoring, it doesn't actually manage any of the interfaces, but I will disable it in case it does interfere.
Unbound worked well with the SBC I was using before, so I don't think that's the problem, but I will test if it makes a difference using resolved instead.

As for the curl command, this is all it prints if I -v it:

#  journalctl -b | curl -v -F 'f:1=<-' ix.io
*   Trying 66.172.11.73:80...
* Connected to ix.io (66.172.11.73) port 80 (#0)
> POST / HTTP/1.1
> Host: ix.io
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 33747974
> Content-Type: multipart/form-data; boundary=------------------------fa48ede97dac3bec
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 17 Jul 2023 12:02:43 GMT
< Server: Apache
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host ix.io left intact

seth · 2023-07-17 12:51:24

sudo journalctl -b | curl -F 'file=@-' 0x0.st

hersi · 2023-07-17 13:27:35

http://0x0.st/Hj61.txt

hersi · 2023-07-17 13:44:00

I might run this again after a reboot, I just found out that my openvpn server was failing constantly and filling the logs with an immense amount of errors

Last edited by hersi (2023-07-17 13:44:30)

seth · 2023-07-17 13:45:29

Explains why ix.io doesn't work - the journal is 35 MB…

I'd first remove all the noise and see whether one of the NICs works realibly in regular mode.
No VPNs, bridges and only one network managing service.

There's a GINORMOUS amount of noise on tun0 before int0 loses the carrier the first time.

hersi · 2023-07-17 14:19:45

Okay, I rebooted and produced a much cleaner log
Sorry about the 35MB log
http://0x0.st/Hj64.txt

hersi · 2023-07-17 14:43:20

The strange thing is that the gigabit WiFi AP I have doesn't get disconnected at all. I tried switching the cables around but any 2.5Gig connections get disconnected after a while regardless of which port it's connected to.
Do you think it could be a firmware or hardware problem on the Realtek chip itself? The builtin kernel module and the realtek driver modules don't seem to have an effect on how frequently it drops connections at all. I already disabled the PCIe power saving in the BIOS as well...

seth · 2023-07-17 16:09:44

There's really no pattern nor obvious trigger.
You could try to limit the throughput in case the cables aren't good enough.

https://www.thegeeksearch.com/how-to-us … -in-linux/
tl;dr

ethtool –s int1 speed 1000 duplex full autoneg off

The "permanent settings" part of the page probalby doesn't apply to systemd-networkd, I'd have to look up whether and how you can control that through its config

hersi · 2023-07-18 10:44:50

seth wrote:

There's really no pattern nor obvious trigger.
You could try to limit the throughput in case the cables aren't good enough.

I don't want to celebrate early, but I think I fixed it.
I completely neglected the possibility of the cable being the problem, since it worked without problems until now.
I got a Cat6 cable out of my drawer, and ever since plugging that in, it hasn't dropped once.

seth · 2023-07-18 12:07:12

Get a face shield, because ewaller is gonna cum all over this thread
Cables were, are and will always remain the #1 problem in IT, electronics and bicycle breaks

hersi · 2023-07-18 13:26:01

Blast shields are up!

It's always the easiest solution that gets overlooked
Thanks for the help!

Arch Linux

#1 2023-07-16 21:36:54

[SOLVED]Network interface connection bouncing up and down on home serv

#2 2023-07-16 21:58:09

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#3 2023-07-16 22:18:12

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#4 2023-07-16 23:43:49

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#5 2023-07-17 06:09:09

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#6 2023-07-17 11:53:13

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#7 2023-07-17 12:27:07

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#8 2023-07-17 12:46:56

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#9 2023-07-17 12:51:24

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#10 2023-07-17 13:27:35

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#11 2023-07-17 13:44:00

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#12 2023-07-17 13:45:29

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#13 2023-07-17 14:19:45

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#14 2023-07-17 14:43:20

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#15 2023-07-17 16:09:44

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#16 2023-07-18 10:44:50

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#17 2023-07-18 12:07:12

Re: [SOLVED]Network interface connection bouncing up and down on home serv

#18 2023-07-18 13:26:01

Re: [SOLVED]Network interface connection bouncing up and down on home serv

Board footer