You are not logged in.
I would like to preface this question by stating that the OS I will be providing information from is Manjaro. I have posed a question on their forums regarding this issue and did not receive much in the way of suggestions or help. I have observed the exact behavior I will describe below when booting to live version of EndeavourOS, Manjaro, and the arch install disk. If this is not allowed I apologize.
After the most recent 6.5 kernel and linux-firmware updates I began to notice very inconsistent network performance. Local network performance could reach 1gb, but is spiky and fluctuates a great deal. Internet performance was what gave the issue away. Speed tests seem to be artificially capped at around 10Mbps, but I get peaks reaching my maximum bandwidth (300Mbps) at times. The first indication that this issue may have been introduced with either the kernel or linux-firmware was when I booted to an older arch install I had running kernel 6.0.8. Unfortunately, after booting back and forth to this install a couple of times the issue followed me there. I do have a journalctl dump from prior to the issue appearing though. I have booted into kernel 6.4, 6.5, and 6.6-rc1 all with the same poor network performance. Given how the problem eventually migrated to the arch install running kernel 6.0, I imagine the kernel and firmware packages may not be the issue at all.
I dual boot Windows 11 on this machine. Windows is installed on a separate drive, with fastboot and hibernation diabled. Any switching between Windows and Linux is done with a system shutdown in between. When in Windows, network performance is flawless. Internet speedtests show a steady connection speed capped at 300+Mbps. Iperf tests between this machine and another arch virtual machine maintain a steady connection state at 1Gbps. Results of these tests will be provided.
To give you a visual idea of the problem I am dealing with, please see this image. I believe the peak at the beginning of the test was near 300Mbps.
Further troubleshooting has involved the following:
Replace network cable
Replace network switch
Disable auto negotiation and set interface speed to 1000 Mbps. This has been set from the switch and through ethtool. I usually see a message in dmesg that manual configuration is not available.
Add pcie_port_pm=off pcie_aspm.policy=performance to my kernel boot parameters. I have ASPM disabled in the BIOS.
Disable ipv6 using kernel boot parameter ipv6.disable=1.
Remove DHCP reservation from router in case of any issue with different host name between Windows and Linux.
Journalctl outputs:
6.6.0-1_Manjaro, pcie_aspm.policy=performance pcie_aspm=off -> http://0x0.st/HJtG.txt
6.6.0-1_Manjaro, ipv6.disabled=1 -> http://0x0.st/HJgC.txt
6.4.16_Manjaro, ipv6.disabled=1 -> http://0x0.st/HJEr.txt
6.0.8-1_Arch the only working boot -> http://0x0.st/HJgd.0_journa
inxi -Fxz
System:
Kernel: 6.4.16-5-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
Desktop: KDE Plasma v: 5.27.8 Distro: Manjaro Linux base: Arch Linux
Machine:
Type: Desktop System: Gigabyte product: X670E AORUS MASTER v: -CF
serial: <superuser required>
Mobo: Gigabyte model: X670E AORUS MASTER serial: <superuser required>
UEFI: American Megatrends LLC. v: F13c date: 08/16/2023
Battery:
ID-1: hidpp_battery_0 charge: 60% condition: N/A volts: 3.9 min: N/A
model: Logitech G502 LIGHTSPEED Wireless Gaming Mouse status: discharging
CPU:
Info: 12-core model: AMD Ryzen 9 7900X bits: 64 type: MT MCP arch: Zen 4
rev: 2 cache: L1: 768 KiB L2: 12 MiB L3: 64 MiB
Speed (MHz): avg: 3136 high: 4700 min/max: 3000/5733 boost: enabled cores:
1: 3000 2: 3000 3: 3000 4: 3000 5: 3000 6: 3000 7: 3000 8: 3000 9: 3000
10: 3000 11: 2879 12: 3000 13: 3000 14: 4700 15: 3000 16: 3000 17: 3000
18: 3000 19: 3000 20: 3000 21: 4700 22: 3000 23: 3000 24: 3000
bogomips: 225698
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
Device-1: NVIDIA GP104 [GeForce GTX 1080] driver: nvidia v: 535.113.01
arch: Pascal bus-ID: 01:00.0
Display: x11 server: X.Org v: 21.1.8 driver: X: loaded: nvidia gpu: nvidia
resolution: 1: 2560x1440~144Hz 2: 2560x1440~60Hz
API: EGL v: 1.5 drivers: nvidia,swrast platforms:
active: gbm,x11,surfaceless,device inactive: wayland,device-1
API: OpenGL v: 4.6.0 compat-v: 4.5 vendor: nvidia mesa v: 535.113.01
glx-v: 1.4 direct-render: yes renderer: NVIDIA GeForce GTX 1080/PCIe/SSE2
API: Vulkan v: 1.3.264 drivers: nvidia surfaces: xcb,xlib devices: 1
Audio:
Device-1: NVIDIA GP104 High Definition Audio driver: snd_hda_intel v: kernel
bus-ID: 01:00.1
Device-2: Creative Sound Blaster X4
driver: cdc_acm,hid-generic,snd-usb-audio,usbhid type: USB bus-ID: 3-3:2
Device-3: Logitech Blue Microphones
driver: hid-generic,snd-usb-audio,usbhid type: USB bus-ID: 9-1.4:4
API: ALSA v: k6.4.16-5-MANJARO status: kernel-api
Server-1: PipeWire v: 0.3.81 status: active
Network:
Device-1: Intel Ethernet I225-V vendor: Lenovo driver: igc v: kernel
port: N/A bus-ID: 0d:00.0
IF: enp13s0 state: up speed: 2500 Mbps duplex: full mac: <filter>
Device-2: Intel Wi-Fi 6 AX210/AX211/AX411 160MHz driver: N/A
bus-ID: 0e:00.0
Bluetooth:
Device-1: Intel AX210 Bluetooth driver: btusb v: 0.8 type: USB bus-ID: 1-9:2
Report: btmgmt ID: hci0 rfk-id: 0 state: up address: <filter> bt-v: 5.3
lmp-v: 12
Drives:
Local Storage: total: 2.77 TiB used: 170.82 GiB (6.0%)
ID-1: /dev/nvme0n1 vendor: solidgm model: SSDPFKKW020X7 size: 1.86 TiB
temp: 46.9 C
ID-2: /dev/nvme1n1 vendor: Smart Modular Tech. model: SHPP41-1000GM
size: 931.51 GiB temp: 51.9 C
Partition:
ID-1: / size: 915.53 GiB used: 170.82 GiB (18.7%) fs: ext4
dev: /dev/nvme1n1p2
ID-2: /boot/efi size: 299.4 MiB used: 316 KiB (0.1%) fs: vfat
dev: /dev/nvme1n1p1
Swap:
Alert: No swap data was found.
Sensors:
System Temperatures: cpu: 43.0 C mobo: N/A gpu: nvidia temp: 48 C
Fan Speeds (rpm): N/A gpu: nvidia fan: 32%
Info:
Processes: 370 Uptime: 27m Memory: total: 32 GiB note: est.
available: 31.09 GiB used: 3.55 GiB (11.4%) Init: systemd Compilers:
gcc: 13.2.1 clang: 16.0.6 Packages: 1524 Shell: Zsh v: 5.9 inxi: 3.3.30iperf3 - Manjaro Client to Arch Server
iperf3 -c 192.168.10.221 -t 15 ✔
Connecting to host 192.168.10.221, port 5201
[ 5] local 192.168.10.160 port 49778 connected to 192.168.10.221 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 11.6 MBytes 97.0 Mbits/sec 94 389 KBytes
[ 5] 1.00-2.00 sec 57.1 MBytes 479 Mbits/sec 46 342 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 948 Mbits/sec 47 426 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec 93 362 KBytes
[ 5] 4.00-5.00 sec 25.8 MBytes 216 Mbits/sec 0 413 KBytes
[ 5] 5.00-6.00 sec 58.3 MBytes 489 Mbits/sec 47 386 KBytes
[ 5] 6.00-7.00 sec 7.83 MBytes 65.6 Mbits/sec 0 396 KBytes
[ 5] 7.00-8.00 sec 12.4 MBytes 104 Mbits/sec 0 417 KBytes
[ 5] 8.00-9.00 sec 1.12 MBytes 9.38 Mbits/sec 0 417 KBytes
[ 5] 9.00-10.00 sec 4.54 MBytes 38.1 Mbits/sec 0 421 KBytes
[ 5] 10.00-11.00 sec 4.47 MBytes 37.5 Mbits/sec 0 426 KBytes
[ 5] 11.00-12.00 sec 3.36 MBytes 28.1 Mbits/sec 0 430 KBytes
[ 5] 12.00-13.00 sec 7.89 MBytes 66.2 Mbits/sec 0 438 KBytes
[ 5] 13.00-14.00 sec 10.1 MBytes 85.0 Mbits/sec 0 451 KBytes
[ 5] 14.00-15.00 sec 5.59 MBytes 46.9 Mbits/sec 46 325 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 435 MBytes 243 Mbits/sec 373 sender
[ 5] 0.00-15.16 sec 432 MBytes 239 Mbits/sec receiveriperf3 - Arch Client to Manjaro Server
iperf3 -c 192.168.10.160 -t 15 1 ✘ 1m 25s patrick@scout
Connecting to host 192.168.10.160, port 5201
[ 5] local 192.168.10.221 port 59844 connected to 192.168.10.160 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 7.23 MBytes 60.6 Mbits/sec 124 48.1 KBytes
[ 5] 1.00-2.00 sec 35.8 MBytes 300 Mbits/sec 197 42.4 KBytes
[ 5] 2.00-3.00 sec 2.36 MBytes 19.8 Mbits/sec 17 46.7 KBytes
[ 5] 3.00-4.00 sec 17.1 MBytes 143 Mbits/sec 280 39.6 KBytes
[ 5] 4.00-5.00 sec 24.6 MBytes 207 Mbits/sec 90 41.0 KBytes
[ 5] 5.00-6.00 sec 16.3 MBytes 137 Mbits/sec 331 49.5 KBytes
[ 5] 6.00-7.00 sec 19.1 MBytes 160 Mbits/sec 319 43.8 KBytes
[ 5] 7.00-8.00 sec 9.01 MBytes 75.6 Mbits/sec 317 36.8 KBytes
[ 5] 8.00-9.00 sec 10.1 MBytes 84.5 Mbits/sec 117 147 KBytes
[ 5] 9.00-10.00 sec 47.7 MBytes 400 Mbits/sec 339 58.0 KBytes
[ 5] 10.00-11.00 sec 109 MBytes 915 Mbits/sec 10 383 KBytes
[ 5] 11.00-12.00 sec 52.4 MBytes 439 Mbits/sec 741 56.6 KBytes
[ 5] 12.00-13.00 sec 70.1 MBytes 588 Mbits/sec 5 286 KBytes
[ 5] 13.00-14.00 sec 52.5 MBytes 440 Mbits/sec 619 96.2 KBytes
[ 5] 14.00-15.00 sec 19.5 MBytes 163 Mbits/sec 362 45.2 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-15.00 sec 493 MBytes 276 Mbits/sec 3868 sender
[ 5] 0.00-15.00 sec 491 MBytes 274 Mbits/sec receiveriperf3 - Windows Client to Arch Server
.\iperf3.exe -c scout.lan -t 20
Connecting to host scout.lan, port 5201
[ 4] local 192.168.10.3 port 49814 connected to 192.168.10.221 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 114 MBytes 951 Mbits/sec
[ 4] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 4] 2.00-3.00 sec 113 MBytes 949 Mbits/sec
[ 4] 3.00-4.00 sec 113 MBytes 948 Mbits/sec
[ 4] 4.00-5.00 sec 113 MBytes 950 Mbits/sec
[ 4] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 4] 6.00-7.00 sec 113 MBytes 949 Mbits/sec
[ 4] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 4] 8.00-9.00 sec 113 MBytes 949 Mbits/sec
[ 4] 9.00-10.00 sec 113 MBytes 948 Mbits/sec
[ 4] 10.00-11.00 sec 113 MBytes 950 Mbits/sec
[ 4] 11.00-12.00 sec 113 MBytes 949 Mbits/sec
[ 4] 12.00-13.00 sec 113 MBytes 949 Mbits/sec
[ 4] 13.00-14.00 sec 113 MBytes 949 Mbits/sec
[ 4] 14.00-15.00 sec 113 MBytes 949 Mbits/sec
[ 4] 15.00-16.00 sec 113 MBytes 949 Mbits/sec
[ 4] 16.00-17.00 sec 113 MBytes 949 Mbits/sec
[ 4] 17.00-18.00 sec 113 MBytes 949 Mbits/sec
[ 4] 18.00-19.00 sec 113 MBytes 949 Mbits/sec
[ 4] 19.00-20.00 sec 113 MBytes 949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-20.00 sec 2.21 GBytes 949 Mbits/sec sender
[ 4] 0.00-20.00 sec 2.21 GBytes 949 Mbits/sec receiveriperf3 - Arch Client to Windows Server (to show retransmissions in comparison to Manjaro runs)
iperf3 -c 192.168.10.3 -t 20
Connecting to host 192.168.10.3, port 5201
[ 5] local 192.168.10.221 port 33658 connected to 192.168.10.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 951 Mbits/sec 0 218 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 0 218 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 945 Mbits/sec 0 218 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 0 218 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 218 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 218 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 15 158 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 211 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 211 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 0 211 KBytes
[ 5] 10.00-11.00 sec 113 MBytes 946 Mbits/sec 0 211 KBytes
[ 5] 11.00-12.00 sec 112 MBytes 942 Mbits/sec 0 211 KBytes
[ 5] 12.00-13.00 sec 112 MBytes 939 Mbits/sec 0 211 KBytes
[ 5] 13.00-14.00 sec 113 MBytes 945 Mbits/sec 0 211 KBytes
[ 5] 14.00-15.00 sec 112 MBytes 940 Mbits/sec 0 211 KBytes
[ 5] 15.00-16.00 sec 113 MBytes 945 Mbits/sec 0 211 KBytes
[ 5] 16.00-17.00 sec 112 MBytes 942 Mbits/sec 0 211 KBytes
[ 5] 17.00-18.00 sec 112 MBytes 938 Mbits/sec 0 211 KBytes
[ 5] 18.00-19.00 sec 113 MBytes 945 Mbits/sec 0 211 KBytes
[ 5] 19.00-20.00 sec 112 MBytes 942 Mbits/sec 0 211 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 2.19 GBytes 943 Mbits/sec 15 sender
[ 5] 0.00-20.00 sec 2.19 GBytes 942 Mbits/sec receiverThe high retransmission rate in iperf in arch/Manjaro compared to the Windows run certainly seems to be an avenue to explore, I'm just not sure where to look. Any help or suggestions are really appreciated.
Offline
From a random majaro boot
Oct 22 17:47:25 patrick-linux kernel: iwlwifi 0000:0e:00.0: no suitable firmware found!
Oct 22 17:47:25 patrick-linux kernel: iwlwifi 0000:0e:00.0: minimum version required: iwlwifi-ty-a0-gf-a0-39
Oct 22 17:47:25 patrick-linux kernel: iwlwifi 0000:0e:00.0: maximum version supported: iwlwifi-ty-a0-gf-a0-78
Oct 22 17:47:25 patrick-linux kernel: iwlwifi 0000:0e:00.0: check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.gitMake sure to install linux-firmware and post the journal of a problematic archlinux boot.
Online
This journal from my arch install tries to load iwlwifi-ty-a0-gf-ao-72.ucode: http://0x0.st/HJlq.txt
I considered bringing this issue up in my original post, since I've seen the journal spammed with iwlwifi errors in every OS I've tried to use since beginning to track this problem down but didn't want to muddy the water any further.
From my boot to the arch install disk, I had to disable wifi to not be spammed to death in the console with these errors. Disabling wifi (rfkill) did not help my original issue however. In manjaro, I followed the advice of another thread here, which I cannot find at the moment, suggesting to rename the iwlwifi-ty-a0-gf-a0-62 to 78 to see if that improved anything. I was still having the issue and think I ended up moving all of that series of firmware to the point it can't find any suitable firmware.
Offline
Yes, but that's "Linux version 6.0.8-arch1-1" and your OP says "6.0.8-1_Arch the only working boot"
Or is it that that boot specifically workd while other runs - on this pretty dated kernel - do not?
Not having any firmware is rather not going to work.
The usual appraoch on suspected regressions is to move the newest versions (forcing the kernel to fall back to an older version) until the system stabilizes.
Replacing firmware versions isn't a good idea b/c the versions might be special-cased in the kernel module, the module might reject the FW b/c the name and internal version mismatch or you might blindly pick a version that's not even considered by your kernel (ie. the rename is inert)
Online
Sorry for the confusion. That is the install that initially did not have network issues, but after booting different distros a few times and going back, the problem became apparent in that install as well. Not sure how to explain that since it has not been updated.
I have hesitated to do a clean install of any distro simply because the process will take quite a while with the current network issues. I'll do a clean install of arch and report back.
Offline
Oct 14 23:01:02 archlinux kernel: pci 0000:0f:00.0: [c0a9:2263] type 00 class 0x010802Shows up in the good but not the bad boot and is apparently an nvme
Other than that there're way more iwlwifi firmware errors in the bad boot, thesy all show up after NM.
Assuming a race condition, what if you add iwlwifi to the initramfs and/or disable NM and only explicitly start it after you got yourself a cup of coffee?
Online
Seth, thanks for your help so far. Here is a clean install of arch, with kernel 6.5.8: http://0x0.st/HJ0_.txt. Still exhibiting the same issues.
I think this is the opposite of what you were suggesting, but I've blacklisted iwlwifi and regenerated initramfs. At the same time, I've disabled NetworkManager.service, and have been starting it after boot. Journal of the above changes: http://0x0.st/HJ0y.txt
Last edited by grindstaffp (2023-10-23 20:38:50)
Offline
iwlwifi isn't in the last journal - the point of blacklisting it is to prevent it's automatic loading. You'll have to manually "modprobe iwlwifi" at some point w/ that setup.
Online
Latest journal. Booting with iwlwifi blacklisted and networkmanager disabled. Upon boot, start network manager and modprobe iwlwifi.
http://0x0.st/HJ0y.txt
Offline
iwlwifi isn't in that journal - the suggested order would be:
- boot
- coffee
- modprobe iwlwifi
- MORE COFFEE
- Start NM
- i dRiNk tOo mUcH CoFfEe
- in doubt explicitly connect the wifi from NM
However, if it happens w/ iwlwifi blacklisted/unavailable the NIC is irrelvant and the iwlwifi/errors a red herring…
Since this happens across systems and even the one good "6.0.8-1_Arch" seems to have been a fluke: try some live distro like grml.org to check the behavior on a completely different SW stack.
Also make sure that fastboot is _still_ disabled (cause windows re-enables it w/ updates)
Also, going back to the OP, the iperfs between windows and arch are actually OK (there's a single short RT burst in one stint, but that's randomly "normal") - the massive problems are only shown in interaction between arch and manjaro - do you have an iperf arch/arch or grml/arch,arch/grml?
Online
Quick update: I think it is safe to say that the issues I was having were hardware related, not software. A few other issues started to present themselves and decided to try a different motherboard. New board has same intel 2.5Gb NIC and it works flawlessly. WiFi adapter is not intel on this board and I haven't tried it out but no error messages either. I appreciate the time you took to help me troubleshoot.
Offline