You are not logged in.

#1 2020-11-01 19:18:22

buzuddha
Member
Registered: 2020-10-02
Posts: 72

[SOLVED] Recent abrupt decline in NFS file transfer speeds

First, thanks for any help anyone is willing to give!

I have two ubuntu 20.04 servers with two different software RAIDs on them. One is music and movies (RAID6) and the other is personal docs and photos (RAIDZ2).

I mount and manage these from my laptop uname -r gives 5.9.1-arch1-1. Over the past 6 months, I bought a new router and have been getting really fast transfer speeds. Typically every file starts at 250MB/s and descends down to a plateau of ~50MB/s(RAID6) or ~70MB/s (RAIDZ2). Great, I can transfer a bluray file in about 2-3 minutes from the file browser or command line without issue.

In the past week or so, file transfers to either NAS start at 250MB/s then descend quickly past the usual plateau, the RAID6 plateaus at 5MB/s and the RAIDZ2 down at 19MB/s. On top of this, when doing file transfers within the file browser, the files will hang at 0% remaining for a 3-10 minutes. I assume this is some recent update on the client as these systems are managed on two different servers with two different softwares (mdadm vs zfs).

mount output (client side) for RAID6 and ZFS servers in order

192.168.192.175:/mnt/Share/raid6 on /home/name/raid6 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.192.164,local_lock=none,addr=192.168.192.175)
192.168.192.201:/mnt/raidz2 on /home/name/raidz2 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.192.164,local_lock=none,addr=192.168.192.201)

/etc/exports on the RAID6 is

/mnt/Share/raid6      192.168.192.1/24(rw,insecure,no_subtree_check,all_squash,anonuid=1000,anongid=1000)

/etc/exports on the RAIDZ2 is

/mnt/raidz2	192.168.192.1/24(rw,no_root_squash,no_subtree_check)

Both mdadm --detail and zpool status -v come back clean for both RAIDs and have more than 50% disk available.

TL;DR - healthy network RAIDS both showed abrupt dramatic decline in write performance

also posted this here too.

Last edited by buzuddha (2020-11-24 21:25:14)

Offline

#2 2020-11-01 21:10:12

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Ignoring all speed-o-meters and looking at a wall clock, stuff really got slower?
Do you experience the same w/ the LTS kernel and/or nfs-cp?
Download from the servers is ok?
Does this effect somehow correlate to the VLC situation (ie. uploading gets slow once you had a hanging VLC connection to the server and SIGKILL'd it)?

Offline

#3 2020-11-02 13:40:10

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Based on a wall clock, writes are down to about 5MB/s for server 1. So yes, stuff got much slower. Especially the hanging at 0% remaining.

nfs-cp is also bad.

Download from server 1: 11-12 MB/s: Bad - need to look into this
Download from server 2: 40-50 MB/s: Good

Not correlated with hanging processes from VLC as far as I can tell (killing it on the client doesn't solve the problem)

Also, if I mount server 1 on to server 2 (LAN connection) and copy a 6.5gb file, it take 97s, so 67MB/s which is a little slow, but not too bad.

Last edited by buzuddha (2020-11-02 13:44:41)

Offline

#4 2020-11-02 13:43:32

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

The question is whether it's already bad *before* any vlc attempts (because force killing vlc will not cleanly close the connection to the server)

nfs-cp is also bad

doesn't seem to be the nfs implementation - did you download via nfs or eg. http?

Offline

#5 2020-11-02 13:46:13

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

I see.

I downloaded via NFS. Don't know how to download via http.

I have tried ye olde turn it all off and turn it all back on. Let me try once more though...

Last edited by buzuddha (2020-11-02 13:47:44)

Offline

#6 2020-11-02 14:03:29

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Ok back up to 50 MB/s down from server 1 after hard reset of entire network, but the write issue remains. No VLC instances were opened, so there shouldn't be anything hanging.

Offline

#7 2020-11-05 23:06:21

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Update: I set up samba for this drive between my laptop and the RAID6 drive and I still get 5 MB/s (wall clock calculation), though no hangs at the end of transfers. Transfers from a USB stick onto the RAID6 are 350 MB/s (wall clock calculation), so the locally everything is very fast.

Offline

#8 2020-11-06 08:53:48

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

So it's likely more the transport layer or lower.

seth wrote:

Do you experience the same w/ the LTS kernel

Also please post a complete system journal (after a slow upload in case there're disconnects) - do you have an alternative NIC available?

Offline

#9 2020-11-11 00:40:03

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Hi Seth! Thanks for getting back @ me
Question
So by "experience the same with the LTS kernel" I'm guessing = "have you tried with Ubuntu?"

A: yes but only on server 2 (houses the RAIDZ2) which is a wired connection - gets 67 MB/s - On server 2, uname -r gives

 5.4.0-52-generic 

I'll put ubuntu on my lenovo C940 (arch) laptop when I have a chance. Other laptops have crap cards in them. Maybe this weekend. TBD

The Journal
The complete system journal is huge! The process:

1. Rebooted
2. Ran

 mount -a 

3. Transferred a 4gb file (slow speeds and hang at the end).
4. Ran

  journalctl

I deleted 2M lines of garbage repeated lines at the beginning.
5. Had trouble uploading to Google drive in the file browser too, so used drag and drop in the web browser to upload to gdrive (quick upload). Can be found HERE . If anyone reads this and thinks this is an insane thing to do from a security standpoint, please post or DM me immediately. I am a linux hobbyist not a pro.

Also:
Plugged in a gigabit USB3.0 stick, put laptop on airplane mode, unmount/remounted everything - got a 14.9gb file to upload to the raid6 server via NFS in ~2.5 minutes via wall clock. No hang at 0% remaining.

Maybe my WiFi is interrupted by the anxiety I'm emitting as a Washington DC resident in the year 2020. I hope there's an update for that...

Last edited by buzuddha (2020-11-11 00:58:36)

Offline

#10 2020-11-11 07:47:04

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

https://www.archlinux.org/packages/core … linux-lts/

The complete system journal is huge!

sudo journalctl -b

See the first link in my signature on how to feed that into a pastebin service.

There should™ be no private data in the stuff you posted but it'll likely also be the session journals (all of them…) what's not helpful here.

Offline

#11 2020-11-11 14:23:02

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Sorry, gotcha!

Here's the journal since boot. Steps to reproduce:

1. Reboot
2. Run

mount -a

3. copy a large file to NFS (experience slow transfer)
4. Run

sudo journalctl -b

http://ix.io/2DMA

I can't see anything obvious in there, but I don't know what I'm looking for.

I think I'll have time to try a live disk of ubuntu later today!

Offline

#12 2020-11-11 14:46:33

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

You've at least dhcpcd and NM enabled, pick one, drop the other.
Sanity check:

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Then there's PIA and (maybe only because of that) a lot of iptables action - if cleaning up the NIC controlling services doesn't cut it: can you try w/o PIA to rule it out as the cause?
(Ultimately post an updated journal)

Offline

#13 2020-11-11 23:23:23

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

So I got rid of DHCP

systemctl stop dhcpcd.service
systemctl disable dhcpcd.service

The Sanity Check
DHCP is OFF

systemctl status dhcpcd.service
● dhcpcd.service - dhcpcd on all interfaces
     Loaded: loaded (/usr/lib/systemd/system/dhcpcd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

Reboot, mounted the drives, copied a 3gb file to the RAID6 server (experienced slow transfer speeds). Noticed that the speed-o-meter started up higher than it usually does ~300MB/s now (never seen it that high for WiFi), but still drops precipitously and can't be useful. I was timing but gave up. Probably took about 7' in my estimate which is about ~7MB/s.

Journal - No DHCP. I noticed that the machine tries to mount the NFS mounts but fails starting at line 1525-1555. I don't know if this is during the boot procedure or not.

Going to try the same for piavpn.service in a second.

systemctl stop piavpn.service
systemctl disable piavpn.service

The Sanity Check

systemctl status piavpn.service
● piavpn.service - Private Internet Access daemon
     Loaded: loaded (/etc/systemd/system/piavpn.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

rebooted, mount all drives, copied a different 3.1gb file to the RAID6 server (experienced slow transfer speeds). By wall clock it took 3100MB/650s = 4.7 MB/s.

Journal - No PIA.

I'll read up on downgrading kernels when time permits. Was really thinking I should just put arch on the servers...is it not smarter to bring the servers up the the newer kernel? They're just personal servers, no one else relies on them, if they're down it's no thang.

Last edited by buzuddha (2020-11-11 23:53:43)

Offline

#14 2020-11-12 07:14:42

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

One last thing: do you use nautilus to copy the files?
What about rsync or cp on the console (no nuatilus, actually no gnome: boot the multi-user.target, see 2nd link in my signature) - there might be too much too much forth and back chatter, also by the file indexing service.

Offline

#15 2020-11-13 03:56:47

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Huzzah! The via LTS kernel I get 1600MB/~32s (clocked) = ~50MB/s. Being able to just swap out a kernel like that is pretty gd amazing! Computers man! Who knew!?

The slow speeds were still apparent with cp. I can still try rsync or pv or something.

I'll swap back to arch kernel and boot to the command line and give it a shot and see what I get in a sec...

Offline

#16 2020-11-13 04:14:19

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Ok, booting into arch kernel without gnome: cp is 4.7 MB/s.

Also now my grub defaults to the LTS kernel.

Offline

#17 2020-11-13 14:10:51

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

So a kernel regression - it seems.
Since nfs-cp is "also bad" and the external dongle is not, it is more likely the interface driver.
Sounds related to https://bbs.archlinux.org/viewtopic.php?id=258042 and at some point the LTS kernel will become slow as well.

Check the pacman log for the updated packages in the critical time frame, notably which kernel you were using before.

Offline

#18 2020-11-17 02:14:18

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Ok so let's see. The Kernels

I would guess this problem popped up in October and I lived with it for a couple weeks before posting about it. So my guess is the one of these updates probably: 2Oct20, 6Oct20, 17Oct20, 25Oct20.

Here's the Full Log since September

I don't really know what I'm looking for but possibilities as far as I can tell:
[2020-10-06T12:57:50-0400] [ALPM] upgraded iw (5.8-1 -> 5.9-1)
[2020-10-06T12:57:51-0400] [ALPM] upgraded linux-firmware (20200817.7a30af1-1 -> 20200916.00a84c5-1)
[2020-10-06T12:57:51-0400] [ALPM] upgraded linux (5.8.12.arch1-1 -> 5.8.13.arch1-1)

Last edited by buzuddha (2020-11-17 02:34:26)

Offline

#19 2020-11-17 07:40:44

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Most likely

[2020-10-17T04:09:38-0400] [ALPM-SCRIPTLET] ==> Starting build: 5.8.14-arch1-1
[2020-10-25T05:48:33-0400] [ALPM-SCRIPTLET] ==> Starting build: 5.9.1-arch1-1

though that's only 1 "couple" of week.

Install 5.8.14-arch1-1 and see whether the behavior is good w/ 5.8

Nov 11 08:26:19 Squarch kernel: iwlwifi 0000:73:00.0: Detected Intel(R) Wi-Fi 6 AX200 160MHz, REV=0x340

There's a whole bunch of semi-recent threads about this chip acting up, here's some module voodoo:
https://bbs.archlinux.org/viewtopic.php … 3#p1918843
You could try whether any of those parameters helps with the situation.

Offline

#20 2020-11-19 00:39:48

damige
Member
Registered: 2016-09-02
Posts: 40

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Just want to drop in to say i have had the same issue (slow uploads using AX200, in my case with SSHFS ~8mbit) on kernel 5.9.8, and when switching to LTS 5.4.77 transfers are back to expected at ~450mbit.
This was at least the case for a week when i first noticed it. So i am assuming it appeared in kernel 5.9.
Is anyone working on this upstream, or are they unaware of this problem?


XPS 15 7590 | i9-9980HK | 32GB RAM | 512G NVME | Intel AX200 | NVIDIA 1650 | OLED 4K

Offline

#21 2020-11-20 19:45:28

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Hey, quick request for help here for someone who's only done the easy lts downgrade

sudo pacman -S linux-lts

.
I have looked at the link Seth posted above to downgrade the kernel to a specific one (5.8.14).

in /var/cache/pacman/pkg ls | grep linux shows the kernel I'm looking for but not the linux-headers nor the virtualbox-host-modules-arch archives. Are these necessary? I'm hesitant to just try it run only to bork the machine and have to do a bunch of chrooting when I'm busy with other things at the moment.

Last edited by buzuddha (2020-11-20 19:46:13)

Offline

#22 2020-11-20 20:29:12

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

You need the vbox modules if you want to run vbox - otherwise not.
The headers are required to compile stuff against the kernel (doen't have to be out of tree modules, but they are the most common cause - see the depending packages here: https://www.archlinux.org/packages/core … x-headers/ )

If you have neithre package installed atm, you don't need to intall them for the downgrade either.

You can still get all of this out of https://wiki.archlinux.org/index.php/Do … ux_Archive anytime you need them.

Offline

#23 2020-11-22 17:59:04

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Aight!

1. Run

sudo pacman -U linux-5.8.14.arch1-1-x86_64.pkg.tar.zst

2. Reboot, Advanced options, choose non-lts kernel
3. Check

uname -r
5.8.14-arch1-1

4. Mount NFS drives
5. Copy a 1.9gb file to server1 and time -> 1900Mb/~35s ~ 54MB/s!

Wonderful detective work Seth! I am in your debt! What is the best way to report this, proceed, etc? Presumably the next time I run pacman -Syu, it's going to update the kernel back to 5.9.x and update the wireless card driver?

Offline

#24 2020-11-22 21:26:43

seth
Member
Registered: 2012-09-03
Posts: 50,012

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

You can ignore the "linux" packe in your pacman.conf but this is probably https://bbs.archlinux.org/viewtopic.php?pid=1939135

For the recent kernels (and likely matching upstream bug report) see https://bugzilla.kernel.org/show_bug.cgi?id=209913#c4

Offline

#25 2020-11-24 21:24:00

buzuddha
Member
Registered: 2020-10-02
Posts: 72

Re: [SOLVED] Recent abrupt decline in NFS file transfer speeds

Ok fixed! Thank you Seth! Big ups!

I've let the bugzilla people know I've experienced this. Seems like I'm not the only one.

1. Ran

 pacman -Syu

2. Restart

 uname -r
5.9.10-arch1-1

3. mount NFS drive
4. Copy a 1.2gb file to server; confirm low upload speeds ~5 MB/s
5. Ran

 ethtool -K wlp115s0 tso off

6. Copy 1.2gb file to server; time 1200MB/21s ~ 57MB/s

Offline

Board footer

Powered by FluxBB