Arch Linux

bepaald

Member

Registered: 2016-10-08

Posts: 28

Slow scp file transfer over local network

Hi! I'm not exactly a Linux noob, but definitely a networking noob, so feel free to speak to me like I'm an idiot.

I used to have a Raspberry Pi 4b on LAN which I could transfer files to from my desktop PC, both wired connections to my router. Transferring files would go at about 80MB/s (though it varied). A few days ago I switched out the RPi4b with a new RPi5, with an nvme drive instead of an SD card. The RPi5 simply replaced the RPi4, using the same ethernet cable to connect to the router. However, now whenever I transfer anything to the RPi5, the transfer speed tops out at around 20MB/s.

Just for testing, I got my old RPi4 out again, and connected it as well (using a new ethernet cable, the RPi5 is still connected with the one that has always worked before), so I could test both at the same time. Running `scp` in verbose mode, shows pretty much the same output, until the end:

On the RPi4:

[...]
Authenticated to rpi4b.local ([192.168.0.200]:666) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /home/bepaald/.ssh/known_hosts for [rpi4b.local]:666 / (none)
debug1: client_input_hostkeys: searching /home/bepaald/.ssh/known_hosts2 for [rpi4b.local]:666 / (none)
debug1: client_input_hostkeys: hostkeys file /home/bepaald/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug1: Remote: /storage/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /storage/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending subsystem: sftp
debug1: pledge: fork
largefile                                                                                                                                                                                                       100%  781MB  85.7MB/s   00:09    
scp: debug1: truncating at 819200000
[...]

On the RPi5:

[...]
Authenticated to libreelec5.local ([192.168.0.167]:666) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /storage/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /storage/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending subsystem: sftp
debug1: pledge: fork
largefile                                                                                                                                                                                                       100%  781MB  20.2MB/s   00:38    
scp: debug1: truncating at 819200000
[...]

I notice, apart from the speed difference, the fast one says debug1: pledge: filesystem, while the slower one says debug1: pledge: network. I couldn't find any info on what these pledges mean, but could that be related to the problem? Does anyone have any other ideas what is going on? Please let me know if there is any more information I should supply.

I also tested the network speed, and the filesystem speed on the RPi5 locally by logging in through ssh, and both are perfectly fine so it's not that the network connector/chip is bad on the new RPi or the storage is slow. Both are in the hundreds of MB/s when testing locally on the device itself.

I also logged into my router (TP-Link Archer C6) to see if there is any setting that would be active for the RPi4 specifically and not the RPi5, but I can't find anything (but I also don't know what I would be looking for).

Thanks!

Last edited by bepaald (Today 06:47:57)

Beemo

Member

Registered: 2024-12-20

Posts: 162

Re: Slow scp file transfer over local network

Although scp now uses sftp in the backend, perhaps try sftp directly?

> cd Downloads/
> sftp user@serv.tld:/home/user/Downloads/test.mkv

If that's still slow, try a speed test between the 2 devices (iperf). E.g.
RPi

$ iperf3 -s

PC

> iperf3 -c iperf3.example.com

Last edited by Beemo (Today 07:33:36)

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 77,944

Re: Slow scp file transfer over local network

I also tested the network speed, and the filesystem speed on the RPi5 locally by logging in through ssh, and both are perfectly fine

Raise the verbosity, compare the KEX results.
Might be costly/un-accelerated encryption?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

bepaald

Member

Registered: 2016-10-08

Posts: 28

Re: Slow scp file transfer over local network

@Beemo:

Thanks for the suggestions. If I'm reading the output correctly, iperf is fast, sftp is (as you suspected) the same as scp:

[~] $ iperf3 -c libreelec5.local
Connecting to host libreelec5.local, port 5201
[  5] local 192.168.0.169 port 59746 connected to 192.168.0.167 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   115 MBytes   962 Mbits/sec    0    987 KBytes       
[  5]   1.00-2.00   sec   110 MBytes   923 Mbits/sec  117    264 KBytes       
[  5]   2.00-3.00   sec   108 MBytes   906 Mbits/sec   92    252 KBytes       
[  5]   3.00-4.00   sec   109 MBytes   912 Mbits/sec   36    318 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   929 Mbits/sec    6    325 KBytes       
[  5]   5.00-6.00   sec   110 MBytes   920 Mbits/sec    9    331 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   931 Mbits/sec    0    472 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   937 Mbits/sec    0    598 KBytes       
[  5]   8.00-9.00   sec   109 MBytes   918 Mbits/sec   42    428 KBytes       
[  5]   9.00-10.00  sec   109 MBytes   916 Mbits/sec    0    537 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec  302            sender
[  5]   0.00-10.00  sec  1.08 GBytes   926 Mbits/sec                  receiver

iperf Done.

[~] $ sftp -P 666 root@libreelec5.local
Connected to libreelec5.local.
sftp> put largefile
Uploading largefile to /storage/largefile
largefile                                                                                                     100%  781MB  24.9MB/s   00:31    
sftp> exit

@seth:

I forgot to mention, I also checked top on the RPi's during the file transfer. On the old RPi4, the CPU was maxed out, on the new one it was at around 10%. I was just not sure what I could conclude from that, the RPi5 obviously has a faster CPU, but there could also indeed be changes in hardware encryption support. I would generally assume the RPi5 to be faster though, not slower.

Not sure what KEX results are, or how to compare them... I've ran the copy with -vvv to both devices, there was pretty much too much output for me to read (~20000 lines), so I then looked for differences in lines containing "kex". Grepping for "kex" returned about 50 lines, but no (meaningful) difference between the two devices:

[~] $ scp -vvv -r -P 666 /home/bepaald/largefile root@RPI4b.local:"/storage/" > SCP_VVV_PI4 2>&1
[~] $ scp -vvv -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/" > SCP_VVV_PI5 2>&1
[~] $ diff <(grep -C1 -i kex SCP_VVV_PI4) <(grep -C1 -i kex SCP_VVV_PI5)
28c28
< debug1: Server host key: ssh-ed25519 SHA256:TxupATv+SHdh2FPtr[...]
---
> debug1: Server host key: ssh-ed25519 SHA256:jQfq25WIcKjBLgAu[...]

I'm thinking this one change is expected (EDIT just noticed that one line doesn't even pertain to KEX, it's there because of the one line context I used for grep). Please let me know if there is anything specific I should look for, or if I need to post the full verbose outputs.

Thanks!

Last edited by bepaald (Today 09:27:10)

Beemo

Member

Registered: 2024-12-20

Posts: 162

Re: Slow scp file transfer over local network

To check what algorithm is being used:

> sftp -vv ...
debug1: kex: algorithm: mlkem768x25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

RPi5 has AES acceleration. If the cipher is "aes..." in the output, then maybe the acceleration is broken...

> cat /proc/cpuinfo | grep aes

Last edited by Beemo (Today 09:55:38)

bepaald

Member

Registered: 2016-10-08

Posts: 28

Re: Slow scp file transfer over local network

Thanks again. Seems the algorithm used is exaclty the same as what you posted:

debug1: kex: algorithm: mlkem768x25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

(this is identical on both the RPi4 and the RPi5) So no AES I think? As for the CPU features, on the RPi5:

LibreELEC5:~ # cat /proc/cpuinfo | grep aes
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

On the RPi4 this command produces no output.

bepaald

Member

Registered: 2016-10-08

Posts: 28

Re: Slow scp file transfer over local network

Well thanks for your help. I have made some progress. It turns out, if I limit the bandwidth of the scp command, the speed actually increases (up to a point).

[~] $ # default:
[~] $ scp -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/"
largefile                                                         100%  781MB  24.2MB/s   00:32    
[~] $ # limit to 409600 mbps (50MB/s)
[~] $ scp -l 409600 -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/"
largefile                                                         100%  781MB  49.8MB/s   00:15    
[~] $ # limit to 737280 mbps (90MB/s)
[~] $ scp -l 737280 -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/"
largefile                                                         100%  781MB  87.9MB/s   00:08    
[~] $ # limit to 819200 mbps (100MB/s)
[~] $ scp -l 819200 -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/"
largefile                                                         100%  781MB  97.3MB/s   00:08    
[~] $ # limit to 901120 mbps (110MB/s), that is pushing my luck
[~] $ scp -l 901120 -r -P 666 /home/bepaald/largefile root@libreelec5.local:"/storage/"
largefile                                                         100%  781MB  23.0MB/s   00:34    

So, the sweet spot seems to be at 100MB/s. As soon as I go over that, the speed completely falls to the floor. This happens in the last line, but apparently also in the first (the default, without limiting). Which is fine, I didn't expect the transfer to ever go faster than that. But, I would like to know why this is happening. And is there something on the RPi5 I can change to cause it to just default to this speed somehow, instead of me having to specify this (arbitrary?) limit every time I transfer something.

Thanks!

EDIT Interestingly, if I limit the bandwidth on the RPi4, the speed drops compared to the default:

[~] $ scp -r -P 666 /home/bepaald/largefile root@rpi4b.local:"/storage/"
largefile                                                         100%  781MB  91.2MB/s   00:08    
[~] $ scp -l 819200 -r -P 666 /home/bepaald/largefile root@rpi4b.local:"/storage/"
largefile                                                         100%  781MB  43.5MB/s   00:17    

Last edited by bepaald (Today 13:20:49)

Beemo

Member

Registered: 2024-12-20

Posts: 162

Re: Slow scp file transfer over local network

Not sure if relevant, but I remember that RPi's default qdisc is pfifo_fast for some reason.

> sysctl net.core.default_qdisc
net.core.default_qdisc = fq_codel

I'm no expert in this, maybe some buffer got overrun.

Also the 100MB/s limit is a bit sus. By any chance your devices think they can go over 1Gbps but either the NIC or cable is causing loss?

> ethtool enp195s0f3u1u4
Speed: 1000Mb/s

EDIT: nope iperf test disproved this

p.s. Maybe something specific to the RPi (having downstream kernel and packages and all)? Consider asking on RPi forum in case the devs know something about it.

Last edited by Beemo (Today 14:47:57)

bepaald

Member

Registered: 2016-10-08

Posts: 28

Re: Slow scp file transfer over local network

Thank you again, I appreciate you trying to help.

The qdisc on the rpi5 is fq_codel, just as what you posted.

I should probably try a different OS on the rpi5, see if that makes a difference. It's running LibreELEC now, surely that does not have the latest kernel (though I would also think that more people would actually encounter this problem on LibreELEC specifically). I'll try a RPI specific forum as well. Thanks again!

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 77,944

Re: Slow scp file transfer over local network

lsmod | grep -i aes

What if you unload and blacklist the aes modules?
Resp.

export OPENSSL_ia32cap="~0x200000200000000"

should™ disable AES (and PCLMULQDQ, not sure why but it frequently gets suggested - maybe just copypasta)

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Beemo

Member

Registered: 2024-12-20

Posts: 162

Re: Slow scp file transfer over local network

Oh I didn't notice you're running LibreELEC, thought it was Raspberry Pi OS.
And yeah, I'd recommend trying the official OS first (e.g. on a USB stick / SD card), for the custom modifications. If that works, Fedora, for ruling out LibreELEC.
Another thing I thought of is the SSD, in case of an unstable connection.