Copying files to SMB mount is slow (not efficient)

gohu · 2020-11-08 04:59:40

I have a NAS (Truenas), from which I mounted a SMB share on my Arch box.

The fstab entry looks like this:

//truenas/media	/mnt/data/media	cifs	_netdev,x-systemd.automount,username=gohu,password=xxxx,uid=1000,gid=1000	0	0

and the mount command returns this:

//truenas/media on /mnt/data/media type cifs (rw,relatime,vers=3.1.1,cache=strict,username=gohu,uid=1000,forceuid,gid=1000,forcegid,addr=192.168.1.25,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,_netdev,x-systemd.automount)

It works OK, but when I want to copy files from a local drive to the share, it's not as efficient as it may be.

Specifically, when I use an rsync command to copy over a directory containing a bunch of big files (multiple GBs), I monitor both the output of the rsync (with --progress) and the network usage (to see how fast the data is sent to the NAS), and I see this little dance:

rsync reads a file from the drive as fast as it can go (>100 MB/s); but nothing is sent over the network to the NAS
when the file is finished reading, rsync continues to the next file but it "blocks" (it doesn't read and shows 0% progress); while the previously read data is finally sent to the NAS (network usage saturates gigabit ethernet, ~120 MB/s) which takes a few tens of seconds for a multi-GB file
when the network transfer is over, rsync is unblocked and we start again from 1.

Consequently, the file copy is half as fast as it could be because it's either reading files, or sending data to the NAS, but never both at the same time. I get about 50-60 MB/s when averaged.

I noticed that if I run the "sync" command when rsync is in the middle of reading a file (and thus when nothing is sent over the network), then data starts being sent over the network, and yet rsync doesn't stop reading the file either. This gave me an idea, running this during the file copy:

while true; do sync; sleep 0.5; done

And I can confirm that when this runs, the copy is much faster, I see both ~100 MB/s reads and ~100 MB/s network usage, sustained all the time, giving a ~100 MB/s average.

But, yeah, the sync in a loop is not a good solution.

I tried a lot of googling, and though complaints about SMB performance are a dime a dozen, I didn't really find a description of this problem and much less a solution. Also, some suggestions I found involve modifying options on the SMB server side (NAS side) and some look "dangerous", data-safety wise. My "sync loop" hack proves that sacrificing data safety is not necessary (since I'm pretty sure "sync" is not dangerous).

So, do you have an idea as to what a (safe) solution may be?

Awebb · 2020-11-08 06:05:09

Try man mount.cifs and search for cache.

gohu · 2020-11-08 17:10:02

Awebb wrote:

Try man mount.cifs and search for cache.

OK, so as confirmed from "mount", the default value is "cache=strict". I tried the two other possible values:

cache=none This is slightly better as both disk read and network send are steady at around ~75 MB/s all the time, so the result is a net copy speed of ~75 MB/s; but this is still far from reaching the >100 MB/s that the hardware is capable of
cache=loose This is exactly the same as my first post. And it seems like it is unsafe.

All in all, I don't think "cache" is the right parameter to change...

seth · 2020-11-08 22:26:08

Try to lower rsize/wsize a bit - rsync is single threaded, so it will either read or write stuff, not both at the same time .
Lowering the read/write chunks (to eg. 2MB) should have a similar effect to syncing faster.

Awebb · 2020-11-09 04:30:36

Yes, the actual problem is rsync. However, I wanted to see if (not) caching could do the trick alone.

Now OP should google "rsync parallel" and "rsync parallel xargs". There are two interesting methods to parallelize rsync floating around the web.

TheSgtBilko · 2020-11-09 19:41:23

I always thought that the rsync "wait/block"-time was when the checksum was calculated over the wire. As in:
1. Copy file over to remote side.
2. Calculate target and (over the wire) destination checksums to confirm successful copy.
3. Back to 1 for next file.

edit: spelling!

Last edited by TheSgtBilko (2020-11-09 19:50:47)

gohu · 2020-11-09 23:17:36

Awebb wrote:

Yes, the actual problem is rsync. However, I wanted to see if (not) caching could do the trick alone.
Now OP should google "rsync parallel" and "rsync parallel xargs". There are two interesting methods to parallelize rsync floating around the web.

seth wrote:

Try to lower rsize/wsize a bit - rsync is single threaded, so it will either read or write stuff, not both at the same time .
Lowering the read/write chunks (to eg. 2MB) should have a similar effect to syncing faster.

No, I'm sorry if my first post was misleading, but this has nothing to do with rsync. I only use it as a verbose cp, there are no special rsync features involved. The exact same issue happens with cp or with thunar. I can use iostat to confirm that the source disk is being read, then stays idle, at times opposite those during which the network is used, just the same as with rsync.

We're not talking about whatever size the rsync buffer is, or whatever size the chunk size of SMB is (I mean the wsize/rsize, I'm not sure of exactly what they govern and of the terminology here). Those are a few MB at most. What we're talking about are files of 1, 2 or 3 GB being fully cached when they are read from the source disk, then being flushed to the NAS when the file is finished reading.

From what I understand, this caching is done in the Linux's page cache. So I think there are two general ideas that could help:

make the page cache smaller (but only for the CIFS mount?) / make it flush often (basically what my sync loop hack does) / simply bypass the page cache (looks like this is exactly what cache=none means, but the performance is obviously not great, some cache is helpful, just not so much that it delays flushes a whole lot) / ?
probably better: make the "flush" non-blocking; I think this is what happens during a copy between two "standard" filesystems: the page cache quickly fills up, then it starts flushing when it becomes too big, but those filling and flushing processes are not aligned with individual files being finished copying or whatever, so they don't block anything

I guess running multiple copy processes in parallel could also help indeed, because by chance some processes could be "reading" when others would be blocked on "flushing"... But it's not a very good solution as I would need to always do the copies with special commands, and also it doesn't help if I only have one (or a few files) to copy.

seth · 2020-11-10 07:01:18

Does lowering vm.dirty_expire_centisecs improve your situation?

gohu · 2020-11-10 22:48:54

seth wrote:

Does lowering vm.dirty_expire_centisecs improve your situation?

It does!

I tried lowering vm.dirty_expire_centisecs to 100 (1 second) and indeed I can see that the file is getting transfered to the network very soon after it starts copying.

I also tried setting vm.dirty_background_bytes to 10 MB, and it works too.

With either of these settings, transfering a single 1 GB file goes from ~17s to ~9s.

That's great... Except that these settings are global and affect all filesystems and I'm not sure if it's not going to hurt somewhere else

seth · 2020-11-11 11:28:38

vm.dirty_writeback_centisecs defaults to 500, setting vm.dirty_expire_centisecs below that is probably moot (ie. you're actually running 5s atm)

Syncing files more often means more actual writes on your drives, wearing them down faster.
However, you only write stuff that actually need to be written, ie. you'll not hammer the SDD if there's no actual data in the cache.

vm.dirty_expire_centisecs defaults to 30 seconds (so everything older than 30s is written by default, mostly for data protection purposes) - the question is whether you're updating some file on disk faster than this (and if: whether you wanted to use a tmpfs here anyway ;-)

Arch Linux

#1 2020-11-08 04:59:40

Copying files to SMB mount is slow (not efficient)

#2 2020-11-08 06:05:09

Re: Copying files to SMB mount is slow (not efficient)

#3 2020-11-08 17:10:02

Re: Copying files to SMB mount is slow (not efficient)

#4 2020-11-08 22:26:08

Re: Copying files to SMB mount is slow (not efficient)

#5 2020-11-09 04:30:36

Re: Copying files to SMB mount is slow (not efficient)

#6 2020-11-09 19:41:23

Re: Copying files to SMB mount is slow (not efficient)

#7 2020-11-09 23:17:36

Re: Copying files to SMB mount is slow (not efficient)

#8 2020-11-10 07:01:18

Re: Copying files to SMB mount is slow (not efficient)

#9 2020-11-10 22:48:54

Re: Copying files to SMB mount is slow (not efficient)

#10 2020-11-11 11:28:38

Re: Copying files to SMB mount is slow (not efficient)

Board footer