Ejecting an SD card after copying is done takes long time (due cache?)

szszoke · 2023-02-25 17:06:55

Hello,

Time to time I copy games that are installed on my computer to an SD card. These games usually take several tens of gigabytes of space. I noticed that whenever I try to eject my SD card after my file explorer (Nautilus) says that copying is done, it just takes forever. Initially the transfer speeds reported by the file explorer are unbelievably high, like 2GB/s and it then slowly decreases down to some tens of MB/s.

My system has 128GB of memory.

$ uname -r
6.1.12-arch1-1

$ cat /proc/sys/vm/dirty_background_ratio 
10

$ cat /proc/sys/vm/dirty_ratio              
20

$ cat /proc/vmstat | grep -E "dirty|writeback"
nr_dirty 5215018
nr_writeback 2048
nr_writeback_temp 0
nr_dirty_threshold 5960634
nr_dirty_background_threshold 2976678

Normally nr_dirty is around 150 but when I am copying to my SD card it goes up really high and keeps increasing. Eventually the copying is "done" and nr_dirty starts to decrease.

I don't quite know if I am looking at the right place and I never had to deal with something like this but ideally there would be a way to configure my system to always write directly to any USB storage medium.

Last edited by szszoke (2023-02-25 17:11:17)

Trilby · 2023-02-25 17:13:26

szszoke wrote:

ideally there would be a way to configure my system to always write directly to any USB storage medium.

There are mount options that can acheive this. I don't know them off the top of my head, but checking `man mount` should give the answer. However I'd discourage using such options. Flash device writes are cached for a reason and making all writes immediately sync will accelerate the wear of the device. Instead, you can periodically run sync (e.g., after you've copied moved content); one the sync is complete, unmounting/removal will be essentially instantaneous.

Note that neither option can change the actual amount of time required to move data to a device, it just changes when the data actually gets sync'ed to the device and starts that process sooner rather than waiting for a umount command.

Last edited by Trilby (2023-02-25 17:15:36)

szszoke · 2023-02-25 17:20:18

I understand that copying takes as long as it takes but in the latest example, I copied about 70GB of data which took maybe 10-20 minutes according to the file explorer and probably another 10-20 minutes of "limbo" until the data was actually written to the disk. It would be much better with more realistic reporting. If it is not good to directly write to the disk, is it possible to make the system write data to the USB storage more often?

Trilby · 2023-02-25 17:25:05

szszoke wrote:

It would be much better with more realistic reporting.

That's an issue with the file manager in use.

szszoke wrote:

... is it possible to make the system write data to the USB storage more often?

Yes, that's what sync does, which is why I recommended it.

seth · 2023-02-25 21:06:42

https://wiki.archlinux.org/title/sysctl#Virtual_memory
Don't mount "sync", it's bad for performance and NAND survival rates.

That's an issue with the file manager in use.

tbf, pretty much every FM exposes this behavior because one would have to actively poll/stat the destination to get realistic results - what's bad for the throughput.
A reasonable approach could be to occasionally poll the destination, estimate the effective speed, interpolate the indicator and base the polling frequency on the accuracy of the prediction.
One would also have to take the vm.dirty_*ratio into account against the read data, because the zero-writing speed before the cache gets flushed would heavily skew the estimation.

Since it's only relvant when writing from fast sources to slow destinations, I guess nobody has ever cared enough. At least I'm not aware of a FM that does this.

Maniaxx · 2023-02-26 03:03:56

Instead of *ratio you could use *bytes so "expectations" might be met more gracefully.
https://unix.stackexchange.com/a/640826

Last edited by Maniaxx (2023-02-26 03:04:34)

szszoke · 2023-02-26 10:41:45

Maniaxx wrote:

Instead of *ratio you could use *bytes so "expectations" might be met more gracefully.
https://unix.stackexchange.com/a/640826

Thanks for the suggestion. I will play around with the numbers.

Trilby · 2023-02-26 13:17:50

Why? Are you going to submit patch the source code of your file manager?

EDIT: oops, yes, for the config - I missed that.

Last edited by Trilby (2023-02-26 18:41:44)

szszoke · 2023-02-26 17:24:10

Trilby wrote:

Why? Are you going to submit patch the source code of your file manager?

No.. I meant that I would play around with the numbers in the config file that is mentioned on the forum post that Maniaxx linked.

Arch Linux

#1 2023-02-25 17:06:55

Ejecting an SD card after copying is done takes long time (due cache?)

#2 2023-02-25 17:13:26

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#3 2023-02-25 17:20:18

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#4 2023-02-25 17:25:05

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#5 2023-02-25 21:06:42

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#6 2023-02-26 03:03:56

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#7 2023-02-26 10:41:45

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#8 2023-02-26 13:17:50

Re: Ejecting an SD card after copying is done takes long time (due cache?)

#9 2023-02-26 17:24:10

Re: Ejecting an SD card after copying is done takes long time (due cache?)

Board footer