Force writing to USB drives to wait (synchronous vs asynchronous)

drcouzelis · 2021-01-29 14:22:29

Sorry if this has been answered, I'm not quite sure how to search for it...

From what I've gleaned over the years, Windows and Linux handle writing to USB drives differently. On Windows, the default behavior is to show a progress bar for the time needed to write the files to the USB drive. On Linux, apparently the default behavior is to show a progress bar for the kernel to tell the files to be written to the USB drive? Which is MUCH faster, but is also the reason why things like the "sync" command are needed before actually ejecting the USB drive. I hate this behavior in Linux.

Is there a way to make the behavior of Linux behave like the default behavior in Windows when writing files to a USB drive?

...in other words, I want the "cp -r /big/files /my/usb/drive" to return when the files are actually written to the USB drive.

Last edited by drcouzelis (2021-01-29 15:29:24)

ugjka · 2021-01-29 14:29:56

Probably some udev rule needed, but i'm interested too

seth · 2021-01-29 15:01:38

https://lonesysadmin.net/2013/12/22/bet … rty_ratio/
https://www.kernel.org/doc/Documentation/sysctl/vm.txt

nb. that lowering those values too much™ (forcing to sync more often) will and can shred your performance and drive (well, it's gonna age faster - in particular NAND)

You could also shadow cp w/ a function that checks the destination mountpoint, then casewise runs "sudo sync" after the actual cp command.

drcouzelis · 2021-01-29 15:28:14

synchronous / asynchronous writes

Those are the search terms! Thanks for the tip in the links.

seth wrote:

https://lonesysadmin.net/2013/12/22/bet … rty_ratio/
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
nb. that lowering those values too much™ (forcing to sync more often) will and can shred your performance and drive (well, it's gonna age faster - in particular NAND)

I didn't think about that... It will affect all drive access. I'll probably leave it at whatever Linus thinks is best. But it's nice to know how to change it if needed.

Also, I just learned the meaning of nb / nota bene. Haha

Thank you!

Last edited by drcouzelis (2021-01-29 15:28:40)

seth · 2021-01-29 16:06:31

Since it's by default a ratio of your RAM, it's perfectly ok to lower the values if you've ALOTOFRAM™ - vbut yes, the behavior isn't FS/device dependent.
The writing client could simply inspect the mountpoint and sync more often if it's a removable drive, but the kernel has -afaik- just a global setting to impact this.

Trilby · 2021-01-29 16:06:44

Also note (nb!) that linux doesn't just totally procrastinate. You ask for files to be copied, and the kernel makes the generally-sound assumption that the actual device will be there for a while, so it says "Sure, we'll handle that, go about your day now". And when there is appropriate IO downtime, the physical writes can and do occur. So no sync would be needed if the device actually stayed mounted long enough for the writes to complete.

The end result of how long it takes before the device is safe to be removed is similar - I suspect it might be slightly longer with the linux approach just because the IO writes are low priority where as a forced sync prioritizes them - but the same amount of data still needs to be written either way.

I'm curious what the use case is where this would be needed. How about rather than trying to make every operation syncronous, just have a tool that displays a remaining progress bar: if you give a cp command then immediately call the remaining-progress indicator, it'd be pretty much like windows; but if you give the cp command, then do a bunch of other stuff, then decide to run the remainin-progresss indicator it might go very quickly or even report that there's nothing left to do.

The remaining-progress indicator would not be for a specific command or operation, but a general metric of queued but not completed IO. Essentially this is just a sync command with a progress indicator.

seth · 2021-01-29 16:16:35

I'm curious what the use case is where this would be needed.

This is actually asked a lot and it's usually about the "accuracy" of GUI progress indicators or why they don't act linear.
Ie. just like with kettles, people love to watch progress indicators ;-)

ugjka · 2021-01-29 16:35:07

Quick things I found from goofing around

man ext4 wrote:

FILE ATTRIBUTES
The ext2, ext3, and ext4 filesystems support setting the following file attributes on Linux systems using the chattr(1) utility:
[...]
D - synchronous directory updates
[...]
S - synchronous updates

man mount wrote:

FILESYSTEM-INDEPENDENT MOUNT OPTIONS
[...]
dirsync
All directory updates within the filesystem should be done synchronously. This affects the following system calls: creat, link, unlink, symlink, mkdir, rmdir, mknod and
rename.
[...]
sync All I/O to the filesystem should be done synchronously. In the case of media with a limited number of write cycles (e.g. some flash drives), sync may cause life-cycle
shortening.

Trilby · 2021-01-29 16:49:23

seth wrote:

people love to watch progress indicators ;-)

I'd prefer paint drying or grass growing. But none-of-the-above really sounds ideal.

ugjka · 2021-01-29 18:01:09

sync option absolutely kills performance, i get about 500-600 kB/s throughput, ha

Edit: on vfat, ext4 performs many times better

Edit2: exfat seems to be on par with ext4 only loosing half the throughput

Edit2: As an experiment this will be my udisks2 conf for a while (I use KDE's removable storage thingy)

[ugjka@archee ~]$ grep -v "#" /etc/udisks2/mount_options.conf

[defaults]
defaults=sync

Just need to stop using vfat and stick to exfat for Windows stuff. And don't use the fuse driver for exfat, that one doesn't support sync

Last edited by ugjka (2021-01-29 20:27:31)

Arch Linux

#1 2021-01-29 14:22:29

Force writing to USB drives to wait (synchronous vs asynchronous)

#2 2021-01-29 14:29:56

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#3 2021-01-29 15:01:38

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#4 2021-01-29 15:28:14

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#5 2021-01-29 16:06:31

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#6 2021-01-29 16:06:44

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#7 2021-01-29 16:16:35

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#8 2021-01-29 16:35:07

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#9 2021-01-29 16:49:23

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

#10 2021-01-29 18:01:09

Re: Force writing to USB drives to wait (synchronous vs asynchronous)

Board footer