Robust file copying (synchronous mount)

M4he · 2013-07-02 07:31:04

When copying files (especially external USB drives) I'm always issueing a:

sync

because the file transfer pretends to have finished but still didn't write everything to the disk. It happened to me more than once, that files got corrupted on the connected disk because I seemingly forgot to issue a sync command and disconnected the device too early.
It happens both with the cp command as well as with any graphical file manager. They pretend to write at insane speeds, finish very quickly, but only when I issue a sync the drive LED flashes like hell for at least 10 seconds while writing the remaining blocks to disk.

Doing some research, I've found the asynchronous mounting to be the culprit, which seems to be the default setting for mounting external drives. However I have not found any clue on how to change that default setting. It happened to me on all distros so far.

What I need is a robust method for copying files, thus synchronously mounted USB devices. I'm still backing up my data partition with robocopy on Windows (shame on me) because I cannot allow any files getting corrupted.

brebs · 2013-07-02 07:55:42

Use the flush mount option. This is a useful udev rule:

ACTION=="add", ENV{ID_FS_TYPE}=="vfat|ntfs", ENV{mount_options}="$env{mount_options},flush,utf8,gid=100,umask=002"

fukawi2 · 2013-07-02 08:44:22

Just a point of clarification... It is the kernel doing this, not the copy tools (cp, rsync, dolphin, thunar etc).

And yes, you're right; async mounting (default) is the culprit and will almost certainly cause you corruption issues if the device isn't sync'ed first.

Which brings me to the fact that a sync is done when the device is unmounted; so if you're not unmounting your devices properly, what did you expect to happen? Even Windows does this with the "Safely Remove Device" dialog.

If you're mounting using fstab, you can use the "sync" flag in the option column, otherwise the udev above is a good choice.

But please... A unicorn cries everytime a device isn't cleanly unmounted

M4he · 2013-07-02 14:49:00

Thanks both of you for your replies!

fukawi2 wrote:

Which brings me to the fact that a sync is done when the device is unmounted; so if you're not unmounting your devices properly, what did you expect to happen? Even Windows does this with the "Safely Remove Device" dialog.

The last time I regularly needed to "safely remove" my devices on Windows was a decade ago on Windows 2000.
I do unmount my devices properly on Linux and Mac though, but I tend to forget to at times due to this.

What I want to achieve is that the file copy progress shown corresponds to the actual copy process. I don't care about the need to unmount that much but I do care about the incorrect progress display and the huge time amount the sync operation takes.

Can someone shed some light on the difference of flush and sync? Is their functionality filesystem dependent? (I've got devices with FAT, NTFS and exFAT)

Lone_Wolf · 2013-07-03 18:31:25

according to man mount, the flush option is only supported on FAT filesystems.

the sync option is more versatile :

The  following  options apply to any filesystem that is being mounted (but not every filesystem actually honors them - e.g., the sync option today has effect only for ext2, ext3, fat,  vfat and ufs):partitions

cfr · 2013-07-14 15:34:18

I've never had problems but I come from OS X so I'm paranoid about unmounting as OS X throws out alarming error messages if you forget to do it and so you learn pretty fast. I pretty much always use safely unmount in Windows although I don't remember needing to do this when I used Windows properly but that was Windows 95...

Arch Linux

#1 2013-07-02 07:31:04

Robust file copying (synchronous mount)

#2 2013-07-02 07:55:42

Re: Robust file copying (synchronous mount)

#3 2013-07-02 08:44:22

Re: Robust file copying (synchronous mount)

#4 2013-07-02 14:49:00

Re: Robust file copying (synchronous mount)

#5 2013-07-03 18:31:25

Re: Robust file copying (synchronous mount)

#6 2013-07-14 15:34:18

Re: Robust file copying (synchronous mount)

Board footer