You are not logged in.

#1 2013-07-02 07:31:04

M4he
Member
Registered: 2011-08-11
Posts: 75

Robust file copying (synchronous mount)

When copying files (especially external USB drives) I'm always issueing a:

sync

because the file transfer pretends to have finished but still didn't write everything to the disk. It happened to me more than once, that files got corrupted on the connected disk because I seemingly forgot to issue a sync command and disconnected the device too early.
It happens both with the cp command as well as with any graphical file manager. They pretend to write at insane speeds, finish very quickly, but only when I issue a sync the drive LED flashes like hell for at least 10 seconds while writing the remaining blocks to disk.

Doing some research, I've found the asynchronous mounting to be the culprit, which seems to be the default setting for mounting external drives. However I have not found any clue on how to change that default setting. It happened to me on all distros so far.

What I need is a robust method for copying files, thus synchronously mounted USB devices. I'm still backing up my data partition with robocopy on Windows (shame on me) because I cannot allow any files getting corrupted.


Arch - makes me feel right at /home

Offline

#2 2013-07-02 07:55:42

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: Robust file copying (synchronous mount)

Use the flush mount option. This is a useful udev rule:

ACTION=="add", ENV{ID_FS_TYPE}=="vfat|ntfs", ENV{mount_options}="$env{mount_options},flush,utf8,gid=100,umask=002"

Offline

#3 2013-07-02 08:44:22

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,216
Website

Re: Robust file copying (synchronous mount)

Just a point of clarification... It is the kernel doing this, not the copy tools (cp, rsync, dolphin, thunar etc).

And yes, you're right; async mounting (default) is the culprit and will almost certainly cause you corruption issues if the device isn't sync'ed first.

Which brings me to the fact that a sync is done when the device is unmounted; so if you're not unmounting your devices properly, what did you expect to happen? Even Windows does this with the "Safely Remove Device" dialog.

If you're mounting using fstab, you can use the "sync" flag in the option column, otherwise the udev above is a good choice.

But please... A unicorn cries everytime a device isn't cleanly unmounted sad

Offline

#4 2013-07-02 14:49:00

M4he
Member
Registered: 2011-08-11
Posts: 75

Re: Robust file copying (synchronous mount)

Thanks both of you for your replies!

fukawi2 wrote:

Which brings me to the fact that a sync is done when the device is unmounted; so if you're not unmounting your devices properly, what did you expect to happen? Even Windows does this with the "Safely Remove Device" dialog.

The last time I regularly needed to "safely remove" my devices on Windows was a decade ago on Windows 2000.
I do unmount my devices properly on Linux and Mac though, but I tend to forget to at times due to this.

What I want to achieve is that the file copy progress shown corresponds to the actual copy process. I don't care about the need to unmount that much but I do care about the incorrect progress display and the huge time amount the sync operation takes.

Can someone shed some light on the difference of flush and sync? Is their functionality filesystem dependent? (I've got devices with FAT, NTFS and exFAT)


Arch - makes me feel right at /home

Offline

#5 2013-07-03 18:31:25

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 10,636

Re: Robust file copying (synchronous mount)

according to man mount, the flush option is only supported on FAT filesystems.

the sync option is more versatile :

The  following  options apply to any filesystem that is being mounted (but not every filesystem actually honors them - e.g., the sync option today has effect only for ext2, ext3, fat,  vfat and ufs):partitions

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#6 2013-07-14 15:34:18

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,127

Re: Robust file copying (synchronous mount)

I've never had problems but I come from OS X so I'm paranoid about unmounting as OS X throws out alarming error messages if you forget to do it and so you learn pretty fast. I pretty much always use safely unmount in Windows although I don't remember needing to do this when I used Windows properly but that was Windows 95...


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

Board footer

Powered by FluxBB