Problem when copying files to USB device

lugge · 2017-04-01 16:12:59

So there is an old issue which I encountered on multiple Linux distributions and cannot handle.

Sometimes when I copy large files (say, video files/mkv) from HDD to an USB device my Standalone-Multimedia-Player (from Western Digital) will stop playing the file somewhere in the middle.
This happened some years ago and I then learned I have to use "sync" in order to clear and write the filesystem buffer.
Since then, I always use "sync" when my filebrowser has finished copying, sometimes even multiple times.
Problem solved (or so I thought).

Now I'm on my arch system and the same issue happens again and again.
Yesterday I copied 5 files from HDD to USB, synced, synced again and shut down my PC before unplugging the USB.
In the middle of file 4/5 the WD-player stopped playing the file.

I now did some resarch.
On my PC using VLC player, the file can be played completely. So, It seems there is an issue within the copied file which my standalone player cannot handle but VLC can.
Did md5sum check: checksum is wrong!

So, I don't know how this can happen.
I synced after copying, and the file was completely copied but now has some errors in it. That will lead to a md5sum mismatch.
I only know for sure that this can happen when copying from HDD to USB.
But I also have multiple HDDs in my PC.
I never thought there can be problems when copying photos or important documents.

TL;DR:
I sometimes have md5sum checksum issues when copying files between disks/USB.
How can this happen?
How can I solve this?
Am I the only one encountering this?
Do I have to run md5sum whenever I copy files because I cannot trust the kernel?!

Regards, lugge

Last edited by lugge (2017-04-01 16:16:32)

WorMzy · 2017-04-01 16:27:56

How big is the file you mentioned, and what filesystem does the USB device use?

seth · 2017-04-01 18:32:12

1st off:
it's "normal" that software players can deal with (partially) broken stuff while HW players can't (ie. handle bit errors gracefully where the DSP isn't very flexible), so no surprise here.

2nd:

I then learned I have to use "sync" in order to clear and write the filesystem buffer

No, you have to unmount the device before unplugging it and notably the unmount needs to finish before unplugging it.
(Though sync, iff finished, should do the really relevant stuff and a reboot should unmount things unless you unplug power early)
This is kinda important as unmounting may cover more things than just syncing the filesystem. It also prevents further access (like when some daemon thinks it's terrible important to write a search index to teh device etc.)

3rd:
You should ensure the HW is good. There's been a time where it was quite popular to "fake" 2GB drives to have 4GB or more memory, but writing actually more data to them would override random other bits; also the device could simply approach EOL. You can do this by trying the device on OtherOS™ and ensure write is good there. You can also try using trustworthy™ copy programs on linux, because

4th:
eg. kio has a record to hikkup on copying large data amounts, so what exactly do you use to copy data? cp, rsync, dd, some fancy GUI file browser ...

5th:
What WorMzy said ;-)
Notably if the answer is "FAT32" and even more notably if that FAT table was written by OtherOtherOS™

lugge · 2017-04-03 18:43:36

WorMzy wrote:

How big is the file you mentioned, and what filesystem does the USB device use?

The files are about 400MB and the Stick is formatted with FAT32 (maybe was formatted on Win machine).

seth wrote:

1st off:
it's "normal" that software players can deal with (partially) broken stuff while HW players can't (ie. handle bit errors gracefully where the DSP isn't very flexible), so no surprise here.

Yes, that was my guess to.

seth wrote:

No, you have to unmount the device before unplugging it and notably the unmount needs to finish before unplugging it.
(Though sync, iff finished, should do the really relevant stuff and a reboot should unmount things unless you unplug power early)
This is kinda important as unmounting may cover more things than just syncing the filesystem. It also prevents further access (like when some daemon thinks it's terrible important to write a search index to teh device etc.)

I somehow remember unmounting directly after copying files took about 1s.
Which means, unmount did not sync.
However, this was years ago on an Ubuntu machine and maybe I don't remember correct. I just sync manually to be sure.

seth wrote:

kio has a record to hikkup on copying large data amounts, so what exactly do you use to copy data? cp, rsync, dd, some fancy GUI file browser ...

I used "fany GUI file browser", Nemo. But internally I guess Nemo will use the same syscalls.
Your point is interesting.
Does this mean kio (part of the kernel) is known for having trouble copying files?! Can't believe it...

seth wrote:

Notably if the answer is "FAT32" and even more notably if that FAT table was written by OtherOtherOS™

Answer is "yes" and "yes" again.
Whats the point in that?

seth · 2017-04-03 21:29:06

this was years ago on an Ubuntu machine

*cough* - so what? ;-P
Seriously, a problem you might have had with "unmounting" (probably using a GUI tool as well?) on another OS is no usable measure.

kio is not part of the kernel but KDEs IO abstraction layer and this is not a matter of "syscalls" but either in the filesystem or bad arrangement of read & write calls to libc (iirc kio suffered from a race condition when movin many files at once)
Please try to reproduce the problem using

cp -v <foo> /run/media/<usb stick label>/

and afterwards also inspect

dmesg | tail -100

for possible IO errors.

If you can reproduce it this way: FAT is not a filesystem but a file allocation table; DOS age technology - it's as robust and reliable as a raw egg.
You can try to repartition the stick and re-create the FAT index (mkfs.vfat, yes i just said it's no FS ;-)

Another thing I just noticed is "the stick" - if you're trying one stick with you operate for years, a broken (degrading) hardware device is more likely than a software problem in this case.

kepler186 · 2017-04-05 14:23:36

Hello I always had some sort of a similar problem on an older PC. Sometimes the copy process would take longer, visually it would appear finished but if i unplugged the USB stick, the full file was not copied.

So i stumbled across some settings which fixed this somehow. Open the /etc/sysctl.conf and add:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

Save and write "sudo sysctl -p" in terminal.

Reboot PC, enjoy

More info at: https://lonesysadmin.net/2013/12/22/bet … rty_ratio/

Last edited by kepler186 (2017-04-05 14:24:10)

severach · 2017-04-12 01:47:02

RMPrepUSB is the only tool that formats FAT32 on flash drives reliably. Old or not, FAT32 is so difficult to do right that even Microsoft utilities sometimes get it wrong.

R00KIE · 2017-04-12 14:54:55

@severach
Why would you recommend what seems to be a windows only tool in a forum dedicated to linux?

@lugge
Are you using any other usb devices at the same time you are copying the files? Specifically usb webcams or any other non mass storage devices (mice do not count here).

WorMzy · 2017-04-12 14:56:23

Severach, that appears to be a Windows-only tool. Please refrain from linking to tools that cannot be used on Linux. You also haven't provided any evidence to support your claims, I have personally never had a problem with mkfs.fat.

Arch Linux

#1 2017-04-01 16:12:59

Problem when copying files to USB device

#2 2017-04-01 16:27:56

Re: Problem when copying files to USB device

#3 2017-04-01 18:32:12

Re: Problem when copying files to USB device

#4 2017-04-03 18:43:36

Re: Problem when copying files to USB device

#5 2017-04-03 21:29:06

Re: Problem when copying files to USB device

#6 2017-04-05 14:23:36

Re: Problem when copying files to USB device

#7 2017-04-12 01:47:02

Re: Problem when copying files to USB device

#8 2017-04-12 14:54:55

Re: Problem when copying files to USB device

#9 2017-04-12 14:56:23

Re: Problem when copying files to USB device

Board footer