You are not logged in.

#1 2010-01-15 18:51:08

SternGerlachExperiment
Member
Registered: 2009-10-11
Posts: 51

Mirroring a drive and verifying written data

I have an internal drive that holds some important personal files. I bought an external USB hard disk for the purpose of backing up the files. What would be the best method to mirror the contents of the internal drive to the external drive and then verify that the data is really there.

The best situation would be that the backup software detects what files have been added / changed (but not deleted), copies them on to the external file and then verifies that the data on the external drive matches the data on the original drive (verifies all the files, not just the recently copied files).

I'm a bit worried if the data will surely be written on the external drive. I've previously had the problem that I've inserted a memory card in a card reader, then copied some files and "safely removed" the card, but then found out that only the names of the files were written, but the files were all empty. So I want the program to make sure the data actually is on the external drive.

I do not want any snapshot file or tar file, but just the original drive mirrored as it is.

Offline

#2 2010-01-15 19:19:55

perbh
Member
From: Republic of Texas
Registered: 2005-03-04
Posts: 765

Re: Mirroring a drive and verifying written data

man rsync

Offline

#3 2010-01-15 19:30:58

perbh
Member
From: Republic of Texas
Registered: 2005-03-04
Posts: 765

Re: Mirroring a drive and verifying written data

ok - here's _one_ way of doing it ...

#!/bin/sh
# for the following to work properly, do not use automount!
#
# assume the first external disk is /dev/sdb1 and is mounted on /usb
# assume your backup disk will be /dev/sdc1
#
# connect the backup-disk before going to bed at night, start up
# this script and then remove the backup-disk in the morning

die() { echo -e "$*"; exit 1; }
df | grep -q '/usb$' || die "-- /usb is _not_ mounted!"
test -b /dev/sdc1 || die "-- no backup disk (/dev/sdc1)"
test -d /backup || mkdir -p /backup
mount /dev/sdc1 /backup || die "-- could not mount backup disk on /backup!"
rsync -av --delete /usb/ /backup || echo "-- backup failed!"
umount /backup
exit 0

Obviously - there are dozens of mutations on this one ...
1) Put the script in crontab and let it run each night
2) Use it as in 1), but only connect the backup disk on the nights you want it - always make sure to remove it the next morning
3) Keep the backup disk connected at all times

Personally I used to use 3) - but I had lightning strike which blew out not only the computer, but both usb-disks as well :-(
These days I go for 2) because of the lesson learned - if its not connected, nothing (apart from a catastrophy) will touch it in any way.
YMMV

Last edited by perbh (2010-01-15 19:39:57)

Offline

#4 2010-01-15 19:35:12

SternGerlachExperiment
Member
Registered: 2009-10-11
Posts: 51

Re: Mirroring a drive and verifying written data

Ok, thanks for the help!

I did have a look at rsync before I posted, but I failed to see that it checks the files were properly copied, and I figured it wouldn't satisfy my need for data verification. But it seems it does.

Thanks again.

Offline

#5 2010-01-15 19:44:03

perbh
Member
From: Republic of Texas
Registered: 2005-03-04
Posts: 765

Re: Mirroring a drive and verifying written data

If the copying goes OK - that (in my simple mind) is sufficient verification.
As to your specific problem in your original post - I bet you used automount  and pulled out the SD too early!
If you manually mount and umount - you _know_ the data is there (and not floating around in cache somewhere ...)

Offline

#6 2010-01-15 20:04:56

SternGerlachExperiment
Member
Registered: 2009-10-11
Posts: 51

Re: Mirroring a drive and verifying written data

Well that's the thing. I made various test, and even with manually mounting and unmounting and copying the files using command line, the files still didn't end up on the memory card. I also used Konqueror and Dolphin, and none of the methods worked. I typed sync on the command line and the light on the card reader blinked rapidly for a while, so I thought that would've flushed the data on to the card, but no. It was empty.

The same hardware (same card, same reader) worked in Windows.

However it seems that rsync actually does some verifying. From the man page:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred

Offline

Board footer

Powered by FluxBB