[solved] Looking for smart backup software

eleius · 2017-07-05 15:18:38

Does anyone know any software that can perform synchronization (backup of a folder and all of its subfolders) between 2 hard drives, and that also recognizes moved and/or renamed files and folders? Since I'm trying to keep a daily backup of my works (mostly videos in the gigabyte range), I'd like to keep file copy operations to a minimum.

I've tried dirsyncpro, freefilesync, rsync with the detect-renamed-lax patch, hrsync, and a couple of windows programs, but all of them happily delete my moved folders and copy gigabytes of files instead of simply renaming them.

If you know of such a program please let me know - I don't care if it's slow, as long as it works. Thanks in advance

Last edited by eleius (2018-06-15 17:35:44)

brebs · 2017-07-05 15:36:43

ZFS as the filesystem?

eleius · 2017-07-05 17:17:22

brebs wrote:

ZFS as the filesystem?

Both drives are formatted as NTFS (for Linux/Windows interoperability)

Last edited by eleius (2017-07-05 17:17:37)

WorMzy · 2017-07-05 17:50:12

Mod note: Normally product recommendation request threads are either binned or moved to TGN (depending on the type of product and the amount of effort the OP has made), but I feel this topic is more about overcoming the technical limitations of backup software, and there is some potential here for a proper discussion.

I may be proven wrong, but for now I'm opting to leave it in Apps/DE discussion.

woodape · 2017-07-05 20:48:47

You could run an apache server locally on your computer and run an instance of Nextcloud with the data stored on the one harddrive and the sync client pointing to some local folder(s) on the other. To help save on RAM overhead you could write a script that starts the httpd and mariadb services (to launch apache and the database software), runs the commandline sync client to sync the data, and then stops the httpd service. Its a very heavy handed solution, but Nextcloud does a good job of handling file renames and files moves.

JohnBobSmith · 2017-07-05 21:48:53

I've never used backup software. Keeping this in mind, why not have a simple shell script for this? You could store dates last modified in a text file using the output from the stat command. Then, we have this: If the current iteration of stat is newer than the one stored in our previously mentioned file, then do cp newFiles /dev/sdb1/backup_20170705/. I've never used cron, but you might be able to to automate the process with a cron job once you're confident the script works.

seth · 2017-07-06 12:23:49

Came up recently, promotes deduplication. No idea whether it can. => https://github.com/bup/bup

Annoyingduck · 2017-07-06 12:33:21

I use Lucky Backup (available in the AUR) which uses Rsync as it's foundation tool. It's a very simple, powerful gui application. To set it up for full content backup, just copy the folder/directories to the target backup drive first. Then set up Luckybackup using the default option "backup source inside destination" and check "do not create extra directory" then save the profile. Name the profile and each individual action per drive you want to backup. Then whenever there is a change to any file, just run Lucky Backup and your backup drive will be fully synced with the source drive. I use it for everything on my main cpu drive to my backup drive (files, music, videos, etc). If any minor detail is changed on my primary drive (ex. I updated embedded tag info in a music file), running Luckybackup will sync that change to the target backup drive. Super useful tool. There are endless uses for it and the encyclopedia on the Lucky Backup website goes over all of the options that are available. I just listed the basic clone folder to another drive option.

Last edited by Annoyingduck (2017-07-06 12:43:56)

eleius · 2017-07-06 13:29:33

Thanks everyone for your suggestions.

@ JohnBobSmith: I was thinking about coding it myself, but keeping track of every moved/renamed/deleted item doesn't seem an easy task for a shell script (unless I computed hashes for every single file, but as I said they're in the GB range...) Not enough spare time ATM, but I'll try as a last resort.

@ Seth: Bup doesn't seem what I had in mind, as IIUC it needs to make a full local backup of all files from my first hdd in order to be able to sync them.

@ Annoyingduck: LuckyBackup works just fine as a sync tool, but it doesn't do what I was looking for specifically: instead of just moving files/folders, it deletes them and then copies them again to their new locations.

Last edited by eleius (2017-07-06 16:12:42)

Annoyingduck · 2017-07-06 13:36:28

You set up lucky backup wrong then. Do exactly what I said, and it'll clone the folder/directory to your target drive. I use it daily, you changed one of the options that caused the target folder to be cut instead of being copied. It's a powerful tool with many options read up on it it will do exactly what you want and do it extremely fast and properly.

eleius · 2017-07-06 15:22:03

I didn't change any other option. Files and folders *are* correctly copied so that the 2 disks are identical, just not the way I wanted.

Let's say I have the following structure on my first and second HDDs:

root
+ folder_A
- file_1
- file_2
+ folder_B
+ folder_C
- file 3
- file_4
- file_5

Now, if I move "folder C" out of "folder B", to the root of the first HDD, sync programs (including LuckyBackup) delete it from the second HDD and copy it again to the new location. (The same goes for renamed files and directories.)

What I want is a simple "mv /folder_B/folder_C /" instead. If it worked that way, the move operation would take a very small time, instead of the many minutes required to copy the gigabytes of data I have in folder_C.

Last edited by eleius (2017-07-06 15:34:56)

Annoyingduck · 2017-07-06 15:37:36

Something is definitely wrong. When I get home later this evening I'll verify that I did not put a typo in my post. But what I just described will not delete the source destination, it will copy it to the Target destination. I assure you that program will do exactly what you want. I just need to verify if I made an error in my post or we need to figure out what setting you have enabled that's causing that action because that should not happen.

eleius · 2017-07-06 15:57:08

Just to be clear (I'm not a native english speaker) what I meant is, instead of this:

mv /disk2/folder_B/folder_C/ /disk2/folder_C/

it does something like this:

cp -a /disk1/folder_C/ /disk2/folder_C/
rm -rf /disk2/folder_B/folder_C/

the result is identical, but obviously the cp/rm operation takes a lot of time.

Last edited by eleius (2017-07-06 16:13:26)

seth · 2017-07-06 16:24:24

You need something that has access to the FS journal and utilizes that. And a sufficiently large journal.
Otherwise you'd have to look for each missing file for a file with an equal hash (what can be slow if you've many small files on a large disks ...)

HiImTye · 2017-07-06 20:15:05

the assumption is that you're not just moving files, you're backing them up, that's why backup software does that (it's in the name). a double pointer is not a backup, the data still exists in only one set. what you want to do is only accomplished, really, through links and at the filesystem level (see: zfs, as mentioned above)

Annoyingduck · 2017-07-07 13:29:46

My original directions are correct. That will not delete the source destination files. Copy the files first before setting up Lucky Backup. Be sure "backup source inside destination" is chosen NOT "synchronize source and destination". Be sure to check the box "do not create extra directory". Select the source, and then select the destination. There will be a warning when creating the profile about files being deleted on the destination. This is normal, as your backing up a source, so any changes to the source may be deleted from the destination (not the source) due to it being a backup or clone of the source destination as files in the source change over time. Be sure to save the profile. Then running Lucky Backup will clone any changes from the source to the destination. Only changes will be written, not the entire source. It's extremely fast. If that is not working for you, than you are doing something wrong or have changed some settings within Lucky Backup.

Trilby · 2017-07-07 13:39:41

Annoyingduck, you don't seem to be reading eleius's descriptions. Despite his apologies, his English is excellent and he's described quite clearly what is missing from programs like Lucky Backup for his use. I don't know the right tool - if there is a right tool - for what he wants, but trying to brow beat him into accepting one that doesn't match his criteria is silly. His issue with Lucky Backup is not that the source is deleted (that's never been said) it's the unnecessary disk IO between the two disks for what could be a local mv operation within the backup disk.

Perhaps the directory example was too complex - a single file example would do. Imagine disk1 with a many GB file named "abigfile". This gets backed up to disk2. That backup proceedure requires many GBs of data to move between the disks. This part is unavoidable. But then the next day the file on disk1 is *renamed* "a_big_file" with absolutely no change to it's content. At this point 2 things could happen:

1) The backup program creates a new file on disk2 called "a_big_file" and again transfers those many GB of data from disk1 to disk2. There are many options here on whether the existing "abigfile" would then be deleted from disk2 or not but those are irrelevant, the problem is that the GBs of data is being moved a second time. Whether or not anything is deleted from disk1 is also irrelevant. The problem is that data is needlessly moved between disks.

2) The backup program recognizes that "a_big_file" is really just a new name for "abigfile" which already exists. So no data is moved, and the file already on disk2 is simply renamed.

The problem is that for 2 to happen, the backup program would need some way of knowing that "abigfile" and "a_big_file" were actually identical. This could (maybe) come from hashing, it may come from a FS journal as Seth suggests, or it may come from some creative use of inode records rather than filenames (though I'm not sure if this could work).

Annoyingduck · 2017-07-07 14:02:15

Trilby wrote:

Annoyingduck, you don't seem to be reading eleius's descriptions. Despite his apologies, his English is excellent and he's described quite clearly what is missing from programs like Lucky Backup for his use. I don't know the right tool - if there is a right tool - for what he wants, but trying to brow beat him into accepting one that doesn't match his criteria is silly. His issue with Lucky Backup is not that the source is deleted (that's never been said) it's the unnecessary disk IO between the two disks for what could be a local mv operation within the backup disk.
Perhaps the directory example was too complex - a single file example would do. Imagine disk1 with a many GB file named "abigfile". This gets backed up to disk2. That backup proceedure requires many GBs of data to move between the disks. This part is unavoidable. But then the next day the file on disk1 is *renamed* "a_big_file" with absolutely no change to it's content. At this point 2 things could happen:
1) The backup program creates a new file on disk2 called "a_big_file" and again transfers those many GB of data from disk1 to disk2. There are many options here on whether the existing "abigfile" would then be deleted from disk2 or not but those are irrelevant, the problem is that the GBs of data is being moved a second time. Whether or not anything is deleted from disk1 is also irrelevant. The problem is that data is needlessly moved between disks.
2) The backup program recognizes that "a_big_file" is really just a new name for "abigfile" which already exists. So no data is moved, and the file already on disk2 is simply renamed.
The problem is that for 2 to happen, the backup program would need some way of knowing that "abigfile" and "a_big_file" were actually identical. This could (maybe) come from hashing, it may come from a FS journal as Seth suggests, or it may come from some creative use of inode records rather than filenames (though I'm not sure if this could work).

No I understand that, and I'm not trying to beat something in. All I am saying is that what I described will do what he wants to do. The source IS NOT deleted - this is what I'm trying to emphasize. Luckybackup uses changes in the source files checksum to write changes, so any file rename or addition to a folder/file will be written to the target destination. The source remains unaltered, and only the changes are then written to the target destination. If an 8gb file has the name changed, and say all of the text within the file has the first letter in each word changed to capital letters, only those changes will be written, not the entire 8gb file. And most importantly the source is unaltered, it's not deleted, moved, or touched in anyway. It's only read and then copied to the target. I'm simply letting the user know that if he is experiencing the source destination being deleted when running LuckyBackup with the settings I have described, that he is doing something wrong or has changed something. It will do exactly what he wants to do. The same way if someone asked "when I go outside and walk around my feet hurt"...and the reply is well, put shoes on. Luckybackup is the shoes in this scenario, I don't know if he is putting them on his hands though...

Trilby · 2017-07-07 14:26:51

Annoyingduck wrote:

The source IS NOT deleted - this is what I'm trying to emphasize... The source remains unaltered... the source is unaltered, it's not deleted, moved, or touched in anyway... if he is experiencing the source destination being deleted when running LuckyBackup with the settings I have described, that he is doing something wrong

You are not listening, at all. This is not the problem. I just tested lucky backup to be sure, and it most definitely doesn't do what the OP wants. I backed up a large file to a flash drive, then I renamed the source file and tried to backup again - lucky backup tried to copy the contents of the file again - in fact it failed because one copy of the file could fit on the flash drive, but two copies couldn't - it ran out of space. If Lucky Backup were actually just renaming the already existing file on the target it would not have ran out of space.

Annoyingduck · 2017-07-07 14:39:19

Trilby wrote:

Annoyingduck wrote:
The source IS NOT deleted - this is what I'm trying to emphasize... The source remains unaltered... the source is unaltered, it's not deleted, moved, or touched in anyway... if he is experiencing the source destination being deleted when running LuckyBackup with the settings I have described, that he is doing something wrong
You are not listening, at all. This is not the problem. I just tested lucky backup to be sure, and it most definitely doesn't do what the OP wants. I backed up a large file to a flash drive, then I renamed the source file and tried to backup again - lucky backup tried to copy the contents of the file again - in fact it failed because one copy of the file could fit on the flash drive, but two copies couldn't - it ran out of space. If Lucky Backup were actually just renaming the already existing file on the target it would not have ran out of space.

I have run into that before, but only with a flash drive regardless of it's format (fat, exfat, ntfs, ext4, etc). I haven't figured out why it does that on flash drives, but for regular hard drives in ntfs/ext4 format internal or external that does not occur. My main 1tb backup disk is nearly maxed out and I run Lucky all of the time with many changes and there is not a drive space error. I don't know, just trying to help.

Trilby · 2017-07-07 14:41:24

You're (still) missing the point. Even if it didn't run out of space, it still would have been moving data. Moving data is the problem, not deletion of anything.

seth · 2017-07-07 14:42:07

Since luckybackup uses rsync, I'd say it can perform in-data deduplication, but not inter-data deduplication.
Iow, it can (should) do delta updates, but rather not replay a log.

eleius · 2017-07-07 15:58:31

@ Trilby: ah, I was starting to think my english was a lot worse than I thought

@ Annoyingduck: my hdds are regular, mechanical hard drives in ntfs format. Anyway thanks for trying to help, at least I can use luckybackup for my backups if I cannot find a solution to my original issue.

@ Seth: there's an "official" patch for rsync at https://download.samba.org/pub/rsync/rs … 1.2.tar.gz that adds an option that should do what I want (in theory, but my tests say otherwise):

"This patch adds the options --detect-renamed-lax and --detect-moved.
These modify the --detect-renamed algorithm to adopt a matching file
without verifying that the content is as expected. The former blindly
accepts a file that matches in size and modified time. The latter
requires that the filename also match (ignoring any renamed files)."

Anyway as I said, even a few specific (commercial) Windows software I've tried didn't help at all, So maybe I'll try whipping up a custom backup program after all. Thanks everyone.

Last edited by eleius (2017-07-07 16:05:00)

x33a · 2017-07-07 17:35:33

My suggestion is to use a fast hashing algorithm like crc32 and use it to decide whether the files have been updated or not.

Edit: A quick search for fast hashing algorithms came up with: https://github.com/Cyan4973/xxHash

Last edited by x33a (2017-07-07 17:37:41)

kokoko3k · 2017-07-07 17:48:38

Try unison, some useful infos here: https://vxlabs.com/tag/unison/

unison is terribly efficient at transferring changes to and fro. It makes use of the rsync-algorithm, with added niceties like duplicate detection shortcuts.

Last edited by kokoko3k (2017-07-07 17:59:12)

Arch Linux

#1 2017-07-05 15:18:38

[solved] Looking for smart backup software

#2 2017-07-05 15:36:43

Re: [solved] Looking for smart backup software

#3 2017-07-05 17:17:22

Re: [solved] Looking for smart backup software

#4 2017-07-05 17:50:12

Re: [solved] Looking for smart backup software

#5 2017-07-05 20:48:47

Re: [solved] Looking for smart backup software

#6 2017-07-05 21:48:53

Re: [solved] Looking for smart backup software

#7 2017-07-06 12:23:49

Re: [solved] Looking for smart backup software

#8 2017-07-06 12:33:21

Re: [solved] Looking for smart backup software

#9 2017-07-06 13:29:33

Re: [solved] Looking for smart backup software

#10 2017-07-06 13:36:28

Re: [solved] Looking for smart backup software

#11 2017-07-06 15:22:03

Re: [solved] Looking for smart backup software

#12 2017-07-06 15:37:36

Re: [solved] Looking for smart backup software

#13 2017-07-06 15:57:08

Re: [solved] Looking for smart backup software

#14 2017-07-06 16:24:24

Re: [solved] Looking for smart backup software

#15 2017-07-06 20:15:05

Re: [solved] Looking for smart backup software

#16 2017-07-07 13:29:46

Re: [solved] Looking for smart backup software

#17 2017-07-07 13:39:41

Re: [solved] Looking for smart backup software

#18 2017-07-07 14:02:15

Re: [solved] Looking for smart backup software

#19 2017-07-07 14:26:51

Re: [solved] Looking for smart backup software

#20 2017-07-07 14:39:19

Re: [solved] Looking for smart backup software

#21 2017-07-07 14:41:24

Re: [solved] Looking for smart backup software

#22 2017-07-07 14:42:07

Re: [solved] Looking for smart backup software

#23 2017-07-07 15:58:31

Re: [solved] Looking for smart backup software

#24 2017-07-07 17:35:33

Re: [solved] Looking for smart backup software

#25 2017-07-07 17:48:38

Re: [solved] Looking for smart backup software

Board footer