Sync between Sweden and China

svanberg · 2011-11-16 21:50:19

Hi,

We have different data on two locations (Sweden and China) that we want to sync and today we using a software called SyncBack for Windows, but the whole synchronization job is taking more than 24 hours to complete and sometimes longer than that. Of the time it takes to complete a job, a couple of minutes is just used to actually copy the data, the big problem is the speed when comparing files on "the other side".

Can rsync handle this faster and more reliable? Do rsync use any local "database" that holds all the changes, on both sides for a faster sync? I have limited experience with rsync so, I would be glad if you guys could help me out.

Thanks in advance!

Ogion · 2011-11-16 22:04:17

Let's say it this way, i haven't yet seen rsync take any longer for syncing than the download…
(Sure, if you continue big files from an interrupted transmission, it'll take a couple minutes sometiems to find the place where it was interrupted and continue on from there, but this was only noticeable for files > 1 gig.)

How about you just try out rsync locally. Make two local folders, with the data, slightly different, and then just run:
rsync -avh --partial --progress folder_one folder_two

That is, this brings changes from folder_one to folder_two. If you have changes on both sides that need to be written to the respective other side, you probably have to use something else for that. I don't really have experience for that.

Ogion

Awebb · 2011-11-16 22:33:12

At work we have two servers holding some sort of a cache, that must be backed up every day. It's about 50GB data, split in hundreds of thousands of folders and subfolders on a few billion files. That takes a few minutes on a rather old dual core xeon and a P4. rsync makes a list on both servers, diffs that list and only syncs the changed files.

How many files are we talking about and what kind of hardware do you use?

fukawi2 · 2011-11-16 23:43:29

Is this a 2-way sync (ie, files on both ends are changed and needs to be sync'ed to the other location), or a 1-way sync (ie, Sweden to China, never China to Sweden)?

For a 1-way sync, rsync will be perfect for you.

You could do a 2-way sync using the --update flag, but this will fail if the same file gets updated at both locations between syncs -- someone's data will get overwritten. I'm not sure how your current solution handles this.

svanberg · 2011-11-17 07:26:13

Awebb wrote:

At work we have two servers holding some sort of a cache, that must be backed up every day. It's about 50GB data, split in hundreds of thousands of folders and subfolders on a few billion files. That takes a few minutes on a rather old dual core xeon and a P4. rsync makes a list on both servers, diffs that list and only syncs the changed files.
How many files are we talking about and what kind of hardware do you use?

The folder is containing 16 828 files and 24 590 folders and right now we using Windows Server 2008 x64 with 4 GB RAM (this virtual machine is hosted on VMware vSphere 4.1 with 2.4 GHz and 16 cores).

fukawi2 wrote:

Is this a 2-way sync (ie, files on both ends are changed and needs to be sync'ed to the other location), or a 1-way sync (ie, Sweden to China, never China to Sweden)?
For a 1-way sync, rsync will be perfect for you.
You could do a 2-way sync using the --update flag, but this will fail if the same file gets updated at both locations between syncs -- someone's data will get overwritten. I'm not sure how your current solution handles this.

2-way sync sure can be a problem, but I try anyway.

jon · 2011-11-17 20:31:29

For a two-way sync I think Unison sounds like it would do the job. I believe it uses rsync on the back-end, and also comes with a GUI.

fukawi2 · 2011-11-17 21:59:05

jon wrote:

For a two-way sync I think Unison sounds like it would do the job. I believe it uses rsync on the back-end, and also comes with a GUI.

That would be my best thought for a 2-way sync. I'm not sure how well it supports scripting/automation since a) it's a GUI and b) it requires interaction (AFAIK) to resolve conflicts.

It would be worth trying it though.

Arch Linux

#1 2011-11-16 21:50:19

Sync between Sweden and China

#2 2011-11-16 22:04:17

Re: Sync between Sweden and China

#3 2011-11-16 22:33:12

Re: Sync between Sweden and China

#4 2011-11-16 23:43:29

Re: Sync between Sweden and China

#5 2011-11-17 07:26:13

Re: Sync between Sweden and China

#6 2011-11-17 20:31:29

Re: Sync between Sweden and China

#7 2011-11-17 21:59:05

Re: Sync between Sweden and China

Board footer