Synchronize multiple computers ?

doupod · 2013-08-24 14:39:05

Hi,

I've some computers that I'd like to sync against my own server, just as what Dropbox or Ubuntu One propose. My idea is to have my desktop and my laptop computer syncing against my server to always have the same structure on both computers (and a backup accessible wherever I am).

I know about rsync (and duplicity) that I use for my server backups. But these don't seem to be able to do a "two-way syncs". What I want may be more like a Git repository : you add files in your home folder, you delete them and it's automatically replicated on the other computer, via the server. But the problem is that I'm not sure a Git repo could handle hundreds of GB (and video files > 10GB)...

Do you have any solution / script to help me deal with it ?

Thanks

progandy · 2013-08-24 14:41:39

I guess unison is a good option if you are ok with scheduled/manual syncs and don't need realtime updates.

doupod · 2013-08-24 14:53:28

Indeed, I saw Unison while searching infos about this on the web. Do you have any feedback about it ?

Some more infos I forgot in my previous post :
- I don't really need realtime updates. Scheduled updates made by Cron are ok also. But, I really need the system to be ok with connection loss (as I'll use it on my laptop, I won't always have an internet access, and I can lost my internet access in the middle of a sync).
- I really need also some bandwith saving system (ie syncing only necessary files and if possible compressing during sync).
- If possible, it would be great to have an option to crypt the backup on my server (just as what you can do with duplicity).

And some ideas I got (if by chance, anyone had an advice about it / a feedback) :
- Git repo ? => Not sure it's a good idea because of the size of the potential repo. Maybe something like Git-Annex (http://git-annex.branchable.com/) ?
- Mounting a remote location on my computer (Webdav for example) and syncing against it with a script.

Last edited by doupod (2013-08-24 15:10:18)

broken pipe · 2013-08-24 15:45:33

i use bittorrent sync (http://labs.bittorrent.com/experiments/sync.html) to keep my home dirs in sync.
transfers are encrypted and split into 4 MB parts. Changes are controlled, monitored and indexed by the bittorrent sync daemon (https://aur.archlinux.org/packages/bittorrent-sync/).
There's a .SyncIgnore file in the parent direcory to control which files are ignored to sync (eg. i sync nearly everything in my home folders except several dotfiles which are listed in the file).
Syncs are distributed via a secret key (whether read only or rw).
Both computers must be online to sync (you can sync over LAN, fullspeed or over the internet, depending on your upload rate).

doupod · 2013-08-24 16:44:12

Bittorrent sync seems to be interesting but it seems that I can't encrypt files on the server side (just as what duplicity does). This is important for me.

For unison, it seems very interesting and powerful (especially the ability to handle connection loss, partial transfers etc.), but I didn't find any way to encrypt the files on the server side. Do you have any idea of how I could do it ?

The best solution I found for now seems to be to have a webdav mount (OwnCloud, maybe better because uses the port 80 which is always opened) or a remote SSH mount somewhere and sync against it (maybe with unison ?), crypting it with encfs.

Last edited by doupod (2013-08-24 16:45:28)

cfr · 2013-08-25 03:01:05

If you search for encryption and unison, there are a bunch of home-grown solutions there - maybe one of them is useful to you? I'd link but I don't see a way to do that with startpage.com and I only use google now for search in case of dire necessity. (They've still got my documents and email but at least they don't have everything!)

EDIT: Can you not set up and encryption container on the server, open it over ssh and run unison over ssh, for example? I'm assuming you want the container closed when not in use - obviously it is easy otherwise as you can just set up the system on luks or whatever.

Last edited by cfr (2013-08-25 03:03:34)

doupod · 2013-08-25 12:11:12

I'm interesting in your home-grown solutions. If you can post any links to some working setups you know, it would be really great !

By now, I think the best solution would be to use a remote mount on my system and syncing against it (with unison for example) with a transparent encryption.

I saw encfs that is often used with webdav. I know about LUKS (actually, my laptop is already a LVM on LUKS). But I'd like the container to be always closed on the server (ie I don't want anybody taking control of my server to have access to my files). So, I'd like an encryption with a passphrase or keyfile that will only be stored on my laptop and my desktop (and never on the server side) and I'd like to mount it transparently on my computer.

cfr · 2013-08-25 13:21:10

Just search on unison and encryption. That's all I did.

progandy · 2013-08-25 17:41:05

cfr wrote:

Just search on unison and encryption. That's all I did.

Those homegrown solutions are not perfect, if my understanding of unison is correct.
- If you synchronise encrypted files you'll get merge problems if conflicts occur (How are you comparing and merging something encrypted?)
- If you have a remote mount, you'll end up with lots of data to transfer since unison has to create all checksums locally and cannot rely on the server to do that. You end up transferring whole files instead of filenames with checksums.

Ideally, unison should sync encrypted data. When the need to merge arises the files will be decrypted, merged and reencrypted.
Can you create a script that uses encfsctl to encode/decode filenames, operate on the decoded name and let unison work with the encrypted result?

The best solution would be this I guess (use encfs --reverse if you want to permanently store unencrypted data on your laptop):

laptop unencrypted <--encfs (--reverse) --> laptop encrypted <--unison--> server encrypted
         \----------------------|-----------------|--------------/
                > merge script using encfsctl to merge in decrypted state (AFAIK does not exist)

Last edited by progandy (2013-08-25 17:43:04)

doupod · 2013-08-25 19:31:22

@propagandy: This scheme might be the easier one but (if I correctly understand what you propose), ~~I'll have to store the data twice on my laptop : one encrypted version and one unencrypted version.~~. I just have to set a script to handle conflicts.

This might not be a great deal for some documents files but I'm looking for a way to sync my whole disks, which means about 80GB of music and hundreds of GB of videos...

EDIT : I thought about something : I'll only run unison from the laptop and the desktop, never from the server. So the cache will always be on my laptop and my desktop. So maybe remote mount point won't be a great deal, no ?

EDIT : A solution could be to sync against a webdav mount. What about unison + any encryption scheme ? Else, Owncloud has an encryption app which could provide the same as Dropbox, no ?

A last solution I found would be to use a Git repository with transparent encryption with something like git-crypt or git-annex (http://git-annex.branchable.com/encryption/).

Finally, if I can't find any better solution, I could made my own script based on duplicity frontend which uses rsync for optimization and supports encryption via GPG. But, I'll lose the benefits from Unison... (always in a correct state no matter connection losses, etc.)

Last edited by doupod (2013-08-25 20:14:15)

cfr · 2013-08-25 20:33:53

If you have an encryption container on the server, you can sync with the server and have the container open just for the time you need to sync, right? You can sync over ssh so the sync will be secure. At least, that's the sort of set up I imagined you wanted.

The recommended way to use unison is to only sync from one side i.e. always sync from laptop, say. If you have other machines to sync, they recommend you use one as the "hub" and sync from everything else to that single hub. When I run unison, it is fast. I am syncing much less data but even so. It is, however, much slower the first time because it has to check everything. But it doesn't do that each time later.

If you want encryption done locally then unison wouldn't be a good solution. (I guess you'd want then to do the equivalent of whatever things like SpiderOak do but I don't know how you would mimic that.)

progandy · 2013-08-25 20:53:48

So maybe remote mount point won't be a great deal, no ?

You need stable inode numbers, or try pretendwin=true in order to ignore them. Then you can use the fastcheck without checksums after the initial sync. Make sure to do your first sync in a gigabit lan.

Last edited by progandy (2013-08-25 20:54:45)

cfr · 2013-08-25 21:26:26

I'm not really seeing the issue here. When I use unison it is incredibly fast. My last sync took under 1 second. This is remote over ssh. It involved transfer of one new directory containing one pdf file from the remote machine to the laptop from which I run unison. All I am doing is running

unison -auto <local-path> ssh://<ssh-server>/<remote-path>

Although the directories I'm syncing are not anywhere near the sizes the OP needs to sync, they are not small either. Basically, I'm syncing all of the articles, books etc. which I have electronic copies of. The first sync took quite a while. Subsequent syncs are incredibly fast.

What am I not understanding here?

progandy · 2013-08-25 21:50:32

What am I not understanding here?

He wants an encrypted backup.

There are two major routes with unison without any decryption on server side
1)
* sshfs,nfs to /mnt/encrypted_server
* encfs /mnt/encrypted_server to /mnt/unencrypted_server
--> here you have to make sure you rarely use checksums since the remote fs transfers the data before it can be calculated
... ... you have to use fastcheck (needs stable inode numbers or pretendwin)
* unison between /home and /mnt/unencrypted_server
2)
* ENCFS7_CONFIG=/some/.encfs7.xml encfs --reverse /home /mnt/local_crypted
* unison /mnt/local_encrypted ssh://SERVER/remote_encrypted
--> here you'll have to make a custom merge script if you expect conflicts (easy: replace one copy)

If you accept temporary decryption on server side there is no real issue:
* log in to server (ssh)
* mount encfs (supply password via encrypted ssh connection)
* unison
* unmount encfs
* disconnect
maybe the mounting can be done with pam_encfs during ssh login?

Xyne · 2013-08-25 22:25:43

Summarizing to check that I have understood your requirements:

a central server will host all files and all other systems (the clients) will sync with it (i.e. star network)
the files must be encrypted on the server
the files must never be unencrypted on the server

If that is correct then here is how you do it:

set up ssh on the server if you haven't done so already
create a local encrypted directory on one of the clients with EncFS or eCryptfs (you may find ecryptfs-simple useful)
mount the local encrypted directory and add your files to it
set up unison to synchronize the underlying encrypted directory (not the unencrypted mountpoint) via ssh on the server
synchronize other clients with the server

With filesystem encryption (EncFS, eCryptfs), the underlying encrypted files are regular files, so you can leverage unison's full potential for keeping these files in sync without having to transfer unencrypted data. This is better than using sshfs as unison will be able to checksum files directly on the server instead of transferring all of the data via ssh(fs) to do it locally.

If you use EncFS, there is a file in the encrypted directory that contains encryption metadata. I would configure unison to ignore it and transfer it manually and directly to each client. Not only will this prevent it from appearing on the server, it will also ensure that an error arising in one copy does not propagate to the others. Each client will have its own independent copy.

For eCryptfs, the metadata is stored in the file itself.

cfr · 2013-08-25 22:36:56

Xyne wrote:

the files must never be unencrypted on the server

This was the bit I wasn't clear on. Now the complications make sense.

doupod · 2013-08-26 07:03:16

Actually, if I could find a solution involving a complete encryption (and files never in an unencrypted state), it would be really great

The fact is that I also host other services on my server. So, there's always a small risk to see someone taking control of my server. In this case, I'd like to prevent this person to access my files.

If I use the technique based on an encrypted container, opened during the sync, and close the rest of the time, when the container is opened for my sync, it is accessible from the server. This means that it can be opened for some minutes every hours, or something like this (I don't have an accurracy estimation of the transfer time for each sync, if run hourly but should be some minutes max with Unison I think). A potential attacker could access my files during this time.

On the contrary, if the files are always stored (as Xyne propose), there aren't any risks to see an attacker get access to my files.

Moreover, I need a hub sync. Each computer must sync against the same server. I think this is the easier solution. Another solution would be to sync my laptop against my desktop and my desktop against my server if this is easier (and if my transfer rates are too slow, because in this case desktop / laptop is gigabit LAN).

cfr · 2013-08-26 13:39:15

In that case I just misunderstood what you wanted.

diepes · 2014-02-27 22:32:55

I am also struggeling with the problem of syncing my backup, photos and scanned documents to multiple location to prevent loss.

One of the things i am worried about is corruption of a file, and then syncing it over the other copies, it seems some kind of saved checksum will be needed to detect good or bad files.

As to the encryption to remote server this i have a idea for, just use the linux encrypted home drive, and then sync the underlying encrypted file system ($HOME/.Private) to the server, can even use rsync.
As each file is also a encrypted file, only new files need to sync, and you can always sync it to a new location and mount with the encryption key.

Arch Linux

#1 2013-08-24 14:39:05

Synchronize multiple computers ?

#2 2013-08-24 14:41:39

Re: Synchronize multiple computers ?

#3 2013-08-24 14:53:28

Re: Synchronize multiple computers ?

#4 2013-08-24 15:45:33

Re: Synchronize multiple computers ?

#5 2013-08-24 16:44:12

Re: Synchronize multiple computers ?

#6 2013-08-25 03:01:05

Re: Synchronize multiple computers ?

#7 2013-08-25 12:11:12

Re: Synchronize multiple computers ?

#8 2013-08-25 13:21:10

Re: Synchronize multiple computers ?

#9 2013-08-25 17:41:05

Re: Synchronize multiple computers ?

#10 2013-08-25 19:31:22

Re: Synchronize multiple computers ?

#11 2013-08-25 20:33:53

Re: Synchronize multiple computers ?

#12 2013-08-25 20:53:48

Re: Synchronize multiple computers ?

#13 2013-08-25 21:26:26

Re: Synchronize multiple computers ?

#14 2013-08-25 21:50:32

Re: Synchronize multiple computers ?

#15 2013-08-25 22:25:43

Re: Synchronize multiple computers ?

#16 2013-08-25 22:36:56

Re: Synchronize multiple computers ?

#17 2013-08-26 07:03:16

Re: Synchronize multiple computers ?

#18 2013-08-26 13:39:15

Re: Synchronize multiple computers ?

#19 2014-02-27 22:32:55

Re: Synchronize multiple computers ?

Board footer