dropbox like functionality

toad · 2009-10-06 10:35:16

I'd love to have an open source programme/script with the following characteristics:

Aim:

- put files on comp #1 in a special folder
- folder detects changes automagically and syncs folder with online storage
- comp #2 has same folder, automagically realizes that changes took place on comp #1 and syncs immediately

Constraints:
- minimal cpu usage
- no bandwidth hogger

I know of a number of closed source applications such as dropbox or spideroak (both in AUR) which do this to a greater/lesser extent. Thing is, they are closed source and I'd prefer to have something doing the above to my server.

Unfortunately my programming skills are negligible and I require large amounts of help (if I can manage this at all).

So, if anybody out there is interested in developing an open source programme/script like that or has some interesting ideas then I'm looking forward to hearing from you.

Vintendo · 2009-10-06 11:46:39

I have been thinking about this as well, you can use curlftpfs but I don't know how many recources it uses, and I don't think it's downloaded to your computer...

scio · 2009-10-06 13:16:51

I had thoughts about this as well, but then I couldn't decide if a special directory was the best option, or a way to register files/directories to sync would be better. Essentially its cron (or fam/gamin) triggering a version control commit/pull. The commit should be easy, the pull would require an open connection that would push updates or polling an address for when changes occur.

toad · 2009-10-06 13:32:48

scio wrote:

I couldn't decide if a special directory was the best option, or a way to register files/directories to sync would be better.

Whatever is easier. I am not really worried about ease of setting the thing up as long as it, once it is running, is a no brainer you can just forget and yet rely on.

scio wrote:

Essentially its cron (or fam/gamin) triggering a version control commit/pull. The commit should be easy, the pull would require an open connection that would push updates or polling an address for when changes occur.

Correct me if I'm wrong, but a cron does things periodically, i.e. is forced into operation "externally" so to speak. I'm after an automatic thingamajig that does nothing if nothing changes but as soon as it does, it kicks into life. The ArchWiki entries on fam and/or gamin were not exactly enlightening but I take your word for it An open connection for this functionality goes without saying, too.

Anyway, is this dead easy then? You make it sound like it is

rson451 · 2009-10-06 13:43:01

incron fires on file specific events, so that isn't the tough part. As scio said, the tough part is telling the second client machine that changes are available on the web and that it needs to sync.

toad · 2009-10-06 13:54:31

rson451 wrote:

the tough part is telling the second client machine that changes are available on the web and that it needs to sync.

What if this icron (whatever that is) existed on the server as well, triggering a chain of syncs with comp #2,3,4, etc.? The only problem I see would be for the server to know where comp #2, etc are located... Could that be overcome with an initial registration to the server on coming online?

scio · 2009-10-06 16:47:05

toad wrote:

What if this icron (whatever that is) existed on the server as well, triggering a chain of syncs with comp #2,3,4, etc.? The only problem I see would be for the server to know where comp #2, etc are located... Could that be overcome with an initial registration to the server on coming online?

Ideally, the server would have no control over the clients. It would just say "hey, stuff changed!", then the client would update as needed. However, this would require a process running on the client side that constantly asks the server "did anything change?" or sits with an open connection to the server waiting for the magic message "stuff changed!" so it can go and sync up. This is my biggest gripe with dropbox for my laptop, it is the worst offender of cpu wakes, sometimes I just run it manually when I know I changed things, but that's a bad habit.

If you controlled the server and the clients, you could have the server remotely connect to your clients and push updates to a folder it had shared rights to.

Now you have me thinking about a whole distributed system where machines register to each other and there is a group or user on all the machines that can log in and change a specific folder. Each machine would just have a list of MAC address and IPs (like an arp table) of machines that wanted to be part of the network.

When a machine registers, the distributed user logs into another machine (could be priority based so you can have a home server be where everyone pulls from / pushes to), and copies all the files down. When a change happens, user logs into machine again, and pushed updates.

Now on that other machine, when a change gets pushed in, it pushes it out to all the other machines.

This is obviously a security risk with the external user and machine registration, but it would end the need for a server holding your data and a constant poll for updates. There are also probably better ways to do this, but that was just my first few thoughts on a distributed version.

dalingrin · 2009-10-06 17:33:10

I don't think I completely understand the objective. Do you want the files to be copied from the special folder of a client to a server and then propagate the changes to other clients?
Or do you mean without a server and just P2P propagation?

Last edited by dalingrin (2009-10-06 17:34:06)

Vintendo · 2009-10-06 17:54:36

What I want is a centralized server that syncs content between client in a special folder. So the first one.

toad · 2009-10-06 17:59:59

scio wrote:

Ideally, the server would have no control over the clients.

and

scio wrote:

However, this would require a process running on the client side that constantly asks the server "did anything change?" or sits with an open connection to the server waiting for the magic message "stuff changed!" so it can go and sync up. This is my biggest gripe with dropbox for my laptop, it is the worst offender of cpu wakes

presents a dichotomy. Both are bad. Question is, which is worse? To eliminate frequent CPU wake ups the server should be the one which tells the client that things have changed. Does this necessarily amount to control over the client? Couldn't it just be that the server says "hey, new stuff" and the client, when switched on, automagically/asks for permission to change said folder/set of folders? I reckon I am confused when you say that ideally the server should have no control...

dalingrin wrote:

I don't think I completely understand the objective.

Objective for me = easiest possible way of syncing files incl. pim stuff and what have you across any number of computers. How this could best be achieved I have no idea. afaik a server is good 'cos none of the clients have to be online at the same time which in turn would be the major draw back of p2p - so from that premise I'd prefer a server set up.

toad · 2009-10-06 18:00:49

vintendo, you beat me to it

rson451 · 2009-10-06 18:42:22

My initial thoughts on this are to use a DVCS. On the clients, have incron commit and push to the server when a file is changed. On the server, have a post-update hook that dumps the latest commit ref in a file. The clients then would have to poll that file via cron and if the ref doesn't match their HEAD, pull.

jasonwryan · 2009-10-06 18:45:16

Vintendo wrote:

What I want is a centralized server that syncs content between client in a special folder. So the first one.

I am quite keen to see the same thing: Dropbox is currently the only closed source software I run, and I am loathe to give it up becuase it works so damn well...

This thread has me thinking: hasn't this been solved (or the bulk of the code already written)? The server is effectively a repo and you want to be able to sync multiple machines? Would it not be easier to repurpose pacman?

toad · 2009-10-06 19:09:38

@ jasonwryan

pacman has, afaik, no functionality that we are after. Indeed, an automatic pacman sync would inevitably result in a broken system sooner or later (which is why there are plenty of threads about this in the forum but each one carries a MASSIVE warning against just such functionality - aka it's no bug but a feature )

But I reckon you're right in that the code is out there - question is which pieces of the puzzle have to be put together to make this picture...

Vintendo · 2009-10-06 19:21:26

jasonwryan wrote:

Vintendo wrote:
What I want is a centralized server that syncs content between client in a special folder. So the first one.
I am quite keen to see the same thing: Dropbox is currently the only closed source software I run, and I am loathe to give it up becuase it works so damn well...
This thread has me thinking: hasn't this been solved (or the bulk of the code already written)? The server is effectively a repo and you want to be able to sync multiple machines? Would it not be easier to repurpose pacman?

I think that svn or another version control system would be more appropriate then using pacman for this.

dalingrin · 2009-10-06 19:38:24

I haven't done it myself but it doesn't seem like it would be too hard to implement a custom push protocol which would initiate a file transfer. The file transfer itself could be prexisting code.

Last edited by dalingrin (2009-10-06 21:08:38)

scio · 2009-10-06 19:50:00

Dropbox is essentially a SVN server, this whole process is just automating check in and updates. The distributed version as rson451 and I have been talking about would be something like git, bzr, or hg.

rson451 has the right idea, but as we said before, the hard part is updating others. Check ins are trivial, the updates are the hard / annoying part.

My distributed rant was mostly a stream of randomness to reduce cpu usage on my laptop.

If you want to have similar functionality quickly, sign up for a free VCS account and create a new repo then just check in and out manually (like many people on the forums do for dot files, http://github.com/ghost1227/dotfiles)

dalingrin · 2009-10-06 20:58:41

I was thinking of a custom push protocol + rsync.

The custom push would just be used to eliminate cron. Rsync would take care of the mirroring so the push protocol would just initiate rsync on the client when something has changed. This would eliminate the CPU wake ups right?

scio · 2009-10-06 23:09:27

dalingrin wrote:

The custom push would just be used to eliminate cron. Rsync would take care of the mirroring so the push protocol would just initiate rsync on the client when something has changed. This would eliminate the CPU wake ups right?

What triggers the push? How would the push initiate rsync on the client?

dalingrin · 2009-10-07 01:37:40

The push would be an open connection between the server and clients sitting idle. My thoughts were something like Inotify or FAM would be on the server watching for changes. Once a change has been made the push mechanism would send info to the clients which would then initiate the rsync. I haven't personally worked with inotify, fam or rsync so perhaps I'm way off.

On the client, they would have a very small memory resident process sitting idle holding the connection open until a message is sent from the server which would then launch a script(rsync).

Last edited by dalingrin (2009-10-07 01:40:44)

toad · 2009-10-07 05:38:09

fwiw - rsync has my cpu going off the scale on a data set of less than 1GB

iphitus · 2009-10-07 08:54:27

There's also SpiderOak.

Give it a look. Just like Dropbox, except they claim to offer complete privacy. All data is encrypted locally, so their servers only ever see encrypted data. The servers never see you password, so If you lose your password, you lose your data.

They have released some code and patches under open source, and intend to release the client/server as open source too. I think the client is python based, so the code is visible, however I'm not sure.

Otherwise inosync -> rsync would work nicely. If you don't have your own server, rsync.net do pretty cheap rsync services and offer discounts for students/foss developers.

That said, I'm happy with dropbox. It works really well. Anything I put in my dropbox folder, I'd just as happily put onto my public http. The sharing features and web ui have been very useful too.

toad · 2009-10-07 09:11:32

Thanks iphitus, but I'd much rather have an open source solution over which I have more control...

I'll have a look at the VCS thingy scio was talking about although it is not what either dropbox or SpiderOak do.

crankyadmin · 2009-10-07 12:04:22

VPS + Unison = Free Open Source Drop Box clone.

I've been doing it for years. And never failed me once.

EDIT: My VPS costs me $15 a month for 50Gb

Last edited by crankyadmin (2009-10-07 12:05:16)

toad · 2009-10-07 12:56:18

Thanks, crankyadmin. A fine thing indeed but expensive and not quite on target afaik 'cos Unison has to be invoked by something and requires a fair amount of interaction.

Arch Linux

#1 2009-10-06 10:35:16

dropbox like functionality

#2 2009-10-06 11:46:39

Re: dropbox like functionality

#3 2009-10-06 13:16:51

Re: dropbox like functionality

#4 2009-10-06 13:32:48

Re: dropbox like functionality

#5 2009-10-06 13:43:01

Re: dropbox like functionality

#6 2009-10-06 13:54:31

Re: dropbox like functionality

#7 2009-10-06 16:47:05

Re: dropbox like functionality

#8 2009-10-06 17:33:10

Re: dropbox like functionality

#9 2009-10-06 17:54:36

Re: dropbox like functionality

#10 2009-10-06 17:59:59

Re: dropbox like functionality

#11 2009-10-06 18:00:49

Re: dropbox like functionality

#12 2009-10-06 18:42:22

Re: dropbox like functionality

#13 2009-10-06 18:45:16

Re: dropbox like functionality

#14 2009-10-06 19:09:38

Re: dropbox like functionality

#15 2009-10-06 19:21:26

Re: dropbox like functionality

#16 2009-10-06 19:38:24

Re: dropbox like functionality

#17 2009-10-06 19:50:00

Re: dropbox like functionality

#18 2009-10-06 20:58:41

Re: dropbox like functionality

#19 2009-10-06 23:09:27

Re: dropbox like functionality

#20 2009-10-07 01:37:40

Re: dropbox like functionality

#21 2009-10-07 05:38:09

Re: dropbox like functionality

#22 2009-10-07 08:54:27

Re: dropbox like functionality

#23 2009-10-07 09:11:32

Re: dropbox like functionality

#24 2009-10-07 12:04:22

Re: dropbox like functionality

#25 2009-10-07 12:56:18

Re: dropbox like functionality

Board footer