Dvcs-autosync: An open source dropbox clone... well.. almost

Dieter@be · 2011-03-27 13:28:01

I found the Dvcs-autosync project on the vcs-home mailing list, which btw is a great list for folks who are doing stuff like maintaining their home directory in a vcs.
In short:

- simple python tool (600 sloc), works with your dvcs of choice (mainly tested/used with git)
- watches for inotify events, performs commits on changes (coalesces some changes together)
- synchronizes with other clones (remotes), uses xmpp for push notifications
- let's you know what's going on through libnotify popups or whatever
- home page: http://mayrhofer.eu.org/dvcs-autosync
- dvcs-autosync gitorious repo: http://gitorious.org/dvcs-autosync/dvcs-autosync
- initial announcement on vcs-home: http://lists.madduck.net/pipermail/vcs- … 00314.html
- AUR git package: http://aur.archlinux.org/packages.php?ID=47643

Use cases:

- you have one or more trees you want to maintain under a VCS (because you want vcs advantages like history and whatnot) but it's not worth to spend time comitting manually
- you want to backup some files transparently
- you want to share / work with others easily
- you like the idea of dropbox, but not the closed-source-ness or the vendor dependence
- you work mainly with relatively small files (or.. read on)

Thoughts:

- very simple to get started
- simpler (and imho saner) code and implementation in comparison to sparkleshare
- bound to the limitations of the dvcs. In case of git: no support for ownership, xattrs, and less suited for bigger files (although git-annex might help?)
- You cannot store a git repo inside a git repo, I think (i.e. I don't think you can keep a -potentially dirty- git clone in dvcs-autosync)
- For a real, filesystem-level dropbox-alike coda might be a better option, though I'm not sure how useful that project is right now

If it sounds like something you need, try it out. And contributions welcome.
Next up to the todolist: more clever heuristic for event coalescing, setting up bugtracker.

Floft · 2011-03-30 00:44:35

I've been playing around with it for the last few days. The only thing I noticed is that I didn't see an option not to show notifications (in the version I had). That's easily fixed though... the way I did it: link removed I do like it overall.

Last edited by Floft (2011-06-16 19:04:54)

Dieter@be · 2011-03-30 09:39:35

that seems like a useful option; if you have github or gitorious (the project is on both) you can do a pull request there

ltpl4y3r · 2011-04-01 02:25:37

Do you know of any good resources for learning how to set this up from scratch? I am hoping to use this to sync between 3 different PCs and have never setup/used git without a PKGBUILD.

Dieter@be · 2011-04-01 08:55:25

ltpl4y3r wrote:

Do you know of any good resources for learning how to set this up from scratch? I am hoping to use this to sync between 3 different PCs and have never setup/used git without a PKGBUILD.

just install it from AUR with a tool like clyde, setting up is easy; instructions are on the project website.

skodabenz · 2011-04-01 11:54:55

I wonder whether it allows maintaining custom arch repos like some users do you in dropbox.

Dieter@be · 2011-04-01 20:17:15

skodabenz wrote:

I wonder whether it allows maintaining custom arch repos like some users do you in dropbox.

hmm that's another interesting use of dropbox (or clone thereof).

answer: it does allow it, you just need to be aware that storing big files in git is not very efficient (your disk usage doubles), until we've figured out how to use git-annex in dvcs-autosync

smartboyathome · 2011-04-01 21:12:24

It looks like a project called Sharepoint on Github is doing something similar, and even uses python as well. Perhaps you two could combine efforts or could learn from each others' code?

Dieter@be · 2011-04-02 09:01:09

smartboyathome wrote:

It looks like a project called Sharepoint on Github is doing something similar, and even uses python as well. Perhaps you two could combine efforts or could learn from each others' code?

it's sharebox, and.. way ahead of you
see the comments @ http://dieter.plaetinck.be/dvcs-autosyn … lmost.html and discussion @ http://lists.madduck.net/pipermail/vcs- … 00351.html

ngoonee · 2011-04-06 06:47:22

How useful would this be for automatically updating a local git repo? In other words I don't want to use all the networked stuff. Just something which would add . and commit -a whenever inotify says something happened.

ltpl4y3r · 2011-04-07 03:10:54

Okay, I have managed to get it working, for the most part. I think your instructions may be missing:

git add -S

before the first commit.

Anyway, I was wondering what use the login information is in the config. Am I going to need to make it so that it can ssh in w/o the need to supply a password?

DEBUG:root:Starting push timer with 5 seconds until push would occur (if no other changes happen in between)
DEBUG:root:Tick 1 / 5
DEBUG:root:Tick 2 / 5
DEBUG:root:Starting coalesce timer with 2 seconds until coalescing events for file /home/nick/autosync/01 Oya Shirazu.mp3 would occur (if no other changes happen in between)
DEBUG:root:Resetting already active coalesce timer to new timeout of 2 seconds until coalescing events for file /home/nick/autosync/01 Oya Shirazu.mp3 would occur
DEBUG:root:Tick 3 / 5
DEBUG:root:Tick 4 / 5
DEBUG:root:Tick 5 / 5
INFO:root:NOTIFICATION: Pushing changes: Pushing last local changes to remote repository
dropbox@192.168.2.201's password: INFO:root:Coalesce event triggered for file /home/nick/autosync/01 Oya Shirazu.mp3

I would like to note that to log into the server, it is a different login name and password than the workstation. Any hints on how to proceed?

EDIT:
Sorry, doesn't seem to be syncing between computers.. Back to the drawing board.

Last edited by ltpl4y3r (2011-04-07 03:13:47)

Dieter@be · 2011-04-07 12:21:22

ltpl4y3r wrote:

Okay, I have managed to get it working, for the most part. I think your instructions may be missing:
git add -S
before the first commit.

What? The instructions clearly mention to add to the index.

ltpl4y3r wrote:

Anyway, I was wondering what use the login information is in the config. Am I going to need to make it so that it can ssh in w/o the need to supply a password?
DEBUG:root:Starting push timer with 5 seconds until push would occur (if no other changes happen in between)
DEBUG:root:Tick 1 / 5
DEBUG:root:Tick 2 / 5
DEBUG:root:Starting coalesce timer with 2 seconds until coalescing events for file /home/nick/autosync/01 Oya Shirazu.mp3 would occur (if no other changes happen in between)
DEBUG:root:Resetting already active coalesce timer to new timeout of 2 seconds until coalescing events for file /home/nick/autosync/01 Oya Shirazu.mp3 would occur
DEBUG:root:Tick 3 / 5
DEBUG:root:Tick 4 / 5
DEBUG:root:Tick 5 / 5
INFO:root:NOTIFICATION: Pushing changes: Pushing last local changes to remote repository
dropbox@192.168.2.201's password: INFO:root:Coalesce event triggered for file /home/nick/autosync/01 Oya Shirazu.mp3
I would like to note that to log into the server, it is a different login name and password than the workstation. Any hints on how to proceed?
EDIT:
Sorry, doesn't seem to be syncing between computers.. Back to the drawing board.

use ssh keys

ngoonee · 2011-04-16 09:53:32

ngoonee wrote:

How useful would this be for automatically updating a local git repo? In other words I don't want to use all the networked stuff. Just something which would add . and commit -a whenever inotify says something happened.

Tried this out, but keep getting the following:-

DEBUG:root:AutosyncJabberbot:unable to connect to server your XMPP id here.
Exception in thread Thread-7:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/usr/bin/autosync.py", line 139, in run
self.expire(self.arg)
File "/usr/bin/autosync.py", line 401, in _real_push
bot.send(sendto, 'pushed %s' % remoteurl)
File "/usr/lib/python2.7/site-packages/jabberbot.py", line 179, in send
self.send_message(mess)
File "/usr/lib/python2.7/site-packages/jabberbot.py", line 166, in send_message
self.connect().send(mess)
AttributeError: 'NoneType' object has no attribute 'send'

Of course, I have not setup any xmpp details. Doesn't seem to affect proper functioning (so far, fingers crossed). So, a question and a request:-

1. I guess wouldn't make sense to have this script without the networking bits, but if I wish to run it that way is it 'safe'? I assume nothing gets touched in the base directory in any case.
2. Could the .autosync-example file be included in the built package? Maybe as /etc/xdg/autosync/example or /usr/share/examples/something. Would make more sense, IMHO.

Dieter@be · 2011-04-16 10:09:42

ngoonee wrote:

1. I guess wouldn't make sense to have this script without the networking bits, but if I wish to run it that way is it 'safe'? I assume nothing gets touched in the base directory in any case.

Not sure what you mean, but supporting `syncmethod = none` seems fairly trivial. (the variable already exists, but we don't use it yet)

ngoonee wrote:

2. Could the .autosync-example file be included in the built package? Maybe as /etc/xdg/autosync/example or /usr/share/examples/something. Would make more sense, IMHO.

true.
Right now my package copies some files, but Rene was gonna implement setup.py/python distutils/setuptools whatever that thing is called. So i'm waiting a bit for that to update my package.

Either way, you can create tickets for both issues @ https://github.com/rmayr/dvcs-autosync (or better: do pull requests

ngoonee · 2011-04-16 10:27:42

Dieter@be wrote:

ngoonee wrote:
1. I guess wouldn't make sense to have this script without the networking bits, but if I wish to run it that way is it 'safe'? I assume nothing gets touched in the base directory in any case.
Not sure what you mean, but supporting `syncmethod = none` seems fairly trivial. (the variable already exists, but we don't use it yet)

Yes, that would be swell!

Dieter@be wrote:

ngoonee wrote:
2. Could the .autosync-example file be included in the built package? Maybe as /etc/xdg/autosync/example or /usr/share/examples/something. Would make more sense, IMHO.
true.
Right now my package copies some files, but Rene was gonna implement setup.py/python distutils/setuptools whatever that thing is called. So i'm waiting a bit for that to update my package.
Either way, you can create tickets for both issues @ https://github.com/rmayr/dvcs-autosync (or better: do pull requests

Hmm, the second is more a packaging suggestion for you, but yes if a 'proper' packaging is planned then I'll wait for that. I'm a python illiterate currently so I'll just file a ticket for the first . Thanks. Btw, why a github address when the project is gitorious?

Dieter@be · 2011-04-16 10:38:44

Dieter@be wrote:

Hmm, the second is more a packaging suggestion for you, but yes if a 'proper' packaging is planned then I'll wait for that. I'm a python illiterate currently so I'll just file a ticket for the first . Thanks. Btw, why a github address when the project is gitorious?

we're still evaluating different options. Rene is more of a gitorious fan (but it doesn't have a bugtracker), I'm the github fan.
the 2nd issue is definitely not just a packaging issue. AFAIK the setup.py thingie should take care of putting all files where they belong (for a given prefix / "root tree"), so that the PKGBUILD file can just call python2 setup.py, similar to how you would deal with makefiles. I'ld still make a ticket for your issue so that you're sure it doesn't get forgotten.

Arch Linux

#1 2011-03-27 13:28:01

Dvcs-autosync: An open source dropbox clone... well.. almost

#2 2011-03-30 00:44:35

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#3 2011-03-30 09:39:35

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#4 2011-04-01 02:25:37

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#5 2011-04-01 08:55:25

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#6 2011-04-01 11:54:55

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#7 2011-04-01 20:17:15

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#8 2011-04-01 21:12:24

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#9 2011-04-02 09:01:09

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#10 2011-04-06 06:47:22

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#11 2011-04-07 03:10:54

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#12 2011-04-07 12:21:22

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#13 2011-04-16 09:53:32

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#14 2011-04-16 10:09:42

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#15 2011-04-16 10:27:42

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

#16 2011-04-16 10:38:44

Re: Dvcs-autosync: An open source dropbox clone... well.. almost

Board footer