Sugestions on minimalistic file syncroniser?

Hessiess · 2017-09-30 23:24:19

For a long time I used Subversion to manage my whole user file system however the work I do involves a lot of large image and video files. SVN handles these flawlessly, but working with it became impractical as it stores two copies of all files in a checkout, and I sometimes need to work on a laptop with no Internet. I tried out numerous file synchronizers and 'self hosted cloud' systems last year and all of where either overcomplicated, did not work reliably or did not have the needed features. In the end I gave up and quickly wrote a tool to solve the problem:

https://github.com/robehickman/simple-http-file-sync

While this tool works, and I'm surprised how well it works given how simplistic it is, it isn't perfect. Before putting in the work to improve it I'm interested if anyone knows of existing projects with similar goals? Here are a summery of my needs:

* Due to poor outbound network performance system must be hosted locally.
* Version tracking on a centralized repository.
* Single focus, I find 'self hosted cloud' systems to be dumping grounds of functionality.
* Command line and configuration file based (no web interface!)
* No dependency on a database
* Must support partial checkouts/ignore list
* No restrictions on file names, I don't use NTFS or FAT, so should not be limited by them.
* Must support multiple 'repositories'
* Leans toward being a version control system for binary files (i.e. merging not needed)
* Preferably HTTP(s) based. I don't currently need to, but it would be useful to function inside restrictive networks.

I'm aware of rsync and unison, neither of which handle version tracking and rsync only works one way. Git Annex started with similar ideas but is overcomplicated. A centralized system works fine for me. SVN would be perfect if it didn't insist on duplicating as noted above.

jasonwryan · 2017-09-30 23:37:17

It doesn't fulfill all your criteria (if anything would), but syncthing is pretty awesome. If you want version tracking, just point it at a repo and it will sync that. You are under no obligation to use the web interface either.

Moving to Apps and DEs...

Trilby · 2017-09-30 23:44:11

I was thinking definitely rsync, until this...

Hessiess wrote:

I'm aware of rsync and unison, neither of which handle version tracking and rsync only works one way.

This depends what you mean by version tracking. What exactly do you need out of version tracking? You can make incremental backup stashes. They can be in a directory with a date, or a sequential number, or both. How is this not versioning? Also, rsync can/does do two-way syncs.

But if you want more of a version control system that a backup syncronization, I'd also suggest fossil - a web interface is available, but there is no need to use it.

Hessiess · 2017-10-03 21:45:22

Thanks for the comments, I have tried syncthing before, however it wasn't able to find the other computer on my network using the code that it gave. I'll have a look at fossil in detail when I get chance, the website says it's distributed, but it may be workable if it's able to leave the history on a server. I can have essentially unlimited storage on a central server given how cheap hard drives are these days, however doing the same on all my client devices is less practical.

What I want is a file synchronizer with some version control features, past versions need to be easy to access, such as issuing a command like 'checkout [version id]'. I realized that a system like I want would be a niche project if it did exist, the problem with which is it can be difficult to find them. I thought that here would be the best place to ask as Arch users tend to go into the niche things more frequently, such as tiling window managers.

progandy · 2017-10-03 23:58:28

Have you tried seafile community editition?

* Due to poor outbound network performance system must be hosted locally. YES
* Version tracking on a centralized repository. YES
* Single focus, I find 'self hosted cloud' systems to be dumping grounds of functionality. YES
* Command line and configuration file based (no web interface!) POSSIBLE
* No dependency on a database NO, mysql or sqlite for server, sqlite for cli client
* Must support partial checkouts/ignore list YES
* No restrictions on file names, I don't use NTFS or FAT, so should not be limited by them. PROBABLY
* Must support multiple 'repositories' YES, libraries
* Leans toward being a version control system for binary files (i.e. merging not needed) YES
* Preferably HTTP(s) based. I don't currently need to, but it would be useful to function inside restrictive networks. YES, requires nginx or apache

What I want is a file synchronizer with some version control features, past versions need to be easy to access, such as issuing a command like 'checkout [version id]'.

I don't know if the existing clients support that, but you could extend them or write your own using the provided APIs.

https://seafile.com
https://manual.seafile.com/
https://freeplant.gitbooks.io/seafile-u … l/content/

Last edited by progandy (2017-10-04 00:11:11)

Arch Linux

#1 2017-09-30 23:24:19

Sugestions on minimalistic file syncroniser?

#2 2017-09-30 23:37:17

Re: Sugestions on minimalistic file syncroniser?

#3 2017-09-30 23:44:11

Re: Sugestions on minimalistic file syncroniser?

#4 2017-10-03 21:45:22

Re: Sugestions on minimalistic file syncroniser?

#5 2017-10-03 23:58:28

Re: Sugestions on minimalistic file syncroniser?

Board footer