You are not logged in.

#1 2012-09-07 09:16:46

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

[no longer relevant] Selfhosted Dropbox alternative

I'm sure this topic has been discussed before, but I haven't been able to find it.

In any case, I'm really tired of DropBox, not least due to it's limitations in regards to storage. Of course you can just pay up, but I find it way too expensive for a product that really isn't that great, especially on a Linux platform.

Further it so happens that I have my own VPS, that is of course not my own as in that I own it, but as in I control it.

So the solution it obvious, but I haven't been able to find much on the subject.


In any case I would very much like to hear your recommendations, especially if you know of a solution that don't require me to set up a webserver first, as I find  the two subject unrelated.


Best regards.


zacariaz wrote:

Thanks for the reply.
I've changed the thread title to better explain the issue.

The links so far did offer one interesting option, unison, though a few issues was pointed out, the thread was also a couple of years old, so I'll investigate further and hope it is useful.

I would however like to point out that I'm not interested in a fancy GUI or anything like that. I simply want a folder on my computer(s) that is automatically and on the fly synced with the server.

Last edited by zacariaz (2012-09-10 14:10:39)


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#2 2012-09-07 09:24:45

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Offline

#3 2012-09-07 10:37:33

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Thanks for the reply.
I've changed the thread title to better explain the issue.

The links so far did offer one interesting option, unison, though a few issues was pointed out, the thread was also a couple of years old, so I'll investigate further and hope it is useful.

I would however like to point out that I'm not interested in a fancy GUI or anything like that. I simply want a folder on my computer(s) that is automatically and on the fly synced with the server.


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#4 2012-09-07 14:03:06

lifeafter2am
Member
From: 127.0.0.1
Registered: 2009-06-10
Posts: 1,332

Re: [no longer relevant] Selfhosted Dropbox alternative

rsync?


#binarii @ irc.binarii.net
Matrix Server: https://matrix.binarii.net
-------------
Allan -> ArchBang is not supported because it is stupid.

Offline

#5 2012-09-07 14:09:14

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

lifeafter2am wrote:

rsync?

I don't think that can work on the fly.


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#6 2012-09-07 14:23:40

lifeafter2am
Member
From: 127.0.0.1
Registered: 2009-06-10
Posts: 1,332

Re: [no longer relevant] Selfhosted Dropbox alternative


#binarii @ irc.binarii.net
Matrix Server: https://matrix.binarii.net
-------------
Allan -> ArchBang is not supported because it is stupid.

Offline

#7 2012-09-07 14:43:54

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: [no longer relevant] Selfhosted Dropbox alternative

@lifeafter2am
unison is basically bidirectional rsync.

@zacariaz
I recommend unison. It may not work on the fly, but you don't really need it to in a single-user setup. Configure a script to run it when you log in and then again when you log out. Use the VPS as your central repo and sync everything with that via ssh. If you only work on one system at a time and sync at the beginning and end of each session then you can fully automate conflict resolution. See e.g. "-prefer newer".

The only thing that it doesn't handle are hardlinks. Hardlinks on one system will be created as separate files on another. If the target directory is on a single device then you can recreate the hardlinks with some scripting and "hardlink" (provided by the "hardlink" package in [community]). My current method is to use "find" to determine existing hardlinks on both systems (via ssh on the remote system), then pass the list to a script after the sync to link the target files on both systems (again, using ssh to run the command on the remote system).

Last edited by Xyne (2012-09-07 14:45:21)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#8 2012-09-07 14:51:33

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Xyne wrote:

@lifeafter2am
unison is basically bidirectional rsync.

@zacariaz
I recommend unison. It may not work on the fly, but you don't really need it to in a single-user setup. Configure a script to run it when you log in and then again when you log out. Use the VPS as your central repo and sync everything with that via ssh. If you only work on one system at a time and sync at the beginning and end of each session then you can fully automate conflict resolution. See e.g. "-prefer newer".

The only thing that it doesn't handle are hardlinks. Hardlinks on one system will be created as separate files on another. If the target directory is on a single device then you can recreate the hardlinks with some scripting and "hardlink" (provided by the "hardlink" package in [community]). My current method is to use "find" to determine existing hardlinks on both systems (via ssh on the remote system), then pass the list to a script after the sync to link the target files on both systems (again, using ssh to run the command on the remote system).

The trouble with unison is first of all that you need to use the same version across all clients, which quite frankly is too much of a bother.
Furthermore I use multiple systems at the same time quite often, so that is also a problem.

The rsync solution is interesting as it's highly configurable, however I fear that it's too complicated for me.

The thing is that I have clients that have an excess of storage and some that don't. Those that do can just do a complete sync, but in other cases another solution may be preferably.

Well, guess I'll have to think about it.

Thanks for now and best regards.


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#9 2012-09-10 09:38:33

teekay
Member
Registered: 2011-10-26
Posts: 271

Re: [no longer relevant] Selfhosted Dropbox alternative

Not sure what exactly your requirements are, but FUSE sshfs comes to mind - with it you can simply mount the folder on the remote VPS on any other box using SSH. Concurrent mounts are no problem, as file locking works the same as on a local box with concurrent users. And as it's a mount, it's on-the-fly of course.

Last edited by teekay (2012-09-10 09:52:28)

Offline

#10 2012-09-10 11:05:25

FreeTheBee
Member
Registered: 2011-01-02
Posts: 125

Re: [no longer relevant] Selfhosted Dropbox alternative

Perhaps this is also interesting,

http://git-annex.branchable.com/

I am not sure how usable it is at the moment though.

Offline

#11 2012-09-10 13:27:23

steelbak
Member
Registered: 2011-02-27
Posts: 19
Website

Re: [no longer relevant] Selfhosted Dropbox alternative

http://ajaxplorer.info/

It runs on a web server, but is great. As a matter of fact, I can even send you a VirtualBox image with it ready to fly on an ArchLinux server.

Last edited by steelbak (2012-09-10 13:28:51)

Offline

#12 2012-09-10 14:10:11

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Thank you all for your suggestions, but I have actually decided that I can no longer afford my VPS, no as in I couldn't, but rather shouldn't.

If there are other alternatives to DropBox, because it still rather sucks in various areas, I'd very much like to hear about them, but other than that, thanks once again, but no thanks. wink

Oh ya, just a little tiny think. I've noticed that DropBox sync both to ~/Dropbox and to /root/Dropbox

If anyone can explain that, that'd be helpful.


Best regards.


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#13 2012-09-11 08:16:26

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

zacariaz wrote:

If there are other alternatives to DropBox, because it still rather sucks in various areas, I'd very much like to hear about them, but other than that, thanks once again, but no thanks. wink

For me the most basic "alternative" to DropBox (given you can always use ssh from where you are) simply is sshfs. You can set up a pretty secure ssh Server at home and access it via a dynamic DNS hoster (no-ip, dyndns, etc). Most routers provide the option to refresh your IP at the dyn DNS provider if your ISP changes it.

ssh is pretty versatile. Beside using it for file services you can route http traffic to it or simply access your systems at home.
If you do not like to expose ssh to the outside world there also would be the option to run a chrooted sftp server and use sftpfs as DropBox-alike.

Today platforms such as a Raspberry Pi enable you to run power-efficient servers at home without a big power bill.

Software such as OwnCloud IMO go far beyond the sole purpose of serving files.

HTH
D$


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#14 2012-09-11 13:59:20

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Darksoul71 wrote:
zacariaz wrote:

If there are other alternatives to DropBox, because it still rather sucks in various areas, I'd very much like to hear about them, but other than that, thanks once again, but no thanks. wink

For me the most basic "alternative" to DropBox (given you can always use ssh from where you are) simply is sshfs. You can set up a pretty secure ssh Server at home and access it via a dynamic DNS hoster (no-ip, dyndns, etc). Most routers provide the option to refresh your IP at the dyn DNS provider if your ISP changes it.

ssh is pretty versatile. Beside using it for file services you can route http traffic to it or simply access your systems at home.
If you do not like to expose ssh to the outside world there also would be the option to run a chrooted sftp server and use sftpfs as DropBox-alike.

Today platforms such as a Raspberry Pi enable you to run power-efficient servers at home without a big power bill.

Software such as OwnCloud IMO go far beyond the sole purpose of serving files.

HTH
D$

Only problem being that I'm on a shared connection behind a router which I do not access to, otherwise I'd never have spend money on a VPS.

So if there's a solution to that problem I'm very interested. big_smile


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#15 2012-09-11 14:07:00

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

zacariaz wrote:

Only problem being that I'm on a shared connection behind a router which I do not access to, otherwise I'd never have spend money on a VPS.
So if there's a solution to that problem I'm very interested. big_smile

Uh oh...a "hostile" network so to speak. Then pagekite might be an option for you to access your system from external:
http://pagekite.net/

They provide some sort of reverse-proxy thingie (sorry, I am not a network guy). By running a simply python script at your local machine you are able to access it under a .pagekite domain from external. Even ssh is possible.

You might want to check their webpage. I think the quickstart provides most of what you can do:
http://pagekite.net/support/quickstart/

Last edited by Darksoul71 (2012-09-11 14:08:50)


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#16 2012-09-11 14:15:49

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Darksoul71 wrote:
zacariaz wrote:

Only problem being that I'm on a shared connection behind a router which I do not access to, otherwise I'd never have spend money on a VPS.
So if there's a solution to that problem I'm very interested. big_smile

Uh oh...a "hostile" network so to speak. Then pagekite might be an option for you to access your system from external:
http://pagekite.net/

They provide some sort of reverse-proxy thingie (sorry, I am not a network guy). By running a simply python script at your local machine you are able to access it under a .pagekite domain from external. Even ssh is possible.

You might want to check their webpage. I think the quickstart provides most of what you can do:
http://pagekite.net/support/quickstart/

I've always been convinced something like this was possible, but I've never been able to find any solutions.

Thanks a bunch, I'll certainly check it out.


Best regards.


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#17 2012-09-11 14:17:42

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

zacariaz wrote:

I've always been convinced something like this was possible, but I've never been able to find any solutions.
Thanks a bunch, I'll certainly check it out.

You are welcome ! While pagekite costs money in the end, you are still able to try out if it works for you smile


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#18 2012-09-11 14:22:57

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Darksoul71 wrote:
zacariaz wrote:

I've always been convinced something like this was possible, but I've never been able to find any solutions.
Thanks a bunch, I'll certainly check it out.

You are welcome ! While pagekite costs money in the end, you are still able to try out if it works for you smile

oh...
That why the package in aur is outdated...

What I can get for that same price as my vpn:

Free pagekite.py software & updates
5 month(s) of service
27 GB of transfer quota
Unlock up to 2 extra .pagekite.me name(s)
The ability to use CNAMEs from your own domain

Of course that's for 5 months instead of just one, but that actually only make things worse, assuming that is not a monthly transfer quota.

No I'm sorry, but it was too good to be true. big_smile


edit:
apparently it is possible to get it for free, but if these are the limitations when you pay...

Last edited by zacariaz (2012-09-11 14:37:11)


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#19 2012-09-11 15:03:47

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

Well, everything in life comes at a certain price wink

Last edited by Darksoul71 (2012-09-11 15:06:29)


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#20 2012-09-11 15:09:16

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Darksoul71 wrote:

Well, everything in life comes for a certain price wink

Indeed, but I don't quite understand why it's necessary with a dynamic tunneled reverse proxy. (No I didn't just make that up) Such a solution will of course cost resource and thus money.

I mean data can be transferred both ways, it's just a matter of creating the connection. I'm no network guru, but I really can't see why this should be so complicated.

Obviously you'd need help from "someone" visible. In my case I have a webserver that could do the job, but why on earth does it have to serve as a proxy for it to work in this fashion?


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#21 2012-09-11 15:18:58

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

Well, I never looked too deeply into their approach. Also I lack the knowledge to do something similar to this.
Accessing services on a local machine behind a firewall which you do not control makes it hard to access them sad

So at least some sort of tunnelling / proxy stuff needs to be done. At least unless you can expose an internal port of your machine to the outside world. Of course all this is possible with existing technology but the blind is not king in between the one eyed smile


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#22 2012-09-11 15:32:07

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

I guess I'm thinking about this the wrong way.

In my mind it perfectly liable to think something like this:

Mr. Smith and Mr. Anderson don't know each others phone number, but they both know the phone number of Mr. Langley. Mr. Langley don't even have to know their phone numbers as they show up on his phone when they call. thus it's rather obvious how to establish a connection.

Sadly it would seem the world doesn't quite work this way. wink


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#23 2012-09-11 20:43:37

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

To my limited understanding you are not thinking the wrong way.

To keep your example:
Mr. Smith calls Mr. Langley which calls Mr. Anderson.

Or more specific for your use case:
Your PC "at home" connects to an external server which IP is know externally (e.g. ssh tunnel). Since communication from your system to an external host causes no problem with most routers, this nearly always works flawlessly. Now for accessing your PC at home from an external IP you "only" need to contact the external server which routes your request to your home system via the established tunnel. Enforcing applications to use a tunnel rather than bounding to a physical NIC can easily be done via proxychains. I guess this is pretty much what pagekite does. If you run their python script it connects to one of their server via ssh tunnel. If you now access their servers with your "subdomain" they route the requests back to your server. Technically no real voodoo but still requires an external server which needs to route the traffic through.


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#24 2012-09-11 21:02:29

zacariaz
Member
From: Denmark
Registered: 2012-01-18
Posts: 539

Re: [no longer relevant] Selfhosted Dropbox alternative

Darksoul71 wrote:

To my limited understanding you are not thinking the wrong way.

To keep your example:
Mr. Smith calls Mr. Langley which calls Mr. Anderson.

Or more specific for your use case:
Your PC "at home" connects to an external server which IP is know externally (e.g. ssh tunnel). Since communication from your system to an external host causes no problem with most routers, this nearly always works flawlessly. Now for accessing your PC at home from an external IP you "only" need to contact the external server which routes your request to your home system via the established tunnel. Enforcing applications to use a tunnel rather than bounding to a physical NIC can easily be done via proxychains. I guess this is pretty much what pagekite does. If you run their python script it connects to one of their server via ssh tunnel. If you now access their servers with your "subdomain" they route the requests back to your server. Technically no real voodoo but still requires an external server which needs to route the traffic through.

Well that's the problem really isn't it? Mr. Smith and Mr. Anderson can't communicate directly, thus all traffic has to go through Mr. Langley. In my work, Mr. Langley would simply SMS the phone number of Mr. Smith to Mr. Anderson and vise versa.

I guess the easiest explanation would be to say that neither Smith nor Anderson are able to receive call?

(I really need to get access to that router)


I am a philosopher, of sorts, not a troll or an imbecile.
My apologies that this is not always obvious, despite my best efforts.

Offline

#25 2012-09-12 06:49:58

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: [no longer relevant] Selfhosted Dropbox alternative

zacariaz wrote:

Well that's the problem really isn't it? Mr. Smith and Mr. Anderson can't communicate directly, thus all traffic has to go through Mr. Langley. In my work, Mr. Langley would simply SMS the phone number of Mr. Smith to Mr. Anderson and vise versa.

I guess the easiest explanation would be to say that neither Smith nor Anderson are able to receive call?

(I really need to get access to that router)

LOL...this will be my last reply on this topic prior a mod moves this to "Threads leading to nowhere" smile

To keep the example:
Mr. Anderson (external PC) wants to call Mr. Smith (home server) via cell phone but unfortunately cell phones do not work for Mr. Smith because he lives beyond a mountain (Router plus firewall which does not allow external access to internal services). So the work around for Mr. Smith is that he calls Mr. Langley via a normal wire-driven phone. Mr. Langley has both a working cell phone and a wire-driven phone. So Mr. Anderson calls Mr. Langley via cell phone who then simple puts the cell phone against the handset of the regular phone on which Mr. Smith is listening.

Most tools which communicate with other tools over routers / firewalls rely on external servers for "handshake". The communication after successful handshake though might take place without routing all traffic across the external server.


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

Board footer

Powered by FluxBB