bidirectionally syncing 2 or more servers

lilmike · 2015-04-21 19:14:45

Hi,
I have gone through about 6 or so different sync applications to try and get some of my folders on two arch linux web machines syncing. To name the ones I can think of off the top of my head, they are: osync, unison, glusterfs, lsyncd, and syncthing with the syncthing-inotify program.
Each one has fallen down in one way or the other. I thought I had syncthing working, but apparently the stress of having to keep track of over 53k files is getting to it, and it now doesn't want to sync. So my question:

Does anyone know of a relatively easy to set up, bidirectional sync solution (preferably that can sync multiple servers, but if I have to set it up in a star configuration I can, that also preferably can sync as soon or very soon after the file is changed/created?
Thanks,
-Michael.

lucke · 2015-04-21 20:19:27

Perhaps btsync?

lilmike · 2015-04-22 11:15:41

Hi,
I've thought about btsync, but the problem there is, as far as I know, it doesn't provide anything like syncthing-inotify to make transfers almost immediately on update/create.
Thanks,
-Michael.

kokoko3k · 2015-04-22 11:24:41

i use this to sync multiple clients, as a dropbox replacement, i really don't know if it would give problems with 53k files...

# cat /etc/systemd/system/unison.service 
[Unit]
Description=unison sync a pi

[Service]
Type=simple
ExecStart=/usr/bin/sudo -H -u kokoko3k /path/to/unison.sh
TimeoutSec=0
#StandardInput=tty
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

# cat  /path/to/unison.sh
#!/bin/bash

#This script will mimic Dropbox behaviour by using unison.
#it is STRONGLY advised to do the first sync manually

#You need:
#  ssh on client and server
#  ability to log in to the server as root with no password using ssh
#  inotifywait
#  sudo
#  unison
#  optionally, notify-send and a notification daemon running.


#Begin of configuration

  LDIR="/home/kokoko3k/Dropbox/"                 #Local Dir  ; Please use full path names
  RDIR="/home/kokoko3k/unison/"                  #Remote Dir ; Please use full path names

  RHOST="pi"                            #Remote Host
  RPORT="40022"                         #Remote Port

  TIMEOUT=60                          #Time in secs to check if something changed remotely

  user=kokoko3k                             #The user to send notifications to
  process=plasma-desktop                #A dbus registered process to send notifications
  ICON=~/scripts/unison.sync.png        #Icon for notify-send

  LOGFILE=/home/kokoko3k/.unison/complete.log        #Complete Logfile
  LOGFILELAST=/home/kokoko3k/.unison/last.log        #Partial Logfile (holds last sync info until a TIMEOUT cycle)
  LOGFILEDEBUG=/home/kokoko3k/.unison/debug.log      #Messages useful for debug

#End of configuration


# Function used to send notifications
notify-send-all(){
    for p in $(pgrep $process); do
                dbus=$(grep -z DBUS_SESSION_BUS_ADDRESS /proc/$p/environ | sed 's/DBUS_SESSION_BUS_ADDRESS=//')
                user=$(grep -m 1 -z USER /proc/$p/environ | sed 's/USER=//')
                dply=$(grep -z DISPLAY /proc/$p/environ | sed 's/DISPLAY=//')
                sudo -u $user sh -c "DBUS_SESSION_BUS_ADDRESS=\"$dbus\" DISPLAY=\"$dply\" notify-send -i $ICON \"$1\" \"$2\" "
        done
        }


# Main sync function
unisync(){
  unison -logfile $LOGFILELAST         \
         -batch -prefer newer -times   \
         -copymax 10 -maxthreads 10    \
         $LDIR ssh://root@$RHOST/$RDIR \
         -sshargs "-p $RPORT -o cipher=arcfour" &> $LOGFILEDEBUG
}

#Main
cd

while true ; do
        rm $LOGFILELAST $LOGFILEDEBUG &>/dev/null
                (unisync) || (sleep 2 ; unisync) || (sleep 3 ; unisync) || (sleep 5 ; unisync)

        if [ -s $LOGFILELAST ] ; then
                notify-send-all "Unison: $(grep failed $LOGFILELAST)" "$(grep -v 'failed\|started\|BGN\|finished' $LOGFILELAST)"
                cat $LOGFILELAST >> $LOGFILE
        fi

        timeout $TIMEOUT inotifywait -r -e modify,move,create,delete $LDIR &>/dev/null
done

-edit

For me, the "server" is a raspberry pi and the clients are 3 Pcs with root ssh access to it (hostkey authentication); I use it to sync my savegames, and it works ; optionally it will notify you about the sync status (if you want the notifications, you have to run a notification daemon and change "plasma-desktop" to something specific to your environment.
through inotify, the changes are immediately synced to the server.
the clients will pull the changes every $TIMEOUT seconds.

* If fileA is newer than fileB, and they have the same name, fileB is overwritten everywhere.
* sync process will sync deletions too.

you may need to raise: /proc/sys/fs/inotify/max_user_watches

If you need clientA to be notified instantly about changes happened on clientB and don't want to wait for $TIMEOUT, you could:

* setup host based authentication from the server to the clients
* set an inotify on the server to the sync folder
* restart the unison server remotely through ssh

...or just set $TIMEOUT to a lower value

Last edited by kokoko3k (2015-04-22 11:37:14)

Ashren · 2015-04-22 12:40:58

Have you tried csync2? Perhaps in combination with lsyncd or incron?

Arch Linux

#1 2015-04-21 19:14:45

bidirectionally syncing 2 or more servers

#2 2015-04-21 20:19:27

Re: bidirectionally syncing 2 or more servers

#3 2015-04-22 11:15:41

Re: bidirectionally syncing 2 or more servers

#4 2015-04-22 11:24:41

Re: bidirectionally syncing 2 or more servers

#5 2015-04-22 12:40:58

Re: bidirectionally syncing 2 or more servers

Board footer