Arch Linux

JohnDeVries

Member

Registered: 2019-07-15

Posts: 14

rclone remote with full local mirror

In order to access various cloud storage services I set up some rclone remotes, which I mount with full caching. Unfortunately this does not really allow for offline access of the remotes' files.

What I would like to do is set up a local directory which will mirror the entire content of the mounted remote, so that offline access is possible. This local copy should update as soon as rclone signals a file change. Additionally, if changes are made to the local copy while offline they should be synchronised to the remote as soon as the system is back online.

Has anyone here managed to create a similar setup?

taumeister

Member

Registered: 2019-01-06

Posts: 31

Re: rclone remote with full local mirror

Hi,
you can only do that in my opinion if you write yourself something around 'rclone'.

The manual of 'rclone' says :

Rclone syncs a directory tree from one storage system to another.

You could write a script that looks for the folder in intervals and matches source and destination.
I created something like this for important documents that I digitize.

Multiple folders are synchronized to Onedrive every X minutes using password- and header-protected Zip.

I could make this available to you if you like.

JohnDeVries

Member

Registered: 2019-07-15

Posts: 14

Re: rclone remote with full local mirror

Yes please! I'd be quite curious to see how you made that.

I would prefer to setup more of an "event driven" system, rather than polling.
For the local side I know there are utilities that can watch folders for changes, etc. Building on top of that would solve the "upload local changes" part. I suppose some system to queue those up would be necessary to make it work offline as well.

As for the "sync online changes" part: is it even possible to "listen" to some service? Or must you poll?

Last edited by JohnDeVries (2019-09-20 18:37:23)

taumeister

Member

Registered: 2019-01-06

Posts: 31

Re: rclone remote with full local mirror

I think you want features from the program that just don't work.
As comprehensive as rclone is, I don't think it's meant to monitor folders or services.
It simply synchronizes from source to target, the rest you have to do yourself.

What my script does is this:

Take two folders and their subfolders and 7zip them with password and headerprotected each into monthly archives in the folder 'upload-to-onedrive'.
Keeps only the last 5 archives and deletes the rest.

e.g.

../upload-to-onedrive
2019.07._Skripte.7z
2019.07._Dokumente.7z

Each additional 7zip operation updates the monthly archive (you might run it every 5 minutes via cronjob, I do that weekly).

Then I update the backup file with rclone into my Onedrive folder.

After that I get a mail about it.

You have to have a close look at the script and adapt it for yourself.

If you need help, let me know.

You need sendEmail and 7zip
You have to adjust everything in the brackets, e.g. <your-path>...

#!/bin/bash

# start Dir, where the script starts
dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# where to backup t0
bcktarget=/<your-backup-path>

# format to 2019-03 , only want monthly archives
date=$(date +"%Y-%m.")

# function _make(), creates or updates the monthly archives

_make()
{
# create the new name
fullpath=$1
basename=$(basename "$1")
newname="$date-$basename.7z"

# create archive with password
echo "Compressing :$Folder"
7z -mhe=on a $bcktarget/$newname $fullpath -p<your-password>

echo "Deleting files older then 5"

#https://krausens-online.de/loeschen-der-letzten-x-dateienverzeichnisse-linux/
ls -td $bcktarget/* | grep $basename.7z | tail -n +5 | xargs rm -rf
} 


# send folders to function and let them compress 
for folder in '>folder1>' '<folder2>' ; do _make $folder ; done

# sync with rclone
echo "rclone sync in progress"
rclone sync <your-backup-path/> remote:<your-remote-path>

# send mail
echo 'Auto Upload to Microsoft Onedrive ' | \
sendEmail -f <from-your-mailadress> -t <to-your-mail-address> -u "Backup to Onedrive" \
-s <smtp-server:port> -xu <your-username> -xp <your-password> -m "Weekly Backup to Onedrive"