Arch Linux

SeagullFish

Member

Registered: 2023-08-10

Posts: 78

How to auto-sync to USB-storage on mount

Hello, all.
I have been researching for a while, to figure out how to accomplish three different objectives, in order to create a private backup system:

Objective 1: Find an application that (on my command) one-way synchronizes a source folder of my choice, to a target folder on an external USB-drive.
Objective 2: Figure out how to set up a list of several source folders and associated target folders, and make the application synchronize according to the list.
Objective 3: Make a predefined setup, so that the solution made in objective 2 executes automatically once the external USB-drive is mounted.

This i what I have figured out on my own, so far:

Objective 1:
I have been looking into both Syncthing and Resilio Sync. Unfortunately, none of them are quite suitable for my use case, because they both are designed to synchronize files between 2 or more devices, and not between 2 folders on the same device. Admittedly, in Syncthing’s own forum, it is mentioned that this can be achived by running two separate instances of Syncthing on the same device. However, it is also mentioned that this is not recommended. Moreover, from a note on the Arch Linux wikipage for Syncthing, I also interpret that such an approach may lead to high CPU consumption.

A third option is rclone. Even though it is mainly designed for synchronizing to cloud services, it seems that it could be used for local folders, like this:

rclone sync /path/to/source /path/to/target

However, rclone seems unable to synchronize symbolic links (according to this page):

• By default, symbolic links are ignored.

• By supplying --copy-links or -L, you can make rclone follow symbolic links, and synchronize the contents of the pointed files or directories. But I assume that this could also pose a danger if a symbolic link points to a folder within the same path as (but at a level above) the symbolic link, so that rclone will be synchronizing in an eternal loop.

• By supplying --links or -l, you can make rclone replace symbolic links (in the target folder) with textfiles with a ’.rclonelink’ suffix. But this doesn't seem like a failsafe solution to me, if there should ever be a need for restoring files from backup later.

• I haven’t found any other solution to this for rclone.

A fourth option could (possibly?) be rsync, like this:

rsync -a -P --delete /path/to/source /path/to/target

However, this web site states that:

rsync is a [...] command-line utility for synchronizing files and directories between two locations over a remote shell, or from/to a remote Rsync daemon. [...] Rsync can be used [...] as a replacement for scp , sftp , and cp commands.

My questions:

• Is rsync mainly designed for synchronizing to ftp rather than local folders?

• Does rsync require to run as a daemon rather than a user application?

• Do you have any comments or suggestions regarding my review?

Objective 2:
I’m not sure how to design this perfectly, but I imagine for example that I could write a bash shell script. I suppose it would be looking something like this:

#!/bin/sh

application [OPTION...] /path/to/source1 /path/to/target1
application [OPTION...] /path/to/source2 /path/to/target2
application [OPTION...] /path/to/source3 /path/to/target3
#And so on...

My question:
Will this be a feasible approach? For example: Is there a risk that the shell script may simply execute all listed commands and exit, without waiting for the commands to finish one by one?
(A prerequisite is that no more than one instance of the application should be running at a time.)

Objective 3:
At section 2.2.4 on the Arch Linux wikipage for Syncthing, it is mentioned that:

The Syncthing systemd user service can be started after a specific (optionally encrypted) device has been mounted, and stopped when the device has been unmounted.

My question:
Could the bash shell script, or alternatively rclone or rsync, be started the same way, after a specific device has been mounted? If yes, then how?

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,208

Re: How to auto-sync to USB-storage on mount

rsync will do all of this, ignore whatever blog post suggests otherwise. It can be used for local files and in that case definitely standalone (the server component is mostly interesting when actual dealing with remote hosts, as the daemon can more accurately determine which files are already present so that only the delta is passed over the wire) .

As for objective 2. If you list that as you mentioned. all commands will run in sequence one after the other. If you want to stop execution on error or so you'd have to check exit codes.

You can configure systemd services to listen to appearance of a device/setting up a mount point and then executing a standalone service running your script. See e.g https://wiki.archlinux.org/title/Udev#M … s_in_rules and the section below it for long running services, and you'd run your script as a service.

Also caveat that one might stumble over if you want to retain symlinks the file system on the target storage needs to support them.

Last edited by V1del (2024-09-19 13:44:27)

cryptearth

Member

Registered: 2024-02-03

Posts: 2,167

Re: How to auto-sync to USB-storage on mount

USB storage already sounds like a bad idea - I would use a network share hosted on a server with at least some sort of mirror or raid
and to prevent ransomware attacks only mount the share when syncing to it - don't make the mistake to have it always mounted