Arch Linux

mouseman

Member

From: Outta nowhere

Registered: 2014-04-04

Posts: 291

[solved] bash script for syncing local repository

Butchered up the script sample from the devwiki:

#!/bin/bash

#if (( $EUID != 0 )); then
#    echo "I work better when run as root."
#    exit 1
#fi

base="/mnt/data"
home="${base}/arch"
target="${home}/repo"
tmp="${base}/tmp"
lock="${base}/tmp/sync-local-repo.lck"
bwlimit=4096 # To use bandwidth limit, add --bwlimit=$bwlimit to rsync below

source='rsync://ftp.nluug.nl/archlinux/'
lastupdate_url="http://ftp.nluug.nl/os/Linux/distr/archlinux/lastupdate"

exec 9>"${lock}"
flock -n 9 || exit 1

# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && diff -b <(curl -s "$lastupdate_url") "$target/lastupdate" >/dev/null; then
        exit 0
elif ! stty &>/dev/null; then
    quiet="-q"
fi

sudo rsync -rtlvH --safe-links --delete-after --progress -h ${quiet} --timeout=600 --contimeout=60 -p --delay-updates --no-motd --temp-dir="${tmp}" ${source} "${target}"

#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"

It works fine but I have a few questions.

This part doesn't seem> to work correctly:

exec 9>"${lock}"
flock -n 9 || exit 1

I think that because it creates the file, but doesn't remove it. Also, it doesn't seem to matter if the file exists, the script still runs. I do a 'watch -n1 ls -al' on the tmp dir and I can see the timestamp change on the file when I start the script manually. It seems to simply reuse the file. I expected it to quit. So I tested it by making a script with 'exec 9>tmp.lck' followed by a sleep to keep the script running. From another terminal I can simply remove the lock file.  So I'm pretty sure I don't understand how this is supposed to work .

Second question is about systemctl status on the sync-service. It doesn't log when it is finished and doesn't seem to show it as active while the script runs. Not sure how I can check for sure, but when I start it manually with systemctl start sync-local-repo.service and immediately follow it up with a status check, it shows as inactive since 1s ago. Code=exited, status=0/SUCCESS. I doubt it runs so fast as the rsync needs to check the source against my local files. When I run the rsync command straight from the terminal it runs at least 3 or 4 seconds, it never finishes in 1 second. - journalctl does show what I am looking for.

As always, thanks for the help guys!

Last edited by mouseman (2016-01-10 09:39:16)

Saint0fCloud

Member

Registered: 2009-03-31

Posts: 137

Re: [solved] bash script for syncing local repository

`flock` simply allows you (un)set file locks, it won't fail if the file already exists. If you want a file that will cause your script to fail if it exists then you'll want to do a conditional test.

if [[ -f <lock_file> ]]; then
    exit 1
fi

Check out the man page for `flock` as well as this page, http://www.gnu.org/software/bash/manual … sions.html, on conditional expressions for bash.

mouseman

Member

From: Outta nowhere

Registered: 2014-04-04

Posts: 291

Re: [solved] bash script for syncing local repository

Thanks for the reply.

I was confused over how those two lines worked but I just figured it out. I thought the 'exec 9>file' also set the lock and that flock only tested it. So when I tried to test it with two scripts, one containing only the exec line followed by a sleep and the other with the 'flock -n 9 || exit 1'  followed by a sleep, it failed.

I see how it works now though.

What I want to do next is search for a few fast and recent mirrors, say once a day, dump it to a file and let the above script pick a random one from it to sync with each run. That way I won't be syncing against the same one all the time (not asking how to do this, just saying. I'll figure it out ).