get, put, rget, and rput scripts

Buce · 2009-09-21 09:46:23

After getting some help on a script in this forum recently, I figured I'd post a few scripts I've been using for a while now. I'm guessing they could still use some optimization, but I suspect they'll be useful for some people.

I originally wrote these to learn BASH scripting, mostly, and also because I can never remember all the fine points of rsync, whose manpage is a beast to read. I could never remember which argument a trailing / mattered for, or what the trailing slash did, for example. Exclude rules were always a pain. Etc, etc.

The scripts are 'get', 'put', 'rget', and 'rput'. They use a configuration file to refer to remote computers, so that you can type 'hostalias' instead of 'user@host.com' or 'user@ipaddress'. The config file also specifies a default directory for files to be uploaded to.

EDIT: Oh, and a word of warning about rget and rput -- Since they recursively copy (and possibly overwrite or remove) data, you should be super careful when you use them. I generally only use them in aliases/functions/scripts that I've tested enough to be confident about, and only rarely from an interactive shell.

##### SYNTAX #####

get <list of files> from <host> # retrieves the list of files from remote host
get <list of files> from <host> to <dir> # does the same, but puts the retrieved files in <dir> instead of the current working directory
put <list of files> on <host> # uploads list of files to the remote host, placing them in the default directory
put <list of files> on <host> in <dir> # same, but puts the files in <dir> instead of the default directory
rget <dir> from <host> # retrieves a whole directory structure (or multiple directory structures, if <dir> is a list of directories) from remote host
rget <dir> from <host> to <localdir> # same, but puts the directories in <localdir>
rput <dir> on <host> # I think you get the idea by now
rput <list of dirs> on <host> in <dir> # etc

##### get #####

#!/bin/bash
# Download files from remote computer
# Requires rsync

# Syntax: get file1 file2 file3 from host [to dir]

# init vars
FILES=
HOST=
DEST=
CONFIG="${HOME}/.usr/etc/hosts.conf"
OPTS="-vz"
SWITCH="from"
OPTSWITCH="to"
ARG_CHECK=`perl -e "print index('$*', $SWITCH)"`
REG_EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}'
REG_IP='[A-Za-z0-9._%+-]+@([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}'

# check arguments
if [ "$ARG_CHECK" = -1 ]
then
    echo "Usage: `basename $0` <file1> <file2> <file3> ... <fileN> $SWITCH <host> [$OPTSWITCH <dir>]"
    exit 1
fi

# make list of files to download
while [ "$1" != "$SWITCH" ]
do
    FILES="${FILES} $1"
    shift
done

shift

# retrieve proper host from config file
HOST=`egrep "^[[:space:]]*$1[[:space:]]+($REG_EMAIL|$REG_IP)" \
     $CONFIG | sed -r 's/^\s*\w*\s+//' | sed -r 's/:.*$/:/'`
# make sure host exists
if [ -z $HOST ]
then
    echo "`basename $0`: invalid host name \`$1', is $CONFIG configured?"
    exit 1
fi

# if optional destination directory is specified, use it
shift
DEST=`pwd`
if [ "$1" = "$OPTSWITCH" ]
then
    DEST="$2"
fi

# internetz away!
rsync ${OPTS} ${HOST}"${FILES}" ${DEST}

exit 0

##### put #####

#!/bin/bash
# Upload files to remote computer
# Requires rsync

# Syntax: put file1 file2 file3 on host [in dir]

# init vars
FILES=
HOST=
DEST=
CONFIG="${HOME}/.usr/etc/hosts.conf"
OPTS="-vz"
SWITCH="on"
OPTSWITCH="in"
ARG_CHECK=`perl -e "print index('$*', $SWITCH)"`
REG_EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}'
REG_IP='[A-Za-z0-9._%+-]+@([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}'

# check arguments
if [ "$ARG_CHECK" = -1 ]
then
    echo "Usage: `basename $0` <file1> <file2> <file3> ... <fileN> $SWITCH <host> [$OPTSWITCH <dir>]"
    exit 1
fi

# make list of files to upload
while [ "$1" != "$SWITCH" ]
do
    FILES="${FILES} $1"
    shift
done

shift

# retrieve proper host from config file
HOST=`egrep "^[[:space:]]*$1[[:space:]]+($REG_EMAIL|$REG_IP)" \
     $CONFIG | sed -r 's/^\s*\w*\s+//'`
# make sure host exists
if [ -z $HOST ]
then
    echo "`basename $0`: invalid host name \`$1', is $CONFIG configured?"
    exit 1
fi

# if optional destination directory is specified, use it
shift
if [ "$1" = "$OPTSWITCH" ]
then
    HOST=`echo $HOST | sed -r 's/:.*$/:/'`
    DEST="$2"
fi

# internetz away!
rsync -vz ${FILES} ${HOST}${DEST}

exit 0

##### rget #####

#!/bin/bash
# Download dirs and their contents from remote computer
# Requires rsync

# Syntax: rget dir1 dir2 dir3 from host [to dir]

# init vars
DIRS=
HOST=
DEST=
CONFIG="${HOME}/.usr/etc/hosts.conf"
EXCLUDE_FILE="${HOME}/.usr/etc/exclude.sync.conf"
OPTS="-avz --delete-after --exclude-from ${EXCLUDE_FILE}"
SWITCH="from"
OPTSWITCH="to"
ARG_CHECK=`perl -e "print index('$*', $SWITCH)"`
REG_EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}'
REG_IP='[A-Za-z0-9._%+-]+@([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}'

# check arguments
if [ "$ARG_CHECK" = -1 ]
then
    echo "Usage: `basename $0` <dir1> <dir2> <dir3> ... <dirN> $SWITCH <host> [$OPTSWITCH <dir>]"
    exit 1
fi

# make list of dirs to download
while [ "$1" != "$SWITCH" ]
do
    # remove trailing slash
    DIRS="${DIRS} `dirname $1`/`basename $1`"
    shift
done

shift

# retrieve proper host from config dir
HOST=`egrep "^[[:space:]]*$1[[:space:]]+($REG_EMAIL|$REG_IP)" \
     $CONFIG | sed -r 's/^\s*\w*\s+//' | sed -r 's/:.*$/:/'`
# make sure host exists
if [ -z $HOST ]
then
    echo "`basename $0`: invalid host name \`$1', is $CONFIG configured?"
    exit 1
fi

# if optional destination directory is specified, use it
shift
DEST=`pwd`
if [ "$1" = "$OPTSWITCH" ]
then
    DEST="$2"
fi

# internetz away!
rsync ${OPTS} ${HOST}"${DIRS}" ${DEST}

exit 0

##### rput #####

#!/bin/bash
# Upload dirs and their contents to remote computer
# Requires rsync

# Syntax: rput dir1 dir2 dir3 on host [in dir]

# init vars
DIRS=
HOST=
DEST=
CONFIG="${HOME}/.usr/etc/hosts.conf"
EXCLUDE_FILE="${HOME}/.usr/etc/exclude.sync.conf"
OPTS="-avz --delete-after --exclude-from ${EXCLUDE_FILE}"
SWITCH="on"
OPTSWITCH="in"
ARG_CHECK=`perl -e "print index('$*', $SWITCH)"`
REG_EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}'
REG_IP='[A-Za-z0-9._%+-]+@([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}'

# check arguments
if [ "$ARG_CHECK" = -1 ]
then
    echo "Usage: `basename $0` <dir1> <dir2> <dir3> ... <dirN> $SWITCH <host> [$OPTSWITCH <dir>]"
    exit 1
fi

# make list of dirs to upload
while [ "$1" != "$SWITCH" ]
do
    # remove trailing slash
    DIRS="${DIRS} `dirname $1`/`basename $1`"
    shift
done

shift

# retrieve proper host from config dir
HOST=`egrep "^[[:space:]]*$1[[:space:]]+($REG_EMAIL|$REG_IP)" \
     $CONFIG | sed -r 's/^\s*\w*\s+//'`
# make sure host exists
if [ -z $HOST ]
then
    echo "`basename $0`: invalid host name \`$1', is $CONFIG configured?"
    exit 1
fi

# if optional destination directory is specified, use it
shift
if [ "$1" = "$OPTSWITCH" ]
then
    HOST=`echo $HOST | sed -r 's/:.*$/:/'`
    DEST="$2"
fi

# internetz away!
rsync ${OPTS} ${DIRS} ${HOST}${DEST}

exit 0

##### PARAMETERS YOU'LL WANT TO CHANGE #####

This declaration is in all of the scripts:

CONFIG="${HOME}/.usr/etc/hosts.conf"

I have a convoluted scheme for the way my home directory is set up, so the weird path has nothing to do with the scripts, you can set it to whatever you want. Your hosts.conf (or whatever you want to call it) should look like this:

alias1      user@myhost.com:tmp
alias2   user@255.255.255.255:tmp
alias3        user@myotherhost.com:www

If this was your hosts.conf, then, for example, 'get <file> from alias1' would try to retrieve the file from user@bwulf.myhost.com. The ':tmp' and ':www' bits at the end only matter for put and rput -- they're the default directories that you'll find the files in after using those scripts. Since rsync (for me at least) ends up in the home directory when connecting to a remote computer, the ones here correspond to '~user/tmp' and ~user/www'. Substituting 'tmp' for '/home/user/tmp' should also work, and '~/tmp' or '${HOME}/tmp' might work too, but I haven't tested this and there might be conflicts between ${HOME} on your local machine and ${HOME} on the remote machine.

There's also these two lines in rget and rput:

EXCLUDE_FILE="${HOME}/.usr/etc/exclude.sync.conf"
OPTS="-avz --delete-after --exclude-from ${EXCLUDE_FILE}"

If you've used rsync for backups before, this shouldn't be too cryptic, but I just wanted to point it out so that you can change EXCLUDE_FILE's path to a file that actually exists on your computer. I'll leave you to google/manpage exclude rules for rsync, but an abridged version of my exclude.sync.conf is below for reference. Also, you can also edit OPTS to suit your purposes (this variable is in get and put, too).

My exclude.sync.conf:

*cache*
*Cache*
*CACHE*
etc/dot/mozilla
etc/dot/.purple
ns.back
ns

That's about it, I think. Comments, suggestions, and constructive criticism are welcome. One deficiency I'm semi-aware of -- I'm guessing the way I use sed and perl to handle the config file could use some optimization, since my knowledge of text-processing utilities is pretty sparse.

Last edited by Buce (2009-09-21 10:07:58)

scragar · 2009-09-21 13:45:36

Buce, the : is often used to specify a port number, you might want to choose a different symbol for a directory, maybe / or even | (Personally I think another space might be the best idea)

Last edited by scragar (2009-09-22 00:39:34)

Buce · 2009-09-21 14:04:37

But in rsync, what comes after the colon is the directory. ' rsync -avz file.mp3 user@host.com:music/ ' is a perfectly normal rsync command, so the : shouldn't throw too many people off -- should it? The port is set by the remote shell that rsync uses, not rsync itself. At least, I think that's how it works.

Also, not that it matters much to me, but there's no 'r' in 'Buce'.

Arch Linux

#1 2009-09-21 09:46:23

get, put, rget, and rput scripts

#2 2009-09-21 13:45:36

Re: get, put, rget, and rput scripts

#3 2009-09-21 14:04:37

Re: get, put, rget, and rput scripts

Board footer