You are not logged in.

#1 2015-12-16 01:48:24

Kardell
Member
From: London a new Babylon
Registered: 2007-02-20
Posts: 225

Help with s3 backup script

Hi all,
   Please can anyone help with finding what is wrong with this script.
It use to work. I have marked line 73 within the code.

SOURCE=(
"/home/john/Backups"
)

IFS=$(echo -en "\n\b")
logFile=/home/log.log
bucket=s3bucket
s3cfg=/home/config.conf
touch $logFile
echo Finding files and performing backup: Please wait...
 
# for loop to go through each item of array SOURCE which should contain the
# directories to be backed up
 
for i in "${SOURCE[@]}"
do
 
# nested for loop to run find command on each directory in SOURCE
 
for x in `find $i`
do
# x is each file or dir found by 'find'. if statement determines if it is a regular file
 
if [ -f "$x" ]
then
# create a hash to mark the time and date of the file being backed up to compare later for
# incremental backups
 
fileSize=`stat -c %s $x`
modTime=`stat -c %Y $x`
myHash=`echo $x $fileSize $modTime | sha1sum`
 
# If statement to see if the hash is found in log, meaning it is already backed up.
# If not found proceed to backup
 
if ! grep -q $myHash $logFile
then
echo Currently uploading $x
 
# s3cmd command to put an encrypted file in the s3 bucket
# s3out var should capture anything in stderr in case of file transfer error or some other
# problem. If s3out is blank, the transfer occurred without incident. if an error occurs
# no output is written to the log file but output is written to an error log and s3out is
# written to the screen.
 
s3out=$(s3cmd -c $s3cfg -e put $x s3://$bucket 2>&1 > /dev/null)

if [ "$s3out" = "" ]
then
echo $x :///: $fileSize :///: $modTime :///: $myHash >> $logFile
else
# s3out had content, but was possibly a warning and not an error.. Checking to see if
# there exist an upload file within the last 2 minutes. If so, the file will be considered
# uploaded. Two minutes is to account for variance between local and remote time signatures.
 
date1=$(date --date="$(s3cmd -c $s3cfg ls s3://$bucket | awk '{print $1 " " $2 " +0000"}')" +%s)
date2=$(date +%s)
 
#line 73 below
datediff=$($date2-$date1)
 
if [[ $datediff -ge -1200 ]] && [[ $datediff -le 1200 ]]
then
echo There was a possible error but the time of the uploaded file was written within
echo the last 20 minutes. File will be considered uploaded and recorded as such.
echo $x :///: $fileSize :///: $modTime :///: $myHash >> $logFile
echo `date`: $x had warnings but seemed to be successfully uploaded and was logged to main log file >> $logFile.err
else
echo $s3out
echo `date`: $s3out >> $logFile.err
fi
echo ------------------------------------------------------------------------------------
fi
fi
fi

done
done

I get this error and the file is not uploaded. This is happening just to some of the files not all.


Currently uploading /home/john/Backups/151214-file.tar.gz.00
date: invalid date 'DIR s3://bucket// +0000\n2015-08-24 00:04 +0000\n2015-08-24 00:07 +0000\n2015-08-23 21:40 +0000\n2015-08-24 00:06 +0000\n2015-08-24 00:04 +0000\n2015-08-23 22:42 +0000\n2015-08-24 00:04 +0000\n2015-08-24 00:04 +0000\n2015-08-26 23:30 +0000\n2015-10-12 01:15 +0000\n2015-10-19 12:57 +0000\n2015-10-19 11:52 +0000\n2015-10-19 11:52 +0000\n2015-10-19 12:32 +0000\n2015-10-19 11:54 +0000\n2015-10-19 12:26 +0000\n2015-10-19 11:57 +0000\n2015-09-11 22:08 +0000\n2015-10-19 11:38 +0000\n2015-10-19 13:00 +0000\n2015-10-19 11:37 +0000\n2015-10-19 12:40 +0000\n2015-10-19 12:06 +0000\n2015-10-19 11:41 +0000\n2015-10-19 12:57 +0000\n2015-10-19 12:06 +0000\n2015-09-29 22:06 +0000\n2015-09-29 00:55 +0000\n2015-10-19 13:01 +0000\n2015-10-19 11:54 +0000\n2015-10-19 13:23 +0000\n2015-10-19 12:49 +0000\n2015-10-19 13:06 +0000\n2015-10-19 11:58 +0000\n2015-10-19 12:57 +0000\n2015-10-19 12:45 +0000\n2015-10-19 12:35 +0000\n2015-10-19 13:32 +0000\n2015-10-19 13:03 +0000\n2015-09-11 22:08 +0000\n2015-10-19 13:00 +0000\n2015-10-19 13:00 +0000\n2015-10-19 11:43 +0000\n2015-10-19 13:27 +0000\n2015-10-19 13:23 +0000\n2015-10-19 12:53 +0000\n2015-10-19 13:10 +0000\n2015-10-19 12:01 +0000\n2015-10-19 11:54 +0000\n2015-10-19 11:50 +0000\n2015-10-19 12:43 +0000\n2015-10-19 12:34 +0000\n2015-10-19 13:00 +0000\n2015-10-19 11:46 +0000\n2015-10-19 11:38 +0000\n2015-10-19 13:06 +0000\n2015-10-19 22:23 +0000\n2015-10-19 22:22 +0000\n2015-10-19 22:23 +0000\n2015-10-19 22:23 +0000\n2015-10-19 22:22 +0000\n2015-10-26 21:05 +0000\n2015-10-27 01:36 +0000\n2015-10-27 04:07 +0000\n2015-10-27 00:47 +0000\n2015-10-27 01:54 +0000\n2015-10-27 02:14 +0000\n2015-10-27 07:21 +0000\n2015-10-27 08:27 +0000\n2015-10-27 04:13 +0000\n2015-10-27 07:40 +0000\n2015-10-27 02:53 +0000\n2015-10-27 19:24 +0000\n2015-10-27 05:09 +0000\n2015-10-27 01:13 +0000\n2015-10-27 04:51 +0000\n2015-10-27 06:20 +0000\n2015-10-27 06:05 +0000\n2015-10-27 06:31 +0000\n2015-10-27 03:58 +0000\n2015-10-27 01:44 +0000\n2015-10-27 01:55 +0000\n2015-10-27 06:41 +0000\n2015-10-27 02:23 +0000\n2015-10-27 08:04 +0000\n2015-10-27 05:34 +0000\n2015-10-27 06:51 +0000\n2015-10-27 07:30 +0000\n2015-10-27 05:55 +0000\n2015-10-27 08:14 +0000\n2015-10-27 04:25 +0000\n2015-10-27 04:34 +0000\n2015-10-27 02:05 +0000\n2015-10-27 05:00 +0000\n2015-10-27 04:15 +0000\n2015-10-27 04:12 +0000\n2015-10-27 00:59 +0000\n2015-10-27 03:39 +0000\n2015-10-27 04:41 +0000\n2015-10-27 05:44 +0000\n2015-10-27 04:15 +0000\n2015-10-27 04:21 +0000\n2015-10-27 07:50 +0000\n2015-10-27 00:48 +0000\n2015-10-27 01:24 +0000\n2015-10-27 02:37 +0000\n2015-10-27 07:11 +0000\n2015-10-27 05:20 +0000\n2015-10-27 05:44 +0000\n2015-10-27 02:32 +0000\n2015-10-27 19:14 +0000\n2015-10-27 03:03 +0000\n2015-10-27 03:18 +0000\n2015-10-27 18:58 +0000\n2015-10-27 05:25 +0000\n2015-10-27 03:54 +0000\n2015-10-27 01:36 +0000\n2015-10-27 18:57 +0000\n2015-10-27 05:50 +0000\n2015-10-27 03:33 +0000\n2015-10-27 03:28 +0000\n2015-10-27 03:49 +0000\n2015-10-27 19:09 +0000\n2015-10-27 02:48 +0000\n2015-10-27 07:01 +0000\n2015-10-27 19:07 +0000\n2015-10-27 18:51 +0000\n2015-10-27 05:00 +0000\n2015-10-27 08:24 +0000\n2015-10-27 06:19 +0000\n2015-10-27 19:09 +0000\n2015-10-27 05:05 +0000\n2015-10-27 05:44 +0000\n2015-10-27 06:15 +0000\n2015-10-27 07:59 +0000\n2015-10-27 01:38 +0000\n2015-10-27 07:06 +0000\n2015-10-27 01:17 +0000\n2015-11-09 21:35 +0000\n2015-11-09 21:32 +0000\n2015-11-09 21:35 +0000\n2015-11-09 21:32 +0000\n2015-11-09 21:33 +0000\n2015-11-22 19:52 +0000\n2015-11-22 20:07 +0000\n2015-11-22 19:47 +0000\n2015-11-22 20:17 +0000\n2015-11-22 20:22 +0000\n2015-11-22 19:53 +0000\n2015-11-22 19:58 +0000\n2015-11-22 20:08 +0000\n2015-11-22 20:12 +0000\n2015-11-22 19:46 +0000\n2015-11-22 20:03 +0000\n2015-11-22 19:52 +0000\n2015-11-22 20:07 +0000\n2015-11-22 20:22 +0000\n2015-04-27 00:01 +0000\n2015-11-18 17:16 +0000\n2015-11-18 17:26 +0000\n2015-11-18 17:34 +0000\n2015-11-18 17:06 +0000\n2015-11-18 16:52 +0000\n2015-11-18 17:16 +0000\n2015-11-23 20:23 +0000\n2015-11-24 01:15 +0000\n2015-11-23 00:46 +0000\n2015-11-23 20:12 +0000\n2015-11-23 02:11 +0000\n2015-11-23 23:46 +0000\n2015-11-22 23:54 +0000\n2015-11-23 21:26 +0000\n2015-11-23 02:59 +0000\n2015-11-23 01:10 +0000\n2015-11-23 19:34 +0000\n2015-11-23 11:06 +0000\n2015-11-23 00:21 +0000\n2015-11-23 01:29 +0000\n2015-11-23 20:33 +0000\n2015-11-23 22:50 +0000\n2015-11-23 01:48 +0000\n2015-11-23 03:38 +0000\n2015-11-23 22:06 +0000\n2015-11-23 02:20 +0000\n2015-11-23 19:14 +0000\n2015-11-23 20:03 +0000\n2015-11-23 21:39 +0000\n2015-11-24 00:56 +0000\n2015-11-23 03:47 +0000\n2015-11-23 01:43 +0000\n2015-11-24 01:21 +0000\n2015-11-23 19:58 +0000\n2015-11-23 20:23 +0000\n2015-11-24 01:15 +0000\n2015-11-23 20:23 +0000\n2015-11-23 00:56 +0000\n2015-11-24 00:13 +0000\n2015-11-23 01:58 +0000\n2015-11-23 10:51 +0000\n2015-11-23 02:39 +0000\n2015-11-23 01:19 +0000\n2015-11-24 00:40 +0000\n2015-11-23 03:28 +0000\n2015-11-23 02:07 +0000\n2015-11-23 03:47 +0000\n2015-11-23 19:23 +0000\n2015-11-23 03:08 +0000\n2015-11-23 20:59 +0000\n2015-11-23 02:49 +0000\n2015-11-24 01:06 +0000\n2015-11-23 19:53 +0000\n2015-11-23 21:32 +0000\n2015-11-23 01:09 +0000\n2015-11-23 01:57 +0000\n2015-11-23 01:01 +0000\n2015-11-23 00:47 +0000\n2015-11-23 03:57 +0000\n2015-11-23 02:30 +0000\n2015-11-23 23:18 +0000\n2015-11-23 19:44 +0000\n2015-11-23 20:13 +0000\n2015-11-23 03:18 +0000\n2015-11-23 00:42 +0000\n2015-11-23 10:41 +0000\n2015-11-23 04:04 +0000\n2015-11-23 01:39 +0000\n2015-11-23 11:03 +0000\n2015-12-14 01:21 +0000\n2015-12-14 01:21 +0000\n2015-12-14 01:21 +0000\n2015-12-16 01:29 +0000\n2015-12-14 01:21 +0000\n2015-12-14 01:21 +0000\n2015-12-14 01:21 +0000\n2015-10-19 12:07 +0000\n2015-10-19 12:26 +0000\n2015-10-19 13:14 +0000\n2015-08-26 23:06 +0000\n2015-03-23 02:47 +0000\n2015-03-23 00:36 +0000\n2015-03-23 00:00 +0000\n2015-03-23 01:39 +0000\n2015-03-23 01:13 +0000\n2015-03-22 23:34 +0000\n2015-10-19 11:43 +0000\n2015-10-19 22:22 +0000\n2015-10-27 05:00 +0000\n2015-11-09 21:33 +0000\n2015-11-22 20:07 +0000\n2015-12-16 01:29 +0000\n2015-12-14 01:16 +0000\n2015-11-18 16:57 +0000'
./s3incbkp.sh:73: command not found: 1450229504-

Thanks


"Those who don't know history are doomed to repeat it." Edmund Burke

Offline

#2 2015-12-16 20:22:44

nixpunk
Member
Registered: 2009-11-23
Posts: 271

Re: Help with s3 backup script

For line 73, assuming you are looking to do arithmetic, use:

$(($date2-$date1))

not

$($date2-$date1)

Offline

#3 2015-12-17 21:38:22

Kardell
Member
From: London a new Babylon
Registered: 2007-02-20
Posts: 225

Re: Help with s3 backup script

Hi nixpunk
  Thank you for suggestion, I have tried what you have suggested, but this time the error is

73:bad math expression: operand expected at end of string

I forgot to mention that this is on ZSH not BASH shell.

I think this has something to do with date comparison of the stuff that is already in the bucket as you can see it is throwing some modification dates of these files.

At the moment it is happening to some of the files, minority of the files, but all of them are tar.gz split archives.

Cheers

Last edited by Kardell (2015-12-17 21:49:31)


"Those who don't know history are doomed to repeat it." Edmund Burke

Offline

#4 2015-12-18 15:43:59

nixpunk
Member
Registered: 2009-11-23
Posts: 271

Re: Help with s3 backup script

The method I posted is POSIX compliant, so it should work with either shell. 

It looks like the error is due to date1 being empty.  To troubleshoot the issue further, I would print the name of each file before line 73 in order to determine when it is failing.

Offline

#5 2015-12-21 02:44:53

Kardell
Member
From: London a new Babylon
Registered: 2007-02-20
Posts: 225

Re: Help with s3 backup script

Hi again,
    I have disabled this date checking and this is what s3cmd is throwing now.

WARNING: Upload failed: /file.tar.gz.00 ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
WARNING: Upload failed: /file.tar.gz.00 ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
WARNING: Upload failed: /file.tar.gz.00 ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
WARNING: Upload failed: /file.tar.gz.00 ([Errno 32] Broken pipe)

Most topics about this error relate to multipart which is actually enabled in the config
Besides this file is just 7.5MB in size
I have to research more on this.
Best regards


"Those who don't know history are doomed to repeat it." Edmund Burke

Offline

Board footer

Powered by FluxBB