esync.sh - easy directory sync'ing

Doomcide · 2011-08-30 12:04:02

A while ago I wrote a little bash script, to synchronize two directories. I had a lot of freetime and wanted to improve my bash-knowledge, also rsync had a lot of stuff I didn't need and was, in my case, slower than the script. (The tests take longer, but if you make changes to large files, rsync is a lot faster) So I thought I might as well share it here:

#!/bin/bash

#get script name
script=`basename $0`

#define quiet as `nž
quiet=n

#define usage
usage ()
{
cat  << EOF
usage: $script [options] DIR1 DIR2 [LISTDIR]

This script synchronizes DIR2 with DIR1.

options:
	-h	Show this message
	-q	Quiet (Exeption: If -y or -n are not given, ask before removing extraneous files in DIR2)
	-c	Create DIR2 if it doesn't exist
	-y	Don't ask before removing extraneous files in DIR2, always remove them
	-n	Don't ask before removing extraneous files in DIR2, never remove them

DIR1: 	 Source-directory (e.g.: /home/user/)
DIR2: 	 Destination-directory / directory to synchronize with DIR1 (e.g.: /media/windows/backup/)
LISTDIR: Directory where the lists will be stored. If missing, lists will be stored in: $HOME/.sync-list-tmp/1/1-1 and removed afterwards
EOF
}

#get options
while getopts ":ynqch" OPTION ${wambo[@]}  ; do 
case $OPTION in 
		y)
		   rmfiles="y"
		   ;;
		n) 
		   rmfiles="n"		
		   ;;
		q) 
		   quiet="y"
		   ;;
		c)
		   create="y"
		   ;;
		h)
		    usage
		    exit 1	
		    ;;
		?)
		   usage
		   exit
		   ;;
esac;  
done


#reset `$@ž and `$#ž
shift $(($OPTIND-1))


#check positional parameters
if [ ! $# == 2 ] && [ ! $# == 3 ]; then
	if [ ! "$quiet" = "y" ]; then
		usage
	fi
	exit

else
	
	#set directories & the date etc.
	
	number=1
	
	dir1="$1"   

	dir2="$2"
	
	if [ ! -z $3 ]; then

		dat=`date +%y%m%d`
	
		hour=`date +%H%M`
	
		bs=`basename $dir1`
	
		listdir="$3/$dat"
	
		rmld="n"

	else 
		
		dat=1
		
		hour=1
		
		bs=1
		
		listdir="/$HOME/.sync-list-tmp/"
		
		rmld="y" #remove listdir afterwards
	
	fi
fi



#check if destination-directory doesn't exists & if -c , if both is true create it
if [ ! -d $dir2 ]; then 
	if [ "$create" = "y" ]; then 
		mkdir -p $dir2
	else
		echo "ERROR - destination-directory: $dir2 was not found & -c flag is not set."
		echo "Quitting..."
		exit
	fi
fi

if [ ! -d $dir1 ]; then
	if [ ! "$quiet" = "y" ]; then
		echo "ERROR - source-directory: $dir1 was not found"
		echo "Quitting..."
	fi
	exit
fi	

#copy new(er) files to $dir2
if [ ! "$quiet" = "y" ]; then
	echo "Starting to synchronize $dir2 with $dir1..."
fi
cd $dir1
cp -ru * $dir2

if [ -a .??* ]; then
	cp -ru .??* $dir2   #copy hidden files
fi

if [ ! "$quiet" = "y" ]; then
	echo "Finished Copying - Searching for differences..."
fi


#create dirlists & log-directorys 

#create listdir if it doesn't already exist
if [ ! -d $listdir ]; then
	mkdir -p $listdir
fi

#create sub-directory in listdir for this execution
while [ -d $listdir/$bs-$hour-$number ]; do
	number=$((number+1))
done	

mkdir $listdir/$bs-$hour-$number

#create dirlists of dir1 and dir2 

cd $dir1
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist1

cd $dir2
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2




#compare files & create list of unique files in dir2

cd $listdir/$bs-$hour-$number

cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2 >> $listdir/$bs-$hour-$number/dirlistall

sort $listdir/$bs-$hour-$number/dirlistall | uniq -u > $listdir/$bs-$hour-$number/dirlistrm-tmp

cd $dir2

while read line;
do
	if [[ -a $line ]]; then
		echo $line >> $listdir/$bs-$hour-$number/dirlistrm
	fi
done < $listdir/$bs-$hour-$number/dirlistrm-tmp

if [[ -f $listdir/$bs-$hour-$number/dirlistrm-tmp ]]; then
	rm $listdir/$bs-$hour-$number/dirlistrm-tmp
fi




#ask for permission to rm unique files in dir2 & do it if permission is given

cd $dir2

if [[ -s $listdir/$bs-$hour-$number/dirlistrm ]]; then
	
	if [ ! "$quiet" = "y" ]; then
		cat $listdir/$bs-$hour-$number/dirlistrm
	fi
	if [ -z $rmfiles ]; then
			echo "Delete the files listed above, in $dir2? (They don't exist in $dir1.)"
			echo "Type "y" for yes or "n" or no!"
			read answer
	fi	
	if [ "$answer" == "y" ] || [ "$rmfiles" == "y" ]; then

		if [ ! "$quiet" = "y" ]; then
			echo "Deleting files..."
		fi	
			cd $dir2
			
			while read line; 
				do rm -rf "$dir2/$line"; 
			done < $listdir/$bs-$hour-$number/dirlistrm
			
		if [ $? == 0 ]; then 
				
				if [ ! "$quiet" = "y" ]; then
					echo "Files deleted"
				fi
		else
				if [ ! "$quiet" = "y" ]; then
					echo "ERROR - See the list in $listdir/$bs-$hour-$number for more information."
					echo "Saving list..."
				fi
		fi
	
	elif [ "$answer" == "n" ] || [ "$rmfiles" = "n" ]; then
		if [ ! "$quiet" = "y" ]; then
			echo "Files not deleted - Saving list..."
		fi
			cd $dir2
			find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2-after
			
			
	
			cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2-after >> $listdir/$bs-$hour-$number/dirlistall-after
			
			sort $listdir/$bs-$hour-$number/dirlistall-after | uniq -u > $listdir/$bs-$hour-$number/check

			
		if [ ! "$quiet" = "y" ]; then
			echo "Quitting..."
		fi
		
		#remove list if listdir is not given
		if [ $rmld = "y" ]; then
			rm -rf $listdir
		fi
		exit	

	else
		if [ ! "$quiet" = "y" ]; then
			echo "Incorrect Input - Quitting..."
		fi

		#remove list if listdir is not given
		if [ $rmld = "y" ]; then
			rm -rf $listdir
		fi
		exit

	fi
else 
	if [ ! "$quiet" = "y" ]; then
		echo "No differences found!"
	fi
fi


		

#create dirlist of dir2 after rm and cp & check if contents  of dir1 and dir2 are identical


cd $dir2
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2-after



cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2-after >> $listdir/$bs-$hour-$number/dirlistall-after

sort $listdir/$bs-$hour-$number/dirlistall-after | uniq -u > $listdir/$bs-$hour-$number/check


#remove the listdir entries from the check-log & determine dir of unique files


cd $dir2

#create list of files, only existing in $dir2

if [ -s $listdir/$bs-$hour-$number/check ]; then
	while read line; do  
		if [[ -f $line ]]; then
			echo $line >> $listdir/$bs-$hour-$number/check-dir2
		fi
	done < $listdir/$bs-$hour-$number/check


	
	cd $dir1

	#create list of files, only existing in $dir1
	while read line; do 
		if [[ -f $line ]]; then
			echo $line >> $listdir/$bs-$hour-$number/check-dir1-tmp
		fi
	done < $listdir/$bs-$hour-$number/check

	

	#go to $listdir
	cd $listdir/$bs-$hour-$number/

	#remove obsolete "check" log & clean up
	rm check  

	if [[ -s check-dir1 ]]; then
		while read line; do
			echo $line | grep $listdir/$bs-$hour-$number >> $listdir/$bs-$hour-$number/log-files-check
		done < $listdir/$bs-$hour-$number/check-dir1
    
		if [[ -s log-files-check ]]; then
			cat check-dir1-tmp log-files-check >> $listdir/$bs-$hour-$number/check-dir1-tmp2
			sort check-dir1-tmp2 | uniq -u > $listdir/$bs-$hour-$number/check-dir1
			rm check-dir1-tmp check-dir1-tmp2
		else 
			mv check-dir1-tmp check-dir1
		fi
            
	fi
fi


#tell user if successfull or not
cd $listdir/$bs-$hour-$number

if [ ! -f check-dir1 ]  || [ ! -s check-dir1 ] && [  ! -f check-dir2 ] || [ ! -s check-dir2 ]; then
	if [ ! "$quiet" = "y" ]; then
		echo  "Synchronization successful - Quitting..."
	fi
else
	if [ ! "$quiet" = "y" ]; then
		echo "ERROR"
		if [ -s check-dir1 ] && [ -s check-dir2 ]; then
			echo "$dir1 & $dir2 are not identical: Unique files in both."
		elif [ -s check-dir1 ] && [ ! -s check-dir2 ]; then
			echo "$dir1 & $dir2 are not identical: Unique files in $dir1."
		elif [ ! -s check-dir1 ] && [ -s check-dir2 ]; then
			echo "$dir1 & $dir2 are not identical: Unique files in $dir2."
		fi
		if [ $rmld = "n" ]; then	        
		echo "See the lists in $listdir/$bs-$hour-$numberH for more information."
		fi
	fi
fi

#remove lists if listdir is not given
if [ $rmld = "y" ]; then
	rm -rf  $listdir
fi

#exit 

exit

I'd be happy about some feedback

For people to lazy to copy and paste: Download

MoonSwan · 2011-08-30 20:14:29

Wow, nice! Out of curiousity, and because I was going to learn to use Unison (it's still being developed, I hope), what's the main difference between your script and something like Unison?

Unia · 2011-08-30 20:39:52

So lets say I want to keep a backup of my home dir on a separate harddrive, by just copying them over. On day one, I bought the HDD, formatted it and copy my complete /home on the HDD. A week later, I have downloaded numerous files, did some work, edited some music and put my holiday photo's on the computer.

If I then run your script, it will sync my home dir with the HDD and thus, copy over any new/changed files automatically, leaving the old files that haven't changed intact?

^ Is that assumption correct? If so I might test it!

Doomcide · 2011-08-31 08:27:25

@MoonSwan: I didn't hear about Unison before, quickly googled it and from what I read the main difference seems to be that this script only synchronizes one way. This means, if you have Dir1 and you want an exact copy of it called Dir2, everytime you run this script like this:

 esync.sh -y Dir1 Dir2

It will update the files that have been updated in Dir1 and add new files that were added to Dir1 in Dir2 and, also will delete files in Dir2, that don`t exist in Dir1 (anymore). The last behavior can be changed, by running it with the '-n' switch. Now if you modified some files in Dir2 and want Dir1 to have the same content, you have to run it like:

esync.sh [-y/n] Dir2 Dir1

(It's also possible to choose interactively if files should be deleted, while the script is running.)

@Unia: Yes, it only copies the files, that have been modified or added, the rest is left unchanged. It's also possible to delete files that only exist in the "backup directory", as described above.

Last edited by Doomcide (2011-08-31 08:27:46)

MoonSwan · 2011-08-31 19:18:18

@Doomicide, very good answer, thank you.

I may play with this when I have some time this week. Much thanks for sharing this little script! It's people like you that I met in my first forays into linux a decade a go and it's fantastic to see people like yourself today.

TLDR: thank you very much and you're awesome

Doomcide · 2011-08-31 20:17:00

Thank you, too for the compliments Feels good to know one's effort is appreciated.

chneukirchen · 2011-09-01 21:12:33

How is that an improvement over rsync?

mhertz · 2011-09-01 23:57:45

Doomcide wrote:

I had a lot of freetime and wanted to improve my bash-knowledge, also rsync had a lot of stuff I didn't need and was, in my case, slower than the script. (The tests take longer, but if you make changes to large files, rsync is a lot faster) So I thought I might as well share it here:

I myself also just use rsync, but nonetheless...

Last edited by mhertz (2011-09-02 00:00:45)

Doomcide · 2011-09-02 08:26:35

chneukirchen wrote:

How is that an improvement over rsync?

As I said in my first post, this is way faster if you don't change big files and is really easy to use. It's not an Improvement it's something different. Of course rsync is way more powerful and flexible, but this script does what I need better and faster.

mhertz wrote:

I myself also just use rsync, but nonetheless...

But "nonetheless..." what? Sorry I don't really get what you want to say.

mhertz · 2011-09-02 23:11:11

Sorry, I just reffered to that even though I didn't myself used it, then the contribution imho where still valid i.e. had a reason for existing and not being useless...

Edit: Hmm, just to be sure that we don't talk between each other, then my last post with your quote and my reply under it, where not to say something to you, but instead to answer the previous posters question: "How is that an improvement over rsync?", where I used your quote to answer him with...

Last edited by mhertz (2011-09-02 23:19:46)

Doomcide · 2011-09-05 14:27:17

Ok now I understand, sorry sometimes it takes some time

Arch Linux

#1 2011-08-30 12:04:02

esync.sh - easy directory sync'ing

#2 2011-08-30 20:14:29

Re: esync.sh - easy directory sync'ing

#3 2011-08-30 20:39:52

Re: esync.sh - easy directory sync'ing

#4 2011-08-31 08:27:25

Re: esync.sh - easy directory sync'ing

#5 2011-08-31 19:18:18

Re: esync.sh - easy directory sync'ing

#6 2011-08-31 20:17:00

Re: esync.sh - easy directory sync'ing

#7 2011-09-01 21:12:33

Re: esync.sh - easy directory sync'ing

#8 2011-09-01 23:57:45

Re: esync.sh - easy directory sync'ing

#9 2011-09-02 08:26:35

Re: esync.sh - easy directory sync'ing

#10 2011-09-02 23:11:11

Re: esync.sh - easy directory sync'ing

#11 2011-09-05 14:27:17

Re: esync.sh - easy directory sync'ing

Board footer