[solved] Did rsync behaviour change? (=> nope, not here at least )

whoops · 2015-01-30 19:46:49

Hi!

I just used one of my "full system backup" hd's and the backup script on it deleted all the backups and itself (still made a current backup, just the incremental ones are gone now) -.-
I didn't use that script from the hd for a while (didn't have changes in big files for almost a year, so I only pushed my home folder to the router hd every day), but back when I was working with big files more, I used it every second day for several years and it always worked reliably.

That being said: My guess would be, that it deleted the "excluded=" folders, which is shouldn't (?) or didn't use to with --delete (the hd root was a clone of my system root, except for one excluded /BACKUP directory that kept the increments from --backup). Can't see any other way.

Did anyone notice a change in rsync behaviour? Anything strange?
Couldn't find anything anything on the net that fits... but I almost remember having fixed something rsync-related somewhere... some time ago... not sure what/if/why though, maybe just déjà-vue...
I have some more distributed rsync scripts and might have to check after a lot of things if there's really been a change in rsync behaviour...

( Please excuse the general fuzzyness of this question/topic... I'm in the middle of one of those weird fits of "I should have made a backup before I started that backup now I need to check EVERYTHING and the server too and omg when was the last time I even ssh'd into the router and looked at the cronjobs!?" )

Last edited by whoops (2015-01-31 09:55:19)

jasonwryan · 2015-01-30 20:17:37

Nope. Without seeing your script, it is impossible to say what happened...

Not a Sysadmin issue, moving to NC...

whoops · 2015-01-30 20:38:37

*sigh*
then I guess I'll update this once I scrape the rest of that self-destroying script out of the raw hd image or get my off-site copy from the bank.

The only thing I've isolated from that mess so far are the two rsync lines out of context (those must have been in some log somewhere separate from the rest of the script):

rsync -aAXvxbiH /* \
$backup_drive --backup-dir=$backup_drive/BACKUP/$(date +%Y-%m-%d) \
--log-file="$backup_drive/BACKUP/$(date +%Y-%m-%d).log" \
--exclude-from="/opt/scripts/backup.exclude" \
--delete --fuzzy \
|| { echo 'incremental backup failed' ; exit 1; }

rsync --delete -aAXv /* "$backup_drive" --exclude={share/Trash/*,/dev/*,/proc/*,/sys/*,/tmp/*,/tmp/.*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,*/BACKUP/*,/home/*/.ccache/*,/home/*/.thumbnails/*,/var/tmp/*,/home/*/.mozilla/firefox/*.default/Cache/*,/home/*/.mozilla/firefox/*.default/thumbnails/*,/home/*/.pulse/*,*.part} \
|| { echo 'full backup failed' ; exit 1; }

according to some of the output I saw, it was the second call of rsync that deleted /BACKUP/*
So as of yet, it looks to me, as if the way rsync interprets
*/BACKUP/*
must have changed at some point. Unless there's something in the context / rest of the script...

I'm pretty sure the same script is running on at least 2 other machines that I don't have access to ATM (and has been for 6 years or something) as a daily.cron...

DAMN YOU, "PAST ME"! -.-

Last edited by whoops (2015-01-30 20:52:29)

jasonwryan · 2015-01-30 20:44:59

That second call doesn't have a hyphen in front of the options...

whoops · 2015-01-30 20:52:09

Oh, thanks, that was a hasty copy paste error (was scared of the --delete ), fixed it now.

I think that script was an offspring of the script I wrote back THEN there *just-found-it*:
https://bbs.archlinux.org/viewtopic.php?id=152570
*sigh* I can't be trusted with a shell -.-

Last edited by whoops (2015-01-30 20:56:25)

jasonwryan · 2015-01-30 20:56:38

whoops wrote:

*sigh* I can't be trusted with a shell -.-

whoops earlier wrote:

or get my off-site copy from the bank.

At least you have learned from experience...

\hbar · 2015-01-30 21:43:20

I don't get it: I see a hyphen before the options. Was it edited?

Anyways, it seems to me that */BACKUP/* would not exclude /BACKUP : the /* files you are sync-ing get expanded into their names, one of which would be 'BACKUP' which does not (and should not) match '*/BACKUP/*'

whoops · 2015-01-31 08:31:37

\hbar wrote:

I don't get it: I see a hyphen before the options. Was it edited?

yes, sorry, it probably wasn't clear...:

whoops wrote:

Oh, thanks, that was a hasty copy paste error (was scared of the --delete ), fixed it now.

... what exactly I "fixed" there. Should have re-posted the line instead of editing the old post .

\hbar wrote:

Anyways, it seems to me that */BACKUP/* would not exclude /BACKUP : the /* files you are sync-ing get expanded into their names, one of which would be 'BACKUP' which does not (and should not) match '*/BACKUP/*'

Hmmm, right, "/BACKUP" should definitely be in there too just to be safe, thanks!
There also shouldn't have been a /BACKUP in the first place, though. Here's a more recent version of the whole script that I found on a different backup drive in the meantime (the one that caused the trouble should be almost identical):

#!/bin/bash

# check if root
[ -d /BACKUP ] && { echo "TRYING TO RUN SCRIPT ON WRONG DRIVE?"; exit 1; }
[ -f /backupthis ] || { echo "NOPE!  This drive does not want to be backed up ('/backupthis' missing)"; exit 1; }

# find backup drive marker file
while read mount; 
	do 
		if [ -a "$mount/BACKUP/backupdriveisabove" ]; 
			then echo $mount; 
			backup_drive=$mount; 
		fi; 
	done < <(mount | grep /media | sed "s/.*on \(.*\) type.*/\1/")

# check if marker file is fine - should match: 
# directory="/media/BACKUPMOUNT"; blkid $(mount | grep "$directory") | sed 's/.*: //' > $directory/BACKUP/backupdriveisabove
grep "$(blkid $(mount | grep $backup_drive) | sed 's/.*: //')" "$backup_drive/BACKUP/backupdriveisabove" \
	|| { echo "$backup_drive is not a valid backup drive." ; exit 1; }
echo -e "\E[32m===== Backup drive found in $backup_drive =====" && tput sgr0

# Help! I'm scared!
for countdown in `seq 0 5`; do echo "death and destruction in `expr 5 - $countdown`"; sleep 1; done;

# Rsync stuff of which incremental backups should be kept in ./BACKUP/$(date +%Y-%m-%d) first
echo -e "\E[32m===== Partial incremental backup =====" && tput sgr0
synctime=1337
while (( $synctime > 15 )) 
	do statime=$(date +%s);
		rsync -aAXvxbiH /* \
			$backup_drive --backup-dir=$backup_drive/BACKUP/$(date +%Y-%m-%d) \
			--log-file="$backup_drive/BACKUP/$(date +%Y-%m-%d).log" \
			--exclude-from="/opt/scripts/backup.exclude" \
			--delete --fuzzy \
				|| { echo 'incremental backup failed' ; exit 1; }
		eval synctime=`expr $(date +%s) - $statime`
		if (($synctime > 14)); then echo "rsync took too long ($synctime) - repeating"; fi;
	done;

# Rsync (almost all) previously excluded files / nonincremental full backup to backup drive root
echo -e "\E[32m===== Full system backup =====" && tput sgr0
rsync aAXv /* "$backup_drive" --exclude={share/Trash/*,/dev/*,/proc/*,/sys/*,/tmp/*,/tmp/.*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,*/BACKUP/*,/home/*/.ccache/*,/home/*/.thumbnails/*,/var/tmp/*,/home/*/.mozilla/firefox/*.default/Cache/*,/home/*/.mozilla/firefox/*.default/thumbnails/*,/home/*/.pulse/*,/home/_*,*.part} \
	|| { echo 'full backup failed' ; exit 1; }


# Try to merge two old backup increments without making too big a gap (if there are more than 60 backups)
echo -e "\E[32m===== Trying to merge 2 backup increments =====" && tput sgr0
minnamecur=0;
treshold=$(ls -d $backup_drive/BACKUP/2???-??-?? | wc -l);
if (($treshold > 60))
	then { 
		treshold=`expr $treshold / 3`
		# Do not create gaps between incremental backups that would be bigger than this many seconds
		mingap=$[60*60*24*30]
		pushd "$backup_drive/BACKUP" || { echo 'incremental backup folder not found' ; exit 1; }
		c=0
		# find backup that would leave the smallest gap
		for dir in 2???-??-??;
			do c="`expr $c + 1`";
			dirsec=$(date -d "$dir 00:00" +%s)
			eval name_$c=$dirsec; 
			if (($c > 4)) && (($mingap > 86399)); 
				then {
					cpre="`expr $c - 4`";
					ccur="`expr $c - 3`";
					cnex="`expr $c - 2`";
					eval namecur=\$name_$ccur;
					eval namepre=\$name_$cpre;
					eval namenex=\$name_$cnex;
					gap=`expr $namenex - $namepre`;
					if (("$gap" < "$mingap"))
						then {
							mingap="$gap"
							export minnamecur="$namecur"
							export minnamenex="$namenex"
							daysgap=`expr $mingap / 86400` 
					} fi;
				# reduce acceptable gap size for newer backup increments depending on amount of backups
				gapreduce="`expr $mingap / $treshold`";
				mingap="`expr $mingap - $gapreduce`";
				# echo "DEBUG: $c -- Folder $namecur -- acceptable gap size reduced to: $daysgap days";
			} fi; 
		done;
		# merge 2 backup increments / remove the old one. 
		if (( $minnamecur > 2000 ))
			then {
				eval rmback=$backup_drive/BACKUP/$(date -d "@$minnamecur" +"%Y-%m-%d");
				eval toback=$backup_drive/BACKUP/$(date -d "@$minnamenex" +"%Y-%m-%d");
				cp -vlrTf $toback $rmback &&\
					rm $toback -rv &&\
						mv -v $rmback $toback &&\
							echo -e "\E[32m===== moved $rmback to $toback, resulting gap: $daysgap days ====="  && tput sgr0
			} else echo "Could not find Folder to delete. Maybe gaps between backups are too big already."
		fi;
} fi;

popd

# install grub to backup-drive
echo grub-install $(mount | grep $(stat -c%m "$backup_drive") | sed "s/[0-9].*//")

# unmount the backup drive :3
umount $backup_drive -v \
	|| { echo "Something might have gone horribly wrong but now it's too late to do anything about it :D" ; exit 1; }

From what I can tell, the only way this can has happened is if somehow /BACKUP got created while the script was already running? Or is there another way that I can't see?

Have to retrace my steps...
1) Wanted to make a backup of my project in kdevelop with the builtin git function. When I pressed the "git button" in kdevelop, it crashed and all source files of the project were deleted.
2) temporarily killed git & other backup solutions. Tried to restore files with git manually, but there were no deleted files registered / all commits were empty for some reason.
3) I connected a SPARE_HD and dumped /dev/sda2 (my swap) to it (used a lot of ram so I hoped the source files were in there)
4) Connected OLD_BACKUP_HD in addition to SPARE_HD and started backup script ("just in case")
5) used extundelete to recover everything deleted withing the last 2 hours to the SPARE_HD
6) Started '# strings /sda | grep -n600 "Text that I luckily had in the first line of all my lost files" > /SPARE_HD/big_text-file'
7) Started rummaging trough the stuff that extundelete recovered on the SPARE_HD to see if anything's missing (only 2 of the deleted files were unrecoverable)
8) Started rummaging trough /SPARE_HD/big_text-file with grep, found the last 2 files I was missing in there (HUZZAH!)
9) Looked at my OLD_BACKUP_HD and saw it had deleted its own incremental BACKUP folder plus itself and its logs...
Hmmm, I can totally not see myself creating /BACKUP there somewhere between step 4 and 9 because during that time I tried to avoid writing ANYTHING to /dev/sda, but I don't see another way this could have happened, so I guess I must have done it anyway -.-

*sigh* Maybe it's time to retire that thing... even though it had a pretty good run (several years on several mostly unsupervised machines including a webserver and a router - and no problems)...

And HOW is it possible that EVERY SINGLE time I loose files it's because something goes wrong with an attempt to make a BACKUP?
The only reason why I even need backups seems to be that I keep deleting files while trying to make backups (even if for a change I don't use any messy self-written scripts) and then I end up doing "good old messy file recovery" instead anyways -.-

Last edited by whoops (2015-01-31 08:32:59)

jasonwryan · 2015-01-31 08:52:37

whoops 2nd rsync invocation wrote:

rsync aAXv /* "$backup_drive"

whoops · 2015-01-31 09:17:31

wtf, where does that keep popping up from in exactly that form!?
Now it's in half of the working copies I made of the script all over the place during file restoration somehow...
Thanks again, I would have never noticed that (well, maybe in a year or two )!

Maybe I introduced that typo earlier than I originally though after all? Pretty sure I didn't sed-i all over my scripts. Only grepped for --delete. Didn't use sed. Yes. Hope I didn't -.-

Unless I'm mistaken that would have prevented anything from getting deleted by the second invocation at all, since the "--delete" is also missing from that line... so... now I'm even more confused.

Not sure if I should continue trying to get to the bottom of this or just delete all of the unholy spawn from that script and hope that what ever happened won't happen again with whatever replacement setup I end up using -.-

edit: Oh no, according to my zsh_history I DID sed-i all over my scripts yesterday. In a "grep -l '--delete' -r | while read" sort of way... after I panicked because I didn't know how my files got deleted . And then a second time later in a different location. The only reason the first rsync invocation survived was probably due to the ^ in front of the -- there - I keep making things worse so consistently, if noone stops me that might just keep on escalating until the whole of humanity is doomed forever for some reason.

Last edited by whoops (2015-01-31 09:31:11)

whoops · 2015-01-31 09:53:54

HOLY SH*T FOUND IT!

I had an ancient forgotten udev rule for the ssid of one of the drives I connected while the backup script was running. That's what created the /BACKUP directory (and rmdir'd it later) . In combination with me having forgotten to add /BACKUP (in addition to */BACKUP/* which, you have to admit, looks sort of similar hidden in a long list ) to the rsync exclude list, that ultimately lead to the deletion of the files in $backup_drive/BACKUP/*

Mystery suddenly [solved], now I can start fixing everything!

( phew, gotta love a good scavenger hunt. Take that, "PAST ME"! Your weird config files that you hid everywhere in random places can't confuse me any longer! )

\hbar · 2015-01-31 21:43:52

whoops wrote:

HOLY SH*T FOUND IT!
I had an ancient forgotten udev rule for the ssid of one of the drives I connected while the backup script was running. That's what created the /BACKUP directory (and rmdir'd it later) . In combination with me having forgotten to add /BACKUP (in addition to */BACKUP/* which, you have to admit, looks sort of similar hidden in a long list ) to the rsync exclude list, that ultimately lead to the deletion of the files in $backup_drive/BACKUP/*
Mystery suddenly [solved], now I can start fixing everything!
( phew, gotta love a good scavenger hunt. Take that, "PAST ME"! Your weird config files that you hid everywhere in random places can't confuse me any longer! )

I'm glad you found the problem. My suggestion was to add 'BACKUP' and not '/BACKUP'

Arch Linux

#1 2015-01-30 19:46:49

[solved] Did rsync behaviour change? (=> nope, not here at least )

#2 2015-01-30 20:17:37

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#3 2015-01-30 20:38:37

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#4 2015-01-30 20:44:59

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#5 2015-01-30 20:52:09

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#6 2015-01-30 20:56:38

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#7 2015-01-30 21:43:20

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#8 2015-01-31 08:31:37

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#9 2015-01-31 08:52:37

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#10 2015-01-31 09:17:31

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#11 2015-01-31 09:53:54

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

#12 2015-01-31 21:43:52

Re: [solved] Did rsync behaviour change? (=> nope, not here at least )

Board footer