Will ‘fdupes -r -d -N’ solve my problem?

WeeDram · 2021-04-02 22:12:59

I have a rather old external hard drive with multiple backups on it from years ago. I had the bad habit of simply writing backups to different directories named according to what computer I was upgrading or replacing or simply backing up. Now there are multiple copies of all sorts of files all over the place on the drive.
I am pretty sure much of the stuff is really of no interest to me at all but I would like to check. However, I have realized that I probably won’t live long enough to do this with all those duplicate files

I was thinking of simply using
fdupes -r -d -N
at the root directory of the hard drive but was wondering:
a. will it pick up all duplicates and leave only one or do I need to do this on a directory by directory basis.
b. will it pick up only exact duplicates
c. how likely is it to go completely wrong.
According to my online searches, I think the answer to a & b is yes. Not sure about c.
Actually, I am at the point with this particular drive that I am not sure if I care if the whole operation crashes and burns.
Cheers,
Ross

seth · 2021-04-03 09:20:57

https://man.archlinux.org/man/community … dupes.1.en

You need to provide a directory to search and make VERY sure that you're *NOT* providing the symlinks option.
In doubt, you could preface a dry-run and only print a summary to get an idea of how bad it is.

Also notice that this can be extremely slow and RAM hungry.
Since you're dealing with likely incremental backups of the same data, you might want to run sequences where you compare the oldest backup to each following one individually, then the second oldest to each following one individually… etcetc.
IF YOU DO this MAKE EXTREMELY SURE that you are NOT PASSING THE SAME DIRECTORY TWICE!!

jdupes is a fork of fdupes that claims better performance and according to its github issues has recently (~1 yr ago) rewritten the deduplication code to be less RAM pressuring.

As always w/ autodeleting stuff: good luck

WeeDram · 2021-04-05 16:17:13

Hi, thanks for your reply. I left out the directory on the command line by accident. After testing fdupes on a few directories, I found that although it worked well (so far), using it just highlighted even more the need for a well thought out over-all plan on file removal and consolidation.

My biggest problem at the moment is dealing with file and directory names with spaces in them. I'd like to remove or replace those spaces because it will make tidying things up much easier. So far, I can't get any of the stuff I have found on various searches to work on my installation. But that is really a different question and one I may ask on the forum at a later date.

Again, thanks for responding to my question.
Ross

seth · 2021-04-05 16:26:36

I'm not sure why the spaces would be an obstacle, but check https://archlinux.org/packages/communit … rl-rename/ (unless you want to "find | xargs" it)

WeeDram · 2021-04-06 00:02:41

Thanks for getting back on this! So here's something you can chuckle about. I got so caught up in trying to figure out how to remove/replace spaces and trying all sorts of ways to do it (including trying to use pearl-rename) that I can't remember now any specific reason WHY I wanted to do it. Probably something to do with planning, executing and keeping records of which directories I used fdupes on.

Arch Linux

#1 2021-04-02 22:12:59

Will ‘fdupes -r -d -N’ solve my problem?

#2 2021-04-03 09:20:57

Re: Will ‘fdupes -r -d -N’ solve my problem?

#3 2021-04-05 16:17:13

Re: Will ‘fdupes -r -d -N’ solve my problem?

#4 2021-04-05 16:26:36

Re: Will ‘fdupes -r -d -N’ solve my problem?

#5 2021-04-06 00:02:41

Re: Will ‘fdupes -r -d -N’ solve my problem?

Board footer