Flatten a directory, overwrite files only if bigger & picture sorting

marakasmalan · 2017-11-02 21:17:18

Good day, I need some advice here relating to the flattening of a folder structure. Studied the man pages for cp, move and rsync but cannot seem to find a solution, or simply to dof to identify a solution.

I have a directory with several subdirectories, varying in depth, something like I tried to sketch below. Many of the sub-folders contain the same files, mostly .jpg pictures. Some has been resized to a smaller size for the purpose of e-mail. Several folders has been copied from different machines. Time stamps can not be trusted.

From all the advice I found, I can flatten the folder and generated different file names for duplicates, or have all duplicates end up in a seperate folder.

What I need is all the files in all the subdirectories to be copied (or moved) to a folder (can be the parent folder), but files should only be overwritten if the source file is bigger than the similarly named destination file. Any advice how to achieve this, preferably from the command line?

Once that task is done, is there any Linux application that can rename the files according to the date on the meta-data (not the file time stamp) and place them into a directory structure based on year, month etc?

Thanks in advance!

mm

marakasmalan · 2017-11-03 03:02:03

Ok, so what I did manage:

find Pictures/  -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 15 > duplicates.txt

Will create a text file that will identify all the duplicates by means of the md5 checksum. It can take some time depending on your computer & n.o. files (What the -w 15 is I still need to determine)

awk 'NF != 0 {++count} END {print count}' duplicates.txt

Will return the total number pictures that are part of duplicated sets.

awk '/^$/{getline;print;}' duplicateses.txt |wc -l

Will determine the number of unique files in amongst the sets of duplicates.

fdupes -rSm Pictures/

will remove duplicates, determined via md5 checksum. fdupes package

The above mentioned can be used to eradicate duplicates, but wil still leave files with the same filename, i.e. the same picture, but saved in different resolution.

fslint is a very usefull GUI that can handle duplicates, as well as identify files with the same filename and will allow you to flatten the directory.

Thanks to Carla Schroder who published this article which I found very usefull.

Now the directory can be flattened:

find . -type f -print0 | xargs -0 -I%%% mv %%% /USB/USB_HDD_7/media/Pictures2

To rename and sort into a directory structure according metadata from the picture/file: exiftool can be used once the directory is flattened (cli). See also Phil Harvey's page

There should be a more elegant solution that you can execute from the cli, that solution has evaded me.

Regards

Trilby · 2017-11-03 11:56:41

Checksums only find actual duplicates - i.e. identical files. In your first post, it sounded like you wanted to compare based on file name, otherwise how would you dedup the images that were resized for emails?

Some notes on the second post:

marakasmalan wrote:

...What the -w 15 is I still need to determine

You haven't tried very hard, it's clearly listed in the man page. Do not copy and paste commands if you have no idea what they do - especially if they come from sources who don't know what they are doing (which is evident from the below)

awk 'NF != 0 {++count} END {print count}' duplicates.txt

No need for awk:

grep -cv '^$' duplicates.txt

awk '/^$/{getline;print;}' duplicateses.txt |wc -l
Will determine the number of unique files in amongst the sets of duplicates.

Will it? How? That counts the number of non-blank lines that follow a blank line in duplicates.txt - I don't see how this will count the number of unique files.

If you want to use that find command, just pass it to `uniq -c`. Then count the number of lines that start with 1 for uniq files, and the number of lines that start with a number greater than 1 for the duplicates.

But really, if you want to determine duplicates based on filenames, that find command with md5sums is completely useless. Do you suspect you have files that have been copied as exact duplicates (no resize) to new files with a different name?

Define the question before you start grabbing random answers of the internet to different questions that likely do not apply.

marakasmalan · 2017-12-10 11:53:48

Hello, Sorry for the very long delay, I was out in the bush working on a remote project in DRC.

Thanks for the response and help, Trilby, much appreciated.

Trilby wrote:

Checksums only find actual duplicates - i.e. identical files. In your first post, it sounded like you wanted to compare based on file name, otherwise how would you dedup the images that were resized for emails?

yes, you are 100% correct. Hence resorting to this post. Just tried to explain what what I tried so far. So, I still do not have a viable solution, but as you pointed out my problem definition should have been quntified with greater clarity.

Trilby wrote:

You haven't tried very hard, it's clearly listed in the man page. Do not copy and paste commands if you have no idea what they do - especially if they come from sources who don't know what they are doing (which is evident from the below)

Noted, thank you. Thorough man page consultation now part of first priorities when encoutering commands new/unknown to me.
For clarity sake, -w states the maximum number of characters to be compared in each line.

Trilby wrote:

awk 'NF != 0 {++count} END {print count}' duplicates.txt
No need for awk:
grep -cv '^$' duplicates.txt
awk '/^$/{getline;print;}' duplicateses.txt |wc -l
Will determine the number of unique files in amongst the sets of duplicates.
Will it? How? That counts the number of non-blank lines that follow a blank line in duplicates.txt - I don't see how this will count the number of unique files.

Yes, it does, as the number of empty lines created in the file equals the number of unique entries. Using your advice:

Trilby wrote:

If you want to use that find command, just pass it to `uniq -c`. Then count the number of lines that start with 1 for uniq files, and the number of lines that start with a number greater than 1 for the duplicates.

Yields exactly the same result. Your solution is better, mine is indirect and I suppose will leave room for error if the situation becomes more complex.

Trilby wrote:

But really, if you want to determine duplicates based on filenames, that find command with md5sums is completely useless. Do you suspect you have files that have been copied as exact duplicates (no resize) to new files with a different name?

Yes, unfortunately there is proper chaos in the folder.

Trilby wrote:

Define the question before you start grabbing random answers of the internet to different questions that likely do not apply.

Ok, Taking note of this as well. Tried to show that I did try, but clearly I was on the wrong path.

So ignoring all of the above, retaining only the knowledge gained, the problem is as follows.

I have a folder in excess of 30 GB, it contains only pictures ( .jpg & .raw only) and some unfortunate thumbnail files from the windows days.
The folder structure has many subfolders, varying in depth
Some files are duplicated, same size, name etc. (i.e. exactly the same) in different subfolders)
Some files are duplicated, but the duplications are of the same filename although the duplication have been resized to a smaller size (for email purposes I presume)
Some files are duplicated, but the duplication (which can reside in the same subfolder or in a different subfolder) has a different filename.

All the metadata of all the files are entact and can be relied on for sorting and renaming purposes.

The end result required is to have a main folder, with all duplicates and files resized to smaller sized removed.
All files should then be arranged into a folder structure arranged by year/month and the filename should be the date ( yyyymmdd.jpg or yyyymmdd.raw ) the picture was taken.

Any assistance will be greatly appreciated, I am struggling on my own. so a basic outline (as opposed to a spoon-feed solution) with references to what I should focus / research in order to deploy the the strategy wiould also be grand.

regards

marakas

Arch Linux

#1 2017-11-02 21:17:18

Flatten a directory, overwrite files only if bigger & picture sorting

#2 2017-11-03 03:02:03

Re: Flatten a directory, overwrite files only if bigger & picture sorting

#3 2017-11-03 11:56:41

Re: Flatten a directory, overwrite files only if bigger & picture sorting

#4 2017-12-10 11:53:48

Re: Flatten a directory, overwrite files only if bigger & picture sorting

Board footer