A quick way to remove duplicates?

fbungarz
Posts: 3163
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: A quick way to remove duplicates?

Post by fbungarz » 31 Oct 15 11:33

Personally I am no longer using SpaceMan, it is powerful for what it does: finding and deleting duplicates. But that is all it does...
If you are looking for a tool to efficiently organize your files, File Commander is indeed an option. An alternative might be XYplorer, which has a powerful duplicate finder built in and allows you to see folder contents side-by-side in dual panes. I hardly ever use Windows Explorer anymore...
Cheers,
Frank

vlad
Posts: 967
Joined: 01 Sep 08 15:20

Re: A quick way to remove duplicates?

Post by vlad » 31 Oct 15 11:34

Hi catface,

Based on your current task description, I would almost certainly detect and clean your ~7k duplicates using a specialized program, before importing and handling all the pictures in Photo Supreme. A couple of hints here:
- make sure you ask the program to identify the duplicates based on binary content and not just filenames
- both Duplicate Cleaner Pro and NoClone are also able to detect similar images, but I have no experience with that feature (although I am interested in it: if you try it, please do let me know how well it works)
- you may want to run the de-duplication procedure in batches (on several folders at a time), since any program might have performance problems scanning and processing ~100k files

Once you get rid of duplicates, you still have the daunting task of managing ~100k pictures. I would say Photo Supreme is a perfect fit for this task, but you may want to clarify what is your objective and mandate from your sister. Is it to simply tag all pictures, or is it also to cull pictures (keeping only those worth keeping - yep, that's subjective, so first make sure your sister trusts your judgement :wink:)? Another aspect to think about: do you want to tag and organize the pictures solely based on their content or also to retain some info about associated events or locations? (Hints about events, locations or persons might be given by some existing folder names, while the content could usually be inferred directly by looking at the pictures.)

If you've already got rid of duplicates but you don't want to mess with the folder structure for now, then you could import your folders in Photo Supreme as they are (but perhaps applying some file renaming rule, as advised by Mike) and then systematically work your way through pictures based on some set criteria (perhaps independent of existing folders, if those have reduced or no relevance), such as the image dates. (Check the Catalog -> By Date view.)

If you cull enough pictures, you might want to organize the remaining pictures in portfolios and collections, while also tagging them. Eventually (as you create portfolios), you could export the organized pictures into fresh, clean folder structures. This way, you could gradually transition from your old folders into new ones, more or less on the go.

(Keep in mind that's just theory - in practice, you may find pictures of cute cats across many input folders and dates, so you may end up continually building and re-organizing your portfolio collections and output folders. Again, I would say that's somewhat easier if you care only about picture content and, say, chronological order rather than other info too.)

HTH - and let us know your plans and thoughts, if you will,
Vlad

catface
Posts: 35
Joined: 22 Oct 15 19:38

Re: A quick way to remove duplicates?

Post by catface » 31 Oct 15 13:05

Thank you all so much for all your help. I will get started on working out an effective workflow. I'll try out some of the tools suggested and see if I can wrap my head around them.

Will keep you updated!

catface
Posts: 35
Joined: 22 Oct 15 19:38

Re: A quick way to remove duplicates?

Post by catface » 31 Oct 15 15:50

Update;

I used Duplicate File Finder to clean up the duplicates from the external harddrive. It was fast and easy to use, so I'd recommend it.

Now I'm gonna import all the files once again, but how should I do that? Is there anything I should make sure to have checked or unchecked? Do I need to copy the images to a new location for example?

vlad
Posts: 967
Joined: 01 Sep 08 15:20

Re: A quick way to remove duplicates?

Post by vlad » 31 Oct 15 16:52

Do you want to work directly on the existing folders or do you want to keep those untouched, for backup or reference?

For the second option:
1) You should enable Copy images to new location in the import profile.
2) Do you want to create a sub-folder structure for the target location? If yes, then set Subfolder as you wish.
3) Do you want to rename your images upon importing (as advised)? If yes, then set File name as you wish. (There are different naming schemes, so that's a different discussion.)
4) For safety, you may wish to set Duplicate Handling to Make name unique (given that by now all your remaining duplicates should be file name duplicates rather than binary duplicates).
5) Are you interested in image versioning? If not, then make sure Apply versioning is disabled.

I would advise to import your images in batches and check that your custom import profile has had the desired effect. You could even consider working in batches and buidling up a clean catalog incrementally.

OTOH, if you're really impatient and don't care about renaming and copying your images to a new folder structure, I guess you could try working on the existing catalog. If you trigger Verify folder (right click on a folder name), then the images you have already deleted from that folder should get deleted from the PSU catalog too. (That's simply my expectation - I haven't actually tried it.)

That's all I can recommend for now - other people might offer you additional or alternative advices.

catface
Posts: 35
Joined: 22 Oct 15 19:38

Re: A quick way to remove duplicates?

Post by catface » 31 Oct 15 21:01

I really appreciate all of your help. Will get started on working now, with all of the duplicates out of the way.

Mke
Posts: 433
Joined: 15 Jun 14 15:39

Re: A quick way to remove duplicates?

Post by Mke » 01 Nov 15 16:01

vlad wrote:If you trigger Verify folder (right click on a folder name), then the images you have already deleted from that folder should get deleted from the PSU catalog too. (That's simply my expectation - I haven't actually tried it.)
That is indeed what it does (it will generate a list of all the images to be removed first, for your optional review).

ethan11james
Posts: 3
Joined: 21 Apr 15 6:38

Re: A quick way to remove duplicates?

Post by ethan11james » 18 Nov 15 8:09

hey if you struggling with duplicate and similar photos problem easily delete are duplicate from the help of tools. There are many tools available for Mac, Windows. iOS and android platform you can use duplicate photo finder tool and delete all duplicate files.

[update Moderator; removed link to 3rd party software]

Post Reply