Importing / Numbering / Renumbering

[Paul C]

Using a sequential number will guarantee uniqueness, but not reproducibility, as I found in the issue that started this discussion. PSU will keep incrementing the sequence number, so if you import the same set of images twice, the second set will have new sequence numbers and so unique names - but they are duplicate images.

You can easily change the offset number when you want images that have already been imported to be imported a second time using the same file name as the first import. Having said that, I can't imagine why you would want to import any image more than once.

I don't want to import images more than once. I'm trying to make sure images never get imported more than once.

[Mike Buckley]

I don't want to import images more than once. I'm trying to make sure images never get imported more than once.

In my mind, the key to that doesn't have anything to do with a file naming convention. Instead, it has to do with developing an effective workflow and religiously implementing it. In the very rare times that you do mess up, the renaming system can provide the clue that that has happened but it should be very, very rare.

[HCS]

In my mind, the key to that doesn't have anything to do with a file naming convention. Instead, it has to do with developing an effective workflow and religiously implementing it. In the very rare times that you do mess up, the renaming system can provide the clue that that has happened but it should be very, very rare.

I'm with you there big time. Make sure your system doesn't allow for duplicates, don't depend on your software for that.

[Stephen]

In my mind, the key to that doesn't have anything to do with a file naming convention. Instead, it has to do with developing an effective workflow and religiously implementing it. In the very rare times that you do mess up, the renaming system can provide the clue that that has happened but it should be very, very rare.

I'm with you there big time. Make sure your system doesn't allow for duplicates, don't depend on your software for that.

I would have a hard time in agreeing with that last statement...

[Stephen]

Update

My numbering strategy does not quite work as planned because PSU does allow duplicates to be inadvertently imported. To continue using it, this will mean changing my workflow.
1. I import in small batches of related images to save time later by adding some key words and descriptions on import.
2. I do not always import consecutively

By default:
PSU marks all files for import and does not show which have already been imported. If you forget to unmark the ones you do not need, then you may import duplicates before you realize your mistake.

Also, it is easy to forget which have been imported and if you use slightly different descriptions, then the same images will be imported twice with the same file name.

Deleting the duplicates is a pain because PSU does not always import the pairs consistently, i.e. sometimes it is the RAW first and then the JPG and sometimes in reverse. So if you want to delete the ones with the "worse" keywords you cannot merely select the "third" and "fourth" file, but you must verify and mark each individually.

Only if the description, keyword and other META data is exact, will the duplicate file be imported as 123(001).RAW otherwise the file name will be identical.

[Mike Buckley]

Stephen,

I agree that you should strongly consider changing your workflow. The change would be to import all images at once. All numbering problems and other issues will be solved once you do that. All of the cataloging that you have been doing as part of the import process can be done after importing and it can be done in the same amount of time.

[vlad]

Only if the description, keyword and other META data is exact, will the duplicate file be imported as 123(001).RAW otherwise the file name will be identical.

Afaik, the duplication detection in PSU works based on the entire file content (or signature), so any difference is significant. (Keep in mind that PSU also has a facility for detecting similar images, in addition to exact duplicates.) But I don't quite understand your statement: do you (re)import the images to different folders? Otherwise, how could they end up with identical names?

[Stephen]

Stephen,

I agree that you should strongly consider changing your workflow. The change would be to import all images at once. All numbering problems and other issues will be solved once you do that. All of the cataloging that you have been doing as part of the import process can be done after importing and it can be done in the same amount of time.

"All problems would be solved"... I'm not so sure.
Current imports come from i.e. folder 247, are renumbering and then imported in groups "Paris Eiffel Tower", "Paris Arc de Tromphe", etc. PSU thus creates a folder named after each group with the exposure. This in my opinion is more useful (and future-proof) than relying 100% on PSUs capacity.

Rarely are images not imported every evening for security reasons, loss, theft, damaged media, etc.

BUT, I am re-thinking it.

[Stephen]

Only if the description, keyword and other META data is exact, will the duplicate file be imported as 123(001).RAW otherwise the file name will be identical.

Afaik, the duplication detection in PSU works based on the entire file content (or signature), so any difference is significant. (Keep in mind that PSU also has a facility for detecting similar images, in addition to exact duplicates.) But I don't quite understand your statement: do you (re)import the images to different folders? Otherwise, how could they end up with identical names?

As far as I could deduce, that is what happened. When I used very basic meta data, like city, state and country, then I got the 123(001).RAW or 123(002).JPG duplicates. I think that is when I inadvertently pressed import when all files on the screen were still marked, i.e. I instructed it to import files which I had already imported. That happened just a few times.

More common (working without two monitors next to each other) I had forgotten where I had last stopped importing, which is very easy to do. Then the same files are imported again and with the same names. The difference is that the data in the profile had been changed and the second imported therefore included meta data which was different to the images first imported and was in this case also incorrect. So the import function appears to interpret different meta data as referring to a different image. Example, I imported all files of the Eiffel Tower with the correct meta data. Then I did a second import of "Arc de Triomph" and inadvertently marked a few of the Eiffel Tower images too. The second duplicates got the same image name on import (it was being changed by batch file) and the only difference was the meta data.

[vlad]

More common (working without two monitors next to each other) I had forgotten where I had last stopped importing, which is very easy to do.

See? Here is precisely the reason, in your own words, why this workflow is fragile: it relies on human memory, which is... reliably fallible.

Then the same files are imported again and with the same names.

That's where I'm losing you: are the duplicates reimported into a different folder? If you select a pair of duplicates and check their respective paths (by hovering over the names bellow the thumbnails), do you see the expected paths? Does it make sense to you that they are different?

The difference is that the data in the profile had been changed and the second imported therefore included meta data which was different to the images first imported and was in this case also incorrect. So the import function appears to interpret different meta data as referring to a different image.

Ok, let me get this straight: you are importing the exact same source image twice, but with different metadata profiles applied the 1st and 2nd time. Do you use image copying, I guess? Is the target folder the same or do you change it the 2nd time? (IIRC, you also mentioned in a previous post that you use "Make name unique" as the duplicate handling option, right?)

Example, I imported all files of the Eiffel Tower with the correct meta data. Then I did a second import of "Arc de Triomph" and inadvertently marked a few of the Eiffel Tower images too. The second duplicates got the same image name on import (it was being changed by batch file) and the only difference was the meta data.

I didn't realize you do your renaming by batch file, I thought you were doing it via the File name option under "Copy images to new location". Which batch file are you using? And what's you renaming rule, once again?

[Stephen]

More common (working without two monitors next to each other) I had forgotten where I had last stopped importing, which is very easy to do.

<<<See? Here is precisely the reason, in your own words, why this workflow is fragile: it relies on human memory, which is... reliably fallible.

Oh, yes, I make lots of mistakes, so I would like hardware and software to support me with that.

Then the same files are imported again and with the same names.

<<<That's where I'm losing you: are the duplicates reimported into a different folder? If you select a pair of duplicates and check their respective paths (by hovering over the names bellow the thumbnails), do you see the expected paths? Does it make sense to you that they are different?

Yes, each import session goes into a different folder, as the names are usually different and therefore more useful. I am enclosing a screen shot of the import process to outline more clearly what I do. However, duplicate files naturally come from the same source. I am coming from a PIEware application (Aperture) which is naturally very different. I am not comparing both applications but just stating that Aperture had a good way of indicating which files had already been imported.

The difference is that the data in the profile had been changed and the second imported therefore included meta data which was different to the images first imported and was in this case also incorrect. So the import function appears to interpret different meta data as referring to a different image.

<<<Ok, let me get this straight: you are importing the exact same source image twice, but with different metadata profiles applied the 1st and 2nd time. Do you use image copying, I guess? Is the target folder the same or do you change it the 2nd time? (IIRC, you also mentioned in a previous post that you use "Make name unique" as the duplicate handling option, right?)

The source is the same, but the target folder is different on every import. Frankly I do not use the folders in PSU, I only work in the catalog.

Example, I imported all files of the Eiffel Tower with the correct meta data. Then I did a second import of "Arc de Triomph" and inadvertently marked a few of the Eiffel Tower images too. The second duplicates got the same image name on import (it was being changed by batch file) and the only difference was the meta data.

<<<I didn't realize you do your renaming by batch file, I thought you were doing it via the File name option under "Copy images to new location". Which batch file are you using? And what's you renaming rule, once again?

Yes, sorry, it is not called batch, it is the custom defined File Name option.

Photo SupremeScreenSnapz029.jpg (291.37 KiB) Viewed 7232 times

[vlad]

Hi Stephen,

Oh, yes, I make lots of mistakes, so I would like hardware and software to support me with that.

We all make mistakes. I do agree that software applications, in particular, should help us avoid or smoothly revert mistakes, whenever possible. Anyway, let's focus on the issue at hand and see if we could come up with specific suggestions or feature requests for Photo Supreme.

Yes, each import session goes into a different folder, as the names are usually different and therefore more useful. I am enclosing a screen shot of the import process to outline more clearly what I do. However, duplicate files naturally come from the same source. I am coming from a PIEware application (Aperture) which is naturally very different. I am not comparing both applications but just stating that Aperture had a good way of indicating which files had already been imported.

Thanks for the explanations and the screenshot. (Off topic: are those images taken in Patagonia?) It's now pretty clear to me how you're working and what's the problem: you expect PSU to detect image duplicates originating from the same source, although:
1) You copy the same source to two different folders on two different import sessions.
2) You assign different sets of metadata during the two import sessions.

Your expectation is not necessarily unreasonable for a DAM application, but it obviously doesn't match how PSU currently works:

1. PSU doesn't compare original (pre-import) sources against images already in the catalog. I actually think that would be a very valuable enhancement, as it would allow the exclusion (unmarking) of all such duplicates from the import session - i.e., preventing the mistake in the first place.

2. Afaik, PSU applies the filename-based duplicate detection (see the "Duplicate Handling" option under "Copy images to new location") only when you attempt to copy duplicate files to the same folder. This doesn't happen in your case, as you specify a different location every time. I don't see how PSU could help you here (lest the previous suggestion is implemented, of course).

3. Once a couple of images from the same source have been imported with different metadata profiles, you're saying they aren't detected by the (binary) duplication detection facility, right? That's actually interesting and I have a couple of questions:

a) Do you have "Automatically write out Catalog changes..." enabled? (That would explain it, since the metadata changes are then applied right away to the physical files, which have now different binary content - therefore the duplicate detection would indeed fail. Not sure what happens if the metadata is not yet synced...)

b) Is the same true for RAW files (assuming you write out metadata to sidecar files)? I'm asking because I have no idea if the duplicate detection takes sidecar files into account - perhaps Hert or someone else could clarify this?

c) Are images with identical content found as Similar Images (as opposed to Duplicate Files)? Check Catalog->By Catalog State.

The difference is that the data in the profile had been changed and the second imported therefore included meta data which was different to the images first imported and was in this case also incorrect. So the import function appears to interpret different meta data as referring to a different image.

As much as you're focused on your own scenario, you need to realize that there are cases where the very same image could indeed have different metadata applied, for legitimate reasons (versions, anyone?). Btw, do you use versioning? If you don't, you could perhaps employ the version detection mechanism simply to detect images with identical (or very similar) names, which are presumably undesired duplicates. (That's just a rough idea, though, as you would want to rule out false positives.)

I think we can all learn from this, so I'm happy to discuss this further and think of improvements w.r.t. both Photo Supreme and our own workflows.

Cheers,
Vlad

[Stephen]

Yes, each import session goes into a different folder, as the names are usually different and therefore more useful. I am enclosing a screen shot of the import process to outline more clearly what I do. However, duplicate files naturally come from the same source. I am coming from a PIEware application (Aperture) which is naturally very different. I am not comparing both applications but just stating that Aperture had a good way of indicating which files had already been imported.

<<<Thanks for the explanations and the screenshot. (Off topic: are those images taken in Patagonia?) It's now pretty clear to me how you're working and what's the problem: you expect PSU to detect image duplicates originating from the same source, although:
<<<1) You copy the same source to two different folders on two different import sessions.

I'm only following best practice recommendations by Krogh in the DAM book, lacking other recommendations anywhere.

<<<2) You assign different sets of metadata during the two import sessions.

That is inadvertent. If PSU would warn me then this would not occur.

<<<Your expectation is not necessarily unreasonable for a DAM application, but it obviously doesn't match how PSU currently works:

It was not obvious to me and it is not explained anywhere, as far as I have seen. It might be time to write a comprehensive manual

<<<1. PSU doesn't compare original (pre-import) sources against images already in the catalog. I actually think that would be a very valuable enhancement, as it would allow the exclusion (unmarking) of all such duplicates from the import session - i.e., preventing the mistake in the first place.

<<<2. Afaik, PSU applies the filename-based duplicate detection (see the "Duplicate Handling" option under "Copy images to new location") only when you attempt to copy duplicate files to the same folder. This doesn't happen in your case, as you specify a different location every time. I don't see how PSU could help you here (lest the previous suggestion is implemented, of course).

See comment above

<<<3. Once a couple of images from the same source have been imported with different metadata profiles, you're saying they aren't detected by the (binary) duplication detection facility, right? That's actually interesting and I have a couple of questions:

<<<a) Do you have "Automatically write out Catalog changes..." enabled? (That would explain it, since the metadata changes are then applied right away to the physical files, which have now different binary content - therefore the duplicate detection would indeed fail. Not sure what happens if the metadata is not yet synced...)

No, I do not.

Photo SupremeScreenSnapz030.jpg (148.92 KiB) Viewed 7209 times

<<<b) Is the same true for RAW files (assuming you write out metadata to sidecar files)? I'm asking because I have no idea if the duplicate detection takes sidecar files into account - perhaps Hert or someone else could clarify this?

Yes

<<<c) Are images with identical content found as Similar Images (as opposed to Duplicate Files)? Check Catalog->By Catalog State.

I don't know

The difference is that the data in the profile had been changed and the second imported therefore included meta data which was different to the images first imported and was in this case also incorrect. So the import function appears to interpret different meta data as referring to a different image.

As much as you're focused on your own scenario, you need to realize that there are cases where the very same image could indeed have different metadata applied, for legitimate reasons (versions, anyone?). Btw, do you use versioning? If you don't, you could perhaps employ the version detection mechanism simply to detect images with identical (or very similar) names, which are presumably undesired duplicates. (That's just a rough idea, though, as you would want to rule out false positives.)

I think we can all learn from this, so I'm happy to discuss this further and think of improvements w.r.t. both Photo Supreme and our own workflows.

Cheers,
Vlad

[vlad]

It's now pretty clear to me how you're working and what's the problem: you expect PSU to detect image duplicates originating from the same source, although:
<<<1) You copy the same source to two different folders on two different import sessions.

I'm only following best practice recommendations by Krogh in the DAM book, lacking other recommendations anywhere.

I also have the DAM book (although, admittedly, I haven't opened it in a long while) - where could I find the relevant best practice that you're alluding to? Regarding PSU-specific recommendations (which shouldn't necessarily contradict Peter Krogh's ones): it is true that there is no single document with clear and easy to follow guidelines - and I'm all for creating such a doc, and much sooner than a hypothetical (and almost utopical) comprehensive manual - but there are quite a few helpful people and good recommendations right in this forum.

<<<2) You assign different sets of metadata during the two import sessions.

That is inadvertent. If PSU would warn me then this would not occur.

I would support any practical feature request for preventing this happen, I'm even willing to brainstorm its feasibility and details. Until such an improvement gets implemented, you might want to adjust your expectations and workflow accordingly.

<<<Your expectation is not necessarily unreasonable for a DAM application, but it obviously doesn't match how PSU currently works:

It was not obvious to me and it is not explained anywhere, as far as I have seen. It might be time to write a comprehensive manual

Fair enough. Quite a number of people have requested a comprehensive manual. My personal opinion is that this is hard to achieve (without freezing the product's development) for a small (but agile) company like IdImager - I would rather see its efforts focused on fixing reported defects or shortcomings, implementing new features, as well as producing a set of practical teaching tools (video tutorials, guideline doc etc.) Your mileage may vary.

<<<3. Once a couple of images from the same source have been imported with different metadata profiles, you're saying they aren't detected by the (binary) duplication detection facility, right? That's actually interesting and I have a couple of questions:

<<<a) Do you have "Automatically write out Catalog changes..." enabled? (That would explain it, since the metadata changes are then applied right away to the physical files, which have now different binary content - therefore the duplicate detection would indeed fail. Not sure what happens if the metadata is not yet synced...)

No, I do not.

Ok, but I'm presuming you previously saved the metadata, at least after the first session? Is that true? If so, then the metadata is still different between the two physical files originating from the same source. The test would be to import the same image twice - with different metadata applied each time, but not yet saved to physical files - and see if the images are still detected as duplicates. Once the metadata is changed in at least one of the file, all bets are off, as the files are no longer true duplicates, are they?

(As an aside, it would also be interesting to check if applying the same metadata to multiple copies of the same original may result in the modified copies no longer being recognized as duplicates. That would be clearly sub-optimal, although it wouldn't suprise me too much if it happened.)

<<<b) Is the same true for RAW files (assuming you write out metadata to sidecar files)? I'm asking because I have no idea if the duplicate detection takes sidecar files into account - perhaps Hert or someone else could clarify this?

Yes

To me, that makes sense: it means that the duplication detection works the same, irrespective of whether the metadata is stored to the image file itself or to a sidecar file. Kudos to Hert for implementing consistent behavior here!

<<<c) Are images with identical content found as Similar Images (as opposed to Duplicate Files)? Check Catalog->By Catalog State.

I don't know

And aren't you curious? What prevents you from checking?

Cheers,
Vlad

[Stephen]

<<<c) Are images with identical content found as Similar Images (as opposed to Duplicate Files)? Check Catalog->By Catalog State.

I don't know

And aren't you curious? What prevents you from checking?
Cheers,
Vlad

PSU crashing and lack of time. It's Saturday here, the day when major backups are always done. Also, my priority has shifted to the colour rendering of edited images in PSU. I have posted a message in Mantis and I'm hoping to find a resolution, as this could be a turning point away from PSU.