Custom XMP in Info vs Details Panel?

Post Reply
fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 31 May 15 9:43

It looks like it worked [mostly?]. The labels are there, the few I checked are mapped correctly.
However, oddly enough some of the categories seem not to have migrated completely. They are arranged differently in the new catalog and they no longer appear marked by a custom color (see the screenshot). I am not sure if that is relevant, but thought you'd better know:
OldPSU-NewPSU.jpg
OldPSU-NewPSU.jpg (188.49 KiB) Viewed 3003 times

Hert
Posts: 21558
Joined: 13 Sep 03 7:24

Re: Custom XMP in Info vs Details Panel?

Post by Hert » 31 May 15 17:19

That's possible. Some of the Categories are "fixed" and can't be overwritten by a script. For those categories (People, Events, Places Miscellaneous) the import script would only read the content for those categories and not the details like colors or its position. This won't matter much as you could easily replace them yourself and change the colors. But for completeness I've updated the Import script to read those details too.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 31 May 15 17:34

OK, the updated version worked and the categories were imported with the same properties as the original catalog. What's next?
Frank

Hert
Posts: 21558
Joined: 13 Sep 03 7:24

Re: Custom XMP in Info vs Details Panel?

Post by Hert » 31 May 15 17:54

Now you have a new database with your existing catalog structure.

- Configure your Preferences how you want them
- Run an import for a smaller folder to see the results
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 31 May 15 20:05

Hi Hert,
I have configured the preferences exactly as in the screenshots that I posted here previously.
Then I have run a test import on one single folder.
Unfortunately Versioning does not work as expected. The folder contains 3 versions for each image a JPG, a DNG and a NEF, all named the same, but differing by their file extension: *.jpg, *.dng, *.nef.
In IDI5 I had the JPG as the main version, the DNG assigned to a custom place holder called "digital negative", and the NEF assigned to one called "Original".
After the import images appear all as versioned, but they are not being assigned to the respective placeholders.

So I tried to do two things:
1) In Preferences I added a custom file mask for these placeholders: Digital Negative = *.dng, Original = *.nef (see screenshot).
That still does not work. Images are not assigned to their respective placeholders.
2) I configured PSU to "temporarily" read the ICS schema thinking that it would read the versions from the ICS that IDI5 wrote into the images.
Still: the versions are not recognized and not assigned to their placeholders.
PSU_VersionConfiguration.jpg
PSU_VersionConfiguration.jpg (85.07 KiB) Viewed 2974 times
In the Import Settings dialog there is only the option to configure which version to detect as main version, JPG or RAW. There seems no way to specify how subversions are being assigned.

What am I doing wrong?

Thanks,
Frank
Attachments
PSU_Marked for import.jpg
PSU_Marked for import.jpg (190.86 KiB) Viewed 2973 times
PSU_Versions Not Assigned NEFs not imported#.jpg
PSU_Versions Not Assigned NEFs not imported#.jpg (146.05 KiB) Viewed 2974 times

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 31 May 15 22:20

Selecting all images in the folder after import and running version detection on them gives inconsistent results:
Of a total of 72 files (24 JPG + 24 DNG + 24 NEF) 17 NEF files are assigned to the place holder "Original" and 7 DNG files are assigned to the place holder "Digital Negative".
Destroying all versions and running version detection again results in an error message "Access violation (0)". The program freezes and can only be closed via the task manager.
Opening the database again, destroying the version sets again and running version detection on all files again results in 18 NEF files assigned to the "Original" and 23 DNG files to the "Digital Negative" placeholder.
Trying it again (destroy + detect) results in 24 NEFs corectly assigned, but only 16 DNGs assigned.
Completely inconsistent.

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 01 Jun 15 7:24

2) I configured PSU to "temporarily" read the ICS schema thinking that it would read the versions from the ICS that IDI5 wrote into the images.
Still: the versions are not recognized and not assigned to their placeholders.
That's strange. Did you have versioning enabled or disabled in the import profile when you read the ICS schema? Could you confirm that ics:VersionPlaceHolder stores the correct value in the imported images?

(Tip: you could add ics:VersionPlaceHolder as a custom field and you should be able to select it in Catalog->By Image Details; however, the ics:VersionPlaceHolder counter stays at 0 for me there - I'm not sure when or how its indexing gets triggered.)

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 9:47

Dear Vlad,
thanks for getting back on this.
It appears that these tags are NOT being written to the ICS schema by IDI5. Perhaps Hert newly introduced them with PSU?
Here a screenshot of IDI5:
IDI5_VersionTags.jpg
IDI5_VersionTags.jpg (155.03 KiB) Viewed 2934 times
The same in PSU3 after importing the image:
PSU3_VersionTags.jpg
PSU3_VersionTags.jpg (86.72 KiB) Viewed 2934 times
This means, for the versions correctly to be recognized, it won't make a difference if ICS reading is enabled or not. The tag is not there and consequently cannot be read.
Now, that still does not explain why the images are recognized as subversions, but not assigned to their correct placeholders during import.
Yes, I have of course versioning enabled during the import and I have configured PSU preferences to match these placeholders using custom masks:
PSU_CustomMask_Placeholder.jpg
PSU_CustomMask_Placeholder.jpg (118.02 KiB) Viewed 2934 times
Now: I thought that perhaps configuring PSU to read the ICS schema might overwrite the version tags otherwise assigned by PSU. So I tried it again with ICS reading turned off. The result, though, is the same. Not one version placeholder is recognized during the import. And: Trying to version the files via SHIFT+CTRL+V after the import returns completely inconsistent results.

:(

Frank

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 10:14

PS: That the versions assigned in IDI5 are not being recognized would not be such a problem if PSU would detect them afterwards using SHIFT+CTRL+V. But even destroying the versions after import and re-versioning the placeholders are simply not assigned to all images. I have even tried to select the images that are not correctly assigned manually and used SHIFT+CTRL+V on these images again. They are then assigned to the correct placeholders, but in turn for other images their placeholders are then removed.

Example:
1) I select all 72 images (24 JPG, 24 DNG, 24 NEF) in the folder after import and destroy the version sets.
2) I select all images and apply SHIFT+CTRL+V
3) 24 JPGs are assigned to the Main Version, 23 DNGs to "Digital Negative", 13 NEFs to "Original".
4) I filter all NEFs that do not have the placeholder "Original" assigned and use SHIFT+CTRL+V again.
5) 24 NEFs now have the placeholder "Original", but now suddenly only 16 DNGs are still assigned to the placeholder "Digital Negative" (previously 23 DNGs were assigned to that placeholder, thus 7 DNGs had their placeholder removed when the NEFs got their placeholder assigned).
6) I try to re-assign the DNGs that now lack a placeholder, selecting them and use SHIFT+CTRL+V, as a result some of the NEFs now have their placeholders removed.

The only way I can correctly assign ALL placeholders is MANUALLY!

Obviously that is not an option when I now have to re-import ca. 80,000 images.

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 11:10

The good news is:
Writing both standard XMP and custom Darwin Core XMP now works perfectly and all my labels with their mapping appear to have migrated correctly. Thank you Hert! This is fantastic.
Now I only have to figure out how to resolve the versioning issue and I am good to go. I am sure I have something not correctly configured. If I only knew what...

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 11:24

I believe I know what might cause the problem:
When I look at the NEFs and DNGs, they are both assigned as subversions to the JPGs BUT sometimes the NEF comes as the first subversion, sometimes the DNG (looking at the little tab on the right of the thumbnails, sometimes the DNG is no. 1, sometimes the NEF). When I manually correct this the DNG always comes first (on the tab labeled 1) and the NEF comes second. Apparently the version detection randomly assigns one of the raw images as a first subversion, either NEF or DNG, but the custom placeholders seem to require that the DNG is recognized as the first subversion followed by the NEF.
Not sure if any of this is by design or should be reported as a bug in Mantis.
PSU_placeholder_sequence.jpg
PSU_placeholder_sequence.jpg (108.65 KiB) Viewed 2924 times

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 15:12

OK, I have just found this post from January 2015 about versioning: http://forum.idimager.com/viewtopic.php ... ng#p108395
That means I am not the only one having trouble with assigning version placeholder automatically according to the file naming masks specified in Preferences. Fortunately, instead of using automatic versioning there seems a relatively rapid way to do this manually:
Once the files are versioned, hover over each of the version tabs. You'll see various menu items. The menu items displayed on the Main Version tab are different than the menu items displayed on the Sub-version tabs. More specific to your question, hovering over a Sub-version tab displays the placeholders. Hovering over a placeholder displays the name of the placeholder. Clicking a placeholder automatically assigns that placeholder to the selected thumbnail(s).
This is still a bit cumbersome when importing thousands of files, but until the versioning bug gets fixed it seems to be the only option.
I have now filed a report in Mantis: http://bugs.idimager.com/view.php?id=28645

BTW: I have found a lot of things in PSU that I really DO like compared to IDI. And I do really appreciate the help migrating my catalog. So: I hope this does not just come across as all negative...

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 01 Jun 15 16:21

fbungarz wrote:Dear Vlad,
thanks for getting back on this.
It appears that these tags are NOT being written to the ICS schema by IDI5. Perhaps Hert newly introduced them with PSU?
Here a screenshot of IDI5:
Frank, your screenshot doesn't prove anything, because it is for a JPG thumb selection - that is, for a main version. Could you please select a newly imported sub-version (NEF or DNG) and check if there's any ics:VersionPlaceHolder field for that image?
This means, for the versions correctly to be recognized, it won't make a difference if ICS reading is enabled or not. The tag is not there and consequently cannot be read.
I'm entirely open to that possibility - but your screenshot does not definitely prove it.
Now, that still does not explain why the images are recognized as subversions, but not assigned to their correct placeholders during import.
Agreed.
Yes, I have of course versioning enabled during the import and I have configured PSU preferences to match these placeholders using custom masks:
If you enable ICS reading and have version placeholder tags in your ICS fields, I am not sure it's a good idea to enable versioning during importing, as I don't know which version placeholder schema (profile specified versioning vs. ICS-based versioning) takes priority. I would appreciate if Hert could clarify this, whether that still matters to Frank or not.
When I look at the NEFs and DNGs, they are both assigned as subversions to the JPGs BUT sometimes the NEF comes as the first subversion, sometimes the DNG (looking at the little tab on the right of the thumbnails, sometimes the DNG is no. 1, sometimes the NEF).
This in itself wouldn't worry me too much, as I am not aware of any designed (that is, documented) behavior that should depend on the order of sub-versions within a version tab.
When I manually correct this the DNG always comes first (on the tab labeled 1) and the NEF comes second.
I don't understand, how do you "manually correct this"?
Apparently the version detection randomly assigns one of the raw images as a first subversion, either NEF or DNG, but the custom placeholders seem to require that the DNG is recognized as the first subversion followed by the NEF.
Not sure if any of this is by design or should be reported as a bug in Mantis.
If you could reproduce this behavior, I would say it classifies as a bug and needs reporting in Mantis: I can't imagine any reason why the custom placeholders should be assigned by design only upon a certain versioning order, so long as the custom file masks do not overlap (which shouldn't be a concern in your case: *.dng vs. *.nef).

(Btw, a more interesting issue arises if the custom filemasks set by someone do overlap and both match the same filename, e.g.: img_1_subversion.nef would match both *version.nef and *version.* - which one does/should PSU consider? Frank, that actually reminds me of something: I noticed in one of your screenshots that you had at one point the album display placeholder set to "First Version Found". That matching criteerion obviously could override the custom filemasks criteria - so I would make sure to set every placeholder except Digital Negative and Original to No Matching.)
I have now filed a report in Mantis: http://bugs.idimager.com/view.php?id=28645
Ah, that's very good - many people complain of problems, but not all of them follow up with Mantis reports.
BTW: I have found a lot of things in PSU that I really DO like compared to IDI.
And I'm pretty sure you'll discover quite a lot more :D
Writing both standard XMP and custom Darwin Core XMP now works perfectly and all my labels with their mapping appear to have migrated correctly.
Sweet - you're 90% on your way to the promised land! :)

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 19:10

Frank, your screenshot doesn't prove anything, because it is for a JPG thumb selection - that is, for a main version. Could you please select a newly imported sub-version (NEF or DNG) and check if there's any ics:VersionPlaceHolder field for that image?
Sorry, my mistake, but I assure you it looks just the same for a DNG and NEF. The field is empty.
If you enable ICS reading and have version placeholder tags in your ICS fields, I am not sure it's a good idea to enable versioning during importing...
Yes, I also thought this might be the problem, but since the version placeholder tag is not written by IDI5 turning ICS on and turning version recognition off during import does not make much of a difference:

Turning off versioning (with ICS read enabled) results in images being imported as non-versioned, i.e., not arranged in version sets. It also displays the imported images as "out-of-sync". Apparently the metadata are thus not being read during the import. This, with auto-sync being enabled, seems odd. However, it perhaps makes a certain sense: to avoid overwriting the ICS schema PSU apparently sets the images out-of-sync, thus suggesting you might still decide if you want to read the ICS data into the database...

Now, subsequently reading the metadata from the JPGs, DNGs and NEFs results in these images now being versioned (the JPG as main, DNG and NEF as subversions). Nevertheless, both DNG and NEF are still not assigned to a placeholder.
This actually makes some sense. IDI5 does not write the ICS placeholder tag to the images. This tag is missing and PSU obviously therefore cannot read it from the ICS schema. IDI writes tags which images is the main version and which other ones are subversions. Therefore, reading the metadata from the file into the database after import results in the images being versioned correctly, yet still not assigned to placeholders.
When I manually correct this the DNG always comes first (on the tab labeled 1) and the NEF comes second.
When I manually correct this the DNG always comes first (on the tab labeled 1) and the NEF comes second.
Simple. you destroy the version set and then one by one drag first the DNG onto the main JPG and then the NEF assigning placeholders while you do it. A rather cumbersome procedure, but it works. The DNG then becomes the first subversion with placeholder "Digital Negative", the NEF becomes the second subversion with "Original" as placeholder.
If you could reproduce this behavior, I would say it classifies as a bug and needs reporting in Mantis: I can't imagine any reason why the custom placeholders should be assigned by design only upon a certain versioning order, so long as the custom file masks do not overlap (which shouldn't be a concern in your case: *.dng vs. *.nef).
The rather random assignment of placeholders despite the custim masks correctly configured is definitely reproducible and in my opinion qualifies as a serious bug (which I have now reported in Mantis). I have no idea of course, if this is related to the way DNGs and NEFs are rather randomly assigned one time to the first, other times to the second version. It is only a hunch.
(Btw, a more interesting issue arises if the custom filemasks set by someone do overlap and both match the same filename, e.g.: img_1_subversion.nef would match both *version.nef and *version.* - which one does/should PSU consider? Frank, that actually reminds me of something: I noticed in one of your screenshots that you had at one point the album display placeholder set to "First Version Found". That matching criteerion obviously could override the custom filemasks criteria - so I would make sure to set every placeholder except Digital Negative and Original to No Matching.)
Yes, I thought about that too, but in my case versioning is rather straight forward. the DNG custom mask is *.dng, the NEF is *.nef and I always have just one single DNG and just one single NEF in each folder with the same name as the main JPG version, e.g.: DX_0001.jpg + DX_0001.dng + DX_0001.nef, DX_0002.jpg + DX_0002.dng + DX_0002.nef, etc.

fbungarz
Posts: 3308
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 01 Jun 15 19:28

OK, I am becoming convinced that this must be related.

(1) I destroy the version set.
(2) I select all images (CTRL+A) and run version detection (CTRL+SHIFT+V).
(3) Some NEFs are assigned to the first subversion. They are assigned the placeholder "Original". NEFs assigned to the second subversion get no placeholder assigned. The DNGs generally get no placeholder assigned, whether they belong to the first or the second subversion makes no difference.
(4) I select one triplet (JPG+DNG+NEF) at a time and run run version detection (CTRL+SHIFT+V) again.
(5) For each triplet the "Original" placeholder is assigned to the NEF, which becomes the first subversion, the DNG becomes the second subversion but remains without placeholder.
(6) I filter the folder to just show me the DNGs, whcih still lack placeholders and on these files now run run version detection (CTRL+SHIFT+V) again.
(7) Now the DNGs are all being assigned their corresponding placeholder "Digital Negative", but the placeholder "Original" previously assigned to the NEFs is again removed from these files!

As far as I can tell this is reproducible. It seems to suggest that version detection (CTRL+SHIFT+V) does not work correctly on more than two files.

Post Reply