Custom XMP in Info vs Details Panel?

Post Reply
fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 03 Jun 15 9:18

I would certainly report this in Mantis. Basically, you are saying that the ICS reading ignores the version placeholders altogether, right? (Undesired side effect: having both ICS reading and writing on also obliterates them from files upon syncing.)
I think I need to do some more testing and then probably update the existing bug report. The problem I have now with that bug report is that I no longer understand exactly why existing ICS placeholder tags in the file conflict with my attempt trying to run version detection on these files again. This is all getting fairly complex...
If you now run the version detection, do you still get inconsistent results? Does that happen even if you import the images with all the discussed options disabled, including the ICS reading?
Yes, that is exactly the problem for which I have not yet figured out what is going on.
"Apparently" if there are ICS tags already present in the files, but not the database and these tags are not read to the database, then trying to run version detection on these files causes some sort of hickup. Essentially it seems like these ICS tags are for some reason not being read to the database at all (no matter how ICS is configured!).
The last thing I tried was:
(1) copying the existing database
(2) deleting all images from that copy
(3) deleting the custom placeholders in preferences
(4) import new folder previously not imported with ICS read on, autosync off, import versioning off

I thought that perhaps the existing, but empty placeholder in the database were the problem, i.e., that for some reason PSU would not recognize the placeholder tags in the files being the same as the ones newly created in the database.
Consequently, I would have expected that the ICS schema being read would create both the versions and again create the placeholders. This did not happen. The versions were created, the placeholder tag in the files, however, was ignored, even if I after the import I selected again all files and read the metadata from the files via CTRL+ALT+S.

No matter what I do. Reading the ICS from the files correctly assigns the JPG as main, the RAW files as subversions, but in no scenario ever is the placeholder tag being read from the file!

Now, I have one final attempt left: Creating a completely new, empty database, import all labels again using Hert's script and then import the files with auto-sync off, ICS reading on, import versioning off, but no custom placeholders created in preferences. Perhaps there is a conflict still (even despite having completely deleted the custom placeholders) in my existing database. Perhaps the conflicting placeholders are still there somewhere behind the scenes.
My "logic" behind this: Theoretically I would assume that importing a file with a placeholder tag in the ICS, reading ICS from the file would create that placeholder in the database if it was not previously present. So, if the placeholders were, however, created before in PSU and if PSU for some reason does not recognize that its own placeholders are identical with the tags, then perhaps this is what causes the conflict.

I'll report back.
Frank

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 03 Jun 15 10:40

fbungarz wrote:I think I need to do some more testing and then probably update the existing bug report. The problem I have now with that bug report is that I no longer understand exactly why existing ICS placeholder tags in the file conflict with my attempt trying to run version detection on these files again. This is all getting fairly complex...
Frank, don't make it more complex than it needs to be!

I understand that your setup and workflow are fairly complex, but we must report specific bug reports (clear, easily reproducible) in oder to help Hert - and ultimately us! Version placeholders getting ignored upon ICS reading is a seperate issue from the reported problem with version detection and therefore requires a seperate bug report. You should then update the versioning ticket to either note that it may be a side effect of the ICS reading issue or to clearly state that it is independent of it (in which case it should be readily reproducible on images which have never been touched by ICS metadata).
Essentially it seems like these ICS tags are for some reason not being read to the database at all (no matter how ICS is configured!).
I think that's all you need to report in the new Mantis ticket. Simple and clear - no need to conflate the version detection aspect (...unless Hert specifically advises that running the version detection is indeed required to process the ICS versioning tags).
The last thing I tried was:
(1) copying the existing database
(2) deleting all images from that copy
(3) deleting the custom placeholders in preferences
(4) import new folder previously not imported with ICS read on, autosync off, import versioning off
I understand what you tested but I'm not surprised that didn't work, given all we have found out.
No matter what I do. Reading the ICS from the files correctly assigns the JPG as main, the RAW files as subversions, but in no scenario ever is the placeholder tag being read from the file!
Ok, here is yet again what you should report - in a new Mantis ticket! (Try also to replace "no matter what I do" with what you actually did - taking as reference the simplest, easily reproducible setup.)
Now, I have one final attempt left: Creating a completely new, empty database, import all labels again using Hert's script and then import the files with auto-sync off, ICS reading on, import versioning off, but no custom placeholders created in preferences. Perhaps there is a conflict still (even despite having completely deleted the custom placeholders) in my existing database. Perhaps the conflicting placeholders are still there somewhere behind the scenes.
That's indeed a theoretical possiblity, but it's a very long shot. Don't be disappointed or frustrated if that fails too.
My "logic" behind this: Theoretically I would assume that importing a file with a placeholder tag in the ICS, reading ICS from the file would create that placeholder in the database if it was not previously present.
Knowing that the ICS is a safety backup/restore mechanism, I would expect the same.
So, if the placeholders were, however, created before in PSU and if PSU for some reason does not recognize that its own placeholders are identical with the tags, then perhaps this is what causes the conflict.
Perhaps, but unlikely.

Frank, I'm still not entirely clear if you have tried something simpler: turn ICS reading OFF (as well as auto-sync and ICS writing), import the images and then run the version detection with your Original and Digital Negative custom filemasks specified in the global preferences. Could you please simply say if this simple sequence creates or not version sets with the correct placeholders? (No need to add extra steps or complexity, please.)

If this does not work, could you try exactly the same sequence on a set of images which do not have any ICS version tags at all? (If you don't already have such ICS-free images, I guess you could produce them by forcing a manual sync with ICS writing turned OFF.)

Good luck!

fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 03 Jun 15 21:04

Hi Vlad,
Frank, I'm still not entirely clear if you have tried something simpler: turn ICS reading OFF (as well as auto-sync and ICS writing), import the images and then run the version detection with your Original and Digital Negative custom filemasks specified in the global preferences. Could you please simply say if this simple sequence creates or not version sets with the correct placeholders? (No need to add extra steps or complexity, please.)
I tried it. It does not work. Same result: some NEFs get Placeholders assigned, some DNGs too, running it again removes some already assigned and assigns some others. Inconsistent. I cannot recognize a pattern, but can only assume it is a conflict between placeholders that exist in the files and PSU trying to assign the same ones again.
If this does not work, could you try exactly the same sequence on a set of images which do not have any ICS version tags at all?
I tried it. It does not work. From a total of 213 imported files 71 JPGs get correctly assigned to the main version, but 64 (not 71) get assigned to the Digital Negative Placeholder and only 7 NEFs to the placeholder "Original".
(If you don't already have such ICS-free images, I guess you could produce them by forcing a manual sync with ICS writing turned OFF.)
That is how I generated those files without ICS to be imported. Looking at the ICS tags in PSU they were empty after the manual sync. But I am no longer sure what to believe. I probably should have checked it with ExifTool to really see if the ICS is completely gone, but I am really getting a bit tired of trying and trying and trying...
Also, I am starting to get worried about XMP metadata corruption. Obviously all these files are still part of my IDI5 database. So far I have played with only few folders for testing purposes. Their metadata (in particular the ICS data) are now probably all quite messed up. I do hope other metadata have not been affected. Just to be on the save side I will now go back into IDI and use it to write back all XMP to the files. As long as the move to PSU does not work, I will need to make sure I can continue to use the IDI database (yes, I do have a backup of it all too, but I'd rather not have to go down that route...).
Now, I have one final attempt left: Creating a completely new, empty database, import all labels again using Hert's script and then import the files with auto-sync off, ICS reading on, import versioning off, but no custom placeholders created in preferences. Perhaps there is a conflict still (even despite having completely deleted the custom placeholders) in my existing database. Perhaps the conflicting placeholders are still there somewhere behind the scenes.
That's indeed a theoretical possiblity, but it's a very long shot. Don't be disappointed or frustrated if that fails too.
I tried it. It failed. The placeholders are not created when the ICS is read.

I can only conclude that there must be two bugs "nicely complementing one another".
(1) ICS custom placeholder tags are not being read to the database correctly and if present in the files cause conflicts with placeholder assigning rules set in preferences during version detection.
(2) Assigning custom placeholder with file naming masks set in PSU preferences on unversioned image files does not work consistently either.

The first one is a new bug, which I will file separately now, the second one is essentially the same one I already filed before.
Not sure how to keep all this simple though I have tried:
http://bugs.idimager.com/view.php?id=2870

Finally: I think your workaround to import major batches and assign the placeholders with Mike Buckley's method would work, but I am not decided, if I should actually start down this route. Now even more so than before. It seems that the more I try to get versioning to work consistently the more problems I do encounter. I am really not sure if it is worth working a few days getting all my images into PSU and then encounter yet another bug with some aspect of the versioning still not working...
I really do hope all this can be resolved soon.
:(
Cheers,
Frank

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 03 Jun 15 21:49

Hi Frank,

Thanks for the update and I'm sorry to hear that all your attempts to nail down the correct placeholders have failed. Quick questions:

How do you get the exact number of all images with a certain version placeholder? Are you simply looking at the counters in Catalog->By Version Placeholders? That's a very long shot too, but could it be possible that the placeholders are actually correctly assigned but the displayed numbers are wrong?

(I'm asking this not to give you any false hope, but because I did see problems with wrong (stale) counters/info before, especially in Catalog->By Image Details.)

Cheers,
Vlad

fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 03 Jun 15 22:29

How do you get the exact number of all images with a certain version placeholder? Are you simply looking at the counters in Catalog->By Version Placeholders? That's a very long shot too, but could it be possible that the placeholders are actually correctly assigned but the displayed numbers are wrong?
Yes, I am looking there first, but then I also check if the thumbnails have the placeholder icons and using the following code for the custom file info below the the thumbs I am also able to see if the tags are there or not:

Code: Select all

<body bgcolor="%code
if Catalog.ImageIsVersion(ImageItem) then
  result := '#00FF00'
else if Catalog.ImageHasVersions(ImageItem) then
  result := '#FFFF00'
else
  result := '#000000';
%/code"><FONT size="10"color="#000000">%CatalogPlaceholderList</FONT></body>
(I'm asking this not to give you any false hope, but because I did see problems with wrong (stale) counters/info before, especially in Catalog->By Image Details.)
That this could be a counter refresh problem was one of my first thoughts when it all started, but unfortunately it is not...

Cheers,
Frank

tstoddard
Posts: 584
Joined: 07 Sep 12 12:51

Re: Custom XMP in Info vs Details Panel?

Post by tstoddard » 03 Jun 15 22:45

I have to say, after following this thread, not too closely, I'm glad my workflow doesn't depend on custom fields and placeholders.

I do have one question for Frank, which may be a stupid question, but if your placeholders are based on file extensions, why do you need placeholders at all? Can't you accomplish the same thing just by filtering on file extensions or file types? I suppose at some point in the future you might start using a different camera or Nikon will change their raw file extension and maybe there will even be a different file type to use as a digital negative but for right now what difference does it make? Even if the extensions change, you can filter on multiple file types pretty easily just be using the filter bar.
Tom Stoddard

fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 04 Jun 15 9:53

Dear Tom,
no question ever is a stupid one :wink:

Keeping the NEFs & DNGs versioned with their main JPG is simply more convenient for my workflow.
I know that most people today have a non-destructive workflow first introduced by lightroom, i.e., that you generate derivatives of your main files on-the-go according how you need them. In programs like lightroom you simply develop a web version, a print version, etc. when you need them and else keep the raw file.
Unfortunately I started with all this years before programs like lightroom existed and thus have a lot of "legacy baggage" to carry along.
For most of my older files I do not simply have three versions (JPG+DNG+NEF), but for many of my "keepers" I often also have at least one more, the print version of the file that I routinely name "*_print.jpg". Occasionally I do have a second one that I call edited "*_edited.jpg" (just like the placeholders for the NEFs and DNGs are not assigned, the ones for these other custom versions are also not being assigned correctly - I did not want to go into that to avoid making things even more complicated). For very files I even have a few more versions...

Until now using versions was a simple, easy way to know what files belong together. Now, you say that apparently the versioning itself is not the problem. In PSU the files are correctly being recognized as subversions - they are just not assigned to their correct placeholders. Now, while the custom version place holder rules for the DNGs and the NEFs are relatively simple (= the file extensions), having to keep a set of different filter rules for all other custom versions of developed JPGs is certainly more cumbersome. It would obviously be more straight forward simply to be able to continue using version placeholders, and a reason why I want to switch to PSU is to do things more simple, not more complicated.
And finally, you may call that nit-picking: it certainly defies the purpose of having versions and not being able to use them for finding and filtering files, instead being required to use the custom filter masks as filters. One might then as well ask, why PSU even still offers that functionality at all.

OK, and now this is getting a bit philosophical...
Insert "Off-topic" warning here :D

You may say that I should re-assess the way I have been doing things and I am actually testing several raw converters and thinking about re-designing my workflow in the future. Still, I have thousands of original files and their edited versions. I don't want to loose that legacy data only because with non-destructive editing it is theoretically now possible to abandon the idea of having to keep a set of versioned files together.
Evaluating raw converters with that objective has been a sobering experience though. Each program has its very own, proprietary way of developing raws. And, seeing what has recently happened to Aperture, who knows which one will survive the fight for dominance among raw converters? So, just keeping the recipe for how to develop a file with a particular raw converter at first seems like a great alternative to bogging down your hard drive with derivatives. It avoids clutter and saves on hard drive space - only developing the files when you actually need them!
But, once a raw converter has reached an evolutionary dead end and software companies that develop that program decide for whatever reason to abandon it, then you are stuck. And you might then think: had I only developed all those files and then kept those developed versions! Or: when your favorite raw converter actually goes "bellies-up" you get into a hectic frenzy and start developing them all? Or you decide going the safe-route, not choosing the raw converter that gives you the best results (Capture One, DxO?), but instead go with one that will be around for years to come (Lightroom) because of its current market dominance...!?

In any case: for now I am not quite decided that I should completely abandon my version-based workflow. It has served me well for many, many years. And: I'd certainly like to keep my legacy data (at least migrate them into PSU).

Cheers,
Frank

tstoddard
Posts: 584
Joined: 07 Sep 12 12:51

Re: Custom XMP in Info vs Details Panel?

Post by tstoddard » 04 Jun 15 12:23

Frank,

Thank you for taking time to so thoroughly answer my question. For the record, I was not questioning your use of versioning, I understand its value, I was questioning your use of placeholders. Considering the scenario you had been using where you only had a main, original and digital negative, I didn't see the need to use placeholders at all. Now that you've explained your need to keep track of other versions that have the same file extensions it makes sense. I can certainly understand using placeholders to identify versions meant for certain purposes like printing, emailing, the web, etc. but to create and use a custom placeholder just to identify the fact that the version is a dng file doesn't seem to add any value. I do hope you manage to get these issues resolved.

Also, I understand your reluctance to rely on a "non-destructive" or "parametric image editing" raw converter to maintain different versions of an original image. In theory, that's a great idea but, as you've pointed out, you are then locked into a specific program forever and if that program's life cycle comes to an end, then what? The same could be said for using versioning in PSU. What other program could make use of the versioning information that PSU embeds in our files? This is why I always strive to maintain consistent naming conventions and folder structures, as have you.

Good luck getting this all sorted out. We all benefit from your (and vlad's) efforts. Thanks!
Tom Stoddard

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 04 Jun 15 12:29

Hi Frank,

I have good news: I wrote you a custom script for fixing the mess left behind by the version detection and/or the ICS reading. (It took me several solid hours of sifting through the API and trying things out.)

The script will look at all NEF files in the current selection and assign them the 'Original' placeholder, if that is not already assigned. It will also remove any other placeholder assignments that you might have. Similarly, it will look at DNG files and replace the currently assigned placeholders (if any) with the 'Original' custom placeholder, if needed. At the end, it will report how many NEF files and how many DNG files it has fixed.

Prerequisites:

- The 'Original' and the 'Digital Negative' custom placeholders must already be defined
- It is your own responsibility to ensure that the current selection contains NEF and DNG files. (Tip: when you're working in a catalog view that displays the main version only, you could still expand the version sets of selected thumbs: Ctrl+Alt+V.)
- Every (File.JPG, File.NEF, File.DNG) triplet must already form a version set, with the JPG file being the main version. (The script checks for that and will warn you of any non-versioned file for the given extensions.)

You should therefore probably run the script on the images imported with ICS reading on, which correcly create the version sets, even if the versions don't have any placeholders assigned. (Alternatively, you could run the script after the version detection, if that also creates the right version sets, even with incorrect/inconsistent placeholders; the script will fix the missing or wrong placeholders!)

Try it out and let me know how it goes. Good luck!

Vlad

P.S. @Hert: is there any way to programatically define placeholders? (I'm talking about placeholder creation, not assignment.)
Attachments
Assign Placeholders By Extension.psc
Custom script for fixing the NEF and DNG placeholders, as desired by Frank
(5.84 KiB) Downloaded 72 times

Mike Buckley
Posts: 4561
Joined: 10 Jul 08 14:18

Re: Custom XMP in Info vs Details Panel?

Post by Mike Buckley » 04 Jun 15 13:58

Frank,

If you don't get the placeholder fiasco resolved, I wonder if my use of color labels would be helpful to you. You have seven colors available to you, the normal five colors, the non-color label and the silver color label produced by using a name in the Details panel's "Color Label" text field that does not match any of the names you have configured to correlate with each color. You could then easily filter on seven color labels. I don't know if it's possible to filter on the silver label combined with other labels, though I assume the "Invert" setting allows you to filter on the silver label by itself.

fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 04 Jun 15 14:00

Dear Vlad,
thank you so much. The script works!

(1) Import with versioning off, but ICS reading and writing on, autosync on.
(2) The files are correctly versioned with JPG main, DNGs and NEFs subversions (that lack placeholders).
(3) Filter & select all DNGs and NEFs that were imported, running the script on these files.
(4) The PSU database now shows the correct placeholder assigned to the images, but the ICS tag has not yet been written to the file because autosync does not recognized that XMP of the images was touched.
(4) Saving the metadata to the files (CTRL+S) writes the tag to the files because ICS writing is on.

And I have just tried it out: the script even works if I apply it to all files, even the JPGs. So. what I will try now is seeing if I can apply the script as a batch directly after the import...

I think this seems to be a viable way now to move forward starting a bigger import. As for my other custom versions (see the discussion with Tom above), it may be the best to do that semi-automatic according to Mike Buckley's method... (and of course I do hope that the bug will eventually be resolved and custom versioning will get fixed...).

OK, what I still now need to to is do some more checking if all other metadata comes across correctly from IDI. Better make sure it works before staring my major import :wink: .

Again: Thank you!

Cheers,
Frank

fbungarz
Posts: 3304
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: Custom XMP in Info vs Details Panel?

Post by fbungarz » 04 Jun 15 14:03

Hi Mike,
I am using all the color labels already, so unfortunately that doesn't really work.
Thanks though.
Cheers,
Frank

vlad
Posts: 969
Joined: 01 Sep 08 15:20

Re: Custom XMP in Info vs Details Panel?

Post by vlad » 04 Jun 15 15:17

fbungarz wrote:Dear Vlad,
thank you so much. The script works!
Dear Frank,

Didn't I tell you that you were 90% to the Photo Supreme destination? (I just didn't mention the 90/10 rule: the final 10% takes 90% of the effort.) And I had almost made it a personal goal to help you get there in 100 posts or less :) It's therefore heartwarming - and a bit mindnumbing - to hear from you (on exactly the 100th post of the thread) that the versioning thingy finally works! :D

The saga is not yet over though, so I still wish you good luck!

weidmic
moderator
Posts: 4500
Joined: 04 Dec 06 22:21

Re: Custom XMP in Info vs Details Panel?

Post by weidmic » 04 Jun 15 19:18

And I had almost made it a personal goal
:)
PSUServer 4.x, PostgreSQL 10.x
My homepage http://www.michaelweidner.com
PSU Tips and Tricks http://www.michaelweidner.com/WP/psu/

weidmic
moderator
Posts: 4500
Joined: 04 Dec 06 22:21

Re: Custom XMP in Info vs Details Panel?

Post by weidmic » 04 Jun 15 19:18

It was really very interesting to follow this threat...
PSUServer 4.x, PostgreSQL 10.x
My homepage http://www.michaelweidner.com
PSU Tips and Tricks http://www.michaelweidner.com/WP/psu/

Post Reply