cataloging with PSU is a breeze!

[fbungarz]

I guess I don't have a reputation for handing out praise easily, so here it comes:
After finally having migrated my catalog from IDI to PSU and using PSU to catalog images, I must say I am much impressed by the speed how quickly labels can be assigned, metadata are written, label sets can be organized.
That in combination with much more powerful ways to search labels and metadata quickly makes cataloging immensely efficient. I find myself really enjoying this. Cataloging with PSU is a breeze!

[PhilBurton]

Frank,

Did you copy your IDI catalog into Photo Supreme or create a new catalog in PSU based on all the metadata in your images? (Or is this the wrong question to ask?)

I think most users tend to complain/critique than to praise a product. i think it's rare on ANY forum to see a post like this one.

Phil Burton

[Phrans]

"I think most users tend to complain/critique than to praise a product."

It would not be very informative to have a forum with just compliments on a product.
I wouldn't trust such a forum.

I do think there are a lot of implicit and explicit compliments in a lot of posts here, throughout the years.
Hert must (and can) be a very proud person!

Continuously improving the software can only be done with the input of the users, don't consider such input as complaining/criticism.

[fbungarz]

Did you copy your IDI catalog into Photo Supreme or create a new catalog in PSU based on all the metadata in your images? (Or is this the wrong question to ask?)

Not the wrong question at all...

I tried both ways (quite an Odyssey actually, here the short version...):
(1) migrated the catalog, opened it in PSU, turned out to be a complete nightmare, my custom metadata no longer worked
(2) because I have a HUGE catalog with thousands of labels mapped to custom XMP Hert kindly provided a script to export and re-import this label structure into a brand-new catalog
(3) I tried importing all images again, one folder at a time, though metadata read/write now worked fine, my versions were a complete mess...
[filed a bug in Mantis that one year later remains still unresolved...]
(4) Vlad wrote a script that would at least assign the DNGs and NEFs to placeholders correctly
(5) ran out of time ...
(6) about a year later: tried it all again using Hert's and Vlad's scripts; this time again a complete mess, almost ready to give up
(7) disabled ICS writing AND reading, Autosync-off, Versioning off, afterwards ran Versioning and used Vlad's script, tested if metadata write/read worked now - it did
(8) verifying the folders showed: almost every single import some images were missed
(9) almost ready to give up again: importing several hundred folders one at a time, running versioning and Vlad's scripts, verifying and still not there? Give me a break!
(10) Finally got a brilliant idea: ICS in IDI and PSU are NOT compatible!!! Though I had filed a bug report about this in Mantis, I did not actually understand the implications of this. If IDI writes ICS to the images and PSU tries to read it, but does not understand it, all efforts trying to import the folders AND read the ICS (portfolios, versions, etc.) will result in the mess I experience, however: if I migrate the complete catalog and then turn ICS off (!!!), it might actually work...
(11) it did and still does

It would not be very informative to have a forum with just compliments on a product.
I wouldn't trust such a forum.

Well, actually I have been particularly bad. For my workflow migrating PSU really has been (and still is) quite a challenge. PSU lack several key features that I regularly use and have been quite accustomed to in IDI. Finding workarounds and alternatives continues to be a challenge. So, I have been complaining a lot about this and Hert's patience was/is still strained a lot. Still, though I have looked for alternatives, there are none. And: not migrating, being stuck in a product no longer actively being developed is not an option either...

So, I am glad I finally have at least a complete and functioning catalog. And for the cataloging, it works very well!

[Mike Buckley]

if I migrate the complete catalog and then turn ICS off (!!!), it might actually work...
(11) it did and still does

Regarding the ICS, are you referring to reading, writing or both?

[fbungarz]

Regarding the ICS, are you referring to reading, writing or both?

Both:
In IDI I tried turning only writing off, but it seems that unless one exports XMP to the image again the IDI-ICS schema remains there and is still read by PSU.
So, "deleting" the IDI-ICS did not quite work (also: it would have required to use IDI to write XMP to more than 100.000 image again, not ideal...)

So, I thought only turning reading ICS off in PSU should do the trick. That seems to work with Import if autosync is "off" too. If "autosync" is turned on the ICS somehow still goes into the way (don't ask me how, I have given up trying to understand it...).
However, since import regularly misses a couple of files, I had to use "verify" on the folders after the import and, no matter how it is configured, using verify somehow sill reads the ICS. Again I have no idea why... (it should NOT if reading is disabled and writing enabled in Preferences).
So, the only thing that actually worked trying to re-create a new database from scratch was turning ICS off completely, both reading and writing! Once I managed to successfully import a folder, I was able to turn writing ICS on again (for that folder, after forcing an XMP re-write to all images in that folder via CTRL+S). This gave me the clue that converting the entire database might actually work if I turn off ICS. It does. I have not enabled it again since. I assume that writing the metadata to all files again with ICS turned on in PSU in theory should replace the old IDI-ICS schema with the new PSU-ICS data. Still, I really don't quite trust this...

I can only assume that there are a couple of bug causing a mess with migrating IDI-ICS to PSU-ICS using custom XMP (autosync, ICS read/write settings, import, verify). The only thing that routinely and consistently works if not using ICS at all.

[Phrans]

Well, actually I have been particularly bad. For my workflow migrating PSU really has been (and still is) quite a challenge. PSU lack several key features that I regularly use and have been quite accustomed to in IDI. Finding workarounds and alternatives continues to be a challenge. So, I have been complaining a lot about this and Hert's patience was/is still strained a lot. Still, though I have looked for alternatives, there are none. And: not migrating, being stuck in a product no longer actively being developed is not an option either...

So, I am glad I finally have at least a complete and functioning catalog. And for the cataloging, it works very well!

I guess we're in the same boat...
I was really, really happy with IDimager for almost 7 years (intensive use!). But performance/stability problems worsened, so sticking to it was not an option anymore. The conversion to PSU went quite OK, with a little help from Hert. Some flaws/bugs made me almost get second thoughts, but I think they will be solved soon. So we're up and runnning again (intensive use! ), and I hope it's for another 7 years. Being a satisfied customer for so many years is the biggest compliment to IDimager / Hert!

To the moderators: this thread is getting sort of entangled...

[vlad]

Hi Frank,

I still remember your previous migration attempt - it was quite a saga! Congratulations on your successful transition this time around! (Btw, I'm glad you've still found my custom script helpful, as I don't think it''s ever going to suit anyone else; as a matter of fact, that may also be true for the custom script from Hert.)

Regarding the ICS, are you referring to reatding, writing or both?

Both:
In IDI I tried turning only writing off, but it seems that unless one exports XMP to the image again the IDI-ICS schema remains there and is still read by PSU.

That makes sense (for both IDI and PSU): disabling the ICS writing will not touch previously created ICS. (Come to think of it, one may end up with stale ICS data by enabling ICS writing and then disabling that option at some point That suggests ICS writing should be a permanent commitment. )

So, I thought only turning reading ICS off in PSU should do the trick. That seems to work with Import if autosync is "off" too. If "autosync" is turned on the ICS somehow still goes into the way (don't ask me how, I have given up trying to understand it...).

What do you mean by "going into the way"? I haven't actually verified it, but here is my hypothesis: once ICS has become part of the image metadata, you may see it in the catalog (if you're looking for it), no matter how you set the ICS options. But the ICS metadata shouldn't have any tangible effect, assuming you set ICS reading off. (If I'm off the mark, then please correct me.)

However, since import regularly misses a couple of files, I had to use "verify" on the folders after the import and, no matter how it is configured, using verify somehow sill reads the ICS.Again I have no idea why... (it should NOT if reading is disabled and writing enabled in Preferences).

Again, you need to clarify if by "still reading the ICS" you simply mean that:

(1) you are able to see the ICS metadata in the Image Details -> Advanced XMP
OR
(2) the ICS metadata has some visible effect (such as label or version assignment)

I wouldn't be surprised by (1), but I would certainly consider (2) to be a bug. Please clarify.

So, the only thing that actually worked trying to re-create a new database from scratch was turning ICS off completely, both reading and writing! Once I managed to successfully import a folder, I was able to turn writing ICS on again (for that folder, after forcing an XMP re-write to all images in that folder via CTRL+S). This gave me the clue that converting the entire database might actually work if I turn off ICS.

I don't follow that logic. If you were able to turn ICS writing on (with no problems, presumbably), then why would you conclude that you had to turn ICS writing off?

It does. I have not enabled it again since.

Again, I'm confused.

I assume that writing the metadata to all files again with ICS turned on in PSU in theory should replace the old IDI-ICS schema with the new PSU-ICS data. Still, I really don't quite trust this...

Yes, it should. Why don't you try and confirm the procedure on a single image?

(Now that you're happy with your catalog transition to PSU, I don't see how you feel comfortable knowing that you still have stale ICS metadata, written by a product which you have left behind and which is no longer supported.)

I can only assume that there are a couple of bug causing a mess with migrating IDI-ICS to PSU-ICS using custom XMP (autosync, ICS read/write settings, import, verify). The only thing that routinely and consistently works if not using ICS at all.

Are you in effect saying that having the old ICS metadata written by IDI is fine, but turning ICS writing on in PSU would still create problems? You may be right, but I don't understand at all why.

[fbungarz]

HI Vlad,
honestly, I don't really know how to answer your questions. Simply, because I really don't understand myself what is happening. All I can say, with ICS (both writing and reading) turned off it now works - finally!

one may end up with stale ICS data by enabling ICS writing and then disabling that option at some point

Yes, I would love a script that actually deletes all ICS from my files to make sure the stale IDI-ICS not again gets into the way. In theory turning on ICS in PSU now should overwrite the out-dated IDI-ICS. But I am not really sure that it actually does. I'd rather not experiment with it again...
Ideally I'd like a script that deletes all ICS from the files and only then do I feel safe to turn ICS writing in PSU on again.

once ICS has become part of the image metadata, you may see it in the catalog

Of course I can see it is there under "Advanced". Unfortunately there is now way to see if it is old IDI-ICS or new PSU-ICS.

If "autosync" is turned on the ICS somehow still goes into the way

I have no idea why it gets into the way. I can only speculate. Fact is, if i have autosync on during the import AND only ICS writing on (not reading!) the database gets corrupted and writing to XMP no longer works. ONLY if both, writing and reading, are off does it work. It does not make sense, but that is what I experienced.

Again, you need to clarify if by "still reading the ICS" you simply mean that:

What I mean is definitely no. (2):

(2) the ICS metadata has some visible effect (such as label or version assignment)

The effect simply is that I end up with a database where writing XMP is a complete mess.

I think there is definitely a bug lingering around here somewhere, but it is clearly beyond me trying to figure out where!

I first thought that the custom XMP that I was using did not work. Then, because of Hert's insistence that he could read and write "my" custom DwC metadata fine, I was convinced that converting the IDI database to a PSU database was to blame. Thus, I was really happy that Hert wrote a script that would export my catalog labels to a clean database. Unfortunately, even with a clean database I was still running into problems, particularly with the versions (as you recall).

Now, with my new migration attempt I still ran into huge problems. I had imported all folders with images shot in 2006 into a clean database and suddenly XMP writing/reading turned out again to work no longer. About a year ago I had only experimented with a few more recent image folders. Then I did not run into these problems, XMP worked just fine. Migrating my versions was the problem...
So, now I was ready to work on a detailed screenshot video to document the mess. Then it struck me that all these issues might be caused by "corrupt" legacy ICS data in the files. Clearly the data are not really "corrupt", but the two versions of ICS that PSU and IDI write are simply incompatible.
(Well, experimenting quite a bit with various settings I can only assume this is the case and that PSU interprets that the IDI-ICS is just fine). In any case: reading the old, legacy ICS from IDI into a PSU database results in a complete mess (at least in my case!).

Actually, I am really tired of trying to figure out why specifically this is the case. I actually don't think I even can! Troubleshooting this is simply beyond me. I have given up trying. Perhaps what I experience is related to the way my IDI database has been set up? Perhaps it only applies to some XMP embedded in fairly old files (the ones from 2006?). Perhaps it has to do with the custom XMP that I am using? Who knows? I don't ...
It is simply too complex. Also, I quite understand that this must be extremely low on Hert's priority list. Apparently now one else has this ICS legacy data problem that I experience. If one user only experiences these problems, why bother...

I assume that writing the metadata to all files again with ICS turned on in PSU in theory should replace the old IDI-ICS schema with the new PSU-ICS data. Still, I really don't quite trust this...

Yes, it should. Why don't you try and confirm the procedure on a single image?

I am simply not convinced this is working properly. My experience is that autosync on and ICS writing on is a dangerous combination. I might again end up with a database where XMP no longer works.

I don't see how you feel comfortable knowing that you still have stale ICS metadata, written by a product which you have left behind and which is no longer supported.

I am not feeling "comfortable" about it at all!
BUT
I cannot afford to have my existing catalog being corrupted again! That's why I'd love to have a script that definitely deletes all old, "corrupt" IDI ICS from the files. Only then will I try turning ICS writing on again. I will not try to do this manually. For now, I prefer to have both ICS writing and reading turned off.

Are you in effect saying that having the old ICS metadata written by IDI is fine, but turning ICS writing on in PSU would still create problems? You may be right, but I don't understand at all why.

That is exactly what I am saying. Writing ICS in IDI has never been a problem. I can only assume that PSU does not interpret the IDI-ICS correctly. And somehow having autosync turned on in PSU and trying to write ICS also does not work. No idea why...
Really - I have lost weeks struggling with this, wrecking my brain trying to understand, why this does not work. I now found a setting that does work. Fortunately.

[Mike Buckley]

And somehow having autosync turned on in PSU and trying to write ICS also does not work. No idea why...

My Supreme catalog was built from the legacy IDI catalog. In the Supreme catalog, I have always left Auto sync and ICS writing turned on without experiencing any problems. Probably the main difference between my catalog and yours is that I don't use custom XMP and you do, so my guess is that the problems you experienced are related to something going wrong with the custom XMP. Just a guess.

[vlad]

Frank, I don't want to drag this on (especially if it brings back flashes of corrupted metadata and other nightmares), but I'll make a simple suggestion: if the combination of autosync and ICS writing doesn't work for you, then why don't you give a thought to turning the auto-syncing off, rather than the ICS writing? At least, you could disable the auto-syncing as a stepping stone, until you write the fresh ICS metadata (via manual saving of metadata, assuming that works fine) for all your images. IMHO, only at that point you will have completely crossed the long and winding bridge from IdImager to Photo Supreme.

More generally: I may be in minority here, but I still regard auto-syncing OFF as the safer setting (when combined with regular backups + metadata saving), as it allows you additional control (when to save the metadata) and the opportunity to inspect what's actually happening when manually syncing (Ctrl+S). (Granted, Ctrl+S also triggers certain catalog updates before saving the metadata to the physical files, so you need to learn what to expect.)

[fbungarz]

so my guess is that the problems you experienced are related to something going wrong with the custom XMP.

Dear Mike,
I agree. In fact, I do think this very likely is the case.
It might have to do with the history of setting up the DwC custom xmp. It is structured XMP, i.e., some of the XMP fields are nested below others. When I first tried to set this up IDI had a bug, that did not allow structured XMP to be written. Hert later fixed that bug, but perhaps during that process some of the XMP in my earlier files got corrupted. Clearly IDI never complained about it, but now PSU apparently does. This might also explain why PSU did not complain last time when I imported a few of my recent files into a clean PSU database (the one, where Hert helped me migrating the mapped label data with his script). Though I ran into trouble with versioning, XMP writing and reading back then seemed to work just fine.
This time though, when I decided to migrate everything, PSU very early on started to act funny. My files are sorted into folders according to dates and of course I started with some of the earliest folders, those that contain the oldest image files. These files are also the ones carrying around most of the legacy metadata, initially written with some of IDIs earlier versions. Who knows, perhaps they carry around some baggage ICS that PSU now no longer understands even though IDI is not bothered by it...
It is interesting that, when I migrated the whole catalog the first time and now when I started to migrate these earlier folders again, in both instances I ran into these XMP issues. Of course I first thought my custom XMP was to blame. When Hert provided the script, I thought the situation was resolved. The trouble with versioning then sidetracked me. Trying to resolve this again was a challenge (in fact versioning in PSU still does not work correctly, the custom file masks, how the placeholders are supposed to be recognized are pretty much ignored by PSU; it would be nice to know that eventually gets fixed).
Having migrated now, finally with a working catalog version, I have spent the last few days reviewing label and XMP data folder per folder taking this as an opportunity to add missing information, in the process re-writing quite a bit of the IDI's legacy metadata with PSU. I can only hope this to be a bit of a cleansing process. Clearly I cannot expect Hert to check metadata of thousands of my files for inconsistencies. Still, I would be worried less about "stale" ICS (as Vlad calls it) if I had a script that definitely would delete the old IDI-ICS from all files...
Cheers,
Frank

[fbungarz]

Hi Vlad,
believe me, I experimented with all kind of settings. It now works. I think I have narrowed it down as much as possible to corrupt/incompatible ICS (though I still cannot explain what exactly has been going on). Until I am really sure there is no "stale" ICS hanging around anymore, I'll not turn it on again

More generally: I may be in minority here, but I still regard auto-syncing OFF as the safer setting

I have actually used it for quite a few "generations" of IDI. When it first came out, it was quite buggy and an enormous resource hog. I am really impressed how smooth it now works in PSU. It seems it still misses a few files and one ever so often has to check "catalog state" for unsynced files, but using PSU quite intensively the past days, I am no too worried...

Cheers,
Frank

[vlad]

Hi Frank,
Just a long shot regarding the "stale" (legacy) ICS metadata: couldn't that be cleared in the Image Details panel? (Probably not - I remember that only certain metadata can be modified or removed - but it doesn't hurt checking.)

Cheers,
Vlad

[fbungarz]

couldn't that be cleared in the Image Details panel?

Nope! As you correctly point out, the panel does not allow for modifying all data...