Can I import key words at a top level?

[vlad]

Hi Mike, that's an excellent question! I can only speculate, but it is possible (and I guess it would be a relief if) Neil's read settings were not set up properly. (The read settings do matter for keyword processing during an import session, don't they?) Also, I have no idea how eactly Neil imported his pictures: I can see there is a specialized "Import from Lightroom" action (in the Data menu), but I don't know how it works and if Neil used that import or the regular import. (Oh, wait: there's also the import wizard vs. the import profile, so there are some details missings from the picture.)

But something keeps telling me that there must be a way for Neil to achieve what he wants in PSU!

[Mike Buckley]

1) Get back to us with your Keyword processing setting. If it's not set to "Merge...", please set it as such, start with a brand-new (empty) catalog, import your keyword file and only subsequently import your images.

I recommend that Neil also makes sure the setting, Read hierarchical keywords, is also enabled, even though it seems from the screen shots that it is already enabled.

All of that is the only other thing to try that I can think of. Even so, I may be misunderstanding something but it seems to me that it shouldn't matter in this situation whether the Read configuration is set to merge the keywords with the catalog labels or to replace the catalog labels. That's because when your catalog hierarchy matches the metadata stored in the image files or when the data in the catalog is not yet assigned to any images, the import results should be the same. I'm baffled.

there must be a way for Neil to achieve what he wants in PSU!

Absolutely!

[vlad]

Looking again over Neil's screenshots, I have some additional remarks:

1) IIUC, the labels immediately under the 'Label Keywords' category (such as 'art' and...ahem, 'CATEGORIES') are duplicated by top-level categories (rather than labels). However there is no way to merge a label to a top-level category - for each duplicated category, Neil would therefore have to merge all its child labels (...and even their descendents, until #2639 is implemented).

2) Unfortunately, there is no multi-label (many:many) merging implemented. Multiple labels can already be merged to the same destination label - but M:N (or even N:N) label merging is not possible.

3) Although it is likely not trivial to implement reliably (and therefore Neil shouldn't be holding his breath), I think it would be truly amazing if a combination of relocation+merging (call it: smart relocation, or smart merging) was implemented! This would allow the user to select multiple labels (perhaps even categories?) and drag them over a label container (either a label or a category), with the effect that the labels with matching names are merged (recursively) inside the target container, while the non-matching labels are relocated (added) to the target container. Before someone says that it's a lot of work for only a handful of fringe cases (not sure if Neil's case qualifies as such, as I remember other migration questions too), let me state my prediction: if this was to be implemented, the support for reliably merging entire PSU catalogs (which has been discussed before, for example in the context of Travel mode use) would be only one step away!

Anyway, I guess discussing what could be implemented (longer term) is not much help to Neil (sorry). Like Mike, I believe Neil's best bet is to end up with a single hierarchy upon the imports, with no de-duplication subsequently needed. There must be a way.

[HCS]

...

3) Although it is likely not trivial to implement reliably (and therefore Neil shouldn't be holding his breath), I think it would be truly amazing if a combination of relocation+merging (call it: smart relocation, or smart merging) was implemented! This would allow the user to select multiple labels (perhaps even categories?) and drag them over a label container (either a label or a category), with the effect that the labels with matching names are merged (recursively) inside the target container, while the non-matching labels are relocated (added) to the target container. Before someone says that it's a lot of work for only a handful of fringe cases (not sure if Neil's case qualifies as such, as I remember other migration questions too), let me state my prediction: if this was to be implemented, the support for reliably merging entire PSU catalogs (which has been discussed before, for example in the context of Travel mode use) would be only one step away!

...

I'm not trying to dismiss the capabilities of the IDimager development team nor your analysis, but this would actually be the only sane way of implementing such a feature. Now, i couldn't do it myself, not being a programmer. But, i've dealt with this situation quite a couple of times in my job (i support business systems, ERP and the likes) in large transition projects. If you can't get relocated data to merge with the target data, then there really is no point, right?

Again, i'm saying this with respect to the development team, because PSU is fine as it is and it's being improved all the time. The use case of moving labels to another branch and leaving them side by side with a branch that happened to exist with the same name is escaping me.

[vlad]

Hi Neil,

Since I enjoy programming, I have tried my hand at writing a label de-deduplicator especially to solve the situation described in your last post (but hopefully it will be useful to other users too).

- The script scans the entire catalog (repetedly, so it may take a very long time for a large catalog and/or many label duplicates) and attempts to rmerge all duplicate labels (meaning: labels with the same name + the same parent), at all levels.
- It does not merge label hierarchies which do not have the same parent - so it should only help you solve the situation with multiple label hierarchies ('agriculture' , 'art' etc.) which are duplicated within the same label or category
- It asks in the beginning if you would like to confirm each duplicate merging - my strong advice (in your case) is to press No, otherwise you may get hundreds or thousands of confirmation dialogs. (And there is no way to cancel them.)
- If it completes successfully (which, again, may take a long time and PSU will not be responsive meanwhile), it will eventually produce a detailed report file. (That report file may contain FAILURE messages, because the underlying merge function calls in the PSU library sometimes do not appear to be succesful, but don't worry too much about them, for now - the script might have actually done its job. The best way to check is to inspect your catalog.)
- Currently, the label name matching is case insensitive (but it's trivial to change this in the script, if anyone needs)
- Please note that this is my very first script written for PSU - it is not vetted by any scripting expert and I am providing it AS IS, so use it at your own risk. (Don't forget to BACKUP YOUR PSU CATALOG before running the script, if the catalog is important for you). FYI, I've learnt the basics and written the script in <2 hours, but I've spent many more hours testing, tweaking and debugging it. (Typical in the software world, especially when your very first program is a bit more complex than "Hello World" )
- Finally, credit where it is due: the script is a rewrite of the Find Duplicates scripts, written by Hert and vailable in the repository. (However, that script is no good for Neil because it simply lists common label names - it does not check the hierarchical info or merge any duplicates.)

Hopefully you'll get a chance to try it - good luck and let me know if it helps you!

Vlad

[ncartwright]

Hi,

Thanks again for all your efforts. vlad, my preferences are set to merge keywords so it is not this. I will certainly add some comments to the ticket you have raised and give the script a go. I was busy yesterday so did not get much opportunity to look further at the issue but I did manage to load my old version of IDImager 5 to see what would happen with this scenario as it is not something I looked at when using IDImager 5 in the past. I observed exactly the same behaviour as with Photo Supreme so I would assume the code has been ported across from that. The annoying thing here is that all other DAM applications do not seem to have the issues I am experiencing here so it does look like catalogue label handling and flexibility is one area that needs to be looked at in Photo Supreme if it is to provide an easy transition from rival applications which seems to be a key selling point

I am still hoping for an improvement in the future as in other respects Photo Supreme looks to be growing into a good replacement for IDImager and one that would meet all my needs with the exception of the keywording limitations.

I will report back if the script proved successful at resolving my situation in the following days.

[vlad]

Hi Neil, I don't know if you've already got a chance to try the script, I'll just add a couple more remarks:

1) I'm not making any checks or guarantees about the label merge direction - the algorithm is currently choosing a rather arbitrary one. This shouldn't be a problem in your case, I believe, but I guess it could be a problem in a scenario where the merged label details are not exactly the same. A general and smart script for merging should check the label details for compatibility and, if the label details are not identical, it should either:
1a) Ask for user feedback.
1b) Determine if the label details are mostly compatiable and apply the label merging in the direction of most specificity (e.g., a public label should merge into a private label, not the other way).
1c) Abort the label merging (which could be a problem if multilevel merging is to be implemented as an atomic operation - I guess all the checks should be done in advance, because I don't think a label merge could be rolled back.)
I could probably improve my script with some of the above, but I don't have time for now and I don't think it's needed for Neil. (N.B. I have no idea if the merging function in the PSU library already performs any such checks - perhaps Hert could clarify.)

2) As already mentioned, the script errs on the conservative side and is not optimised for speed -I should probably build and use a per-level merging queue rather than rescan the catalog for structural changes after each individual merge. But, to fully optimize the script, I probably need some details from Hert on which data structures may change after a label merge - and, again, I don't have the time to work on this. Still, even if the script takes a long time, you should be able to see some labels being merged live in the catalog explorer, if you have it open and expanded with duplicate labels in view.

3) If the script seems only partially successful (not all the duplicates have been resolved), then please try to run it again!

HTH,
Vlad

[ncartwright]

Hi,

I have not been able to look further at this issue for a week or so due to work commitments but have now had a chance to look further.

vlad, I tried your script, it took a significant time to run (around 12 hours) on my label structure and had some success but did not totally work. I think the script itself actually did what it was supposed to do but Photo Supreme is still causing issues which defeated the intent of the script. Some labels were successfully merged but with others I ended up still with two labels, with some images having one of the labels assigned to them and other images having the duplicate label assigned to them. A strange situation given I started with one of the labels being assigned to no images!

Having tried to move all label assignments for the appropriate images just to one of the duplicate labels so I could delete the other manually showed that some images were randomly being associated to one or the other of the duplicate labels each time I tried to move all label assignments to one of the duplicates. I hope this is understandable as it is difficult to describe the behaviour I was seeing as it makes no sense. I therefore think there are some fundamental issues with the way Photo Supreme handles imported keywords or keyword lists.

I have solved the issue for myself by assigning all keywords in Lightroom to an image and then importing that image into Photo Supreme. This was a lengthy process and not ideal but it does mean I now have my controlled vocabulary complete within Photo Supreme. I still have reservations moving to Photo Supreme given that the behaviour I have experienced whilst trying to resolve this issues gives me some concern over the inherent stability of Photo Supreme when it comes to handling Labels. In addition to the issues getting my keyword structure into Photo Supreme it also managed to delete a lot of keywords totally from my images which I had to recover using Lightroom. If more work was done to improve this area of the application I would be far happier. Having thought back to my initial move to IDImager 4 many years ago from ACDsee I think some of the issues I have seen in Photo Supreme I had at that time in IDImager but blamed them on ACDsee.

Thanks for all the assistance.

Neil

[vlad]

Hi Neil and thanks for getting back to us with your experience.

vlad, I tried your script, it took a significant time to run (around 12 hours) on my label structure and had some success but did not totally work.

I'm not suprised by either of these facts. I had started with a fairly fast (naive?) version of the algorithm but then I hit a number of fatal errors during repeated testing and tweaking, so I finally ended up tweaking the algorithm towards the "slowest but safest" extreme. Even so, I indeed noticed that some merging function calls invoked on the underlying PSU library are simply not successful (for reasons and under conditions that completely elude me) - that's why I decided to not turn such failures into fatal errors, while I also recommended running the script multiple times. (In theory, my advice still holds, although I admit it's hardly practical at 12 hrs per running time. Btw, if the script completed without any fatal error and produced a detailed report, feel free to upload it here or on some server if you would like me to have a quick look over it.)

I think the script itself actually did what it was supposed to do but Photo Supreme is still causing issues which defeated the intent of the script. Some labels were successfully merged but with others I ended up still with two labels, with some images having one of the labels assigned to them and other images having the duplicate label assigned to them. A strange situation given I started with one of the labels being assigned to no images!

Yep, I've also noticed strange problems with the label merging in PSU. I'm pretty sure we could eventually work them out with Hert, but that may require significant time + a more technical discussion (perhaps within one or more Mantis tickets).

Having tried to move all label assignments for the appropriate images just to one of the duplicate labels so I could delete the other manually showed that some images were randomly being associated to one or the other of the duplicate labels each time I tried to move all label assignments to one of the duplicates. I hope this is understandable as it is difficult to describe the behaviour I was seeing as it makes no sense. I therefore think there are some fundamental issues with the way Photo Supreme handles imported keywords or keyword lists.

I kind of understand what you're saying and easily believe there are some importing & merging problems indeed.

I have solved the issue for myself by assigning all keywords in Lightroom to an image and then importing that image into Photo Supreme.

Ah, that's a smart workaround! (No, it can't be called a proper solution.)

This was a lengthy process and not ideal but it does mean I now have my controlled vocabulary complete within Photo Supreme.

That's great, congratulations!
(In my opinion, if you are now having a solid base and you give up any subsequent keywording outside PSU, chances are that the major hurdles are behind you and things are going to go pretty smoothly moving forward.)

I still have reservations moving to Photo Supreme given that the behaviour I have experienced whilst trying to resolve this issues gives me some concern over the inherent stability of Photo Supreme when it comes to handling Labels. In addition to the issues getting my keyword structure into Photo Supreme it also managed to delete a lot of keywords totally from my images which I had to recover using Lightroom. If more work was done to improve this area of the application I would be far happier. Having thought back to my initial move to IDImager 4 many years ago from ACDsee I think some of the issues I have seen in Photo Supreme I had at that time in IDImager but blamed them on ACDsee.

I think one key to improving this area is to come up with some detailed and reproducible reports in Mantis. (From then on, it's clearly IdImager's responsibility to act on them.) Granted, this may not be straightforward when it comes to migration and merging problems, which may involve a large number of images and/or labels/keywords.

Regards,
Vlad

[newmikeman]

This is also a critical issue for me, but I'm migrating from a different software but using the so-called Lightroom keyword import feature of PSU.

I should have thought that this was a critical issue for anyone migrating their photos to a new DAM and I'm amazed that no attention has been paid to the issue.

I tried Lightroom but ditched it because it was predominantly an image editor and seemed massively complex for cataloging and retrieving my photos. PSU seems much better with the exception of this half-baked keyword/label management issue. From my point of view as a user there should be no difference between a Category and a Label; the top-level is just a special case and I should be able to drag a label to the top level or drag a category from the top level to become a subordinate to an existing Label or Category.
I can't see the Mantis entry because Mantis has not yet sent me the activation link email. (BTW I used Mantis at my last employer and it was a great ticketing system, especially with a horde of enhancements implemented by our support team).

I think I have solved the issue of importing my keyword hierarchy, but I'm still concerned about the mentions of random mis-assignment on import. We'll see.

Meanwhile here's my solution.

In PSU I'm starting with just the standard Categories, then I import my hierarchical keywords' text file and I get the new Category "Lightroom Keywords". Underneath Lightroom Keywords is the top-level hierarchy of my imported keywords, now Labels - Activities, Events, People, Places, Styles, Things and a few more.
I select the existing Category "Events", and in the middle bit of the screen are squares for each of the Labels under Events. I select all these (Ctrl-A) and press Del, then confirm the deletion.
Now I select the Label "Events" underneath Lightroom Keywords and select all the child labels in the middle bit. I click and drag all these and drop them onto the Events Category.
I select the Events category and lo and behold all my Labels/Keywords in the middle bit.
I repeated this for all my imported keywords at the top-level, of which there were only a dozen, so that wasn't too tedious.
In Places I rather liked the little icons, and as I couldn't find how to set them up I dragged my continents onto PSU's Places.
What was tedious was resolving all the duplicates - if I drag my Europe onto PSU's Places I get all the countries there twice. Inconsistent. Grrr...
Nevertheless it got me there in the end.
Now I can delete the by now empty Lightroom Keywords Category and get on with importing some photos.

I imported a few photos from one folder and some of the keywords were identified but NONE were referenced from the hierarchy just established and ALL were added as new Labels under the Miscellaneous Category.

Synchronize Settings, Read: Read hierarchical keywords ON, Read delimited keywords (|) ON,
Keywords processing = Merge keywords with existing Catalog labels.
Synchronize Settings, Write: Write hierarchical keywords ON,
Keywords processing = Merge Catalog labels with keywords.

I guess I'll have to go and look at that other product Dam something. Pity, as PSU has a lot of good points, and only one show-stopper. So far.

[newmikeman]

Progress... I changed a setting in the source system to write hierarchical keywords with a pipe delimiter. Obvious really, but I was convinced I had done this months ago. Now I have exported a thousand or so photos and it makes a big difference. Like Neil above I imported these images without having set up the hierarchy of labels, and it all looks reasonable now. Once I have exported the remaining thousands I'll import those and check out how things match up.

So let's drop this for now, as the only issue is about importing keywords sensibly and we have a workaround for that. From a commercial point of view I would expect this to be fixed sooner rather than later as it makes a very negative impression on new users who like me are evaluating the product.

Regards
Mike

[vlad]

From my point of view as a user there should be no difference between a Category and a Label; the top-level is just a special case and I should be able to drag a label to the top level or drag a category from the top level to become a subordinate to an existing Label or Category.

I realize your expectation may fit the initial intuition of a new user, but keep in mind that certain PSU categories (Events, Persons, Places) have special semantics (they map the underlying labels to certain metadata fields). Once you become aware of that, you will probably realize that dragging and dropping a category (i.e., top level node) under an existing label or category would raise some questions (e.g., is the special semantics preserved within the new label sub-tree?). For now, it's probably simpler to think of labels and categories as distinct entities (labels vs label containers).

I can't see the Mantis entry because Mantis has not yet sent me the activation link email.

Hopefully you have received it by now.

[newmikeman]

Has PSU been corrected yet, regarding the unfortunate effects observed by Neil and others?

[vlad]

What is your specific use case and what are your specific concerns?

(IIRC, Hert mentioned many months ago that he had improved the label merging implementation and, in particular, his script for merging duplicate labels.)

[vlad]

Btw, you could add a comment to FR #2639, if you're concerned about "unfortunate effects" of label merging.