going out of sync

[vkfoto]

I had 0 out of sync just recently. My images are all stored in folders by date, a year folder at the root and all the images of that year in their own month / day folders.

I was recently labeling some 6000 images in a subfolder of 2007 as well as moving some of them to their correct folders.

When I looked again in the out of sync, it had over 50,000! Some for other years too!

In the catalog state view, how is the out of sync value calculated? Is this a dynamic system, constantly checking the diskdrives to see if there are any changes? I know that PSu recommends that file changes be done within the program but if do something directly in the folder on the disk, how is PSu aware of that?

[vlad]

I was recently labeling some 6000 images in a subfolder of 2007 as well as moving some of them to their correct folders.

I understand from your recent posts that you don't currently have any version sets, right? Next question: did your recent labeling session only involved assigning labels, or did it involve creating, modifying or deleting labels too? Finally, did you change any global preference(s)?

When I looked again in the out of sync, it had over 50,000! Some for other years too!

That is not necessarily incorrect (as there are all kind of possible dependencies affecting metadata). I asked the above questions on the off chance that your answer would enable me to offer a clear explanation.

In the catalog state view, how is the out of sync value calculated? Is this a dynamic system, constantly checking the diskdrives to see if there are any changes? I know that PSu recommends that file changes be done within the program but if do something directly in the folder on the disk, how is PSu aware of that?

Generally, it's your own responsibility to inform PSU of external changes. (IIRC, it is able to automatically pick up on some changes, such as file renaming. ) You could always run Verify Folders or Verify Files to trigger the external change detection. You could also re-map a folder to the correct (updated) physical location.

[vkfoto]

I was recently labeling some 6000 images in a subfolder of 2007 as well as moving some of them to their correct folders.

I understand from your recent posts that you don't currently have any version sets, right? Next question: did your recent labeling session only involved assigning labels, or did it involve creating, modifying or deleting labels too? Finally, did you change any global preference(s)?

Until I better understand how they work and how to use versioning correctly, I'm avoiding them for now. Sort of like when moving from a flat file to a relational database, it takes a while to change the way of organizing the data.

As fo the labels, it has been a combination of assigning existing labels and creating others on the fly as needed. Most of my images are already separated in common subfolders by subject so each folder gets the overall location labels plus specific labels for that folder only. I then go back and select images that have things in common with other images elsewhere and give them those labels too. I may have modified some existing labels. Changed the include parent option on a few that didn't have it set. No label was removed from the catalog. Removed a few from some images when I assigned them in error.

No global preferences were changed

[vlad]

I may have modified some existing labels. Changed the include parent option on a few that didn't have it set.

Keep in mind that the parent option change only applies going forward. If you want to apply it on previously labelled images, you will need to unassign and reassign the modified label to all applicable images.

And no, it's no clear to me, from your description, why you ended up with so many OOS images.

[fbungarz]

I may have modified some existing labels. Changed the include parent option on a few that didn't have it set.

Did you perhaps relocate/merge any labels? Did you edit any label to perhaps write metadata to files? Did you change Preferences to write GEO-locations? All these changes will affect your image sync-status.
A single change of one label might easily affect thousands of images, depending on the change - if that label applies to all these images. Particularly merging or re-locating labels, even moving them to another category, often has consequences for many, many files that may not immediately be apparent. Changing the settings of a label also can affect a lot of images. As Vlad pointed out the "assign parent" option will not immediately mark images as out-of-sync. Also the option to "Unassign labels with same parent" will not affect existing images, but is only applied next time you apply that label or force a sync (CTRL+S) to the file.
However: mapping labels to metadata fields or even configuring it to "Apply detail profile" or to write a GEO-location into the file will put all images assigned to that label as "out-of-sync".

[vkfoto]

This really underlines the need to fully understand the interrelations among the various party of PSu. As mentioned above, one small change can have a huge knock-on effect and affect many other images.

Part of my problem is probably applying PSu to an existing set of over 250,000 images now in my catalog while learning the program at the same time. As I find and remove unintentional duplicates, label and relabel the images the large size makes everything more cumbersome.

Option: reduce the catalog while preserving the current work vs just creating a new and smaller catalog. Then add the older images a small bit at a time once I'm more comfortable with the program and have settled on the settings that work best for me. Can the all the labels be exported and applied to a new catalog?

[jstartin]

Can the all the labels be exported and applied to a new catalog?

This is one of the things that "sync" allows. Your labels initially go into the catalog database. Then, depending on settings in the preferences synchronisation section, they are either automatically written out to the image file, or to an xmp sidecar file, or can be so written out on demand.

If the preferences for write and read are properly configured then reimporting the images will recreate the label hierarchy and other settings in a new catalog and assign imported images to the correct labels.

The write settings needed are:
Keywords processing "Replace keywords with catalog labels"
Tick "Write catalog data to IDimager ICS scheme"
Tick "Write hierarchical keywords"

When you come to reimport the read settings need to be temporarily changed to:
Keywords processing "Replace catalog labels with existing keywords"
Tick "Read hierarchical keywords"
Tick "Read IDimager ICS scheme"

The last settings above should be deselected in normal use to avoid a performance penalty.

I recommend you backup the catalog first and keep the copy separate, just in case.

By the way, unless you are expecting to have to re/reate a catalog in this way, or are about to share files in a way that requires them to have the keyword tags present then it does not really matter if the file are left out of sync for a while. In normal operation PSu uses its catalog and not the information written into your files.

Oh, and if you want to force writes to go into sidecar xmp files and not the image files then set all of the latter to read-only.

[vlad]

Option: reduce the catalog while preserving the current work vs just creating a new and smaller catalog.

How many unintended duplicates do you expect to discover? Perhaps you could first get rid of those duplicates and only then engage in serious cataloguing.

Can the all the labels be exported and applied to a new catalog?

In addition to Jim's explanations, I'd like to point out that you could transfer your hierarchical label structure via a formatted data file. Check out: Data -> Export to Formatted Vocabulary File and the corresponding Import action.

[fbungarz]

I definitely think doing this in relatively small batches starting with a small catalog is a much better way forward. I think someone here already mentioned that you are quite courageous to start out with a database of 250,000 images...
PSu is a fantastic program, but it is complex as well.
I am not so sure what might be the best strategy in your case.

Yes, you can sync all the metadata into your files and that data will then be available during import. That is the point of writing metadata to image files, it is a kind of backup of the data, which is independent from the database.

On the other hand: it is convenient to start out with a fairly well organized catalog of keywords. Building that up only by importing images will be cumbersome. Lots of labels that are now there will be missing from a brand-new, fresh catalog. However, you can actually export the catalog labels and import them into a clean database.

Here is probably what I would do:
(1) Use 'Data - Export to Formatted Vocabulary File".
(2) Create a clean database and import the Vocabulary File.
(3) Without any images in your new catalog, check out and cleanup your catalog (i.e., the Vocabulary that you just imported).
(4) Import small batches of images, folder per folder and experiment with settings, labeling etc.
(6) Once confident, import the rest...

Cheers,
Frank

[vkfoto]

Thanks to everyone for their helpful suggestions and support. Now to start on v2 of my grand catlaog.

One of the nice side affects of this work is looking again at all of the images I've be accumulating over the years.

[PhilBurton]

This really underlines the need to fully understand the interrelations among the various party of PSu. As mentioned above, one small change can have a huge knock-on effect and affect many other images.

Part of my problem is probably applying PSu to an existing set of over 250,000 images now in my catalog while learning the program at the same time. As I find and remove unintentional duplicates, label and relabel the images the large size makes everything more cumbersome.

Why not create a catalog wiht a small number of representative images and work through all your concerns and get the workflow worked out. THEN do your main catalog, but even then don't import all 250 K images at one go. And don't forget to back up!

Phil

[vkfoto]

This really underlines the need to fully understand the interrelations among the various party of PSu. As mentioned above, one small change can have a huge knock-on effect and affect many other images.

Part of my problem is probably applying PSu to an existing set of over 250,000 images now in my catalog while learning the program at the same time. As I find and remove unintentional duplicates, label and relabel the images the large size makes everything more cumbersome.

Why not create a catalog wiht a small number of representative images and work through all your concerns and get the workflow worked out. THEN do your main catalog, but even then don't import all 250 K images at one go. And don't forget to back up!

Phil

That's what I meant by "v2 of my grand catalog." It will start with the current year's images and then once things settle down to a routine, I'll start pulling in previous years.

[vkfoto]

Exported the labels to file.
Created a new catalog
Imported the label file.

The imported label name was not merged with the built-in one of the same name. The imported name also lost its image. Also the assigns parents option was set to No even if it was Yes in the original catalog.

Is that supposed to happen?

[vlad]

Exported the labels to file.
Created a new catalog
Imported the label file.

Note that this sequence is supposed to transfer only your label hierarchy, not the label properties and not the label assignments.

The imported label name was not merged with the built-in one of the same name.

Interesting. Could you give an example? Btw, the label file import may simply be designed to recreate the label hierrachy exactly as in the file, rather than merge it with pre-existing labels in the current catalog. (Not sure - I haven't tested, but I'd like to know.)

The imported name also lost its image. Also the assigns parents option was set to No even if it was Yes in the original catalog. Is that supposed to happen?

Yes:
1. If you want to transfer the image-label associations, then reread Jim's post and follow that path.
2. Unfortunately, the label file does not encode the labels' details, such as the parent assignment options. The good news: you can select multiple labels and edit all of them at once.

[vkfoto]

One more time. this time using Jim's recipe

When you come to reimport the read settings need to be temporarily changed to:
Keywords processing "Replace catalog labels with existing keywords"
Tick "Read hierarchical keywords"
Tick "Read IDimager ICS scheme"

And the label hierarchy got imported but not 100% and only applied to 9 images. All 11,906 images from one set were imported though.

I looked in Explorer and saw tags in the JPGs and in the XMP files associated with the CR2s. Tried a Verify Folder but nothing extra was added.

Then I removed a few of the subfolders and re-ran the import. It found them again, and the Synchronize Service started. Then PSu became Not-responding and the Sync Service stayed at 0.0% for a long time. After more than 30 minutes I gave up.

Did I miss something?
Going back, at least the original catalog has all the labels I had applied.