Crashing when importing 'duplicates'

[Stephen]

Computer is NOT under load, but PSu keeps locking up (beach ball in Mac terms).
I am importing small groups of maybe 50-60 files from one folder to another and renaming them in the process. Is that an unreasonable task to request? Everything is clean, database + thumbnails compacted, etc. Any clues?

Stephen, This may be a long shot. Create a whole new catalog, empty. Redo the import, etc, and observe the results. Phil

My tiny new test catalog with 240 images has just locked up. I moved the cursor to the top to change to the main catalog and PSu stopped working (beach ball). The fans are getting louder, PSu is now taking most of the processes, although the CPU is only working at 16%. At this stage I would normally force quit the application. I quit my browser to see if that helps. CPU usage dropped to 12%. Fan speed slower. Only 25% of available RAM being used. PSu using 100% of processes (maybe an inaccurate counter, as I still have a few other apps open).

Internal hard drive with PSu has 25% free space and the external data drive has 24%. Both getting low, but not critical! 30 mins of inaction, so I forced quit.

Full disclosure:
The groups of files I am importing are already known by PSu and on import they are being renamed and copied from one directory to another in my filing system. These are the reasons:

* I place the original filename in the headline and document title of each file on import.
Often the names of my files are changed by third parties, so this helps tracking the original.

* I could move the folders and rename the files, but then I would not have the original names in those fields. There is possibly more chance that PSU recognizes more of the metadata and avoids my redoing work.

* I can verify the completeness of the "duplicates" before removing PSU's link to the original location.

PSu does not have image recognition, so why should this be causing the problem?

[vlad]

Hi Stephen,

The groups of files I am importing are already known by PSu and on import they are being renamed and copied from one directory to another in my filing system.

Do I understand correctly that you use an import profile to:
1) perform file copying and renaming for a bunch of images, which were previously imported
2) import the new (duplicated) images
?

I don't really understand why would you do that.

I could move the folders and rename the files, but then I would not have the original names in those fields. There is possibly more chance that PSU recognizes more of the metadata and avoids my redoing work.

If the original names are already filled in, then why can't you simply move and rename the files? (These wouldn't affect the metadata, would it?)

I can verify the completeness of the "duplicates" before removing PSU's link to the original location.

What do you mean by that?

PSu does not have image recognition, so why should this be causing the problem?

Are you sure that PSU hanged because the imported images were duplicates?

[Stephen]

Thanks, Vlad

The groups of files I am importing are already known by PSu and on import they are being renamed and copied from one directory to another in my filing system.

Do I understand correctly that you use an import profile to:
1) perform file copying and renaming for a bunch of images, which were previously imported
2) import the new (duplicated) images?
I don't really understand why would you do that.

1) I am renaming them during this process, as PSu does not know the new names
2) Yes, only the images are duplicates, but the names are new.

I could move the folders and rename the files, but then I would not have the original names in those fields. There is possibly more chance that PSU recognizes more of the metadata and avoids my redoing work.

If the original names are already filled in, then why can't you simply move and rename the files? (These wouldn't affect the metadata, would it?)

a) I tried just renaming them, but the import has an advantage as I can save the renaming profile, which I did not find a way to do when just renaming. The original files have different naming schemes, so I can save different renaming profiles to use during import.
b) Also (and I tried it) if I just rename them, then the original file names will remain in the Document Title field (Details pane).
c) I am consolidating the new folders, so many import sessions will go in the same folder. I know that I could merge folders and rename them, but my methodology appeared more logical.
I made a mistake re my metadata comment above.

I can verify the completeness of the "duplicates" before removing PSU's link to the original location.

What do you mean by that?

I skim through the descriptions in the Details pane and verify and / or duplicate them in all new imported files in case just some files had details written. A bit of housekeeping work.

PSu does not have image recognition, so why should this be causing the problem?

Are you sure that PSU hanged because the imported images were duplicates?

No I am not, but I presumed that somebody might suggest that this could be the reason.

[sanphotgn]

I am kind of confused if you want the original file name in the fields or new duplicate file name in the fields. Either way - you could do that via a macro. I apply a macro in the Headline field when I want to use a file.

If original: macros, follow vlad's suggestion of moving and naming within PSU.

If duplicate: follow vlad's suggestion of moving and naming within PSU, then macros.

You can save your rename scripts from the Rename window.

Rename Save to File.png (44.29 KiB) Viewed 8110 times

[Stephen]

I am kind of confused if you want the original file name in the fields or new duplicate file name in the fields. Either way - you could do that via a macro. I apply a macro in the Headline field when I want to use a file.

If original: macros, follow vlad's suggestion of moving and naming within PSU.

If duplicate: follow vlad's suggestion of moving and naming within PSU, then macros.

You can save your rename scripts from the Rename window.

Rename Save to File.png

Many thanks, I could not find that button to save the macros! That might be the solution and is VERY useful to know

[sanphotgn]

From the manual: "TIP: While editing the fields, you can use Macro Commands. Right click in the field to select any available macro command. Then when applying the metadata, the macro commands will be replaced with their appropriate value."

Using the file name in the Headline field example ... Open the Details panel. Select the photos. Right click in the Headline field, and go to Image, File Name.

Detail Panel Field Macro Commands 01.png (22.07 KiB) Viewed 8041 times

Detail Panel Field Macro Commands 02.png (16 KiB) Viewed 8041 times

You should see this in the Headline field:

Code: Select all

%FileName

Headline File Name.png (6.08 KiB) Viewed 8041 times

Click the OK button.

You can can use more than one Macro Command.

If you want file name and extension (note you will have to manually enter the . if you choose the fields from the short-cut menu):

Code: Select all

%FileName.%FileExtension

Once you have used a macro in a field, you can quickly use it in the future in the same field: just start typing the macro and the macro should auto-fill.

[Stephen]

Many thanks. That opens another box of opportunities... hopefully not a Pandora's box

Unfortunately, the idea of only renaming will just be a short term. Excepting this test in a new catalog, I have all files in one PSu catalog. The principal directories of which are:
library - all images taken before using PSu
library B - all subsequent images

- Until recently I have not been using labels very much (the problem of crashing has now increased considerably)

- I am now working on moving blocks of images (and renaming according to the library B scheme, as mentioned earlier). Primarily those are the one I need to currently edit. (I have not been able to work productively today.)

- The renaming proposed functioned perfectly, thanks however, I want to move them to library B, because:
(a) that is where where the C1 library is located.
(b) the belief (hope) that PSu will correct any xmp errors during that process.
(c) better to have everything together.
(d) In the process, I am consolidating.

- The crashes have not decreased, even when moving (re-importing) the renamed files from "library" to "library B".

Before every import I now close PSu. After every import I close PSu. Then I set the labels and close it again before the next import. Sometimes it works, but not always. I have had to force quit PSu (equiv Windows Atl Ctrl Del) at least 15 times today!

Everything is compacted. Is there a way to re-index PSu or is that not the problem? Is the problem importing images from several folders and importing them into ONE folder...?

[vlad]

Many thanks. That opens another box of opportunities... hopefully not a Pandora's box

Unfortunately, the idea of only renaming will just be a short term.

The crashes have not decreased, even when moving (re-importing) the renamed files from "library" to "library B".

Do you move the renamed files from "library" to "library B" inside or outside PSU? Why do you still have to re-import?

Before every import I now close PSu. After every import I close PSu. Then I set the labels and close it again before the next import. Sometimes it works, but not always. I have had to force quit PSu (equiv Windows Atl Ctrl Del) at least 15 times today!

That sounds terrible.

Everything is compacted. Is there a way to re-index PSu or is that not the problem? Is the problem importing images from several folders and importing them into ONE folder...?

I don't know if that's the problem, but it should not be the problem. To be honest, though, I still don't understand your workflow - maybe you could describe it step by step. Anyway, if you have just compacted the DB and there's no improvement, I don't know what else you could do. Have you tried restarting your machine?

Hert stated that Supreme verifies the DB integrity upon running the catalog backup procedure. I'm wondering if there's any other way to invoke such a check and get notified in case of a problem with the DB. (Btw, do you have a recent backup that you could restore to?)

[Stephen]

The crashes have not decreased, even when moving (re-importing) the renamed files from "library" to "library B".

Do you move the renamed files from "library" to "library B" inside or outside PSU? Why do you still have to re-import?

Before every import I now close PSu. After every import I close PSu. Then I set the labels and close it again before the next import. Sometimes it works, but not always. I have had to force quit PSu (equiv Windows Atl Ctrl Del) at least 15 times today!

That sounds terrible.

Everything is compacted. Is there a way to re-index PSu or is that not the problem? Is the problem importing images from several folders and importing them into ONE folder...?

I don't know if that's the problem, but it should not be the problem. To be honest, though, I still don't understand your workflow - maybe you could describe it step by step. Anyway, if you have just compacted the DB and there's no improvement, I don't know what else you could do. Have you tried restarting your machine?

Hert stated that Supreme verifies the DB integrity upon running the catalog backup procedure. I'm wondering if there's any other way to invoke such a check and get notified in case of a problem with the DB. (Btw, do you have a recent backup that you could restore to?)

This is my workflow to bring old (pre PSu) images from library to library b. (Everything is in the same catalog).
The old images were labelled inconsistently because the label was just a reminder of the content and used when importing into Aperture, but it goes something like this.
These are the folder names in "library" (which is my library for old files):
Paris Eifel Tower
Paris Champs Elysee
Louvre Paris
Paris Notre Dame
Bar at Georges V

That was sufficient for my key wording / labeling requirements at the time with Aperture. I now need more, so all of these keywords / labels are added after importing and then all of these 5 folders can be put into one, which I name:
France Paris filedate

The numbering scheme of the older files was also inconsistent, so I am either renumbering them first according to several schemes to get a uniform result OR I am renumbering them when importing / moving them.

So now import Paris Eifel Tower from "library" to "library b" and create the new folder France Paris filedate. Then I add the labels. Then I remove folder Paris Eifel Tower from the catalog.
The process continues with folder Paris Champs Elysee, also adding the files to the newly created folder France Paris filedate, in this case bundling them all in the same folder.

The next day in Paris continues the same way so that I end up with 4 folders for the 4 day trip, rather than maybe 20 which were originally filed according to the topic.

There are certainly many ways to accomplish this, but this seemed logical for me to move:
2010-06-01_11_24_33-_DSC0981.JPG from "library" to "library b" and rename it at that time to:
20140601-112433-0981.JPG (which is my consistent numbering scheme) rather than renaming it first so that PSu has 2 files with the same name (albeit in different locations) to contend with.

However, neither way is working, but that might not be connected with my operation.

I have many PSu backups, but I don't usually make then daily.

Vlad, thanks for your help in this matter

[vlad]

This is my workflow to bring old (pre PSu) images from library to library b. (Everything is in the same catalog).

Ok, so just to clarify (as the term confused me when I first read it): your "pre PSu" images are already in the PSU catalog. You presumably refer them to "pre PSu" because they were already tagged (keyworded) before the PSU import. Correct?

The old images were labelled inconsistently because the label was just a reminder of the content and used when importing into Aperture, but it goes something like this.
These are the folder names in "library" (which is my library for old files):
Paris Eifel Tower
Paris Champs Elysee
Louvre Paris
Paris Notre Dame
Bar at Georges V

That was sufficient for my key wording / labeling requirements at the time with Aperture. I now need more, so all of these keywords / labels are added after importing and then all of these 5 folders can be put into one, which I name:
France Paris filedate

The numbering scheme of the older files was also inconsistent, so I am either renumbering them first according to several schemes to get a uniform result OR I am renumbering them when importing / moving them.

That's where I'm getting confused: your older files are already in the catalog. What does "importing / moving" them involve?

So now import Paris Eifel Tower from "library" to "library b" and create the new folder France Paris filedate.

If "Paris Eifel Tower", "library" and "library b" are all folders, then what do you mean by "importing Paris Eifel Tower"? Do you mean that you copy all image files from "library/Paris Eifel Tower" to "library B/France Paris <filedate>" and in the process you rename the image files as well, then import the copies? Do you perform all these actions via an import profile?

Then I add the labels.

Wait a sec... Which labels do you add? If the "old" images in "library" had labels (keywords), don't the "new" images in "library B" have the same labels already assigned upon importing?

Then I remove folder Paris Eifel Tower from the catalog.
The process continues with folder Paris Champs Elysee, also adding the files to the newly created folder France Paris filedate, in this case bundling them all in the same folder.

Ok, so all your "library" images taken on <filedate> end up right in the "France Paris <filedate>" folder, right? (And you do ensure that the names of newly copied files do not clash with and overwrite previously copied files, right?)

The next day in Paris continues the same way so that I end up with 4 folders for the 4 day trip, rather than maybe 20 which were originally filed according to the topic.

Ok, I think I've finally understood your workflow.

There are certainly many ways to accomplish this, but this seemed logical for me to move:
2010-06-01_11_24_33-_DSC0981.JPG from "library" to "library b" and rename it at that time to:
20140601-112433-0981.JPG (which is my consistent numbering scheme) rather than renaming it first so that PSu has 2 files with the same name (albeit in different locations) to contend with.

You keep referring to file and folder moves, but you actually achieve "the moves" by copying and renaming the files during import, and then removing the old "library" files and folders. Did I get that right? And do you only remove the "library" folders from the PSU catalog or do you delete them from disk too?

However, neither way is working, but that might not be connected with my operation.

By "not working" you mean the PSU freezes, right? I need to think a bit about your workflow (it's...interesting), but first I'd like your confirmation that I've got it right.

I have many PSu backups, but I don't usually make then daily.

I was (indirectly) asking if reverting to the latest "working" backup is an option for you. Also, did you restart your machine?

Vlad, thanks for your help in this matter

You are welcome, Stephen, but please notice that my help is akin to that special kind of help offered by some shrinks: I'm hoping to ask the right questions in order for you to figure out the right answers to your own problems

[Stephen]

This is my workflow to bring old (pre PSu) images from library to library b. (Everything is in the same catalog).

Ok, so just to clarify (as the term confused me when I first read it): your "pre PSu" images are already in the PSU catalog. You presumably refer them to "pre PSu" because they were already tagged (keyworded) before the PSU import. Correct?

Sorry, I must be careful with my descriptions. "pre PSu" should be called "ex Aperture". They are all the folders rescued from Aperture. (PSu's Aperture import did not work for me). Few had keywords, but some had descriptions. I also had some information in the folder names.
All of them (the whole "library") which is one directory on my hard drive were imported in one step. Now I am cleaning up.

The old images were labelled inconsistently because the label was just a reminder of the content and used when importing into Aperture, but it goes something like this.
These are the folder names in "library" (which is my library for old files):
Paris Eifel Tower
Paris Champs Elysee
Louvre Paris
Paris Notre Dame
Bar at Georges V

That was sufficient for my key wording / labeling requirements at the time with Aperture. I now need more, so all of these keywords / labels are added after importing and then all of these 5 folders can be put into one, which I name:
France Paris filedate

The numbering scheme of the older files was also inconsistent, so I am either renumbering them first according to several schemes to get a uniform result OR I am renumbering them when importing / moving them.

That's where I'm getting confused: your older files are already in the catalog. What does "importing / moving" them involve?

I am copying them folder by folder from the old "library" (directory on the hard drive) to the new "library b" in a different directory on the same hard drive. I then tell PSu to forget the folders in the old "library". Some of the xmp files are inconsistently written (file format) so I am hoping that PSu is repairing those and replacing missing ones during import.

So now import Paris Eifel Tower from "library" to "library b" and create the new folder France Paris filedate.

If "Paris Eifel Tower", "library" and "library b" are all folders, then what do you mean by "importing Paris Eifel Tower"? Do you mean that you copy all image files from "library/Paris Eifel Tower" to "library B/France Paris <filedate>" and in the process you rename the image files as well, then import the copies? Do you perform all these actions via an import profile?

I use an import profile (with different renaming procedures) and copy the files to the new location at that time.

Then I add the labels.

Wait a sec... Which labels do you add? If the "old" images in "library" had labels (keywords), don't the "new" images in "library B" have the same labels already assigned upon importing?

Some of the "new" images had keywords, but I had never assigned labels. I am doing that after this import procedure.

Then I remove folder Paris Eifel Tower from the catalog.
The process continues with folder Paris Champs Elysee, also adding the files to the newly created folder France Paris filedate, in this case bundling them all in the same folder.

Ok, so all your "library" images taken on <filedate> end up right in the "France Paris <filedate>" folder, right? (And you do ensure that the names of newly copied files do not clash with and overwrite previously copied files, right?

Correct. They cannot (should not) clash because they are going into a new folder in a different directory. The way I understand PSu is that some people might live with a catalog where the files are spread over many directories and even many hard drives.

The next day in Paris continues the same way so that I end up with 4 folders for the 4 day trip, rather than maybe 20 which were originally filed according to the topic.

Ok, I think I've finally understood your workflow.

There are certainly many ways to accomplish this, but this seemed logical for me to move:
2010-06-01_11_24_33-_DSC0981.JPG from "library" to "library b" and rename it at that time to:
20140601-112433-0981.JPG (which is my consistent numbering scheme) rather than renaming it first so that PSu has 2 files with the same name (albeit in different locations) to contend with.

You keep referring to file and folder moves, but you actually achieve "the moves" by copying and renaming the files during import, and then removing the old "library" files and folders. Did I get that right? And do you only remove the "library" folders from the PSU catalog or do you delete them from disk too?

In fact my comment "that PSu has 2 files with the same name" should not be a problem and indeed I have just stated the opposite.

However, neither way is working, but that might not be connected with my operation.

By "not working" you mean the PSU freezes, right? I need to think a bit about your workflow (it's...interesting), but first I'd like your confirmation that I've got it right.

Yes, I mean freezing.

I have many PSu backups, but I don't usually make then daily.

I was (indirectly) asking if reverting to the latest "working" backup is an option for you. Also, did you restart your machine?

The freezes do not appear to be destructive and have occurred for 10 days now, so backups are useless. Indeed when I repeat the import, the folder I wanted to import is shown in the import dialog, so that was remembered by PSu. I restart the machine every morning.

Vlad, thanks for your help in this matter

You are welcome, Stephen, but please notice that my help is akin to that special kind of help offered by some shrinks: I'm hoping to ask the right questions in order for you to figure out the right answers to your own problems

Yes, I think the dialog is useful and maybe even contains info which will help others.

[Stephen]

Hert stated that Supreme verifies the DB integrity upon running the catalog backup procedure. I'm wondering if there's any other way to invoke such a check and get notified in case of a problem with the DB. (Btw, do you have a recent backup that you could restore to?)

Will that help the current DB or only the backup? But that would not make sense unless one restores the backup.
If it verifies the current DB, then I need to keep searching.

I am getting freezing with new imports from a memory card too!

[fbungarz]

Does the database compact?
Generally I would start from a brand new database and only import the images you actually want. I think your strategy to import images that are already inside the database is a recipe for disaster...

[Stephen]

Does the database compact?

Yes, perfectly and I do it regularly.

Generally I would start from a brand new database and only import the images you actually want.

What do you mean by "only the images" I want? I want all of them. Is it not normal to consolidate? I am basically just copying files to an existing directory (via PSu, so that it knows what is happening.

I think your strategy to import images that are already inside the database is a recipe for disaster...

But PSu presumably does not analyze the image, but looks at the file name and maybe a few other data fields? So if the file name is different, then surely it will be different enough to be recognized as a different image? Besides, in my experience with PSu, that is how duplicate images occur, even inadvertently.

[fbungarz]

Hi Stephen,
sorry that I was rash.
If I correctly understand what you are doing (which I find rather difficult reading your explanations) then I think the crashes you are experiencing may be caused by messing up the catalog seriously from inside PSu.

Here is the scenario:
You have X number of files in your catalog (probably quite a few?). They all have some (rudimentary) metadata, keywords, labels, XMP already assigned to them. Now you copy these files inside PSU into different folders using the PSu import wizard, renaming them in the process.
The result are duplicates that have the same metadata as your originals, for each you have JPGs (perhaps even several?) and RAWs. That is two sets of files (named differently), different formats (JPGs and RAWs) - all with the same labels, all inside the same catalog. Right?
You have mentioned elsewhere you do not use versions, you instead store derivatives in subfolders. I am not even sure if you are using strict naming conventions to name all these different derivatives of the same files. And you do not use autosync.

Now you want to start cleaning up your catalog.
I do not know about you: I could NOT handled this complexity! For me it would be the perfect recipe for disaster! I would most certainly completely mess up everything, loose track of what I am actually doing.

If I were you I would try to set up a very clean and simple, strategic workflow:
(1) start with a completely new database.
(2) Import your files using the wizard into a clean, new folder structure, that is organized not according to topics, but dates, rename your files following a strict renaming template, again one that avoids including topics, but a pure, clean numbering system (perhaps with the name of the person who took the photo as a prefix; PSu includes many great templates for renaming and setting up a general folder structure.
(3) The import wizard allows you to label your files during import using labels, you can also add keywords. Use them sparingly to write common sets to the batches of your files you import.
(4) Import only small batches
(5) Once you imported the first small batch, inspect what metadata the files contain, check XMP (Details Panel) and labels (Assign Panel). Delete XMP entries you do not want, un-assign labels you do not need or like, inspect the catalog tree for redundant label that you do not want and delete those or re-locate them where they belong, or merge them with duplicates elsewhere in the tree.
(6) Now you have a small, clean database.
(7) Continue with the next small batch, your database grows step-by-step, you clean and re-organize the metadata and build a clean catalog that is easy to navigate.

This, I believe is a strategy that should work. I think you are trying to deal with too many issues at the same time. In my experience a huge, complex messy catalog frequently stalls, freezes up, becomes sluggish. Call yourself lucky that your database still compacts without a glitch. That means it is at least not corrupted. But that does not mean it hasn't become too unwieldy to function well.
I cannot look over your shoulder, what exactly you are doing, but I can only assume you are trying too many things at the same time. And I have a really, really hard time imagining any scenario, where cleaning up an existing folder structure from within PSu will work well. The Import Wizard is designed for importing images not yet inside the database and not for creating duplicates of existing files that you then later remove again.

Like I said: I really have difficulties understanding what you are actually doing. Maybe I am totally wrong, but I find your explanations extremely confusing. My suggestions might be completely off and if that is the case I apologize. I am trying to help, but perhaps without completely understanding the steps of your workflow, my advice is not all that useful.

Good luck,
Frank