Crashes when importing a large collection of files

[Bobbert]

I have 90,000 images catalogued in Exposure X7. Each file has an xmp sidecar. I have been using my own controlled vocabulary which has worked for me for years. I wanted to use a dedicated DAM to get better cataloging, and PSU seems like a great alternative. What could go wrong?.

First I tried to import my whole photo collection. CRASH! Since my files are organized by year, with each month as subfolder, I tried to import 1 year at a time. CRASH!

It seems that Photo Supreme cannot handle a large import of say 5000 files at one go.

Here's what I ultimately had to do. I imported a year, and PSU was able to at least recognize the folder and file names before crashing. I then reopen PSU and navigate to the "out of sync" files, select all and read the metadata from the files. That works; sloooowly, but otherwise good. I then tried to use the "tools/build missing Thumbnails and Previews" for all files in the last import - CRASH!.

It seems PSU can't sync the database and build thumbs at the same time, and it can't build a large number of thumbs.

I had to build thumbs for each subfolder, one at a time. SLOW but that too worked and now I have a good PSU catalogue.

Now I need to merge categories because some end up in Miscellaneous, and that is grunt work, but at least I have a good, compacted and uncorrupted catalog, backed up of course, to work with.

NB: this is NOT a problem with corrupted files. The same files that seem to bring PSU to its knees in a large import are just fine in a small import. It is NOT a problem with hardware. I have a gaming laptop with a fast 8 core i7 processor, an NVIDIA graphics card with plenty of graphics memory, and 32 gig of RAM. I use it for music production with no problems. My photos are on an external USB3 hard drive and the PSU catalog database is on an internal ssd drive.

Issues for the developer...
1.LOTS of random crashes when importing large numbers of files at one time.
2. Inability to process .xmp data and build thumbs at the same time with a large number of files. I don't know how many I can do at one time... not willing to spend the time doing the experiment, but I know it can read metadata from 8000 files if I don't do ANYTHING else while it is working.
3. Building thumbs is simply a pain in the ass. No quick way to do it.


Hopefully now that I have a stable database, when I import NEW pics from my camera and keyword them I won't have any more issues.

Why did I put up with all this? I'm trialing the software, but if all goes well from now on, there is NO other alternative that is as intuitive and clever. In other words, the 5 DAYS of work I put in is actually worth it. I’ll never have to do this kind of import again so I looked at it as the upfront cost to get started. But it would be so much more worth it if the crashes could be fixed. I have read about “poof crashes” on this forum going back years. Each time they are supposed to be resolved with an update, but they keep on happening. Time I think to find and fix these issues once and for all.

[G8DHE]

Just a thought having random crashes when your using a lot of memory, might well be indicative of RAM problems, might be worth using a copy of Memtest86 to check your memory perhaps ? The free copy is usually sufficient to identify problems https://www.memtest86.com/

[RobiWan]

Hi,

I think you have some Hardware issues. It's correct - PSU will be a bit slowly after many thousends of images are imported.
Crash can happen I mean only in 2 situations:
1. no more RAM is available
2. RAM is defective

What about your OS version? Do you use some AV software? If yes you can try to exclude Catalog-directory and maybe PSU process

To check your Hardware I can recomend you use - Prime95. There are some tests to check - CPU, RAM, GPU. Maybe is timming/ overclocking your problem.

[Hesmith1029]

I am bringing over a collection of about 200,000 files from Adobe Organizer. A gaming quality PC with 32GB of RAM. The process was slow and prone to hanging the PC. The existing metadata in my file is dirty to say the least, Adobe kept everything in the keywords. So I have a LOT of clean up to do.

Photo Supreme will go into a non responsive state on almost any operation involving more than few items. The time to delete a few dozen un-needed labels can be a couple of minutes. Deleting a few hundred at the same time can be forever. PS will in an out a non responsive state multiple times. And when Resource Monitor shows it responsive it will likely have wait chain of 4 to 10 processes. Similar things will happen if are moving files from one folder to another. Similar behavior when moving imported labels from Miscellaneous to their correct spot.

Today I decided that I did not want to spend time trying to label people on all the photos I have labeled as Junk. So wanted to drop the NO PEOPLE label on everything with a Junk label. About 5000 photos. Selected the Junk label from left panel. Then in the right panel select labels and typed in NO PEOPLE in search and selected it. And off it went. Figured it would a few minutes. And hour later I am still waiting. Giving me plenty of time to write this post. It is using one core of eight. It is showing no disk activity, thus no writing the database at all.

You are correct that the architecture of Photo Supreme is excellent. There is nothing else like that I have found after years of looking to move away from Adobe Organizer. There are tools to accomplish almost anything. And it seems centered on WHO, WHEN, WHAT, and WHERE, not a f-stops, shutter speeds, ISO ratings like so many other tools are. The discussion in the opening pages of documentation is excellent. The concepts that the program is built around are excellent. SQLite is stable robust database engine that is more than capable. There is so much that is being done right. But it would be great if something could be done to improve its handling of large transactions.

[RobiWan]

Hi,

PSU can handle a lot of data (my catalog on SQLite and PostgreSQL has over 300000 images). That is not the problem. The problem I suggested to Hert long time ago was - writing logs. So you would see where the problem is. So you are always flying blind.
In my experience - if something happens like in your case, then 2 tables in the database are damaged (corrupt content). idSearchData and idSearchDataValue.

[Bobbert]

I should have been clearer. Most of the crashes are when PSU goes into a "Not Responding" state. If I wait until it becomes responsive it takes... well it never seems to become responsive again in any reasonable period of time. It is faster to close PSU and restart than to wait it out. SOME of the crashes are actual crashes... PSH closes unexpectedly. But mostly I'm referring to the first kind.

It is not a hardware problem. I have 32 gigs of RAM, enough to run real time audio applications and I don't get memory or paging errors with any other application.

[RobiWan]

Neither @G8DHE nor I wanted to attack you.
I have a larger catalog in PSU. When I started, I had about 140,000 photos that I imported all at once. Noch problems
What you describe, even if you don't want it to be true, may very well be a hardware problem. It doesn't have to be, but it can be. If you do not want to do tests, I can only say - good luck.

[G8DHE]

Indeed, I have just recovered my database on my workstation after suffering random crashes since mid-December all down to problems with RAM, no other programs seemed to be affected and all normal, browsing, email, accounts etc worked fine only when using PSU and I guess because it was using a lot more memory at any given time, I'm now down to 28GB rather than 32GB now one stick of memory has been removed and the others shuffled around

[Mke]

I had similar problems with DXO crashing when I first got that (years ago), despite everything else working flawlessly.

After extensive testing with Windows Driver Verifier and Who Crashed I eventually tracked it down to an incorrectly configured memory setting in BIOS. So it's certainly worth running some tests.