Crashes when importing a large collection of files

[Bobbert]

I have 90,000 images catalogued in Exposure X7. Each file has an xmp sidecar. I have been using my own controlled vocabulary which has worked for me for years. I wanted to use a dedicated DAM to get better cataloging, and PSU seems like a great alternative. What could go wrong?.

First I tried to import my whole photo collection. CRASH! Since my files are organized by year, with each month as subfolder, I tried to import 1 year at a time. CRASH!

It seems that Photo Supreme cannot handle a large import of say 5000 files at one go.

Here's what I ultimately had to do. I imported a year, and PSU was able to at least recognize the folder and file names before crashing. I then reopen PSU and navigate to the "out of sync" files, select all and read the metadata from the files. That works; sloooowly, but otherwise good. I then tried to use the "tools/build missing Thumbnails and Previews" for all files in the last import - CRASH!.

It seems PSU can't sync the database and build thumbs at the same time, and it can't build a large number of thumbs.

I had to build thumbs for each subfolder, one at a time. SLOW but that too worked and now I have a good PSU catalogue.

Now I need to merge categories because some end up in Miscellaneous, and that is grunt work, but at least I have a good, compacted and uncorrupted catalog, backed up of course, to work with.

NB: this is NOT a problem with corrupted files. The same files that seem to bring PSU to its knees in a large import are just fine in a small import. It is NOT a problem with hardware. I have a gaming laptop with a fast 8 core i7 processor, an NVIDIA graphics card with plenty of graphics memory, and 32 gig of RAM. I use it for music production with no problems. My photos are on an external USB3 hard drive and the PSU catalog database is on an internal ssd drive.

Issues for the developer...
1.LOTS of random crashes when importing large numbers of files at one time.
2. Inability to process .xmp data and build thumbs at the same time with a large number of files. I don't know how many I can do at one time... not willing to spend the time doing the experiment, but I know it can read metadata from 8000 files if I don't do ANYTHING else while it is working.
3. Building thumbs is simply a pain in the ass. No quick way to do it.


Hopefully now that I have a stable database, when I import NEW pics from my camera and keyword them I won't have any more issues.

Why did I put up with all this? I'm trialing the software, but if all goes well from now on, there is NO other alternative that is as intuitive and clever. In other words, the 5 DAYS of work I put in is actually worth it. I’ll never have to do this kind of import again so I looked at it as the upfront cost to get started. But it would be so much more worth it if the crashes could be fixed. I have read about “poof crashes” on this forum going back years. Each time they are supposed to be resolved with an update, but they keep on happening. Time I think to find and fix these issues once and for all.

[G8DHE]

Just a thought having random crashes when your using a lot of memory, might well be indicative of RAM problems, might be worth using a copy of Memtest86 to check your memory perhaps ? The free copy is usually sufficient to identify problems https://www.memtest86.com/

[RobiWan]

Hi,

I think you have some Hardware issues. It's correct - PSU will be a bit slowly after many thousends of images are imported.
Crash can happen I mean only in 2 situations:
1. no more RAM is available
2. RAM is defective

What about your OS version? Do you use some AV software? If yes you can try to exclude Catalog-directory and maybe PSU process

To check your Hardware I can recomend you use - Prime95. There are some tests to check - CPU, RAM, GPU. Maybe is timming/ overclocking your problem.

[Hesmith1029]

I am bringing over a collection of about 200,000 files from Adobe Organizer. A gaming quality PC with 32GB of RAM. The process was slow and prone to hanging the PC. The existing metadata in my file is dirty to say the least, Adobe kept everything in the keywords. So I have a LOT of clean up to do.

Photo Supreme will go into a non responsive state on almost any operation involving more than few items. The time to delete a few dozen un-needed labels can be a couple of minutes. Deleting a few hundred at the same time can be forever. PS will in an out a non responsive state multiple times. And when Resource Monitor shows it responsive it will likely have wait chain of 4 to 10 processes. Similar things will happen if are moving files from one folder to another. Similar behavior when moving imported labels from Miscellaneous to their correct spot.

Today I decided that I did not want to spend time trying to label people on all the photos I have labeled as Junk. So wanted to drop the NO PEOPLE label on everything with a Junk label. About 5000 photos. Selected the Junk label from left panel. Then in the right panel select labels and typed in NO PEOPLE in search and selected it. And off it went. Figured it would a few minutes. And hour later I am still waiting. Giving me plenty of time to write this post. It is using one core of eight. It is showing no disk activity, thus no writing the database at all.

You are correct that the architecture of Photo Supreme is excellent. There is nothing else like that I have found after years of looking to move away from Adobe Organizer. There are tools to accomplish almost anything. And it seems centered on WHO, WHEN, WHAT, and WHERE, not a f-stops, shutter speeds, ISO ratings like so many other tools are. The discussion in the opening pages of documentation is excellent. The concepts that the program is built around are excellent. SQLite is stable robust database engine that is more than capable. There is so much that is being done right. But it would be great if something could be done to improve its handling of large transactions.

[RobiWan]

Hi,

PSU can handle a lot of data (my catalog on SQLite and PostgreSQL has over 300000 images). That is not the problem. The problem I suggested to Hert long time ago was - writing logs. So you would see where the problem is. So you are always flying blind.
In my experience - if something happens like in your case, then 2 tables in the database are damaged (corrupt content). idSearchData and idSearchDataValue.

[Bobbert]

I should have been clearer. Most of the crashes are when PSU goes into a "Not Responding" state. If I wait until it becomes responsive it takes... well it never seems to become responsive again in any reasonable period of time. It is faster to close PSU and restart than to wait it out. SOME of the crashes are actual crashes... PSH closes unexpectedly. But mostly I'm referring to the first kind.

It is not a hardware problem. I have 32 gigs of RAM, enough to run real time audio applications and I don't get memory or paging errors with any other application.

[RobiWan]

Neither @G8DHE nor I wanted to attack you.
I have a larger catalog in PSU. When I started, I had about 140,000 photos that I imported all at once. Noch problems
What you describe, even if you don't want it to be true, may very well be a hardware problem. It doesn't have to be, but it can be. If you do not want to do tests, I can only say - good luck.

[G8DHE]

Indeed, I have just recovered my database on my workstation after suffering random crashes since mid-December all down to problems with RAM, no other programs seemed to be affected and all normal, browsing, email, accounts etc worked fine only when using PSU and I guess because it was using a lot more memory at any given time, I'm now down to 28GB rather than 32GB now one stick of memory has been removed and the others shuffled around

[Mke]

I had similar problems with DXO crashing when I first got that (years ago), despite everything else working flawlessly.

After extensive testing with Windows Driver Verifier and Who Crashed I eventually tracked it down to an incorrectly configured memory setting in BIOS. So it's certainly worth running some tests.

[afarrar]

Hello - I'm experiencing crashes while importing large numbers of files - 7000+.
I have a new computer - it's totally clean.
MacBook Pro
Apple M2 Max
64 GB
Ventura 13.2.1
I can import 300 files via an external HD - no problem. But I work in 10's of thousands of files.
Anyone here use Mac? Suggestions?

[RobiWan]

Suggestions?

Du you have an import Profile you use?
In my opinion PSU has an issue when you use import Profile and you imgaes have GPS data you will reversegeocoding. PSU needs very long time before
you can see "something happens here"

[HCS]

A number of challenges are presented in this thread, perhaps good for others than the OP to open their own thread to get more appropriate and dedicated answers.

Also, i think it is good to separate the concerns: 1. importing large images sets and 2. working with large image databases.

Ad 1. While i understand that importing 10's of thousands of images in a few seconds sounds attractive, how often would one do that? It may be necessary to split up your to be imported images in batches.
Ad 2. I also have a rather large database with well over 150K images and have no problems with any operation. I am running PSU on Windows 10 with PostgreSQL 14. PSU has no problems with large datasets.

To the OP, if your system gives unexpected issues when running larger loads, regardless of how other apps behave, it may indeed point to some system issues. While we are not PSU support in this forum, we are very willing to help diagnose your situation and help you use PSU to its best. A system diagnosis may be in order and that info might help us put you on the right track.

[fbungarz]

I can confirm that import of large amounts of images all at once causes PSu to become very sluggish, even unresponsive. I very much doubt this is a hardware program. There is simply a huge amount of data to process, lots of memory necessary and this slows things down.

Occasionally it helps to restart PSu if it becomes unresponsive, but this is actually quite dangerous, because the database may become corrupted and then things will eventually be even worse. Other, regular operations will become sluggish, unresponsive too. Database corruption is not necessarily everything totally "kaputt" vs. everything working like a breeze. Some tables like the search indices may be corrupted, and the database overall still works, only not very well any more. Database maintenance like regular compacting may resolve some of these issues, but best practice is not even to risk corruption.

So, I would recommend doing this, when dealing with substantial image imports:
(1) break it down into smaller, more manageable batches and restart PSU in between these batches.
(2) if that's not possible (why not?) give the program all the time it actually needs, i.e., run the import overnight or even days; quite often it looks like the program is unresponsive, but in Task Manager you see that memory use is still fluctuating, which means PSu is still running; be patient, don't just crash it and risk corruption.

Clearly, in my opinion, braking it into batches is the better strategy ...

[bimo]

• To me the underlying problem seems to be the same as handled in viewtopic.php?t=33704.

• @fbungarz

  Occasionally it helps to restart PSu if it becomes unresponsive, but this is actually quite dangerous, because the database may become corrupted and then things will eventually be even worse.

  Since 2007 I personally never detected a corrupted database (neither in IDImager nor in PSU). If your observation is right then that would be a mayor flaw in the overall design of PSU. It's decades that db write/update operations are transaction based, which means that logically associated db operations are regarded as a single unit. To achieve this the db system has to be informed that a boundary for all preceding writes/updates has been reached ("COMMIT". On starting the db system always looks for unfinished transactions (missing COMMIT) and wipes them away - thus no corruptive data can be produced.
  
  I have no doubt PSU is built that way.

[Hert]

The Single Edition uses SQLite as the database system and unlike a "full RDMS", SQLite is a desktop database that uses a single file. SQLite is very robust but can become corrupt. The most common reason for corruption is when the computer power fails during a write operation to the database file. As with any file that is important for the user, a good backup strategy is always required.

https://www.sqlite.org/howtocorrupt.html