Import very slow

Post Reply
barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Import very slow

Post by barneagle » 19 Nov 16 20:55

I'm currently importing a folder containing 14,000 photos, which should take the total in PS from 28,000 to 42,000. The import seems set to take between 4 and 5 hours (1 hour until thumbnail import was finished then another 3 or 4 hours to complete synchronization).

Is it normal to take this long? My first import (of 5,300 photos) took about 20 minutes. Does it get slower as the database grows?

Should I import in smaller batches?

Are there housekeeping steps I should carry out to improve performance?

Hert
Posts: 6336
Joined: 13 Sep 03 7:24

Re: Import very slow

Post by Hert » 20 Nov 16 3:57

Try to compact the database
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Re: Import very slow

Post by barneagle » 20 Nov 16 21:17

Thanks. I have done. I'll compact regularly and see how I get on.

jstartin
Posts: 409
Joined: 23 Aug 06 13:47
Location: UK

Re: Import very slow

Post by jstartin » 21 Nov 16 0:08

Do you have automatic version detection switched on in the import profile? I think that would cause import to slow as the number of images grows.
Jim (Photo Supreme: AMD Quad-Core A8-5500 Accelerated Processor 3.2 GHz; internal AMD Radeon™ HD7560D; 4GB DDR3 SDRAM; Win10x64)

barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Re: Import very slow

Post by barneagle » 21 Nov 16 10:49

Thanks for that, Jim. I'll have a look at the suggestion. I'll have a look at that when my current import batch is finished!

vlad
Posts: 895
Joined: 01 Sep 08 15:20

Re: Import very slow

Post by vlad » 21 Nov 16 23:21

jstartin wrote:Do you have automatic version detection switched on in the import profile? I think that would cause import to slow as the number of images grows.
I'm not sure. Does the importer's version detection check against images imported in previous sessions?

fbungarz
Posts: 1574
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Import very slow

Post by fbungarz » 22 Nov 16 13:18

For the record: I have recently experienced a "program hang" during one import session as well.
Here is what happened:
(1) started first import, import, sorting into subfolders and renaming went well, no problems.
(2) without closing PSu, I started a different import of a different folder otherwise using the same profile.
(3) got asked if I wanted to save the changes of the profile (= different import folder) - yes.
(4) started the import - my renaming dialog popped up, but nothing happened (FYI I am using the "Rename & Download by Prefix Import Script" from the resource repository at: http://repository.idimager.com/openreso ... 2408FCE4B0)
(5) clicked the start import button a second time, the renaming dialog popped up again, and again nothing happened...
(6) Odd, I decided to close/kill PSu
(7) Checking the folders in Windows File Explorer: the subfolders (named by date) had all been created and wo different copies of the files had been placed inside, the naming of these files were a mess...
(8) back in PSu: the folders and the files were there, the names here a mess to...
(9) trying to delete the folders in PSu and PSu completely stalls, have to kill it.
(10) deleting the messed up subfolders in Windows File Explorer
(11) deleting the thumbnails in PSu
(12) deleting the non-existent folders in PSu takes for ever, but eventually, witing long enough they are all gone
(13) Compacting the catalog...
(14) trying the same import again; this time everything works...

I have occasionally experienced fairly slow import sessions before. But before it never was quite so bad. Perhaps this happens, because I tried importing a large amount of files.
What I did notice: restarting PSu from one import session to the next usually works a lot better. The import runs typically much faster after a program restart. No idea why...

Cheers,
Frank

barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Re: Import very slow

Post by barneagle » 22 Nov 16 15:00

Thanks, Frank, for your encouragement.

I have tried compacting the database, turning off versioning, quitting all running programs, rebooting Windows. None of them seemed to make a significant improvement. I'm currently importing a batch of ~16,000 photos which should take the catalog from 60,000 to 76,000. Synchronize Service is unbelievably slow ... it's running at about 42 photos per minute, so at this rate the import will take 6 or 7 hours on top of thumbnail creation.

The only adjustment I've made to the default import profile is to turn off versioning so I believe that all PSu is doing is reading the basic file information (path, name, date/time, etc) and metadata and putting that in the Catalog.

How do people with 250,000 or more photos in their Catalog manage?

My other recent experience of cataloging photos is with WLPG (Windows Essentials Photo Gallery) and the Android app F-Stop. With WLPG I was able to bring in my whole collection of ~150,000 photos and not worry about how long it took. The job was certainly finished overnight and I can bring in batches of several thousand and start using the program pretty much immediately. With F-Stop the import of metadata and thumbnail creation for ~100,000 photos took just a few hours (probably 4 or 5). To repeat, this is an Android app running on much less powerful hardware. How can PSu be so slow? What is it doing?

I'll try one more batch later, with compacted databases, rebooted Windows and just the one program running. But if that is no quicker I think I'll have to abandon my trial of this impressive program.

jstartin
Posts: 409
Joined: 23 Aug 06 13:47
Location: UK

Re: Import very slow

Post by jstartin » 22 Nov 16 15:48

vlad wrote:
jstartin wrote:Do you have automatic version detection switched on in the import profile? I think that would cause import to slow as the number of images grows.
I'm not sure. Does the importer's version detection check against images imported in previous sessions?
I thought it did but, judging from a brief test, I think I was mistaken.
Jim (Photo Supreme: AMD Quad-Core A8-5500 Accelerated Processor 3.2 GHz; internal AMD Radeon™ HD7560D; 4GB DDR3 SDRAM; Win10x64)

fbungarz
Posts: 1574
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Import very slow

Post by fbungarz » 22 Nov 16 17:21

FYI - I use PSU to manage a catalog of ca. 106,000 images, all extensively labeled and with detailed metadata. I imported these over several the years. Importing them from scratch in several batches would take a considerable amount of time. Unless you are starting out anew with a huge collection (like yours) it is hardly ever necessary to import several thousand images in one go. I don't think even wedding photographers ever shoot that much.

For example, I recently imported ca. 400 images in one go. This took not longer than perhaps 10 minutes. This, in my opinion is not unreasonable, given that PSU does a considerable amount of work during an import: it generates thumbs and previews, creates subfolders and renames files, applies labels and metadata.

The only larger import I did fairly recently was adding lots of DVDs with scanned, high-resolution TIFF files. Building the thumbs for these images was excruciatingly slow. But I imply let PSu do his work, when I was not working on the PC (mostly overnight). I also recently migrated from delimited keywords to simple ones and had to re-write XMP data for all the files which contained DarwinCore metadata. This means I had synchronize almost all 100,000+ images. It takes a while, but can again be done in batches. And again having to rewrite almost the entire catalog to the files is nothing you will repeatedly have to do.

barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Re: Import very slow

Post by barneagle » 22 Nov 16 19:30

Perhaps my problem is that the program is doing more than I need. All I want is for it to read the files and store file info and metadata (particularly keywords and ratings) in the database. I have experimented with deleting the thumbs database and found that I could happily allow the program to create thumbs and previews on the fly when needed, so I could easily dispense with the creation of thumbnails and previews at the time of import.

However, I recognise that if I adopt the program I shall have to accept that it works the way it does.

One of my concerns has been that I might have to go through the whole process again if I ever need to regenerate the database. Can I be confident that if I backup both the database and the photos that the restore process will quickly get me running again should I, for example, ever need to move to a new computer?

fbungarz
Posts: 1574
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: Import very slow

Post by fbungarz » 22 Nov 16 20:55

One of my concerns has been that I might have to go through the whole process again if I ever need to regenerate the database. Can I be confident that if I backup both the database and the photos that the restore process will quickly get me running again should I, for example, ever need to move to a new computer?
I have been using IDImager since 2006, I believe back then I started with version 3 of that program. I wasn't too happy when PhotoSupreme first came out as replacement for IDI version 5 and perhaps only two years ago finally decided to upgrade. Essential to my workflow is the DarwinCore, when Hert decided to fully support that metadata standard, I was extremely happy and I have been using PSu ever since.
Not once during the ten years I have lost a database or any important metadata. I am very meticulous about backups and a few times had to use them, mostly due to my own mistakes; if I remember correctly only once my database got corrupted. Since then I have learned to do regularly maintenance (compacting the database) and I am also careful to backup before compacting as well as keeping a recent backup when I do major changes (like changes preferences how metadata are written or re-locating/merging a catalog label that has a huge amount of images assigned to it; I am typically using the catalog with auto-sync on, but again, when I make major changes to the catalog prefer to sync back these changes to the files manually, breaking the sync-queue down into smaller batches).
I have not yet had to restore a complete catalog from the metadata stored inside the images (from a backup yes, but not from the metadata inside the image files). The migration process from IDI to PSu was not very straight forward, mainly because I had gotten used to a lot of customization and settings different from the default options, and of course, the problems that I experienced migrating the DarwinCore metadata. But every step on the way Hert has been unusually supportive - despite me being very vocal at first, when he decided to introduce PSu as a replacement of IDI.

As you probably know by now, there are two catalog files that PSu creates per default (you can rename them or even create several additional ones): the actual catalog "idimager.cat.db" and the thumbnail database "idimager.thumbs.db. As long as you regularly backup these two files you should be fairly safe. It is good, though, also to backup the application folder (scripts, import profiles, etc. etc.). I keep everything in one folder that I regularly backup with SyncBack. I have never been all that fond of PSu's built in backup (simply because it takes longer than using SyncBack). So, I have never actually used the built-in restore either and therefore cannot comment on how well it works.
One advantage of using SyncBack: I use it to keep the catalog in-sync across two different computers, my laptop and my desktop. One thing I am not all that happy about: the two databases can grow fairly large, quite a nuisance moving them back and forth between the two computers and fairly slow doing backups. I filed a suggestion in Mantis to implement some sort of incremental backup routines, essentially the idea is that one would only need to save the latest changes. No idea how likely that request will be implemented (that's another one of my few complaints: I wish Mantis would keep some kind of timeline/schedule telling you which bugs are being fixed and which requests/suggestions have a chance of getting implemented).

One thing: nothing in computers will be around forever. The market is extremely dynamic. IDI may go bankrupt, Hert may at some point retire (we all do), someone else may decide to buy the company, who knows. Sh*t happens!

The good thing using PSu: it is one of the few programs that meticulously adheres to XMP, which means that your catalog metadata are fully being duplicated as XMP inside the image files. So, the data are there to stay and can be accessed from any other program that reads XMP (only custom XMP may be inaccessible). Fate may eventually force you to migrate to another program, but at least your data will be there, because this info resides inside the files. Of course in that case you may have to go through the arduous, time-consuming process again of reconstructing your database in a different program using the image metadata to do so (but at least in that case you will still be able to do that ...).

Hey, perhaps someone should pay me to write these emails :wink:
But seriously: browse the forum, you will find a lot of vocal complaints that I authored. I do criticize a lot. So this is no advertisement, but I honestly have not found another image cataloging software that fits my needs as well as PSu.

barneagle
Posts: 91
Joined: 13 Nov 16 13:18

Re: Import very slow

Post by barneagle » 22 Nov 16 21:23

Thanks for all your reassurance!

vlad
Posts: 895
Joined: 01 Sep 08 15:20

Re: Import very slow

Post by vlad » 01 Dec 16 11:47

Hi Jim,
jstartin wrote:
vlad wrote:
jstartin wrote:Do you have automatic version detection switched on in the import profile? I think that would cause import to slow as the number of images grows.
I'm not sure. Does the importer's version detection check against images imported in previous sessions?
I thought it did but, judging from a brief test, I think I was mistaken.
I'm not surprised. Afaik, automatic version detection is looking for possible versions only in the current folder and its sub-folders, by design. In your test, did you import (additional) images from a folder already in the catalog or from a brand new folder? (I think only the first scenario could possibly trigger a check against previously imported images within the same folder tree - but I doubt it does.)

Post Reply