PSU building thumbs dbase extremely slow from TIFFs?

Post Reply
fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

PSU building thumbs dbase extremely slow from TIFFs?

Post by fbungarz » 09 Feb 14 6:36

Hi all,
testing PSU v2 I have migrated my IDI database and let it start building the thumbs database to find out how large it will get using the smallest thumbs settings. With 76,389 images in my database progress was pretty quick until thumbs for about 20% of images got built. Then the thumbnail building process virtually stalled with a single thumb taking a minute or two to get build. After four hours running PSU it reached 25% and I stopped PSU. No better luck after restart. still excruciatingly slow !!!
I noticed that the files that apparently almost stalled PSU seem to be 120MB TIFF files, digital scans of my slide collection. In Windows Explorer thumbs of these files are displayed virtually instantaneously, likewise in Lightroom, IDI Pro also has no problems creating thumbs and previews of the files. Photoshop opens the entire files more rapidly than PSU creates the thumbs. Of course I have no idea if the file format (TIFF) and posibly their size are really the reason why the process is so slow. I only noticed that in the progress box names of JPGs, DNGs and NEFs fly across the screen so rapidly that one can hardly read their names, while names of the TIFFs seem to grind the process almost to a halt.
Any suggestions?
Thanks,
Frank

Hert
Posts: 6685
Joined: 13 Sep 03 7:24

Re: PSU building thumbs dbase extremely slow from TIFFs?

Post by Hert » 09 Feb 14 9:57

Maybe lower your image processing limit to 100MB. All images with a file size above that limit will then use the Windows engine to build thumbs. Windows doesn't support as much tiff kinds as PSU does but if it builds yours and you're more happy with its speed then use that
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: PSU building thumbs dbase extremely slow from TIFFs?

Post by fbungarz » 09 Feb 14 18:04

Hi Hert,
thanks for the tip. I tried it. No difference at all, he thumb building process is still excruciatingly slow - and actually opening the folders where the files are stored it seems PSU is now not even building thumbs for these files at all!
To investigate I canceled the preview/thumbs building and switched to the folder view of PSU and looked at the folders where the large TIFFs are stored. The folders are all pale red and PSU tells me they are not in the catalog. Rather odd, because They are definitely a part of IDI Pro's catalog and I migrated this database to the PSU catalog...
And: the thumbs info, file name, keywords, XMP etc. etc. is all there! Only PSU tells me it is not part of the catalog!? Searches retrieve the files - only the thumbs are missing! When I verify the folder PSU suggests to add them to the catalog, if I accept this a second identically named hard drive letter G:\ (identical and a duplicate with the letter already there! with the same label ImageArchive!, but PSU apparently adds the serial number now?) gets mounted and added to the catalog. Thumbs for that "additional" drive do not get build (or if they do, the process is EXTREMELY slow).
Very, very weird.

Off-topic (a bit): Noticing that PSU adds the serial number of a drive to the catalog I have a different question. I already mentioned that I use SyncBack to keep my desktop database and laptop database always in-sync. Now the desktop catalog accesses all image files on an internal hard disk, the laptop connects to an USB drive. Again I use SyncBack to keep the files on these drives in-sync. This works well without any glitches using IDI Pro, because IDI identifies he drives by their drive letter, not their serial. I only have to make sure the USB drive is always mounted to the same drive letter (which is easy using USBDLM's ini file to force windows to mount the drive to the same letter each time). If PSU identifies the drive by its serial this workflow ill no longer work, but I'' constantly have to tell PSU to re-map the folders...!?
Can this feature be disabled so PSU does NOT check the drives serial no.?

Thanks,
Frank

Hert
Posts: 6685
Joined: 13 Sep 03 7:24

Re: PSU building thumbs dbase extremely slow from TIFFs?

Post by Hert » 09 Feb 14 19:53

fbungarz wrote:Very, very weird.
Not weird at all. A drive is identified by medium type, medium serial, and medium name. If you change either of these three properties then PSU doesn't recognize the drive. For example; when you change the file volume name, or you replace the hard drive with another one, all cause the drive to no longer be recognized. This is no different than in IDI. If you have a drive G: that is different as another drive G: and you're sure that it is the correct one, then remap the "wrong" drive to the correct physical folder.
because IDI identifies he drives by their drive letter, not their serial.
That is wrong. IDI identifies a drive by medium type, medium serial, and medium name. The drive letter is of absolutely no meaning. Not in IDI, not in PSU. In IDI you need to "relocate" a drivee if either of these three properties change. In PSU the "relocate" feature is renamed to "Map to correct physical folder", which is better covering what it does. But it's exactly the same.
Can this feature be disabled so PSU does NOT check the drives serial no.?
If you use a different USB drive then it's a different medium, no matter the drive letter. If you insert a different DVD disc in a DVD drive then the drive letter is also the same, but it's a different medium. How would software know? Because the disc''s serial number is different. The same for an USB drive...if you insert another drive in the same USB port then it's a different medium.

You'll need to come up with better procedures for your USB drives.

Hert
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: PSU building thumbs dbase extremely slow from TIFFs?

Post by fbungarz » 09 Feb 14 20:51

Hert,
I am really sorry, but what you say about IDI Pro is not correct!
I have been using IDI Pro for YEARS the way I described it. When I open the database on my laptop it recognizes the external USB as drive G. Copying over the database to my desktop the exact same database there connects without a hitch directly to the internal drive G (which is of course a different drive!). NO re-mapping needed!
AND:
If I mount the USB to a different drive letter on my laptop, then the database does NOT recognize the drive and THEN I do have to re-map it! This is why I am using USBDLM. It reads the serial and makes sure the drive is mapped always to G.
Ironically this works well for my workflow, but it has been quite a pain using different databases on different PCs throughout the research station. We very regularly had huge issues with external drives not being recognized by IDI because people had used an USB stick and the drive letter that IDI was using for an external drive now suddenly pointed to the stick, not the drive. So people ended up with having some folders on F, some on G, which both corresponded to the exact same drive. Only they had been working with the drive one time mounted to F, the other time to G. Until I discovered USBDLM this was quite a pain...
So, seriously, I am NOT making this up! (And I can obviously see the benefits of PSU handling drives differently! Only for they way I a plan to use it I wish it did not...).

The following also confirms that PSU and IDI handle drives differently; in PSU I now have two drives when I open folders:
1)
Drive G with Label "ImageArchive": most folders (but not all!) are pale red. They contain "empty" thumbs, i.e.,ones that lack thumbnails. Nevertheless, I can open them from PSU in an external editor, I can read their XMP, I can add keywords, metadata. As far as I can tell the files are there, PSU recognizes they are there, but still tells me they are not in the catalog!? :?
On that same drive there are other folders not red. They have thumbs with thumbnails. I guess these are the ones that were build when I ran the "Build Missing Thumbnails" from the Tools menu. So the "build missing thumbs" command does not check the serial, it even ignores that PSU thinks these images are not in the database, it simply adds the thumbs for these files (though very slowly so).

2)
Running the "verify" command on one "red" folder I now also have a second drive "G" labeled "ImageArchive, xxxxxx" (where the xxxxx is the serial of he drive). That second drive contains only that one folder, which PSU "verified". All other folders are on the drive without the serial.
This means that the "verify" command in PSU apparently reads and adds the drive serial, which obviously is not part of the converted IDI Pro database. This clearly confirms the scenario above where IDI Pro ignores the serial, PSU does not. It means, if I use a converted IDI database and do never use the verify command, the database will then lack the drive serial and I do not have to remap each time. Of course this is impractical, because "verify"is quite central and essential to so much of PSU's functionality...
You'll need to come up with better procedures for your USB drives.
Well, it simply means PSU is no longer as portable because I essentially now have to do a re-map every time I switch computers :(

- sync database
- sync images
- remap drive

One more step. I just hope the re-mapping in PSU is fast. In IDI it can take ages (apparently not designed for doing it regularly...).

Back to the original topic:
Any suggestions on why the thumbs building from the TIFFs is so slow even if I force IDI to let Windows build the thumbs? Like I said, in Explorer they are built really quickly, if I limit reading in PSU to 100 MB as you suggested I do not see any noticeable difference. PSU does take just as long as before and not as quick as Explorer.
Perhaps DNGs and NEFs are so much faster because PSU accesses their previews, not the original file? But then, JPGs also are fast. So, probably it is overall file size? Would it help to convert all TIFFs to DNGs?

Cheers,
Frank

Hert
Posts: 6685
Joined: 13 Sep 03 7:24

Re: PSU building thumbs dbase extremely slow from TIFFs?

Post by Hert » 09 Feb 14 21:13

Sorry Frank, but IDimager does not identify drives by their drive letter. Maybe you are using the media browser in idi and then don't notice that media are identified differently in the catalog...in the IDI media explorer you're not working with the catalog.

Come up with a procedure where you work with one usb drive as leading and make backups separately without exchanging the use of the drive and the backup drive

Ps. In case you forgot, I'm the one who developed both tools and so I know how they both work ;)
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Post Reply