Versioning on Import Extremely Slow

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Versioning on Import Extremely Slow

Post by gcorbin » 06 Sep 20 13:37

Versioning on Import Extremely Slow

When I import my photos into Photo Supreme with the importer, everything works fine except the versioning stage takes forever, about 3 seconds per photo. It might take an hour to version 800 photos (400 JPG ORF pairs) with the rest of the import taking a minute or two. During the versioning period, my i7 CPU sits at about 30% usage and the SSD disc sits at 0% usage.

It has been like this for many versions, so I am confident that it is something to do with my configuration, but I cannot see what is wrong. I am currently using PS 5.5.1.3084 on Windows 10 2004, i7 CPU, 16GB RAM, SSD hard disk. I have compressed the database many times and rebooted many times. My database is now up to 333098 images so is getting big. I recently reinstalled windows from blank and re-installed all my software which did not fix this issue, so I don’t think this issue is caused by a windows issue. The issue happened before Win 10 2004 so again, I don’t think this is the issue. I use windows defender and have excluded the database from scanning. Everything in Photo Supreme runs fast except for the versioning on import.

The relevant Import profile settings and version settings are:-
Import Profile
Copy images to new location
%yyyy
%mm
%dd
\
%FileExtension
\

File name Original Filename
Apply Versioning Ticked

Versioning
Version Set display As a Set in Categories
Version Set detection mask %FileName*.*
Version Set offset folder Current and Sibling Folders
Main version detection method First JPEG else RAW

Place holder Match As
album Display First Version Found
web display No Matching
slide show No Matching
print No Matching
e-mail No Matching
Newsletter Custom File Mask *.png
Gimp No Matching
Affinity Photo Custom File Mask *.afphoto
custom 5 No Matching
custom 3 No Matching


Another minor issue which may be related is after the import completes, the new folder/folders are not displayed in the folder list. I have to close the parent folder and reopen the parent folder to display the newly importer folders. Again, I presume this is not normal.

Anyone got any suggestions what is wrong with my configuration as waiting for half an hour to an hour for versioning on import is a pain.

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 07 Sep 20 8:12

I run on an i7 (10th gen) with 16GB (2666Ghz memory) and a NVMe SSD (3400MB/s).

I just tried Version Detect here and performance is as expected.
As your report shows, Version Detection is unrelated to the database because with your version detection rule (%filename*.*) the version detection mainly takes place on the file system. The volume of images in the catalog is hardly of any effect for version detection.

1. Can you share the number of images that you are importing. Of course it could be that you checked for more images than I checked.

2. Is there a performance difference when you run Versioning as part of the import or selecting all imported files (Last Import) and then right click -> Versions -> Version Detection (Ctrl+Shift+V). It shouldn't because the import versioning is the exact same thing.

3. Can you give an estimate as to how many images are in the "current folder and sibling folders". The number of files could have an impact because for every file that PSU has to "version", PSU has to compare the version rule with every single file that it finds in the "current and sibling folders".

If there are 100 files being imported and there are 1000 files in the "current and sibling folders" then PSU has to do about 100.000 (100 * 1000) version rule comparisons. If there are 10000 files being imported and there are 1000 files in the "current and sibling folders" then PSU has to do 10.000.000 (10000 * 1000). Or import 10000 files and there are 10000 files in the configured offset folder then that number becomes 100,000,000. See how fast that number grows? And that is why you see the CPU work while the SSD is enjoying a Cuba Libre.

For these comparisons, PSU is of course optimized.

4. Can you give an estimate about what percentage of the imported files eventually end up being versioned?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 07 Sep 20 23:30

1 I am importing the days shooting from an SD card, so typically 50 to 250 JPG ORF (Olympus RAW) photo pairs. ie. 100 to 500 files. The last import for example was 108 JPG, 108 ORF and 1 MOV.

The files being imported are going into a new date directory being created for the import, so empty.

2 I have turned off populating Import Sessions (I have long forgotten where) so that is not populated. If I open the 20200905/JPG folder and version detect the 108 files, this takes about 3 seconds. The RAW files in 20200905/ORF also take about 3 seconds to version detect. The one MOV file in the 20200905/MOV directory is almost instantaneous.

If I select all 217 files in the 20200905 directory and version detect these, it takes around 175 seconds.


3 The number of files in my structure are:-
Image


4 I always shoot JPG & RAW and these are version paired. Only MOV files end up not be versioned. I rarely shoot video (129 in my catalogue). Thus, virtually 100% of my photos end up in version pairs.

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 08 Sep 20 12:18

I did some further testing and it gets weirder. I dragged the 20200905 directory structure to the '000 Finished' directory and rerun the version detection on the 217 files. This time, the versioning in the activity window finishes in 5 seconds with 0% CPU 0% disc activity during that period but then the IDimagerSU.exe app process runs at 30% CPU no disk activity for another 7 minutes 25 seconds while he PS activity window says no activity after which the CPU activity drops to zero. Repeating the version detection gives exactly the same results with 5 sceonds followed between 7 and 7.5 minutes, so it is repeatable. Dragging the 20200905 back to its original location in "Photos from Rocoh" and repeating the version detection on the 217 files gives the same result of 5 seconds for version detection followed by 7 to 7.5 minutes of 30% CPU.

During the first minute or so of the 30% CPU, the thumbnail custom details for the first JPG and ORF version pair blanks as a pair and around five seconds later redisplays as a pair. About half way through the blank period, the next version pair thumbnail custom info blanks for 5 seconds before redisplaying as a pair. This repeats for each version pair visible on the screen. ie. a new pair of thumbnail custom info blanks around every 2.5 seconds.

Incase it is important, my 4 lines of custom thumbnail info are:-
%xmp:photoshop:Headline{encode=html}
%hh:%nn:%ss %dd/%mm/%yyyy %xmp:tiff:Model{encode=html}
%code result := iif(ImageItem._HasRecipe, '<FONT color="#FF00CC00">Recipe </FONT>', ''); %/code%exif:FNumber %xmp:exif:ExposureTimes ISO%xmp:exif:ISOSpeedRatings %xmp:exif:FocalLength
%xmp:dc:subject

I dragged the 20200905/JPG directory up one level under "Photos From Ricoh" and did a version detection on the 217 JPGs and it took 2 minutes 30 seconds at 30% CPU and PS activity showing versioning.

I don't know what I am seeing, but hopefully this makes sense to you Hert.

Some extra background which may assist.
I am trying to import my days photos each day from the SD card to a new date directory with subdirectories for JPG and ORF. I also want the JPG and ORF to be detected as version pairs. For the versioning to detect the pairs, I need to set the Version Set offset folder to "Current and Sibling Folders". While this works, the versioning is painfully slow as already reported. Am I doing something wrong?

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 08 Sep 20 14:34

Hmm, how you describe it, it would be the change in display, switching from single to version thumb that slows down the operation.

Try this;
1. Select the files again as you did before
2. Start the version detection
3. Select a different folder so that the images being versioned are no longer loaded and no change of thumb is needed

Does that make it faster? If so that confirms my suspicion.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 08 Sep 20 15:10

When I select a different folder, the CPU still remains at 30% for the 7 odd minutes. Interestingly however, all the thumbnails in the directory are displayed with a second but the custom thumbnail info for each is blank. The custom thumbnail info is populated at about 1 photo every 3 seconds.

By the way, if I exit Photo Supreme during the 7 minutes of 30% CPU by hamburger exit, the GUI disappears but the IDimagerSU.exe background process remains still consuming 30% CPU and does not terminate even when the CPU usage goes to zero percent.

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 09 Sep 20 14:35

The custom thumbnail info is populated at about 1 photo every 3 seconds.
Now I'm curious what custom thumb info lines you have defined. Can you share each line here? Thank you
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 10 Sep 20 0:01

I listed my custom thumbnail info previously, but here it is again.

%xmp:photoshop:Headline{encode=html}

%hh:%nn:%ss %dd/%mm/%yyyy %xmp:tiff:Model{encode=html}

%code result := iif(ImageItem._HasRecipe, '<FONT color="#FF00CC00">Recipe </FONT>', ''); %/code%exif:FNumber %xmp:exif:ExposureTimes ISO%xmp:exif:ISOSpeedRatings %xmp:exif:FocalLength

%xmp:dc:subject


I did some more testing for you. I turned the thumbnail Style to Normal (turning off the Custom Thumbnail Info) and the timings are the same. Doing a version detection on a date directory still takes about 5 seconds of versioning activity plus about 7 minutes of 30% CPU with no displayed PS activity. Presumably, this means that the custom thumbnail info is not the issue as turning it off does not change the behavior.

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 10 Sep 20 8:44

Can you try this:
1. Select one of the files in the 20200905\JPG folder (not more than one)
2. Run the script below with Tools -> Scripter
3. Post the results you're getting here

Code: Select all

    function ParentFolderForFile(AFileName: WideString): WideString;
    var
      ADrive: WideString;
      ATokens: TTntStringList;
      i: Integer;
    begin
      ADrive    := WideIncludeTrailingBackslash(WideExtractFileDrive(AFileName));
      AFileName := MidStr(AFileName, Length(ADrive) + 1, Length(AFileName));

      ATokens := TTntStringList.Create;
      try
        Tokenize(WideExcludeTrailingBackslash(WideExtractFilePath(AFileName)), '\', ATokens);

        if ATokens.Count >= 2 then
          ATokens.Delete(ATokens.Count - 1);

        result := ADrive;
        for i := 0 to ATokens.Count - 1 do
          result := result + ATokens.Strings[i] + '\';
      finally
        ATokens.Free;
      end;
    end;

var
  AItem: TImageItem;
  AOffset, ATemp: WideString;
  i: Integer;

  APath: TCatalogFilePath;
  APaths: TCatalogFilePaths;
begin
  if Selected.Count <> 1 then
  begin
    Say('Select just one file');
    exit;
  end;

  AItem := Selected.Items[0];
  AOffset := ParentFolderForFile(AItem.FileName);
  Say2('Offset folder', AOffset);

  APaths := TCatalogFilePaths.Create(TCatalogFilePath, '');
  APath := TCatalogFilePath.Create(nil);
  try
    APath.FilePath := AOffset;
    APath.UpdateFileInfo;
    PublicCatalog.EnumFilePathsForParent(APath, APaths);

    ATemp := '';
    for i := 0 to APaths.Count - 1 do
      ATemp := ATemp + CrLfW + APaths.Items[i].FilePath;

    Say2('Folder to scan', ATemp);
  finally
    APath.Free;
    APaths.Free;
  end;
end;
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 10 Sep 20 15:00

The results are:-
Image
Image

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 10 Sep 20 18:22

Thank you. PSU scans the correct folders to detect versions in. I wanted to double check because you mentioned it being so much faster when moved to another folder. But that shouldn't make a difference.

Can you replace your 3rd custom thumb line with this:

Code: Select all

%code result := iif(ImageItem._HasRecipe, '<FONT color="#FF00CC00">Recipe </FONT>', ''); %/code%xmp:exif:FNumber %xmp:exif:ExposureTimes ISO%xmp:exif:ISOSpeedRatings %xmp:exif:FocalLength
Instead of %exif:FNumber it uses %xmp:exif:FNumber, which is faster to retrieve as the info is in the database where %exif displayes directly from the file, meaning it needs to load the exif for every file.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 11 Sep 20 13:17

Thank you for that suggestion Hert. I have updated my third line. I haven't noticed any difference, but as you said, it should make the thumbnail info faster.

It seems like we are running out of ideas. My only other thought is my synchronization settings. I have verified the first page of settings are as per the default settings, but I am not sure if the the second page of settings are as default. Would any of my synchronization settings cause my slow versioning?

Image
Image
Image
Image

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 11 Sep 20 14:24

Your synbc settings are fine.
Can you try disabling the placeholder detection one by one and see if it improves the performance? Maybe those are the culprit(?).
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

gcorbin
Posts: 59
Joined: 21 Aug 06 12:31
Location: Brisbane

Re: Versioning on Import Extremely Slow

Post by gcorbin » 12 Sep 20 21:00

Ok. I have extensively played around with all the version detection settings and have worked out how to reproduce this issue. Disabling all my version rules and reverting to just the default rule does not change the version detection speed.

If you set the Version Configuration settings as below, you can create or avoid the slow version detection when doing version detection on the photos in a folder structure like 20200905.
Image
Unfortunately, the versioning when importing is still slow when I set Version Set Display to Always together as a Set and Current and Siblings Folder.

I also updated PS from 5.5.1.3084 to 5.5.1.3117 and importing now gives the error:-
[Import Graham.profile] {Assigning Portfolio} “4173605001431001” is an invalid integer (0)
I cancelled the import and tried to read metadata from the images and PS couldn’t read the metadata from my images.

I updated to the 5.5.1.3119 and the import now stops at 50% with no disk activity and no CPU activity. I can still use PS normally, but the import did not complete. The images have no metadata.

I reverted back to my tried and true 3084 and everything is fine again. I can import from my SD card as expected with all metadata read although the versioning is still slow. Is there an issue with importing with these later versions of PS or does my catalogue have some issue?

Any thoughts on what else I can try?

Hert
Posts: 6333
Joined: 13 Sep 03 7:24

Re: Versioning on Import Extremely Slow

Post by Hert » 13 Sep 20 6:45

Always check https://whatsnew.idimager.com to see changes in builds.

Can you
1. Send me a (zipped) copy of your database (only the cat.db, no need for thumbs.db)
2. Send me some of your ORF JPG combinations (at least 4 sets so I can check the version detection)?
3. Send me your import profile

FWIW; importing works fine here, then again I don't know what you have configured in your import profile and so I can only guess

Thank you
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Post Reply