anyone having problems with Verify Folders for Files?

[Hert]

My images are all on a network share (Windows server) and I'm testing with a Wifi (AC) connected machine.

Performance for sync read won't be impacted by TTP. What can impact import performance significantly are CRS edits from Lightroom or ACR in your XMP. Especially when you use brushing in Lightroom, the CRS metadata can quickly become HUUUUUUGE. And PSU has to index that all.

[G8DHE]

OK interesting result, the network is a LAN not WiFi;
LocalImage result;

SyncTimingLaptopLocalImage.JPG (128.06 KiB) Viewed 6117 times

RemoteImage result;

SyncTimingLaptopRemoteImage.JPG (127.36 KiB) Viewed 6117 times

So a bit over double the time or half the speed, when the current sync has finished on my main workstation I'll repeat the test on the same image.

[Hert]

0.86 seconds is a good score. And since the difference with your network is large I think that means that your network has above average latency. is that a NAS or a Windows share?

As mentioned before, the image I imported was on a Windows network share and that took 0.61 seconds to import.

I now copied that image to my local SSD drive. With this result:

Sync Read Speed 2.png (49.81 KiB) Viewed 6116 times

Network vs local is almost the same. Actually I'm a bit surprised about that. Guess that means that my network hardly has any latency.

[G8DHE]

Right same script on the Work Station currently running the larger Sync job mentioned above 5072 files with the Activity pane closed;

SyncTimingWorkStationRemoteImage.JPG (80.11 KiB) Viewed 6110 times

I'll run it again later when the larger Sync job has finished as it will no doubt have had a serious impact on the run time.

Both the Laptop and Work Station are running W10 Pro, the file images are on a windows share to an old W2k3 server.

[G8DHE]

OK, well the second large Sync run has completed OK, about the normal speed on this setup. Which now with 2 runs without locking up looks promising, this is with the Activity panel closed, so maybe this is something that can be explored further ? I have some more sizable runs to complete but not quite sure on when I will be able to fit them in, possibly one tonight then a gap as I'll be remote for several days - or possibly longer depending on the Covid-19 rules!

[vkfoto]

New build 3361
Ran the script and averaged 0.3 sec per image

done in dot3.JPG (17.5 KiB) Viewed 6035 times

[vkfoto]

I've been banging away at applying labels to many hundreds of images, some that didn't have any at all and others that were missing some. Not a sync freeze in sight.

The only thing that changed was the latest upgrade to build 3361.
Either something was fixed or changed that was causing me so many issues,
or
Something in the previous upgrade went wrong in my installation.

[Hert]

@vkfoto; 0.3 seconds is a more than excellent score!!

@Geoff; can you re-run the script now that your import has completed? I'm curious how it scores on your workstation (was 36.03 sec), on your laptop+remote access (was 1.95 sec). I would expect that the workstation+remote should be able to achieve a similar score as the laptop+remote

[yar]

On my desktop the script runs in 0.12 to 0.24 sec. And I use normal thumb view

But my problem has not gone with the new build. If I chose 'export data to file' after 'verify folder quick' the synchronize process got stuck on folders created in 2020. If I, however, restart PSU, select all files in the same directory, and press Ctrl-s synchronization works without any problem.

So it is not a big deal as I can simply rewrite data for all images in the unfortunate 2020.

[G8DHE]

Right Hert yes with no other process running the result is

sync4.jpg (50.71 KiB) Viewed 5979 times

Which is what I would expect. Also now running 3361 in thumbnail view.

[Hert]

That’s a great result Geoff. Now can you sync-read test for a 100 images? If one image takes 0.34 seconds then 100 image should come around 35 seconds (+/- 10 secs).

[G8DHE]

Just tried running the single file script a number of times and there is obviously caching going on as the first attempt is around 0.55 seconds and then subsequent runs on the same file are around 0.17 seconds. Also tried it on a variety of different files a single time only and the variation was from 0.2 to 11.06 seconds over about 6 attempts, I haven't controlled for file size so random file sizes mostly. Anyway running a Compact on both databases and will try again shortly and then see what gives on Syncing on batches of larger file number.
Any chance of a Script to log each attempt and to repeat the batch of files say 10 times ? Then we would have a more repeatable set of results for comparison.

[G8DHE]

OK well Read Metadata for a batch 0f 100 similar sized JPG files yields 6:08 Minutes or 368 seconds so that is an average of 3.68 seconds each .....

[Hert]

6 minutes to import 100 images...that's slow!

I've created a new script and attached it below. This script allows you to select multiple files and it calculates the time needed for syncing.

This time, again select 100 files and run the script.

Code: Select all

var
  AOffsetStart, AOffsetEnd, AStart, AEnd, ASum, AAvg: Cardinal;
  i: Integer;
  AImage: TImageItem;
  APro: TxomProgress;
begin
  if Selected.Count = 0 then
  begin
    Say('Please select at least 1 thumbnail');
    exit;
  end;

  APro := TxomProgress.Create(nil);
  try
    APro.Caption := 'Sync Read Speed Benchmark';
    APro.Max := Selected.Count;

    AOffsetStart := GetTickCount;
    ASum := 0;
    for i := 0 to Selected.Count - 1 do
    begin
      AImage := Selected.Items[i];

      APro.Pos := i + 1;
      APro.ProgressText := AImage.FileNameOnly;

      AStart := GetTickCount;
      PublicCatalog.ReadXMPForImageAdvanced(AImage, PublicOptions.XMPSyncReadSettings, nil);
      AEnd := GetTickCount;

      ASum := ASum + (AEnd - AStart);
    end;
    AOffsetEnd := GetTickCount;
  finally
    APro.Free;
  end;

  AAvg := ASum / Selected.Count;
  Say2(
        IntToStr(Selected.Count) + ' files',
        'Done in ' +
          FloatToStr(RoundToPrecision((AOffsetEnd - AOffsetStart) / 1000, 2)) + ' sec. ' +
        'Total time for sync is ' +
          FloatToStr(RoundToPrecision((ASum) / 1000, 2)) + '. ' +
        'Average per image is ' +
          FloatToStr(RoundToPrecision((AAvg) / 1000, 2)) + ' sec' +
        '');
end;

This is the result on my machine.

Sync Read Speed 3.png (57.32 KiB) Viewed 5880 times

As you can see, sync'ing-read 100 files takes about 40 seconds to complete. Keep in mind that the application will be even faster because then screen updates are optimized, which a script can't do.

If I sync-read the same 100 files with right click -> Metadata -> Read metadata from file then these same 100 files takes about 7 seconds to import. I've recorded it in this video: https://cp.idimager.com/support/videos/ ... dSpeed.mp4
This screen was recorded on a $300 low end machine equipped with an Intel i3-8gen running latest Windows 10 Pro (virgin install, nothin else installed but PSU) and running PSU with all preferences at default. Images are on an external USB3 connected 4TB Seagate HDD(!) drive. Catalog is on the internal M2-SSD (256GB Corsair MP510). This is the lowest end machine I have available here. Anyone running a modern machine should be able to achieve these same timings.

And I'm very curious to learn what the causes is why some people get different results. The difference is most likely in the hardware.

PS. When I run this same import of 100 files on the fastest machine I have here (an i7-10gen with a fast 512GB SanDisk Extreme M2 SSD) then the same sync-read import takes 4 seconds to complete. A sync write of 100 files takes 12 seconds to complete.

[G8DHE]

Right using the script on a 100 files gives times all in the 40-50 second range;

multi-sync-4.JPG (17.65 KiB) Viewed 5828 times

Doing a Manual read of the same files took 16-17 seconds on the one shown above.

There seems to be a lot of variability at times, despite the machine not having anything significant running on it, I'm accessing it remotely over VNC as I'm away from home at present.

Machine;
OS Name Microsoft Windows 10 Pro
Version 10.0.18363 Build 18363
System Type x64-based PC
Processor Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz, 3501 Mhz, 6 Core(s), 12 Logical Processor(s)
The programs and database are on a 1Tbyte SSD and the images are via LAN connection to a Windows server machine.

PSU is 5.6.3361