anyone having problems with Verify Folders for Files?

[fbungarz]

The only thing that seems to work most of the time is to just close PSU and then open it again. Then I can sync any images that were not snyced the last time.

Same here, unfortunately.

This did NOT happen in the past. I remember, I used to regularly sync 1,000+ images (took forever, but did not stall). Now it stalls at ca. 100 or less...

[fbungarz]

BTW - a workaround is stopping the sync queue via x, re-start and then start syncing again, until it stalls, stopping the queue, restart again...
OR: sync in small batches only, ca. 20 at a time, until that no longer works, then stop it, restart, etc.
It basically means syncing of large batches of images in one go is no longer possible...
Upon restart syncing always works really fast, then slows down considerably and eventually completely stalls.

[Hert]

I sync thousands of images every day here and never have problems. At least once a day I sync-read and sync-write 20k images in one go.

What strikes me is that both of you run the database from an external drive. Is that correct? If so, are you also both using HDD instead of SSD?

[vkfoto]

My database as well as the program itself are on separate internal SSD. The images themselves are on a external HDD NAS. Up until the recent upgrade, I don't recall having so many sync issues.

[fbungarz]

Almost the same here; database and program on the same internal SSD, images on internal HDD.

[vkfoto]

Getting tired of constantly having to restart PSU, I thought I would cleanup my label categories. I wasn't too consistent when I started them. So I removed all the plural versions and made them singular, made the first letter uppercase and merged and relocated some. All in all, close to 20k images had to be resynced and it worked without a hitch. Let it run overnight and by morning it was all done.
So changing an existing label in an image works as expected
Adding a new label to an images causes a lot of stalls.
Is there some sub-routine in the sync process that is only invoked for new labels that could account for this?

[vkfoto]

After a detour to compact and then backup, I continued to add new labels to existing images. A few images synced correctly. Then I created a new label and added an existing label to a set of six images. Synchronize Service has been stuck at 0.0% for more than ten minutes. Sigh

[vkfoto]

I think I may have found something.
When I got stuck again, I noticed that some of the images on disk had either been deleted or had been damaged in someway, i.e. they were not the same as what was shown in the folder view. It seems that the synchronize service gets stuck on them instead of ignoring and moving on.
With over 280k images and many that can be modified by other programs, especially the derivative images like tif and jpg, this will be a constant issue.

If my hypothesis is correct, can the synchronize service be adjusted to just ignore missing or damaged files, perhaps issue an exception report, instead of freezing the program?

When PSU freezes, db-wal and db-shm files are left behind in the catalog folder and there is a IDImageSU.exe in a suspended state with no threads as per resource monitor. The only way I found to delete them is to reboot.

[yar]

I did some more tests and it seems that the synchronize service got stuck after verifying folders with images created in some time window in 2020.
The steps to reproduce the error are:
"Verify folder all" reports that there is no changes;
"Verify folder quick" reports that some files have been changed;
When for these files "Export data to file" is chosen the synchronize service got stuck for folders created in 2020. I could not determine exact time limits yet.
For older folders synchronizations (of =<40 files) works fine.
My folder structure is: pictures/year/day. jpg and cr2 files are combined into version sets

[Hert]

Yesterday I did another test and I can synchronize (read and write) 20K as stored on a network share without issues here. Altered files and volumes can not be the issue here, unless there's some kind of corruption in the file that trips PSU in an unpredictable way (fat chance as PSU will just skip it). And, knowing how the sync process is implemented, I don't think that the sync process is capable of hanging at all.

A few days back I've asked to try but nobody reported back:
1. Don't click anywhere while the progress runs. No difference? then try:
2. Keep the progress box closed (upper right icon in the progress box) and then sync.
Also for (2) don't click anywhere in the application while it runs
And I'll add this question:
3. Are you in Grid or Thumb view?

[G8DHE]

OK just trying this Hert, Verify Quick, has identified 3262 files changed, I know there have been no image data changes so killed the Thumbnail rebuild, then closed the progress window started at 09:36 running now ........... will report back when I see CPU activity drop to a low level for a while!

[G8DHE]

OK, well it ended 13:11 so that's 3:05Hrs so its 3.4 seconds per sync, which seems reasonable, but it is only a single test of the sequence and others, usually smaller in size, have locked up previously.
I'll have another large batch to be checked later today so will time that with the Activity panel closed and see what goes.

[G8DHE]

OK second test coming up
[ external image ]
So 5072 to be sync'ed
started at 13:24 so 4:47 if the same speed ....

[Hert]

3.4 seconds per sync is on the slow side. Speed may depend on the size of your metadata. Here one file is sync'ed (read) in under one second.

You can benchmark with the script below.

My import performance timings are here; 21K images import in 36 minutes:
viewtopic.php?f=57&t=28271&p=125282#p125280

Code: Select all

var
  AStart, AEnd: Cardinal;
  AImage: TImageItem;
begin
  if Selected.Count <> 1 then
  begin
    Say('Please select exactly 1 thumbnail');
    exit;
  end;

  AImage := Selected.Items[0];

  AStart := GetTickCount;
  PublicCatalog.ReadXMPForImageAdvanced(AImage, PublicOptions.XMPSyncReadSettings, nil);
  AEnd := GetTickCount;

  Say2(AImage.FileName, 'Done in ' + FloatToStr(RoundToPrecision((AEnd - AStart) / 1000, 2)) + ' sec');
end;

PS. Can you also answer my questions?

[G8DHE]

Yes, I would agree I normally expect a few files to be done in very few seconds, but when I get these situations where other programs (TagThatPhoto) do mass updates its always been a LOT slower, the database is on the SSD but the images are all over the network on a separate server.
I'll try the script on my laptop which has some images on the local HD and others over the network to the same server and see how it compares.