anyone having problems with Verify Folders for Files?

fbungarz
Posts: 1618
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: anyone having problems with Verify Folders for Files?

Post by fbungarz » 20 Dec 20 1:08

The only thing that seems to work most of the time is to just close PSU and then open it again. Then I can sync any images that were not snyced the last time.
Same here, unfortunately.
:(
This did NOT happen in the past. I remember, I used to regularly sync 1,000+ images (took forever, but did not stall). Now it stalls at ca. 100 or less...

fbungarz
Posts: 1618
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: anyone having problems with Verify Folders for Files?

Post by fbungarz » 20 Dec 20 1:12

BTW - a workaround is stopping the sync queue via x, re-start and then start syncing again, until it stalls, stopping the queue, restart again...
OR: sync in small batches only, ca. 20 at a time, until that no longer works, then stop it, restart, etc.
It basically means syncing of large batches of images in one go is no longer possible...
Upon restart syncing always works really fast, then slows down considerably and eventually completely stalls.

Hert
Posts: 6557
Joined: 13 Sep 03 7:24

Re: anyone having problems with Verify Folders for Files?

Post by Hert » 20 Dec 20 4:27

I sync thousands of images every day here and never have problems. At least once a day I sync-read and sync-write 20k images in one go.

What strikes me is that both of you run the database from an external drive. Is that correct? If so, are you also both using HDD instead of SSD?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

vkfoto
Posts: 190
Joined: 19 Oct 16 3:51

Re: anyone having problems with Verify Folders for Files?

Post by vkfoto » 20 Dec 20 4:57

My database as well as the program itself are on separate internal SSD. The images themselves are on a external HDD NAS. Up until the recent upgrade, I don't recall having so many sync issues.

fbungarz
Posts: 1618
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: anyone having problems with Verify Folders for Files?

Post by fbungarz » 20 Dec 20 7:32

Almost the same here; database and program on the same internal SSD, images on internal HDD.

vkfoto
Posts: 190
Joined: 19 Oct 16 3:51

Re: anyone having problems with Verify Folders for Files?

Post by vkfoto » 20 Dec 20 22:59

Getting tired of constantly having to restart PSU, I thought I would cleanup my label categories. I wasn't too consistent when I started them. So I removed all the plural versions and made them singular, made the first letter uppercase and merged and relocated some. All in all, close to 20k images had to be resynced and it worked without a hitch. Let it run overnight and by morning it was all done.
So changing an existing label in an image works as expected
Adding a new label to an images causes a lot of stalls.
Is there some sub-routine in the sync process that is only invoked for new labels that could account for this?

vkfoto
Posts: 190
Joined: 19 Oct 16 3:51

Re: anyone having problems with Verify Folders for Files?

Post by vkfoto » 21 Dec 20 2:28

After a detour to compact and then backup, I continued to add new labels to existing images. A few images synced correctly. Then I created a new label and added an existing label to a set of six images. Synchronize Service has been stuck at 0.0% for more than ten minutes. Sigh

vkfoto
Posts: 190
Joined: 19 Oct 16 3:51

Re: anyone having problems with Verify Folders for Files?

Post by vkfoto » 21 Dec 20 3:29

I think I may have found something.
When I got stuck again, I noticed that some of the images on disk had either been deleted or had been damaged in someway, i.e. they were not the same as what was shown in the folder view. It seems that the synchronize service gets stuck on them instead of ignoring and moving on.
With over 280k images and many that can be modified by other programs, especially the derivative images like tif and jpg, this will be a constant issue.

If my hypothesis is correct, can the synchronize service be adjusted to just ignore missing or damaged files, perhaps issue an exception report, instead of freezing the program?

When PSU freezes, db-wal and db-shm files are left behind in the catalog folder and there is a IDImageSU.exe in a suspended state with no threads as per resource monitor. The only way I found to delete them is to reboot.

yar
Posts: 7
Joined: 19 Jul 10 15:50

Re: anyone having problems with Verify Folders for Files?

Post by yar » 21 Dec 20 8:54

I did some more tests and it seems that the synchronize service got stuck after verifying folders with images created in some time window in 2020.
The steps to reproduce the error are:
"Verify folder all" reports that there is no changes;
"Verify folder quick" reports that some files have been changed;
When for these files "Export data to file" is chosen the synchronize service got stuck for folders created in 2020. I could not determine exact time limits yet.
For older folders synchronizations (of =<40 files) works fine.
My folder structure is: pictures/year/day. jpg and cr2 files are combined into version sets

Hert
Posts: 6557
Joined: 13 Sep 03 7:24

Re: anyone having problems with Verify Folders for Files?

Post by Hert » 21 Dec 20 9:05

Yesterday I did another test and I can synchronize (read and write) 20K as stored on a network share without issues here. Altered files and volumes can not be the issue here, unless there's some kind of corruption in the file that trips PSU in an unpredictable way (fat chance as PSU will just skip it). And, knowing how the sync process is implemented, I don't think that the sync process is capable of hanging at all.

A few days back I've asked to try but nobody reported back:
1. Don't click anywhere while the progress runs. No difference? then try:
2. Keep the progress box closed (upper right icon in the progress box) and then sync.
Also for (2) don't click anywhere in the application while it runs
And I'll add this question:
3. Are you in Grid or Thumb view?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

User avatar
G8DHE
Posts: 274
Joined: 21 Aug 17 13:58

Re: anyone having problems with Verify Folders for Files?

Post by G8DHE » 21 Dec 20 10:58

OK just trying this Hert, Verify Quick, has identified 3262 files changed, I know there have been no image data changes so killed the Thumbnail rebuild, then closed the progress window started at 09:36 running now ........... will report back when I see CPU activity drop to a low level for a while!
Geoff Mather (G8DHE)

User avatar
G8DHE
Posts: 274
Joined: 21 Aug 17 13:58

Re: anyone having problems with Verify Folders for Files?

Post by G8DHE » 21 Dec 20 14:16

OK, well it ended 13:11 so that's 3:05Hrs so its 3.4 seconds per sync, which seems reasonable, but it is only a single test of the sequence and others, usually smaller in size, have locked up previously.
I'll have another large batch to be checked later today so will time that with the Activity panel closed and see what goes.
Geoff Mather (G8DHE)

User avatar
G8DHE
Posts: 274
Joined: 21 Aug 17 13:58

Re: anyone having problems with Verify Folders for Files?

Post by G8DHE » 21 Dec 20 14:25

OK second test coming up
Image
So 5072 to be sync'ed
started at 13:24 so 4:47 if the same speed ....
Geoff Mather (G8DHE)

Hert
Posts: 6557
Joined: 13 Sep 03 7:24

Re: anyone having problems with Verify Folders for Files?

Post by Hert » 21 Dec 20 14:40

3.4 seconds per sync is on the slow side. Speed may depend on the size of your metadata. Here one file is sync'ed (read) in under one second.

You can benchmark with the script below.

My import performance timings are here; 21K images import in 36 minutes:
viewtopic.php?f=57&t=28271&p=125282#p125280

Code: Select all

var
  AStart, AEnd: Cardinal;
  AImage: TImageItem;
begin
  if Selected.Count <> 1 then
  begin
    Say('Please select exactly 1 thumbnail');
    exit;
  end;

  AImage := Selected.Items[0];

  AStart := GetTickCount;
  PublicCatalog.ReadXMPForImageAdvanced(AImage, PublicOptions.XMPSyncReadSettings, nil);
  AEnd := GetTickCount;

  Say2(AImage.FileName, 'Done in ' + FloatToStr(RoundToPrecision((AEnd - AStart) / 1000, 2)) + ' sec');
end;
PS. Can you also answer my questions?
Attachments
Sync Read Speed.png
Sync Read Speed.png (54.5 KiB) Viewed 563 times
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

User avatar
G8DHE
Posts: 274
Joined: 21 Aug 17 13:58

Re: anyone having problems with Verify Folders for Files?

Post by G8DHE » 21 Dec 20 14:57

Yes, I would agree I normally expect a few files to be done in very few seconds, but when I get these situations where other programs (TagThatPhoto) do mass updates its always been a LOT slower, the database is on the SSD but the images are all over the network on a separate server.
I'll try the script on my laptop which has some images on the local HD and others over the network to the same server and see how it compares.
Geoff Mather (G8DHE)

Post Reply