Compacting thumbnail database takes a long time

[Larry56]

I just updated to new version on Windows 10 and was asked if I wanted to update catalog. I said yes and PSu started. Took a little time to complete the catalog - about 5 minutes. The thumbnail catalog though has been working for 25 minutes and it's only 22 GB.

I don't remember compacting taking this long in the past. Anyone else notice a longer time with this version?

[Sigi]

Thumbnail compacting took always significantly longer on my computer compared to the catalog.

[G8DHE]

I must admit since updating to the last release 5.3.0.2592 a couple of days ago, compacting the database has slowed to an absolute crawl and display of thumbnails is also badly affected with it taking an age to display the full image in the meantime a base colour is displayed which is change from previous ?

[Hert]

How long a compact takes is outside of PSU's control. That is SQLite at work.
And the display of thumbnails is faster in 5.3 when compared to 5.2. You're correct that the base color pre-display is new. In 5.2 the pre-display was a blank gap.

[G8DHE]

Would installing the server version with one of the other databases be worth while in producing a speed up ?

[Hert]

Can you define the performance?
For example,
1. First compact the database to rule out performance drops due to fragmentation
2. Then restart PSU
3. Open a folder with about 3,000 images (click the counter).

How fast does the folder open? Here that is around 0,7 second

4. Open a folder with about 15,000 images

How fast does the folder open? Here that is around 2,5 seconds

Also share some of your hardware details. I'm testing on an i5 8259U with 16GB RAW, with a 512 SSD (nvme). The catalog is stored on the SSD (of course). The images are all stored on a network share.

[Hert]

In addition to my previous post;
Performance can be also heavily impacted by your thumbs settings. Start with setting the thumbs at 100%. If your thumbs are too big to paint then PSU needs to load the large previews so that it can still paint a crisp thumbnail. That heavily impacts performance (up to 8 times slower). And if you set your thumbs too small that the large number of thumbs that fit in one screen also impact the performance. You should find a good balance there.

[G8DHE]

Thanks for getting back so quickly Hert, appreciated.
I'll get the info together for my setup, and at some point over the holidays I'll get some basic timings done!
Trouble at this time of year is that others seem to have ideas on what one should be doing other than looking at screens

[Larry56]

When I said it took 25 minutes I should have also said I had to force quit (via Task Manager) as I couldn't let run any longer.

Today I started PSu 5.3.0.2592 on Windows 10 and selected Compact Thumbnail Database. It's been running 1 hour and 32 minutes and is still compacting. I had last compacted about 2 weeks ago under an older version.

Thinking there may be a problem on my notebook I decided to test on my desktop. This installation has a small thumb database (17gb vs 22gb) and had v5.1.0.2446 on Windows 10. I started it and updated a couple folders and then ran the compact. The catalog database took about 1 minute and the thumbnail database took 13 minutes.

I then updated to 5.3.0.2592 and let it update the database at startup. When done I ran the compact database and it took about 2 minutes. I then ran the compact thumbnail database. It's still compacting after 60 minutes where under the old version it was only 13 minutes.

I currently have both computers compacting thumbnail database so the times will be actually longer.

Is this expected or is something else going on?

[G8DHE]

Something has obviously changed as there are now two additional .db files created .db-shm and .db-wal the .db-shm file was changing in size during the compacting process.

[Larry56]

I ended up force quiting via Task Manager since normal way's to stop didn't work. The notebook was still working after 2 hours 19 minutes and the desktop 2 hours 10 minutes.

I did some further testing. I uninstalled 5.3.0.2592 on my notebook and found a v5.2.0.2513 to install. I then copied an old catalog and thumb catalog to the catalog folder and installed 5.2.0.2513. Starting it I was asked to update the catalog and clicked OK. When done I updated some folders that were missing in the catalog. Then, to be safe I backed up to a new folder on an external SSD. I exited and when I went back into PSu I received the compact message and said yes to compact. The regular catalog took 1 minute and the thumb catalog took 14 minutes to compact under v5.2.0.2513.

Since the compacting thumb catalog had to be force stopped after 2 hours on two computers with different catalogs but same newest version I'm assuming there is something with this new version.

[G8DHE]

Ah ha, it appears that the two new .db-shm|wal files are from sqlite3.7 version
https://stackoverflow.com/questions/777 ... -databases

[Larry56]

I can confirm both files were created on both the notebook and desktop. Your link indicates the should be deleted on completion? They may be left because I early terminated.?

[G8DHE]

I've never seen them appear during or after before, apparently used with Write Ahead Logic and SHared Memory, so maybe its been turned on in this latest version ??

[Mikki]

Hi all

I can confirm that database compacting process has slowed down significantly on my installation too, since I yesterday upgraded to version 5.3.0.2593.
The databases are stored locally on a 1 TB SSD (idimager.cat.db=3.2 GB and idimager.thumbs.db=34.7 GB). Images are stores locally on a conventional 4 TB hard drive. The PC is i7-8700 with 32 GB RAM installed.

The compacting process has so far been ongoing for 5 hours and 30 min. Currently the status bar indicates “Compacting catalog database…”. Task Manager indicates the IDimagerSU.exe is consuming only 0.2% – 0.4% CPU.

I hope the above can contribute to bringing down time needed to conduct the compact exercise.
If more info I needed, please let me know.

BR
Mikki

database sizes.png (20.36 KiB) Viewed 7660 times

Catalog Settings.png (77.33 KiB) Viewed 7660 times