compression very lenthy

johnbarrett
Posts: 18
Joined: 27 Jun 18 15:36

compression very lenthy

Post by johnbarrett » 13 Feb 20 19:12

This had speed up but was there a change in the last update as it took over an hour. It was under 1/2 that befor today.

Hert
Posts: 6257
Joined: 13 Sep 03 7:24

Re: compression very lenthy

Post by Hert » 14 Feb 20 0:00

I assume you mean compacting? The last change was in build 2663. As can be found on https://whatsnew.idimager.com

Build 2663; In the Single User edition, compacting the database will also do a rebuild of the indexes

That unfortunately does cost time, but has proven to be a necessity.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

johnbarrett
Posts: 18
Joined: 27 Jun 18 15:36

Re: compression very lenthy

Post by johnbarrett » 14 Feb 20 9:33

Will that be done each time now?

Hert
Posts: 6257
Joined: 13 Sep 03 7:24

Re: compression very lenthy

Post by Hert » 14 Feb 20 9:48

Hmmm, that's a good remark and food for thought.
Currently, as it is, the answer is yes.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Hert
Posts: 6257
Joined: 13 Sep 03 7:24

Re: compression very lenthy

Post by Hert » 14 Feb 20 10:14

BTW; I did some tests and reindexing adds about 15% to the total compacting time. That is not even close to the 100% "longer" that you mentioned. Maybe you can re-verify your timings

Also make sure that you exclude the catalog folder for any possible virus scanner that you have installed (even the one that may come with your operating system).
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

johnbarrett
Posts: 18
Joined: 27 Jun 18 15:36

Re: compression very lenthy

Post by johnbarrett » 14 Feb 20 10:25

Will do so when home and have the time.
John

johnbarrett
Posts: 18
Joined: 27 Jun 18 15:36

Re: compression very lenthy

Post by johnbarrett » 17 Feb 20 11:58

I added the contents of the catalogue folder into Norton. Started compression at 9.47am and it finished at 10.45am. The size was 7.75Gb cpu usage was around 18% , memory usage was around 160MB and disk between 7 and 25MB/s. The time taken before was well under 20 m. If the index rebuilding is remaining permanently could there be an option to close the program and PC down after finishing and it could be set running unattended.

The basic spec is Physical Memory: 16304 MB
Processor: Intel(R) Core(TM) i7-4900MQ CPU @ 2.80GHz
Processor Speed: 2794 MHz
1 SSD HD 500Mb
1 SSD HD 1Tb

fbungarz
Posts: 1560
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: compression very lenthy

Post by fbungarz » 18 Feb 20 23:03

I think it is most likely a good thing that index-rebuilding is now part of the routine. Previously, my database kept growing and growing - despite compacting it regularly. As a hack, I finally figured out a way to delete the index table in SQLite Expert and then run a batch command to rebuild the index. Without the index the main database (not the thumbnail db) is minute. Re-building the index usually took a long time, but was worth doing it ever so often, because after re-building the index the database was still a lot smaller than doing a simple compacting.
So, even if the whole process might take a lot longer, I believe it is most likely worth the wait.

My only concern now is this:
For speed, I have the database on my relatively small SSD drive. Last time I tried compacting the thumbnail database, I had to copy it to another drive, else the SSD would have run out of space. In my experience, you need essentially twice the hard disk space of the full size of the database, else you'll hit a ceiling and the compacting will fail.
I am not sure if there is any way to avoid having to move my database to another drive each time I want to compact - and with the addition of a nag-screen telling me ever so often to please compact, this procedure is actually quite a nuisance. Any chance the compacting could be configured to utilize the space on another drive for the temporary copy and not the primary SSD, where my database is located?

Hert
Posts: 6257
Joined: 13 Sep 03 7:24

Re: compression very lenthy

Post by Hert » 18 Feb 20 23:50

You can now run this script to disable the startup compacting. Keep in mind that you still should compact regularly;

Code: Select all

WriteToRegistry('', 'CheckLastCompactDatabaseOnStartup', 0);
To restore default behaviour run this script:

Code: Select all

WriteToRegistry('', 'CheckLastCompactDatabaseOnStartup', 1);
Previously, my database kept growing and growing - despite compacting it regularly.
This was fixed in build 2705. When running that build or higher then run Tools -> Build Missing Thumbnails and Previews at least once. That will clean up unused thumbs/previews.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

andrew.heard
Posts: 171
Joined: 16 Jun 10 0:36

Re: compression very lenthy

Post by andrew.heard » 19 Feb 20 10:20

I haven't used the scripter before now. The above script worked, because registry value REG_SZ \HKEY_CURRENT_USER\SOFTWARE\IDimager Systems, Inc.\IDI\CheckLastCompactDatabaseOnStartup is set to 0. However it is disconcerting that there is no user feedback when the Run button is clicked. I clicked it a few times, expecting something to happen. I guess I could add an extra line with a msgbox, but should it be necessary? Maybe a msgbox with a checkbox "dont show this again" could be displayed by default after running a script?

Username
Posts: 262
Joined: 18 Feb 18 22:21

Re: compression very lenthy

Post by Username » 19 Feb 20 10:41

@Hert
I have autovacuum setup and do reindexing by script for my postgres 12.
Does PSu notice that or does it have its own timer for reminders?
PSu Server 5 & Postgres 12 on macOS 10.14
PSO 5 on Windows Server 2016

- I'm the user

Hert
Posts: 6257
Joined: 13 Sep 03 7:24

Re: compression very lenthy

Post by Hert » 19 Feb 20 10:54

I have autovacuum setup and do reindexing by script for my postgres 12.
Great, that's the way to do it for a professional database system.
Compacting in the Server Edition takes seconds to complete. And unlike in the Single User edition, the Server Edition gives the option to remind again after x days. And is only presented when the admin logs in.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Username
Posts: 262
Joined: 18 Feb 18 22:21

Re: compression very lenthy

Post by Username » 19 Feb 20 14:41

Thanks Hert.
I'm more than happy for that info. :)
Last edited by Username on 19 Feb 20 22:09, edited 1 time in total.
PSu Server 5 & Postgres 12 on macOS 10.14
PSO 5 on Windows Server 2016

- I'm the user

rfcat
Posts: 69
Joined: 21 Feb 07 11:38
Location: Australia

Re: compression very lenthy

Post by rfcat » 29 Feb 20 0:24

Compacting using on a single user version takes a long time as previously noted. PSU uses a lot CPU time and never shows progress.
The lack of progress information is the annoying part.
Latest PSU and MAC OS installed.
Ian
Ian M
Photography, understanding how to catalogue so I can find my photos. :D .

johnbarrett
Posts: 18
Joined: 27 Jun 18 15:36

Re: compression very lenthy

Post by johnbarrett » 29 Feb 20 9:26

I agree about the lack of progress information, you have no idea how it's doing. My experience is its way longer than just a small increase in time. I had asked for an option to shut the pc down if it's left to run, but it's so long I have to stop my pc from going into hibernation more than once. I would have to setup a power option to not use this or the program needs one along with a turn off when finished

Post Reply