Compacting Catalog not where the catalog is.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Compacting Catalog not where the catalog is.

Post by emha » 19 Dec 20 23:37

During I compact my catalog which is on an external drive, the internal drive looses capacity around the gb of the catalog.
Is it possible to generate the temporary files there where the catalog is?

vkfoto
Posts: 214
Joined: 19 Oct 16 3:51

Re: Compacting Catalog not where the catalog is.

Post by vkfoto » 20 Dec 20 1:40

See here
viewtopic.php?f=57&t=28028&p=124872&hilit=temp#p124855
All about where your temp folder is located.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 20 Dec 20 9:15

vkfoto wrote:
20 Dec 20 1:40
See here
viewtopic.php?f=57&t=28028&p=124872&hilit=temp#p124855
All about where your temp folder is located.
Thank you, but I’m on Mac and the solution seems to be for windows.

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 20 Dec 20 14:17

That's not different on macOS. Specify a folder on your system to use as the temp dir.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 20 Dec 20 17:42

Hert wrote:
20 Dec 20 14:17
That's not different on macOS. Specify a folder on your system to use as the temp dir.
Obviously it is different. Or I did something wrong.
I opened the scripter an write:
WriteToRegistry('', 'SQLiteTempDir', '/Volumes/psuExtern/Bilder/tmp');

I pressed run an quit the scripter. I closed psu, start it again and compact the database.
It didn't use the new location. It used furthermore the internal drive.

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 21 Dec 20 8:34

Maybe the SQLite support for temp folders on macOS is different (which I doubt). If I remember well the temp folder setting in SQLite was deprecated and they may have completely stopped supporting it. They deprecated it because SQLite wants to use only the OS's temp folder.

The temp folder is, afaik, not used to build a new catalog during vacuum. Afaik, SQLite uses the same folder as where the database file is stored to build a new catalog when it processes the vacuum command. There's no way for PSU to change that behavior. If you have insufficient space then you could temporarily copy the file, compact it and copy the file back. Or expand drive space

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 21 Dec 20 9:46

Hert wrote:
21 Dec 20 8:34
If you have insufficient space then you could temporarily copy the file, compact it and copy the file back.
Which file you mean?

I have the catalog on an external drive with more than 1tb free space.
inside catalog are these files:
photosupreme.cat.db
photosupreme.cat.db-shm
photosupreme.cat.db-wal
photosupreme.thumbs.db
photosupreme.thumbs.db-shm
photosupreme.thumbs.db-wal
These files are all on the external drive.

I do not understand which file i should temporarily copy now from where to where, because the database is already on the external drive.

P.S. I cannot expand the internal space because you cannot change anything on the Mac hardware.

P.P.S. When I start the compacting, the .db-shm and .db-wal files disappear. The files appear again after finishing compacting.During the files disappeared there is a .db-journal file visible.

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 21 Dec 20 10:56

This is all SQLite at work. .shm, .wal, and .journal files are temporary files created by SQLite during database operations. SQLite manage these files and you should never delete them. These files only exist while the database is open, so after closing PSU these temporary SQLite files will no longer exist, and if they do then again don't delete them as then SQLite will handle them when the database is again opened.

The catalog database files are:
photosupreme.cat.db
photosupreme.thumbs.db

The cat.db contains all the catalog information
The thumbs.db file contains the thumbnail and previews images

NEVER copy the files while PSU is open (hence the .wal and .shm files exists).

If the database is already on the external drive then don't copy it of course. In that case change the TEMP folder for your OS if you think there's not enough space in the temp folder.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 21 Dec 20 12:12

Hert wrote:
21 Dec 20 10:56

If the database is already on the external drive then don't copy it of course. In that case change the TEMP folder for your OS if you think there's not enough space in the temp folder.
Exact this is the question i asked.
So, it is not possible for a mac user to compact a catalog which is on an external drive, if the internal space is not at least the size of the database at the external drive. Right?

Sorry, i‘m only a Photographer not an it-developer.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 24 Dec 20 9:16

So, my question is yes still open, is it that as a mac user I have to be careful that the catalog doesn't get too big because there is no way to compact that catalog externally?

So you have to buy your Macbook with sufficiently large SSD if you have a lot of photos, if you want to use Photo Supreme.
Too bad. Have a nice Christmas anyway.

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 24 Dec 20 10:50

I can't change how SQLite operates. As I've mentioned you can always:
change the TEMP folder for your OS if you think there's not enough space in the temp folder.

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 24 Dec 20 12:21

Yes, I had already understood that.
Only, I am not a software developer. I have no idea how to change such a system process. I also don't know if that's good for macOS to just change the tmp directory.

Maybe a note on the store page would be good too, that as a mac user you need at least an internal 1tb ssd. ;-)

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 24 Dec 20 12:47

emha wrote:
24 Dec 20 12:21
Maybe a note on the store page would be good too, that as a mac user you need at least an internal 1tb ssd. ;-)
The database disk space required depends on the number of images you want to catalog and how you configure the storage (mainly the preview sizes).
Someone who configures PSU to use 640px previews needs a LOOOOOT less diskspace for the database than someone who configures PSU to use 3200px previews. And someone who wants to catalog 1 million photos also needs a lot more than a 1TB drive ;)

FYI; the person with the most images in the catalog that I know of has 1.8 million images cataloged. Of course higher volumes requires one to also think about capacity and infrastructure. I'm fairly sure that this user needs more than 1TB of diskspace

In other words: maybe you could consider using smaller previews if you need a smaller database size.

This what SQLite does when it compacts your database:
https://www.sqlitetutorial.net/sqlite-vacuum/

Hert
Posts: 6778
Joined: 13 Sep 03 7:24

Re: Compacting Catalog not where the catalog is.

Post by Hert » 24 Dec 20 13:07

While reading that link I sent you it gave me an idea for a script:
Try running this script. Set your folder at the top. I've set it to what you mentioned above;

Code: Select all

const
  VacuumFolder = '/Volumes/psuExtern/Bilder/tmp';

var
  ADs: TDBXOMClientDataSet;
  AFileName: WideString;
begin
  // compact catalog
  ADs := PublicCatalog.NewDataSet;
  try
    AFileName := WideIncludeTrailingBackslash(VacuumFolder) + PublicCatalog.SQLDBNamePlain;
    if WideFileExists(AFileName) then
    begin
      if not AskYN('Would you like to overwrite the file ' + AFileName) then
        exit;
      WideDeleteFile(AFileName);
    end;

    ADs.CommandText := 'VACUUM main INTO ''' + AFileName + ''' ';
    ADs.ExecuteSQLNative;
  finally
    PublicCatalog.FreeDataSet(ADs);
  end;

  // compact thumbnails
  ADs := Thumbnailer.NewDataSet;
  try
    AFileName := WideIncludeTrailingBackslash(VacuumFolder) + WideExtractFileName(Thumbnailer.DatabaseFileName);
    if WideFileExists(AFileName) then
    begin
      if not AskYN('Would you like to overwrite the file ' + AFileName) then
        exit;
      WideDeleteFile(AFileName);
    end;

    ADs.CommandText := 'VACUUM main INTO ''' + AFileName + ''' ';
    ADs.ExecuteSQLNative;
  finally
    Thumbnailer.FreeDataSet(ADs);
  end;

  Say('finished');
end;

emha
Posts: 21
Joined: 17 Oct 20 22:27

Re: Compacting Catalog not where the catalog is.

Post by emha » 25 Dec 20 21:27

Hert wrote:
24 Dec 20 13:07
While reading that link I sent you it gave me an idea for a script:
Try running this script. Set your folder at the top. I've set it to what you mentioned above;
A thousand thanks for the effort. It almost works. It actually gets into tmp on the external drive.
At the end comes the message finished. But, the compacted cat.db and thumb.db are not written
from tmp to catalog, but remain in tmp. Can I move them from tmp to catalog myself?
Or is there an addition for the script?

Post Reply