question about database size

[fbungarz]

Hi all,
using PSU after my migration from IDI for about two weeks now, I am starting to get worried a little bit about size. I found a pretty good workaround to avoid copying a gigantic thumbnail database between laptop and desktop (http://forum.idimager.com/viewtopic.php?f=57&t=24084).
Now, though, after intensively using PSU I am surprised by the sharp increase in file size of the main, catalog database too.
For comparison:
My old IDimager catalog database is 3.39 GB.
After converting to PSU and compacting the new PSU database was about 4 GB.
After two weeks using PSU regularly every day, the database (again after compacting) grew to 6.2 GB. That means in only two weeks of (heavy) use the database size increased about a third!
For reference, the catalog contains 103.893 images. I have not been adding new images, just used the database for cataloging. Many of my labels are mapped to custom XMP, most of my images are versioned, so perhaps that explains some of the drastic increase in file size. Also, since tagging images in PSU is so much faster, I enjoyed finally to work on a backlog of quite a few images that couldn't be bothered about using IDI. I am sure, the intensity of using PSU will eventually slow down again...
Still, are 6GB = 100.000+ images about to be expected? I'd be much interested to get some feedback about their database sizes from other users here on the forum.
Currently I manage ALL my images in one database only. I find this the most convenient solution. But I am getting worried about these gigantic files...
Cheers,
Frank

[sanphotgn]

Frank,

To provide a comparison ... I created a new catalog in PSU in the fall of last year. It has just under 12,500 images. I have 7 custom XMP fields. I probably have 50 labels. Maybe more. All are directly under a Category. None are mapped. No versions. Latest compacted database: 740,000 KB.

So, let's multiply by 10: 125,000 images in a 7,400,000 KB database.

Thus, 125,000 images in a 7.4 GB database.

You have 100,000+ images in a 6 GB database.

Hopefully, I did the simple math correctly. I realize the 1:1 in reality is anything but reality, but it gives you an idea.

Kevin

[Stephen]

I too have everything in one database:
100,000 images
idimager.cat.db = 1.9 GB
idimager.thumbs.db = 17.5 GB
remaining files in the PSu backup directory = minimal

I don't have the custom xmp stuff, but I hope this helps anyway.

[Mike Buckley]

I doubt that you're going to get meaningful information by comparing the size of databases relative to the number of photos. As an example, I have 30,000 images and my catalog database is only 700 MB. That's consistent with Stephen's report but not at all with Frank's or Kevin's reports. 99% of my images are versioned but I have no custom XMP.

[fbungarz]

I doubt that you're going to get meaningful information by comparing the size of databases relative to the number of photos.

Probably not, but it seems that the amount of labels present in a catalog and the metadata written to the image files have a GIGANTIC impact. My catalog database is getting huge (8 GB now!). And I am still not nearly done with cataloging.
I have adopted a routine of regularly compacting the database, but gain very little from that. I begin to wonder how sustainable that is.
I used to be quite worried about the size of the thumbnail database (see http://forum.idimager.com/viewtopic.php?f=57&t=24084), but now I thing the catalog database might be more of a problem...

Frank

[Preston B]

Here's what I am showing...

5249 Images
Catdb = 164 MB
Thumbsdb = 5.9 GB

Image Preview Size = 1680 px
Image Preview Quality = 95 %

Note: I have a large number of very large high resolution TIFF and PSD files (many well over 1.0 GB). I do not version, nor do I use any custom XMP.

Just making a SWAG here, but it would appear that original image size, as well as 'Preview Size' and 'Preview Quality' would affect the size of the Thumbsdb. Did a guess correctly?

--P

[fbungarz]

Hi Preston,
the size of thumbnail database indeed appears directly related to the number of images and preview quality and preview size. It does not surprise that this file is substantially larger in PSu, which supports higher quality previews with larger previews than in IDI.
Since it is possible to use the same catalog database with different thumbnail databases (see http://forum.idimager.com/viewtopic.php?f=57&t=24084), size of the thumbnail database is no longer not such a concern for me...

However, with my catalog database still growing quite substantially, I am starting to get worried. I already noticed some performance decrease, which may be related to database size. For example, I am using "right click on a label - show in a new tab" a lot. I love that feature to quickly review if I did not forget assigning some metadata or labels to a particular image. Initially the new tab would be displayed virtually instantaneously, now, with my catalog database almost twice its original size (8 GB!), it takes a few seconds to load...

Well, I guess the performance decrease (if indeed related to the database size) is the prize I pay for managing more metadata, versions, labels - the prize essentially for a more complex database !?

Form what others have posted here, it appears that especially using a lot of XMP metadata significantly blows up the catalog database size. Again: apparently the prize I have to pay...

Now, what does worries me more than performance issues: I can only assume that a larger catalog is also more prone to corruption. Perhaps I should at least turn on "synchronous mode" (see http://www.senoiaphoto.com/psu) to be safe. But turning that option on likely results again in an even more drastic performance decrease...

It would be helpful to know if the server version of PSU works more efficient and is smaller than the SQLite version? Does the server version run under WAMP? That might be an option to avoid having to install a local server. Also: is there a size limit for SQLite? Am I likely to "hit the roof" here? Presumably the server version is designed to handle larger databases more efficiently?

Cheers,
Frank