PSUv3 is slow when the database is stored on a NAS

Post Reply
aaturner
Posts: 10
Joined: 02 Oct 12 13:04

PSUv3 is slow when the database is stored on a NAS

Post by aaturner » 31 Oct 14 11:43

This subject may have been covered in another thread, but I have not spotted it.

I am test driving PS v3 as a trial version. I have attempted to import 20,000 images from LR5 to a new catalogue. After about 18 hours the process was completed but every image was unsynchronized and very few thumbnails were visible. Not much success there.

I read some of the forum threads, one of which suggested that it would be quicker to import directly from the original file folders. I set up a new catalogue and started to ingest the 20,000 images. After 18 hours the thumbnail service has completed its task, but the synchronize service is still running and only 33% complete after 24 hours; at this rate it will be three days before the process is complete! It all seems very slow for a product that is described as very fast. Is this normal or have I screwed things up somewhere?

FYI my images are on a dedicated internal hard disk (Industrial strength 7200 rpm Western Digital Black). The catalogue is located on an external NAS via a 1Gb ethernet connection. The OS is win7 pro running a latest version i7 quad core processor with 16Gb ram.

Any suggestions as to how I might improve import performance will be gratefully received.

Many thanks.

[update Hert; changed the title to better cover the subject; original subject was "PSv3 Ingesting and/or importing from LR 5 is very slow"]

Hert
Posts: 20323
Joined: 13 Sep 03 7:24

Re: PSv3 Ingesting and/or importing from LR 5 is very slow

Post by Hert » 31 Oct 14 12:27

The catalogue is located on an external NAS via a 1Gb ethernet connection.
And there's your culprit.

Storing a SQLite database on a network is never a good idea. This applies to Lightroom (which also uses SQLite) and also to Photo Supreme Standard (SQLite). A database is a high translation system and SQLite will create journal files on the NAS all the time. Even with a 1Gbit network that will have a drastic impact on performance.
On top of that, SQLite was never designed for network usage. If Photo Supreme detects that you're running the database on a non-fixed drive than it will switch to "high safety" which is also slower than the safety mode in which it runs when the database is stored on a local drive. And last but not least; the high safety setting will prevent most of the problems that can occur when using a SQLite database on a network drive but when you get network disturbance anytime during a journal file transaction then your database *will* be corrupted. It's not a matter of "will it corrupt", but "when is it corrupt". It can run smooth for months but sooner or later your database will corrupt.

Best to keep the database on your local machine. What you could consider is to replicate the local database to the network so that you can use that to query the database on other clients.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

stevehughes
Posts: 86
Joined: 19 Jan 13 3:46
Location: Brisbane, Australia

Re: PSUv3 is slow when the database is stored on a NAS

Post by stevehughes » 02 Nov 14 6:36

Just a thought. If you want to put the DB on the NAS then you might want to consider the PostgreSQL Server version of PSU. Depending on the NAS, you may find that you can install PostgreSQL on it (e.g. my QNAP supports the installation of all sorts of cool extras). That would give you multi-user, multi-machine access to the DB.

I run the PostgreSQL version of PSU but I admit I have not tried it against a PostgreSQL installation on my NAS.

Steve

Hert
Posts: 20323
Joined: 13 Sep 03 7:24

Re: PSUv3 is slow when the database is stored on a NAS

Post by Hert » 02 Nov 14 9:06

It indeed appears to be possible to install PostgreSQL on a Qnap NAS.
http://wiki.makeitfit.ch/QPKG-Packages_ ... .28QPKG.29

Also some Synology NAS appear to have an installation opportunity for PostgreSQL

Keep in mind that PSU requires v9 of PostgreSQL
Hert
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

oversampling
Posts: 27
Joined: 13 Aug 15 17:39

Re: PSUv3 is slow when the database is stored on a NAS

Post by oversampling » 09 Nov 15 9:10

I can confirm that the trial version of PS (PostgreSQL edition) can connect to the PostgreSQL DB (v.9.3.4.1, available as an esay-to-install "qpkg" format) on my Qnap NAS (TS870 firmware version 4.2) after adjusting a bit the ".conf" PostgreSQL files according to the PS documentation.
PS created the DB on the NAS and connected to it but the trial just expired so I could not test any longer.
However keeping both the images and the related DB on the NAS looks an interesting use case to me...

Roberto

peterpix
Posts: 10
Joined: 01 Jun 16 15:36

Re: PSUv3 is slow when the database is stored on a NAS

Post by peterpix » 11 Apr 17 17:24

I've got my image files on a NAS and the database on the computer. The synchronise service is still slow though. I guess it is all that writing to the NAS. The trouble is that I have so many images they won't fit on a standard USB drive. And the NAS has double-redundancy, which helps me sleep at night (it is backed up too!).

I would be interested to hear how others handle image file storage and backup.

Peter

Stephen
Posts: 533
Joined: 01 Oct 14 10:15

Re: PSUv3 is slow when the database is stored on a NAS

Post by Stephen » 12 Apr 17 14:18

I don't use NAS, but I do keep all images on external drives. The database is on the internal computer drive.

You don't state the size of your library but larger USB drives are available now. You could even use a swappable housing / hub like the one made by ICY.

I currently connect the external drive via Firewire 800 (on a Mac) and then have 4 additional Firewire 800 drives daisy chained. The 2 main libraries are on different drives and are synced as often as I like to additional drives as backups. Next week the connection will be changed to a Thunderbolt hub and from there to the Firewire 800 drives. The main library drives are exchanged ever 2 years.

There are probably similar solutions available for Windows users.

I hope this helps?
Never say never change, but using Mac since 2005. Photo Supreme 3.3.0.2604. I stand behind the interoperability of files between applications and systems.

Mke
Posts: 265
Joined: 15 Jun 14 15:39

Re: PSUv3 is slow when the database is stored on a NAS

Post by Mke » 13 Apr 17 19:33

peterpix wrote:I would be interested to hear how others handle image file storage and backup.
I have a RAID1 array ('fake RAID') on my desktop machine with a pair of 3TB WD Red hard drives dedicated to image storage, with the catalogs on an SSD, and a second smaller RAID array for non-photo data. Of course you need a motherboard that supports multiple RAID arrays (or separate hardware RAID controllers, which I looked into but discounted) for such a setup - one from a well established motherboard supplier so that in case of board failure you can substitute a compatible board to access your RAID.

Backup is to a series of LaCie Rugged drives.

Post Reply