resilio sync anyone?

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

resilio sync anyone?

Post by fbungarz » 14 Dec 18 1:22

I recently bought a new laptop with both an SSD and HDD.
The idea is to establish a workflow that allows two people to both use PSu. At work I will be mostly databasing images that are work-related. At home my wife will use the laptop to database our private images.

Currently ALL our images are managed in one and the same PSu desktop database.

From that database have been thinking to "split off" a separate PSu database just for the private files, store them on the laptop HDD and periodically synchronize the laptop HDD with the desktop HDD. Until now I have used SyncBack to back-up files and synchronize folders between my desktop and my old notebook.

Here is the challenge:
Though my wife does not need access to my work images, I would like to be able to keep having access to the private ones from my desktop PSu database.

I have been looking at Resilio Sync, a program developed by Bit Torrent, that allows computers be synchronized peer-to-peer via the internet. The program uses block-chain synchronization, which means that not the entire files, but only parts of the files that have changed are being synchronized.

I am thinking that this likely will still not work to sync PSu's database and use that same database on the two different computers (sync conflicts very likely when we are accessing the database more or less the same time without checking if sync completed). However, it might be an option to keep the private image files in-sync.
But then, even if those files are kept in sync: how do I get the metadata changes applied with the database at home back into the desktop PSu database?

Is there a script that, upon start-up, can automatically verify specific folders and load any changes into the database?

Anyone using Resilio Sync, free or pro?

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 14 Dec 18 15:20

I use Unison for very similar purposes, as well as backing up copies of data with the mirroring turned off so that changes at the backup can't alter the "master" copy. I like it as it will work across OS's, local networks, across the Internet and can also with a bit of thought handle copies across multiple machines in a round robin fashion if needed.

[edit Hert; link removed. no links to 3rd party software allowed]
Geoff Mather (G8DHE)

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: resilio sync anyone?

Post by fbungarz » 14 Dec 18 16:26

Thanks for sharing the link.
I use Unison for very similar purposes
Would you mind explaining your workflow in a bit more detail?

As I tried to explain: I am not interested here in a backup or mirroring a slave to a master drive. I am interested in bi-directional syncing.
I think that will work reasonably well with these tools for the image files, because at the different workstations we will be working mostly on different files (desktop/work images vs. laptop/private pictures) . So, provided the syncing is reasonably fast and reliable, it should keep these files the same. When I use PSu on one computer to change metadata in these files, the changes should be propagated to the other computer, and vice versa...

However, my main challenge: While the image file metadata changes will thus be synced, how then do I get the metadata changes into the other PSu database on the other computer.

I would happily change to the PSu server version, AFAIK but that does not work across the internet, does it?

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 14 Dec 18 21:11

Unison in its simplest mode will do just what you want;
You start by setting up a config file, which points to the highest (identical) level in each file system you want to sync - the roots,
you may want to include file types to include or exclude from the sync process,

Code: Select all

# Unison preferences
batch = true
contactquietly = true
fastcheck = true
ignore = Name temp.*
ignore = Name TEMP.*
ignore = Name *.temp
ignore = Name *.TEMP
ignore = Name *.tmp
ignore = Name *.TMP
ignore = Name pspbrwse.jbf*
ignore = Name Thumbs.db*
ignore = Name RECYCLER
ignore = Name System Volume Information
log = true
logfile = w:\WEBSHARE\LogFiles\unison\geoffpassfile1.log
maxthreads = 4
root = \\Shack\C\Program Files (x86)\Raccoonworks\PassFile
root = B:\geoffpassfile
sortbysize = true
times = true
ui = text
xferbycopying = true
You then run on one machine Unison as service, which waits to be called from another machine,
then from the other machine you run Unison this then compares the two sets of folders, and changes that have been made to individual files will be transferred to the other machine. If both files have been changed then it can cope with this either by using a specified program to merge the changes into both ends or flags it to the User to handle.
In my case for the above I have it setup in the Scheduler to run every 5 minutes.
In another case which syncs some web pages and files that are being frequently changed then its setup in a batch file to run all the time, that way I keep a backup on another machine in case the web server machine fails for any reason, in which case a seperate system watches for a server failure and alters the routing to pass traffic to the fallback machine, once the the server comes back then it switches back to the original machine, and all the files will have been maintained correctly no matter which machine they were altered on.

[edit Hert, link removed: no links to 3rd party software allowed]
Geoff Mather (G8DHE)

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: resilio sync anyone?

Post by fbungarz » 14 Dec 18 23:34

If both files have been changed then it can cope with this [...] by using a specified program to merge the changes ...
So, with this setup, you are suggesting one could even work on two separate files of the same PSu database on two different computers, changing both PSu database files and Unison would even then merge these changes into both PSu database files on both computers ???

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 15 Dec 18 11:14

Not unless Hertz has provided you with a program to merge the two altered DB files :-)
It will keep the individual image files updated at both ends in step, but there is no obvious way to merge the actual database files together if both have been changed. My own approach to this problem would be to run a Verify at each machine to read the image files and compare with the local copy of the database, that way any changes made will be incorporated back into the database. I guess this assumes you write all the metadata to the image files as you go.
The above is one reason I like the new command line feature to run a script, so that first it Sync's any out-of-sync images, Backup the database and then Compact the database and exit, I can schedule that nightly. Then before I work on an area I Verify that part of the structure to read back any changes from the image files themselves.
Geoff Mather (G8DHE)

snowman1
Posts: 305
Joined: 01 Jan 07 3:13
Location: UK

Re: resilio sync anyone?

Post by snowman1 » 15 Dec 18 18:53

Frank, just some thoughts:
- "Replaying" the image file data back into the "other" database should work, but it does feel a bit clunky and potentially prone to error.
- If you are not both working on the private images at the same time, could you not just transfer the whole database - i.e. catalog files? As well as images, if indeed you need the images themselves and not just the thumbs, on your work machine?
- Or if this is too much of an overhead then a blockwise transfer - i.e. changed blocks only - could work, though I'd test it thoroughly first as it may cause a massive fail when used on database files (NB For the avoidance of confusion, "blockchain" as per the OP is a different concept, as used in bitcoin etc)
- The usual way to do this in a professional environment would be to use the database's native mechanism for synching a remote database: this generally works by taking the database's transaction log - which is generally produced by the database anyway as part of the mechanism for ensuring transactional integrity - shipping those log files to the remote database location and replaying the transactions into that database. However I do not know if the database used by PSU personal edition has this facility - complexities like this are often dispensed with in domestic databases (as they are not always needed in a single user/single machine environment, add complexity and consume resources)
Good luck with this and please do report back on how it goes!
Snowman1
http://www.flickr.com/photos/snowman-1/
--------------------------------------

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: resilio sync anyone?

Post by fbungarz » 17 Dec 18 18:17

Hmmmm...
This all does seem a bit more complex than anticipated.

Unison seems like a great freeware, but I know myself and I am not too great at hard-coding these things via a command-line interface. So, I though Resilio still might be a better option for me. However, when Geoff mentioned that one could use Unison to merge files, I got all excited again...
Turns out, however, that's not quite a straight forward and currently virtually impossible for the database itself.

I also now think that a direct sync over the internet via peer-to-peer is not likely to work well. I am not a big fan of letting my desktop run all the time (overnight, during the holidays). Using Unison or Resilio seems to only work well, if computers are connected most of the time. So, I did a bit more research and I think syncing via the cload may be a better option. Seems like they have some very decent deals at pCloud.

So, probably, keeping separate databases, use the cloud to sync the personal image files and use verify (perhaps even running it as a script as Geoff suggested) is the way to go.
The above is one reason I like the new command line feature to run a script, so that first it Sync's any out-of-sync images, Backup the database and then Compact the database and exit, I can schedule that nightly. Then before I work on an area I Verify that part of the structure to read back any changes from the image files themselves.
Can you point me to that script? Does it work on PSu standalone?
Do you schedule this via Windows to run nightly and the turn your PC off once finished?

Snowman - thanks for your suggestions, but, like I said, it seems getting everything, images and databases, in-sync seems too much of a challenge and, like you say, "prone to error". I thought, if there was an easy & safe way, I might try it out. But it all seems a bit too risky...

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 17 Dec 18 20:22

I'm in the process at the moment of putting all the script together, I have the Sync all out of Sync images based on the existing available script, the backup is still being done by copying the files on an overnight schedule, but the Psu API has;
function BackupDataToFile (AFileName: WideString; AWithVerify: Boolean = True): Boolean;
Which appears to be the required call
and have yet to work on the Compact database I'm hoping the API;
procedure CompactAndRepairDatabase;
mentioned in the API documentation is the relevant section https://repository.idimager.com/documen ... atalog.htm
Hopefully over the Xmas/New Year break I will have time to finish it!

I must admit both my Workstation and of course the Server do run 24/7, only the Laptop when out of the house doesn't run 24/7 as I run Unison scripts on all those machines at all sort of times for different functions. For many years that included a complete copy of all my images on the server to my son's machine(s) and his backups to me that ran every night, but the size of the drive and volumes involved was beginning to get out of hand with over 2Tbytes additional volume space required on each of our machines - however it did work as I did need on one occasion to restore the complete 1Tbyte of images, as it was at that time, which we did by physically moving the drive from London to home here and copying it back as it was considerably quicker than doing it over the net - one of the problems which Cloud services haven't overcome either!

Oh yes Unison does also have a GUI interface, which is simple to use, ideal for testing and then the command line is very simple to implement in the scheduler;
Image
Geoff Mather (G8DHE)

Hert
Posts: 6685
Joined: 13 Sep 03 7:24

Re: resilio sync anyone?

Post by Hert » 18 Dec 18 13:31

This script illustrates the compacting;
https://repository.idimager.com/openres ... 8F0795224C
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 18 Dec 18 17:38

Ah brilliant, it covers both thank you!
Geoff Mather (G8DHE)

fbungarz
Posts: 1628
Joined: 08 Dec 06 5:03
Location: Arizona, USA

Re: resilio sync anyone?

Post by fbungarz » 18 Dec 18 23:46

Just a quick comment. I would love to get my hands on that scrip, once you have it ready.
However: I am not all that fond of PSu's built-in backup. It takes ages, longer than simply copying the database file.
Perhaps, the backup component could be an optional part of that script...?

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 19 Dec 18 11:24

I have to agree about the backup time! Is there a reason Hert why the internal backup takes longer ? Is anything else being done that isn't obvious but might be useful to keep, any checking or does it Compact at the same time or something ?
Geoff Mather (G8DHE)

Hert
Posts: 6685
Joined: 13 Sep 03 7:24

Re: resilio sync anyone?

Post by Hert » 19 Dec 18 12:14

When doing a backup on the Single User edition then the database files are first checked for integrity and then copied.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

User avatar
G8DHE
Posts: 297
Joined: 21 Aug 17 13:58

Re: resilio sync anyone?

Post by G8DHE » 19 Dec 18 15:59

Makes sense, is that covered by the call;
function BackupDataToFile (AFileName: WideString; AWithVerify: Boolean = True): Boolean;
or is the checking another call ?
Incidentally is there a more detailed description of what the calls above do anywhere, at the moment I'm using a suck and see approach which doesn't always give a lot of info back :-)
Geoff Mather (G8DHE)

Post Reply