Versioning controlled by labels

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Versioning controlled by labels

Post by David Grundy » 09 Dec 12 13:39

I have been reading the discussions about versioning, and thinking about how this could be made generic.

I think versioning really is inherently complicated, and anybody who has an opinion on how it should be used will have their own preferences on workflow. So a general approach is needed which can cater to all file structures and workflows.

Therefore I think it would make sense for versioning to have
(1) a default of being switched off
(2) a well-documented automatic mode - whatever Hert thinks works - which can be turned on by the user in the preferences. (I can almost guarantee I would not use as it's unlikely to satisfy my requirements.)
(3) a user-controlled "Advanced Versioning" mode.

Actually #3 can be set up in a quite simple way which gives enormous flexibility, using labels to control versioning and to know afterwards what happened Here's my suggestion - I'm sure others can improve on this, then just need to persuade Hert to do it! ;-)
The idea is that you would first find all the potential main versions, and add a specific label to all those; then find all the potential subsidiary versions, and add a specific label to those. Then the version detection would match up Mains with Versions only where they have those predefined labels, and additionally meet the file naming rules.

Then all the complexity is handled by advanced users, and there's no need for arguments about what workflow to use.

Sample dialog fields in the attachment.

Notes:
Users might prefer to make all the [Versioning] labels in a private category so as not to write them out to the files. It might make it easier to undo mistakes.
The labels shown in the dialog are just my idea - of course you could select any labelling scheme.
Line4 Options: [Use alternative placeholder] / [Do not add version]
Grey out Line5 if "Do not add version" is selected in Line4.
Line6 Options: [Combine all labels] / [Main version labels replace other version]
Line7 Options: [Version filename starts with Main filename] / [Main filename starts with Version filename] / [Match first N characters]
Grey out Line8 if not needed

... David
Attachments
VersioningDialog1.jpg
VersioningDialog1.jpg (63.37 KiB) Viewed 8879 times

weidmic
moderator
Posts: 835
Joined: 04 Dec 06 22:21

Re: Versioning controlled by labels

Post by weidmic » 10 Dec 12 8:41

Why not having just a basic versioning in PSU and then asking Hert nicely (very nicely) for a script that can do all the stuff we would like to have?
This way Hert can keep PSU as simple as possible without making PSU more complicated and some of the users with a more complicated file hierarchic structure or different workflow HAPPY :)

Regards,
Michael
PSUServer 5.x, PostgreSQL 10.x
My homepage http://www.michaelweidner.com
PSU Tips and Tricks http://www.michaelweidner.com/WP/psu/

jstartin
Posts: 408
Joined: 23 Aug 06 13:47
Location: UK

Re: Versioning controlled by labels

Post by jstartin » 10 Dec 12 9:58

David Grundy wrote:(3) a user-controlled "Advanced Versioning" mode.
Actually #3 can be set up in a quite simple way which gives enormous flexibility, using labels to control versioning and to know afterwards what happened Here's my suggestion - I'm sure others can improve on this, then just need to persuade Hert to do it! ;-)
The idea is that you would first find all the potential main versions, and add a specific label to all those; then find all the potential subsidiary versions, and add a specific label to those. Then the version detection would match up Mains with Versions only where they have those predefined labels, and additionally meet the file naming rules.
David,
It's a nice idea with some practical problems, I think. One is that, in PSU, labels for versioned files are a property of the whole version set. If you pre-label files as "Main"/"Version" then any added versions would inherit both labels. Needs another mechanism, adding complexity to PSU.
weidmic wrote:Why not having just a basic versioning in PSU and then asking Hert nicely (very nicely) for a script that can do all the stuff we would like to have?
This way Hert can keep PSU as simple as possible without making PSU more complicated and some of the users with a more complicated file hierarchic structure or different workflow HAPPY :)
If it can be done by a script then I support this.
Jim (Photo Supreme: AMD Quad-Core A8-5500 Accelerated Processor 3.2 GHz; internal AMD Radeon™ HD7560D; 4GB DDR3 SDRAM; Win10x64)

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 10 Dec 12 14:56

I don't think I explained myself properly, since responses so far seem to miss my point.

It's all very well saying "Ask Hert to write a script". But you still have to define exactly what the script will do.

I am trying to think about "How should the advanced versioning work, once it is invoked?"

I don't mind whether advanced versioning is implemented via a script or via a menu option or any other way, as long as it works and it lets me work the way I want. But if you implement a script that works for your preferred way, it probably won't work for my preferred way.

So I want Hert to implement a flexible approach so that each user can decide:
(1) Which files to search for master versions
(2) Which files to search for the subsidiary versions
(3) What happens when versions are found.

I don't want someone else to tell me which files can be masters and which can be versions. I don't want to have to adapt to Michael's way or Hert's way. So the labels idea is a way to completely generalise this, and a single version of the dialogue could do everything. You just have to think a bit differently to see how it works.

So ...

You assign the label "Versioning | Main" (or whatever you prefer instead) to one group of files. You can choose any set of files you like, regardless of where they are organised. It might be 10 files on 10 different disk drives. You don't have to select them all at once, you can take your time to find the ones you want to include in the versioning. Heck, you could include every file in the database if you like. It's quick to assign an extra label to all of them.

You assign the label "Edited" (or whatever) to a different set of files. Again, complete flexibility to decide which files. There may be some overlap with the "Versioning | Main" set. No problem.

You have complete flexibility to include or exclude specific files or folders when versioning.

When you are finished you can check that every candidate versioned file found its main version, by searching for files with the label "Edited" (or whatever label you used) and which are not versioned.

After running and checking the versioning, you then unassign all "Versioning" labels. You don't need them assigned to these files any more because their job is done, and you can reuse them next time for a new versioning project.

(I think of labels in the databse as being tools for any purpose I like. For some purposes (most of the labelling I do now) they are permanent attributes of the relevant files. But in the versioning workflow they would be assigned and unassigned as needed. Hence my suggestion to use private labels for this purpose - there is no need to write them to the files.)

... David

weidmic
moderator
Posts: 835
Joined: 04 Dec 06 22:21

Re: Versioning controlled by labels

Post by weidmic » 10 Dec 12 16:11

David,

I did understood your request and there is no "one way fits it all approach"
There are just too many views on how a image should be versioned. Though, I found the way on how V5 handled the versioning was very usable!

...
I have thought about a label approach a long time ago too.
The problem is that this wont work, just because of the way Hert is thinking about the labels within a version set.
In short: "All labels have to be always in every image of a version set" period
I totally (and some others) disagree that it should be that way, but it is how it is...

Cheers,
Michael
PSUServer 5.x, PostgreSQL 10.x
My homepage http://www.michaelweidner.com
PSU Tips and Tricks http://www.michaelweidner.com/WP/psu/

Mike Buckley
Posts: 1194
Joined: 10 Jul 08 14:18

Re: Versioning controlled by labels

Post by Mike Buckley » 10 Dec 12 16:16

David,

As much as I like your creativity, it seems to me that your system involves a workaround (adding versioning catalog labels and later removing them) for which a workaround already exists (adhering to a nested system of file folders at least temporarily while versioning is taking place). The dialog menu that you proposed seems to provide total flexibility though at the compromise of also increasing complexity, when that complexity goes against one of the primary goals of being able to easily use Supreme.

I am extremely interested in the use of versioning in Supreme, so I wonder whether you have been able to find reasonable workarounds that meet your needs. I ask because I believe, at least theoretically, the current implementation of versioning when combined with slight revisions to my workflow would meet all of my needs. I wouldn't know of course until I dive deeply into my workflow using Supreme, which I have not yet done.

Mike Buckley
Posts: 1194
Joined: 10 Jul 08 14:18

Re: Versioning controlled by labels

Post by Mike Buckley » 10 Dec 12 16:18

weidmic wrote:The problem is that this wont work, just because of the way Hert is thinking about the labels within a version set.
In short: "All labels have to be always in every image of a version set" period
I disagree, Michael. The system David describes would indeed work despite that characteristic that you mention. It would work because the versioning catalog labels would be manually removed after the version sets are created.

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 10 Dec 12 17:58

Mike, you're right that this is a workaround. (I am famous in my workplace for finding ways to work around system requirements; so this word doesn't shock me. :wink: )

I think that the system which Hert has implemented may not always work for me, but I am still thinking about it.

I agree that my suggestion is more complicated than would be ideal. However I also think that versioning is inherently complicated unless you are content to have little control over what files get allocated to what placeholders in batch operations. The approach Hert has implemented requires some careful documentation so that users know exactly what will happen. I'm not sure this is really less complicated than a dialog box which tells you exactly what is going to happen. The same degree of care is needed by the new user to avoid problems when learning to use it. (I don't know whether my example actually achieves this objective. It's hard to self-critique on this sort of stuff.)

I'm happy for there to be a default approach but I'd also like a flexible approach to be available. If it's in a script so that average users never see it, that's fine, especially if it means less need to spend time on documentation for new users.

My intention is to propose an approach which allows anyone to implement anything they might want (or at least, anything that I can think of within a broad range of variations). To me, my approach seems simple because I can work with the files wherever they happen to be, and I can take my time selecting the right set of files to work on, and I can see exactly what will happen when it runs.

In Hert's implementation I may have problems if the two sets of images for versioning are on different drives (which in my case they often are), since moving them may be seeveral hours' worth of file copying from drive to drive in some cases. Instead of waiting I could deliberately reassign part of the external directory tree to an empty subfolder on the local drive ... which somehow worries me. Also, not sure I 100% understand and haven't tried it yet, but I think Hert's system won't cope well with the situation where the files for versioning are spread across many subfolders - for example, I have a folder structure for original files which often goes down to specific days because my shooting is typically either 0 or several hundred shots per day, with not much in between. I have a separate set of folders, organised differently, for any derivatives. Figuring out how to safely move all this stuff around - and then move it back again later - feels complicated to me. It feels much more natural to leverage things that PSu already does well - in particular, searching and labelling - and leave the file structure in place.

I tend to think in terms of "what could go wrong" when designing workflow, since there will sometimes be accidents, and sometimes I will be called away in the middle of something. So personally I would rather assign my well-defined labels temporarily than move files around temporarily. It seems easier to recover from forgetting where I am up to.

(Incidentally I think it's perverse for a DAM - which is supposed to free me from the constraints of folder structures - to impose a folder structure requirement for a particular operation such as versioning. That doesn't necessarily mean it's a bad idea, though.)

I think there is more than one way for things to be complicated. In my implementation, I think the complication is mainly in learning to think of labels as temporary tags when that's useful.

... David

tstoddard
Posts: 578
Joined: 07 Sep 12 12:51

Re: Versioning controlled by labels

Post by tstoddard » 10 Dec 12 19:37

David,

What you are proposing is the design of a version detection wizard that can be run on files that are already in the existing database. This might be worth pursuing but for those of us who would prefer to control the creation of version sets at the time files are imported into the database it would be more complicated than necessary. Your "version detection wizard" might only be needed one time and then future versioning could be done as new files are imported then you would never need the version detection wizard again. For this reason, I believe that a script would be a better solution. That script could be shared and easily modified to accommodate other workflows. Also, I think that having to do so much labeling could become tedious, error prone and inefficient if it was required every time versioning was to take place. I would prefer to have Hert focus on improving the way that automatic versioning works.

The issue of how versioning is done when a folder verification takes place is somewhat different than doing it for files that are already in the database but some of your suggestions have given me ideas that I think would improve that process. As I see it, the issues everyone is struggling with are:
  • How to control what set of files to search in
    Which files should become the Main version
    What placeholders should be used for the version that is not the Main version
Your idea of using labels could be part of the answer but would not be required if a user implements a hierarchical folder structure to store derivative files. All that PSU would have to do is to prompt the user for where to look for the matching files before it imports new files. It could give users the option to look in:
  • the current folder and its subfolders
    the entire database
    a set of files that have a certain label
This way, if I want to use the label based system that you suggest I can, but if I already organize my files in a hierarchical folder structure, I don't need to. Also, I would only need to label the files that I want to search and no others.

Determining which files are made the Main version and what placeholders to use for the others gets a little more complicated but I think there should be some settings, accessible through preferences and/or import profiles, that tell PSU how to make that decision. My preference would be to decide this based on which files are already in the catalog and which aren't. In my thought process, if I create a derivative of a file that is already in the catalog, I would want the original file to be the Main version and the derivative to use one of the other placeholders. I realize others prefer the opposite but this solution would accommodate that as well. By making these decisions at the time files are imported, users and PSU wouldn't need to do all of the labeling that your system requires.

Users could check results by looking at the files in the Import Session that is created automatically by PSU. (I'm assuming that PSU creates sessions for files imported during a folder verification. I think it does but I'm not at home to test that. Also, I'm not sure if version sets will appear in the Import Session collections but I believe that they do.)
Tom Stoddard

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 11 Dec 12 17:02

I would also agree that when versioning on import logically it will usually make sense to me for the image already in the database to be the main image. And usually, versioning could be done on import. Although I don't really want to alter my file management to get versions in folders below the main version, this could perhaps be done systematically. I haven't yet thought through all the cases.

One example from my own experience where care is needed to get the versioning right:
- I buy a new camera (OM-D - Yay!!)
- Then I find that no raw converter that I would consider using actually supports it yet (not so Yay!)
- I shoot Raw+jpg for a while, and any derivatives in the meantime come from the jpgs (Yay-ish). For the time being the camera jpg is the main version.
- Later I can convert the .ORFs to .DNGs, and I want these to be the main versions. And I can delete the ORFs at this point.

Am I going to have to remove the existing files from the database, and add them back again, in order to get the DNGs as the main version? That is doable but it seems peculiar to me ... like breaking something and having to fix it again afterwards. I want flexibility to cover this and other unexpected situations without moving lots of folders around.

Tom, I think you are saying there should be a "simple" way and a flexible way. I also said that earlier, but now I"m not so sure. See how we all struggle with understanding the best way to do this, and there were debates in IDI5 days and earlier about the same topic. People definitely don't agree on how they want versions to work, and different ways suit different people.

Frankly i suspect the average "basic" level user when trying to use versions is going to end up with inconsistencies as to which version is main and which is some other placeholder. I think versioning is very powerful but needs to be seen as inherently an advanced function. I don't think it's usually going to work well for someone who isn't prepared to think a bit about their workflow, and I do think that someone ready to do this level of thinking will appreciate more flexibility. Overall, I think if it's worth doing versioning, then it's worth providing flexibility to do it comprehensively.

... David.

tstoddard
Posts: 578
Joined: 07 Sep 12 12:51

Re: Versioning controlled by labels

Post by tstoddard » 11 Dec 12 19:53

David Grundy wrote:Frankly i suspect the average "basic" level user when trying to use versions is going to end up with inconsistencies as to which version is main and which is some other placeholder. I think versioning is very powerful but needs to be seen as inherently an advanced function. I don't think it's usually going to work well for someone who isn't prepared to think a bit about their workflow, and I do think that someone ready to do this level of thinking will appreciate more flexibility. Overall, I think if it's worth doing versioning, then it's worth providing flexibility to do it comprehensively.
David, I couldn't agree more. I've spent the last few months trying to familiarize myself with the various concepts and methods for cataloging a collection of media files and during that time I've spent much of it puzzling over versioning and how best to accomplish it. Given the approach that PSU uses, I don't believe it's possible to efficiently create an interface that would allow you to do all of the things you'd like to be able to do. I would like, for the purpose of discussion, to suggest a fundamental change in the approach that is currently in use.

Since there is little or no documentation for the internal implementation of version sets in PSU, I will have to make some assumptions. These may or may not be correct. I am assuming that "Main" is basically just a placeholder. I say this because it appears to be mutually exclusive to other placeholders. In other words, a "Main" version can't have a "Album" placeholder. PSU may treat "Main" versions differently than versions with other placeholders for the sake of synchronization but it doesn't appear to me to be the case, except, perhaps on the initial synchronization that occurs when the set is created.

It seems to me that many of the complications involving version sets come down to the ability to select or specify which file is to be the "Main" version. Again, my assumptions could be incorrect but if "Main" was implemented as a flag (a simple bit field) in a version set that is separate from the placeholder field then much of the complexity could be eliminated and it would be a lot easier to allow users to make the changes they want to make. Database integrity rules could easily guarantee that only one file per version set is set as "Main". Any file that is in a version set could have a placeholder and could also be considered the "Main" version. In fact, I'm not really sure what being "Main" status even means other than it's the image that appears on top of the stack of the version set in the thumbnail view. Perhaps someone can enlighten me as to what other aspects of the version set the "Main" version controls in PSU. Hert stated in a different topic that the only metadata that is synchronized between version set members are labels. If that is so then there isn't much need for a "Main" version at all. Perhaps it should really be called the "Display" to "Top" version.

In any case, if the "Main" designation was represented by a flag in the version's record then changing which version is the "Main" version should be a fairly trivial thing. Changing it wouldn't necessitate changing placeholders so all that the program would have to do is prompt the user for a preference. It could be done in a batch command called "Version Set Main". The user could be given the option to specify criteria for the selection preference such as; by file type or by file date or by datetime file was imported into the catalog or by placeholder. If new version sets are created when images are imported, the decision would only need to be made between the new file, and the existing file. Since subsequent changes could be done easily in batches after the import, that decision wouldn't be so critical.

I hope I'm making sense. My point is that if we separate the concept of "placeholder" from the concept of "Main" things get much simpler. This is because changing or assigning placeholders or changing which version is the "Main" version is done independently of the other.

Of course, initial version detection is yet another independent function that could be performed as a batch operation. I can imagine batch commands that could be done in sequence to Detect Potential Versions, Create Version Sets, Set "Main" Version, and Assign Version Placeholders.

I could go on but I'm running out of time. I hope someone finds my ramblings of interest.
Tom Stoddard

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 12 Dec 12 18:14

I found it quite odd at first to see in the view for Catalog "by version placeholder" that the options were
- Version sets
- Album Display
- Camera jpg
- etc

I really had to stop and question what it could mean and how it might relate to how I think about versions. "Version sets" seemed - and still seems - not to belong in this list as it's not a placeholder. I find it confusing. I think "Main Version" and "Version set" are two quite different concepts, but I conclude that in PSu the terms are not properly distinguished. I think this is a symptom of not having a completely coherent model of how versioning is supposed to work.

For the moment I am holding off on doing any versioning work in PSu because it doesn't feel mature yet. But ... if nobody tries to use the tools there won't be much feedback and maturity will not arrive ...

... David

PS I think the existing implementation makes it quite difficult to automatically display exactly the versions you prefer to see, since Album Display doesn't seem to do it. I do have a workaround which I am pretty sure will work for the general case (using labels and dynamic search), but I have delayed implementing it while I wait to see whether PSu versions evolve into something I think I can use. I will probably try it out this weekend anyway.

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 15 Dec 12 17:56

Mike, you said
I am extremely interested in the use of versioning in Supreme, so I wonder whether you have been able to find reasonable workarounds that meet your needs. I ask because I believe, at least theoretically, the current implementation of versioning when combined with slight revisions to my workflow would meet all of my needs. I wouldn't know of course until I dive deeply into my workflow using Supreme, which I have not yet done.
Unfortunatley I just don't see yet how it's going to work for me. I have many images - many thousands - which have edited versions in parallel folder structure to the originals with which I want to associate them. I didn't finish this mammoth task in IDI, and I can't see how I'm going to be able to do it in PSu. So as far as I can see there is now no DAM tool available which will allow me to accomplish this task, so I can stop worrying about it and just assume it's not going to happen.

Some impediments to doing this in PSu, that I can see:
  • I have large parallel folder structures for original and derivative files before about 2008. It's rather infeasible to get this to work via a subfolder structure as there will be thousands of folder moves to do.
    Originals and derivatives for 2008-2012 are in different trees but they are not even parallel.
    Derivatives have names which usually (but not always) include the original file but are not the same.
    (I could rename to a standard scheme, but I don't see any way to make a standard scheme work separately in the originals folders and the derivatives folders, given that there is not a 1:1 mapping from originals to derivatives. Sometimes one original has several derivtives; some originals have no derivatives. Timestamp-based filenames won't always work because of high-speed bursts which have the same time, to the limit of accuracy of the original timestamp in the file.)
    I can't see any way in PSu to control what version placeholder is assigned to added versions. There's no way I'm going to do that by hand for all the versions of my images.
Basically the reason I want flexibility is to accommodate fixing the mess coming from past iincomplete efforts to get things organised. I also want to know that I will be able to change my mind later about how things are organised, and accomplish the change using batch-type operations rather than one file at a time. I also want to know that I will be able to recover from serious mistakes in versioning, even if they are discovered after some time has passed so that "restore from backup" would mean losing a lot of subsequent work. I just don't see anywhere the intention to provide facilities such as swapping versions on large numbers of version sets; reassigning versions from one placeholder to another; viewing different versions by default at different times and for different purposes (while defaulting to main if the desired version doesn't exist); and a few other things in the back of my mind. Basically this feels to me like a toy implementation, not treating versioning as the powerful tool it so easily could be. I can find workarounds for a some of these things, but it feels like I'm fighting the intention of the software rather than working with it. That means that subsequent program upgrades might break elements of any workaround that I implement.

I don't seee a reasonable workaround for the parallel file structures though.

I am still thinking about how what I need might be accomplished, but I am pessimistinc, given the overriding priority given to making PSu look simple.

lippe
Posts: 296
Joined: 12 Aug 06 12:26
Location: Wondelgem (Belgium)

Re: Versioning controlled by labels

Post by lippe » 16 Dec 12 11:46

David Grundy wrote: I don't seee a reasonable workaround for the parallel file structures though.
David, you could use a Portfolio to store the Originals, Main versions, Derivates in the same collection. In PSu, the Portfolio is "intended to allow you to build a structure with images that belong together", but they are folder independent. So you cannot build a collection automatically by defining a folder structure and have to add the images manually to the collection.

You can also import all folders with Originals, Main versions, Derivates and store the paths to a Dynamic Search and store this to you favorites. But this method is also cumbersome as it cannot donne automatically.

The truth is that there is no support in PSu for using different folders to store the Originals, Main versions and Derivates separately as proposed by thedambook.com and dpbestflow.org. While that folder based workflow allows the use of simple backup and archive methodes outside the DAM software, it is currently a not supported workflow by that DAM software.

Lippe,
Photo Supreme V3, LR6, FPV, PSE14 - vaio i5 @ 2.5GHz + 8GB , 850 EVO 500GB - WD 1TB - Windows 10 Pro 64 bits- DS216play - EOS 600D

David Grundy
Posts: 241
Joined: 13 May 07 16:40
Location: Hong Kong

Re: Versioning controlled by labels

Post by David Grundy » 16 Dec 12 14:06

Lippe, I guess if I am associating files manually I can still do it one file at a time by dragging files onto other files in the collection viewer to create version sets. But really, the manual approach doesn't appeal to me! My style is always to automate ... because I know I am too lazy to keep a manual system going, whereas the challenge of automating is interesting in itself.

... David

Post Reply