Issue with XMP file creation

Post Reply
Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Issue with XMP file creation

Post by Al_M » 20 Oct 19 2:36

This week I imported 108 .CR2 files from my 7D MkII.
I used an import profile to rename them and place them in my file structure.
After assigning geotags and some labels I Sync'ed metadata (writing to xmp). But I noticed 5 of the xmp had a naming convention I had not seen before.
EOS7D2-19-2299_Mac-Pro_Oct-19-200604-2019_CaseConflict.xmp
where typically it would be
EOS7D2-19-2299.xmp
Every time I go back and modify metadata and re-sync, a new XMP with updated timestamp (in the filename) is created, so there are multiple XMPs for a single raw file. Normally, the XMP is updated/overwritten.

After a bit of investigation I remembered that these 5 particular files I had rated in Canon's Digital Photo Professional 4 ( I sometimes cull in DPP prior to importing to PS). So, I can understand some mixup in the XMP metadata upon initial import to PSu.
***EDIT: I messed up. :oops: The software is actually Fast Raw Viewer (FRV). NOT DPP. Apparently I didn't have enough coffee, when I wrote that... :roll: ***

So I deleted the 5 offending files from PSu (from the catalog and disk). Compacted the catalog and thumbnail database. Then
proceeded to re-import the original files directly from my CF card. Using the same import profile but this time direct to PSU (not using DPP *** EDIT: FRV *** first). I add metadata and sync, and now the same thing is happening! I'm a bit surprised to see this as I thought it for sure was the DPP ratings that were causing it.

Has anyone seen this type of behavior in XMP files before? I'm not sure how to clean it up or fix it. I know the XMP files don't take up much space, but something isn't right and it worries me that it may affect the integrity of my metadata.
Last edited by Al_M on 22 Oct 19 15:09, edited 1 time in total.
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

Hert
Posts: 5866
Joined: 13 Sep 03 7:24

Re: Issue with XMP file creation

Post by Hert » 20 Oct 19 7:48

Psu *always* uses the original file name with XMP extension. So your files are not a result of PSU.
Could it be that you have some product installed that monitors the file system?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Username
Posts: 207
Joined: 18 Feb 18 22:21

Re: Issue with XMP file creation

Post by Username » 20 Oct 19 9:46

Does not DPP features a feature for renaming images which can be ran when even you move or re-arrange the sorting order in DPP itself?

Could that have been enabled by mistake?
PSu Server 5 & Postgres 10 on macOS 10.14
- I'm the user

Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Re: Issue with XMP file creation

Post by Al_M » 22 Oct 19 15:58

UPDATE:
For clarification and for the benefit of others reading this thread, I would like to update what I have found.
  • First, the software was Fast Raw Viewer (FRV), NOT Digital Photo Professional (DPP). I edited the original post to reflect this.
  • DPP actually does not create XMP sidecars. Since it's a Canon tool it writes metadata (ratings) to the CR2 file directly.
  • Since I used FRV to 'cull' and assign ratings to the 5 files in question, it was the first to create the XMP sidecar file.
  • Once imported into the PSu catalog, editing metadata in the database and performing a sync caused the new, weird XMP file to be created (with the "_Mac-Pro_Oct-19-.....CaseConflict" in the filename). As best as I can tell the format for the XMP is one that PSu does not like, and once PSu tries to write to it, this conflict file is created as a result. Each subsequent time that you sync in PSu, a new "conflict" file is created. I confirmed this by monitoring the directory in Finder.
  • If the XMP file is created first by PSu, by importing into the catalog and sync'ing, THEN I edit metadata/ratings, colors, etc. with FRV, PSU is perfectly happy and properly reads back metadata even though it was edited by FRV. Obviously this does not work with my workflow of culling and rating with FRV. So I will need to work with the developer of FRV to see if they can update their XMP standard.
I am by no means an expert in XMP file standards, so I have attached one of the files created by FRV for anyone who is interested in looking at it to see what might be causing the incompatibility with PSu.
EOS7D2-19-2299_TEST.xmp
XMP file created by Fast Raw Viewer
(522 Bytes) Downloaded 12 times
Next Steps:
  • Contact FRV developer about XMP file standard.
  • Come up with new workflow for culling and rating prior to import to PSu. (if possible)
  • Continue to update this thread as information becomes available.
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

Hert
Posts: 5866
Joined: 13 Sep 03 7:24

Re: Issue with XMP file creation

Post by Hert » 22 Oct 19 20:55

This sample XMP file is a "half baked" XMP. It only contains a few data elements. The first tool that creates the XMP is responsible to do the conversion of existing metadata (Exif and IPTC-IIM) to XMP. This file doesn't have the metadata converted.

Try this, right click on the thumbnail and select Metadata -> Convert metadata to XMP. That will fix the missing data in the XMP.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Re: Issue with XMP file creation

Post by Al_M » 23 Oct 19 0:47

I agree that the XMP file generated by Fast Raw Viewer is 'minimal'. Considering how it works with PSu, 'half baked' may be a better description... :D

I tried your suggestion (Convert Metadata to XMP) and the "CaseConflict" XMP file generation persists.

At this point I have to 'train' myself not to rate photos while culling and perform the rating in PSu exclusively (at least until a resolution or a workaround surfaces).

FYI, I have attached the XMP file modified by PSu with the Metadata -> Convert metadata to XMP option. Perhaps you can glean some useful bit of information from it.
EOS7D2-19-2777_Mac-Pro_Oct-22-183030-2019_CaseConflict.xmp
XMP (created by FRV and "Converted to XMP" by PSu)
(4.06 KiB) Downloaded 10 times
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

sanphotgn
Posts: 308
Joined: 26 Aug 07 18:06

Re: Issue with XMP file creation

Post by sanphotgn » 23 Oct 19 3:07

Even though the original XMP file only contains a few elements, shouldn't those elements be read by PSu if they are written correctly?

When I compare the original XMP with my examples (written by PSu and/or Lightroom) and examples in the XMP Specification Part 1, dated April 2012, I see differences. Whether those are allowed differences in the standard I don't know.
Photo Supreme 3.3.0.2602 (64 bits) (Windows)

Hert
Posts: 5866
Joined: 13 Sep 03 7:24

Re: Issue with XMP file creation

Post by Hert » 23 Oct 19 8:04

sanphotgn wrote:
23 Oct 19 3:07
Even though the original XMP file only contains a few elements, shouldn't those elements be read by PSu if they are written correctly?
That is correct, but even then I recommend using "Convert Metadata to XMP" to get the XMP complete.
Perhaps you can glean some useful bit of information from it.
EOS7D2-19-2777_Mac-Pro_Oct-22-183030-2019_CaseConflict.xmp
I've checked that file and it indeed looks like a PSU XMP file....but...PSU never uses this file name convention. Some kind of tool is interfering with your file system and renamed PSU's file to this file.

Are your files stored locally or on a NAS? I'm asking because a quick Google search came up with this: https://community.synology.com/enu/forum/17/post/84238
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

sanphotgn
Posts: 308
Joined: 26 Aug 07 18:06

Re: Issue with XMP file creation

Post by sanphotgn » 23 Oct 19 13:31

Al_M - are you using Dropbox?
A case conflict happens when you name a file or folder with the same name as a file or folder in the same location, but with different upper-case and lower-case letters. For example, two files named “records” and “Records”.

Although you can store these two files in the same place on dropbox.com, you can’t store them in the same place in the Dropbox folder on Windows, Mac, and other file systems. Instead, one will be appended with the words (Case Conflict).
https://help.dropbox.com/installs-integ ... e-conflict
Photo Supreme 3.3.0.2602 (64 bits) (Windows)

Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Re: Issue with XMP file creation

Post by Al_M » 23 Oct 19 19:25

Are your files stored locally or on a NAS? I'm asking because a quick Google search came up with this: https://community.synology.com/enu/forum/17/post/84238
Yes! That's it.

I store my files on an external SSD connected via Thunderbolt2. This is the drive PSu reads and writes to directly.

I also have a Synology NAS. Using the Synology Drive client I created a Sync Task which synchronizes that external drive to a backup directory on the NAS pretty much instantly (as soon as the client detects a change on the SSD).

This is a Two-way sync. So (whatever the reason that Synology is detecting a conflict, I'm not really sure why yet) it creates this "...CaseConflict.xmp" file (on the NAS file system) and then is written back to the SSD (because it is a two-way operation).

Now the Synology Drive client does offer a One-way upload option that may remedy the situation.

Another option is to disable the Sync Task all together and instead use a Backup Task on a schedule.

In any case I'll have some testing to do.

Hert, I realize this isn't a Photo Supreme issue, but for the benefit of others in similar situations I would like to keep updating the thread if that's ok with you.
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

Hert
Posts: 5866
Joined: 13 Sep 03 7:24

Re: Issue with XMP file creation

Post by Hert » 24 Oct 19 7:41

Great that you found the culprit.
Al_M wrote:
23 Oct 19 19:25
Hert, I realize this isn't a Photo Supreme issue, but for the benefit of others in similar situations I would like to keep updating the thread if that's ok with you.
This is a user-to-user forum, So as long as topics comply with the forum rules you can post whatever you like.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Re: Issue with XMP file creation

Post by Al_M » 25 Oct 19 2:39

UPDATE:
Fisrt, thanks to sanphotgn. I re-read his post:
Al_M - are you using Dropbox?
No I'm not using DropBox, but it seems that Synology Drive (sync server/client) is operating the same way. So I did a little further searching and found this: https://www.timeexposure.com/support_ce ... cle&id=872

Here is a step-by-step order of operations and my observations:
  • 1. Cull and rate with FRV (Fast Raw Viewer) on local, internal SSD
  • 2. FRV generates the first XMP file on my local, internal SSD --> example.xmp with lower-case extension.
  • 3. Open PSu and Import to Catalog.
  • 4. The raw file and corresponding sidecar (with lower-case .xmp) is copied to the local, external SSD where all my catalogued files live.
  • 5. For whatever reason, when PSu copies the sidecar file it writes to the drive with an upper-case extension --> example.XMP
  • 6. Synology Drive (client) in turn synchronizes (copies) the new files detected on my external SSD to the NAS backup directory, also with upper-case extension.
  • 7. Now, I make a change to metadata say for example assign a color. Metadata > Save Metadata to File. PSu appears to over-write the sidecar file changing the extension back to lower case in the process. This is on the local (external) SSD.
  • 8. Synology Drive dutifully recognizes a change and synchronizes the file to the NAS now with a lower-case .xmp extension.
  • 9. Based on what I read in the above-referenced article the NAS file system interprets the file as a NEW file and writes it to the NAS with the "...CaseConflict..." appended to the filename and lower-case extension. The original upper-case extension file also remains, on the NAS.
    So at this point there are 2 versions of the sidecar on the NAS. One with upper-case extension and one with lower-case extension (and modified name).
  • 10. On the Mac external SSD, only the lower-case (with modified name) sidecar remains. I gather this is because macOS (and Windows for that matter) use case-insensitive file systems and since it doesn't care over-wrote the upper-case version of the sidecar with the lower case version. The NAS in turn re-named the lower-case version with the "...CaseConflict..." and that's the version I see on the macOS volume (external SSD).
I know it's a little hard to follow and there is a lot going on. But I have played with this for hours now trying to get to the root of the issue. I have made lots of notes and summarized them in the above list.

A couple of questions arise as a result:
  • Why does Photo Supreme change the case of the sidecar extension from lower-case to upper-case upon importing to the catalog (specifically with the option to Copy to a different location (than the Source) ? [Step 5 above]
  • Why does PSu chage the case again from upper-case to lower-case when writing to the existing XMP ? [Step 7 above]
  • Can an option to "preserve sidecar extension case upon copy" or "Force lower-case (or upper-case) .xmp extension upon write" be implemented?
FYI, I used the only other tool that I have which can modify XMP metadata (HoudahGEO, which uses exiftool) to test if the case was changed from lower- to upper- (or vice versa) upon writing to the sidecar file. What I found was that it did not modify the case of the extension and therefore avoiding the issue I am having. If PSu could act in the same manner, I believe that would be ideal.
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

Hert
Posts: 5866
Joined: 13 Sep 03 7:24

Re: Issue with XMP file creation

Post by Hert » 25 Oct 19 9:38

Al_M wrote:
25 Oct 19 2:39
  • Why does Photo Supreme change the case of the sidecar extension from lower-case to upper-case upon importing to the catalog (specifically with the option to Copy to a different location (than the Source) ? [Step 5 above]
I was surprised to read that myself. This is changed in the next build.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

Al_M
Posts: 7
Joined: 21 Jun 19 14:09
Location: Kansas City, USA

Re: Issue with XMP file creation

Post by Al_M » 25 Oct 19 17:26

WOW!

Problem solved. I downloaded and installed the latest build and performed some testing. The problem is no longer.
Thank you Hert! I can't believe how quickly you made the necessary changes.

Only issue I see is that per your list of fixes:
Fixes

1. Build 2465; When copying an XMP sidecar files the lowercase .xmp extension is used instead of uppercase
Maybe this will cause issues in the future. (for others)
I wonder if a more flexible and permanent solution would be an option that the user could set under the Preferences. If that's possible, I will post a feature request under Mantis.

I am a relatively new user. Trialed the software in June, and purchased a license in July. I am thoroughly impressed by your customer service. You have earned a loyal customer and I have begun recommending Photo Supreme to anyone interested in a DAM solution.
Photo Supreme V5 (build 2465)
macOS 10.14.6 (Mojave)
Mac Pro (Late 2013), 3.7 GHz Quad-Core Xeon E5, 32 GB DDR3

Post Reply