Problem with "build missing thumbnails and previews"

[PhilBurton]

Is there any way to run an "image validate" function that would detect and optionally remove corrupted images?

Not that I'm aware of.

Is this something that should go into Mantis?

Phil

[vlad]

I don't know to what extent PSU (or any other program) could detect corrupted images and signal them as such. I guess one question is how is "corruption" defined and what would be the scope of corruption detection.

(For example, this discussion concerns thumbnail (re)building, but aren't those images which require "Convert Metadata to XMP" corrupt in some way (i.e. non-compliant with the standard), too? I don't want to hijack this thread, but I have always wondered how could I know to run that operation and on which image selection, unless I find out which images are missing proper XMP data in the first place.)

[Mike Buckley]

how could I know to run that operation [ "Convert Metadata to XMP"] and on which image selection, unless I find out which images are missing proper XMP data in the first place.

I don't think it will hurt anything to run it, so you could run it on all images. As an example, I always have to run it on all vertically oriented raw files that have been edited in Capture NX2. Rather than take the time to select only the vertically oriented images, I select all of them.

[PhilBurton]

I don't know to what extent PSU (or any other program) could detect corrupted images and signal them as such. I guess one question is how is "corruption" defined and what would be the scope of corruption detection.

Let's define corruption as some issue with the image file or its sidecar XMP file that causes PSu to misfunction or even crash. The misfunction could be an inability to display the thumbnail, inability to import more images, or a general freezeup.

Maybe what's needed is some kind of diagnostic reporting, like the kind you get when Windows bluescreens.

Phil

[n8vz]

I don't know of any such routine, either. It would be a nice addition in a future update. It seemed to me that there used to be more "fix it" routines in the old IDI than in PSU.

That brings me back to the problem. I tried Mike's latest suggestion, and I still have not resolved the issue. I've also noted that PSU seems to be running slower than it used to.

[Hert]

Slower? Compact the database regularly, a weekly frequency is a good average maintenance cycle.

[Mike Buckley]

Slower? Compact the database regularly, a weekly frequency is a good average maintenance cycle.

If it really should be done weekly, that information should be included in the "Maintenance" Quick Manual. It only says that compacting should be done "on a regular basis." A regular basis could be every six months or every week.

Personally, I have gotten to the point that I compact only when I make changes to the system of catalog labels though I would also compact if I used the custom XMP capabilities and the like. I have never experienced any of the problems experienced by people who don't compact.

[Hert]

There's no reason why you should compact after changing your catalog label system or using custom XMP. Main reason to compact is to clean up fragmentation in the database file(s).

[n8vz]

Slower? Compact the database regularly, a weekly frequency is a good average maintenance cycle.

I do this regularly.

[Mke]

Is there any way to run an "image validate" function that would detect and optionally remove corrupted images?

Not that I'm aware of.

No - but fixity checking to guard against digital decay is certainly something that I'd like to see in PSU. Without it, it could be years before a corruption problem is discovered, by which time it is likely that any backups would also contain corrupted versions of the file, unless you use abnormally robust archiving. Only a small sample, but doing some quick math on the Getty Research Institute report at http://blogs.getty.edu/iris/preventing-digital-decay/ suggests that there is a likely complete corruption of around 2 images per decade per 1,000 images. (In fact maybe that rate would now be higher, since image files are now much bigger - so more data to corrupt?)

With an in-depth knowledge of the image file structure would I imagine it would be possible to separate the data for the image (which wouldn't be expected to change) with the metadata in the file (which would, for those of use who write metadata for the files) and to create a checksum for the former. I thought I'd put in a Mantis request for it already, but as I can't find it it looks like I didn't.

The only photo software that offers a partial solution (restricted to DNG files, which have inbuilt checksums) is Lightroom (from version 5).

Of course if you never change your files once their created, you can do it with 3rd party solutions - such as ExactFile http://www.exactfile.com/ (donationware).
Or for the ultimate solution... ...archive your images to analog film, perhaps (https://web.archive.org/web/20150330191 ... _sheet.htm)?

[Mike Buckley]

there is a likely complete corruption of around 2 images per decade per 1,000 images.

That's only 0.02% per year! Considering that PhotoSupreme is available via a perpetual license of only $100, if I was the developer that statistic would be the best economic motivation I could imagine for not taking the effort to build the corruption-detection capability into the software. I would leave that up to a third-party provider. That's especially true considering that there are free checksum products.

[Mke]

there is a likely complete corruption of around 2 images per decade per 1,000 images.

That's only 0.02% per year! Considering that PhotoSupreme is available via a perpetual license of only $100, if I was the developer that statistic would be the best economic motivation I could imagine for not taking the effort to build the corruption-detection capability into the software. I would leave that up to a third-party provider. That's especially true considering that there are free checksum products.

It's also a total loss of 200 images out a collection of 100,000 over 50 years - plus all the partial corruptions - more if file size plays a part, as I suspect; whether that's good or bad is subjective. Compared to the total loss of zero images in my old analog collection (which dates back over 130 years - no I didn't take them all) it doesn't seem particularly good to me.

In marketing terms, adding a feature that makes a real difference in protecting your images would be a good selling point. Adobe seem to think so, even if they've only picked off an easy partial solution.

[Mike Buckley]

In marketing terms, adding a feature that makes a real difference in protecting your images would be a good selling point. Adobe seem to think so, even if they've only picked off an easy partial solution.

Actually, I looked for the selling point at Adobe's website and couldn't find it. Even if it's buried somewhere so deeply that I missed it, it's clear to me that they don't think it's a particularly good selling point. If it was, it would be displayed more prominently or at all.

[n8vz]

I still have not solved this problem. I can't find the remaining apparently corrupt files that are presumably at fault. This is very frustrating. PSU really needs a utility to identify the files that it is having trouble with and that may be corrupt.

[Mike Buckley]

Have you tried the suggestions I made in my 5th and 6th posts in the thread?