download/import skips images

[fbungarz]

Hi all,
I have recently downloaded quite a few images and for some reason it repeatedly keeps missing some of the files marked for download.
In my workflow I shoot JPG+NEF and download the images in pairs into folders named by date picture taken, e.g.: 2018-10-02

The pairs get renamed in pairs from something like "_DSC8222.JPG+_DSC8222.NEF" to "PX_10624.JPG+PX_10624.NEF" using a script that Hert provided (the script allows me to enter a prefix and adds a sequence number storing the last number used for that particular prefix in the registry).

Then I use the DNG converter on the NEFs to generate DNG copies of all the NEFs, use "Verify Folder" to make sure the DNGs show up and then use "Detect Versions" to keep the triplets of JPG+NEF+DNG together.
Doing that I first thought the version detection wasn't working, because I ended up with different versions of images in the same triplet. But then I noticed the files are named the same, but the images were different:

PSu_download_not-working.jpg (133.7 KiB) Viewed 295143 times

Comparing the images with the folder from which I downloaded the files showed that quite a few files had not been downloaded AT ALL. PSu simply had skipped these files...
It seems that more often the JPGs are skipped, but occasionally also the NEFs are not there.


This is really annoying, because it means I can no longer trust the downloader to get ALL images that I want into PSu...
I initially started to copy the missing files manually and rename then and then re-version by hand. That worked for folders with only a few missing images. But in one case almost 10 images out of 30 were not there. Which files are not copied seems entirely random, I cannot detect any pattern...

Anybody else having the same problems?

Frank

[fbungarz]

I figured out what was wrong. One of the NEFs from the pairs was missing (didn't get copied from the memory card?).
Thus, when the script renamed the following images it messed it all up.
None of PSu'S fault. Sorry for the confusion...

[fbungarz]

Actually I was mistaken.
I double-checked that there were NEF+JPG pairs for ALL images!
Then I started the download again.
Turns out only some 44 of a total of 86 images were downloaded this time.
When I look at the downloader again, only the images downloaded are unmarked, the ones that were skipped are still marked for download. You can see from the screenshot that sometimes entire HPG+NEF pairs are not downloaded, sometimes only a NEF or a JPG gets skipped.

Images_skipped_during_download.jpg (309.56 KiB) Viewed 295135 times

Something is definitely not working...


I could file a report in Mantis, but I am not sure if this only affects my system. Perhaps the renaming script no longer functions correctly...

[Hert]

Nothing has changed in the Import routines for "ages". And if your script wouldn't work then nothing would get copied I presume. Could be that the drive of the target folder is full?

Best to do a clean test. Empty card, shoot a few pictures, import, check results.
fwiw; the items that are "pre-selected" are files that have their attribute flag not on the source file. PSU will set the flag after download; but sometimes the files already have the flag set by some other application or operating system...depending on the history of the source files.

[fbungarz]

Nothing has changed in the Import routines for "ages". And if your script wouldn't work then nothing would get copied I presume. Could be that the drive of the target folder is full?

Hhhmmm.
Something must have changed. This always worked without a glitch before. Never had this happening before. At least not since you supplied me with an update of this script. Could it be that an update of the PSu database to version 4 affected how scripts are being loaded?

Best to do a clean test. Empty card, shoot a few pictures, import, check results.

I never download/import directly from the memory card. I use FastPicture Viewer to first cull and rate the images and then import from the HDD folder onto which I copied the images with FastPicture viewer.
In PSu I always mark all images from that folder then to be imported.
At first I thought there might have been a copying glitch, when moving the JPG+NEF files from the memory card to the folder using Fast Picture Viewer. That would mess up the naming during import and perhaps cause some to the files not to be copied. So I double-checked. Tried with a small batch of pairs. The same: PSu simply does not import some of the files.
I tried several different imports yesterday. All failed to download ALL images in that folder. And every import different files were left over. No consistent pattern discernible which ones would be skipped.
The folder from where the images are imported is just an ordinary folder, not specific security settings, not on the system drive; so is the folder created by PSu to which the images are downloaded...

fwiw; the items that are "pre-selected" are files that have their attribute flag not on the source file. PSU will set the flag after download; but sometimes the files already have the flag set by some other application or operating system...depending on the history of the source files.

The files are fresh from the camera, copied via FastPictureViewer. I have used that same version of FastPictureViewer for ages, never had any problem. Even if I manually deselect all images in the import window and then re-select all, some files are still being skipped! And again: how many files are skipped and which particular ones is always different. Sometimes 5 of 86 total images, sometimes 20, sometimes 43 are not imported. Completely inconsistent.

In the end I manually named all 86 files (43 image pairs), copied them to the folder and used Verify Folder to get them into PSu. Not particularly convenient...


Can an import profile get corrupted?

I'll try taking some fresh pictures today and do some tests using perhaps also a different profile.

[snowman1]

Can an import profile get corrupted?

I doubt it's relevant but it seems completely random whether my import profile comes up with a destination folder of F: or F:/dcim/
It should be F: (my laptop's memory card drive) - the reason being that my Panasonuc cameras place videos in a folder that is not under /dcim/ - but no matter how many times I save the profile it is still random which one it comes up with. But like I say, probably totally unrelated to your issue.

[fbungarz]

OK - the downloader is completely broken!
I have used it to import many different files from different folders using different import profiles. EACH time a significant amount of images does NOT get imported!
An example - importing lots of images: 134 images (= 67 NEF+JPG image pairs) > only 70 images get imported, sometimes NEFs skipped, sometimes JPGs, sometimes both.

Another example - importing only 10 images: only 8 get imported ;-(

The ONLY thing that is consistent: before the import all images are marked to be imported, afterwards the ones that have been skipped are still marked, the ones that have been imported are now now longer marked!

[fbungarz]

Also: the script no longer remembers the last number from the registry. Instead it now always displays the last number entered.
Has there been a modification where the registry values that PSu uses is stored has changed?
I am using a custom shortcut with a path to a custom registry:
"C:\Program Files\IDimager Products\Photo Supreme x64\IDimagerSU.exe" -regfile "C:\PSU Catalog\mydata.idreg" -A "C:\PSU Catalog"

[fbungarz]

I'll try compacting the database tomorrow. Maybe that helps...

[Hert]

I have done a quick test here with the same startup parameters as you're using. That is working fine and registry values are persisted.

You could try by renaming the mydata.idreg file to something else in the "C:\PSU Catalog" folder. Then start PSU. You will then start with a new registry file

[fbungarz]

You could try by renaming the mydata.idreg file to something else in the "C:\PSU Catalog" folder. Then start PSU. You will then start with a new registry file

Hhmm, yes if that is the reason for the hickups, I'll probably need to do this. I am a bit reluctant though, because it's not just the number counts that are stored in that custom file, but quite a few database customizations. I have been thinking that these too might cause the problems...

I have experimented a bit further. What seems to work is if all images were taken on the same exact date and are stored together in one and the same folder (no subfolders). At least for smaller batches no images are then skipped.

And it is not that copying from the memory card onto the HDD via Fast PictureViewer is to blame either. Yesterday, in the lab,I used ControlMyNikon to shoot images directly to the HDD via tethered USB. ControlMyNikon automatically names the images and downloads them from the camera immediately onto my HDD via the tethered connection (skipping the camera's memory card entirely):

---folder Lichen name
---folder Collector A
--- folder Collector number 1
--- folder Collector number 2
--- folder Collector number 3
---folder Collector B
--- folder Collector number 1
--- folder Collector number 2

Now, if I download from the top folder "Lichen name", tons of images get skipped, apparently random. And with each run, different images are skipped. If I download from the lowest subfolders, one subfolder at a time no images are skipped. But those folders typically contain only 4-5 pairs of images anyway. And it would be much more conventient to be able to download everything in one go (as I always did before).

Well, I'll try more troubleshooting, compacting, using a different regfile. Seems like nobody else here is reporting similar problems ...
wich probably means it will be hard to resolve

[fbungarz]

Even if downloading a single folder from the same date, if there are more than 10 images, 1 or 2 get skipped...

[fbungarz]

After compacting both the main and the thumbs database I tried it again with 66 brand new images. Only 42 got imported.
Tried replacing the reg-file. No difference.
No idea what else to do. This has worked flawlessly many, many, many times before updating to version 4...

[fbungarz]

Just to give you an idea: all the ones that have the green check mark were NOT downloaded. 24 of 66 images total = 36% = more than a third!

PSu_after-import_marked-skipped-images_03.jpg (405.96 KiB) Viewed 295000 times

[Hert]

If you remove the copy script, will it then copy all your images?
And you've never used the import feature in V4 before this?