download/import skips images

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 06 Dec 18 17:32

Next step: running it with "make file names unique"...
Now everything gets downloaded, but the names of files that previously were skipped get an additional (001) appended to their name. NOT what I want.
That the file renaming was messed up by the new multi-threading download feature makes sense.
It would make sense, that the resulting renaming mess, would also result in some of the files being considered duplicates and thus not downloaded at all...

But why would the program suddenly now consider some files duplicates, which it previously never did? Files that have a different name AND different time stamp???
PSU_pseudo-duplicates.jpg
PSU_pseudo-duplicates.jpg (252.75 KiB) Viewed 400 times

Hert
Posts: 21197
Joined: 13 Sep 03 7:24

Re: download/import skips images

Post by Hert » 06 Dec 18 17:57

Did you copy the new script and apply it to your profile? And did you save your profile?

Set duplicate handling to Skip
And a file is duplicate only if the target file name already exists.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 06 Dec 18 20:19

A new script? Where? Did not see that in your post...
I copied the new renaming rule and applied it to my profile...
[And, like I said in one of my previous posts, compiling the rename rule I received this error:
Unknown identifier or variable is not declared: 'Session'. Source position: 28,72
But I assumed that was OK because you suggested it will raise an error..]
Set duplicate handling to Skip
That is what I did!
And a file is duplicate only if the target file name already exists.
Well, the target name DOES NOT exist! How can it? The files are copied to an empty folder!

But perhaps I am doing this wrong and need somehow to download also a new pre-script and in addition to the renaming rule apply that one to the profile...
Where can I get the script? I have just looked at the script repository and form there only the old version is available:
https://repository.idimager.com/openres ... 2408FCE4B0

Hert
Posts: 21197
Joined: 13 Sep 03 7:24

Re: download/import skips images

Post by Hert » 07 Dec 18 1:05

Can you double check that the script (I mean the rename rule, which is a script too) was copied to your profile?

And yes, that is the error that you can ignore. Though you should use build 1790 or higher.
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 07 Dec 18 17:58

Can you double check that the script (I mean the rename rule, which is a script too) was copied to your profile?
Absolutely, it certainly was!

AFAIK, I am using the most recent build: 1793.

Hert
Posts: 21197
Joined: 13 Sep 03 7:24

Re: download/import skips images

Post by Hert » 07 Dec 18 18:38

I did a test here today and could import 300 raw with jpg pairs without any issue. All names were consistent. Also no skips or duplicates. So I’m sure that the customized script works well.

Can you share your changed import profile here?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 07 Dec 18 21:24

Here it is:
ASU_Macro_2018-12-07.zip
(2.36 KiB) Downloaded 4 times

Hert
Posts: 21197
Joined: 13 Sep 03 7:24

Re: download/import skips images

Post by Hert » 07 Dec 18 23:13

Thank you. The profile does contain the new script.

I ran another test with this profile (to rule out that there are additional changes) and again no duplicates and consistent naming.
AfterImport.png
AfterImport.png (886.87 KiB) Viewed 296 times

Could it be that you're copying to a folder that already contains files with the numbers in the range?
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 08 Dec 18 1:40

I ran another test with this profile (to rule out that there are additional changes) and again no duplicates and consistent naming.
Very strange...
Could it be that you're copying to a folder that already contains files with the numbers in the range?
No.
Well, at least in PSU, I deleted all the files previously downloaded to this folder.
One of the obvious problems, I have a limited number of completely "fresh" files and with all the troubleshooting, I keep downloading and then (if the download did not work) delete the files again...
Could it be that PSU does not correctly delete all references when choosing "delete images from disk and database"?

On Monday, I'll delete these files again, do a database compact and the try downloading the files again...
Then, I will also do a completely fresh new download of files that were never downloaded/imported into PSu.

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 11 Dec 18 0:33

OK, I just tried it with a completely new, fresh set of images, never having been touched by PSu.
Renaming works perfectly, image pairs correctly renamed.

BUT ...
Out of 38 image files, only 24 are imported, 14 skipped.
No idea why.

:(

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 11 Dec 18 0:37

Just to illustrate a screenshot.
After downloading and opening the importer there remain several pairs of images that are still marked for download, which the downloader simply ignored...
:(
Attachments
Images_skipped_after-download.jpg
Images_skipped_after-download.jpg (316.79 KiB) Viewed 117 times

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 11 Dec 18 1:08

Not using the rename rule and only downloading JPGs, not pairs, no images are skipped.

Hert
Posts: 21197
Joined: 13 Sep 03 7:24

Re: download/import skips images

Post by Hert » 11 Dec 18 10:31

fbungarz wrote:
11 Dec 18 0:33
Out of 38 image files, only 24 are imported, 14 skipped.
Did you check that numbers in the series didn't already exist on the hard drive?

I just checked the PSU code and the script code and I can't think of any reason why this wouldn't work. And as mentioned, it works flawless here. So something must be different. Already existing files in the target folder in the range could be an explanation.

Anyway, here's a version for your rename script that has locking hard coded in the script. I don't expect this to make any difference since PSU already does name rule parsing inside a lock of the import session...and so this is a wild shot. If this too doesn't work for you then I can only imagine that the files already exist in your target folder from a previous download. Or that somehow you're not using the download profile that you expect to be using.

Code: Select all

//    - This script is designed to be used with the downloader pre-script
//      "Download by Prefix", as it manages some of the registry keys used
//      by this script.
// JAG, 16-Apr-2007
// HVZ, 23-Jun-2016 (converted to Photo Supreme)
// HVZ, 06-Dec-2018 (made script independent of the sequence as with multithreading, the sequence is not predictable)
// HVZ, 11-Dec-2018 (added critical section)

const
  cScriptName = 'DownloadByPrefixList';
var
  highestNumber, lastNumber: Integer;
  prefix:     WideString;

  AStackName: WideString;
  AFiles: TTntStringList;
  ALock: TxomCriticalSection;
  idx: Integer;
 
begin
  ALock := TxomCriticalSection(StrToPtrInt(Nvl(PopFromStack(cScriptName), '0')));
  if ALock = nil then
  begin
    ALock := TxomCriticalSection.Create(cScriptName);
    PushToStack(cScriptName, PtrIntToStr(ALock));
  end;

  ALock.Acquire;
  try
    prefix        := ReadFromRegistry ('dlByPrefixCurrent', 'prefix', '');
    if (prefix = '') then prefix := 'UNKNOWN_'; 
    highestNumber := ReadFromRegistry ('dlByPrefixCurrent', 'value', 0); 

    AStackName := cScriptName + prefix;
    AFiles := TTntStringList(StrToPtrInt(Nvl(PopFromStack(AStackName), '0')));
    if AFiles = nil then
    begin
      AFiles := TTntStringList.Create;
      PushToStack(AStackName, PtrIntToStr(AFiles));
    end;
    if VarToStr(Nvl(PopFromStack(AStackName + 'session'), 0)) <> Session.GUID then
    begin
      AFiles.Clear;
      PushToStack(AStackName + 'session', Session.GUID);
    end;

    idx := AFiles.IndexOf('%FileName');
    if idx = -1 then
    begin
      lastNumber := highestNumber + 1;
      AFiles.AddObject('%FileName', lastNumber);  // maintain the number used for this filename
    end
    else
    begin
      // reuse the same number for existing files (e.g. TEST.JPG and TEST.NEF)
      lastNumber := AFiles.Objects[idx];
    end;

    if lastNumber > highestNumber then
    begin
      WriteToRegistry ('dlByPrefixLastVals', prefix, lastNumber);
      WriteToRegistry ('dlByPrefixCurrent', 'value', lastNumber);
    end

    // return our newly constructed filename
    result := prefix + AddLeadingChars (IntToStr(lastNumber), '0', 4, False);
  finally
    ALock.Release;
  end;
end;
This is a User-to-User forum which means that users post questions here for other users.
Feature requests, change suggestions, or bugs can be logged in the ticketing system

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 11 Dec 18 16:28

Did you check that numbers in the series didn't already exist on the hard drive?
The target folders do not even exist prior to the download. The profile creates them as new subfolders in "..\yyyy\yyyy_mm_dd\". The parent of these folders named by date is a folder with prefix of the script. In this particular case I used a fresh, newly created parent folder with a new prefix for testing this out.
However, I did not create a new folder for each test. So, like I said in a previous post: I try run the download, when it does not work I delete the images "from database and disk" in PSu. If the database somehow "remembers" any of these names, I guess it is then possible it gets confused.

Also, perhaps it still has to do with storing/accessing the registry. The idea is that the script would store the" last number used+1" for a particular prefix in the registry and from there starts renaming the files. Now, since for these particular test runs I always had to delete the incompletely downloaded files, I also had to reset this counter number. If there is something messed up with how the numbers are stored in the registry, then perhaps that could explain some of the skipped files? Just speculating. No idea really...

fbungarz
Posts: 3250
Joined: 08 Dec 06 5:03
Location: Galapagos, Ecuador

Re: download/import skips images

Post by fbungarz » 11 Dec 18 22:01

Same results with the new rename rule: of 36 images selected, only 24 get downloaded, 14 skipped.

Post Reply