Parallel processing of files: script ends early

dejhost · October 20, 2020

Hello everybody.

I am writing an autoit-script that launches a python-script. The python script is processing files that are listed in the array "$iList". This array is created using _FileListToArray, so $iList[0] contains the number of files in the array.

The python-script is supposed to run $k = 16 times in parallel. I delay the start of processing the files $t = 10 * 1000ms.

This is my code:

Local $k = 16   ;  Amount of files processed simultaniously
Local $t = 10 * 1000 ; Time to wait inbetween launching python
Local $aProcessList[1]
Local $i = 1

While $iList[0] > 0
    While $iList[0] > 0 AND Ubound($aProcessList) <= $k
        $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[1]
        Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE)            ; C = Close, K = Keep Open; Returning Process ID
        Sleep($t)
        $aProcessList = ProcessList("python.exe")
            ConsoleWrite("Python is running times = " & Ubound($aProcessList) &  @CRLF)
        Local $return =_ArrayDelete($iList, 1)
            ConsoleWrite("$return = " & $return & "        @error = " & @error  &  @CRLF)
        $iList[0] = $return - 1
    Wend
    _ArrayDisplay($iList, "READY NOW") ; this Array is never shown. Why?
    Sleep($t)
    $aProcessList = ProcessList("python.exe")
Wend

This seems to work to a certain degree. I am asking for your help to get it working all the way and simplify the code. Here are the things I wonder:

I believe that the inner WHILE-loop is not left at any stage. The Array with the title "READY NOW" is never shown. Why is that so?

The last lines of the console:

Python is running times = 14
$return = 217        @error = 0
+>10:53:24 AutoIt3.exe ended.rc:0
+>10:53:24 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 75.12

Why is the Python running 14 times? It's supposed to be $k = 16
Not all files are processed. The script just ends at the stage shown above. Why aren't all files processed.

Thanks in advance for your help.

Jos · October 20, 2020

There is no way we can run this code to reproduce what you are seeing as $iList isn't defined, so what about you post something that does reproduce the issue when run?
You do pause 10 seconds in each inner loop, so maybe python processes simply ended in the mean time?
No idea why the code suddenly ends without any message/error... hence the need for a reproducer script.

Jos

Edited October 20, 2020 by Jos

dejhost · October 20, 2020

5 minutes ago, Jos said:

There is no way we can run this code to reproduce what you are seeing as $iList isn't defined, so what about you post something that does reproduce the issue when run?

Jos

Thank you for your input, Jos.

The code creating the $ilList-array:

Local $iList = _FileListToArray("C:\Users\Jost\Documents\RnD\Subsea_Images\01SelectionCopy\", "*.jpg", $FLTA_FILES, True)

The result looks like this:

Quote

You do pause 10 seconds in each inner loop, so maybe python processes simply ended in the mean time?

Correct. Python processes may last a few seconds up to a few minutes. My intention is to always check how many python.exe-processes are currently running, and launch python until 16 files are processed in parallel, or there are no more files to process.

I am aware that this is not a reproducer script. I am trying to come up with something useful.

dejhost · October 20, 2020

For testing purposes, I set $k = 2. Now, the Array with the title "READY NOW" is appearing. So basically the delays by the sleep()-command are so high, that the processing is faster. Good to know.

Still I wonder why not all files are being processed.

I created a script to reproduce the problem. Instructions:

Create sleep_short.bat and sleep_long.bat on your Desktop. Write "timeout 5" and "timeout 180" in them.
Change the path in the following code to match you own desktop.

#include <Array.au3>

;~CREATE ARRAY ilist
Local $iList[101]
$iList[0] = 100
Local $i

For $i = 1 to 99 STEP 2
    $iList[$i] = "C:\Users\Jost\Desktop\sleep_short.bat"
Next

For $i = 2 to 100 STEP 2
    $iList[$i] = "C:\Users\Jost\Desktop\sleep_long.bat"
Next

_Arraydisplay($iList)


Local $k = 5     ;  Amount of files processed simultaniously
Local $t = 1 * 1000 ; Time to wait inbetween launching cmd
Local $aProcessList[1]
Local $i = 1

While $iList[0] > 0
    While $iList[0] > 0 And UBound($aProcessList) <= $k
        $CMD = $iList[1]
        Run(@ComSpec & " /c " & $CMD, "", @SW_SHOW)            ; C = Close, K = Keep Open; Returning Process ID
        Sleep($t)
        $aProcessList = ProcessList("cmd.exe")
        ConsoleWrite("CMD.EXE is running times = " & UBound($aProcessList) & @CRLF)
        Local $return = _ArrayDelete($iList, 1)
        ConsoleWrite("$return = " & $return & "        @error = " & @error & @CRLF)
        $iList[0] = $return - 1
    WEnd
    _ArrayDisplay($iList, "READY NOW")
    Sleep($t)
    $aProcessList = ProcessList("cmd.exe")
WEnd

This seems to work alright!?

Any wild guesses are appreciated!

Jos · October 20, 2020

17 minutes ago, dejhost said:

This seems to work alright!?

Any wild guesses are appreciated!

So this reproducer script is working (so not a reproducer) ? :think:

... and your real script is still stopping unexpectedly without any error or strange returncode?
The original script doesn't have any exitloop or exit statements in those Close-Loops?

Jos

dejhost · October 20, 2020

Quote

So this reproducer script is working (so not a reproducer) ?

Correct. The "reproducer" uses the same code (as far as possible) and runs fine. All 100 files are beeing "processed".

Quote

... and your real script is still stopping unexpectedly without any error or strange returncode?

Yes. In my first post you can see that it starts processing 223 files, and stopped when down to 217. No error-message or so. Right now I got:

Python is running times = 7
$return = 179        @error = 0
Python is running times = 8
$return = 178        @error = 0
Python is running times = 9
$return = 177        @error = 0
Python is running times = 10
$return = 176        @error = 0
Python is running times = 11
$return = 175        @error = 0
Python is running times = 12
$return = 174        @error = 0
Python is running times = 11
$return = 173        @error = 0
Python is running times = 12
$return = 172        @error = 0
+>15:29:41 AutoIt3.exe ended.rc:0
+>15:29:42 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 524.5

So still having this problem. I inserted "$iList[0]" in the Consolewrite and ran the script again. Python was still running in the background (because of the last execution). The entire output shows:

Python is running times = 10
$return = 223        $iList[0] = 223      @error = 0
Python is running times = 10
$return = 222        $iList[0] = 222      @error = 0
Python is running times = 10
$return = 221        $iList[0] = 221      @error = 0
Python is running times = 10
$return = 220        $iList[0] = 220      @error = 0
+>15:39:54 AutoIt3.exe ended.rc:0
+>15:39:54 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 42.46

So while "$iList[0]" contains correct values, the script ends very fast. Do you have an idea on how to find out why?

Quote

The original script doesn't have any exitloop or exit statements in those Close-Loops?

Correct. If I am not mistaken, I do not need such statements:

The outer WHILE loop re-triggers the inner WHILE loop as long as there are files left to process.
The inner WHILE-loop is entered if less than $k = 16 files are currently processed AND there are files left to process.

There are certainly more elegant ways to do this (happy for suggestions!), but it seems to work in the reproduction-script!?

Jos · October 20, 2020

I probably would have done it with a for next loop to process all entries in an array... something like this:

For $x = 1 To $iList[0]
    ; wait till slot is available
    Do
        $aProcessList = ProcessList("python.exe")
        sleep(1000)
    Until UBound($aProcessList) <= $k
    ; run next process
    $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x]
    Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE)            ; C = Close, K = Keep Open; Returning Process ID
    ConsoleWrite("Python is running times = " & UBound($aProcessList) & @CRLF)
    Sleep($t)
Next

As to your issue with the posted code .... not idea and will need that replicator script to debug.

dejhost · October 21, 2020

I testet your code. I just changed the content of the consolewrite. This is what I got in return:

$x = 66        $iList[0] = 223      UBound($aProcessList) = 13        @error = 0
$x = 67        $iList[0] = 223      UBound($aProcessList) = 13        @error = 0
$x = 68        $iList[0] = 223      UBound($aProcessList) = 13        @error = 0
$x = 69        $iList[0] = 223      UBound($aProcessList) = 14        @error = 0
$x = 70        $iList[0] = 223      UBound($aProcessList) = 14        @error = 0
$x = 71        $iList[0] = 223      UBound($aProcessList) = 14        @error = 0
+>09:49:35 AutoIt3.exe ended.rc:0
+>09:49:35 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 778.2

So basically the same behaviour.

So my conclusion is, that the problem is not within the autoit script. Two more thoughts:

Maybe the script aborts because of overload of computer resources (CPU/Memory)? But shouldn't there be some message in this case? Anyhow: I will reduce to $k = 11, try again and come back to you.
It is also likely that there is an error in the python-script. But if python returns an error and aborts - this shouldn't make autoit-script abort too, right?

Edited October 21, 2020 by dejhost

JockoDundee · October 21, 2020

35 minutes ago, dejhost said:

So basically the same behaviour.

Regarding the job that fails, when it does fail, does it fail at exactly the same point and same way?

If it doesn’t, it points to some sort of resource/permission issue or race condition.

Jos · October 21, 2020

1 hour ago, dejhost said:

So my conclusion is, that the problem is not within the autoit script. Two more thoughts:

Maybe the script aborts because of overload of computer resources (CPU/Memory)? But shouldn't there be some message in this case? Anyhow: I will reduce to $k = 11, try again and come back to you.

It is also likely that there is an error in the python-script. But if python returns an error and aborts - this shouldn't make autoit-script abort too, right?

Neither make sense to me as I would expect and hard crash with some high value RC or in case 2 it would only terminate the python pid, not the AutoIt3 process.
Do you have some Anti Virus software acting up and killing the Autoit3 process?

dejhost · October 21, 2020

24 minutes ago, JockoDundee said:

Regarding the job that fails, when it does fail, does it fail at exactly the same point and same way?

It does not fail at the same point, but it does fail the same way. Explanation:

The python script uses quite some PC-resources during the first few seconds (about 6 seconds, 40% of CPU). It then decides whether

a) the current file has not been processed before => let's start processing. This takes a few minutes, but uses only about 4% of PC-resources from now on.

b) the current file has been processed before => so it ends the python-program. In this case, AutoIT will initiate the next file.

Starting the Autoit-script over and over again yesterday and today, more and more files get processed, and python chooses more and more often b) when starting.

So it cannot fail at the exact same point since the same file is not processed twice.

However, it fails the same way:

$x = 98        $iList[0] = 223      Python is running times = 12        @error = 0
$x = 99        $iList[0] = 223      Python is running times = 12        @error = 0
$x = 100        $iList[0] = 223      Python is running times = 12        @error = 0
$x = 101        $iList[0] = 223      Python is running times = 11        @error = 0
$x = 102        $iList[0] = 223      Python is running times = 12        @error = 0
$x = 103        $iList[0] = 223      Python is running times = 12        @error = 0
+>10:58:35 AutoIt3.exe ended.rc:0
+>10:58:35 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 3182

While I receive this message from autoIT, there are still plenty of python-scripts continuing to run for several minutes in the background...

The first time I ran the script today, 47 files were created. The second time, 119 files were created.

Jos · October 21, 2020

Post the latest version of your script please so I can see whether we can some extra debugging options to it.

Jos

dejhost · October 21, 2020

1 hour ago, Jos said:

Post the latest version of your script please so I can see whether we can some extra debugging options to it.

Jos

It's basically your script.

Local $k = 10     ;  Amount of files processed simultaniously
Local $t = 10 * 1000 ; Time to wait inbetween launching python
Local $aProcessList[1]
Local $i = 1


For $x = 1 To $iList[0]
    ; wait till slot is available
    Do
        $aProcessList = ProcessList("python.exe")
        sleep(1000)
    Until UBound($aProcessList) <= $k
    ; run next process
    $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x]
    Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE)            ; C = Close, K = Keep Open; Returning Process ID
        ConsoleWrite("$x = " & $x & "        $iList[0] = " & $iList[0] & "      Python is running times = " & UBound($aProcessList) & "        @error = " & @error & @CRLF)
    Sleep($t)
Next

msgbox(1, "processing finnished", "processing finnished")

Ok, here are the good news: It processed all files by now. The script ran all the time through, from 13:45 to 15:22, so roughly 1,5 hours. That's about 20min longer than the previous run. There are 2 things that were different:

PC-resources: I reduced $k = 10. Instead of 60%-70% CPU-usage, I get about 50%. So maybe there were peaks in CPU-load or the memory that led to the stop.
Windows Defender had something to do with it. I had it deactivated during the last run.

For testing purposes, I will turn on the Windows Defender and process the files over again. I will report to you once this is done. If this works fine, I will set $k = 20 and see if the following crash shows the same symptoms as so far: a clean end of the script.

Jos · October 21, 2020

1 minute ago, dejhost said:

Windows Defender had something to do with it. I had it deactivated during the last run.

99.99999999999999% sure this was your culprit as that is the only way I could explain for processes to suddenly end without any RC other than 0.
...and lets face it: it isn't that common that a process shells soo many other processes in the legitimate world.

Jos · October 21, 2020

Something like this would be my next try to get the reason for the unexpected process closure:

Local $k = 10     ;  Amount of files processed simultaniously
Local $t = 10 * 1000 ; Time to wait inbetween launching python
Local $aProcessList[1]
Local $i = 1
Dim $iList[2] = [1, "test"] ; just added for testing
OnAutoItExitRegister("Script_Ended")

ConsoleWrite("-processing For-Next items " & $iList[0] &  @CRLF)
For $x = 1 To $iList[0]
    ; wait till slot is available
    Do
        $aProcessList = ProcessList("python.exe")
        Sleep(1000)
    Until UBound($aProcessList) <= $k
    ; run next process
    $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x]
    Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE)            ; C = Close, K = Keep Open; Returning Process ID
    ConsoleWrite("+ $x = " & $x & "        $iList[0] = " & $iList[0] & "      Python is running times = " & $aProcessList[0][0] & "        @error = " & @error & @CRLF)
Next
ConsoleWrite("-processing For-Next done." & @CRLF)

Func Script_Ended()
    Switch @exitCode
        Case 0
            $sMsg = "normal"
        Case 1
            $sMsg = "by exit"
        Case 2
            $sMsg = "by click"
        Case 3
            $sMsg = "by logoff"
        Case 4
            $sMsg = "windows shutdown"
        Case Else
            $sMsg = "unknown?"
    EndSwitch
    ConsoleWrite("+processing ending code: " & @exitCode & " - " & $sMsg & @CRLF)
EndFunc   ;==>Script_Ended

dejhost · October 22, 2020

I changed my mind - and turned off Windows Defender, while increasing to $k = 12. The script processed all files and ended properly. I will increase $k a bit more and add exclusions for Defender in order to avoid troubles from this end. Some more tests during the next days will hopefully support my current conclusion: problem solved!

The "OnAutoItExitRegister("Script_Ended")" - Function is a handy thing. Good to know that something like this exists!

Thank you so much for your support! This forums is still one of the amazing places in the internet.

JockoDundee · October 22, 2020

Jos declaring 14 9’s of certainty is proof enough for me

dejhost · October 22, 2020

1 hour ago, JockoDundee said:

Jos declaring 14 9’s of certainty is proof enough for me

Agreed 🙂

Parallel processing of files: script ends early

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members