dejhost Posted October 20, 2020 Share Posted October 20, 2020 Hello everybody. I am writing an autoit-script that launches a python-script. The python script is processing files that are listed in the array "$iList". This array is created using _FileListToArray, so $iList[0] contains the number of files in the array. The python-script is supposed to run $k = 16 times in parallel. I delay the start of processing the files $t = 10 * 1000ms. This is my code: Local $k = 16 ; Amount of files processed simultaniously Local $t = 10 * 1000 ; Time to wait inbetween launching python Local $aProcessList[1] Local $i = 1 While $iList[0] > 0 While $iList[0] > 0 AND Ubound($aProcessList) <= $k $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[1] Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE) ; C = Close, K = Keep Open; Returning Process ID Sleep($t) $aProcessList = ProcessList("python.exe") ConsoleWrite("Python is running times = " & Ubound($aProcessList) & @CRLF) Local $return =_ArrayDelete($iList, 1) ConsoleWrite("$return = " & $return & " @error = " & @error & @CRLF) $iList[0] = $return - 1 Wend _ArrayDisplay($iList, "READY NOW") ; this Array is never shown. Why? Sleep($t) $aProcessList = ProcessList("python.exe") Wend This seems to work to a certain degree. I am asking for your help to get it working all the way and simplify the code. Here are the things I wonder: I believe that the inner WHILE-loop is not left at any stage. The Array with the title "READY NOW" is never shown. Why is that so? The last lines of the console: Python is running times = 14 $return = 217 @error = 0 +>10:53:24 AutoIt3.exe ended.rc:0 +>10:53:24 AutoIt3Wrapper Finished. >Exit code: 0 Time: 75.12 Why is the Python running 14 times? It's supposed to be $k = 16 Not all files are processed. The script just ends at the stage shown above. Why aren't all files processed. Thanks in advance for your help. Link to comment Share on other sites More sharing options...
Developers Jos Posted October 20, 2020 Developers Share Posted October 20, 2020 (edited) There is no way we can run this code to reproduce what you are seeing as $iList isn't defined, so what about you post something that does reproduce the issue when run? You do pause 10 seconds in each inner loop, so maybe python processes simply ended in the mean time? No idea why the code suddenly ends without any message/error... hence the need for a reproducer script. Jos Edited October 20, 2020 by Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 20, 2020 Author Share Posted October 20, 2020 5 minutes ago, Jos said: There is no way we can run this code to reproduce what you are seeing as $iList isn't defined, so what about you post something that does reproduce the issue when run? Jos Thank you for your input, Jos. The code creating the $ilList-array: Local $iList = _FileListToArray("C:\Users\Jost\Documents\RnD\Subsea_Images\01SelectionCopy\", "*.jpg", $FLTA_FILES, True) The result looks like this: Quote You do pause 10 seconds in each inner loop, so maybe python processes simply ended in the mean time? Correct. Python processes may last a few seconds up to a few minutes. My intention is to always check how many python.exe-processes are currently running, and launch python until 16 files are processed in parallel, or there are no more files to process. I am aware that this is not a reproducer script. I am trying to come up with something useful. Link to comment Share on other sites More sharing options...
dejhost Posted October 20, 2020 Author Share Posted October 20, 2020 For testing purposes, I set $k = 2. Now, the Array with the title "READY NOW" is appearing. So basically the delays by the sleep()-command are so high, that the processing is faster. Good to know. Still I wonder why not all files are being processed. I created a script to reproduce the problem. Instructions: Create sleep_short.bat and sleep_long.bat on your Desktop. Write "timeout 5" and "timeout 180" in them. Change the path in the following code to match you own desktop. expandcollapse popup#include <Array.au3> ;~CREATE ARRAY ilist Local $iList[101] $iList[0] = 100 Local $i For $i = 1 to 99 STEP 2 $iList[$i] = "C:\Users\Jost\Desktop\sleep_short.bat" Next For $i = 2 to 100 STEP 2 $iList[$i] = "C:\Users\Jost\Desktop\sleep_long.bat" Next _Arraydisplay($iList) Local $k = 5 ; Amount of files processed simultaniously Local $t = 1 * 1000 ; Time to wait inbetween launching cmd Local $aProcessList[1] Local $i = 1 While $iList[0] > 0 While $iList[0] > 0 And UBound($aProcessList) <= $k $CMD = $iList[1] Run(@ComSpec & " /c " & $CMD, "", @SW_SHOW) ; C = Close, K = Keep Open; Returning Process ID Sleep($t) $aProcessList = ProcessList("cmd.exe") ConsoleWrite("CMD.EXE is running times = " & UBound($aProcessList) & @CRLF) Local $return = _ArrayDelete($iList, 1) ConsoleWrite("$return = " & $return & " @error = " & @error & @CRLF) $iList[0] = $return - 1 WEnd _ArrayDisplay($iList, "READY NOW") Sleep($t) $aProcessList = ProcessList("cmd.exe") WEnd This seems to work alright!? Any wild guesses are appreciated! Link to comment Share on other sites More sharing options...
Developers Jos Posted October 20, 2020 Developers Share Posted October 20, 2020 17 minutes ago, dejhost said: This seems to work alright!? Any wild guesses are appreciated! So this reproducer script is working (so not a reproducer) ? ... and your real script is still stopping unexpectedly without any error or strange returncode? The original script doesn't have any exitloop or exit statements in those Close-Loops? Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 20, 2020 Author Share Posted October 20, 2020 Quote So this reproducer script is working (so not a reproducer) ? Correct. The "reproducer" uses the same code (as far as possible) and runs fine. All 100 files are beeing "processed". Quote ... and your real script is still stopping unexpectedly without any error or strange returncode? Yes. In my first post you can see that it starts processing 223 files, and stopped when down to 217. No error-message or so. Right now I got: Python is running times = 7 $return = 179 @error = 0 Python is running times = 8 $return = 178 @error = 0 Python is running times = 9 $return = 177 @error = 0 Python is running times = 10 $return = 176 @error = 0 Python is running times = 11 $return = 175 @error = 0 Python is running times = 12 $return = 174 @error = 0 Python is running times = 11 $return = 173 @error = 0 Python is running times = 12 $return = 172 @error = 0 +>15:29:41 AutoIt3.exe ended.rc:0 +>15:29:42 AutoIt3Wrapper Finished. >Exit code: 0 Time: 524.5 So still having this problem. I inserted "$iList[0]" in the Consolewrite and ran the script again. Python was still running in the background (because of the last execution). The entire output shows: Python is running times = 10 $return = 223 $iList[0] = 223 @error = 0 Python is running times = 10 $return = 222 $iList[0] = 222 @error = 0 Python is running times = 10 $return = 221 $iList[0] = 221 @error = 0 Python is running times = 10 $return = 220 $iList[0] = 220 @error = 0 +>15:39:54 AutoIt3.exe ended.rc:0 +>15:39:54 AutoIt3Wrapper Finished. >Exit code: 0 Time: 42.46 So while "$iList[0]" contains correct values, the script ends very fast. Do you have an idea on how to find out why? Quote The original script doesn't have any exitloop or exit statements in those Close-Loops? Correct. If I am not mistaken, I do not need such statements: The outer WHILE loop re-triggers the inner WHILE loop as long as there are files left to process. The inner WHILE-loop is entered if less than $k = 16 files are currently processed AND there are files left to process. There are certainly more elegant ways to do this (happy for suggestions!), but it seems to work in the reproduction-script!? Link to comment Share on other sites More sharing options...
Developers Jos Posted October 20, 2020 Developers Share Posted October 20, 2020 I probably would have done it with a for next loop to process all entries in an array... something like this: For $x = 1 To $iList[0] ; wait till slot is available Do $aProcessList = ProcessList("python.exe") sleep(1000) Until UBound($aProcessList) <= $k ; run next process $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x] Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE) ; C = Close, K = Keep Open; Returning Process ID ConsoleWrite("Python is running times = " & UBound($aProcessList) & @CRLF) Sleep($t) Next As to your issue with the posted code .... not idea and will need that replicator script to debug. SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 21, 2020 Author Share Posted October 21, 2020 (edited) I testet your code. I just changed the content of the consolewrite. This is what I got in return: $x = 66 $iList[0] = 223 UBound($aProcessList) = 13 @error = 0 $x = 67 $iList[0] = 223 UBound($aProcessList) = 13 @error = 0 $x = 68 $iList[0] = 223 UBound($aProcessList) = 13 @error = 0 $x = 69 $iList[0] = 223 UBound($aProcessList) = 14 @error = 0 $x = 70 $iList[0] = 223 UBound($aProcessList) = 14 @error = 0 $x = 71 $iList[0] = 223 UBound($aProcessList) = 14 @error = 0 +>09:49:35 AutoIt3.exe ended.rc:0 +>09:49:35 AutoIt3Wrapper Finished. >Exit code: 0 Time: 778.2 So basically the same behaviour. So my conclusion is, that the problem is not within the autoit script. Two more thoughts: Maybe the script aborts because of overload of computer resources (CPU/Memory)? But shouldn't there be some message in this case? Anyhow: I will reduce to $k = 11, try again and come back to you. It is also likely that there is an error in the python-script. But if python returns an error and aborts - this shouldn't make autoit-script abort too, right? Edited October 21, 2020 by dejhost Link to comment Share on other sites More sharing options...
JockoDundee Posted October 21, 2020 Share Posted October 21, 2020 35 minutes ago, dejhost said: So basically the same behaviour. Regarding the job that fails, when it does fail, does it fail at exactly the same point and same way? If it doesn’t, it points to some sort of resource/permission issue or race condition. Code hard, but don’t hard code... Link to comment Share on other sites More sharing options...
Developers Jos Posted October 21, 2020 Developers Share Posted October 21, 2020 1 hour ago, dejhost said: So my conclusion is, that the problem is not within the autoit script. Two more thoughts: Maybe the script aborts because of overload of computer resources (CPU/Memory)? But shouldn't there be some message in this case? Anyhow: I will reduce to $k = 11, try again and come back to you. It is also likely that there is an error in the python-script. But if python returns an error and aborts - this shouldn't make autoit-script abort too, right? Neither make sense to me as I would expect and hard crash with some high value RC or in case 2 it would only terminate the python pid, not the AutoIt3 process. Do you have some Anti Virus software acting up and killing the Autoit3 process? SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 21, 2020 Author Share Posted October 21, 2020 24 minutes ago, JockoDundee said: Regarding the job that fails, when it does fail, does it fail at exactly the same point and same way? It does not fail at the same point, but it does fail the same way. Explanation: The python script uses quite some PC-resources during the first few seconds (about 6 seconds, 40% of CPU). It then decides whether a) the current file has not been processed before => let's start processing. This takes a few minutes, but uses only about 4% of PC-resources from now on. b) the current file has been processed before => so it ends the python-program. In this case, AutoIT will initiate the next file. Starting the Autoit-script over and over again yesterday and today, more and more files get processed, and python chooses more and more often b) when starting. So it cannot fail at the exact same point since the same file is not processed twice. However, it fails the same way: $x = 98 $iList[0] = 223 Python is running times = 12 @error = 0 $x = 99 $iList[0] = 223 Python is running times = 12 @error = 0 $x = 100 $iList[0] = 223 Python is running times = 12 @error = 0 $x = 101 $iList[0] = 223 Python is running times = 11 @error = 0 $x = 102 $iList[0] = 223 Python is running times = 12 @error = 0 $x = 103 $iList[0] = 223 Python is running times = 12 @error = 0 +>10:58:35 AutoIt3.exe ended.rc:0 +>10:58:35 AutoIt3Wrapper Finished. >Exit code: 0 Time: 3182 While I receive this message from autoIT, there are still plenty of python-scripts continuing to run for several minutes in the background... The first time I ran the script today, 47 files were created. The second time, 119 files were created. Link to comment Share on other sites More sharing options...
Developers Jos Posted October 21, 2020 Developers Share Posted October 21, 2020 Post the latest version of your script please so I can see whether we can some extra debugging options to it. Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 21, 2020 Author Share Posted October 21, 2020 1 hour ago, Jos said: Post the latest version of your script please so I can see whether we can some extra debugging options to it. Jos It's basically your script. Local $k = 10 ; Amount of files processed simultaniously Local $t = 10 * 1000 ; Time to wait inbetween launching python Local $aProcessList[1] Local $i = 1 For $x = 1 To $iList[0] ; wait till slot is available Do $aProcessList = ProcessList("python.exe") sleep(1000) Until UBound($aProcessList) <= $k ; run next process $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x] Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE) ; C = Close, K = Keep Open; Returning Process ID ConsoleWrite("$x = " & $x & " $iList[0] = " & $iList[0] & " Python is running times = " & UBound($aProcessList) & " @error = " & @error & @CRLF) Sleep($t) Next msgbox(1, "processing finnished", "processing finnished") Ok, here are the good news: It processed all files by now. The script ran all the time through, from 13:45 to 15:22, so roughly 1,5 hours. That's about 20min longer than the previous run. There are 2 things that were different: PC-resources: I reduced $k = 10. Instead of 60%-70% CPU-usage, I get about 50%. So maybe there were peaks in CPU-load or the memory that led to the stop. Windows Defender had something to do with it. I had it deactivated during the last run. For testing purposes, I will turn on the Windows Defender and process the files over again. I will report to you once this is done. If this works fine, I will set $k = 20 and see if the following crash shows the same symptoms as so far: a clean end of the script. Link to comment Share on other sites More sharing options...
Developers Jos Posted October 21, 2020 Developers Share Posted October 21, 2020 1 minute ago, dejhost said: Windows Defender had something to do with it. I had it deactivated during the last run. 99.99999999999999% sure this was your culprit as that is the only way I could explain for processes to suddenly end without any RC other than 0. ...and lets face it: it isn't that common that a process shells soo many other processes in the legitimate world. SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
Developers Jos Posted October 21, 2020 Developers Share Posted October 21, 2020 Something like this would be my next try to get the reason for the unexpected process closure: expandcollapse popupLocal $k = 10 ; Amount of files processed simultaniously Local $t = 10 * 1000 ; Time to wait inbetween launching python Local $aProcessList[1] Local $i = 1 Dim $iList[2] = [1, "test"] ; just added for testing OnAutoItExitRegister("Script_Ended") ConsoleWrite("-processing For-Next items " & $iList[0] & @CRLF) For $x = 1 To $iList[0] ; wait till slot is available Do $aProcessList = ProcessList("python.exe") Sleep(1000) Until UBound($aProcessList) <= $k ; run next process $CMD = " python C:\Users\Jost\Documents\RnD\rePho\filter_kirsch\kirsch_extended.py " & $iList[$x] Run(@ComSpec & " /c " & $CMD, "", @SW_HIDE) ; C = Close, K = Keep Open; Returning Process ID ConsoleWrite("+ $x = " & $x & " $iList[0] = " & $iList[0] & " Python is running times = " & $aProcessList[0][0] & " @error = " & @error & @CRLF) Next ConsoleWrite("-processing For-Next done." & @CRLF) Func Script_Ended() Switch @exitCode Case 0 $sMsg = "normal" Case 1 $sMsg = "by exit" Case 2 $sMsg = "by click" Case 3 $sMsg = "by logoff" Case 4 $sMsg = "windows shutdown" Case Else $sMsg = "unknown?" EndSwitch ConsoleWrite("+processing ending code: " & @exitCode & " - " & $sMsg & @CRLF) EndFunc ;==>Script_Ended SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
dejhost Posted October 22, 2020 Author Share Posted October 22, 2020 I changed my mind - and turned off Windows Defender, while increasing to $k = 12. The script processed all files and ended properly. I will increase $k a bit more and add exclusions for Defender in order to avoid troubles from this end. Some more tests during the next days will hopefully support my current conclusion: problem solved! The "OnAutoItExitRegister("Script_Ended")" - Function is a handy thing. Good to know that something like this exists! Thank you so much for your support! This forums is still one of the amazing places in the internet. Link to comment Share on other sites More sharing options...
JockoDundee Posted October 22, 2020 Share Posted October 22, 2020 Jos declaring 14 9’s of certainty is proof enough for me Code hard, but don’t hard code... Link to comment Share on other sites More sharing options...
dejhost Posted October 22, 2020 Author Share Posted October 22, 2020 1 hour ago, JockoDundee said: Jos declaring 14 9’s of certainty is proof enough for me Agreed 🙂 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now