_MultiProcess UDF : run multiple commands at the same time

[jguinch 393]

What is MutiProcess UDF ?
MutiProcess allows you to run several external programs at the same time and retrieve the exit code of each process. Originally, I created this UDF for my own need, to speed up the execution of many programs.

How does it work ?
MutiProcess works with jobs and tasks. A job is a list of tasks to execute. A task is the execution of an external program (like you can do with Run() )
When you launch a job, MutiProcess executes a specified number of tasks simultaneously. Each time a task finishes to execute, a new task is executed, in order to keep the maximum number of tasks simultaneously.
Example, you want to launch 50 instances of notepad.exe (odd, isn't it?) with a maximum of 10 instances simultaneously. MutiProcess starts the first 10 processes, then it waits for one process to be finished. Since a processes is finished, MutiProcess launchs the next waiting task and so on, until the end of the tasks list.
You can launch several jobs at the same time, each one with a lot of tasks if you want. Each job and task is identified by an uniq identifier (ID).

How can I use it ?
First, you have to create a MultiProcess job, using _MultiProcess_Create. It returns the ID of the job.
Next, you have to add the tasks you want to execute in your job (for example : notepad.exe, ping, ...some external command), using _MultiProcess_AddTask. It returns the ID of the task.
After that, you can launch the job using _MultiProcess_Run
You can wait the end of execution of a job by using  _MultiProcess_WaitClose
You can cancel the execution of a job using _MultiProcess_Close, even if some tasks are in execution (it's possible to force the process to close)
You can use _MultiProcess_SetOnEvent to call you own functions each time a job or task changes of state (start running, finished...). It's a good way to get the progression of the job.
After all, you can get the result of each task, using _MultiProcess_GetResult. The result of a task is the exit code of its process.

Example
Here I want to process a "ping" command on 50 computers in my network. I could use the AutoIt Ping() function, but I use MultiProcess to launch 5 "ping" commands at the same time, to increase the time of execution .
The external command that I want to launch is cmd /c ping 192.168.xxx.xxx -n 1 | find "TTL" : this command returns 0 on success and 1 on error.
In the example, I give 2 methods to do the same thing : the first one with MultiProcess , the second with the AutoIt Ping() function. It's just to show you the difference of time execution.

expandcollapse popup

; EXAMPLE 1 

#Include <Array.au3> ; Just used for _ArrayDisplay

#Include "multiProcess.au3"

; Build an array of hosts to Ping (the 2nd element of the array will contain the result (online/offline)
Local $sSubNet = "192.168.1."
Local $aHosts[50][2]
For $i = 0 To UBound($aHosts) - 1
    $aHosts[$i][0] = $sSubNet & ($i + 1)
Next


; MultiProcess method ================================================================================
ConsoleWrite("Ping " & UBound($aHosts) & " hosts with MultiProcess method, please wait..." & @CRLF)
Local $hTimer = TimerInit()
; Create a Multiprocess job
_MultiProcess_Create()

For  $i = 0 To UBound($aHosts) - 1
;~  ; Add a task, (external command) - returns an ID
    $aHosts[$i][1] = _MultiProcess_AddTask(@ComSpec & ' /c ping ' & $aHosts[$i][0] & ' -n 1 | find "TTL"', @SystemDir, @SW_HIDE)
Next

; Launch the MultiProcess job, with a max of 5 simultaneous processes
_MultiProcess_Run(5)
; Wait until the job finishes to execute
_MultiProcess_WaitClose()

; Get the results of each task
;~ Local $aResult = _MultiProcess_GetResult()
For $i = 0 To UBound($aHosts) - 1
    $iExitCode = _MultiProcess_GetResult($aHosts[$i][1]) ; Retrieve to exit code of the process
    $aHosts[$i][1] = $iExitCode ? "------" : "Online"    ; The command returns 0 on sucess
Next
ConsoleWrite("Elapsed time (in ms) : " & TimerDiff($hTimer) & @CRLF)
_ArrayDisplay($aHosts)
; ====================================================================================================


; Classic method =====================================================================================
; Now, Classic method, using the Ping() function
ConsoleWrite(@CRLF & "Ping " & UBound($aHosts) & " hosts with AutoIt Ping method, please wait..." & @CRLF)
$hTimer = TimerInit()
For  $i = 0 To UBound($aHosts) - 1
    $iRes = Ping($aHosts[$i][0]) ? "Online" : "------"
Next
ConsoleWrite("Elapsed time (in ms) : " & TimerDiff($hTimer) & @CRLF)
_ArrayDisplay($aHosts)
; ====================================================================================================

I don't know if someone will by interested by the UDF, so if needed, I will provide more examples.

Download link : multiProcess.au3

Edited February 1, 2017 by jguinch

[spudw2k 189]

Looks neat and useful.  I was a bit confused as to what Muti meant, but I see the UDF and functions are Multi.

[nhardel 0]

Any ideas how you might pull off a job as an admin user without running the whole script as that user.  Previously I used runas but, that isn't really an option here.  Ideas or possible a new feature.

[gil900 86]

Cool, I just did such a thing to my needs. I thought exactly the same idea

Edited March 14, 2017 by gil900