Jump to content
Chimp

Multi-Task (easily run and mange many processes)

Recommended Posts

Chimp

This is a simple and practical way to start and manage many processes. I use it to quickly query many remote clients for varyous informations. For processes I mean external programs that can run autonomously and that allows I/O redirection.
Here the advantage is that when you start a process, you can also determine which function in your main program should receive results when the external process finishes executing. So you can have different processes and different "callback" functions. The whole works nearly autonomously and asynchronously.
While spawned processes works, you can perform other tasks in the main script and leave the worry of receiving and managing the results, to the pre-established functions.
As I sayd, everything is almost completely automatic and the processes do their work asynchronously and independently of the flow of the main script.
To check and manage processes status, you have only  to call as often as possible the _TasksCheckStatus() function  that will quickly handle the running processes. AdlibRegister() could be used for this ripetitive "polling".
In short:
1) setup single commandlines that will "generate" your wanted result
2) establish wich function of your main script should receive the above result
3) an optional Timeout in seconds is allowed (task from point 1 will be killed at timeout)
see function header for details.

The provided example of use, needs to be executed in a network environment with preferably many clients, and you have to be administrator. Populate  the $aIPList[] array with ClientNames (or ip addresses), and this example script will query all the clients about some informations by running many processes asincronously. Then it will collect results from the processes as soon as "answers" are available and will populate the ListView with that data. I used a ListView so that you can see that all the cells will fill (quite quickly) in an asynchronous way. If you are not in a network with many clients, then this example is nearly useless... (sorry)

Hope it can be of use... (suggestions, bug reports and emrovements are welcome)

MultiTask.udf

#include-once
; #include "Array.au3"

; #INDEX# =======================================================================================================================
; Title .........: Multi Task Management (tasks spawned by the user via the _TaskRun function of this udf)
; AutoIt Version :
; Language ......:
; Description ...: Functions that assists to execute and manage user's spawned tasks.
; Author(s) .....: Chimp
; ===============================================================================================================================

; #CURRENT# =====================================================================================================================
; _TaskRun              Run a new task and stacks its references for later use
; _TasksCheckStatus     Check the status of all running tasks and call the CallBack function when a task completed
; ===============================================================================================================================

; #INTERNAL_USE_ONLY#============================================================================================================
; __TaskRemove          frees task's stacked references
; __NOP                 No OPeration function
; ===============================================================================================================================
Global $aTasks[1][7] = [[0]] ; Create and initialize the task reference stack to 0

; #FUNCTION# ====================================================================================================================
; Name ..........: _TaskRun
; Description ...: Runs a new task as specified in $sCommand
; Syntax ........: _TaskRun($vIndex, $sCommand[, $sParser = "__NOP"[, $vTimeout = False]])
; Parameters ....: $vIndex              - A variant value passed by the caller as a reference to this task.
;                                         this value will be returned to the callback function along with results.
;
;                  $sCommand            - A string value containing the full command to be executed
;
;                  $sParser             - [optional] The name of the function to be called at the end of this task.
;                                         Default is __NOP that is a "do nothing" function.
;                                         P.S.
;                                         the $aParser function will be called by the _TasksCheckStatus() function when
;                                         this task will finish to run:
;                                         an 1D array with 4 elements will be passed to the called function:
;                                         element [0]   the caller's index reference of this task
;                                         element [1]   the StdOut result of this task
;                                         element [2]   the StdErr result of this task
;                                         element [3]   the time spent by this task
;
;                  $vTimeout            - [optional] A variant value. Default is False.
;                                         if specified is the number of seconds for the Timeout.
;                                         After this amount of seconds, if this task is still running, it is killed
;
; Return values .: the total number of stacked tasks
; Author ........: Chimp
; Modified ......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......:
; ===============================================================================================================================
Func _TaskRun($vIndex, $sCommand, $sParser = "__NOP", $vTimeout = False)
    ;
    ; Stack structure:
    ;
    ;   +-----+
    ; 0 | nr. | <-- [0][0] number of running tasks
    ;   +-----+-----+-----+-----+-----+-----+-----+
    ; 1 |  0  |  1  |  2  |  3  |  4  |  5  |  6  |
    ;   +-----+-----+-----+-----+-----+-----+-----+
    ; n |  *  |  *  |  *  |  *  |  *  |  *  |  *  |
    ;      |     |     |     |     |     |     |
    ;      |     |     |     |     |     |   [n][6] setted with the TimerInit() value at the start of this task.
    ;      |     |     |     |     |     |
    ;      |     |     |     |     |   [n][5] contains the required Timeout in seconds (Default is set to False = no timeout)
    ;      |     |     |     |     |
    ;      |     |     |     |   [n][4] The CallBack function name that will receive the results of this task
    ;      |     |     |     |
    ;      |     |     |   [n][3] the error message returned by the StdErr stream of this task
    ;      |     |     |
    ;      |     |   [n][2] the result of the command returned by the StdOut stream of this task
    ;      |     |
    ;      |   [n][1] the reference of this task passed by the user (will be passed back to the caller along with results)
    ;      |
    ;    [n][0] the PID of this running task
    ;
    ReDim $aTasks[$aTasks[0][0] + 2][7] ; add a new element to the stack
    $aTasks[0][0] += 1 ; add 1 to the number of running tasks

    ; Run the passed command with the I/O streams redirected
    $aTasks[$aTasks[0][0]][0] = Run($sCommand, "", @SW_HIDE, 6) ; 6 -> $STDOUT_CHILD (0x2) +  $STDERR_CHILD (0x4)

    ; store references of this task to the stack
    $aTasks[$aTasks[0][0]][1] = $vIndex
    $aTasks[$aTasks[0][0]][4] = $sParser

    $aTasks[$aTasks[0][0]][5] = $vTimeout ; can be False or the number of seconds for the timeout of this task
    $aTasks[$aTasks[0][0]][6] = TimerInit() ; store the TimerInit() value for this task

    Return $aTasks[0][0] ; return the total number of running tasks
EndFunc   ;==>_TaskRun

; #FUNCTION# ====================================================================================================================
; Name ..........: _TasksCheckStatus
; Description ...: Scans the status of all active tasks and checks if some task has finished its job
;                  This function should be called as aften as possible
; Syntax ........: _TaskCheckStatus()
; Parameters ....: None
; Return values .: number of still running tasks;  * see remarks
; Author ........: Chimp
; Modified ......:
; Remarks .......: When a task finish, it is removed from the stack and results are passed to the callback function.
;                  An 1D array with 4 elements will be passed to the called function:
;                  element [0]  the caller's index reference of this task
;                  element [1]  the StdOut result of this task
;                  element [2]  the StdErr result of this task
;                  element [3]  the time spent by this task (approximative time)
;                               is an approximate datum because it also includes the delay
;                               added by the main loop before calling _TasksCheckStatus()
; Related .......:
; Link ..........:
; Example .......:
; ===============================================================================================================================
Func _TasksCheckStatus()

    If $aTasks[0][0] = 0 Then Return 0 ; if no tasks then return
    Local $bEndTask ; will be setted to True if the checked task has finished or is killed by timeout
    Local $iPointer = 1 ; start checking from first task in the stack

    Local $aResults[4] ; this Array will be loaded with the results of the task
    Local $sCallBack ; the name of the function to call when a task ends its job
    Local $aArgs[2] = ["CallArgArray", ""] ; "special" array will be loaded with parameters for the CallBack function

    While $iPointer <= $aTasks[0][0]

        $bEndTask = False

        $aTasks[$iPointer][2] &= StdoutRead($aTasks[$iPointer][0]) ; read and store the StdOut stream
        $aTasks[$iPointer][3] &= StderrRead($aTasks[$iPointer][0]) ; read and store the StdErr stream

        ; If @error Then ; if there is an @error is because this task has finished its job
        If Not ProcessExists($aTasks[$iPointer][0]) Then ; if this task has finished its job
            $bEndTask = True ; flag the end of the work

        ElseIf $aTasks[$iPointer][5] <> False Then ; if task is still running see if a Timeout check was rquired and if so if is been reached
            If Int(TimerDiff($aTasks[$iPointer][6]) / 1000) >= $aTasks[$iPointer][5] Then ; timeout reached or exceeded!!
                $aTasks[$iPointer][3] = "@error Timeout " & $aTasks[$iPointer][3] ; insert an error message at the beginning of the StdErr stream
                StdioClose($aTasks[$iPointer][0]) ; close I/O streams
                ProcessClose($aTasks[$iPointer][0]) ; kill this process
                $bEndTask = True ; flag the end of this task
            EndIf
        EndIf

        If $bEndTask Then ; if this task has finished, get its results and send to the CallBack function

            $aResults[0] = $aTasks[$iPointer][1] ; ............. Index (the caller reference)
            $aResults[1] = $aTasks[$iPointer][2] ; ............. the StdOut generated by this task
            $aResults[2] = $aTasks[$iPointer][3] ; ............. the StdErr generated by this task
            $aResults[3] = TimerDiff($aTasks[$iPointer][6]) ; .. time spent by this task

            $sCallBack = $aTasks[$iPointer][4] ; the name of the function to be called

            $aArgs[1] = $aResults ; second element of the "CallArgArray" special array contains the $aResults array
            ;                       loaded with the parameters to be send to the CallBack function. (array in array)
            ;                       (the CallBack function will receive only the 1D 4 elements array $aResults)

            __TaskRemove($iPointer) ; remove references of this task from the stack

            ; call the CallBack function and pass the results of this task.
            ; (CallBack function should return as soon as possible because it stops the CheckStatus for the other tasks)
            ;
            Call($sCallBack, $aArgs) ; Call CallBack function  --->---+
            ;                                                         |
            ;  <--- and return here  ---------------------------------+
        EndIf

        $iPointer += 1 ; check next task
    WEnd
    Return $aTasks[0][0]
EndFunc   ;==>_TasksCheckStatus

; Internal use. Remove the task references from the stack
Func __TaskRemove($iElement)
    #cs
        If $iElement > $aTasks[0][0] Or $iElement < 1 Then Return
        StdioClose($aTasks[$iElement][0])
        _ArrayDelete($aTasks, $iElement)
    #ce
    ; - new --------------------------------------------------
    ; remove element without the _Array* udf
    If $iElement > 0 And $iElement <= $aTasks[0][0] Then
        StdioClose($aTasks[$iElement][0])
        If $aTasks[0][0] > 1 Then
            For $i = 0 To UBound($aTasks, 2) - 1
                $aTasks[$iElement][$i] = $aTasks[$aTasks[0][0]][$i]
            Next
        EndIf
    Else
        Return ; queue is empty or the required element is out of bound
    EndIf
    $aTasks[0][0] -= 1
    ReDim $aTasks[$aTasks[0][0] + 1][UBound($aTasks, 2)]
    Return $aTasks[0][0] ; returns the number of tasks still running
EndFunc   ;==>__TaskRemove

; Internal use. An empty function
Func __NOP($aDummy)
    ; NOP (No OPeration)
EndFunc   ;==>__NOP

 

Example of use

; #RequireAdmin
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GuiListView.au3>
#include '.\MultiTask.au3'
;
; IMPORTANT following array should be populated with many real HostNames
; here are just nonsense items used as placeholders
Global $aIPList[] = [@ComputerName, @IPAddress1, @ComputerName, @ComputerName, 'InexistentClient', @ComputerName]

Global $hGrid ; The ListView Handle
Global $ahGrid[1] ; An array to keep handles of any listview row

Example()
MsgBox(0, "Debug:", "Done" & @CRLF & "Hit OK to exit")

Func Example()
    Local $Form1 = GUICreate("Clients status", 760, 400)
    ;
    ; Create the ListView
    $hGrid = GUICtrlCreateListView("HostName|IP|Info|Last reboot|CPU|Last logon|Current logon", 0, 0, 760, 400)
    _GUICtrlListView_SetColumnWidth($hGrid, 0, 140) ; HostName
    _GUICtrlListView_SetColumnWidth($hGrid, 1, 100) ; IP
    _GUICtrlListView_SetColumnWidth($hGrid, 2, 80) ; Ping info ms or Off or Unknown or Timeout)
    _GUICtrlListView_SetColumnWidth($hGrid, 3, 120) ; Last Reboot
    _GUICtrlListView_SetColumnWidth($hGrid, 4, 40) ; cpu load
    ; last 2 columns a mutually exclusive. If there is a user logged it's shown on the last column
    ; if there is not a user logged then the last user that was logged  is shown on column 4 instead
    _GUICtrlListView_SetColumnWidth($hGrid, 5, 140) ; Last logged UserId (if nobody is logged now)
    _GUICtrlListView_SetColumnWidth($hGrid, 6, 140) ; Currently logged UserId
    _GUICtrlListView_SetExtendedListViewStyle($hGrid, BitOR($LVS_EX_GRIDLINES, $LVS_EX_FULLROWSELECT)) ; show grid; select whole row
    ;
    GUISetState(@SW_SHOW)

    ; following line is needed if you have to refill the listview
    ; _GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($hGrid)) ; empty the listview

    ReDim $ahGrid[UBound($aIPList)]

    ; this loop will run all needed tasks at once.
    ; The results of the tasks will be managed by the callback functions (nearly) autonomously and asynchronously
    For $i = 0 To UBound($aIPList) - 1
        $ahGrid[$i] = _GUICtrlListView_AddItem($hGrid, "") ; create a listview row
        ;
        ; ... for each client ...
        ;
        ; spawn ping commands. result will be send to the _PingParse() function
        _TaskRun($i, "Ping -a -n 1 -4 " & $aIPList[$i], "_PingParse")

        ; spawn commands to recall LoggedOn User. Result will be send to the _LoggedOn() function
        _TaskRun($i, "wmic /node:" & $aIPList[$i] & " computersystem get username /value", "_LoggedOn", 5) ; 5 = timeout in 5 seconds

        ; spawn commands to recall Last reboot time. result will be send to the _LastReboot() function
        _TaskRun($i, "wmic /node:" & $aIPList[$i] & " os get lastbootuptime /value", "_LastReboot", 5)

        ; spawn commands to recall % of CPU load. result will be send to the _CPU_load() function
        _TaskRun($i, "wmic /node:" & $aIPList[$i] & " cpu get loadpercentage /value", "_CPU_load", 7)
    Next

    ; now, while all tasks are running
    ; we could perform other activities in the meantime
    ; or we can wait the end of all the tasks
    Do
        Sleep(250)
        ; you could perform other jobs here while waiting for the tasks to finish
    Until Not _TasksCheckStatus() ; <-- this function performs management of running tasks
    ;                                   and should be called as often as possible

EndFunc   ;==>Example

; below are CallBack functions

; #FUNCTION# ====================================================================================================================
; Name ..........: _PingParse (a callback function)
; Description ...: this function analize the output of a ping command and "extract" needed infos
;                  it fills columns 0 1 and 2 of the list view ($aTarget[0] is the line number of the listview to be filled)
; Syntax ........: _PingParse($aTarget[, $bDomain = True])
; Parameters ....: $sTarget             - An array with Ping results passed by the MultiTasks as soon as any ping task ends
;                                         the passed array contains following data:
;                                         $aTarget[0] Index for this task
;                                         $aTarget[1] StdOut from ping (the whole output of the ping)
;                                         $aTarget[2] StdErr from ping
;                                         $aTarget[3] Time spent by this task to complete (is NOT the ping roundtrip time)
;                  $bDomain             - [optional] A binary value. Default is True. (keep domain info in host name)
;
; Return values .: None. It Fills Listview columns 0, 1 and 2
;                                         column 0 : resolved HostName or ""
;                                         column 1 : IP address or "" (this can contain last known IP even if now is offline)
;                                         column 2 : roundtrip time or "Unknown" or "timeout" or "Off"
;
; Author ........: Chimp
; ===============================================================================================================================
Func _PingParse($aTarget, $bDomain = True)
    ; $aTarget contains 4 elements: [0] Index, [1] StdOut, [2] StdErr, [3] time spent by this task

    Local $sOutput = $aTarget[1] ; stdout
    Local $0, $1, $2, $3, $aMs
    ; Local $iAnswer = -1, $iName = -1
    Local $aResult[3] = ["", "", ""] ; [0]ms, [1]HostName, [2]IP

    $aMs = StringRegExp($sOutput, "([0-9]*)ms", 3)
    If Not @error Then ; Ping replayed
        $aResult[0] = $aMs[UBound($aMs) - 1] ; average ms
    Else
        ; $aResult[0] = "off"
    EndIf

    $0 = StringInStr($sOutput, "Ping")
    $1 = StringInStr($sOutput, "[") ; HostName decoded?

    If $1 Then ; HostName decoded
        $2 = StringInStr($sOutput, "]")
        $3 = StringInStr($sOutput, " ", 0, -2, $1)
        $aResult[1] = StringMid($sOutput, $3 + 1, $1 - $3 - 1) ; HostName
        $aResult[2] = StringMid($sOutput, $1 + 1, $2 - $1 - 1) ; IP
    Else

        If $0 Then ; pinging an IP address?
            ; $aResult[1] = "" ; no HostName
            Local $aFindIP = StringRegExp($sOutput, "\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", 3)
            If Not @error Then $aResult[2] = $aFindIP[0]
        Else
            ; unknown HostName
            $aResult[0] = "Unknown"
            $aResult[1] = $aIPList[$aTarget[0]] ; retrieve HostName from the $aIPList array

        EndIf
    EndIf

    If $bDomain = False Then
        Local $aSplit = StringSplit($aResult[1], ".", 2) ; 2 = $STR_NOCOUNT
        $aResult[1] = $aSplit[0] ; romove .domain part from the HostName
    EndIf

    If StringLeft($aTarget[2], 14) = "@error Timeout" Then $aResult[0] = "Timeout"

    ; Now that we have the infos, we compile related cells in ListView
    ;                            grid    row-handle            data         column
    _GUICtrlListView_SetItemText($hGrid, $ahGrid[$aTarget[0]], $aResult[1], 0) ;    first column "HostName"
    _GUICtrlListView_SetItemText($hGrid, $ahGrid[$aTarget[0]], $aResult[2], 1) ;    second column "IP address"
    _GUICtrlListView_SetItemText($hGrid, $ahGrid[$aTarget[0]], $aResult[0], 2) ;    third column "Infos about the ping"

    ; ConsoleWrite("Debug: " & "-> " & $aResult[0] & @TAB & $aResult[1] & @TAB & $aResult[2] & @CRLF)
EndFunc   ;==>_PingParse

Func _LastReboot($aParameters) ; Last reboot DateTime
    $aParameters[1] = StringStripWS($aParameters[1], 8)
    Local $equal = StringInStr($aParameters[1], "=")
    If $equal Then
        _GUICtrlListView_AddSubItem($hGrid, $ahGrid[$aParameters[0]], WMIDateStringToDate(StringMid($aParameters[1], $equal + 1)), 3) ; column 3
    EndIf
EndFunc   ;==>_LastReboot

Func _CPU_load($aParameters) ; % of CPU load
    $aParameters[1] = StringStripWS($aParameters[1], 8)
    Local $equal = StringInStr($aParameters[1], "=")
    If $equal Then
        _GUICtrlListView_AddSubItem($hGrid, $ahGrid[$aParameters[0]], StringMid($aParameters[1], $equal + 1), 4) ; column 4
    EndIf
EndFunc   ;==>_CPU_load

Func _LoggedOn($aParameters) ; User now logged
    $aParameters[1] = StringStripWS($aParameters[1], 8)
    ; if none is logged, then find the user that was last logged (also using _TaskRun)
    If $aParameters[1] = "" Or $aParameters[1] = "UserName=" Then
        ; following syntax is by @iamtheky (thanks)
        ; https://www.autoitscript.com/forum/topic/189845-regexp-pattern-in-findstr-dos-command/?do=findComment&comment=1363106
        Local $sCmd = 'cmd /c FOR /F "usebackq skip=2 tokens=1-3" %A IN (`REG QUERY ' & '"\\' & $aIPList[$aParameters[0]] & _
                '\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI" /REG:64 /v LastLoggedOnUser 2^>nul`) Do @echo %C'
        _TaskRun($aParameters[0], $sCmd, "_LastLogged", 5) ; find last logged user and, when ready, send result to _LastLogged()
    Else
        ; if someone is logged then write username to column 6
        Local $aUser = StringSplit($aParameters[1], "=", 2) ; 2 = $STR_NOCOUNT
        _GUICtrlListView_AddSubItem($hGrid, $ahGrid[$aParameters[0]], $aUser[UBound($aUser) - 1], 6) ; column 6
    EndIf
EndFunc   ;==>_LoggedOn

Func _LastLogged($aParameters)
    _GUICtrlListView_AddSubItem($hGrid, $ahGrid[$aParameters[0]], $aParameters[1], 5) ; column 5
EndFunc   ;==>_LastLogged


Func WMIDateStringToDate($dtmDate) ; thanks to @kylomas
    ; https://www.autoitscript.com/forum/topic/169252-wmi-password-age-issue/?do=findComment&comment=1236082
    ; reformat date to mm/dd/yyyy hh:mm:ss and zero fill single digit values
    Return StringRegExpReplace(StringRegExpReplace($dtmDate, '(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2}).*', '$2/$3/$1 $4:$5:$6'), '(?<!\d)(\d/)', '0$1')
EndFunc   ;==>WMIDateStringToDate

 

Edited by Chimp
changed topic tile (added "run"
  • Like 4

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites
RooperGee
Posted (edited)

Loving this code you came up with!  Have you made any mods to it lately?  I'll be testing it out over the next week and provide feedback.  Thanks!

Edited by RooperGee

Progress lies not in enhancing what is, but in advancing towards what will be. - Kahlil Gibran

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×