ChildProc (Parallel Processing) UDF

[seangriffin]

This UDF is a continuation of the superb work already done by Piccaso in his CoProc UDF.

It provides users with a way to create and communicate with child processes within an AutoIT script. It does not provide the more complex process management functions that the CoProc UDF already has. The emphasis of this UDF is instead on synchronisation and communication between the calling script and it's child processes.

It's got the same core functionality as CoProc UDF, but includes more functions around inter-process communication. Rather than compliment the CoProc UDF I found it easier to just incorporate a subset of it's functionality in this UDF.

REQUIREMENTS:

AutoIt3 3.2 or higher

LIST OF FUNCTIONS:

_ChildProc_Start($sFunction = Default, $vParameter = Default, $max_children = 0)

_ChildProc_WriteToFile($name, $text = Default, $pid = @AutoItPID)

_ChildProc_ReadFromFile($pid, $name, $fOption = 1)

_ChildProc_ReadFromAllFiles()

_ChildProc_WriteToEnvVar($sName, $vValue = Default)

_ChildProc_ReadFromEnvVar($sName, $fOption = 1)

_ChildProc_WriteToRegistry($sName, $vValue = Default, $sRegistryBase = Default)

_ChildProc_ReadFromRegistry($sName, $fOption = 1, $sRegistryBase = Default)

_ChildProc_GetChildCount()

_ChildProc_WriteToCOPYDATA($name, $text = Default, $pid = "")

_ChildProc_ReadFromCOPYDATA($name, $pid = "")

_ChildProc_ReadFromAllCOPYDATA()

_ChildProc_SendCOPYDATA($hWnd, $sData)

_ChildProc_WM_COPYDATA($hWnd, $msgID, $wParam, $lParam)

EXAMPLE:

Example #1 - Parallel Processing using Environment Variables and Files

So far, I believe this example demonstrates the most efficient way to use the UDF, and yields the best performance. It's my recommendation for usage of the UDF.

The following example demonstrates how an AutoIT script can start 10 other child processes (using the ChildProc UDF), pause until the child processes have finished, and then report their output back to the main script.

The user is presented with an option to run a benchmark when the script is run. The benchmark option will report to the user the total time it took the script at the end. Results can be compared with other examples in this post.

The flow of the script is as follows. A loop is performed to run a repetitive task in multiple child processes (_ChildProc_WriteToEnvVar and _ChildProc_Start calls). Variables and values are defined in each _ChildProc_WriteToEnvVar call to be passed into the "MultiProcessFunction". The variable names are all unique, in that each one is prefixed with the number of the current iteration. This ensures that variables from previous calls aren't accidentally overwritten.

The child processes (in parallel within the "MultiProcessFunction" function) consume the variables set by the "main" calling script, using "_ChildProc_ReadFromEnvVar". The last line of the "MultiProcessFunction" function (_ChildProc_WriteToFile) writes the output of the child process to a unique file (named according to it's PID) to be collected later by the main script.

Once the main script completes it's first loop (and all child processes have started) it then goes into a "while" loop, calling "_ChildProc_GetChildCount" periodically until there are no child processes left (ie. they have all ended). This is a synchronisation point to tell the main script to then continue and collect the output from all the children (using "_ChildProc_ReadFromAllFiles").

Note that compiling this example, and running it as an executable, should produce the same result as above.

This example benchmarks at 1056 ms on my dual-core 2.5MHz Thinkpad.

ChildProc_EnvVar_File_test.au3

Example #2 - Parallel Processing using Environment Variables and WM_COPYDATA

The following example demonstrates how an AutoIT script can start 10 other child processes (using the Windows WM_COPYDATA functions), and retrieve the output of each process as it finishes back to the main script.

The user is presented with an option to run a benchmark when the script is run. The benchmark option will report to the user the total time it took the script at the end. Results can be compared with other examples in this post.

The example is quite complex, due to the nature of trying to implement WM_COPYDATA across multiple concurrent processes. WM_COPYDATA is synchronous in nature, much like a pipe attached to a process / window. For multiple processes to communication through it to other processes / windows, the data transfer must be synchronised across all processes, so that no data is lost. The synchronisation process seems to be resource expensive, and slows the script significantly.

Note that there seems to be a synchronisation issue in this example in that many times it "falls out of sync" and leaves some child processes pending forever. If this happens, you'll need to manually kill the AutoIt3.exe processes that have hung, using Windows Task Manager. My apologies. Since this example is much slower than Example #1 above, I'm not in a hurry to fix it (it's not my preferred method of inter-processs communication).

This example benchmarks at 6326 ms on my dual-core 2.5MHz Thinkpad.

ChildProc_COPYDATA_test.au3

Example #3 - Sean's eBay Bargain Hunter

This example I've stored in a topic all it's own. Mainly because I want to version control it as a separate application.

It's a brilliant demonstration of the power of Example #1 above, applied to an actual app. Click here to visit and download the Bargain Hunter.

I have previously used this app. to conduct over 100 parallel searches in eBay, with all processes returning their results in as little as 60 seconds (which I think is pretty cool).

DOWNLOAD:

Latest Version - v0.4 (02/06/10)

ChildProc.au3

Edited June 2, 2010 by seangriffin

[Shafayat]

Nice looking.

But why does the ChildProc.au3 say "ebay.au3" in the header description?

Filename:eBay.au3

[seangriffin]

Nice looking.

But why does the ChildProc.au3 say "ebay.au3" in the header description?

Filename:eBay.au3

I used my eBay UDF as a template for this UDF, and looks like a missed a few updates.

Fixed now in v0.2 above. Thanks for the review!

[Shafayat]

I used my eBay UDF as a template for this UDF, and looks like a missed a few updates.

- I figured that much out.

[seangriffin]

Version 0.3 is now available for download.

Changes include:

• Removed the "wait_for_availability" parameters from all functions (not really required). Performance improvement.
• Added environment variable functions.

This version includes a major performance improvement over 0.2. In 0.2 I used the "wait_for_availability" parameter as a synchronisation method between child and parent processes, but found that performance (the speed) of executing the script was significantly impacted. I've discovered that the parameter within the function "_ChildProc_Start" is actually sufficient enough to ensure variables and values are unique, and I no longer need the synchronisation of "wait_for_availability". The result is the child processes are spawn very fast. No delays.

The introduction and use of environment variables in v0.3 I think is also a slight performance boost over using the registry for variable storage.

Example above has been updated accordingly.

Edited May 29, 2010 by seangriffin

[KaFu]

The introduction and use of environment variables in v0.3 I think is also a slight performance boost over using the registry for variable storage.

"A environment variable set in this way will only be accessible to programs that AutoIt spawns (Run, RunWait). Once AutoIt closes, the variables will cease to exist." I found WM_COPYDATA to be perfectly suited for inter-script communication. Edited May 29, 2010 by KaFu

[seangriffin]

"A environment variable set in this way will only be accessible to programs that AutoIt spawns (Run, RunWait). Once AutoIt closes, the variables will cease to exist." I found WM_COPYDATA to be perfectly suited for inter-script communication.

Gudday KaFu,

I've updated the first post with a new version of the UDF that incorporates WM_COPYDATA. It took me a while to get my head around how it works, and to implement it in a multi-child UDF such as this one, which is why this is such a late reply.

I was quite excited to learn about WM_COPYDATA, and couldn't wait to implement it. Though sadly my results are quite disappointing. I did not get the performance results I was expecting, and the solution I find is actually not as good as my original Environment Variable + File method of communication (example #1 above).

The examples in the top-post have been updated, with a new example for WM_COPYDATA. You might like to review the example and the UDF if you have time, to see if I'm using WM_COPYDATA efficiently. The main concern I have with WM_COPYDATA is that it's like a synchronous pipe. I like to describe it as like a two ended pipe, with one end connected to a window (hwnd), and the other being free for any other process to send data / communicate through. Other examples I've found, such as Yashied's script, in the forums have always been demonstrating the service with two processes. I wonder if I'm the first person in the forums to explore it's use in a multiple child process environment? With two processes, WM_COPYDATA is quite easy to implement. Process 1 and 2 poll for data through their attached pipes. Process 1 grabs the pipe attached to process 2 and sends data to it. Process 2 sees that data and performs a function. Likewise process 2 can send data to process 1 and it performs a function. Nice and simple. With a number of child processes communicating with a parent, things get more difficult. The parent has one pipe connected to it, and a number of children are competing to talk through it to the parent. Synchronisation becomes the key. And once you start synchronising processes, you begin to loose the asynchronous nature of multi-process scripting, bottlenecks occur, reducing the performance of the overall script significantly. This is what I've found first hand. The benchmarks against the examples tell the story.

Something wierd I've found with WM_COPYDATA is that it doesn't seem to be able to communicate with the hwnd of the internal AutoIT window. The internal AutoIT window being that which is accessible via the AutoItWinSetTitle and AutoItWinGetTitle functions. Data just doesn't seem to come out of the other end of the pipe - so to speak. So my UDF gets around this by having each process create it's own additional and uniquely named GUI. WM_COPYDATA works using these GUIs. The need for an additional GUI (above the internal one) for each process is not the preferred solution, and makes me wonder why I'm even using WM_COPYDATA, when I could just interact between processes by sending and receiving data directly via these GUIs themselves! I haven't tried this yet, but I'm considering a test where the parent script creates a GUI, and then a control for every child process it spawns. The children then communicate back to the parent via the control created specifically for that child in the parent's GUI. Whether inter-process communication via GUI controls is efficient is yet to be seen. I might conduct some tests to see if an AutoIT GUI is capable of accepting megabytes worth of data with a GUI control.

Whilst example #1 above I think is blindingly fast (considering what it's doing), I'm also considering the option of implementing a RAM disk to hold the files created in example #1. So instead of writing these temporary files, used for the purposes of IPC, to the hard disk, I could write them to a RAM disk, for extra speed. I love the concept of file handling and environment variables more that pipe-like structures, such as WM_COPYDATA, because they inherently have identifiers built into them (such as filenames, and variable names), and are more-so persistent. Piped data isn't easily identified. I've gotten around this in the UDF by sending a pseudo variable name through WM_COPYDATA prior to the variable's data. But again the synchronisation issue of this kills performance.

What other semi-persistent storage mechanisms does Windows offer, aside from env. variables and files? Writing directly to memory, and retaining that data after a process ends, is a nice thought, but I haven't found a good facility in AutoIT for this. Seems to be a few DLLs around that do it. But I worry that if a child hangs, or the DLL, then the memory won't be freed, and my UDF would leak memory as a result.

Edited June 2, 2010 by seangriffin

[FuryCell]

Very cool. This will come in handy when I run into threading issues which is likely now that I'm using AutoIt more and more for writing actual apps.

[seangriffin]

Very cool. This will come in handy when I run into threading issues which is likely now that I'm using AutoIt more and more for writing actual apps.

It's very cool, I think, but I wrote it

Check out Example #3 I just added to the top post. It's the best demo of the UDF so far.

[KaFu]

You might like to review the example and the UDF if you have time, to see if I'm using WM_COPYDATA efficiently.

seangriffin, I will definitely take an in-depth look at the examples you posted in the coming days (too busy with work atm ), as I'm excited too to learn about the performance advantage of envget.

Something wierd I've found with WM_COPYDATA is that it doesn't seem to be able to communicate with the hwnd of the internal AutoIT window.

I've had that issue too, and like you I decided to go with an extra (hidden) communication child GUI in SMF to sync the main script and the report script. In both scripts I've implemented a sync-state array to detect if sended data has been received and vice versa.

So my UDF gets around this by having each process create it's own additional and uniquely named GUI.

That's also my way to go, where the hidden child GUIs titel is something like this:

Global Const $GUID_Report = 'bd50eeed-19e4-4fda-a32e-cc4290a0b0bf' ; created with http://www.guidgen.com/Index.aspx
guicreate("WMCD_Helper_" & $GUID_Report)

Additionally my child GUIs have some Input controls on them where I post information the receiving script can extract, e.g. the hwnd of a third window is always located in a specific input control:

$hwnd_Report = HWnd(ControlGetText("WMCD_Helper_" & $GUID_Report, "", "[CLASS:Edit; INSTANCE:2]"))

But for sure I'll do some testing with you method and I won't insist on sticking to wm_copydata

Thanks a lot for all your efforts put into this one!

Edit: One thing I forgot to mention. I used WM_COPYDATA because with this you can trigger functions in the target script even while processing in a tight loop. You don't have to specifically poll for data but it's more like a kind of callback. Maybe the best way would be to combine the two methods? Use WM_COPYDATA to tell the target script that it should poll EnvGet?

Edited June 2, 2010 by KaFu

[ParoXsitiC]

I have used CoRoutine for years, and today I found CoProc. I was impressed how he could send data to and from child processes and the code definitely looks complex. However, this is probably the best multi-process UDF I have used simply because its ease of use. It's so mind blowing simple, with the use of making files and creating env vars you'd think there would be performance issues, but no. I guess the "keep it simple" sayings do apply.

CoRoutine accomplishes sending of arrays and complicated data to and from child and parent processes with using the stdin and weird delimits to parse arrays (It can't do objects). Thats the only real advantage it has. I tried to creating a throttling loop for it and it consists of like 1 while loop and 2 for loops, all you have to do here is define a max child for _ChildProc_Start.

The errors I found were to _ChildProcReciver, it was making an error for me. Also try not to use the _ArrayAdd function, so it doesnt require to include array.au3.

I'd like to see some use of a callback function for when child processes die, that way data can be processed as they create & die simultaneous. You could also use this communication from back and forth to create processes that just stay alive and process as it comes (Although this is better suited for CoProc). So instead of eventually running and killing 100 processes, 10 at a time. Just create 10 persistent processes that process data 10 times each.

I have attached a very basic example basically mimicking what you do in ebay.au3:

#include <ChildProc.au3>
#include <Array.au3>
#include <File.au3>
#include <Date.au3>


$Files = _FileListToArray("C:\")
_ArrayDelete($Files, 0)


For $i = 0 to (UBound($Files) - 1)
    _ChildProc_WriteToEnvVar($i & "file", $Files[$i])
    _ChildProc_WriteToEnvVar($i & "timestamp", _Now())
    _ChildProc_Start("Worker", $i, 10)
    ConsoleWrite("Process " & ($i + 2) & " Pid: " & $_ChildProc_pid[($_ChildProc_num_children - 1)] & " Started: " & @HOUR & ":" & @MIN & ":" & @SEC & @CRLF)
Next

Do
    sleep(100)
Until _ChildProc_GetChildCount() <= 0


$Text = _ChildProc_ReadFromAllFiles()
$Array = StringSplit($text, "&#%^#", 1)
_ArrayDelete($Array, 0)

_ArrayDisplay($Files)
_ArrayDisplay($Array)


Func Worker($sMsg)
    $file = _ChildProc_ReadFromEnvVar($sMsg & "file")
    $timestamp = _ChildProc_ReadFromEnvVar($sMsg & "timestamp")
    $final = $file & "      -       The time is " & _Now() & ", yet the varible was made " & $timestamp
    _ChildProc_WriteToFile("", $final & "&#%^#")
EndFunc   ;==>Worker

This will display an array of the contents of your C:\ first...for comparison. Take note of whats there and how many elements, then the next array will display the same thing but all processes via child processes, there is also some timestamp for example

Edited July 14, 2010 by ParoXsitiC

[ParoXsitiC]

OK so the speed isn't quite what it seemed to be when I had 10 going at once, yeah you can always just add more threads to makeup for lost time. However, I just needed to create 1 background thread that simply went and does a BinaryToString & InetRead of an URL and returned the HTML... Without using multi processing, it took on average around 1 seconds...with it...2 seconds. Granted its probably related to file writing and reading and array functions.

Feels like there is room for improvement though.

[Juvigy]

Hi ,

I get an _ChildProcReciver undefined function. Cant find this function in any of the au3 files.

Please help.

[jherazob]

Expanding on what Juvigy said, when running the demo scripts within SciTE4AutoIt3, which calls AU3Check first, it comes up with a dialog box showing a warning about a possibly undeclared variable and an error about an undefined function:

C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc.au3(146,29) : WARNING: $sMsg_Rcvd: possibly used before declaration.
        while StringLen($sMsg_Rcvd)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc.au3(713,54) : ERROR: _ChildProcReciver(): undefined function.
    If Not IsKeyword($vPar) Then _ChildProcReciver($vPar)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc_EnvVar_File_test.au3 - 1 error(s), 1 warning(s)

A cursory look at the UDF shows that the variable is used several times on that block so it may just need a "Local $sMsg_Rcvd" or something like that, but the function is never mentioned anywhere in the script apart from the offending line. Since the code runs perfectly i assume the mentioned function is imported from somewhere in the Windows APIs (haven't really checked out the udf so i dunno) and the variable really is there, but the warning is annoying (the program allows you to "Continue anyway" so you can run it even with the error), and if you do include the udf in your programs this'll get in the way of debugging.

Apart from that, the udf looks like a great job which i'll put to use very soon

[jherazob]

Problems here, reading and writing from env vars doesn't work well, seem to be read-once or something.

Here's an unholy mess of cut-and-paste an example:

expandcollapse popup

#NoTrayIcon
#include <Date.au3>
#include <Array.au3>
#include <ChildProc.au3>

#include <Constants.au3>
#NoTrayIcon

Opt("TrayOnEventMode", 1)
Opt("TrayMenuMode", 1)
Opt("TrayAutoPause", 0)


TraySetOnEvent($TRAY_EVENT_PRIMARYDOUBLE,"TR_Event_Doubleclick")
TrayCreateItem("Exit", -1)
TrayItemSetOnEvent(-1, "TR_Event_Exit")
TraySetState()

_ChildProc_WriteToEnvVar("salir", "No")
_ChildProc_WriteToEnvVar("contador", "1")
$childpid = _ChildProc_Start("monitorprocess")

While 1
    Sleep(100)
WEnd

Func TR_Event_Doubleclick()
    MsgBox(0, "from guiprocess","contador is " & _ChildProc_ReadFromEnvVar("contador",False) & @CRLF & "salir is " & _ChildProc_ReadFromEnvVar("salir", False ) )
EndFunc

Func TR_Event_Exit()
    _ChildProc_WriteToEnvVar("salir", "yes")
    TrayTip("Exiting", "Closing child processes...", -1, 1)
    While _ChildProc_GetChildCount() > 0
        Sleep(250)
    WEnd
    TrayTip("Done","All done",-1,1)
    Sleep(1500)
    Exit
EndFunc

Func monitorprocess()
    $contador = Number(_ChildProc_ReadFromEnvVar("contador",False))
    $salir = _ChildProc_ReadFromEnvVar("salir", False)
    MsgBox(0, "fromchild, firstrun", "here, salir is " & $salir & "and contador is " & String($contador))
    While 1
        $salir = _ChildProc_ReadFromEnvVar("salir", False)
        $contador = $contador + 1
        MsgBox(0, "fromchild,inloop", "here, salir is " & _ChildProc_ReadFromEnvVar("salir", False) & " and contador is " & _ChildProc_ReadFromEnvVar("contador", False))
        If $contador = 30 Then $contador = 1
        _ChildProc_WriteToEnvVar("contador", String($contador))
        If $salir == "yes" Then ExitLoop
        Sleep(2000)
    WEnd
    MsgBox(0, "fromchild,end", "here, salir is " & _ChildProc_ReadFromEnvVar("salir", False) & " and contador is " & _ChildProc_ReadFromEnvVar("contador", False))
EndFunc

Child process never exits because "salir" never changes, and if you doubleclick the tray icon so that it shows the values those vars have on the parent process you see that the constantly changing "contador" stays at 1 (or whatever initial value you write on it on line 32). The values never are exchanged between processes again

I don't really like the idea of a file been written once every 250ms or something, and i loathe the idea of using the registry for process comunication. Will try out copydata next but i don't really like the fact that it seems to be much much slower

Edit: That undeclared variable in the warnings before is inside _ChildProc_WriteToCOPYDATA, and it really seems to be necessary as it errors me out in testing, lines 178 and 239.

Edited September 24, 2010 by jherazob

[tito]

Expanding on what Juvigy said, when running the demo scripts within SciTE4AutoIt3, which calls AU3Check first, it comes up with a dialog box showing a warning about a possibly undeclared variable and an error about an undefined function:

C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc.au3(146,29) : WARNING: $sMsg_Rcvd: possibly used before declaration.
        while StringLen($sMsg_Rcvd)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc.au3(713,54) : ERROR: _ChildProcReciver(): undefined function.
    If Not IsKeyword($vPar) Then _ChildProcReciver($vPar)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\usuario\Mis documentos\Downloads\ChildProc_EnvVar_File_test.au3 - 1 error(s), 1 warning(s)

A cursory look at the UDF shows that the variable is used several times on that block so it may just need a "Local $sMsg_Rcvd" or something like that, but the function is never mentioned anywhere in the script apart from the offending line. Since the code runs perfectly i assume the mentioned function is imported from somewhere in the Windows APIs (haven't really checked out the udf so i dunno) and the variable really is there, but the warning is annoying (the program allows you to "Continue anyway" so you can run it even with the error), and if you do include the udf in your programs this'll get in the way of debugging.

Any solution found ?

[laffo16]

hope am ok to bump this, when running _ChildProc_Start() in a looped fashion (5sec intervals, each process takes aprox 2 secs), the mouse cursor becomes an egg timer for a split second in each loop. is their anyway of stopping windows from running the egg timer once run() is launched. i had a look a GUICtrlSetCursor, Mouse Management, AutoITSetOptions but couldnt see a way of preventing this. Perhaps their is a run() parentheses which can do this or some other workaround? i really just want the multi-processing to work in the background and require no status of its operation. any help apprecaited.