AutoItIPC UDF for Inter Process Communication

[Kanashius]

This is a UDF (User defined functions) library for AutoIt, providing a simple way to start sub processes and communicate with them.

You can also find the AutoIt-IPC UDF on Github.

The UDF can be used to detect, if the script is started as a sub process by the main process and start the corresponding routines.
The UDF also provides the ability to send commands as well as data between the main and the sub process.
The UDF uses TCP in the background to transfer the data.

The IPC-Example files showcase different ways to use the UDF.
The only required file to be included in the script is the "IPC.au3".

The following functions are available:

; #CURRENT# =====================================================================================================================
; __IPC_StartUp
; __IPC_Shutdown
; __IPC_GetScriptExecutable
; __IPC_StartProcess
; __IPC_SubGetPID
; __IPC_SubCheck
; __IPC_SubConnect
; __IPC_SubSend
; __IPC_SubSendCmd
; __IPC_MainSend
; __IPC_MainSendCmd
; __IPC_ProcessStop
; __IPC_SubProcessing
; __IPC_MainProcessing
; __IPC_Log
; ===============================================================================================================================

Simple example on how to send commands with data between the main and sub process:

expandcollapse popup

#cs ----------------------------------------------------------------------------

     AutoIt Version: 3.3.18.0
     Author:         Kanashius

     Script Function:
        Example script for the IPC InterProcessCommunication UDF.
        This example shows how commands can be sent and received.
        This includes commands with additional data.

#ce ----------------------------------------------------------------------------
#include "IPC.au3"

Global Const $iCOMMAND_START = 1, $iCOMMAND_END = 2, $iCOMMAND_PROGRESS = 3, $iCOMMAND_UNKNOWN = 4

; check if the call is a sub process and start the respective function
__IPC_SubCheck("_SubProcess", "_MainProcess")
If @error Then __IPC_Log($__IPC_LOG_ERROR, "__IPC_SubCheck: "&@error&":"&@extended)

; main/sub process both should call shutdown before exit
__IPC_Shutdown()
Exit
; registered as callback in __IPC_StartProcess to be called when data from the sub process is received
; the main process main method, registered in __IPC_SubCheck to be called when the script is running as main process (no sub process command line arguments detected)
Func _MainProcess()
    ; start a sub process calling the same script.
    ; the _CallbackMain method is called for messages received from the sub process
    ; 100 is the parameter provided to the sub process (total items)
    Local $hProcess = __IPC_StartProcess("_CallbackMain", "11")
    ; wait for the sub process to finish
    While ProcessExists(__IPC_SubGetPID($hProcess)) And Sleep(10)
    WEnd
EndFunc

; registered as callback in __IPC_StartProcess to be called when data from the sub process is received
Func _CallbackMain($hSubProcess, $iCmd, $arData)
    ; $hSubProcess can be used to differentiate between different sub processes (if multiple are started with the same callback method)
    ; $iCmd contains the command send by the server, or Default if only data was sent
    ; $arData contains an array with all the send data or Default if only a command was sent
    Switch $iCmd
        Case $iCOMMAND_START
            ConsoleWrite("Start processing "&$arData[0]&" items"&@crlf)
        Case $iCOMMAND_END
            ConsoleWrite("Finished processing"&@crlf)
        Case $iCOMMAND_PROGRESS
            Local $iTotal = $arData[0]
            Local $iItemsDone = $arData[1]
            Local $iPerc = ($iItemsDone=$iTotal)?100:Mod($iItemsDone, $iTotal)
            ConsoleWrite("Progress: "&$iItemsDone&"/"&$iTotal&" = "&Round($iItemsDone/$iTotal, 2)&" => "&$iPerc&"%"&@crlf)
        Case Default
            ConsoleWrite("Data received"&@crlf)
        Case Else
            ConsoleWrite("Unknown command ["&$iCmd&"] with arData["&UBound($arData)&"] "&@crlf)
    EndSwitch
EndFunc

; the sub process main method, registered in __IPC_SubCheck to be called when the script is running as a sub process
Func _SubProcess($hSubProcess)
    Local $iTotalItems = 10
    If UBound($CmdLine)>1 Then $iTotalItems = Int($CmdLine[1])
    ConsoleWrite("Process ["&$hSubProcess&"]: Start processing items: "&$iTotalItems&@crlf)
    __IPC_SubSendCmd($iCOMMAND_START, $iTotalItems)
    If @error Then __IPC_Log($__IPC_LOG_ERROR, "Failed sending", @error, @extended) ; to check for errors when sending
    For $i=0 to $iTotalItems-1
        __IPC_SubSendCmd($iCOMMAND_PROGRESS, $iTotalItems, $i+1)
    Next
    __IPC_SubSendCmd($iCOMMAND_END) ; to sent only a command, make $data empty. Otherwise, the command will be sent as data
    __IPC_SubSendCmd($iCOMMAND_UNKNOWN, "some command")
    ConsoleWrite("Process ["&$hSubProcess&"]: Finished"&@crlf)
EndFunc

Changelog:

Version 1.0.0
- release of the first version
Version 1.1.0
- seperated __IPC_MainSend into __IPC_MainSend for data and __IPC_MainSendCmd for commands
- seperated __IPC_SubSend into __IPC_SubSend for data and __IPC_SubSendCmd for commands
- when sending data/commands, up to 10 parameters can be added. To send more, put them in other datatypes (e.g. arrays) for sending
- the receiving callbacks now have a $iCmd and $arData parameter with $arData having as many elements as data parameters used with the send methods
- it is now possible to sent any autoit data type directly (They are automatically converted to send over tcp and then converted back)
  (except currently Object (ObjCreate) and DllStructs (DllStructCreate), still looking into that)

I hope you find this UDF useful and please leave a comment, if you have any suggestions to make it better or if you found any bugs.

Thank you to everyone who helps to improve the UDF

Download everything as a zip file: Github AutoIt-IPC download

 

IPC.au3 IPC-Example-Gui.au3 IPC-Example-LargeData.au3 IPC-Example-ManualHandling.au3 IPC-Example-Mixed.au3 IPC-Example-SeparatedMain.au3 IPC-Example-SeparatedSub.au3 IPC-Example-Commands.au3 IPC-Example-Data.au3 IPC-Example-DataStructures.au3

Edited February 14 by Kanashius
v1.1.0

[argumentum]

17 minutes ago, Kanashius said:

Thank you to everyone who helps to improve the UDF

You're welcome 👼
Why not just point the download to GitHub's main.zip ? or post the ZIP instead of all these individual files 🤔

[Kanashius]

6 hours ago, argumentum said:

Why not just point the download to GitHub's main.zip

Hey, good idea, I added the link to the github zip file.
I want the UDF to be downloadable without the examples, so that was the main reason.
With the github zip download, I will leave it like this for now.
Maybe use the download counter as a little indicator, how most probably use the UDF  

[water]

Added your UDF to the wiki  

[WildByDesign]

@Kanashius I'm just getting a chance to really test out your IPC UDF right now.

So from my understanding, we can send 1D arrays. Is there a limit to the size of the 1D array?

[Kanashius]

2 hours ago, WildByDesign said:

So from my understanding, we can send 1D arrays. Is there a limit to the size of the 1D array?

I just put a new version out there, to allow for sending all datatypes (except Object/DllStruct, still have to look into those).
So, with the new version you can send Arrays.

There is no limit from my side, but you can run into limits from AutoIt.
Everything is encoded as Binary with some additional overhead to transmit the dimensions/item count/item type/...
So the limit is probably, how much binary data can be put into one binary variable in AutoIt.

With a quick look: MAX_BINARYSIZE: 2,147,483,647
So: 

For all sent data: DataElementCountType(Int32/Int64) (4 byte) + DataElementCount (4/8 byte) + 
- if sent as command additonal: CommandType(Int32/Int64) (4 byte) + CommandInt (4/8 byte)

For all Arrays: iDataType (4byte) + BinaryLenType(Int32/Int64) (4 byte) + BinaryLen (4/8 byte) + DimensionType(Int32/Int64) (4 byte) + Dimension (4/8 byte)

For 1D: ItemCountType(Int32/Int64) (4 byte) + ItemCount (4/8 byte)
For 2D: ItemCount1DType(Int32/Int64) (4 byte) + ItemCount1D (4/8 byte) + ItemCount2DType(Int32/Int64) (4 byte) + ItemCount2D (4/8 byte)
For 3D: ItemCount1DType(Int32/Int64) (4 byte) + ItemCount1D (4/8 byte) + ItemCount2DType(Int32/Int64) (4 byte) + ItemCount2D (4/8 byte) + ItemCount3DType(Int32/Int64) (4 byte) + ItemCount3D (4/8 byte)
For Every Item: ItemDataType (4 byte) + Item (? bytes)
=> for ?: Item size depends on the datatype (Int32=>4 byte, Int64=>8 byte, Double=>8 byte, Bool=> 1 byte,... with dynamic types like arrays/maps/... having similar calculations, which are done here for arrays)

As an example: For a 1D-Array with 1000 small integers (int32) sent as command (e.g. __IPC_SubSendCmd($iCommand, $arNumbers) 😞
- Sending: 4+4 =8
- Command: 4+4 =8
- All Arrays: 4+4+4+4+4 =20
- 1D-Array: 4+4 =8
- 4*1000+4*1000 =8000
=> 8044 bytes
That is smaller then the MAX_BINARYSIZE, so it will work
With 100 strings and a string length of 200 characters:
- [4 (DataType) + 4 (String length) + 2*200 (string 2 byte per character) ]*100
=> 41644 bytes

I think you will have little problems with these limits

Edited February 13 by Kanashius