Need help getting two scripts communicating

[DrJeseuss]

First off, my goal is to get two scripts to talk. I'll have a script running that is grabbing data from a com port and logging it. I'll have a second script running to display the data in a GUI. Due to speed, I've made this two scripts so the GUI script doesn't delay the Com grabber and all data will properly logged. The first script is able to log about 30 entries a second, but if I integrate the GUI it drops to only about 10. I know the GUI will likely only update with 10-15 per second, but as long as the first app logs it, that's not a problem.

So, I'd like to get the Com grabber to do it's thing, and should send out it's current data to the GUI app on each cycle. The GUI app will only have time to grab 1/2 to 1/3 of those sends, but must only use data from the most current send. From what I've seen with ConsoleWrite or StdInWrite there's a buffer that would back up if I'm only grabbing every few cycles, or I'll be grabbing several cycles of data at once, most of which will already be outdated.

Any suggestions?

[DrJeseuss]

In experimenting I've been able to get a semifunctional workaround. In the first script a log is created using time as filename (@Hour&@Min&@Sec), then the second script is ran from the first. The second script takes the time (as before) and puts to a variable $Time. The first script gets data from the Com port and writes it to the log file. The second script then opens the log file (using the time variable to know the name used). Using _FileCountLines - 1, I'm able to read the last complete line in the log. This is then read using FileReadLine and updates the GUI. The first major issue is the log naming. I need the log name to include @Hour&@Min&@Sec, but it's possible that the second script will be a second off and therefore attempt to open the wrong filename (thus no file exists). The second issue is speed. I'm getting only about 2 refreshes per second. I know the GUI code can handle at least 10+ refreshes when the data is hardcoded, so I'm led to believe the FileOpen, CountLines, FileRead, FileClose sequence is slowing things down terribly.

I've tried using StdInWrite, but the data goes to the buffer far too fast for the GUI script to keep it clear. The result is a full buffer and a hung Com grabbing script (as warned in the help file). I've also tried ConsoleWrite with similar result.

Does anyone know a fast, reliable way to pass 63 hex bytes from one script to another? The sender should send as often as possible without waiting, and the reciever should get only the most current send if possible. Seems like it should be asy, but I can't seem to think of a way.

[DrJeseuss]

Since no suggestions, I've been working away at an answer. Here's what I came up with. First I create a virtual com port pair. Then using Martin's CommMg Serial UDF I open a com port per script I need to "talk" to. Each open port has it's own channel and thus can be selected using the _CommSwitch function of the UDF. The main script spawns the subscripts using Run commands. Each subscript then opens the other half of the virtual com pair. Then the main script begins dishing out data at a feverish pace. The subscripts have more processing to do and are unable to grab every cycle sent to it. So the main script sends the data, then clears the buffer and sends again. This way the buffer will only ever have my most current data. The reciever only has to grab the buffer when it can to get the most current data. Very fast. Not too pretty. But hey, it works.

[JFee]

Haha. Or you could use TCP stuff (lol I'm writing a program with TCP muttley), but in all honesty it would work.

Have the GUI script listen for TCP messages, then have the logging program send a message to the GUI. Send the data in whatever format works for you, then parse it in the GUI script.

This would also let you remotely watch the the COM port

[martin]

In experimenting I've been able to get a semifunctional workaround. In the first script a log is created using time as filename (@Hour&@Min&@Sec), then the second script is ran from the first. The second script takes the time (as before) and puts to a variable $Time. The first script gets data from the Com port and writes it to the log file. The second script then opens the log file (using the time variable to know the name used). Using _FileCountLines - 1, I'm able to read the last complete line in the log. This is then read using FileReadLine and updates the GUI. The first major issue is the log naming. I need the log name to include @Hour&@Min&@Sec, but it's possible that the second script will be a second off and therefore attempt to open the wrong filename (thus no file exists). The second issue is speed. I'm getting only about 2 refreshes per second. I know the GUI code can handle at least 10+ refreshes when the data is hardcoded, so I'm led to believe the FileOpen, CountLines, FileRead, FileClose sequence is slowing things down terribly.

I've tried using StdInWrite, but the data goes to the buffer far too fast for the GUI script to keep it clear. The result is a full buffer and a hung Com grabbing script (as warned in the help file). I've also tried ConsoleWrite with similar result.

Does anyone know a fast, reliable way to pass 63 hex bytes from one script to another? The sender should send as often as possible without waiting, and the reciever should get only the most current send if possible. Seems like it should be asy, but I can't seem to think of a way.

Sending messages With AutoIt is quite easy. Search for WM_COPYDATA, and have a look here.

The code below is bits from various scripts which work, but in copying and changing them I might have made some mistakes.

The first thing to decide on is the message number to use which should be above $WM_USER.

The easiest way is to let Windows give you a number which isn't already in use.

$MNum = RegisterWindowMessage("SomeUniqueString098761234")

Func RegisterWindowMessage($sText)
    Local $aRet = DllCall('user32.dll', 'int', 'RegisterWindowMessage', 'str', $sText)
    Return $aRet[0]
EndFunc  ;==>RegisterWindowMessage

The window receiving the message needs To register the message like this

GUIRegisterMsg($MNum,"GetINfo")
Func GetInfo($hWnd,$MsgID,$Wparam,$LParam)
;something to do here
    
EndFunc

The process Sending the message can get the same msg number in the same way;

$MNum = RegisterWindowMessage("SomeUniqueString098761234")
;It can Send the message like this
_SendMessage($hWnd,$MNum,1,0x4b21)

The Numbers (1,0x4b21) can be anything you want and mean whatever you want you just write the function In the receiving program to understand them.

You can only Send two 32 bit Integers although this doen't stop you sending more complicated information as I will show later.

You want To Send 63 Hex bytes, so one way would be

For $n = 0 To 16
    _SendMessage($hWnd,$MNum,$n,$dword[$n])
Next
_SendMessage($hWnd,$MNum,17,1);tell program we've done

The receiving program needs to do this

Func GetInfo($hWnd,$MsgID,$Wparam,$LParam)
    Switch $Wparam
        Case 0 To 16
             $Dword[$Wparam] = $LParam
         Case 17
             If $Lparam = 1 Then $Complete = True.
     EndSwitch
    
EndFunc

This is a bit cumbersome but it is quite simple, and it will be quick.

More difficult version

To Send the Information In one hit requires reading memory.

I use NomadMemory.au3

The sending program needs to know the PID of the sending program. The easiest way is with a message

_SendMessage($hWnd,$MNum,1,@AutoItPID)

To send the data we need to tell the rec prog where the data is and how much there is.

I am assuming a String is sent.

;Sending program
Func SendString($sOut)
;put the string in a struct
    Local $tempstruct = DllStructCreate('char[' & StringLen($sOut) + 1 & ']')
    DllStructSetData($tempstruct, 1, $sOut)

;create another struct for the address of the first struct and the length of the string
    Local $MsgStruct = DllStructCreate("int;int")
    DllStructSetData($MsgStruct, 1, DllStructGetPtr($tempstruct))
    DllStructSetData($MsgStruct, 2, StringLen($sOut) + 1)

;get handle of window we are sending to
    Local $hAM = WinGetHandle("The title of the window we are send this to")
    If @error = 1 Then
        Exit
    EndIf
    SendMessage($hAM, $MNum, 2, DllStructGetPtr($MsgStruct))
    
EndFunc

The rec program has this

#include "nomadmemory.au3"
    
$MNum = RegisterWindowMessage("SomeUniqueString098761234")

GUIRegisterMsg($MNum,"GetInfo")



Func GetInfo($hWnd,$MsgID,$Wparam,$LParam)
    Switch $Wparam
        Case 1;we're being told the PID
            $iPID = Number($LParam)
        Case 2;$LParam is the address of the struct
            $hMem = _MemoryOpen ($iPID)
        ;$LParam is the address of the struct
             $DataAdd = _MemoryRead ($LParam,$hMem,"int")
             $DataLen = _MemoryRead (AddrCalc($LParam,4),$hMem,"int")
             $Data = _MemoryRead (AddrCalc($DataAdd,0),$hMem,"char[" & $DataLen & "]")
            ;now $Data is the string sent to us
     EndSwitch
    
EndFunc


Func AddrCalc($StartAddr, $Offset)
    Return '0x' & Hex(Number($StartAddr) + $Offset, 8)
EndFunc  ;==>AddrCalc

[DrJeseuss]

Martin,

I've been working on a few other things that have kept me from this recently, but have pondered your examples. I think the NomadMem seems more straightforward (I think... maybe)... So if I understand it, that requests system memory, then writes the data. If needed the data could be written in the same place overwriting the last write thus keeping the data current without first learing a buffer, etc. Then the other end would simply read from that address location any time it needed to grab a copy of the current data. Do I have the basics of it down? Or am I at least close? Since it's using memory instead of hard drive, I imagine it's much faster. But for speed, would I be faster that way, or using a com port/TCP/WinMsg instead? My priorities are speed, efficiency, ease of use, in that order (though often ease of use get's it's way). I also need to be sure at any given time, I'm grabbing only the most recent frame of data. Thanks again for the help and suggestions!

[martin]

Martin,

I've been working on a few other things that have kept me from this recently, but have pondered your examples. I think the NomadMem seems more straightforward (I think... maybe)... So if I understand it, that requests system memory, then writes the data. If needed the data could be written in the same place overwriting the last write thus keeping the data current without first learing a buffer, etc. Then the other end would simply read from that address location any time it needed to grab a copy of the current data. Do I have the basics of it down? Or am I at least close? Since it's using memory instead of hard drive, I imagine it's much faster. But for speed, would I be faster that way, or using a com port/TCP/WinMsg instead? My priorities are speed, efficiency, ease of use, in that order (though often ease of use get's it's way). I also need to be sure at any given time, I'm grabbing only the most recent frame of data. Thanks again for the help and suggestions!

Yes that's correct, once you know the address you can keep reading it for updates. I do exactly that in my debugger script. I have a struct which consists of 7 integers and 34 strings. Once the address is known the program which creates the struct reads some of the integers to see what information the other program wants and then writes it into the other integers and strings. The other program reads the state of the reply and if there is a reply reads the data. This way the two programs pass 34 strings and 7 integers back and forth every 80ms, but at the same time a third program, the editor, is told the line number of the debugged program being executed and passed any ConsoleWrite outputs to display, and of course the debugged program itself is being run by the program which sets all the data in the struct. So I think speed is not a problem.

[DrJeseuss]

Martin,

I've read this through, looked at NomadMemory, your Debugger, and help files, but I'm strugling to understand what's happening here. As best I can tell, in the first script you've created a struct with data, then another with details of the data (len, etc.). In the second script, is it directly reading the memory where the struct was stored? What is the purpose then of the SendMessage in the first script? I'm not sure I've understood this properly.

Also, If I wanted to have multiple scripts running at the same time to multithread, I assume I could put the send and recieve function into each script allowing them all a two-way communication ability. For example, allocated memory of (for example) 5 command bytes (one per script), and 200 data bytes (shared). Each script could check it's command byte each loop. If 0 do nothing, if 1 do this, 2 do that, etc. This would allow script 3 to request data from scrip 2. Then script 2 could signal script 3 it's available. Then script 1 could force script 5 to update itself, etc. Is this correct that several scripts could share the same space?

[martin]

Martin,

I've read this through, looked at NomadMemory, your Debugger, and help files, but I'm strugling to understand what's happening here. As best I can tell, in the first script you've created a struct with data, then another with details of the data (len, etc.). In the second script, is it directly reading the memory where the struct was stored? What is the purpose then of the SendMessage in the first script? I'm not sure I've understood this properly.

Also, If I wanted to have multiple scripts running at the same time to multithread, I assume I could put the send and recieve function into each script allowing them all a two-way communication ability. For example, allocated memory of (for example) 5 command bytes (one per script), and 200 data bytes (shared). Each script could check it's command byte each loop. If 0 do nothing, if 1 do this, 2 do that, etc. This would allow script 3 to request data from scrip 2. Then script 2 could signal script 3 it's available. Then script 1 could force script 5 to update itself, etc. Is this correct that several scripts could share the same space?

Trying to understand my debugger is not an easy way to understand messages, mainly because of the poor way I write programs, (although I am trying to improve) but also because there are too many other things happening. It does show an example of using a largish struct though.

The main idea is to create a struct to hold the data because that is the only way in AutoIt to know where in memory data is stored. One program creates the struct. This program can write data to the struct with DllStructSetData and read from it with DllStructGetData.

Another program can read from and write to this structure using memory read and memory write functions, but first it has to know the address to use. That is the purpose of the message. The program which creates the struct sends a message to another program to tell it the memory address of the struct.

It is possible to have many programs reading the same memory.

Compile this script below. Then run up to 10 instances of it.(After 10 it fails though I don't know why.) Every time you run it again drag the new window somewhere so you can see all the little windows. All these windows are reading the memory of the first one created. They each have a title which shows the order in which they were created. If you click the Finish button on one of the windows they will all close, with a delay of a few seconds for the first one to close after all the others have closed.

expandcollapse popup

#include <guiconstants.au3>
#include <misc.au3>
Const $TOKEN_ADJUST_PRIVILEGES = 0x0020, $TOKEN_QUERY = 0x0008;, $SE_PRIVILEGE_ENABLED = 0x0002
#include <nomadmemory.au3>
#include <sendmessage.au3>


Opt("GUIOnEventMode",1)
GUICreate("ms_finisher",250,70)
$labmem = GUICtrlCreateLabel("mem",20,20,120,28)
$ButF = GUICtrlCreateButton("Finish",20,40,120,26)
GUISetState()

GUICtrlSetOnEvent($ButF,"finish")
GUISetOnEvent($GUI_EVENT_CLOSE,"done")
Const $Stop_LOOPING = 1
Const $SharedVariable = 1
Global $instance = True, $inst = 0, $highSlave = 0,$htm = "started"
While $Instance = true
    $inst += 1
    _Singleton("mary_script" & $inst,1)
    $instance = @error = 183
WEnd
WinSetTitle("ms_finisher","","ms_finisher" & $inst)

If $inst = 1 Then
    $master = True;this program is the master
    $d = DllStructCreate("int")
    DllStructSetData($d,1,0);0 = not finished
    $Address = DllStructGetPtr($d)
    GUIRegisterMsg(2001,"tellAddr"); respond to requests for the address to use
    GUICtrlSetData($labmem,Hex($Address))
Else
    $master = False;this program is one of the slaves
;get info on master
    Global $pid = WinGetProcess("ms_finisher1"),$hid = WinGetHandle("ms_finisher1")

    Global $MemOpen = _MemoryOpen($pid), $memadd = 0

   $memadd = _SendMessage($hid,2001,$inst);send message to master with our instance so it knows how many of us there are
                                         ;the reply will be the memory address of the data
EndIf

If $master Then
    While 1
        If DllStructGetData($d,1) = $Stop_LOOPING then MasterExit(); we exit program

        If StringInStr($htm,"finish") And DllStructGetData($d,1) <> $Stop_LOOPING then
            DllStructSetData($d,$SharedVariable,$Stop_LOOPING); to inform the other instances of the same script
            MasterExit()
        EndIf
    WEnd
Else
    
    While 1

        If Get($SharedVariable) = $Stop_LOOPING then SlaveExit(); we exit program

        If StringInStr($htm,"finish") And Get($SharedVariable) <> $Stop_LOOPING Then
            Set($SharedVariable,$Stop_LOOPING); to inform the other instances of the same script
            SlaveExit()
        EndIf

    WEnd
EndIf

Func Finish()
    $htm = 'will finish this now'
;ConsoleWrite("finishing" & @CRLF)
EndFunc

Func Get($var)
;only reads one int in this script so $var ignored
    Return _MemoryRead($memadd, $MemOpen, "int")
    
EndFunc

Func Set($var,$val)
;only write one int in this script so $var ignored
    _MemoryWrite($memadd, $MemOpen, $val,"int")
EndFunc

Func Done()
    Exit
EndFunc


;if we are the first inst then this is a message from a slave telling us its number
;so will will send the address as the reply
Func tellAddr($hWndGUI, $MsgID, $WParam, $LParam)
    If $MsgID = 2001 Then
        $Sn = Number($Wparam)
        $titlen = "ms_finisher" & $Sn
        If $sn > $highSlave Then $HighSlave = $Sn
        Return $Address
    EndIf
    
EndFunc



Func SlaveExit()
;MsgBox(0,"SLAVE EXIT",$inst)
    _MemoryClose($MemOpen)
    Exit
    
EndFunc


Func MasterExit()
;wait For all slaves To die
    
    For $nn = 2 To $highSlave
        $Instance = True
        $inst = $nn
        While $Instance = true
            _Singleton("mary_script" & $nn,1)
            $instance = @error = 183
        WEnd
    Next
    $d = 0;kill struct
    Exit
EndFunc

[DrJeseuss]

Martin,

Success! I had a pair of scripts built. The first grabbed data from Com1, logged it, and passed a copy out on Com5(Virtual). The second looked for data in on Com6(Virtual) to display in a ListView window on the GUI. After understanding your examples using Nomad, I've been able to eliminate the virtual com ports completely. Now the first script gets data from Com1, logs, then writes to a struct. The second grabs directly from the memory and displays in the ListView. The refresh rate is now incredible! I knew it should be faster, but it's far better than expected. And with your help, it was surprisingly easy! Now to expand on the concept and get my GUI built. Thanks again Martin. You must eat, sleep, and breath code!

[martin]

Martin,

Success! I had a pair of scripts built. The first grabbed data from Com1, logged it, and passed a copy out on Com5(Virtual). The second looked for data in on Com6(Virtual) to display in a ListView window on the GUI. After understanding your examples using Nomad, I've been able to eliminate the virtual com ports completely. Now the first script gets data from Com1, logs, then writes to a struct. The second grabs directly from the memory and displays in the ListView. The refresh rate is now incredible! I knew it should be faster, but it's far better than expected. And with your help, it was surprisingly easy! Now to expand on the concept and get my GUI built. Thanks again Martin. You must eat, sleep, and breath code!

Well done DRJeseuss, I'm glad you have made it work. Yes, messages are very fast. I tried using ControlSend once for communicating between scripts and then changed to using messages and it from pedal power to rocket power.

[Heron]

Well done DRJeseuss, I'm glad you have made it work. Yes, messages are very fast. I tried using ControlSend once for communicating between scripts and then changed to using messages and it from pedal power to rocket power.

Hi there,

I stumbled upon this thread by searching for multithreads (no pun intended). The reason is I'm exploring the same area of communicating between scripts, mainly to overcome the limitation of AutoIt being single threaded. I know of several options: pipes, environment variables, memory structures, windows messages and even file transfer, but did not experiment with them yet. But I did manage to communicate interscriptical by using straightforward AutoIt. Why not use _GUICtrlEdit_SetText() to write in the (hidden) edit box of a second script? First get the handle with ControlGetHandle() each time you want to write. It's much faster then I had expected. First I used polling to signal new data but this was to slow and gave complications when more senders were writing to the same script. Therefore I learned working with events, which was easier than I feared, for triggering and synchronizing. This way I could sent up to 2000 lines per second to another script! However I still don't know how to process the messages fast enough. When the receiving script is slow then the sending script is still delayed. I guess I have to use queues to cope with this but that's another story.

- Heron -

[Yashied]

Compile and run both scripts.

Sender

#Include <WindowsConstants.au3>

Global Const $RECEIVER_ID = 'My_Receiver'

Global $Count = 0

While 1
    _SendData('0x' & Hex($Count))
    Sleep(10)
    $Count +=1
WEnd

Func _SendData($sData)

    Local $tCOPYDATA = DllStructCreate('dword;dword;ptr')
    Local $tMsg = DllStructCreate('char[' & StringLen($sData) + 1 & ']')

    DllStructSetData($tMsg, 1, $sData)
    DllStructSetData($tCOPYDATA, 2, DllStructGetSize($tMsg))
    DllStructSetData($tCOPYDATA, 3, DllStructGetPtr($tMsg))
    Local $Ret = DllCall('user32.dll', 'lparam', 'SendMessage', 'hwnd', WinGetHandle($RECEIVER_ID), 'int', $WM_COPYDATA, 'wparam', 0, 'lparam', DllStructGetPtr($tCOPYDATA))
    If (@error) Or (Not $Ret[0]) Then
        Return 0
    EndIf
    Return 1
EndFunc   ;==>_SendData

Receiver

expandcollapse popup

#Include <WindowsConstants.au3>

Global Const $RECEIVER_ID = 'My_Receiver'

Global $sMsg, $Int = False, $Pause = False

GUICreate($RECEIVER_ID, 400, 400)
$Label = GUICtrlCreateLabel('', 40, 40, 320, 14)
$Button = GUICtrlCreateButton('Pause', 165, 368, 70, 23)
GUIRegisterMsg($WM_COPYDATA, 'WM_COPYDATA')
GUISetState()

While 1
    Switch GUIGetMsg()
        Case -3
            Exit
        Case $Button
            If Not $Pause Then
                GUICtrlSetData($Button, 'Continue')
            Else
                GUICtrlSetData($Button, 'Pause')
                $Int = 0
            EndIf
            $Pause = Not $Pause
    EndSwitch
    If ($Int) And (Not $Pause) Then
        GUICtrlSetData($Label, $sMsg)
        $Int = 0
    EndIf
WEnd

Func WM_COPYDATA($hWnd, $iMsg, $wParam, $lParam)

    Local $tCOPYDATA, $tData

    If Not $Int Then
        $Int = 1
        $tCOPYDATA = DllStructCreate('dword;dword;ptr', $lParam)
        $tData = DllStructCreate('char[' & DllStructGetData($tCOPYDATA, 2) & ']', DllStructGetData($tCOPYDATA, 3))
        $sMsg = DllStructGetData($tData, 1)
    EndIf
    Return 1
EndFunc   ;==>WM_COPYDATA

[Heron]

Compile and run both scripts.

Hi,

thanks for your code, it is very clean, should be an example in the help file! However it doesn't fit my needs. First it seems limited to max approx 100 messages per second and secondly, it misses massages when the receiver is not fast enough (try a Sleep(1000) in the GUIGetMsg() loop).

But thanks for thinking with me, really appreciated!

- Heron -

[Yashied]

You can send messages in two directions. Script1 sends data to Script2 and waits for a response to continue. In turn, Script2 processes the data and sends the message "CONTINUE" in Script1 etc. 100% synchronization. Something like I did in my 3D Axis. Progress for drawing graphs are exactly on this principle.

If you make a queue, it will very quickly overflow.

Edited December 25, 2009 by Yashied