Websockets and AutoIt

[Danp2]

@TheXmanI had not noticed the added log. Thanks for pointing it out. Can you share more details on how you were processing multiple responses from the webdriver server? I previously was using an EOL marker to separate requests, but it doesn't appear that the tool from eSeGeCe includes one in its output.

[TheXman]

1 hour ago, Danp2 said:

Can you share more details on how you were processing multiple responses from the webdriver server?

I assume that you meant responses from the websocket server because I haven't been using webdriver.

Of course details of how one would implement the processing of multiple responses, from a websocket server subscription, depends on numerous variables like:

• On average and at peak times, how often may the responses be coming?
• How big are the responses?
• In what form are the responses sent (clear text, base64, binary)?
• How much post-receipt processing of the response is required?
• Etc...

Are you asking for more details as to how I was able to differentiate and enqueue multiple responses that were received at once or are you asking for more details regarding how I actually processed the multiple responses after they were received and enqueued?  The former is a question related to how I polled and process data from the websocket client's TCP socket.  The latter is more related to how I processed the JSON event messages and turned the JSON data into information.  Those 2 topics are related in some ways and different in others.  I don't want to drill down on the wrong topic.

Edited November 12, 2022 by TheXman

[Danp2]

22 minutes ago, TheXman said:

I assume that you meant responses from the websocket server because I haven't been using webdriver.

Yes... freudian slip. 😆

24 minutes ago, TheXman said:

The former is a question related to how I polled and process data from the websocket client's TCP socket. 

This is what I am interested in ATM. Previously I was using this to capture the data via TCP and return a complete line --

Func __TCPRecvLine($iSocket, $bEOLChar = 0x0A)
    Local Static $bReceived = Binary("")     ; Buffer for received data
    Local $bResult = Binary("")
    Local $bData = TCPRecv($iSocket, 4096, $TCP_DATA_BINARY)
    If @error Then
        Return SetError(1, 0, $bResult)
    EndIf

    local Static $itemp = 0

    $bReceived = $bReceived & $bData
    Local $iLength = BinaryLen($bReceived)

    If $iLength <> $itemp Then
        ;### Debug CONSOLE ↓↓↓
        ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iLength = ' & $iLength & @CRLF )
        ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $bReceived = ' & ($bReceived) & @CRLF )
        FileWriteLine("__TCPRecvLine.log", $bData)
        $itemp = $iLength
    EndIf

    If $iLength Then
        For $i = 1 To $iLength
            If BinaryMid($bReceived, $i, 1) = $bEOLChar Then
                $bResult = BinaryMid($bReceived, 1, $i)    ; Save the found line and
                $bReceived = BinaryMid($bReceived, $i + 1) ; remove it from the buffer
                ExitLoop
            EndIf
        Next
    EndIf

    Return $bResult
EndFunc   ;==>__TCPRecvLine

Is there a marker that can be used with the data returned by sgcWebSocketClient or do you recommend another approach?

[TheXman]

The function below is from the script that connects to the Yahoo Finance server that I referred to earlier.  I have separate functions for enqueuing events and processing events.  Basically, the function below is called to wait for and enqueue responses from the websocket server.  It polls until there is data and then reads the data until there is no more being sent.  As you can see, I can also set how long I want the function to wait for data before timing out and returning.  Once this function returns, I usually call a function that processes the queued data.

expandcollapse popup

Func enqueue_received_events($iWsSocketId, $iTimeoutTicks = 3000)
    Local $sReceivedEvents = "", _
          $sJqCmdOutput    = ""

    Local $hTimeoutTimer   = -1

    Local $iPrevDataLen    = 0


    ;Clear the event queue
    ReDim $gaEventQueue[0][2]

    ;Initialize timeout timer
    $hTimeoutTimer = TimerInit()

    ;Receive until timeout or data received
    Do
        Sleep(100)
        $sReceivedEvents &= TCPRecv($iWsSocketId, 2048)
    Until TimerDiff($hTimeoutTimer) >= $iTimeoutTicks Or $sReceivedEvents <> "" Or $gbStopMonitoring

    If StringLen($sReceivedEvents) = 0 Then Return SetError(1, 0, "") ;Timeout occurred waiting for events or user aborted

    ;Receive until no more data received
    Do
        Sleep(100)
        $iPrevDataLen    =  StringLen($sReceivedEvents)
        $sReceivedEvents &= TCPRecv($iWsSocketId, 2048)
    Until StringLen($sReceivedEvents) = $iPrevDataLen

    ;Parse JSON events into the event queue
    $sJqCmdOutput = _jqExec($sReceivedEvents, $JQ_PARSE_EVENTS)
    If @error Then Return SetError(2, 0, "")

    _ArrayAdd($gaEventQueue, $sJqCmdOutput)      ;Add new events to the queue
    If @error Then Return SetError(3, 0, @error)

    Return True
EndFunc

$sReceivedEvents, once I've received all of the events that the server has sent at the time, will look something like this:

{"event": "message", "text": "This is a test."}
{"event": "message", "text": "This is a test."}
{"event": "message", "text": "This is a test."}

I have no need for EOL or other markers.  jq, which is what I use to process the JSON events from sgcWebSocketClient, handles the multiple responses just as they are (a collection of JSON objects) and filters out the information that I want to enqueue (in a single call/execution).  I have jq output a pipe-delimited list of data that I pass to ArrayAdd().  My event queue is a 2D array that contain event type and any additional details that I need for processing.  The event type is necessary so I can identify (and process) all of the events (connect, disconnect, message, & error).

Edited November 13, 2022 by TheXman

[Danp2]

19 hours ago, TheXman said:

    $sJqCmdOutput = _jqExec($sReceivedEvents, $JQ_PARSE_EVENTS)

Can you share the definition of $JQ_PARSE_EVENTS?

[TheXman]

The jq filter that I used in my Yahoo Finance example looked like this:

Const $JQ_PARSE_EVENTS = _
    '['                                                   & _
    '  .event,'                                           & _
    '  if   .event == "message"      then .text'          & _
    '  elif .event == "error"        then .description'   & _
    '  elif .event == "disconnected" then .code'          & _
    '  else ""'                                           & _
    '  end'                                               & _
    ']'                                                   & _
    '|'                                                   & _
    'join("|")'

Basically it creates a pipe-delimited list that contains the event type and the details of the specific event.

Please keep in mind that the filter you use to process the incoming JSON events, especially as it relates to the "message" events, depends on the details of those events.  The filter that I used above was based on message events that looked like this:

{"event":"connected"}
{"event":"message", "text": "CgRBTVpOFXtUt0IYgI7inYxhKgNOTVMwCDgARVA/zUBl0MywQNgBBA=="}
{"event":"message", "text": "CgNXTVQVAEAOQxiQ2eOdjGEqA05ZUTAIOABFHyP/P2WA6zFA2AEE"}
{"event":"message", "text": "CgRBTVpOFXE9t0IYsPjjnYxhKgNOTVMwCDgARWKTy0BlMFyvQNgBBA=="}
{"event":"disconnected", "code": 0}

Those "message" events are base64 encoded protobuf messages.  That, when decoded, provide the ticker info.  Based on the filter and data above, the jq result looks like this:

In your case, for WebDriver, your "message" events are JSON objects.  So the filter that you would use would look a bit different depending on the information that you want to capture for your event queue (especially as it relates to the message events). 

FYI:
In a single call to jq, it can remove all of the double quotes around the JSON objects, in the .text field of your responses, as well as filter the info from those JSON objects into a format that you can use to create your event queue.  Of course you could do some of that processing using AutoIt functions but it would probably be more efficient and easier to maintain if you let jq handle the JSON parts. 

 

Edited November 13, 2022 by TheXman

[Danp2]

I want to be able to pass the complete results to the calling routine since this is for the Webdriver UDF where we don't know how the data will be used.

Here's what I've come up with thus far --

$sJSON = _
    '{"event": "connected"}' & _
    '{"event": "message", "text": {"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}}' & _
    '{"event": "message", "text": {"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}}'

$sFilter = _
    '[' & _
    '  if   .event == "message" then   ' & _
    '    if .text.id then "response", .text.id, .text' & _
    '    else "event", "", .text' & _
    '    end' & _
    '  elif .event == "error"        then null, .description' & _
    '  elif .event == "disconnected" then null, .code' & _
    '  else .event, "", ""' & _
    '  end' & _
    ']' & _
    '|[.[]|tostring]' & _
    '| join("|")'

_jqInit()
$sJqCmdOutput = _jqExec($sJSON, $sFilter)
$aJSON = _ArrayFromString($sJqCmdOutput)
_ArrayDisplay($aJSON)

Any suggestions on way to improve the jq filter?

[TheXman]

Sure, I have plenty of suggestions.   However, I prefer to provide answers, guidance, or help, to specific questions, issues, or obstacles.  Generic and open-ended questions like "any suggestions..." assumes that I have some idea about your where you want to go with your project -- which I don't.

Is there something specific that your filter is or isn't providing that you want to change?

Edited November 14, 2022 by TheXman

[Danp2]

The goal is to capture both events and responses to prior WS requests. Responses will have an ID key, so that's how they are distinguished from events. At this point I'm satisfied with the output from jq. However, there may be a more efficient way to achieve the same output without the array manipulation and using "tostring".

[TheXman]

In the future, please provide either a full script that runs or at least an image of the output so that I can see the result.  And if there is a difference between the result and the expected result, please explain what those differences are, in detail.

The output of the example script below is slightly different than yours.  It probably does not use the correct WD BiDi terminology but I think it gets the point across.  I'm not sure if you would consider it more efficient, but it is probably closer to the way that I would do it (given my limited understanding of what it is you are trying to do). 

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w 5 -w 6 -d

#include <Constants.au3>
#include <Array.au3>
#include <MyIncludes\jq\jq.au3> ;<== Modify path as needed


Const  $SAMPLE_EVENTS = _
    '{"event": "connected"}' & _
    '{"event": "message", "text": {"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}}' & _
    '{"event": "message", "text": {"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}}' & _
    '{"event":"disconnected", "code": 0}'

Const $JQ_FILTER_FULL_TEXT = _
    '['                                                   & _
    '  .event, '                                          & _
    '  if .event == "message" then'                       & _
    '    if .text|has("id") then'                         & _
    '      "response"'                                    & _
    '    else'                                            & _
    '      "event"'                                       & _
    '    end'                                             & _
    '  else'                                              & _
    '    ""'                                              & _
    '  end,'                                              & _
    '  if   .event == "message"      then .text|tostring' & _
    '  elif .event == "error"        then .description'   & _
    '  elif .event == "disconnected" then .code'          & _
    '  else ""'                                           & _
    '  end'                                               & _
    ']'                                                   & _
    '|@tsv'


jq_example($JQ_FILTER_FULL_TEXT)

Func jq_example($sJqFilter)
    Local $sResult   = ""
    Local $aEventQueue[0][3]

    ;Init jq
    _jqInit("C:\Utils\jq\jq-win64.exe") ;<== Modify path as needed
    If @error Then Return MsgBox($MB_ICONERROR, "Error", "_jqInit failed with @error = " & @error)

    ;Get TSV list of event info
    $sResult = _jqExec($SAMPLE_EVENTS, $sJqFilter)
    If @error Then
        ConsoleWrite("ERROR:" & @CRLF)
        ConsoleWrite($sResult & @CRLF)
        MsgBox($MB_ICONERROR, "Error", "_jqExec failed.")
        Return
    EndIf

    ;Load result into the event queue
    _ArrayAdd($aEventQueue, $sResult, 0, @TAB)
    If @error Then Return MsgBox($MB_ICONERROR, "Error", "_ArrayAdd failed with @error = " & @error)

    ;Display result
    _ArrayDisplay($aEventQueue, "Event Queue", "", 0, Default, "WS Event|WD Event|Details")
EndFunc

Result

Edited November 14, 2022 by TheXman

[Danp2]

36 minutes ago, TheXman said:

In the future, please provide either a full script that runs or at least an image of the output so that I can see the result.  And if there is a difference between the result and the expected result, please explain what those differences are, in detail.

Not sure if you tried to run the prior snippet of code that I posted, but the only thing omitted was the #includes AFAICS. And as I stated in the subsequent post, I'm satisfied with the output but looking for ways to improve / optimize the filter. 🙂

Thanks for the additional example. I was trying to be more efficient by combining the two "event" columns into one. However, it may make more sense to do it your way.

 

[TheXman]

45 minutes ago, Danp2 said:

the only thing omitted was the #includes

I guess you use _ArrayFromString() so often that you think it is a part of the AutoIt install?    I'm sure I could have done a search to find it somewhere, but a reference to the actual function would have been nice.

There are a couple more subtle differences in the filter other than what it outputs.  For example, the use of the has() function is a better way to test for the existence of a key than to check whether a key has a value.

Edited November 14, 2022 by TheXman

[Danp2]

28 minutes ago, TheXman said:

I guess you use _ArrayFromString() so often that you think it is a part of the AutoIt install? 

No, but it is part of the standard AutoIt install. See here for online help. 😏

Quote

There are a couple more subtle differences in the filter other than what it outputs.  For example, the use of the has() function is a better way to for test the existence of a key than to check whether a key has a value.

[TheXman]

2 minutes ago, Danp2 said:

No, but it is part of the standard AutoIt install.

Yes, you're right.  It was added in 3.3.16.0.  I am still on 3.3.14.5. 

[jugador]

Start sgcWebSocketServerWin32.exe from command prompt

{"message": "start", "params":{"host": "127.0.0.1", "port":9222}}

 

when run sgcWebSocketClientWin32.exe from command prompt all good.
but when try to run sgcWebSocketClientWin32 using below code it not working properly
> not sending text message sometime
> or sending only 1st text message
> or not sending Close Connection message
> if all good then getting error at end 

***FREE FOR NON-COMMERCIAL USE***
written by eSeGeCe.
version 2022.5.0
   copyright c 2022
   Email : info@esegece.com
   Web : http://www.esegece.com
Server Listening on 127.0.0.1:8080
Client F0049B06CC062F83A7585297C73E0EC1 connected from 127.0.0.1
{"event": "connected"}
{"event": "disconnected", "code": 0}
Client F0049B06CC062F83A7585297C73E0EC1 disconnected from 127.0.0.1
Socket Error # 10054
Connection reset by peer.

@TheXman need help

expandcollapse popup

#include <WinAPIError.au3>

__ExampleA()
Func __ExampleA()
;   Local $o_Url = 'ws://127.0.0.1:9222'
    Local $sIPAddress = "127.0.0.1"
    Local $iPort = 8080
    Local $o_ByteSend, $iError
    Local $o_Data = ''
    Local $o_Command = ' /c sgcWebSocketClient.exe -server'
    Local $o_Pid = Run(@ComSpec & $o_Command)

    Sleep(250)
    TCPStartup()
    Local $iSocket = TCPConnect($sIPAddress, $iPort)
    If @error Then
        $iError = @error
        ConsoleWrite("TCPConnect, Error code: " & $iError & @Tab & _WinAPI_GetErrorMessage($iError) & @Crlf)
    Else
        ConsoleWrite("> Connection successful " & @Crlf)

        ;~~~~ Open a new WebSocket Connection
        $o_Data = '{"message": "open", "params":{"url": "ws://127.0.0.1:9222"}}'
        TCPSend($iSocket, $o_Data)
        If @error Then
            $iError = @error
            ConsoleWrite("+ Client : Could not send the data, Error code: " & $iError & @Crlf)
        EndIf

        ;~~~~ Send Text message
        For $i = 1 To 3
            Sleep(250)
            $o_Data = '{"message": "write", "params":{"text": "Test Message No ' & $i & ' send."}}'
            $o_ByteSend = TCPSend($iSocket, $o_Data)
            If @error Then
                $iError = @error
                ConsoleWrite("+ Client : Data send failed, Error code: " & $iError & @Crlf)
            Else
                ConsoleWrite("+ Client : Byte Send: " & $o_ByteSend & @Crlf)
            EndIf
        Next

        ;~~~~ Close Connection
        Sleep(1000)
        $o_Data = '{"message": "close"}'
        TCPSend($iSocket, $o_Data)
        If @error Then
            $iError = @error
            ConsoleWrite("+ Client : Could not send the data, Error code: " & $iError & @Crlf)
         EndIf
    EndIf

    TCPCloseSocket($iSocket)
    If @error Then
        $iError = @error
        ConsoleWrite("TCPCloseSocket, Error code: " & $iError & @Crlf)
    EndIf
    TCPShutdown()
EndFunc

 

[Danp2]

@jugadorI'm not sure if this is your issue, but I had to allow some time for the client to load before the TCP connection would succeed. I ended up adding a loop --

For $i = 0 To 10 Step 1
    Sleep(100)
    $iSocket = TCPConnect($sIPAddress, $iPort)
    If Not @error Then ExitLoop
Next

 

[jugador]

@Danp2

That's not the problem, putting sleep(250) after run command work well.
problem is getting "Socket Error # 10054" at end

it seems adding TCPRecv after Close Connection call solve the (Socket Error # 10054) problem.
but don't know why? 

Sleep(250)
Local $sReceived = TCPRecv($iSocket, 2048)
ConsoleWrite($sReceived & @Crlf)

 

Edited November 23, 2022 by jugador

[Danp2]

4 minutes ago, jugador said:

it seems adding TCPRecv after Close Connection call solve the (Socket Error # 10054) problem.

That actually makes sense because the client is sending the status of your close request. If you change the code to this --

Local $sReceived = TCPRecv($iSocket, 2048)
ConsoleWrite($sReceived & @Crlf)

you should get this is the console --

{"event": "connected"}{"event": "disconnected", "code": 0}

These are responses to your connect and disconnect requests. When you launch the client with TCP server enabled, it sends responses via TCP instead of writing to  the console.

[TheXman]

On 11/23/2022 at 9:12 AM, Danp2 said:

When you launch the client with TCP server enabled, it sends responses via TCP instead of writing to  the console.

Enabling the WebSocket client's internal TCP server has no effect on the messages that you may enter at the console or that are received from the connected WebSocket server to the console.  When you enable the internal WebSocket client's TCP server, responses from the connected WebSocket server go to both the console and the WebSocket client's enabled TCP server socket.  The internal TCP server merely allows you to interact with the console, over a TCP connection, as if you were at the console -- which you can still do while the TCP server is enabled.

On 11/23/2022 at 8:57 AM, jugador said:

it seems adding TCPRecv after Close Connection call solve the (Socket Error # 10054) problem.
but don't know why?

To word it a little differently than Danp2, the reason that you got the 10054 error was because you closed the connection to the internal TCP server while there was still data waiting to be received on the socket.  The client's TCP server let you know by telling you that the connection was prematurely closed/reset.  When the internal TCP server is enabled, the JSON event responses that you see in the console are also sent to the internal TCP server's socket.  In your case, getting the error really didn't matter because you weren't doing anything with the responses anyway.  However, in most cases, when you send a message to a WebSocket server, you should see what the response is and proceed accordingly.  As Danp2 pointed out, when you connected to & disconnected from the WebSocket server, it sent back responses confirming that those requests were successful.  Your TCPRecv() read the responses that were sitting on the socket.  So when you closed the TCP connection, it closed gracefully.

Edit (11/25/2022):
Although it wasn't the reason in this case, you can also get a 10054 error if you do a TCPShutdown() without issuing a TCPCloseSocket() first, even if there's no data on the socket.  Or to say it another way, you can get a 10054 error if you do a TCPShutdown() while there is still an open socket, regardless of whether the socket has data on it.

Edited November 25, 2022 by TheXman
Added additional 10054 info

[Danp2]

2 hours ago, TheXman said:

The internal TCP server merely allows you to interact with the console, over a TCP connection, as if you were at the console -- which you can still do while the TCP server is enabled.

Thanks for the correction. I thought that I had experienced a different behavior previously, but it's currently working as you've described.