Parsing binary data frames from an asynchronous source

1 post in this topic

#1 ·  Posted (edited)

I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.


Author: ToasterKing
This is an example of a way to parse streaming binary data that follows a strict format with a header and footer.
In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD.
The _BinaryParse() function accumulates incoming data in a buffer.  Once a footer is found, it searches backward
for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle,
then moves on to looking for the next frame.

; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable.
Local $fSomeData
$fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame.
_BinaryParse($fSomeData) ; Call the function with the received data.  It isn't a complete frame, so it is just stored in the buffer until more data is received.
$fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame.
_BinaryParse($fSomeData) ; The function should be able to parse both frames now.

Func _BinaryParse($fNewData)
    Local Static $fBinaryReceived = Binary("") ; Buffer for received data
    ConsoleWrite("Hey, the function is called!" & @CRLF)
    ; Add new data to the buffer.
    ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt.  It must be converted to strings first.
    ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data.  To be converted back to binary properly, only one instance
    ; of "0x" must exist at the beginning of the string.
    $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x",""))
    ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF)
    Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data
    If $iLength > 0 Then
        Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on
        Local $fByte1,$fByte2
        For $i = 1 To $iLength
            If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found
                ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF)
                If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that
                ConsoleWrite("Header found before the footer!" & @CRLF)
                    $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer)
                    $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer)
                    ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes.  Normally, you'd do something more useful with it here.
                $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp


Edited by toasterking

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • tarretarretarre
      By tarretarretarre
      Yep yep, this is pretty much an attempt to port the existing project's concept to AutoIt's Codebase. So i will not go in to so much detail.
      This is how the communication is done Each client is isolated to the server Features
      Easy API VarType Translation (Example: If the server sends an int, the client will receive an int and vice versa) Fully featured examples Data encryption (Using Autoit's UDF Crypt.au3) Limitations / Drawbacks
      Every Broadcast/Emit is limited to a maximum of 16 parameters Every Callback Function has to have the same number of parameters as the Broadcasted/Emited event It is not possible to Broadcast/Emit objects Only 1D-arrays are allowed to be Broadcasted/Emitted (2D arrays will probably never be supported) Changelog
      Version 1.4.0 (This update DOES NOT break scripts)
      Added a new server method: _Io_getSockets which will return an array of all sockets. See more in the doc Added a banning-system, see more at: _Io_getBanlist, _Io_Ban, _Io_Sanction, _Io_IsBanned Added a new default event for clients banned. See more at default events Added two new client and server methods _Io_setEventPreScript And _Io_setEventPostScript. The intent for these is to not DRY when doing debug \ tasks that requires to be ran before or after events. Added a new client and server method _Io_ClearEvents. Added a third optional parameter to _Io_On called $socket, you may only pass the socket returned from _Io_Listen or _Io_Connect. The intent for this change is to allow for server + client in the same envoirment. Added a second parameter to _Io_Loop called $WhoAmI which should used with the new enums $_IO_SERVER and $_IO_CLIENT. The intent for this change is to allow for server + client in the same envoirment. Added a new client method _Io_TransferSocket. Added a new server method _Io_getActiveSocketCount. Optimations, avoiding Redims and unnecessary nested arrays as good as possible etc. Version 1.3.0 (This update DOES NOT break scripts)
      Got rid of unnecessary Redims with sockets and subscriptions in the main loop (This increased write performence greatly) Changed $iMaxDeadSocketsBeforeTidy from 100 to 1000 Changed _Io_setRecvPackageSize($nPackageSize = 2048) to _Io_setRecvPackageSize($nPackageSize = 4096) because 2017. Added Tests for both subscriptions and the automatic TidyUp Added a new server method: _Io_getMaxConnections Added a new server method: _Io_getMaxDeadSocketsCount Added a fifth parameter to the _Io_Listen method called $iMaxConnections which defaults to 100000. If the iMaxConnection + 1 user connects, they will be instantly disconnected. Added a parameter to _Io_Disconnect called $socket which defaults to null. If the iMaxConnections + 1 client connects, they will be instantly disconnected. Version 1.2.0 (This update DOES NOT break scripts)
      Added an option to set the packet-size of TCP-transports, see _Io_setRecvPackageSize Got rid of unnecessary StringLen's in _Io_loop Changed __Io_TidyUp to _Io_TidyUp and added it to the public Api reference list. Changed $iMaxDeadSocketsBeforeTidy default value from 1000 to 100 and added an option to disable it, read more at _Io_Listen Changed $bAutoReconnect from False to True. Fixed gitignore epicZ fail Improvemend Documentation Version 1.1.0 (This update DOES NOT break scripts)
      Fixed bug when Emitting / Broadcasting without any parameters causing a $fCallback crash Optimized Package-handling once again. Added 1D-Array support (Endless nestning). Added Subscriptions (See _Io_Subscribe _Io_Unsubscribe and _Io_BroadcastToRoom). Added new example for subscriptions (Be sure to use different room names when joining with clients) Added Unit testing (See Tests\Runner.au3 and Tests\Tests.au3, to run tests you need a udf found here: Version 1.0.0
      (This update DOES NOT break scripts) Added data encryption (Using Autoit's UDF Crypt.au3) See more at _Io_EnableEncryption Added new method _Io_Disconnect which can be used with both servers and clients Improved package-handling to increase performance Increased the limit of Broadcasted/Emit parameters from 10 to 16 Api methods
      Server methods
      _Io_Listen($iPort, $iAddress = @IPAddress1, $iMaxPendingConnections = Default, $iMaxDeadSocketsBeforeTidy = 1000, $iMaxConnections = 100000) _Io_Subscribe(ByRef $socket, $sRoomName) _Io_Unsubscribe(ByRef $socket, $sRoomName = null) _Io_Broadcast(ByRef $socket, $sEventName, $p1, $p2, ...$p16)  
      _Io_BroadcastToAll(ByRef $socket, $sEventName, $p1, $p2, ...$p16) _Io_BroadcastToRoom(ByRef $socket, $sDesiredRoomName, $sEventName, $p1, $p2, ...$p16) _Io_socketGetProperty(ByRef $socket, $sProp = Default) _Io_getSockets($bForceUpdate = False, $socket = $__g_io_mySocket, $whoAmI = $__g_io_whoami) _Io_getDeadSocketCount() _Io_getSocketsCount() _Io_getActiveSocketCount() _Io_getMaxConnections() _Io_getMaxDeadSocketsCount() _Io_getBanlist($iEntry = Default) _Io_Ban($socketOrIp, $nTime = 3600, $sReason = "Banned", $sIssuedBy = "system") _Io_Sanction($socketOrIp) _Io_IsBanned($socketOrIp) _Io_TidyUp() Client methods
      _Io_Connect($iAddress, $iPort, $bAutoReconnect = True) _Io_Reconnect(ByRef $socket) Server and Client methods
      _Io_setEventPreScript($fCallback) _Io_setEventPostScript($fCallback) _Io_getVer() _Io_On(Const $sEventName, Const $fCallback, $socket = $__g_io_mySocket) _Io_Emit(ByRef $socket, $sEventName, $p1, $p2, ...$p16) _Io_Loop(ByRef $socket, $whoAmI = $__g_io_whoami) _Io_LoopFacade() _Io_EnableEncryption($sFileOrKey, $CryptAlgId = $CALG_AES_256) _Io_Disconnect($socket = null) _Io_setRecvPackageSize($nPackageSize = 4096) _Io_ClearEvents() _Io_TransferSocket(ByRef $from, ByRef $to) Default events
      Server events
      connection Client events
      banned Server and Client events
      disconnect View source on github
     (OLD!) (OLD) (OLD) (NEWEST 2017-08-11)
    • timmalos
      By timmalos
      Hello all.
      In case this is interesting few of you, I share my AMCP 2.1 protocol UDF in AutoIT. This protocol is used by CasparCG server, which is a Windows and Linux software used to play out professional graphics, audio and video to multiple outputs as a layerbased real-time compositor. It has been in 24/7 broadcast production since 2006. It's free and opensource.
      The UDF I share allows communication between an AutoIt based client and the CasparCG, based on following documentation :
      If you want more details on CasparCG :  official WebSite or have a look to this video
      I'm currently building a full Client based on AutoIt, with many features like drag-and-drop layers, but sadly I can't share it right now, might come later. Don't hesitate to ask questions if you have any or need a basic example.
      The only requirement for this UDF is the other Event-driven TCP UDF by Kip

    • JustinZandee
      By JustinZandee
      I need a random string generator which creates 15 letters/numbers.
      How can I make that?
    • anthonyjr2
      By anthonyjr2
      Hi guys,
      I am pretty bad with regex, and am having some trouble trying to come up with an expression for a certain type of string. Basically I want to be able to tell if a string is of the format:
      Where the A's are any letter from A-Z and the #'s are any digit from 0-9.
      I've been playing around with a regex tester online for a while but I can't really seem to grasp the concept very well. Could anyone give me any tips?
      This isn't exactly an AutoIt specific question which is why I didn't post it in General Help & Support.
    • MazeM
      By MazeM
      here's another UDF for the serial port. It is very similar to CommAPI using kernel32.dll, but all code is packed into a single file without any dependencies, not even using WinAPI.au3. It differs from existing UDF that it doesn't allow a timeout when reading, instead it always returns immediately, either with the requested amount ob bytes read or with a failure status. And of course there is a function provided to query the amount of available bytes in the receive buffer. The reason behind this design decision: You can do 1000 other things in the main loop while checking from time to time if enough data bytes arrived. There's no point to block the program waiting for the serial port.
      It is currently a work-in-progress, as I didn't test all functions yet. The code was developed and tested on Windows 7 64 bit.  The ComUDF-Tests.au3 shows some tests and basic usage of the UDF. Maybe there's no reason to use this UDF, given the existence of the others UDFs, but I did it to get to know DllCall better - I use structs no only to pass but also to get data back (I don't use the array returned by DllCall to read that data, unless required). You're welcome to test it on older and newer Windows versions.
      Here's a list of the implemented functions:
      ; _ComListPorts ; _ComOpenPort ; _ComSetTimeouts ; _ComClosePort ; ; _ComSetBreak ; _ComClearBreak ; _ComGetInputcount ; _ComGetOutputcount ; _ComClearOutputBuffer ; _ComClearInputBuffer ; ; _ComSendByte ; _ComReadByte ; _ComSendBinary ; _ComReadBinary ; ; _ComSendChar ; _ComReadChar ; _ComSendCharArray ; _ComReadCharArray ; _ComSendString ; _ComReadString ; ; __ComClearCommError ; __PurgeComm Maze