Parsing binary data frames from an asynchronous source

1 post in this topic

#1 ·  Posted (edited)

I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.


Author: ToasterKing
This is an example of a way to parse streaming binary data that follows a strict format with a header and footer.
In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD.
The _BinaryParse() function accumulates incoming data in a buffer.  Once a footer is found, it searches backward
for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle,
then moves on to looking for the next frame.

; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable.
Local $fSomeData
$fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame.
_BinaryParse($fSomeData) ; Call the function with the received data.  It isn't a complete frame, so it is just stored in the buffer until more data is received.
$fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame.
_BinaryParse($fSomeData) ; The function should be able to parse both frames now.

Func _BinaryParse($fNewData)
    Local Static $fBinaryReceived = Binary("") ; Buffer for received data
    ConsoleWrite("Hey, the function is called!" & @CRLF)
    ; Add new data to the buffer.
    ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt.  It must be converted to strings first.
    ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data.  To be converted back to binary properly, only one instance
    ; of "0x" must exist at the beginning of the string.
    $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x",""))
    ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF)
    Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data
    If $iLength > 0 Then
        Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on
        Local $fByte1,$fByte2
        For $i = 1 To $iLength
            If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found
                ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF)
                If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that
                ConsoleWrite("Header found before the footer!" & @CRLF)
                    $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer)
                    $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer)
                    ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes.  Normally, you'd do something more useful with it here.
                $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp


Edited by toasterking

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • Dragonfighter
      By Dragonfighter
      I'm searching a way to do xor and shift and if possible also other operations. Thanks in advance for the replies.
    • rudi
      By rudi
      I'm too stupid to see my mistake:
      To investigate the internal "dictionary" of TIFF files I'd like to read in the files in binary mode and to check, if there are more than one pages "in" this TIFF.
      Notepad++, "View as Hex" is presenting the first bytes as "49 49 2a 20 08 20 20 20 12" for the TIF attached to this posting
      The "TIFF Header Format" is easy:
      Offset 00h, 2 Byte = Byte Order, "II"=intel, "MM"=motorola. (I = 0x49)
      --> II
      Offset 02h, 2 Byte = Version Nr.
      Offset 04h, 4 Byte = pointer to first IFD entry
      Description of TIFF header:

      Howto read and analyse the binary content correctly? This is my messy, not operational code:
      $sampleTiff="H:\daten\tif\11\11\111111.TIF" $h=FileOpen($sampleTiff,16) $content=FileRead($h) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $content = ' & $content & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console FileClose($h) $type=VarGetType($content) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $type = ' & $type & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $ToString=BinaryToString($content) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $ToString = ' & $ToString & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ConsoleWrite(@CRLF & @CRLF) $content=StringTrimLeft($content,2) ; cut off the leading "0x" ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $content = ' & $content & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console for $i = 1 to 8 step 8 $next=StringMid($content,$i,2) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $next = ' & $next & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $Chr=BinaryToString($next) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $Chr = ' & $Chr & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ConsoleWrite(@CRLF & "---" & @CRLF) Next Regards, Rudi.
    • ur
      By ur
      When I am trying to compile the autoit files with aut2exe.
      I am getting below error.
      There is no issue in code as the same code is getting compiled on different machine.
      I tried reinstalling the AUtoIT, but the issue replicates.

      Any suggestions?
    • cheeroke
      By cheeroke
      Hi all,
      I got this code and would like to be able to change Baud Rate and instead of sending character by character i would like to be able (if possible) to send whole string. But i don't know how to change it.
      I am taking input from file and processing whole line (this is done in FilesHandling.au3).
      To execute this i am just calling SendData("FileName", int) in "main" script.
      Any help very appreciated.
      #include <WinAPI.au3> #include <Array.au3> #include "FilesHandling.au3" ;init DLL function, we need handle to call the function $h = DllCall("Kernel32.dll", "hwnd", "CreateFile", "str", "\\.\COM19", "int", BitOR($GENERIC_READ,$GENERIC_WRITE), "int", 0, "ptr", 0, "int", $OPEN_EXISTING, "int", $FILE_ATTRIBUTE_NORMAL, "int", 0) $handle=$h[0] Func SendData($FileName, $LineNumber) ;string to be send $c = readFile($FileName, $LineNumber) $cLenght = StringLen($c) $aArray = StringSplit($c, "") ;_ArrayDisplay($aArray, "", Default, 64) For $i = 1 To $cLenght writeChar($handle, $aArray[$i], $cLenght) Next ;move to next line writeChar($handle, @CR,1) EndFunc ;write a single char func writeChar($handle,$c,) $stString = DLLStructCreate("char str") $lpNumberOfBytesWritten = 0 DllStructSetData($stString, 1, $c) $res = _WinAPI_WriteFile($handle, DllStructGetPtr($stString, "str"), 1,$lpNumberOfBytesWritten) if ($res<>true) then ConsoleWrite ( _WinAPI_GetLastErrorMessage() & @LF) EndIf EndFunc  
    • FroVN
      By FroVN
      Hi, i have a problem :" can't set the name of file with a special character like: \;/;";|;...  have anyway to short the StringInSrt and Stringreplace? i am using this code but too long
         if StringInStr($title,'\') or StringInStr($title,'/') or StringInStr($title,':') or StringInStr($title,'*') or StringInStr($title,'?') or StringInStr($title,'"') or StringInStr($title,'<') or StringInStr($title,'>') or StringInStr($title,'|') Then