Jump to content

Parsing binary data frames from an asynchronous source

Recommended Posts


I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.


Author: ToasterKing
This is an example of a way to parse streaming binary data that follows a strict format with a header and footer.
In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD.
The _BinaryParse() function accumulates incoming data in a buffer.  Once a footer is found, it searches backward
for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle,
then moves on to looking for the next frame.

; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable.
Local $fSomeData
$fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame.
_BinaryParse($fSomeData) ; Call the function with the received data.  It isn't a complete frame, so it is just stored in the buffer until more data is received.
$fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame.
_BinaryParse($fSomeData) ; The function should be able to parse both frames now.

Func _BinaryParse($fNewData)
    Local Static $fBinaryReceived = Binary("") ; Buffer for received data
    ConsoleWrite("Hey, the function is called!" & @CRLF)
    ; Add new data to the buffer.
    ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt.  It must be converted to strings first.
    ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data.  To be converted back to binary properly, only one instance
    ; of "0x" must exist at the beginning of the string.
    $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x",""))
    ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF)
    Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data
    If $iLength > 0 Then
        Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on
        Local $fByte1,$fByte2
        For $i = 1 To $iLength
            If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found
                ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF)
                If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that
                ConsoleWrite("Header found before the footer!" & @CRLF)
                    $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer)
                    $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer)
                    ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes.  Normally, you'd do something more useful with it here.
                $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp


Edited by toasterking

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • kartune
      By kartune
      Hello, I am getting this error message when running my script:
      Case $aButiD[0] To $aButiD[$iTotButtons - 1]
      Case ^ ERROR
      Error:  Array variable has incorrect number of subscripts or subscript dimension range exceeded.
      The script's purpose is to read my ini file and create buttons for the sections and section contents.
      The first part of the script creates buttons for the sections of the ini, once you click on the section button it should open another set of buttons of all the contents under the section
      In this case, the section in the .ini file is called "TPA"
      on the Func flavorco()
      It opens the contents of "TPA" into buttons.
      Right now, the section it is reading is determined by $sSection.
      It is only when I try to change $sSection = "TPA"
      $sSection = $flavorco
      that the error starts to appear
      Here is the script
      #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> #include <Array.au3> #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> #include <ButtonConstants.au3> $optbar = IniReadSectionNames("test.ini") If Not IsArray($optbar) Then Exit Local $Button[UBound($optbar) - 1] $mGUI = GUICreate('Restock Flavor Ingredients', 10 + (130 * UBound($optbar)), 100) For $x = 1 To UBound($optbar) - 1 $var = IniReadSection("test.ini", $optbar[$x]) If @error Then ContinueLoop $X_Coord = 10 + (($x - 1) * 100) $Button[$x - 1] = GUICtrlCreateButton($optbar[$x], $X_Coord, 30, 70, 30) Next GUISetState() While 1 $mMsg = GUIGetMsg() If $mMsg = $GUI_EVENT_CLOSE Then GUIDelete($mGUI) ExitLoop EndIf For $i = 0 To UBound($Button) - 1 If $mMsg = $Button[$i] Then MsgBox(0, '$Msg = ' & $mMsg, GuiCtrlRead($Button[$i]) & ' Pressed', 2) Global $flavorco = $Button ; sets the flavor company's variable Call('flavorco') EndIf Next WEnd Func flavorco() ;opened flavor company's flavors MsgBox(0, '$Msg = ' & $mMsg, GuiCtrlRead($Button[$i]) & $flavorco, 0) Local $sFilePath = @ScriptDir & "\Test.ini" Local $sSection = $Button Local $aArray = IniReadSection($sFilePath, $sSection) _ArrayDelete($aArray, 0) ;_ArrayDisplay($aArray) Local $iIndex Local $iTotButtons = UBound($aArray) Local $iNumPerRow = 5 Local $iButWidth = 100 Local $iButDepth = 30 Local $aButiD[$iTotButtons] Local $hGUI = GUICreate('Ingredients Stock "' & $sFilePath & '"', 10 + ($iButWidth + 10) * $iNumPerRow, _ 10 + Ceiling($iTotButtons / $iNumPerRow) * ($iButDepth + 10), -1, $WS_EX_TOPMOST) For $i = 0 To $iTotButtons - 1 ; $x = $iXBorder + (($iRectWidth + $iSpacing) * Mod($i, $iNumCols)) $x = 10 + (($iButWidth + 10) * Mod($i, $iNumPerRow)) ; $y = $iYBorder + (($iRectDepth + $iSpacing) * Int($i / $iNumCols)) $y = 10 + (($iButDepth + 10) * Int($i / $iNumPerRow)) $aButiD[$i] = GUICtrlCreateButton($aArray[$i][0], $x, $y, $iButWidth, $iButDepth) ;, $BS_NOTIFY) Next GUISetState(@SW_SHOW) While 1 $msg = GUIGetMsg() Switch $msg Case $GUI_EVENT_CLOSE GUIDelete($hGUI) ExitLoop Case $aButiD[0] To $aButiD[$iTotButtons - 1] ;clicking on a flavor name $iIndex = ($msg - $aButiD[0]) $RSINGRflavor = $aArray[$iIndex][0] ;flavor name $RSINGRml = $aArray[$iIndex][1] ; ml $RSINGRgal = $aArray[$iIndex][1]*0.000264172 ;converts ml to gal Call('openflavor') EndSwitch WEnd EndFunc Func openflavor() #Region ### START Koda GUI section ### Form=F:\Karl\AutoIt\FJ-Stock JP\GUI Forms\RSINGRSUBFORM.kxf ;opens flavor to view stock and restock submit $RSINGRSUBFORM = GUICreate($sSection & " " & $RSINGRflavor, 242, 213, 530, 269) $RSINGRFlavorTitle = GUICtrlCreateLabel($sSection & " " & $RSINGRflavor, 32, 8, 200, 28) GUICtrlSetFont(-1, 16, 800, 0, "MS Sans Serif") $RSINGRstockcaption = GUICtrlCreateLabel("Current Stock in ml:", 16, 48, 96, 17) $RSINGRstockml = GUICtrlCreateLabel($RSINGRml, 120, 48, 114, 25) GUICtrlSetFont(-1, 8, 800, 0, "MS Sans Serif") $RSINGRInput = GUICtrlCreateInput("ml", 40, 112, 153, 21) $RSINGRSUBMIT = GUICtrlCreateButton("SUBMIT", 16, 152, 81, 33) $RSINGRCLOSE = GUICtrlCreateButton("CLOSE", 145, 151, 81, 33) $RSINGRgalcaption = GUICtrlCreateLabel("Current Stock in gal:", 16, 80, 97, 17) $RSINGRstockgal = GUICtrlCreateLabel($RSINGRGAL, 120, 80, 114, 25) GUICtrlSetFont(-1, 8, 800, 0, "MS Sans Serif") GUISetState(@SW_SHOW) #EndRegion ### END Koda GUI section ### ; $msg = "" While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE GUIDelete($RSINGRSUBFORM) ExitLoop Case $RSINGRCLOSE GUIDelete($RSINGRSUBFORM) ExitLoop EndSwitch WEnd EndFunc  
      Any help is much appreciated!  Thanks.
      My guess is that $Button isn't an actual string??  Whenever I use msgbox it shows up the string exactly, no numbers attached.  So i'm not sure whats going on here.
    • FMS
      By FMS
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
    • ripdad
      By ripdad
      I have had several people ask for this, so I decided to work the algorithm for it and this is the result.
      What is it?
      A Gateway Proxy Sends and Receives Data Unmodified.
      What is it used for?
      You can use it as a gateway, relay or router between two known static IP addresses.
      More information is in the header of the script.
      Download: LocalGatewayProxy.au3
      You will need WSA_NBTCP.au3 from here:
      If you have any questions or problems, let me know.
    • XinYoung
      By XinYoung
      Hello again,
      I'm trying to use the FileMove function to rename a bunch of text files.
      Some of the files have "_1" at the end of their name, for example, "File123_1.txt". I want to remove the "_1" so it would become "File123.txt".
      What I currently have...
              FileMove($sSource & "\*_1.txt", $sDestination & "\*.txt")
      But since I'm using a wildcard, it doesn't appear to be working. I think it's just replacing .txt with .txt. Am I going about this the wrong way? How can I use wildcards and still accomplish this?
      Any help is greatly appreciated ^__^
    • ripdad
      By ripdad

      This script is based on algorithm code from EnrMa.
      Updated: January 22, 2018
      Made improvements. Changes are in the script header.
      Known Issues: AutoIt x64 does not work properly with this script.
      Download: LocalProxyServer_v1.00.zip

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.