Jump to content

Parsing binary data frames from an asynchronous source

Recommended Posts

I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.


Author: ToasterKing
This is an example of a way to parse streaming binary data that follows a strict format with a header and footer.
In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD.
The _BinaryParse() function accumulates incoming data in a buffer.  Once a footer is found, it searches backward
for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle,
then moves on to looking for the next frame.

; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable.
Local $fSomeData
$fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame.
_BinaryParse($fSomeData) ; Call the function with the received data.  It isn't a complete frame, so it is just stored in the buffer until more data is received.
$fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame.
_BinaryParse($fSomeData) ; The function should be able to parse both frames now.

Func _BinaryParse($fNewData)
    Local Static $fBinaryReceived = Binary("") ; Buffer for received data
    ConsoleWrite("Hey, the function is called!" & @CRLF)
    ; Add new data to the buffer.
    ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt.  It must be converted to strings first.
    ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data.  To be converted back to binary properly, only one instance
    ; of "0x" must exist at the beginning of the string.
    $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x",""))
    ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF)
    Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data
    If $iLength > 0 Then
        Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on
        Local $fByte1,$fByte2
        For $i = 1 To $iLength
            If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found
                ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF)
                If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that
                ConsoleWrite("Header found before the footer!" & @CRLF)
                    $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer)
                    $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer)
                    ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes.  Normally, you'd do something more useful with it here.
                $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp


Edited by toasterking
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Colduction
      Hi AutoIt programmers, excuse me for bothering you with multiple topics.

      In AutoIt we can use Number() function to convert Hex string to number but it's output is different of C# output & and i wanna make it's output like AutoIt code.

      For e.g I use this in AutoIt:
      Local $dBinary = Binary("Hello") ; Create binary data from a string. Local $dExtract = Number(BinaryMid($dBinary, 1, 5)) ConsoleWrite($dExtract & @CRLF) And i use this for C#:
      using System; using System.Text; //NameSpace Is Use of Project Name namespace TEST { class Program { public static void Main(string[] args) { //declaring a variable and assigning hex value string dd = ToHex("Hello", Encoding.ASCII); decimal d = Int64.Parse(dd, System.Globalization.NumberStyles.HexNumber); Console.WriteLine("Result: " + d); //hit ENTER to exit Console.ReadLine(); } public static string ToHex(string sInput, Encoding oEncoding, bool b0x_Prefix = false) { byte[] a_binaryOutput = oEncoding.GetBytes(sInput); string sOutput = BitConverter.ToString(a_binaryOutput).Replace("-", ""); if (b0x_Prefix == false) { return sOutput; } else { return "0x" + sOutput; } } } }
      I say once again that excuse me for creating new topic, in fact i'm making a library for GAuthOTP from a topic in AutoIt.
    • By Colduction
      Hi AutoIt Programmers, i wanna figure out how to use Binary functions in C# like:

      IsBinary and other basic ones were too ez, but those two were hard to noob like me.

      I appreciate for your helps/hints.
    • By t0nZ
      Computer Plucker is a fork and a "restoration" of Computer stats utility by @Rogue5099 (link)
      You can obtain various data about local machine, as from the screenshot.

      But my goal is not (only) to provide another PC stats tool, but to develop a very basic remote inventory tool, like (a small subset of) the abandoned Spiceworks or the Microsoft SCCM.
      So this script can be deployed on a machine as a standalone executable (yes you need only ONE file to distribute) , and launching the .EXE it installs itself to a folder (es. c:\autoit\computerplucker)
      In tools menù you can set two tasks and not only
      one for creating every week a file.ini with all the data of the machine. another one  for creating every week a file.ini with all the data of the machine PLUS uploading a CRYPTED record of the machine in a free mysql DB online For my needs, I am  deploying the .exe and schedule the second task, uploading data one time a week.
      In tools menù you will find also commands for:
      manual creating the .ini data file  to upload the data in crypted records to upload the data in clear text records (for testing..) total uninstall of ComputerPlucker, with a simple security code. many other tasks in not so logical order.. next version I have to tidy the menù... So my plan is to deploy this on my machines, and have the data in a online DB, this DB will be for me only a "cache" for data records to be downloaded (and deleted) by another script.
      As today the account of this TEST DB is embedded in code, using the "File to Base64 String' Code Generator" by @UEZ (used also for embedding icons and XML task definitions)
          Local $mysqlhost = "sql11.freemysqlhosting.net"
          Local $mysqlport = "3306"
          Local $mysqldb = "sql11403701"
          Local $mysqluser = "sql11403701"
          Local $mysqlpsw = "QtL7T9U5GV"
      The DB is one table of 30 mysql tinytext fields... very simple indeed...
      You can try to upload data to this DB and access (phpmyadmin or a random mysql browser app) to see the results.
      Or you can use my ComputerPluckerDBbrowser (see at the end of this post) a script to collect and manage the data.

      PRIVACY warning: you are uploading data of your PC account and PC data, test as your risk (or test in another account...)
      I also created a small post in a blog for not so geek friends, and I have a more advanced version of the data browser, you can have more info lookin' here. 😀
      Attached you will find the code, and all the necessary includes are at this >link<.
      ComputerPlucker.EXE is downloadable >HERE<
      ComputerPluckerDBbrowser.EXE is downloadable >HERE<

    • By jmor
      For some reason, I can't convert to Hex a Decimal color that has been calculated.
      For example, this works:
      Local $iColor = 3310546 ConsoleWrite("$iColor (dec): " & $iColor & ", $iColor (hex): " & Hex($iColor, 6) & @CRLF) giving the expected output:
      $iColor (dec): 3310546, $iColor (hex): 3283D2
      But if I want to take an average of several colors, I have a problem.  For example this doesn't work:
      $iColor = (3310546 + 3310546) / 2 ConsoleWrite("$iColor (dec): " & $iColor & ", $iColor (hex): " & Hex($iColor, 6) & @CRLF) giving the wrong output:
      $iColor (dec): 3310546, $iColor (hex): 000000
      I observe this behavior after any processing of a decimal color.
      Is this a bug?
    • By tarretarretarre
      Version 2.x.x and 3.x.x has been moved to branch 3.x
      About Autoit-Socket-IO
      Autoit-Socket-IO is a event driven TCP/IP wrapper heavily inspired from Socket.IO with focus on user friendliness and long term sustainability.
      I constantly want to make this UDF faster and better, so if you have any suggestions or questions (beginner and advanced) Do not hesitate to ask them, I will gladly help!
      Key features
      Simple API 99% data-type serialization thanks to Autoit-Serialize Can easily be extended with your own functionality thanks to Autoit-Events "Educational" examples Data encryption thanks to _<Crypt.au3> Limitations
      Speed. This UDF will sacrifice some speed for convenience Getting started
      Download the script from AutoIt or pull it from the official github repo git@github.com:tarreislam/Autoit-Socket-IO.git and checkout the tag 4.0.0-beta Check out the documentaion Take a look in the examples/ folder Changelog
      To see changes from 3.x.x and 2.x.x please checkout the 3.x branch
      Version 4.0.0-beta (This update break scripts.)
      Code base fully rewritten with Autoit-Events and decoupled to improve code quality and reduce bloat. The new UDF is very different from 3.x.x so please checkout the UPGRADE guide to fully understand all changes Added new documentation documentaion Success stories
      Since December 2017-now I have used version 1.5.0 in an production environment for 150+ clients with great success, the only downtime is planned windows updates and power outages.
      Newest version (2020-09-15!)

      Older versions (Not supported anymore)
      Autoit-Socket-IO-1.0.0.zip Autoit-Socket-IO-1.1.0.zip Autoit-Socket-IO-1.3.0.zip Autoit-Socket-IO-1.4.0.zip Autoit-Socket-IO-1.5.0.zip
  • Create New...