Jump to content
toasterking

Parsing binary data frames from an asynchronous source

Recommended Posts

I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.

 

#cs
Author: ToasterKing
This is an example of a way to parse streaming binary data that follows a strict format with a header and footer.
In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD.
The _BinaryParse() function accumulates incoming data in a buffer.  Once a footer is found, it searches backward
for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle,
then moves on to looking for the next frame.
#ce

; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable.
Local $fSomeData
$fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame.
_BinaryParse($fSomeData) ; Call the function with the received data.  It isn't a complete frame, so it is just stored in the buffer until more data is received.
$fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame.
_BinaryParse($fSomeData) ; The function should be able to parse both frames now.


Func _BinaryParse($fNewData)
    Local Static $fBinaryReceived = Binary("") ; Buffer for received data
    ConsoleWrite("Hey, the function is called!" & @CRLF)
    ; Add new data to the buffer.
    ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt.  It must be converted to strings first.
    ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data.  To be converted back to binary properly, only one instance
    ; of "0x" must exist at the beginning of the string.
    $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x",""))
    ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF)
    Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data
    If $iLength > 0 Then
        Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on
        Local $fByte1,$fByte2
        For $i = 1 To $iLength
            If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found
                ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF)
                If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that
                ConsoleWrite("Header found before the footer!" & @CRLF)
                    $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer)
                    $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer)
                    ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes.  Normally, you'd do something more useful with it here.
                EndIf
                $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp
            EndIf
        Next
    EndIf
EndFunc

 

Edited by toasterking

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By jmp
      i am trying to get number from string using this code :
      #include <IE.au3> $oIE = _IEAttach ("Edu.corner") Local $aName = "Student name & Code:", $iaName = "0" Local $oTds = _IETagNameGetCollection($oIE, "td") For $oTd In $oTds If $oTd.InnerText = $aName Then $iaName = $oTd.NextElementSibling.InnerText $iGet = StringRegExpReplace($iaName, "\D", "") EndIf Next MsgBox(0, "", $iGet) it was get number like 52503058
      But, I want to get only student code 5250. (Different student have different code, sometime its 3 digits, Sometime 4)

       
    • By Inpho
      Hi All,
      I didn't know where to put this; my apologies if this is the wrong location.
      When you plug a Samsung mobile device (phone, tablet, etc.) into the USB of a Windows PC, you can right-click the device in My Computer and select Properties. Here, it will show you the correct serial number of the device. Clearly Windows doesn't use adb to get the SN so I'm stuck at how to get the SN without adb and where the device doesn't have a drive letter assigned to it.
      When I last picked this up, I tried seemingly everything I could from wmi(?) and winmgmts(?) but either it's hidden cloak-and-dagger style or I can't see the forest for the trees...
      Does anyone know what API Windows uses when getting the serial number of a device Windows calls a Portable Media Player?

    • By Inpho
      Hi All,
      I intend on keeping custom functions/UDFs (works in progress) here; if anyone wants to use any code, feel free.
      String functions:
      #AutoIt3Wrapper_AU3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 #include-once ; #FUNCTION# ==================================================================================================================== ; Name ..........: _DateTimeGet ; Description ...: Returns the date and time formatted for use in sortable filenames, logs, listviews, etc. ; Syntax ........: _DateTimeGet(iType = 1[, $bHumanFormat = False]) ; Parameters ....: $iType - [optional] an integer value. Default is 1. ; 1 - Date and time in file-friendly format; 20190115_113756 ; 2 - Date in file-friendly format; 20190115 ; 3 - Time in file friendly format; 113756 ; $bHumanFormat - [optional] a boolean value. Default is False. ; True - Includes slashes in the date and colons in the time with a space inbetween ; False - No slashes or colons included with an underscore inbetween ; Return values .: Success - String ; Failure - Sets @error to non-zero and returns an empty string ; Author ........: Sam Coates ; =============================================================================================================================== Func _DateTimeGet($iType = 1, $bHumanFormat = False) If $iType < 1 Or $iType > 3 Then Return (SetError(-1, 0, "")) ;; Param1: ;; 1 = Date and time in file friendly format: 20190115_113756 ;; 2 = Date in file friendly format: 20190115 ;; 3 = Time in file friendly format: 113756 ;; Param2: ;; True = Use human-readable format: 15/01/2019 11:37:56 Local $sTime = @HOUR & ":" & @MIN & ":" & @SEC Local $sDate = @MDAY & "/" & @MON & "/" & @YEAR If $iType = 1 Then If $bHumanFormat = False Then $sTime = StringReplace($sTime, ":", "") $sDate = StringReplace($sDate, "/", "") $sDate = StringTrimLeft($sDate, 4) & StringMid($sDate, 3, 2) & StringLeft($sDate, 2) Return ($sDate & "_" & $sTime) Else Return ($sDate & " " & $sTime) EndIf ElseIf $iType = 2 Then If $bHumanFormat = False Then $sDate = StringReplace($sDate, "/", "") $sDate = StringTrimLeft($sDate, 4) & StringMid($sDate, 3, 2) & StringLeft($sDate, 2) EndIf Return ($sDate) ElseIf $iType = 3 Then If $bHumanFormat = False Then $sTime = StringReplace($sTime, "/", "") EndIf Return ($sTime) EndIf EndFunc ;==>_DateTimeGet ; #FUNCTION# ==================================================================================================================== ; Name ..........: _FileToFileExtension ; Description ...: Returns a file extension from a filename/FQPN (Fully Qualified Path Name) ; Syntax ........: _FileToFileExtension($sPath) ; Parameters ....: $sPath - a string value. ; Return values .: Success - String ; Failure - Empty string as returned from StringTrimLeft() ; Author ........: Sam Coates ; =============================================================================================================================== Func _FileToFileExtension($sPath) Return (StringTrimLeft($sPath, StringInStr($sPath, ".", 0, -1))) EndFunc ;==>_FileToFileExtension ; #FUNCTION# ==================================================================================================================== ; Name ..........: _FileToFileName ; Description ...: Returns a filename from a FQPN (Fully Qualified Path Name) ; Syntax ........: _FileToFileName($sPath[, $bIncludeExtension = True]) ; Parameters ....: $sPath - a string value. ; $bIncludeExtension - [optional] a boolean value. Default is True. ; Return values .: Success - String ; Failure - Empty string as returned from StringLeft() ; Author ........: Sam Coates ; =============================================================================================================================== Func _FileToFileName($sPath, $bIncludeExtension = True) Local $sReturn = StringTrimLeft($sPath, StringInStr($sPath, "\", 0, -1)) If $bIncludeExtension = False Then $sReturn = StringLeft($sReturn, StringInStr($sReturn, ".", 0, -1) - 1) Return ($sReturn) EndFunc ;==>_FileToFileName ; #FUNCTION# ==================================================================================================================== ; Name ..........: _FileToFilePath ; Description ...: Returns a folder path from a FQPN (Fully Qualified Path Name) ; Syntax ........: _FileToFilePath($sPath) ; Parameters ....: $sPath - a string value. ; Return values .: Success - String ; Failure - Empty string as returned from StringLeft() ; Author ........: Sam Coates ; =============================================================================================================================== Func _FileToFilePath($sPath) Return (StringLeft($sPath, StringInStr($sPath, "\", 0, -1) - 1)) EndFunc ;==>_FileToFilePath ; #FUNCTION# ==================================================================================================================== ; Name ..........: _StringLeft ; Description ...: Searches for a string inside a string, then removes everything on the right of that string ; Syntax ........: _StringLeft($sString, $sRemove[, $iCaseSense = 0, $iOccurrence = 1]) ; Parameters ....: $sString - a string value. The string to search inside. ; $sRemove - a string value. The string to search for. ; $iCaseSense - an integer value. Flag to indicate if the operations should be case sensitive. ; $iOccurrence - an integer value. Which occurrence of the substring to find in the string. Use a ; negative occurrence to search from the right side. ; Return values .: Success - String ; Failure - Empty string as returned from StringLeft() ; Author ........: Sam Coates ; =============================================================================================================================== Func _StringLeft($sString, $sRemove, $iCaseSense = 0, $iOccurrence = 1) Return (StringLeft($sString, StringInStr($sString, $sRemove, $iCaseSense, $iOccurrence) - 1)) EndFunc ;==>_StringLeft ; #FUNCTION# ==================================================================================================================== ; Name ..........: _StringRandom ; Description ...: Returns a string of random characters ; Syntax ........: _StringRandom($iAmount[, $iType = 1]) ; Parameters ....: $iAmount - an integer value. Length of returned string ; $iType - [optional] an integer value. Default is 1. ; 1 - Return digits (0-9) ; 2 - Return hexadecimal (0-9, A - F) ; 3 - Return Alphanumeric upper (0-9, A - Z) ; 4 - Return Alphanumeric (0-9, A - Z, a - z) ; 5 - Return Alpha upper (A - Z) ; 6 - Return Alpha (A - Z, a - z) ; Return values .: Success - String ; Failure - Empty string and @error flag as follows: ; @error : 1 - $iAmount is not a positive integer ; 2 - $iType is out of bounds ; Author ........: Sam Coates ; =============================================================================================================================== Func _StringRandom($iAmount, $iType = 1) If $iAmount < 1 Or IsInt($iAmount) = 0 Then Return (SetError(-1, 0, "")) Local $sString = "" Local $iRandomLow = 1, $iRandomHigh = 62 #Tidy_Off Local Static $aCharId[63] = [0, Chr(48), Chr(49), Chr(50), Chr(51), Chr(52), Chr(53), Chr(54), Chr(55), Chr(56), Chr(57), Chr(65), Chr(66), Chr(67), _ Chr(68), Chr(69), Chr(70), Chr(71), Chr(72), Chr(73), Chr(74), Chr(75), Chr(76), Chr(77), Chr(78), Chr(79), Chr(80), _ Chr(81), Chr(82), Chr(83), Chr(84), Chr(85), Chr(86), Chr(87), Chr(88), Chr(89), Chr(90), Chr(97), Chr(98), Chr(99), _ Chr(100), Chr(101), Chr(102), Chr(103), Chr(104), Chr(105), Chr(106), Chr(107), Chr(108), Chr(109), Chr(110), Chr(111), _ Chr(112), Chr(113), Chr(114), Chr(115), Chr(116), Chr(117), Chr(118), Chr(119), Chr(120), Chr(121), Chr(122)] #Tidy_On If $iType = 1 Then ;; digits: 1 - 10 $iRandomHigh = 10 ElseIf $iType = 2 Then ;; hexadecimal: 1 - 16 $iRandomHigh = 16 ElseIf $iType = 3 Then ;; alnumupper: 1 - 36 $iRandomHigh = 36 ElseIf $iType = 4 Then ;; alnum: 1 - 62 $iRandomHigh = 62 ElseIf $iType = 5 Then ;; alphaupper: 11 - 36 $iRandomLow = 11 $iRandomHigh = 36 ElseIf $iType = 6 Then ;; alpha: 11 = 62 $iRandomLow = 11 $iRandomHigh = 62 Else Return (SetError(-2, 0, "")) EndIf For $i = 1 To $iAmount $sString &= $aCharId[Random($iRandomLow, $iRandomHigh, 1)] ;; append string with corresponding random character from ascii array Next Return ($sString) EndFunc ;==>_StringRandom ; #FUNCTION# ==================================================================================================================== ; Name ..........: _StringTrimLeft ; Description ...: Searches for a string inside a string, then removes everything on the left of that string ; Syntax ........: _StringTrimLeft($sString, $sRemove[, $iCaseSense = 0, $iOccurrence = 1]) ; Parameters ....: $sString - a string value. The string to search inside. ; $sRemove - a string value. The string to search for. ; $iCaseSense - an integer value. Flag to indicate if the operations should be case sensitive. ; $iOccurrence - an integer value. Which occurrence of the substring to find in the string. Use a ; negative occurrence to search from the right side. ; Return values .: Success - String ; Failure - Empty string as returned from StringTrimLeft() ; Author ........: Sam Coates ; =============================================================================================================================== Func _StringTrimLeft($sString, $sRemove, $iCaseSense = 0, $iOccurrence = 1) Return (StringTrimLeft($sString, StringInStr($sString, $sRemove, $iCaseSense, $iOccurrence) + StringLen($sRemove) - 1)) EndFunc ;==>_StringTrimLeft Examples:
      ConsoleWrite(_StringRandom(100, 6) & @CRLF) ConsoleWrite(_StringTrimLeft("C:\Windows\System32\cmd.exe", "C:\Windows\System32\") & @CRLF) ConsoleWrite(_StringLeft("C:\Windows\System32\cmd.exe", "cmd.exe") & @CRLF) ConsoleWrite(_FileToFileName("C:\Windows\System32\cmd.exe") & @CRLF) ConsoleWrite(_FileToFilePath("C:\Windows\System32\cmd.exe") & @CRLF) ConsoleWrite(_FileToFileExtension("C:\Windows\System32\cmd.exe") & @CRLF) ConsoleWrite(_StringRandom(6, 4) & "-" & _StringRandom(4, 4) & "-" & _StringRandom(4, 4) & "-" & _StringRandom(4, 4) & "-" & _StringRandom(6, 4)& @CRLF)  
    • By tarretarretarre
      Autoit-Socket-IO
      Introduction
      Autoit-Socket-IO is a event driven TCP/IP wrapper heavily inspired from Socket.IO with focus on user friendliness and long term sustainability.
      I created this UDF because I was fascinated how Socket.IO made a such scary task "reliable and secure networking" so simple for the developer. So this was my main motivation.
      I constantly want to make this UDF faster and better, so if you have any suggestions. Do not hesitate to make requests!
      Features
      Flexiable and easy to understand API Above avarage documentation "Fully featured" examples Security in form of data encryption and middleware-support Limitations
      Speed. Because I want this UDF to be as flexible and simple as possible. Sometimes speed is sacrificed, but that does not mean i don't try to . It is not possible to emit objects mainly because autoit does not support serialization. Only 1D-arrays can be emitted (2D arrays will probably never be supported) Success story
      Since December 2017-now I have used version 1.5.0 in an production environment for 40+ clients with great success, the only downtime is planned windows updates and power outages.
      Getting started
      Download the script from AutoIt or pull it from the official github repo git@github.com:tarreislam/Autoit-Socket-IO.git and checkout the tag 2.0.0 The documentation is located at Docs\index.html Take a look in the examples/ folder Changelog
      Version 2.0.0 (This update break scripts. Please consult the upgrade.md for guidance)
      All global internal variables has been renamed. Added a bunch of new API methods: _Io_RegisterMiddleware, _Io_whoAmI, _Io_IsClient, _Io_IsServer, _Io_getAllByProperty and _Io_getFirstByProperty and some more. Read more about these in the documentation. _Io_socketGetProperty now has a setter method called _Io_socketSetProperty which can be used to set custom properties. _Io_socketGetProperty now has a third parameter "default" which is used when a property is not found Removed _Io_setEventPostScript and _Io_setEventPretScript in favor of _Io_RegisterMiddleware Improved documentation (It still needs some love though) Improved the verbosity of _Io_DevDebug  
      Newest version (2019-09-29!)
      Autoit-Socket-IO-2.0.0.zip
      Older versions (Not supported anymore)
      Autoit-Socket-IO-1.0.0.zip Autoit-Socket-IO-1.1.0.zip Autoit-Socket-IO-1.3.0.zip Autoit-Socket-IO-1.4.0.zip Autoit-Socket-IO-1.5.0.zip
    • By AoRaToS
      I started working on this program in the summer of 2008 then I stopped cause I faced some problems I couldn't overcome back then. Now that I've practiced more and have become a better scripter/programmer I'm releasing the program to the public to get some opinions. I know it's not a new concept but it's the first program I started besides some small stuff I did just for practice! I won't post the source code yet because it's still under construction, although I'm sure I've posted early stages of the code with bugs in the past in some topic...
      What I wanted was a simple, small, serverless program that would work without installation cause I wanted it for where I work, so I ended up with this!
      I have attached some images of various versions, also visit the forum thread.
      The package includes s!mpL3 LAN Messenger and the full change log.
      Current version 2.9.9.1! [04/07/2019]
      Check the Change Log below!
       
       
      http://www.autoitscript.com/forum/index.php?showtopic=88782
       
       
       
      Read the license before using this software.
       
×
×
  • Create New...