Jump to content
CSL

How to get arbitrary section of bits from a binary

Recommended Posts

I want to take a binary data from any source (string,number,files,etc..) and iterate over each X bits of it in a loop, say take bits 1-5, then 6-10, etc.. Then I want to convert these bits to their corresponding decimal value.

but all the binary functions I found in autoit only work with full bytes, and do not let me get smaller sections of bits, like "BinaryMid()" that "Extracts a number of bytes from a binary variant"

Can anyone tell me if this is possible to do and how? and also if there is a function to convert those bits to/from decimals?

I'm not that familiar with dealing directly with binary, so any help or tips will be very appreciated

Context:

I'm trying to make a function to encode/decode any given binary data into/from a given array of characters. just like Base64 but using different bases then [a-z A-Z 0-9 +/= ]. My approach currently is to figure out how many bits of binary I can encode with each character, read those bits and convert them to a decimal number, then I will use that number as an index and take the character at that index from the character array and add it to the result string.

I'm aware that there may be some padding needed.

 

 

Share this post


Link to post
Share on other sites

This is possible to do with the DllStruct- and the Bit-commands. If you want 5 bit blocks you can store 5 bytes at a time from your binary data into a 5 byte structure (40 bits). Then you can handle each of the 5 bit blocks one by one. 8 blocks in a total.

The CPU only handles 1, 2, 4 or 8 (64 bit CPU) bytes at a time. It does not handle 5 or 7 bytes. To handle 5 bytes you can for example split them up in 2 bytes, 2 bytes and 1 byte. If some of the 5 bit blocks have bits spread over byte 2 and 3 or byte 4 and 5 then you must deal with it.

To do all this you will need to be able to handle binary data. And you must be able to convert from binary to decimal numbers in your head. At least for the first 8 bits.

If you have large amounts of binary data AutoIt is not the fastest programming language. You'll need a compiled language.

What's the purpose for this? I do not think it's possible to make better algorithms for simple encoding/decoding, than those that already exists.

Share this post


Link to post
Share on other sites
14 minutes ago, LarsJ said:

This is possible to do with the DllStruct- and the Bit-commands. If you want 5 bit blocks you can store 5 bytes at a time from your binary data into a 5 byte structure (40 bits). Then you can handle each of the 5 bit blocks one by one. 8 blocks in a total.

The CPU only handles 1, 2, 4 or 8 (64 bit CPU) bytes at a time. It does not handle 5 or 7 bytes. To handle 5 bytes you can for example split them up in 2 bytes, 2 bytes and 1 byte. If some of the 5 bit blocks have bits spread over byte 2 and 3 or byte 4 and 5 then you must deal with it.

To do all this you will need to be able to handle binary data. And you must be able to convert from binary to decimal numbers in your head. At least for the first 8 bits.

If you have large amounts of binary data AutoIt is not the fastest programming language. You'll need a compiled language.

What's the purpose for this? I do not think it's possible to make better algorithms for simple encoding/decoding, than those that already exists.

Thanks for the reply LarsJ,  I did some searching before and tried the bit shifting method but it got complicated fast, I will only deal with small binary data so I instead chose to convert the binary to 0/1 strings and handling that instead, pretty dirty and slow, I Know.

Are there any existing encoding functions/libraries that can encode to an arbitrary list of characters? this is exactly what I need.

This function is for a little encryption based game I'm doing for fun, I know autoit may not be the best fit but it's fast to write & easy to use, and it's just a small project.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By Viszna
      Hello.
      I create a function that saves the log to an html file.
      File html are included picture (conversion to Base64)
      Everything works OK.
      But I do the actions:
      screenshot to the png file (smaller than bmp) per disk (  _ScreenCapture_Capture(@ScriptDir & "\screenshot.png")  ) convert image from disk to base64 I need help to optimize the script:
      - screenshot to memory (do not save to disk) - convert this object to png in memory - invoking the conversion of png image from memory to base64 The first step _ScreenCapture_Capture("") create handle to an HBITMAP in memory
      How to convert image in memory to png?
      How to use Func _ConvertToBase64
      I attach my code
      #include <ScreenCapture.au3> #include <Date.au3> Global $RaportFileName = @YEAR & "-" & @MON & "-" & @MDAY & "_" & @HOUR & "_" & @MIN & "_" & @SEC & ".html" $text = "This is first line text" & @CRLF &"and this is next line" _Raport($text, 1) FileWrite(@ScriptDir & "\" & $RaportFileName, "</pre></html>") ; #FUNCTION# ==================================================================================================================== ; Name ..........: _Raport ; Description ...: ; Syntax ........: _Raport($sText1[, $Screen = 0]) ; Parameters ....: $sText1 - a string value. ; $Screen - [optional] an unknown value. Default is 0. ; 0 - Default - do not screenshot ; 1 - added screenshot full desktop ; Return values .: None ; Author ........: Your Name ; Modified ......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; =============================================================================================================================== Func _Raport($sText1, $Screen=0) Local $sText = "" Local $sHead = "" ; Define HTML file header and style $sHead = '<html>' & @CRLF & '<head><meta charset="utf-8"></head>' & @CRLF $sHead = $sHead & '<style>img{border:3px solid #FF0000;}</style>' & @CRLF $sHead = $sHead & '<style>pre{font-family: monospace;}</style>' & @CRLF $sHead = $sHead & '<style>pre{font-size: large;}</style>' & @CRLF $sHead = $sHead & '<pre>' & @CRLF If NOT FileExists(@ScriptDir & "\" & $RaportFileName) Then ; If file Raport not exist then create FileOpen(@ScriptDir & "\" & $RaportFileName, 258) FileWrite(@ScriptDir & "\" & $RaportFileName, $sHead) EndIf If StringInStr($sText1, @CRLF) > 0 Then ; @CRLF (ENTER) change the @CRLF and 11 space (indentation on width "[GG:MM:SS] ") $sText1 = StringReplace($sText1, @CRLF, @CRLF & '&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;') EndIf $sText = $sText & $sText1 If $Screen <> 0 Then _ScreenCapture_Capture(@ScriptDir & "\screenshot.png") $sText = $sText & @CRLF & '<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;' & '<img src="data:image/png;base64,' & _ConvertToBase64(@ScriptDir & "\screenshot.png") & '"/>' EndIf FileWrite(@ScriptDir & "\" & $RaportFileName, "[" & _NowTime(5) & "]&nbsp;" & $sText & "<br><br>"&@CRLF) ; write to file Raport EndFunc Func _ConvertToBase64($fFile) ;Xroot 2011 ;ClipPut("") ;$FN=@ScriptDir & "\screenshot.png" $FN=$fFile $dat=FileRead(FileOpen($FN,16)) $objXML=ObjCreate("MSXML2.DOMDocument") $objNode=$objXML.createElement("b64") $objNode.dataType="bin.base64" $objNode.nodeTypedValue=$dat ClipPut("") $Wynik = "" ;ClipPut($objNode.Text) $Wynik = $objNode.Text Return $Wynik EndFunc P.S.
      Excuse me my not good English.
    • By Stew
      (Edited from original.  Please note that I AM NOT AN AUTOIT EXPERT.  I write code using Autoit frequently but I am no expert, especially when it comes to I/O.  So any remarks that start with "Why did you..." can be answered by referring to the first sentence.  This project was done in Autoit because of an interface I built to display the data.)
      Attached is a program and ascii input file I wrote to read stock price data, convert it to binary and then read it back into the program in binary.  The goal was to show increased performance for reading the files in binary and provide a demo on how to read/write binary for int32, int64, double and strings for anyone who might find it helpful.  The results on my PC show the following:
      Time to read ascii file only: 456.981951167202
      Ascii read & process time: 6061.83075631701
      Binary write file time: 14787.9184635239
      Time just to read binary file: 42.418867292311
      Binary read and process time: 4515.16129830537
      A couple things to note:
      1) The 32 MB ascii file took 10x longer to read than the 15 MB binary file.  Not entirely sure why.  Both were read into a buffer.
      2) The Binary write takes a long time but I made no effort to optimize this because the plan was to write this file one time only so I don't mind if it takes longer to write this file.  I care much more about how long it takes to read the file because I will be reading it many times.
      3) There was a modest gain in converting the ascii file to binary in terms of file size and reading speed.
      So big picture... not sure it's worth the effort to convert the files to binary even though most of the data is numerical data in the binary file.  That was actually surprising as I expected there would be more of a difference.  Any ideas on how to get the binary data to read at a faster rate would be great.
       
      binary.au3
      2019_02_08.zip
    • By FMS
      Hello,
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
      TEST_labels.idx1-ubyte
    • By Ascer
      Hello.
      A couple week ago i've started learning API Interface for different websites. In this fact sometimes you have to encode your "photo" or "document.body" to send request.
      Functions to encode and decode Base64 was already created by others. Unfortunately i have troubles with running it on Windows R 2008, also speed was terrible.
      I try to find alternative way to code data. 
      I've read about Microsoft "XMLDOM" and created a one simple function to Encode / Decode data to base64binary, base64url
      Thanks for Ghads on Wordpress i coverted a part of his lines from VBscript to AutoIT
      ;============================================================================================================================== ; Function: base64($vCode [, $bEncode = True [, $bUrl = False]]) ; ; Description: Decode or Encode $vData using Microsoft.XMLDOM to Base64Binary or Base64Url. ; IMPORTANT! Encoded base64url is without @LF after 72 lines. Some websites may require this. ; ; Parameter(s): $vData - string or integer | Data to encode or decode. ; $bEncode - boolean | True - encode, False - decode. ; $bUrl - boolean | True - output is will decoded or encoded using base64url shema. ; ; Return Value(s): On Success - Returns output data ; On Failure - Returns 1 - Failed to create object. ; ; Author (s): (Ghads on Wordpress.com), Ascer ;=============================================================================================================================== Func base64($vCode, $bEncode = True, $bUrl = False) Local $oDM = ObjCreate("Microsoft.XMLDOM") If Not IsObj($oDM) Then Return SetError(1, 0, 1) Local $oEL = $oDM.createElement("Tmp") $oEL.DataType = "bin.base64" If $bEncode then $oEL.NodeTypedValue = Binary($vCode) If Not $bUrl Then Return $oEL.Text Return StringReplace(StringReplace(StringReplace($oEL.Text, "+", "-"),"/", "_"), @LF, "") Else If $bUrl Then $vCode = StringReplace(StringReplace($vCode, "-", "+"), "_", "/") $oEL.Text = $vCode Return $oEL.NodeTypedValue EndIf EndFunc ;==>base64  
       
    • By Dragonfighter
      I'm searching a way to do xor and shift and if possible also other operations. Thanks in advance for the replies.
×
×
  • Create New...