CSL

How to get arbitrary section of bits from a binary

4 posts in this topic

I want to take a binary data from any source (string,number,files,etc..) and iterate over each X bits of it in a loop, say take bits 1-5, then 6-10, etc.. Then I want to convert these bits to their corresponding decimal value.

but all the binary functions I found in autoit only work with full bytes, and do not let me get smaller sections of bits, like "BinaryMid()" that "Extracts a number of bytes from a binary variant"

Can anyone tell me if this is possible to do and how? and also if there is a function to convert those bits to/from decimals?

I'm not that familiar with dealing directly with binary, so any help or tips will be very appreciated

Context:

I'm trying to make a function to encode/decode any given binary data into/from a given array of characters. just like Base64 but using different bases then [a-z A-Z 0-9 +/= ]. My approach currently is to figure out how many bits of binary I can encode with each character, read those bits and convert them to a decimal number, then I will use that number as an index and take the character at that index from the character array and add it to the result string.

I'm aware that there may be some padding needed.

 

 

Share this post


Link to post
Share on other sites



This is possible to do with the DllStruct- and the Bit-commands. If you want 5 bit blocks you can store 5 bytes at a time from your binary data into a 5 byte structure (40 bits). Then you can handle each of the 5 bit blocks one by one. 8 blocks in a total.

The CPU only handles 1, 2, 4 or 8 (64 bit CPU) bytes at a time. It does not handle 5 or 7 bytes. To handle 5 bytes you can for example split them up in 2 bytes, 2 bytes and 1 byte. If some of the 5 bit blocks have bits spread over byte 2 and 3 or byte 4 and 5 then you must deal with it.

To do all this you will need to be able to handle binary data. And you must be able to convert from binary to decimal numbers in your head. At least for the first 8 bits.

If you have large amounts of binary data AutoIt is not the fastest programming language. You'll need a compiled language.

What's the purpose for this? I do not think it's possible to make better algorithms for simple encoding/decoding, than those that already exists.

Share this post


Link to post
Share on other sites
14 minutes ago, LarsJ said:

This is possible to do with the DllStruct- and the Bit-commands. If you want 5 bit blocks you can store 5 bytes at a time from your binary data into a 5 byte structure (40 bits). Then you can handle each of the 5 bit blocks one by one. 8 blocks in a total.

The CPU only handles 1, 2, 4 or 8 (64 bit CPU) bytes at a time. It does not handle 5 or 7 bytes. To handle 5 bytes you can for example split them up in 2 bytes, 2 bytes and 1 byte. If some of the 5 bit blocks have bits spread over byte 2 and 3 or byte 4 and 5 then you must deal with it.

To do all this you will need to be able to handle binary data. And you must be able to convert from binary to decimal numbers in your head. At least for the first 8 bits.

If you have large amounts of binary data AutoIt is not the fastest programming language. You'll need a compiled language.

What's the purpose for this? I do not think it's possible to make better algorithms for simple encoding/decoding, than those that already exists.

Thanks for the reply LarsJ,  I did some searching before and tried the bit shifting method but it got complicated fast, I will only deal with small binary data so I instead chose to convert the binary to 0/1 strings and handling that instead, pretty dirty and slow, I Know.

Are there any existing encoding functions/libraries that can encode to an arbitrary list of characters? this is exactly what I need.

This function is for a little encryption based game I'm doing for fun, I know autoit may not be the best fit but it's fast to write & easy to use, and it's just a small project.

Share this post


Link to post
Share on other sites

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • Jibberish
      By Jibberish
      I need to read log files into an array to search for errors. However when I display the array I get garbage or "chinese characters". Our developers say they are using UTF-8, but FileGetEncoding says the logs are "2048" or $FO_UTF16_BE_NOBOM (2048) = Use Unicode UTF16 Big Endian (without BOM) from the Encoding codes in FileOpen().
      There is an app called Detenc that detects the encoding used by files. You have to guess, but it returns correctly when I set the Encoder for UTF-8. I understand Encoding is not etched in stone, but the first character of the file is a capital B, using HxD Hex Editor.
      I even have another  topic here about running PowerShell to reencode the file so AutoIt will store the file properly in the array - See:
      So I am trying to figure out why AutoIt thinks my logs are not UTF-8.
      Here is sample code:
      #include <array.au3> #include <File.au3> Local $aRetArrayFile _FileReadToArray("C:\Logs\Myplayer1.log", $aRetArrayFile) _ArrayDisplay($aRetArrayFile) I won't post the results as it is illegible, but I did attach a screenshot of the _ArrayDisplay results, and this is the first line of the Log file:
      BANNER 10/10/2017 15:56:00 ====================================================================== And the Hex from the beginning of the file:
      42 41 4E 4E 45 52 20 31 30 2F 31 30 2F 32 30 31 37 20 31 34 3A 33 31 3A 33 35 20 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 0D 0A 42 41 4E 4E 45 52 20 So I don't understand why AutoIt thinks the file is UTF16 BE.
      If I can get the Powershell script running, I have a workaround.
      BTW none of my other arrays display as garbage, just the log files.
      Weird.
      Rereading my post, what seems to be missing is the question. I guess my question is, does anyone know why these logs are being displayed incorrectly?
      Cheers
      Jibs

    • Jibberish
      By Jibberish
      Hi all,
      I need to read a log file into an array, but the log file is encoded as $FO_UTF16_BE_NOBOM (2048) = Use Unicode UTF16 Big Endian (without BOM) per FileGetEncoding (it returns 2048).
      I have searched how to convert these log files to UTF-8 and finally found a Powershell command. Since then I have been racking my brain trying to get the function to work. The command itself works from a Powerscript prompt:
      C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command Get-Content C:\Logs\Myplayer_10-10-17-02-31.log | Set-Content -Encoding utf8 C:\Logs\Myplayer1.log This is my sandbox;
      #include <array.au3> #include <File.au3> Local $aArrayLogFile Local $sLogDir = "C:\Logs\" Local $sLogFile = "Myplayer_10-10-17-02-31.log" Local $sConvertedLog = "ConvertedLog.log" Local $sLogDirFile = $sLogDir&$sLogFile RunWait("C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command Get-Content "&$sLogDirFile&" | Set-Content -Encoding utf8 "&$sConvertedLog,$sLogDir) _FileReadToArray($sLogDirFile, $aArrayLogFile) _ArrayDisplay($aArrayLogFile) Also tried
      RunWait("C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command Get-Content "&$sLogDirFile&" | Set-Content -Encoding utf8 "&$sConvertedLog,$sLogDir) and
      ShellExecuteWait("C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"," -Command Get-Content "&$sLogDirFile&" | Set-Content -Encoding utf8 "&$sConvertedLog,$sLogDir) Tried without -Command and a bunch of other parameters that were sprinkled throughout the internet from people trying to get this to work.
      Thanks
      Jibs
    • ur
      By ur
      When I am trying to compile the autoit files with aut2exe.
      I am getting below error.
      There is no issue in code as the same code is getting compiled on different machine.
      I tried reinstalling the AUtoIT, but the issue replicates.

      Any suggestions?
    • wakillon
      By wakillon
      BinaryToAu3Kompressor v1.0.5.4
       

       
      It's now possible to see the best compression ratio using LZMA, LZNT and Base64 compressions with differents combinations.
      Nothing too complicate, you drag'n drop a file on the picture and script Test all compression types and return the ratios.
      ( Test duration depends of file size, slowest compression is LZNT, but all decompressions are fast  )
      Free to you after, to choose the compression(s) you want...
      Yes, LZMA needs a dll ( embedded & compressed in script ) but brings a powerfull compression. 
      It opens scite with your file compressed to an au3 script with or without decompression function as you want.
      Hold Left Shift key when clicking button for just copy script to clipboard.
      Use the 3 compressions at a time works but doesn't give a good ratio, that's why i don't display it.
      Usefull for little files you want include in your scripts !
      No externals files needed, they are already in script.
      Previous downloads : 1103
      Source and Executable
      BinaryToAu3Kompressor will be added to the next version of >SciTEHopper
      Thanks to Ward for his >Base64.au3 and LZMA.au3, and trancexx for his >LZNT functions and his >Base64Decode function.