Ward

Binary UDF

20 posts in this topic

#1 ·  Posted (edited)

"Binary" is a built-in variable type in AutoIt. But functions that deal with binary variable are inadequate. Someone says string functions can also deal with binary variable, but actually, it may cause some problem. For example, using StringInStr to search "0x11" in "0x0110" will return a false result.

 

This UDF solves the problem. It provide a lot of binary related functions. Furthermore, most functions in the UDF are optimized by machine code, so the speed is incredible.

 

Here is the function list:

 

_BinaryAnd($Binary1, $Binary2)
_BinaryOR($Binary1, $Binary2)
_BinaryXOR($Binary1, $Binary2)
_BinaryNot($Binary)
_BinaryShift($Binary, $Shift)
_BinaryRotate($Binary, $Shift)
_BinaryReverse($Binary)
_BinaryInBin($Binary, $Search, $Occur = 1, $Start = 1)
_BinaryReplace($Binary, $Search, $Replace, $Occur = 0)
_BinarySplit($Binary, $Search)
_BinaryRight($Binary, $Count)
_BinaryLeft($Binary, $Count)
_BinaryTrimLeft($Binary, $Count)
_BinaryTrimRight($Binary, $Count)

The usage of these functions are just like the built-in StringXXXX or BitXXXX functions.

 

Attentions:

BinarySplit has a little bit different, it don't support flag parameter in StringSplit.

BinaryInBin and BinaryReplace also don't need casesense parameter.

 

 

_BinaryPeek($Binary, $Start, $Type = "byte")
_BinaryPoke($Binary, $Start, $Value, $Type = "byte")
_BinaryToInt32($Binary)
_BinaryFromInt32($Value)
_BinaryToInt64($Binary)
_BinaryFromInt64($Value)
_BinaryRandom($Length, $Start = 0, $To = 255)

These functions are some helper function.

 

BinaryPeek and BinaryPoke work like stardand BASIC keyword peek and poke. They can handle types defined in DllStructCreate and DllCall (include str and wstr). The $Start are also like other AutoIt functions, count from 1, not from 0.

 

BinaryRandom generate a random binary data for test or something. It has built-in Mersenne Twister PRNG.

 

BinaryToXXX and BinaryFromXXX convert variable type between binary and int32/int64.

 

 

_BitShift64($Value, $Shift)
_BitRotate64($Value, $Shift)
_BitNOT64($Value)
_BitOR64($Value1, $Value2, [$Value3, ...])
_BitAND64($Value1, $Value2, [$Value3, ...])
_BitXOR64($Value1, $Value2, [$Value3, ...])
_Hex64($Value, $Length = 16)
_Dec64($HexString)

These functions are just the same as built-in functin in the same name, but these can handle 64 bit signed integer.

(Built-in function only for 32 bit integer)

 

Binary UDF.zip

 

 

2011/09/15 Update Note:

  • Add following functions:

     

    _BinaryToInt16($Binary)
    _BinaryFromInt16($Value)
    _BinaryToDouble($Binary)
    _BinaryFromDouble($Value)
    _BinaryToMemory($Binary, $Ptr)
    _BinaryFromMemory($Ptr, $Size)
    _BinaryToDLLStruct($Binary)
    _BinaryFromDLLStruct(ByRef $DLLStruct)
    _BinaryFillDLLStruct($Binary, ByRef $DLLStruct)
  • _BinaryRandom can specify the random seed now. $Seed = 0 mean no seed is specified.

     

    _BinaryRandom($Length, $Start = 0, $To = 255, $Seed = 0)
  • Use static variable to store the binary code to avoid repeat memory allocation.

Binary UDF.zip

Edited by Ward
2 people like this

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites



Excellent! The 64-bit binary functions will really be useful.

Share this post


Link to post
Share on other sites

Hi Ward!

Thank you for this UDF

I have a problem: I need to replace a few bytes of binary data. But replace the data is longer than the original data. In this situation it would fit function _BinaryReplace, but it changes, not where I want. I know the exact start Offset from which data and their length.

example:

Input data: 00 00 11 00 11 00 11 00 00 00 ; I need to replace the red

Output data: 00 00 11 00 FF FF 00 11 00 00 00 ; like this

or

Input data: 00 00 11 00 11 00 11 00 00 00

Output data: 00 00 11 00 FF 11 00 00 00

I do not know exactly what it is first or last, but I know the offset and size

It is possible to refine the function _BinaryReplace, so she was not looking for a string and tell it the exact offset

sorry for bad english

Share this post


Link to post
Share on other sites

Hi Ward!

Thank you for this UDF

I have a problem: I need to replace a few bytes of binary data. But replace the data is longer than the original data. In this situation it would fit function _BinaryReplace, but it changes, not where I want. I know the exact start Offset from which data and their length.

Since you know the offset to start replace, maybe you can use BinaryMid() to get part of binary data before _BinaryReplace().

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Interesting :) ... Bummer I don't have 64bit capabilities yet. :)

---

Collection some stuff while playing around with it ...

  • Could use some '#forceref's (5x) to make it not trip on main scripts that use '#AutoIt3Wrapper_AU3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7'
  • Some way to control random with some seed feature. (if possible) :)
  • _BinaryRandom() ... speed is, well :mellow:
    [1] |St| A | 1.74 mSec.   | 256 * Random()
    [2] |St| B | 145.95 mSec. | 65536 * Random()
    [3] |St| A | 105.52 mSec. | 256 * _BinaryRandom(8)
    [4] |St| B | 26.31 Sec.   | 65536 * _BinaryRandom(8)
    Probably because function startup penalty. ... (taking a other approche to using it) ...
    [4] |St| B | 31.68 mSec.  | _BinaryRandom(8*65536)
    That's more like it. But Random() not leaving much slack to play around with.
  • ...

Edited by iEvKI3gv9Wrkd41u

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

@iEvKI3gv9Wrkd41u

Thanks for your testing.

_BinaryRandom (or other functions use machine code) convert HEX to binary and call it every time.

A way to avoid it is use static variable, for example:

Func _BinaryRandom2($Length, $Start = 0, $To = 255)
    If $Length = 0 Then Return Binary("")
    Static $CodeBufferPtr

    If Not $CodeBufferPtr Then
        Local $Code
        If @AutoItX64 Then
            $Code = Binary("0x4157415641554154555756534881ECF80900004538C876034587C185D20F847F0100004C8D5C2410450FB6C9410FB6C04183C10183EA0148C78424D0090000000000004129C1498D8338060000488D5C1101498D730431D2498DBB3406000048894424084D8D93C0090000EB4789C2C1EA0B31C289D0C1E0072580562C9D31D031D289C5C1E50F81E50000C6EF31C589E8C1E81231E841F7F1418D141088114883C1014839D90F84F6000000488B9424D00900004885D20F84850000004C8BBC24E00900004C8BA424D80900004D8D6F044C89E0448B308B2A4C89A424D00900004983C4044C89AC24E00900004C89A424D80900004489F081E5000000804183E60125FFFFFF7F41F7DE09E84181E6DFB00899D1E84431F04133074D39D5890274164D39D40F8542FFFFFF4C899C24D8090000E935FFFFFF4C899C24E0090000EBE00F3131D04989F4BD010000008944241089C2C1EA1E31C269C26589076C01E883C501418904244983C40481FD7002000075DE4C899C24D00900004889B424D80900004889F04889BC24E00900004C8B6C24084C89DA4989FF4989F4E932FFFFFF4881C4F80900005B5E5F5D415C415D415E415FC3")
        Else
            $Code = Binary("0x5557565381ECE40900000FB68424000A0000884424130FB68424040A000038442413760A0FB654241392885424138B9424FC09000085D20F84740100000FB65424130FB6C031DB83C0018DB424D8090000C78424D8090000000000008D4C241829D031D28944240C893424EB5289C2C1EA0B31C289D0C1E0072580562C9D31D031D289C6C1E60F81E60000C6EF31C689F0C1E81231F0F774240C8B8424F80900000254241388141883C301399C24FC0900000F86F90000008B9424D809000085D20F84850000008BAC24E00900008BBC24DC0900008D750489F8897424048B008B3289BC24D809000083C70489BC24DC0900008944240825FFFFFF7F81E60000008009F08B742408D1E883E601F7DE81E6DFB0089931F08B342433450089028B54240439F2899424E009000074153B3C240F8536FFFFFF898C24DC090000E92AFFFFFF898C24E0090000EBE20F3131D0BE010000008944241889C2C1EA1E31C269C26589076C01F08904B183C60181FE7002000075E38D44241C8D94244C06000089C78DB42450060000899424E00900008DAC244C06000089CA898C24D8090000898424DC09000089742404E92DFFFFFF81C4E40900005B5E5F5DC21000")
        EndIf
        $CodeBufferPtr = __BinaryCodeBufferAlloc($Code)
    EndIf

    Local $Buffer = DllStructCreate("byte[" & $Length & "]")
    Local $Ret = DllCall($__Binary_User32Dll, "uint", "CallWindowProc", "ptr", $CodeBufferPtr, _
                                                                        "ptr", DllStructGetPtr($Buffer), _
                                                                        "uint", $Length, _
                                                                        "byte", $Start, _
                                                                        "byte", $To)
    Return DllStructGetData($Buffer, 1)
EndFunc

This function is faster than the old one.

But the memory space can't be free (it seems not important).

Is convert all functions to this format necessary?


新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

I think the Static method is good practice to avoid repeated memory allocations. You could always build in a memory clearing feature to the function if it is important to free that buffer at some point.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

This function is faster than the old one.

Definitely puts a nice dent in the time needed.
[5] |St| B    | 7.51 Sec.    | 65536 * _BinaryRandom_Static(8)

But the memory space can't be free (it seems not important).

Yep. Speed versus Memory use. Don't think where talking that much memory here to. (The Asm code only by the look of it.)

Is convert all functions to this format necessary?

I personally would use Static ... Without it the general function call(s) seem way to slow for any serious(high call rate) use. Even with Static use the function startup penalty's are not to be ignored. (for those functions that are available in AutoIt itself -> 32-bit one's)

e: forum troubles ...

---

Alternative (older) way/topic for some 64-bit support: by jchd.

Edited by iEvKI3gv9Wrkd41u

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

@iEvKI3gv9Wrkd41u

What you suggested is added now.


新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

Interesting thread. I occasionally use bitwise functions, but I mostly use string functions with binary strings. Why? Perhaps, as you say,. because the available bitwise functions seem inflexible and extremely limited. The motivation for creating my modal UDF was aimed at modular arithmatic, binary being the first target base. I don't know all the methods and I still believe that there must be other solutions (perhaps using Struct which I haven't tried yet). My recent three dimensional bit spin topic uses strings of ones and zeros, because the bitwise functions don't hack it.

Share this post


Link to post
Share on other sites

@iEvKI3gv9Wrkd41u

What you suggested is added now.

Cool. :graduated:

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Just to see if _BinaryRandom() is compatible in some way with AutoIt's Random().

#include "Binary.au3"
Global Const $INT32u = Int('0x0100000000') ;; (2^32), int64. (4294967296)

_Test5(8, 2, 1000) ;; Count, Seed, LoopIt.

Func _Test5(Const $iCount = 1, Const $iSeed=0, Const $iLoopIt = 1) ;; $iSeed=0 -> random seed.
    Local Const $iBytes = 1, $iMax = Int(2^(8*$iBytes))-1 ;0, 255, 65535, ...
    Local $R = ''

    For $i = 1 to $iLoopIt

    ;; AutoIt random() output.
    SRandom($iSeed)
    Local $bFirstBytes = Binary('')
    For $1 = 1 To $iCount
        $R = Floor(Random() * ($INT32u + 1))
        $bFirstBytes &= BinaryMid($R,1,1)
    Next

    ;; _BinaryRandom() output.
    $R =  _BinaryRandom($iCount, 0, $iMax, $iSeed)

    If $R <> $bFirstBytes Then
        MsgBox(0,'MsgBox','Oops: $R <> $bFirtsBytes ?')
        Exit 911
    EndIf

    Next
    MsgBox(0,'MsgBox',' :) ')
EndFunc

;)

Although I do wonder a bit what happened to the other bytes. :graduated:

E: Oops, forgot some things.

+ _BinaryRandom() probably using one Byte where Random it using 4 bytes.

Edited by iEvKI3gv9Wrkd41u

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

Just thought I'd toss out there that the UDF works great with the new DllCallAddress() function in 3.3.7.15 beta (ignore the Au3Check warnings about wrong number of parameters, that will be fixed in the next beta). Anyone feel like running a speed test versus CallWindowProc? This would also allow for an eventual rewrite with more sane parameters.

Edited by wraithdu

Share this post


Link to post
Share on other sites

Finally we have standard support for DllCallAddress. Not need my MemoryFuncCall anymore. :graduated:. I can develop my machine code library more easily.

I thank DllCallAddress should much faster than MemoryFuncCall, but have no much difference to CAllWindowProc. (just guess, not yes test.)


新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

Hi Ward,

Sorry to post on this old topic. however, I looking for assistance, I am looking parse large binary data, most of it is 0x00, which I can replace with binaryreplace. So that works amazing.

but I am looking to remove everything below 0x20(except 0xA and 0xD) and 0x80 and above. While I could use binaryreplace to go through all of them using loop, I was wondering if there was a way to use this function like _BinaryReplace($Binary, '>0x7F', '0x')

Thanks,

Stephen

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

Try this:

Local $ansi
For $i = 0 To 255
    $ansi &= Chr($i)
Next
Local $ascii7 = StringRegExpReplace($ansi, "[^\x0A\x0D\x20-\x7F]", "")
_ArrayDisplay(StringToASCIIArray($ascii7))

BTW do you really want to keep 0x7F (the "delete" control character)?

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

Thanks jchd, I do not think I would have ever thought about using regex this way.

And your right I did not want 0x7F

Edited by step887

Share this post


Link to post
Share on other sites

#18 ·  Posted

Where can I find this UDF?

It seems the links are not valid anymore ...

Share this post


Link to post
Share on other sites

#20 ·  Posted

The one you have (from 2011) is the most recent version, I'm not sure why Ward removed all of his UDFs.

1 person likes this

UHJvZmVzc2lvbmFsIENvbXB1dGVyZXI=

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now