Sign in to follow this  
Followers 0
trancexx

_Base64Encode, _Base64Decode

11 posts in this topic

#1 ·  Posted (edited)

I guess there is no need to explain what Base64 encoding is and what is used for.

What is special about this way. Well, I did some testing and it turns that it's fast. About three times faster than the fastest way posted here on this forum (machine code implementation).

It's calling functions "CryptBinaryToString" and "CryptStringToBinary" in Crypt32.dll

I'm in the phase when discovering functions DllCall(), DllStructCreate(),... and related, so please be gentle to me, lol

There is no error checking mainly because of the sentence before this one. If someone is willing to add that to the code below (or correct possible mistakes), I'm willing to learn out of that. So, please do.

Here's the script:

Updated 29th October 2008 - error checking added and some code optimization

$encoded = _Base64Encode("Testing")
MsgBox(0, 'Base64 Encoded', $encoded)


$decoded = _Base64Decode($encoded)
MsgBox(0, 'Base64 Decoded - binary', $decoded)
MsgBox(0, 'Base64 Decoded - string', BinaryToString($decoded))

Func _Base64Encode($input)
    
    $input = Binary($input)
    
    Local $struct = DllStructCreate("byte[" & BinaryLen($input) & "]")
    
    DllStructSetData($struct, 1, $input)
    
    Local $strc = DllStructCreate("int")
    
    Local $a_Call = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($struct), _
            "int", DllStructGetSize($struct), _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($strc))
    
    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf
    
    Local $a = DllStructCreate("char[" & DllStructGetData($strc, 1) & "]")
    
    $a_Call = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($struct), _
            "int", DllStructGetSize($struct), _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($strc))
    
    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error encoding
    EndIf
    
    Return DllStructGetData($a, 1)

EndFunc   ;==>_Base64Encode


Func _Base64Decode($input_string)
    
    Local $struct = DllStructCreate("int")
    
    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf

    Local $a = DllStructCreate("byte[" & DllStructGetData($struct, 1) & "]")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)
    
    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error decoding
    EndIf

    Return DllStructGetData($a, 1)
    
EndFunc   ;==>_Base64Decode

There is @CRLF every 64 chars in return of _Base64Encode().

_Base64Decode() will return binary data. That is intentional to avoid Chr(0) issue. Convert it to string using BinaryToString()

Microsoft about requirements:

Client - Requires Windows Vista or Windows XP.

Server - Requires Windows Server 2008 or Windows Server 2003.

Edited by trancexx
2 people like this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites



Ha... Nice find.


Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

Share this post


Link to post
Share on other sites

LOL, Nice indeed! Post some times so people get an idea of how fast it is.

I bet Crypt32.dll goes back to win9x. Anyone know what other potentially useful functions are inside the dll?

http://msdn.microsoft.com/en-us/library/aa...28VS.85%29.aspx

Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

LOL, Nice indeed! Post some times so people get an idea of how fast it is...

Here are some times, but I guess to get it right more tests should be done (and not in one script because of memory allocation... thing). Three ways are compared. Output size is the number of characters in output string (not counting @CRLF) to verify results. I used older version of machine code _Base64Encode() because the new one tend to crash (Ward, do something).

File size: 0.00664329528808594 MB
Encoding times:

Crypt32.dll: 0.669358815156675 ms
Output size: 9288 

Machine code: 0.57353658076655 ms
Output size: 9288

Mikeytown2, blindwig: 89.5381954969137 ms
Output size: 9288 

---------------------------------------------------------------------------

File size: 0.726992607116699 MB
Encoding times:

Crypt32.dll: 60.3733156029607 ms
Output size: 1016412 

Machine code: 107.127302492356 ms
Output size: 1016412

Mikeytown2, blindwig: 7199.77846346393 ms
Output size: 1016412

---------------------------------------------------------------------------

File size: 5.83719635009766 MB
Encoding times:

Crypt32.dll: 579.908060940706 ms
Output size: 8160992 

Machine code: 838.349008044318 ms
Output size: 8160992 

Mikeytown2, blindwig: 224483.446918533 ms
Output size: 8160992

---------------------------------------------------------------------------

File size: 21.3670444488525 MB
Encoding times:

Crypt32.dll: 8545.42518672066 ms
Output size: 29873296

Machine code: 17715.4873035539 ms
Output size: 29873296

Mikeytown2, blindwig: ~

Difference is even bigger for larger files but that's too extreme, even to write about.

btw, i got this poking dlls (out of curiosity) with this


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

Here are some times, but I guess to get it right more tests should be done (and not in one script because of memory allocation... thing). Three ways are compared. Output size is the number of characters in output string (not counting @CRLF) to verify results. I used older version of machine code _Base64Encode() because the new one tend to crash (Ward, do something).

Difference is even bigger for larger files but that's too extreme, even to write about.

btw, i got this poking dlls (out of curiosity) with this

OKOK, I will fix it. The newer base64 UDF crash because the formula to count the buffer size is wrong (A stupid bug). Thank you for reminding me.

However, your time trial has a little problem. In fact, you are testing the speed of disk reading, autoit string handling, garbage collection and so on.

Because the speed of base64 encoding is much faster than these actions.

In following code I don't set dllstruct to 0, and then it should run as fast as using crypt32.dll.

BTW, a better way to encode a file into base64 is to split it and encode part by part. Handling a huge string in script language is always slow.

Func _Base64Decode($Data)
    Local $Opcode = "0xC81000005356578365F800E8500000003EFFFFFF3F3435363738393A3B3C3DFFFFFF00FFFFFF000102030405060708

090A0B0C0D0E0F10111213141516171819FFFFFFFFFFFF1A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132338F

45F08B7D0C8B5D0831D2E9910000008365FC00837DFC047D548A034384C0750383EA033C3D75094A803B3D75014AB00084C0

751A837DFC047D0D8B75FCC64435F400FF45FCEBED6A018F45F8EB1F3C2B72193C7A77150FB6F083EE2B0375F08A068B75FC

884435F4FF45FCEBA68D75F4668B06C0E002C0EC0408E08807668B4601C0E004C0EC0208E08847018A4602C0E00624C00A46

038847028D7F038D5203837DF8000F8465FFFFFF89D05F5E5BC9C21000"
    
    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    Local $Ouput = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    Local $Ret = DllCall("user32.dll", "int", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "str", $Data, _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "int", 0, _
                                                    "int", 0)

    Return BinaryMid(DllStructGetData($Ouput, 1), 1, $Ret[0])
EndFunc

Func _Base64Encode($Data, $LineBreak = 76)
    Local $Opcode = "0x5589E5FF7514535657E8410000004142434445464748494A4B4C4D4E4F505152535455565758595A61626364656667

68696A6B6C6D6E6F707172737475767778797A303132333435363738392B2F005A8B5D088B7D108B4D0CE98F0000000FB633

C1EE0201D68A06880731C083F901760C0FB6430125F0000000C1E8040FB63383E603C1E60409C601D68A0688470183F90176

210FB6430225C0000000C1E8060FB6730183E60FC1E60209C601D68A06884702EB04C647023D83F90276100FB6730283E63F

01D68A06884703EB04C647033D8D5B038D7F0483E903836DFC04750C8B45148945FC66B80D0A66AB85C90F8F69FFFFFFC607

005F5E5BC9C21000"

    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    $Data = Binary($Data)
    Local $Input = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Input, 1, $Data)

    $LineBreak = Floor($LineBreak / 4) * 4
    Local $OputputSize = Ceiling(BinaryLen($Data) * 4 / 3) 
    $OputputSize = $OputputSize + Ceiling($OputputSize / $LineBreak) * 2 + 4

    Local $Ouput = DllStructCreate("char[" & $OputputSize & "]")
    DllCall("user32.dll", "none", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "ptr", DllStructGetPtr($Input), _
                                                    "int", BinaryLen($Data), _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "uint", $LineBreak)
    Return DllStructGetData($Ouput, 1)
EndFunc

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

I absolutely love the simplicity of your machine code functions. I found them to be beautiful (like a beautiful woman). Probably I'm crazy and everything, khm, khm...

You should make a tutorial on how to get machine code. Don't be lazy, do it. You should find rewarding having people learn of you.

Yes, you are right, seems that now they are as fast.

Return variable of DllStructCreate() in our case is local and releasing allocated memory by setting this variable to 0 is not needed. That was my thinking. Is this true, but completely true?

About splitting...

That should be done very consideringly to avoid possible mistakes due to the fact how base64 encoding is done (3 --> 4). Size of the portions would also depend on how long is every line of output sting (@CRLF we add for SMTP servers).

So, portion lenght should be (4*3*LINE_BREAK)*n. "n" should be determined experimentally to accomplish best performance.

Maybe would be faster, but looping with AutoIt is relatively slow... Someone should do it and test it.

I updated first post (still no error checking).

This how Wards _Base64Encode() looks with me:

Func _Base64Encode($Data, $LineBreak = 76)
    
    Local $Opcode = "0x5589E5FF7514535657E8410000004142434445464748494A4B4C4D4E4F505152" _
                  & "535455565758595A6162636465666768696A6B6C6D6E6F707172737475767778" _
                  & "797A303132333435363738392B2F005A8B5D088B7D108B4D0CE98F0000000FB6" _
                  & "33C1EE0201D68A06880731C083F901760C0FB6430125F0000000C1E8040FB633" _
                  & "83E603C1E60409C601D68A0688470183F90176210FB6430225C0000000C1E806" _
                  & "0FB6730183E60FC1E60209C601D68A06884702EB04C647023D83F90276100FB6" _
                  & "730283E63F01D68A06884703EB04C647033D8D5B038D7F0483E903836DFC0475" _
                  & "0C8B45148945FC66B80D0A66AB85C90F8F69FFFFFFC607005F5E5BC9C21000" 

    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    $Data = Binary($Data)
    Local $Input = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Input, 1, $Data)

    $LineBreak = Floor($LineBreak / 4) * 4
    Local $OputputSize = Ceiling(BinaryLen($Data) * 4 / 3) 
    $OputputSize = $OputputSize + Ceiling($OputputSize / $LineBreak) * 2 + 4

    Local $Ouput = DllStructCreate("char[" & $OputputSize & "]")
    DllCall("user32.dll", "none", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "ptr", DllStructGetPtr($Input), _
                                                    "int", BinaryLen($Data), _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "uint", $LineBreak)
    Return DllStructGetData($Ouput, 1)
    
EndFunc

It's beautiful, right? Specially when you think what job it does.

... I'm crazy :)

Edited by trancexx
1 person likes this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

I absolutely love the simplicity of your machine code functions. I found them to be beautiful (like a beautiful woman). Probably I'm crazy and everything, khm, khm...

You should make a tutorial on how to get machine code. Don't be lazy, do it. You should find rewarding having people learn of you.

I don't make a tutorial because it has a little complicated. The bigest problem is a machine code function using in AutoiIt can't have a static memory reference.

As a result, we can't use windows API, and even a simple string needs special technique to locate.

I think a better way is to use MemoryDllCall, because everyone know how to make a DLL. And it is much easier than getting machine code.

If you are really interested in it, here is a demo. It is the source of my Base64Encode written in C--.

Maybe a little assembly knowledge is necessary to understand it.

#pragma option dll

int _export Base64Encode(dword input, len, buffer, lineBrak) uses ESI EDI EBX
{
    dword newLine = lineBrak;

    $CALL NEAR START
    $DB "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
START:
    $POP EDX

    EBX = input;
    EDI = buffer;
    ECX = len;

    while(signed ECX > 0)
    {
        DSBYTE[EDI] = DSBYTE[DSBYTE[EBX] >> 2 + EDX];

        EAX = 0;
        IF(ECX > 1) EAX = DSBYTE[EBX + 1] & 0xf0 >> 4;
        DSBYTE[EDI + 1] = DSBYTE[DSBYTE[EBX] & 3 << 4 | EAX + EDX];

        IF(ECX > 1)
        {
            EAX = DSBYTE[EBX + 2] & 0xc0 >> 6;
            DSBYTE[EDI + 2] = DSBYTE[DSBYTE[EBX + 1] & 0xf << 2 | EAX + EDX];
        } ELSE DSBYTE[EDI + 2] = '=';

        IF(ECX > 2) DSBYTE[EDI + 3] = DSBYTE[DSBYTE[EBX + 2] & 0x3f + EDX];
        ELSE DSBYTE[EDI + 3] = '=';

        EBX += 3;
        EDI += 4;
        ECX -= 3;
        newLine -= 4;
        IF(ZEROFLAG)
        {
            newLine = lineBrak;
            AX = 0x0A0D;
            $stosw
        }
    }

    DSBYTE[EDI] = 0;
    return;
}

Yes, you are right, seems that now they are as fast.

Return variable of DllStructCreate() in our case is local and releasing allocated memory by setting this variable to 0 is not needed. That was my thinking. Is this true, but completely true?

Yes, it is true, I had tested it.

But I still don't know why setting it to 0 is so slow.

Edited by Ward

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

I don't make a tutorial because it has a little complicated. The bigest problem is a machine code function using in AutoiIt can't have a static memory reference.

As a result, we can't use windows API, and even a simple string needs special technique to locate.

I think a better way is to use MemoryDllCall, because everyone know how to make a DLL. And it is much easier than getting machine code.

If you are really interested in it, here is a demo. It is the source of my Base64Encode written in C--.

Maybe a little assembly knowledge is necessary to understand it.

#pragma option dll

int _export Base64Encode(dword input, len, buffer, lineBrak) uses ESI EDI EBX
{
    dword newLine = lineBrak;

    $CALL NEAR START
    $DB "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
START:
    $POP EDX

    EBX = input;
    EDI = buffer;
    ECX = len;

    while(signed ECX > 0)
    {
        DSBYTE[EDI] = DSBYTE[DSBYTE[EBX] >> 2 + EDX];

        EAX = 0;
        IF(ECX > 1) EAX = DSBYTE[EBX + 1] & 0xf0 >> 4;
        DSBYTE[EDI + 1] = DSBYTE[DSBYTE[EBX] & 3 << 4 | EAX + EDX];

        IF(ECX > 1)
        {
            EAX = DSBYTE[EBX + 2] & 0xc0 >> 6;
            DSBYTE[EDI + 2] = DSBYTE[DSBYTE[EBX + 1] & 0xf << 2 | EAX + EDX];
        } ELSE DSBYTE[EDI + 2] = '=';

        IF(ECX > 2) DSBYTE[EDI + 3] = DSBYTE[DSBYTE[EBX + 2] & 0x3f + EDX];
        ELSE DSBYTE[EDI + 3] = '=';

        EBX += 3;
        EDI += 4;
        ECX -= 3;
        newLine -= 4;
        IF(ZEROFLAG)
        {
            newLine = lineBrak;
            AX = 0x0A0D;
            $stosw
        }
    }

    DSBYTE[EDI] = 0;
    return;
}
This actually helps a lot. Can you please post some simpler example (adding two numbers). Original language source and complete function in AutoIt using machine code. That would help me with patterns. I will try to comprehend.
1 person likes this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

Is there a COMPLETED UDF for this anywhere?


Everseeker

Share this post


Link to post
Share on other sites

The code in first post is a complete UDF.

Since the code is only implementing wrapper functions for CryptBinaryToString and CryptStringToBinary in crypt32.dll, the code should be working, if the functions in the DLL are working (provided that the wrapper functions are coded correctly, but I think they are).

Although this seems to be an old example, the age really has no meaning, since it's only wrapper functions. What has meaning in terms of age, is the age of the DLL. Even on my old XP crypt32.dll is dated 2013-10-07.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0