8 replies to this topic

[trancexx]

I guess there is no need to explain what Base64 encoding is and what is used for.

What is special about this way. Well, I did some testing and it turns that it's fast. About three times faster than the fastest way posted here on this forum (machine code implementation).
It's calling functions "CryptBinaryToString" and "CryptStringToBinary" in Crypt32.dll

I'm in the phase when discovering functions DllCall(), DllStructCreate(),... and related, so please be gentle to me, lol
There is no error checking mainly because of the sentence before this one. If someone is willing to add that to the code below (or correct possible mistakes), I'm willing to learn out of that. So, please do.

Here's the script:

Updated 29th October 2008 - error checking added and some code optimization

AutoIt        expandcollapse  popup

$encoded = _Base64Encode("Testing")
MsgBox(0, 'Base64 Encoded', $encoded)


$decoded = _Base64Decode($encoded)
MsgBox(0, 'Base64 Decoded - binary', $decoded)
MsgBox(0, 'Base64 Decoded - string', BinaryToString($decoded))

Func _Base64Encode($input)
    
    $input = Binary($input)
    
    Local $struct = DllStructCreate("byte[" & BinaryLen($input) & "]")
    
    DllStructSetData($struct, 1, $input)
    
    Local $strc = DllStructCreate("int")
    
    Local $a_Call = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($struct), _
            "int", DllStructGetSize($struct), _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($strc))
    
    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf
    
    Local $a = DllStructCreate("char[" & DllStructGetData($strc, 1) & "]")
    
    $a_Call = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($struct), _
            "int", DllStructGetSize($struct), _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($strc))
    
    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error encoding
    EndIf
    
    Return DllStructGetData($a, 1)

EndFunc   ;==>_Base64Encode


Func _Base64Decode($input_string)
    
    Local $struct = DllStructCreate("int")
    
    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf

    Local $a = DllStructCreate("byte[" & DllStructGetData($struct, 1) & "]")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)
    
    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error decoding
    EndIf

    Return DllStructGetData($a, 1)
    
EndFunc   ;==>_Base64Decode

There is @CRLF every 64 chars in return of _Base64Encode().
_Base64Decode() will return binary data. That is intentional to avoid Chr(0) issue. Convert it to string using BinaryToString()

Microsoft about requirements:
Client - Requires Windows Vista or Windows XP.
Server - Requires Windows Server 2008 or Windows Server 2003.

Edited by trancexx, 29 October 2008 - 10:04 PM.

[SmOke_N]

Ha... Nice find.

[mikeytown2]

LOL, Nice indeed! Post some times so people get an idea of how fast it is.

I bet Crypt32.dll goes back to win9x. Anyone know what other potentially useful functions are inside the dll?

[SmOke_N]

LOL, Nice indeed! Post some times so people get an idea of how fast it is.

I bet Crypt32.dll goes back to win9x. Anyone know what other potentially useful functions are inside the dll?

http://msdn.microsoft.com/en-us/library/aa...28VS.85%29.aspx

[trancexx]

LOL, Nice indeed! Post some times so people get an idea of how fast it is...

Here are some times, but I guess to get it right more tests should be done (and not in one script because of memory allocation... thing). Three ways are compared. Output size is the number of characters in output string (not counting @CRLF) to verify results. I used older version of machine code _Base64Encode() because the new one tend to crash (Ward, do something).

Plain Text        expandcollapse  popup

File size: 0.00664329528808594 MB
Encoding times:

Crypt32.dll: 0.669358815156675 ms
Output size: 9288 

Machine code: 0.57353658076655 ms
Output size: 9288

Mikeytown2, blindwig: 89.5381954969137 ms
Output size: 9288 

---------------------------------------------------------------------------

File size: 0.726992607116699 MB
Encoding times:

Crypt32.dll: 60.3733156029607 ms
Output size: 1016412 

Machine code: 107.127302492356 ms
Output size: 1016412

Mikeytown2, blindwig: 7199.77846346393 ms
Output size: 1016412

---------------------------------------------------------------------------

File size: 5.83719635009766 MB
Encoding times:

Crypt32.dll: 579.908060940706 ms
Output size: 8160992 

Machine code: 838.349008044318 ms
Output size: 8160992 

Mikeytown2, blindwig: 224483.446918533 ms
Output size: 8160992

---------------------------------------------------------------------------

File size: 21.3670444488525 MB
Encoding times:

Crypt32.dll: 8545.42518672066 ms
Output size: 29873296

Machine code: 17715.4873035539 ms
Output size: 29873296

Mikeytown2, blindwig: ~

Difference is even bigger for larger files but that's too extreme, even to write about.

btw, i got this poking dlls (out of curiosity) with this

[Ward]

Here are some times, but I guess to get it right more tests should be done (and not in one script because of memory allocation... thing). Three ways are compared. Output size is the number of characters in output string (not counting @CRLF) to verify results. I used older version of machine code _Base64Encode() because the new one tend to crash (Ward, do something).

Difference is even bigger for larger files but that's too extreme, even to write about.

btw, i got this poking dlls (out of curiosity) with this

OKOK, I will fix it. The newer base64 UDF crash because the formula to count the buffer size is wrong (A stupid bug). Thank you for reminding me.

However, your time trial has a little problem. In fact, you are testing the speed of disk reading, autoit string handling, garbage collection and so on.
Because the speed of base64 encoding is much faster than these actions.

In following code I don't set dllstruct to 0, and then it should run as fast as using crypt32.dll.

BTW, a better way to encode a file into base64 is to split it and encode part by part. Handling a huge string in script language is always slow.

Plain Text        expandcollapse  popup

Func _Base64Decode($Data)
    Local $Opcode = "0xC81000005356578365F800E8500000003EFFFFFF3F3435363738393A3B3C3DFFFFFF00FFFFFF000102030405060708

090A0B0C0D0E0F10111213141516171819FFFFFFFFFFFF1A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132338F

45F08B7D0C8B5D0831D2E9910000008365FC00837DFC047D548A034384C0750383EA033C3D75094A803B3D75014AB00084C0

751A837DFC047D0D8B75FCC64435F400FF45FCEBED6A018F45F8EB1F3C2B72193C7A77150FB6F083EE2B0375F08A068B75FC

884435F4FF45FCEBA68D75F4668B06C0E002C0EC0408E08807668B4601C0E004C0EC0208E08847018A4602C0E00624C00A46

038847028D7F038D5203837DF8000F8465FFFFFF89D05F5E5BC9C21000"
    
    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    Local $Ouput = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    Local $Ret = DllCall("user32.dll", "int", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "str", $Data, _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "int", 0, _
                                                    "int", 0)

    Return BinaryMid(DllStructGetData($Ouput, 1), 1, $Ret[0])
EndFunc

Func _Base64Encode($Data, $LineBreak = 76)
    Local $Opcode = "0x5589E5FF7514535657E8410000004142434445464748494A4B4C4D4E4F505152535455565758595A61626364656667

68696A6B6C6D6E6F707172737475767778797A303132333435363738392B2F005A8B5D088B7D108B4D0CE98F0000000FB633

C1EE0201D68A06880731C083F901760C0FB6430125F0000000C1E8040FB63383E603C1E60409C601D68A0688470183F90176

210FB6430225C0000000C1E8060FB6730183E60FC1E60209C601D68A06884702EB04C647023D83F90276100FB6730283E63F

01D68A06884703EB04C647033D8D5B038D7F0483E903836DFC04750C8B45148945FC66B80D0A66AB85C90F8F69FFFFFFC607

005F5E5BC9C21000"

    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    $Data = Binary($Data)
    Local $Input = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Input, 1, $Data)

    $LineBreak = Floor($LineBreak / 4) * 4
    Local $OputputSize = Ceiling(BinaryLen($Data) * 4 / 3) 
    $OputputSize = $OputputSize + Ceiling($OputputSize / $LineBreak) * 2 + 4

    Local $Ouput = DllStructCreate("char[" & $OputputSize & "]")
    DllCall("user32.dll", "none", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "ptr", DllStructGetPtr($Input), _
                                                    "int", BinaryLen($Data), _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "uint", $LineBreak)
    Return DllStructGetData($Ouput, 1)
EndFunc

[trancexx]

I absolutely love the simplicity of your machine code functions. I found them to be beautiful (like a beautiful woman). Probably I'm crazy and everything, khm, khm...
You should make a tutorial on how to get machine code. Don't be lazy, do it. You should find rewarding having people learn of you.


Yes, you are right, seems that now they are as fast.

Return variable of DllStructCreate() in our case is local and releasing allocated memory by setting this variable to 0 is not needed. That was my thinking. Is this true, but completely true?

About splitting...
That should be done very consideringly to avoid possible mistakes due to the fact how base64 encoding is done (3 --> 4). Size of the portions would also depend on how long is every line of output sting (@CRLF we add for SMTP servers).
So, portion lenght should be (4*3*LINE_BREAK)*n. "n" should be determined experimentally to accomplish best performance.
Maybe would be faster, but looping with AutoIt is relatively slow... Someone should do it and test it.


I updated first post (still no error checking).

This how Wards _Base64Encode() looks with me:

AutoIt        expandcollapse  popup

Func _Base64Encode($Data, $LineBreak = 76)
    
    Local $Opcode = "0x5589E5FF7514535657E8410000004142434445464748494A4B4C4D4E4F505152" _
                  & "535455565758595A6162636465666768696A6B6C6D6E6F707172737475767778" _
                  & "797A303132333435363738392B2F005A8B5D088B7D108B4D0CE98F0000000FB6" _
                  & "33C1EE0201D68A06880731C083F901760C0FB6430125F0000000C1E8040FB633" _
                  & "83E603C1E60409C601D68A0688470183F90176210FB6430225C0000000C1E806" _
                  & "0FB6730183E60FC1E60209C601D68A06884702EB04C647023D83F90276100FB6" _
                  & "730283E63F01D68A06884703EB04C647033D8D5B038D7F0483E903836DFC0475" _
                  & "0C8B45148945FC66B80D0A66AB85C90F8F69FFFFFFC607005F5E5BC9C21000" 

    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]")
    DllStructSetData($CodeBuffer, 1, $Opcode)

    $Data = Binary($Data)
    Local $Input = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Input, 1, $Data)

    $LineBreak = Floor($LineBreak / 4) * 4
    Local $OputputSize = Ceiling(BinaryLen($Data) * 4 / 3) 
    $OputputSize = $OputputSize + Ceiling($OputputSize / $LineBreak) * 2 + 4

    Local $Ouput = DllStructCreate("char[" & $OputputSize & "]")
    DllCall("user32.dll", "none", "CallWindowProc", "ptr", DllStructGetPtr($CodeBuffer), _
                                                    "ptr", DllStructGetPtr($Input), _
                                                    "int", BinaryLen($Data), _
                                                    "ptr", DllStructGetPtr($Ouput), _
                                                    "uint", $LineBreak)
    Return DllStructGetData($Ouput, 1)
    
EndFunc

It's beautiful, right? Specially when you think what job it does.

... I'm crazy

Edited by trancexx, 27 September 2008 - 01:16 PM.

[Ward]

I absolutely love the simplicity of your machine code functions. I found them to be beautiful (like a beautiful woman). Probably I'm crazy and everything, khm, khm...
You should make a tutorial on how to get machine code. Don't be lazy, do it. You should find rewarding having people learn of you.

I don't make a tutorial because it has a little complicated. The bigest problem is a machine code function using in AutoiIt can't have a static memory reference.
As a result, we can't use windows API, and even a simple string needs special technique to locate.
I think a better way is to use MemoryDllCall, because everyone know how to make a DLL. And it is much easier than getting machine code.

If you are really interested in it, here is a demo. It is the source of my Base64Encode written in C--.
Maybe a little assembly knowledge is necessary to understand it.

Plain Text        expandcollapse  popup

#pragma option dll

int _export Base64Encode(dword input, len, buffer, lineBrak) uses ESI EDI EBX
{
    dword newLine = lineBrak;

    $CALL NEAR START
    $DB "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
START:
    $POP EDX

    EBX = input;
    EDI = buffer;
    ECX = len;

    while(signed ECX > 0)
    {
        DSBYTE[EDI] = DSBYTE[DSBYTE[EBX] >> 2 + EDX];

        EAX = 0;
        IF(ECX > 1) EAX = DSBYTE[EBX + 1] & 0xf0 >> 4;
        DSBYTE[EDI + 1] = DSBYTE[DSBYTE[EBX] & 3 << 4 | EAX + EDX];

        IF(ECX > 1)
        {
            EAX = DSBYTE[EBX + 2] & 0xc0 >> 6;
            DSBYTE[EDI + 2] = DSBYTE[DSBYTE[EBX + 1] & 0xf << 2 | EAX + EDX];
        } ELSE DSBYTE[EDI + 2] = '=';

        IF(ECX > 2) DSBYTE[EDI + 3] = DSBYTE[DSBYTE[EBX + 2] & 0x3f + EDX];
        ELSE DSBYTE[EDI + 3] = '=';

        EBX += 3;
        EDI += 4;
        ECX -= 3;
        newLine -= 4;
        IF(ZEROFLAG)
        {
            newLine = lineBrak;
            AX = 0x0A0D;
            $stosw
        }
    }

    DSBYTE[EDI] = 0;
    return;
}

Yes, you are right, seems that now they are as fast.

Return variable of DllStructCreate() in our case is local and releasing allocated memory by setting this variable to 0 is not needed. That was my thinking. Is this true, but completely true?

Yes, it is true, I had tested it.
But I still don't know why setting it to 0 is so slow.

Edited by Ward, 27 September 2008 - 01:37 PM.

[trancexx]

I don't make a tutorial because it has a little complicated. The bigest problem is a machine code function using in AutoiIt can't have a static memory reference.
As a result, we can't use windows API, and even a simple string needs special technique to locate.
I think a better way is to use MemoryDllCall, because everyone know how to make a DLL. And it is much easier than getting machine code.

If you are really interested in it, here is a demo. It is the source of my Base64Encode written in C--.
Maybe a little assembly knowledge is necessary to understand it.

Plain Text        expandcollapse  popup

#pragma option dll

int _export Base64Encode(dword input, len, buffer, lineBrak) uses ESI EDI EBX
{
    dword newLine = lineBrak;

    $CALL NEAR START
    $DB "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
START:
    $POP EDX

    EBX = input;
    EDI = buffer;
    ECX = len;

    while(signed ECX > 0)
    {
        DSBYTE[EDI] = DSBYTE[DSBYTE[EBX] >> 2 + EDX];

        EAX = 0;
        IF(ECX > 1) EAX = DSBYTE[EBX + 1] & 0xf0 >> 4;
        DSBYTE[EDI + 1] = DSBYTE[DSBYTE[EBX] & 3 << 4 | EAX + EDX];

        IF(ECX > 1)
        {
            EAX = DSBYTE[EBX + 2] & 0xc0 >> 6;
            DSBYTE[EDI + 2] = DSBYTE[DSBYTE[EBX + 1] & 0xf << 2 | EAX + EDX];
        } ELSE DSBYTE[EDI + 2] = '=';

        IF(ECX > 2) DSBYTE[EDI + 3] = DSBYTE[DSBYTE[EBX + 2] & 0x3f + EDX];
        ELSE DSBYTE[EDI + 3] = '=';

        EBX += 3;
        EDI += 4;
        ECX -= 3;
        newLine -= 4;
        IF(ZEROFLAG)
        {
            newLine = lineBrak;
            AX = 0x0A0D;
            $stosw
        }
    }

    DSBYTE[EDI] = 0;
    return;
}

This actually helps a lot. Can you please post some simpler example (adding two numbers). Original language source and complete function in AutoIt using machine code. That would help me with patterns. I will try to comprehend.