45 replies to this topic

[Ward]

I have already published a lot of AutoIt UDF about algorithm, but all of them only support 32 bits or so called X86 system. Recently I got a computer with Windows 7 64 bits, so I finally added X64 support to most of my old projects. Besides, I also added some new. For example, some compression algorithm and SHA3 Candidates.

Following are the algorithms list:

Plain Text        expandcollapse  popup

Checksum
  CRC16
  CRC32
  ADLER32

Compression
  FastLZ
  LZF
  LZMA
  LZMAT
  MiniLZO
  QuickLZ

Encode
  Base64
  ARC4
  XXTEA
  DES
  AES

Hash
  Checksums (CRC16/CRC32/ADLER32)
  MD2
  MD4
  MD5
  SHA1
  SHA2 (SHA224/256/384/512)
  SHA3 Candidates
    BLAKE
    BMW (Blue Midnight Wish)
    CUBEHASH
    ECHO
    SHABAL
    SKEIN

Some points to mention:

• All of the subroutines have one or more examples to demonstrate the usage. Since the function and usage of subroutine are easy to understand. A complete subroutines and parameters list are unavailability now. Sorry for my lazy.
  
• All of the subroutines here invoked by Lazycat's method (through CallWindowProc API). My MemoryDLL UDF is not necessary this time. Although MemoryFuncCall (part of MemoryDLL) is still good, but inevitably, it is slower than CallWindowProc.
  
• Some subroutines have the same name with my old machine code version UDF. But for some reason, I rearrange the position of the parameters. Please not mix up.
  
• If you notice, yes, checksums are duplicated. But they receive different parameters. One is the old style, and another use the same interface as other hashes. Choose what you like, but don't use them in the same time.
  
• Some algorithm already supported by the standard UDF "Encryption.au3". But I still provide them, because some system lack of the full support of Windows Crypt Library.
  
• If you are looking for only one hash algorithm, for example, used in encryption, I suggested "SHABAL_TINY.au3". Although it is a bit slower then SHABAL, but it is smaller, and it supports different size of output (from 32 to 512 bits).

 AutoIt Machine Code Algorithm Collection.zip   341.39K   2862 downloads

[DarkAngel]

Great Job

[KaFu]

This UDF is an excellent addition to my library , thanks a lot for the work you put into it !

[trancexx]

Thank you Ward.
...compression/decompression functions are what I'm after.

[llewxam]

I am getting a different MD5 with your routine than using Crypt.au3's _Crypt_HashFile:

$sourceHash = _Crypt_HashFile($source, $CALG_MD5)
$sourceHash2 = _MD5($source)
MsgBox(0, "compare", $sourceHash & @CR & $sourceHash2)

This produces different hashes. Your _MD5 is able to return the same value each time I ask it to for any given file, so at least it is consistent, but would be of limited value since an MD5 doesn't change if the file itself doesn't....

Ian

[ProgAndy]

Hi, Ilewxam.

_MD5 will work on the data you specify via the parameter, just like _Crypt_HashData.
_Crypt_HashFile will treat the param as filename and creates the hash for the content of the file.

[Skrip]

Ooohhhh, I am loving these functions. Thank you so much! Also, it's great you added Skein! Was hoping to see this soon.

Edited by Skrip, 12 November 2010 - 07:06 AM.

[llewxam]

Hi, Ilewxam.

_MD5 will work on the data you specify via the parameter, just like _Crypt_HashData.
_Crypt_HashFile will treat the param as filename and creates the hash for the content of the file.

Sorry, maybe it's the time of day causing me to be dim, but are you saying that the values being different is OK because the routines work differently?

Thanks
Ian

[KaFu]

...but are you saying that the values being different is OK because the routines work differently?

_Crypt_HashFile takes the filename, reads the file and hashs the content. _MD5 will just hash the filename itself if you provide it to the function. To get the same result as _Crypt_HashFile it's your own responsibility to open the file, read the content and parse that to the md5 function. Doing so should result in the same hash.

Edit:

#include <MD5.au3>
_MD5_Startup()
ConsoleWrite("_MD5" & @tab & @tab & @tab & "0x" & _MD5(FileRead(@ScriptName)) & @crlf)
_MD5_Exit()

#Include <Crypt.au3>
_Crypt_Startup()
ConsoleWrite("_Crypt_HashFile" & @tab & @tab & "0x" & _Crypt_HashFile(@ScriptName, $CALG_MD5) & @crlf)
_Crypt_Shutdown()

Edited by KaFu, 12 November 2010 - 01:06 PM.

[Ward]

I suggested read file in binary mode. For example:

#Include <MD5.au3>
Dim $Timer = TimerInit()
Dim $File = FileOpen(@AutoItExe, 16)
ConsoleWrite(_MD5(FileRead($File)) & @CRLF)
FileClose($File)
ConsoleWrite(TimerDiff($Timer) & @CRLF)

#Include <Crypt.au3>
Dim $Timer = TimerInit()
_Crypt_Startup()
ConsoleWrite(_Crypt_HashFile(@AutoItExe, $CALG_MD5) & @CRLF)
_Crypt_Shutdown()
ConsoleWrite(TimerDiff($timer) & @CRLF)

Edited by Ward, 12 November 2010 - 01:44 PM.

[MiserableLife]

WoW, a very good UDF. Thanks for sharing it.


I have a problem. Is this the right way to compress the files in chunks? I can't decompress it...

AutoIt        expandcollapse  popup

#Include "LZMA.au3"

$Original_Filename = FileOpenDialog("Open File", "", "Any File (*.*)")
If $Original_Filename = "" Then Exit

$Compressed_Filename = FileSaveDialog("Save File", "", "Any File (*.*)")
If $Compressed_Filename = "" Then Exit

$BufferSize = 0x80000 ;8MB
$FileSize = FileGetSize($Original_Filename)

$hFile1 = FileOpen($Original_Filename, 16)
$hFile2 = FileOpen($Compressed_Filename, 2 + 16)

For $i = 1 To Ceiling($FileSize / $BufferSize)
    $Original_Data = FileRead($hFile1, $BufferSize)
    $Compressed_Data = _LZMA_Compress($Original_Data, 5)

    FileWrite($hFile2, $Compressed_Data)
    FileFlush($hFile2)
Next

FileClose($hFile1)
FileClose($hFile2)

[Ward]

WoW, a very good UDF. Thanks for sharing it.

I have a problem. Is this the right way to compress the files in chunks? I can't decompress it...

You are welcome.

Current version only supports memory block compression.
To compress file in chunks needs additional codes on "Stream Mode" operation.
Maybe I will add it in next release.

[llewxam]

AAAHHHH, thank you KaFu, ProgAndy, and of course Ward! Now I understand, LOL, makes sense!

Ian

[Ward]

Here is another interesting example, count MD5 in another thread. "Real Multithreading!"
This method should support all the machine code UDF.
Counting hashes in 2 or more threads in the same time is also possible.

Plain Text        expandcollapse  popup

; -----------------------------------------------------------------------------
; MD5 Thread Example
; Purpose: Run MD5 Machine Code UDF In Another Thread
; Author: Ward
; -----------------------------------------------------------------------------

#Include "MD5.au3"

Func _MD5Input_Thread(ByRef $Context, $Data)
    If Not IsDllStruct($_MD5_CodeBuffer) Then _MD5_Startup()
    If Not IsDllStruct($Context) Then Return SetError(1, 0, 0)

    $Data = Binary($Data)
    Local $InputLen = BinaryLen($Data)
    Local $Input = DllStructCreate("byte[" & $InputLen & "]")
    DllStructSetData($Input, 1, $Data)

    If @AutoItX64 Then
        Local $Opcode = '0x4883EC084989C8488B5110498B004D8B4820488B49084D8B40184883C408FFE0'
    Else
        Local $Opcode = '0x83EC1C8B4424208B50108954240C8B500C895424088B5008895424048B5004891424FF1083EC1083C41CC20400'
    EndIf
    $Opcode = Binary($Opcode)

    Local $CodeBufferMemory = _MemVirtualAlloc(0, BinaryLen($Opcode), $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]", $CodeBufferMemory)
    DllStructSetData($CodeBuffer, 1, $Opcode)

    Local $ThreadParam = DllStructCreate("ptr addr; ptr var1; ptr var2; uint var3; int var4")
    DllStructSetData($ThreadParam, 'addr', DllStructGetPtr($_MD5_CodeBuffer) + $_MD5_InputOffset)
    DllStructSetData($ThreadParam, 'var1', DllStructGetPtr($Context))
    DllStructSetData($ThreadParam, 'var2', DllStructGetPtr($Input))
    DllStructSetData($ThreadParam, 'var3', $InputLen)
    DllStructSetData($ThreadParam, 'var4', 0)

    Local $Ret = DllCall("kernel32.dll", "hwnd", "CreateThread", "ptr", 0, "uint", 0, "ptr", _
                        DllStructGetPtr($CodeBuffer), "ptr", DllStructGetPtr($ThreadParam), "uint", 0, "uint", 0)
    Local $ThreadHandle = $Ret[0]
    Do
        Sleep(10)
        ConsoleWrite("MD5 Running..." & @CRLF)
        $Ret = DllCall("kernel32.dll", "int", "GetExitCodeThread", "hwnd", $ThreadHandle, "uint*", 0)
    Until $Ret[2] <> 259 ; STILL_ACTIVE
EndFunc


Dim $BufferSize = 0x100000
Dim $Filename = FileOpenDialog("Open File", "", "Any File (*.*)")
If $Filename = "" Then Exit

Dim $Timer = TimerInit()
Dim $State = _MD5Init()

Dim $FileHandle = FileOpen($Filename, 16)
For $i = 1 To Ceiling(FileGetSize($Filename) / $BufferSize)
    _MD5Input_Thread($State, FileRead($FileHandle, $BufferSize))
Next
Dim $Hash = _MD5Result($State)
FileClose($FileHandle)

ConsoleWrite(StringFormat('%i ms %s', Round(TimerDiff($Timer)), $Hash) & @CRLF)

[lemony]

Hey thanks Ward! I was looking for something like this

[tip]

Thanks Ward these are great

Is there a chance that you could add gzip and deflate compression/decompression to your UDF?


Regards
tip

[Ward]

I will try it if I have more free time. But not soon.
To use deflate/inflate algorithm or to compress/decompress gzip files, the best choice for now is zlib dll.

Here are some example I just written:

AutoIt        expandcollapse  popup

Global $_ZLIBDLL = DllOpen("zlib1.dll")

Func _compressBound($SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "uint:cdecl", "compressBound", "uint", $SourceLen)
    Return $Ret[0]
EndFunc

Func _compress($Dest, ByRef $DestLen, $Source, $SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "compress", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc

#cs
     The compression level must be Z_DEFAULT_COMPRESSION(-1), or between 0 and 9:
   1 gives best speed, 9 gives best compression, 0 gives no compression at all
   (the input data is simply copied a block at a time).  Z_DEFAULT_COMPRESSION
   requests a default compromise between speed and compression (currently
   equivalent to level 6).
#ce

Func _compress2($Dest, ByRef $DestLen, $Source, $SourceLen, $Level)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "compress2", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen, "int", $Level)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc

Func _uncompress($Dest, ByRef $DestLen, $Source, $SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "uncompress", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc


Func Compress($Data, $Level = Default)
    Local $DataLen = BinaryLen($Data)
    Local $Source = DllStructCreate("byte[" & $DataLen & "]")
    DllStructSetData($Source, 1, $Data)
    
    Local $DestLen = _compressBound($DataLen)
    Local $Dest = DllStructCreate("byte[" & $DestLen & "]")
    
    Local $Ret
    If IsKeyword($Level) Then
        $Ret = _compress(DllStructGetPtr($Dest), $DestLen, DllStructGetPtr($Source), $DataLen)  
    Else
        $Ret = _compress2(DllStructGetPtr($Dest), $DestLen, DllStructGetPtr($Source), $DataLen, $Level) 
    EndIf
    
    If $Ret = 0 Then
        Return BinaryMid(DllStructGetData($Dest, 1), 1, $DestLen)
    Else
        Return SetError($Ret, 0, Binary(''))
    EndIf
EndFunc

Func Uncompress($Data, $UncompressedSize)
    Local $DataLen = BinaryLen($Data)
    Local $Source = DllStructCreate("byte[" & $DataLen & "]")
    DllStructSetData($Source, 1, $Data)
    
    Local $Dest = DllStructCreate("byte[" & $UncompressedSize & "]")
    
    Local $Ret = _uncompress(DllStructGetPtr($Dest), $UncompressedSize, DllStructGetPtr($Source), $DataLen) 
    
    If $Ret = 0 Then
        Return DllStructGetData($Dest, 1)
    Else
        Return SetError($Ret, 0, Binary(''))
    EndIf
EndFunc

AutoIt        expandcollapse  popup

Global $_ZLIBDLL = DllOpen("zlib1.dll")

Func _gzopen($Path, $Mode)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzopen", "str", $Path, "str", $Mode)
    Return $Ret[0]
EndFunc

Func _gzclose($File)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzclose", "ptr", $File)
    Return $Ret[0]
EndFunc

Func _gzwrite($File, $Buf, $Len)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzwrite", "ptr", $File, "ptr", $Buf, "uint", $Len)
    Return $Ret[0]
EndFunc

Func _gzread($File, $Buf, $Len)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "gzread", "ptr", $File, "ptr", $Buf, "uint", $Len)
    Return $Ret[0]
EndFunc


Func GZFileWrite($Path, $Data, $Mode = "wb")
    Local $Buffer = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Buffer, 1, $Data)

    Local $GZFile = _gzopen($Path, $Mode)
    _gzwrite($GZFile, DllStructGetPtr($Buffer), BinaryLen($Data))
    
    _gzclose($GZFile)
EndFunc

Func GZFileRead($Path, $BufferSize = 1024)
    Local $Buffer = DllStructCreate("Byte[" & $BufferSize & "]")

    Local $GZFile = _gzopen($Path, "rb")
    Local $Data = Binary('')
    Do
        Local $Len = _gzread($GZFile, DllStructGetPtr($Buffer), $BufferSize)
        If $Len < 0 Then
            _gzclose($GZFile)
            Return SetError(1, 0, Binary(''))
        EndIf
        $Data &= BinaryMid(DllStructGetData($Buffer, 1), 1, $Len)
    Until $Len = 0
    
    _gzclose($GZFile)
    Return $Data
EndFunc

Edited by Ward, 17 May 2011 - 08:55 AM.

[tip]

Thank you very much Ward. They are working as expected.

[Zedna]

Nice!

[llewxam]

I am having trouble with the AES, both AESTest files are giving this error:

C:\Users\Tech\Desktop\AutoIt Machine Code Algorithm Collection\Encode\AES.au3(16,14) : ERROR: syntax error
Global Const Enum

Changing to:

Global $AES_CBC_MODE, $AES_CFB_MODE, $AES_OFB_MODE

resolves it and all is well.

My only request at this point is for salt support, or is that already somehow possible?

Thanks
Ian