Ward

AutoIt Machine Code Algorithm Collection

75 posts in this topic




Great Job :graduated:

Share this post


Link to post
Share on other sites

Thank you Ward.

...compression/decompression functions are what I'm after.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

I am getting a different MD5 with your routine than using Crypt.au3's _Crypt_HashFile:

$sourceHash = _Crypt_HashFile($source, $CALG_MD5)
$sourceHash2 = _MD5($source)
MsgBox(0, "compare", $sourceHash & @CR & $sourceHash2)

This produces different hashes. Your _MD5 is able to return the same value each time I ask it to for any given file, so at least it is consistent, but would be of limited value since an MD5 doesn't change if the file itself doesn't....

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Hi, Ilewxam.

_MD5 will work on the data you specify via the parameter, just like _Crypt_HashData.

_Crypt_HashFile will treat the param as filename and creates the hash for the content of the file.


*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Ooohhhh, I am loving these functions. Thank you so much! Also, it's great you added Skein! Was hoping to see this soon.

Edited by Skrip

[left][sub]We're trapped in the belly of this horrible machine.[/sub][sup]And the machine is bleeding to death...[/sup][sup][/sup][/left]

Share this post


Link to post
Share on other sites

Hi, Ilewxam.

_MD5 will work on the data you specify via the parameter, just like _Crypt_HashData.

_Crypt_HashFile will treat the param as filename and creates the hash for the content of the file.

Sorry, maybe it's the time of day causing me to be dim, but are you saying that the values being different is OK because the routines work differently? :graduated:

Thanks

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

...but are you saying that the values being different is OK because the routines work differently? :graduated:

_Crypt_HashFile takes the filename, reads the file and hashs the content. _MD5 will just hash the filename itself if you provide it to the function. To get the same result as _Crypt_HashFile it's your own responsibility to open the file, read the content and parse that to the md5 function. Doing so should result in the same hash.

Edit:

#include <MD5.au3>
_MD5_Startup()
ConsoleWrite("_MD5" & @tab & @tab & @tab & "0x" & _MD5(FileRead(@ScriptName)) & @crlf)
_MD5_Exit()

#Include <Crypt.au3>
_Crypt_Startup()
ConsoleWrite("_Crypt_HashFile" & @tab & @tab & "0x" & _Crypt_HashFile(@ScriptName, $CALG_MD5) & @crlf)
_Crypt_Shutdown()
Edited by KaFu

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

I suggested read file in binary mode. For example:

#Include <MD5.au3>
Dim $Timer = TimerInit()
Dim $File = FileOpen(@AutoItExe, 16)
ConsoleWrite(_MD5(FileRead($File)) & @CRLF)
FileClose($File)
ConsoleWrite(TimerDiff($Timer) & @CRLF)

#Include <Crypt.au3>
Dim $Timer = TimerInit()
_Crypt_Startup()
ConsoleWrite(_Crypt_HashFile(@AutoItExe, $CALG_MD5) & @CRLF)
_Crypt_Shutdown()
ConsoleWrite(TimerDiff($timer) & @CRLF)
Edited by Ward

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

WoW, a very good UDF. Thanks for sharing it. :graduated:

I have a problem. Is this the right way to compress the files in chunks? I can't decompress it...

#Include "LZMA.au3"

$Original_Filename = FileOpenDialog("Open File", "", "Any File (*.*)")
If $Original_Filename = "" Then Exit

$Compressed_Filename = FileSaveDialog("Save File", "", "Any File (*.*)")
If $Compressed_Filename = "" Then Exit

$BufferSize = 0x80000 ;8MB
$FileSize = FileGetSize($Original_Filename)

$hFile1 = FileOpen($Original_Filename, 16)
$hFile2 = FileOpen($Compressed_Filename, 2 + 16)

For $i = 1 To Ceiling($FileSize / $BufferSize)
    $Original_Data = FileRead($hFile1, $BufferSize)
    $Compressed_Data = _LZMA_Compress($Original_Data, 5)

    FileWrite($hFile2, $Compressed_Data)
    FileFlush($hFile2)
Next

FileClose($hFile1)
FileClose($hFile2)

Share this post


Link to post
Share on other sites

WoW, a very good UDF. Thanks for sharing it. :graduated:

I have a problem. Is this the right way to compress the files in chunks? I can't decompress it...

You are welcome.

Current version only supports memory block compression.

To compress file in chunks needs additional codes on "Stream Mode" operation.

Maybe I will add it in next release.


新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

AAAHHHH, thank you KaFu, ProgAndy, and of course Ward! Now I understand, LOL, makes sense!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Here is another interesting example, count MD5 in another thread. "Real Multithreading!"

This method should support all the machine code UDF.

Counting hashes in 2 or more threads in the same time is also possible.

; -----------------------------------------------------------------------------
; MD5 Thread Example
; Purpose: Run MD5 Machine Code UDF In Another Thread
; Author: Ward
; -----------------------------------------------------------------------------

#Include "MD5.au3"

Func _MD5Input_Thread(ByRef $Context, $Data)
    If Not IsDllStruct($_MD5_CodeBuffer) Then _MD5_Startup()
    If Not IsDllStruct($Context) Then Return SetError(1, 0, 0)

    $Data = Binary($Data)
    Local $InputLen = BinaryLen($Data)
    Local $Input = DllStructCreate("byte[" & $InputLen & "]")
    DllStructSetData($Input, 1, $Data)

    If @AutoItX64 Then
        Local $Opcode = '0x4883EC084989C8488B5110498B004D8B4820488B49084D8B40184883C408FFE0'
    Else
        Local $Opcode = '0x83EC1C8B4424208B50108954240C8B500C895424088B5008895424048B5004891424FF1083EC1083C41CC20400'
    EndIf
    $Opcode = Binary($Opcode)

    Local $CodeBufferMemory = _MemVirtualAlloc(0, BinaryLen($Opcode), $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
    Local $CodeBuffer = DllStructCreate("byte[" & BinaryLen($Opcode) & "]", $CodeBufferMemory)
    DllStructSetData($CodeBuffer, 1, $Opcode)

    Local $ThreadParam = DllStructCreate("ptr addr; ptr var1; ptr var2; uint var3; int var4")
    DllStructSetData($ThreadParam, 'addr', DllStructGetPtr($_MD5_CodeBuffer) + $_MD5_InputOffset)
    DllStructSetData($ThreadParam, 'var1', DllStructGetPtr($Context))
    DllStructSetData($ThreadParam, 'var2', DllStructGetPtr($Input))
    DllStructSetData($ThreadParam, 'var3', $InputLen)
    DllStructSetData($ThreadParam, 'var4', 0)

    Local $Ret = DllCall("kernel32.dll", "hwnd", "CreateThread", "ptr", 0, "uint", 0, "ptr", _
                        DllStructGetPtr($CodeBuffer), "ptr", DllStructGetPtr($ThreadParam), "uint", 0, "uint", 0)
    Local $ThreadHandle = $Ret[0]
    Do
        Sleep(10)
        ConsoleWrite("MD5 Running..." & @CRLF)
        $Ret = DllCall("kernel32.dll", "int", "GetExitCodeThread", "hwnd", $ThreadHandle, "uint*", 0)
    Until $Ret[2] <> 259 ; STILL_ACTIVE
EndFunc


Dim $BufferSize = 0x100000
Dim $Filename = FileOpenDialog("Open File", "", "Any File (*.*)")
If $Filename = "" Then Exit

Dim $Timer = TimerInit()
Dim $State = _MD5Init()

Dim $FileHandle = FileOpen($Filename, 16)
For $i = 1 To Ceiling(FileGetSize($Filename) / $BufferSize)
    _MD5Input_Thread($State, FileRead($FileHandle, $BufferSize))
Next
Dim $Hash = _MD5Result($State)
FileClose($FileHandle)

ConsoleWrite(StringFormat('%i ms %s', Round(TimerDiff($Timer)), $Hash) & @CRLF)
1 person likes this

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

Hey thanks Ward! I was looking for something like this :)

Share this post


Link to post
Share on other sites

Thanks Ward these are great :unsure:

Is there a chance that you could add gzip and deflate compression/decompression to your UDF?

Regards

tip


[center]MsgBox_Tipped: Eye candy msgboxes/inputboxes/loginboxes. | CreateBlankBox: Semi-transparent layers with borders and rounded corners.[/center]

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

I will try it if I have more free time. But not soon.

To use deflate/inflate algorithm or to compress/decompress gzip files, the best choice for now is zlib dll.

Here are some example I just written:

Global $_ZLIBDLL = DllOpen("zlib1.dll")

Func _compressBound($SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "uint:cdecl", "compressBound", "uint", $SourceLen)
    Return $Ret[0]
EndFunc

Func _compress($Dest, ByRef $DestLen, $Source, $SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "compress", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc

#cs
     The compression level must be Z_DEFAULT_COMPRESSION(-1), or between 0 and 9:
   1 gives best speed, 9 gives best compression, 0 gives no compression at all
   (the input data is simply copied a block at a time).  Z_DEFAULT_COMPRESSION
   requests a default compromise between speed and compression (currently
   equivalent to level 6).
#ce

Func _compress2($Dest, ByRef $DestLen, $Source, $SourceLen, $Level)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "compress2", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen, "int", $Level)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc

Func _uncompress($Dest, ByRef $DestLen, $Source, $SourceLen)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "uncompress", "ptr", $Dest, "uint*", $DestLen, "ptr", $Source, "uint", $SourceLen)
    $DestLen = $Ret[2]
    Return $Ret[0]
EndFunc


Func Compress($Data, $Level = Default)
    Local $DataLen = BinaryLen($Data)
    Local $Source = DllStructCreate("byte[" & $DataLen & "]")
    DllStructSetData($Source, 1, $Data)
    
    Local $DestLen = _compressBound($DataLen)
    Local $Dest = DllStructCreate("byte[" & $DestLen & "]")
    
    Local $Ret
    If IsKeyword($Level) Then
        $Ret = _compress(DllStructGetPtr($Dest), $DestLen, DllStructGetPtr($Source), $DataLen)  
    Else
        $Ret = _compress2(DllStructGetPtr($Dest), $DestLen, DllStructGetPtr($Source), $DataLen, $Level) 
    EndIf
    
    If $Ret = 0 Then
        Return BinaryMid(DllStructGetData($Dest, 1), 1, $DestLen)
    Else
        Return SetError($Ret, 0, Binary(''))
    EndIf
EndFunc

Func Uncompress($Data, $UncompressedSize)
    Local $DataLen = BinaryLen($Data)
    Local $Source = DllStructCreate("byte[" & $DataLen & "]")
    DllStructSetData($Source, 1, $Data)
    
    Local $Dest = DllStructCreate("byte[" & $UncompressedSize & "]")
    
    Local $Ret = _uncompress(DllStructGetPtr($Dest), $UncompressedSize, DllStructGetPtr($Source), $DataLen) 
    
    If $Ret = 0 Then
        Return DllStructGetData($Dest, 1)
    Else
        Return SetError($Ret, 0, Binary(''))
    EndIf
EndFunc

Global $_ZLIBDLL = DllOpen("zlib1.dll")

Func _gzopen($Path, $Mode)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzopen", "str", $Path, "str", $Mode)
    Return $Ret[0]
EndFunc

Func _gzclose($File)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzclose", "ptr", $File)
    Return $Ret[0]
EndFunc

Func _gzwrite($File, $Buf, $Len)
    Local $Ret = DllCall($_ZLIBDLL, "ptr:cdecl", "gzwrite", "ptr", $File, "ptr", $Buf, "uint", $Len)
    Return $Ret[0]
EndFunc

Func _gzread($File, $Buf, $Len)
    Local $Ret = DllCall($_ZLIBDLL, "int:cdecl", "gzread", "ptr", $File, "ptr", $Buf, "uint", $Len)
    Return $Ret[0]
EndFunc


Func GZFileWrite($Path, $Data, $Mode = "wb")
    Local $Buffer = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Buffer, 1, $Data)

    Local $GZFile = _gzopen($Path, $Mode)
    _gzwrite($GZFile, DllStructGetPtr($Buffer), BinaryLen($Data))
    
    _gzclose($GZFile)
EndFunc

Func GZFileRead($Path, $BufferSize = 1024)
    Local $Buffer = DllStructCreate("Byte[" & $BufferSize & "]")

    Local $GZFile = _gzopen($Path, "rb")
    Local $Data = Binary('')
    Do
        Local $Len = _gzread($GZFile, DllStructGetPtr($Buffer), $BufferSize)
        If $Len < 0 Then
            _gzclose($GZFile)
            Return SetError(1, 0, Binary(''))
        EndIf
        $Data &= BinaryMid(DllStructGetData($Buffer, 1), 1, $Len)
    Until $Len = 0
    
    _gzclose($GZFile)
    Return $Data
EndFunc
Edited by Ward

新版 _ArrayAdd 的白痴作者,不管是誰,去死一死好了

 

Share this post


Link to post
Share on other sites

Thank you very much Ward. They are working as expected. :unsure:


[center]MsgBox_Tipped: Eye candy msgboxes/inputboxes/loginboxes. | CreateBlankBox: Semi-transparent layers with borders and rounded corners.[/center]

Share this post


Link to post
Share on other sites

I am having trouble with the AES, both AESTest files are giving this error:

C:\Users\Tech\Desktop\AutoIt Machine Code Algorithm Collection\Encode\AES.au3(16,14) : ERROR: syntax error
Global Const Enum

Changing to:

Global $AES_CBC_MODE, $AES_CFB_MODE, $AES_OFB_MODE
resolves it and all is well.

My only request at this point is for salt support, or is that already somehow possible?

Thanks

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now