CRC32, MD4, MD5, SHA1 -for files

[trancexx]

Sure does. Any idea what MS means by 'incoherent' then?

Probably for some specific flags.

[jchd]

Thank you for this nice tool.

BTW do you know which combination of flags would allow for processing already opened files? I keep getting error from CreteFileW for those, but I can read them with various tools without problem. Since we certainly have no intention of changing a single bit of the files we compute hash on , it could be handy for those who need to routinely check if a file's content has changed when this file is e.g. an SQLite database or whatever.

[trancexx]

Thank you for this nice tool.

BTW do you know which combination of flags would allow for processing already opened files? I keep getting error from CreteFileW for those, but I can read them with various tools without problem. Since we certainly have no intention of changing a single bit of the files we compute hash on , it could be handy for those who need to routinely check if a file's content has changed when this file is e.g. an SQLite database or whatever.

Can you read them with built-in AutoIt's FileRead()?

Try changing third parameter of CreteFileW from 1 (FILE_SHARE_READ) to 3 (FILE_SHARE_READ|FILE_SHARE_WRITE).

[jchd]

You are right. I knew I could at least read them because I can run my applications in this situation and these are mostly SQLite databases that I'm using, also 24/7 open in a third-party DBMgr.

(FILE_SHARE_READ|FILE_SHARE_WRITE) works like a charm in this situation.

Thank you very much.

[trancexx]

You are right. I knew I could at least read them because I can run my applications in this situation and these are mostly SQLite databases that I'm using, also 24/7 open in a third-party DBMgr.

(FILE_SHARE_READ|FILE_SHARE_WRITE) works like a charm in this situation.

Thank you very much.

No, thank you.

I'll make the correction.

[zodiac]

First of all, thanks for the amazing work you did with this. The more I see the things you and others come up with, the more I realize I'm not a programmer. That said, when testing this I found that I couldn't get results for files over 400MB in size. Is this expected?

308MB file:

! CRC32 took 807.727508647628 ms

Result: AD9E3A65

! MD4 took 617.432683521882 ms

Result: 0010CADEFD65A7890837AB4B6B45FD51

! MD5 took 824.942514579817 ms

Result: 682058B274566D8AB46B28D45D2B66AF

! SHA1 took 956.555522951724 ms

Result: 5A9398E47DFC73632F7C51B72BAC2C231C8CB1E8

3.10GB file:

! CRC32 took 0.34496967063817 ms

Result:

! MD4 took 0.424828634928062 ms

Result:

! MD5 took 0.366830874120616 ms

Result:

! SHA1 took 0.279530437979379 ms

Result:

[lemony]

trancexx, great work.

I've been testing with some random files and it seems to be working great. And pretty fast too.

I did however find a tiny bug while hashing 0 byte files (empty files). Instead of returning the hash it returns a blank string.

These are the correct values for 0 byte files:

crc32: 00000000
  md4: 31d6cfe0d16ae931b73c59d7e0c089c0
  md5: d41d8cd98f00b204e9800998ecf8427e
 sha1: da39a3ee5e6b4b0d3255bfef95601890afd80709

[Splash]

Thanks! Very usefull!!!

[trancexx]

trancexx, great work.

I've been testing with some random files and it seems to be working great. And pretty fast too.

I did however find a tiny bug while hashing 0 byte files (empty files). Instead of returning the hash it returns a blank string.

These are the correct values for 0 byte files:

crc32: 00000000
  md4: 31d6cfe0d16ae931b73c59d7e0c089c0
  md5: d41d8cd98f00b204e9800998ecf8427e
 sha1: da39a3ee5e6b4b0d3255bfef95601890afd80709

CreateFileMapping fails (10062) with empty files.

@zodiac, probably the size of your ram causes that limitation.

Edited January 9, 2010 by trancexx

[lemony]

CreateFileMapping fails (10062) with empty files.

It's not that big of a problem anyway. If anyone wanted to prevent it, it can easily be done with a simple check of the file size.

thanks trancexx

[zodiac]

@zodiac, probably the size of your ram causes that limitation.

The computer I ran this on had 3GB of RAM.

[SandelPerieanu]

The computer I ran this on had 3GB of RAM.

this example works with any sizes...

I have no explanation for

If $ ichunksize> 1001783296 Then $ ichunksize = 1001783296

this number arose after several attempts, it is upper limit

expandcollapse popup

Func _MD5ForFirstFileChunk($sfile)
   Local $ichunksize = FileGetSize($sfile)
   If $ichunksize > 1001783296 Then $ichunksize = 1001783296
   Local $a_hcall = DllCall("kernel32.dll", "hwnd", "CreateFileW", _
        "wstr", $sfile, _
        "dword", 0x80000000, _ ; GENERIC_READ
        "dword", 1, _ ; FILE_SHARE_READ
        "ptr", 0, _
        "dword", 3, _ ; OPEN_EXISTING
        "dword", 0, _ ; SECURITY_ANONYMOUS
        "ptr", 0)
   If @error Or $a_hcall[0] = -1 Then
    Return SetError(1, 0, "")
   EndIf
   Local $hfile = $a_hcall[0]
   $a_hcall = DllCall("kernel32.dll", "ptr", "CreateFileMappingW", _
        "hwnd", $hfile, _
        "dword", 0, _ ; default security descriptor
        "dword", 2, _ ; PAGE_READONLY
        "dword", 0, _
        "dword", 0, _
        "ptr", 0)
   If @error Or Not $a_hcall[0] Then
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfile)
    Return SetError(2, 0, "")
   EndIf
   DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfile)
   Local $hfilemappingobject = $a_hcall[0]
   $a_hcall = DllCall("kernel32.dll", "ptr", "MapViewOfFile", _
        "hwnd", $hfilemappingobject, _
        "dword", 4, _ ; FILE_MAP_READ
        "dword", 0, _
        "dword", 0, _
        "dword", $ichunksize)
   If @error Or Not $a_hcall[0] Then
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfilemappingobject)
    Return SetError(3, 0, "")
   EndIf
   Local $pfile = $a_hcall[0]
   Local $ibuffersize = $ichunksize
   Local $tmd5_ctx = DllStructCreate("dword i[2];" & _
        "dword buf[4];" & _
        "ubyte in[64];" & _
        "ubyte digest[16]")
   DllCall("advapi32.dll", "none", "MD5Init", "ptr", DllStructGetPtr($tmd5_ctx))
   If @error Then
    DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pfile)
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfilemappingobject)
    Return SetError(4, 0, "")
   EndIf
   DllCall("advapi32.dll", "none", "MD5Update", _
        "ptr", DllStructGetPtr($tmd5_ctx), _
        "ptr", $pfile, _
        "dword", $ibuffersize)
   If @error Then
    DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pfile)
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfilemappingobject)
    Return SetError(5, 0, "")
   EndIf
   DllCall("advapi32.dll", "none", "MD5Final", "ptr", DllStructGetPtr($tmd5_ctx))
   If @error Then
    DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pfile)
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfilemappingobject)
    Return SetError(6, 0, "")
   EndIf
   DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pfile)
   DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hfilemappingobject)
   Local $smd5 = Hex(DllStructGetData($tmd5_ctx, "digest"))
   Return SetError(0, 0, $smd5)
EndFunc   ;==>_MD5ForFirstFileChunk

[Slym]

Hope that it can help :

This is a very simple example of MD5 Hash using Crypt.au3 ( _Crypt_HashData ) + implementing a progress bar :

I find this function to be working very well even on 64 Bits systems (which is not the case for the first one of this topic)

To compute SHA1 instead of MD5, just replace $CALG_MD5 by $CALG_SHA1.

expandcollapse popup

#include <Crypt.au3>
_Crypt_Startup()

$Filename= FileOpenDialog("Select files to hash", @ScriptDir & "\", "All Files (*.*)", 5)

    If $FileName = "" Then
        Exit
    EndIf

    $multiple_files = StringInStr($FileName, "|")

    if $multiple_files > 0 Then
        $files = StringSplit($FileName, "|")
        $folder= $files[1] &"\"
        $lines="------------------------------- Start "& @MDAY & "-" & @MON & "-" & @YEAR & " (" & @HOUR & "h" & @MIN & "s" & @SEC &") -------------------------------"&@CRLF
        $j=2
        $hashes=""
        While $j < $files[0]+1
            $lines &= $files[$j] & " = " & MD5($folder & "\" &$files[$j]) & @CRLF
            $hashes &= $files[$j] & " = " & MD5($folder & "\" &$files[$j]) & @CRLF
            $j=$j+1
        Wend
        $lines &= "------------------------------- End "& @MDAY & "-" & @MON & "-" & @YEAR & " (" & @HOUR & "h" & @MIN & "s" & @SEC &") -------------------------------"&@CRLF
        $file = FileOpen(@ScriptDir &"\MD5 HASHES.txt", 1)
        FileWrite($file, $lines)
        FileClose($file)
        ClipPut($hashes)
        MsgBox(64, "Result", $hashes & @CRLF & "It has been put in your clipboard, you just have to paste it." )
    Else
        $hash = MD5($FileName)
        ClipPut($hash)
        MsgBox(64, "Result", "MD5 hash of file "& $Filename & " is :" & @CRLF & @CRLF & @TAB & $hash & @CRLF & @CRLF & "It has been put in your clipboard, you just have to paste it." )
    EndIf

_Crypt_Shutdown()

Func MD5($FileToHash)

    Local $filehandle = FileOpen($FileToHash, 16)
    Local $buffersize=0x20000,$final=0,$hash=""

    $iterations = Ceiling(FileGetSize($FileToHash) / $buffersize)

    ProgressOn("Computing hash", "File : " & path_to_name($FileToHash), "0 %", -1, -1, 16)
    For $i = 1 To $iterations
        if $i=$iterations Then $final=1
        $hash=_Crypt_HashData(FileRead($filehandle, $buffersize),$CALG_MD5,$final,$hash)
        $percent_md5 = Round(100 * $i / $iterations)
        ProgressSet($percent_md5, $percent_md5 & " %")
    Next
    FileClose($filehandle)

    ProgressSet(100, "100%" ,"File hashed")
    ProgressOff()
    Return StringTrimLeft($hash, 2)
EndFunc


Func path_to_name($filepath)
    $short_name = StringSplit($filepath, '\')
    Return ($short_name[$short_name[0]])
EndFunc   ;==>unix_path_to_name

[natong]

Please help.

I using AutoIT v3.2.0.1 on Win7 x64 and return nothing.

expandcollapse popup

Opt("TrayIconHide", 1)
if $CmdLine[0] = 0 then
exit
EndIf
Global $sFile = $CmdLine[1]
;------------------------------------------------------------------------
; Name...........: _CRC32ForFile
; Description ...: Calculates CRC32 value for the specific file.
; Syntax.........: _CRC32ForFile ($sFile)
; Parameters ....: $sFile - Full path to the file to process.
; Return values .: Success - Returns CRC32 value in form of hex string
;                         - Sets @error to 0
;                 Failure - Returns empty string and sets @error:
;                 |1 - CreateFile function or call to it failed.
;                 |2 - CreateFileMapping function or call to it failed.
;                 |3 - MapViewOfFile function or call to it failed.
;                 |4 - RtlComputeCrc32 function or call to it failed.
; Author ........: trancexx
;------------------------------------------------------------------------
Func _CRC32ForFile($sFile)
    Local $a_hCall = DllCall("kernel32.dll", "hwnd", "CreateFileW", _
            "wstr", $sFile, _
            "dword", 0x80000000, _ ; GENERIC_READ
            "dword", 3, _ ; FILE_SHARE_READ|FILE_SHARE_WRITE
            "ptr", 0, _
            "dword", 3, _ ; OPEN_EXISTING
            "dword", 0, _ ; SECURITY_ANONYMOUS
            "ptr", 0)
    If @error Or $a_hCall[0] = -1 Then
        Return SetError(1, 0, "")
    EndIf
    Local $hFile = $a_hCall[0]
    $a_hCall = DllCall("kernel32.dll", "ptr", "CreateFileMappingW", _
            "hwnd", $hFile, _
            "dword", 0, _ ; default security descriptor
            "dword", 2, _ ; PAGE_READONLY
            "dword", 0, _
            "dword", 0, _
            "ptr", 0)
    If @error Or Not $a_hCall[0] Then
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFile)
        Return SetError(2, 0, "")
    EndIf
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFile)
    Local $hFileMappingObject = $a_hCall[0]
    $a_hCall = DllCall("kernel32.dll", "ptr", "MapViewOfFile", _
            "hwnd", $hFileMappingObject, _
            "dword", 4, _ ; FILE_MAP_READ
            "dword", 0, _
            "dword", 0, _
            "dword", 0)
    If @error Or Not $a_hCall[0] Then
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(3, 0, "")
    EndIf
    Local $pFile = $a_hCall[0]
    Local $iBufferSize = FileGetSize($sFile)
    Local $a_iCall = DllCall("ntdll.dll", "dword", "RtlComputeCrc32", _
            "dword", 0, _
            "ptr", $pFile, _
            "int", $iBufferSize)
    If @error Or Not $a_iCall[0] Then
        DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(4, 0, "")
    EndIf
    DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
    Local $iCRC32 = $a_iCall[0]
    Return SetError(0, 0, Hex($iCRC32))
EndFunc
;------------------------------------------------------------------------
$title = "Copy CRC32 to Clipboard"
$filename = StringSplit($sFile, "")
$hash = _CRC32ForFile($sFile)
$message = $filename[$filename[0]] & " : " & $hash
ClipPut($hash)
MsgBox(64, $title, $message)

[SmOke_N]

Returning nothing? Like not even a messagebox?

If so, you're telling it to exit if you don't pass it a command line parameter.

Other than that, Win7 x64, after removing your global variables and putting in my own file to hash, it worked fine.

[natong]

I tried use 100% your code on the #1 post, it show only

Result:

I try install v3.3.8.1 and got the same result.

Please help what are wrong. I love your code which are clean and easy to use.

Edited August 16, 2012 by natong

[AvvA]

Hello
 
I'm making a duplicate unifier program which requires a lot of checksum calculation, so I did some tests with what AutoIt proposes to do so. I put the results here so this OP function gets what it deserves, a lot of flowers , because it is from far the fastest way:

;AutoIt 3.3.8.0 / core2duo e8500 (3GHz) with 4GB RAM

;file number - file size
;Returned result (MD5 or Error code) / what algorythm was used / time in second(s)

expandcollapse popup

;~ empty reference
;~ -1                                   AutoIt:             0.00804
;~ 0xD41D8CD98F00B204E9800998ECF8427E   Ward:               0.00125
;~ 0x                                   trancexx:           7e-005 --error code: 1
;~ 0xD41D8CD98F00B204E9800998ECF8427E   trancexx KaFu loop: 0.00687

;~ file 1 - 18 421 716 284 bytes
;~ 0xF4E9DD62A332BA84B117CF28724F6C60   AutoIt:             208.96297
;~ 0xF4E9DD62A332BA84B117CF28724F6C60   Ward:               228.21975
;~ 0x                                   trancexx:           0.00029 --error code: 3
;~ 0xF4E9DD62A332BA84B117CF28724F6C60   trancexx KaFu loop: 349.84716

;~ file 2 - 4 544 836 158 bytes
;~ 0x58537EC5AE153A7329233273601C2021   AutoIt:             52.01713
;~ 0x58537EC5AE153A7329233273601C2021   Ward:               57.13209
;~ 0x                                   trancexx:           0.00025 --error code: 3
;~ 0x58537EC5AE153A7329233273601C2021   trancexx KaFu loop: 83.66316

;~ file 3 - 4 269 469 696 bytes
;~ 0x7FD7DB11C916BBDE489AF65773A18EDA   AutoIt:             63.08034
;~ 0x7FD7DB11C916BBDE489AF65773A18EDA   Ward:               63.29612
;~ 0x                                   trancexx:           0.00028 --error code: 3
;~ 0x7FD7DB11C916BBDE489AF65773A18EDA   trancexx KaFu loop: 90.67639

;~ file 4 - 1 691 638 535 bytes
;~ 0x5300F75099D47E275A9F6C43AC501469   AutoIt:             19.84701
;~ 0x5300F75099D47E275A9F6C43AC501469   Ward:               16.96491
;~ 0x5300F75099D47E275A9F6C43AC501469   trancexx:           3.74323
;~ 0x5300F75099D47E275A9F6C43AC501469   trancexx KaFu loop: 4.37193

;~ file 5 - 1 061 624 491 bytes
;~ 0x0DA1C47D0666154B8CA0F717F272DF68   AutoIt:             11.75191
;~ 0x0DA1C47D0666154B8CA0F717F272DF68   Ward:               10.53561
;~ 0x0DA1C47D0666154B8CA0F717F272DF68   trancexx:           2.34701
;~ 0x0DA1C47D0666154B8CA0F717F272DF68   trancexx KaFu loop: 2.74187

;~ file 6 - 874 719 232 bytes
;~ 0xB9803CCA589DDD2282FC72A98DB190C5   AutoIt:             11.08526
;~ 0xB9803CCA589DDD2282FC72A98DB190C5   Ward:               8.85178
;~ 0xB9803CCA589DDD2282FC72A98DB190C5   trancexx:           1.94553
;~ 0xB9803CCA589DDD2282FC72A98DB190C5   trancexx KaFu loop: 2.27487

;~ file 7 - 502 623 560 bytes
;~ 0xCBA697798DF0F97F7F77F69323844305   AutoIt:             6.16212
;~ 0xCBA697798DF0F97F7F77F69323844305   Ward:               4.99015
;~ 0xCBA697798DF0F97F7F77F69323844305   trancexx:           1.11381
;~ 0xCBA697798DF0F97F7F77F69323844305   trancexx KaFu loop: 1.30208

;~ file 8 - 465 462 751 bytes
;~ 0x5C853441C5607BC368AF1A3AB6F9197C   AutoIt:             5.07992
;~ 0x5C853441C5607BC368AF1A3AB6F9197C   Ward:               4.8146
;~ 0x5C853441C5607BC368AF1A3AB6F9197C   trancexx:           1.03213
;~ 0x5C853441C5607BC368AF1A3AB6F9197C   trancexx KaFu loop: 1.21896

;~ file 9 - 417 966 540 bytes
;~ 0xE4CF02FF783E6F8E43C457F1D0B18C2A   AutoIt:             4.48574
;~ 0xE4CF02FF783E6F8E43C457F1D0B18C2A   Ward:               4.17559
;~ 0xE4CF02FF783E6F8E43C457F1D0B18C2A   trancexx:           0.92816
;~ 0xE4CF02FF783E6F8E43C457F1D0B18C2A   trancexx KaFu loop: 1.08682

;~ file 10 - 308 718 465 bytes
;~ 0x4CE09676A6DCD8861C72E08C1F5CD674   AutoIt:             3.02671
;~ 0x4CE09676A6DCD8861C72E08C1F5CD674   Ward:               3.0758
;~ 0x4CE09676A6DCD8861C72E08C1F5CD674   trancexx:           0.68433
;~ 0x4CE09676A6DCD8861C72E08C1F5CD674   trancexx KaFu loop: 0.80289

;~ file 11 - 257 518 137 bytes
;~ 0xA3BFA6A0446153943BE1BC0A42B36049   AutoIt:             3.38988
;~ 0xA3BFA6A0446153943BE1BC0A42B36049   Ward:               2.58057
;~ 0xA3BFA6A0446153943BE1BC0A42B36049   trancexx:           0.5766
;~ 0xA3BFA6A0446153943BE1BC0A42B36049   trancexx KaFu loop: 0.68594

;~ file 12 - 154 540 032 bytes
;~ 0xBE0EF9F1417B402C704A96C4E1ACBC3F   AutoIt:             1.7613
;~ 0xBE0EF9F1417B402C704A96C4E1ACBC3F   Ward:               1.55319
;~ 0xBE0EF9F1417B402C704A96C4E1ACBC3F   trancexx:           0.3431
;~ 0xBE0EF9F1417B402C704A96C4E1ACBC3F   trancexx KaFu loop: 0.41976

;~ file 13 - 103 694 336 bytes
;~ 0xC613B584EEDAA2EBDAA632D4FEA384E0   AutoIt:             1.17679
;~ 0xC613B584EEDAA2EBDAA632D4FEA384E0   Ward:               1.04872
;~ 0xC613B584EEDAA2EBDAA632D4FEA384E0   trancexx:           0.2305
;~ 0xC613B584EEDAA2EBDAA632D4FEA384E0   trancexx KaFu loop: 0.27598

;~ file 14 - 51 460 096 bytes
;~ 0x1EC05C6087BCC09B206E955765D36AA5   AutoIt:             0.58349
;~ 0x1EC05C6087BCC09B206E955765D36AA5   Ward:               0.52024
;~ 0x1EC05C6087BCC09B206E955765D36AA5   trancexx:           0.11452
;~ 0x1EC05C6087BCC09B206E955765D36AA5   trancexx KaFu loop: 0.1405

;~ file 15 - 23 477 240 bytes
;~ 0x09B4DF75B57669F89F7B0AAA542B5701   AutoIt:             0.2437
;~ 0x09B4DF75B57669F89F7B0AAA542B5701   Ward:               0.23622
;~ 0x09B4DF75B57669F89F7B0AAA542B5701   trancexx:           0.05252
;~ 0x09B4DF75B57669F89F7B0AAA542B5701   trancexx KaFu loop: 0.06804

;~ file 16 - 16 777 234 bytes
;~ 0x7E54D27E183ECAADCD40CC3678A88D00   AutoIt:             0.21001
;~ 0x7E54D27E183ECAADCD40CC3678A88D00   Ward:               0.16981
;~ 0x7E54D27E183ECAADCD40CC3678A88D00   trancexx:           0.03751
;~ 0x7E54D27E183ECAADCD40CC3678A88D00   trancexx KaFu loop: 0.04984

;~ file 17 - 11 171 648 bytes
;~ 0x7629625F3CAA89F2E49FFB6A546FD3B5   AutoIt:             0.13932
;~ 0x7629625F3CAA89F2E49FFB6A546FD3B5   Ward:               0.11384
;~ 0x7629625F3CAA89F2E49FFB6A546FD3B5   trancexx:           0.02517
;~ 0x7629625F3CAA89F2E49FFB6A546FD3B5   trancexx KaFu loop: 0.0355

;~ file 18 - 5 592 560 bytes
;~ 0x6F44D565E29560EA0EB60564F78D3F05   AutoIt:             0.08437
;~ 0x6F44D565E29560EA0EB60564F78D3F05   Ward:               0.05616
;~ 0x6F44D565E29560EA0EB60564F78D3F05   trancexx:           0.0127
;~ 0x6F44D565E29560EA0EB60564F78D3F05   trancexx KaFu loop: 0.02193

;~ file 19 - 2 348 343 bytes
;~ 0xF0DBB928FF2F13E256E78519BE115AA7   AutoIt:             0.04418
;~ 0xF0DBB928FF2F13E256E78519BE115AA7   Ward:               0.0231
;~ 0xF0DBB928FF2F13E256E78519BE115AA7   trancexx:           0.00562
;~ 0xF0DBB928FF2F13E256E78519BE115AA7   trancexx KaFu loop: 0.01193

;~ file 20 - 1 081 045 bytes
;~ 0xE69D04B2D335AE2902388DE5B1AE4ADD   AutoIt:             0.08075
;~ 0xE69D04B2D335AE2902388DE5B1AE4ADD   Ward:               0.01595
;~ 0xE69D04B2D335AE2902388DE5B1AE4ADD   trancexx:           0.00419
;~ 0xE69D04B2D335AE2902388DE5B1AE4ADD   trancexx KaFu loop: 0.01144

;~ file 21 - 523 938 bytes
;~ 0xAB7A34FD227F297F87E00BA480995970   AutoIt:             0.08225
;~ 0xAB7A34FD227F297F87E00BA480995970   Ward:               0.00772
;~ 0xAB7A34FD227F297F87E00BA480995970   trancexx:           0.00232
;~ 0xAB7A34FD227F297F87E00BA480995970   trancexx KaFu loop: 0.00912

;~ file 22 - 272 006 bytes
;~ 0xD47914C5C493FF6CDC72A8F5E12BC278   AutoIt:             0.00359
;~ 0xD47914C5C493FF6CDC72A8F5E12BC278   Ward:               0.00292
;~ 0xD47914C5C493FF6CDC72A8F5E12BC278   trancexx:           0.0009
;~ 0xD47914C5C493FF6CDC72A8F5E12BC278   trancexx KaFu loop: 0.00772

;~ file 23 - 262 104 bytes
;~ 0x608DA9EF056FFCC4A801B02D60C1F4F6   AutoIt:             0.06753
;~ 0x608DA9EF056FFCC4A801B02D60C1F4F6   Ward:               0.00315
;~ 0x608DA9EF056FFCC4A801B02D60C1F4F6   trancexx:           0.00135
;~ 0x608DA9EF056FFCC4A801B02D60C1F4F6   trancexx KaFu loop: 0.00799

;~ file 24 - 129 514 bytes
;~ 0x78C1F2C312EEBE871F5EA2DF4A379DDD   AutoIt:             0.06359
;~ 0x78C1F2C312EEBE871F5EA2DF4A379DDD   Ward:               0.00197
;~ 0x78C1F2C312EEBE871F5EA2DF4A379DDD   trancexx:           0.00095
;~ 0x78C1F2C312EEBE871F5EA2DF4A379DDD   trancexx KaFu loop: 0.00756

;~ file 25 - 65 180 bytes
;~ 0x9375FB719F32BE2997291ADF30251A9F   AutoIt:             0.06624
;~ 0x9375FB719F32BE2997291ADF30251A9F   Ward:               0.00109
;~ 0x9375FB719F32BE2997291ADF30251A9F   trancexx:           0.00065
;~ 0x9375FB719F32BE2997291ADF30251A9F   trancexx KaFu loop: 0.00782

;~ file 26 - 32 641 bytes
;~ 0x25F9184E7C1504D4D772F6E7FBD26EBE   AutoIt:             0.00227
;~ 0x25F9184E7C1504D4D772F6E7FBD26EBE   Ward:               0.00071
;~ 0x25F9184E7C1504D4D772F6E7FBD26EBE   trancexx:           0.00056
;~ 0x25F9184E7C1504D4D772F6E7FBD26EBE   trancexx KaFu loop: 0.00729

;~ file 27 - 16 235 bytes
;~ 0xCB62BF22679273F50688032240B4C9FF   AutoIt:             0.00119
;~ 0xCB62BF22679273F50688032240B4C9FF   Ward:               0.00057
;~ 0xCB62BF22679273F50688032240B4C9FF   trancexx:           0.00055
;~ 0xCB62BF22679273F50688032240B4C9FF   trancexx KaFu loop: 0.00732

;~ file 28 - 6 357 bytes
;~ 0xE19B7AF1827971780B398917B2547A93   AutoIt:             0.01017
;~ 0xE19B7AF1827971780B398917B2547A93   Ward:               0.00028
;~ 0xE19B7AF1827971780B398917B2547A93   trancexx:           0.00028
;~ 0xE19B7AF1827971780B398917B2547A93   trancexx KaFu loop: 0.00531

;~ file 29 - 1 120 bytes
;~ 0x254505EFA186EB805BFB53DF215170E5   AutoIt:             0.00848
;~ 0x254505EFA186EB805BFB53DF215170E5   Ward:               0.00025
;~ 0x254505EFA186EB805BFB53DF215170E5   trancexx:           0.00027
;~ 0x254505EFA186EB805BFB53DF215170E5   trancexx KaFu loop: 0.00533

;~ file 30 - 832 bytes
;~ 0x7DB8F0455C4540F2C8D4D4AA54F3A6BF   AutoIt:             0.01178
;~ 0x7DB8F0455C4540F2C8D4D4AA54F3A6BF   Ward:               0.00025
;~ 0x7DB8F0455C4540F2C8D4D4AA54F3A6BF   trancexx:           0.00028
;~ 0x7DB8F0455C4540F2C8D4D4AA54F3A6BF   trancexx KaFu loop: 0.00551

;~ file 31 - 41 bytes
;~ 0xDF23D4FCFBD134ABA539FEC167EFFF42   AutoIt:             0.00063
;~ 0xDF23D4FCFBD134ABA539FEC167EFFF42   Ward:               0.00024
;~ 0xDF23D4FCFBD134ABA539FEC167EFFF42   trancexx:           0.00028
;~ 0xDF23D4FCFBD134ABA539FEC167EFFF42   trancexx KaFu loop: 0.00606

;~ file 32 - 0 bytes
;~ 0xD41D8CD98F00B204E9800998ECF8427E   AutoIt:             0.00055
;~ 0xD41D8CD98F00B204E9800998ECF8427E   Ward:               0.00016
;~ 0x                                   trancexx:           0.00011 --error code: 2
;~ 0xD41D8CD98F00B204E9800998ECF8427E   trancexx KaFu loop: 0.00616

Here is the code I used to do these tests:

expandcollapse popup

#include <crypt.au3> ;for AutoIt way
#include <md5.au3> ;for Ward way
Func numbersplit($number, $separator = ' ')
    $number = Round($number)
    $count = StringLen($number)
    If $count < 4 Then Return $number

    $countsplit = Ceiling($count / 3)
    $firstsplit = Mod($count, 3)

    $finalstring = StringLeft($number, $firstsplit)
    If $firstsplit <> 0 Then $finalstring &= $separator

    For $i=0 To $countsplit Step 1
        $start = 1 + $firstsplit + 3*$i
        $finalstring &= StringMid($number, $start, 3)
        If $i < $countsplit-1 Then $finalstring &= $separator
    Next

    If $firstsplit <> 0 Then $finalstring = StringTrimRight($finalstring, 1)
    Return $finalstring
EndFunc

#include<winapiex.au3>
; #FUNCTION# ;===============================================================================
;
; Name...........: _MD5ForFile
; Description ...: Calculates MD5 value for the specific file.
; Syntax.........: _MD5ForFile ($sFile)
; Parameters ....: $sFile - Full path to the file to process.
; Return values .: Success - Returns MD5 value in form of hex string
;                          - Sets @error to 0
;                  Failure - Returns empty string and sets @error:
;                  |1 - CreateFile function or call to it failed.
;                  |2 - CreateFileMapping function or call to it failed.
;                  |3 - MapViewOfFile function or call to it failed.
;                  |4 - MD5Init function or call to it failed.
;                  |5 - MD5Update function or call to it failed.
;                  |6 - MD5Final function or call to it failed.
; Author ........: trancexx
;
;==========================================================================================
Func _MD5ForFile($sFile)
;_WinAPI_Wow64EnableWow64FsRedirection(False) ;It's only for system files redirected by WOW3264, don't use it for every files.
    Local $a_hCall = DllCall("kernel32.dll", "hwnd", "CreateFileW", _
            "wstr", $sFile, _
            "dword", 0x80000000, _ ; GENERIC_READ
            "dword", 3, _ ; FILE_SHARE_READ|FILE_SHARE_WRITE
            "ptr", 0, _
            "dword", 3, _ ; OPEN_EXISTING
            "dword", 0, _ ; SECURITY_ANONYMOUS
            "ptr", 0)

    If @error Or $a_hCall[0] = -1 Then
;       _WinAPI_Wow64EnableWow64FsRedirection(True) ;it's only for system files redirected by WOW3264
        Return SetError(1, 0, "")
    EndIf
;_WinAPI_Wow64EnableWow64FsRedirection(True) ;it's only for system files redirected by WOW3264
    Local $hFile = $a_hCall[0]

    $a_hCall = DllCall("kernel32.dll", "ptr", "CreateFileMappingW", _
            "hwnd", $hFile, _
            "dword", 0, _ ; default security descriptor
            "dword", 2, _ ; PAGE_READONLY
            "dword", 0, _
            "dword", 0, _
            "ptr", 0)

    If @error Or Not $a_hCall[0] Then
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFile)
        Return SetError(2, 0, "")
    EndIf

    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFile)

    Local $hFileMappingObject = $a_hCall[0]

    $a_hCall = DllCall("kernel32.dll", "ptr", "MapViewOfFile", _
            "hwnd", $hFileMappingObject, _
            "dword", 4, _ ; FILE_MAP_READ
            "dword", 0, _
            "dword", 0, _
            "dword", 0)

    If @error Or Not $a_hCall[0] Then
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(3, 0, "")
    EndIf

    Local $pFile = $a_hCall[0]
    ;_WinAPI_Wow64EnableWow64FsRedirection(False)
    Local $iBufferSize = FileGetSize($sFile)
    ;_WinAPI_Wow64EnableWow64FsRedirection(True)

    Local $tMD5_CTX = DllStructCreate("dword i[2];" & _
            "dword buf[4];" & _
            "ubyte in[64];" & _
            "ubyte digest[16]")

    DllCall("advapi32.dll", "none", "MD5Init", "ptr", DllStructGetPtr($tMD5_CTX))

    If @error Then
        DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(4, 0, "")
    EndIf

    DllCall("advapi32.dll", "none", "MD5Update", _
            "ptr", DllStructGetPtr($tMD5_CTX), _
            "ptr", $pFile, _
            "dword", $iBufferSize)

    If @error Then
        DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(5, 0, "")
    EndIf

    DllCall("advapi32.dll", "none", "MD5Final", "ptr", DllStructGetPtr($tMD5_CTX))

    If @error Then
        DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
        DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)
        Return SetError(6, 0, "")
    EndIf

    DllCall("kernel32.dll", "int", "UnmapViewOfFile", "ptr", $pFile)
    DllCall("kernel32.dll", "int", "CloseHandle", "hwnd", $hFileMappingObject)

    Local $sMD5 = Hex(DllStructGetData($tMD5_CTX, "digest"))

    Return SetError(0, 0, $sMD5)

EndFunc   ;==>_MD5ForFile

#include<WinAPI.au3>
Func md5_KaFu($Checksum_Filename)
    local $nBytes
    ;$Checksum_Filename = @ScriptDir & "\Testfile.exe"

    Local $MD5_CTX = DllStructCreate("dword i[2];" & _
            "dword buf[4];" & _
            "ubyte in[64];" & _
            "ubyte digest[16]")

    DllCall("advapi32.dll", "none", "MD5Init", "ptr", DllStructGetPtr($MD5_CTX))
;_WinAPI_Wow64EnableWow64FsRedirection(False)
    $hFile = _WinAPI_CreateFile($Checksum_Filename, 2, 2)
;_WinAPI_Wow64EnableWow64FsRedirection(True)
;_WinAPI_Wow64EnableWow64FsRedirection(False)
    $iFileGetSize_Save = FileGetSize($Checksum_Filename)
;_WinAPI_Wow64EnableWow64FsRedirection(True)
    $iBuffersize = 1024 * 1024 * 10
    Local $tBuffer = DllStructCreate("byte[" & $iBuffersize & "]")

    For $i = 1 To Ceiling($iFileGetSize_Save / $iBuffersize)
        _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), $iBuffersize, $nBytes)
        DllCall("advapi32.dll", "none", "MD5Update", _
                "ptr", DllStructGetPtr($MD5_CTX), _
                "ptr", DllStructGetPtr($tBuffer), _
                "dword", $nBytes)
    Next

    If IsHWnd($hFile) Then _WinAPI_CloseHandle($hFile)

    DllCall("advapi32.dll", "none", "MD5Final", "ptr", DllStructGetPtr($MD5_CTX))

    ;ConsoleWrite('MD5 = 0x' & Hex(DllStructGetData($tMD5_CTX, "digest")) & @crlf)
    Return '0x' & Hex(DllStructGetData($MD5_CTX, "digest"))
EndFunc

;~ ;WarD way
Local $BufferSize = 0x100000
Func md5_w($sFile)
    $State = _MD5Init()
    Dim $hFile = FileOpen($sFile, 16)
    For $i = 1 To Ceiling(FileGetSize($sFile) / $BufferSize)
        _MD5Input($State, FileRead($hFile, $BufferSize))
    Next
    $digest = _MD5Result($State)
    FileClose($hFile)
;ConsoleWrite(@LF & $bDigest & " - " & $sFile & '  Ward')
    Return $digest
EndFunc

;~ ;AutoIt way
Func md5_ai($sFile, $iALG_ID=$CALG_MD5)
    $bDigest=_Crypt_HashFile($sFile,$iALG_ID)
;ConsoleWrite(@LF & $bDigest & " - " & $sFile & '  AutoIt')
    Return $bDigest
EndFunc

Local $aFiles[2] = ["c:\windows\explorer.exe","c:\windows\regedit.exe"]

Local $stat = '', $stats = ''
For $i = 0 To UBound($aFiles)-1; Step -1
    $stat = @LF & 'file ' & $i+1 & ' ' & $aFiles[$i] & " - " & numbersplit(FileGetSize($aFiles[$i])) & ' bytes'
ConsoleWrite($stat)
    $stats &= $stat
    $timee = TimerInit()
    ;_Crypt_Startup()
    $digest = md5_ai($aFiles[$i])
    $time = TimerDiff($timee)/1000
    ;_Crypt_Shutdown()
    $stat = @LF & $digest & ' ' & '  autoit:' & @TAB & Round($time, 5)
    $stats &= $stat
ConsoleWrite($stat)
    $timee = TimerInit()
    $digest = md5_w($aFiles[$i])
    $time = TimerDiff($timee)/1000
    $stat = @LF & $digest & ' ' & '  ward:' & @TAB & Round($time, 5)
    $stats &= $stat
ConsoleWrite($stat)
    $timee = TimerInit()
    $digest = md5_kafu($aFiles[$i])
    $time = TimerDiff($timee)/1000
    $stat = @LF & $digest & ' ' & '  tx_kafu:' & @TAB & Round($time, 5)
    $stats &= $stat
ConsoleWrite($stat)
    $timee = TimerInit()
    $digest = _MD5ForFile($aFiles[$i])
    $error = @error
    $time = TimerDiff($timee)/1000
    $stat = @LF & '0x' & $digest & ' ' & '  tx:' & @tab & Round($time, 5) & ' --error code: ' & $error
    $stats &= $stat
ConsoleWrite($stat & @lf)
Next

These tests allowed me to spot several things:
- trancexx MD5 (OP) is the fastest for 6KB files and upper, up to, I guess, memory available.
- Ward (here) is the fastest for files below 6KB (but really close to trancexx results).
- Kafu loop with trancexx idea (post #13) is slower than OP function, but really faster than Ward and AutoIt function above 1MB but only up to, I guess, memory available, afterward it becomes the slowest.
 
So, I decided to use trancexx function, and if @error was different from 0, switch to AutoIt or Ward way. This way I could get a fast MD5 calculator without doing further test (trancexx error code is returned really fast).
 
Then I felt on some problems with trancexx OP function:
- the function doesn't read target informations from reparse points (which it should do with CreateFileW parameters given), funky thing is that it returns a correct empty file hash ^^, there is no problem for files that are part of a folder RP.

    -> a quick solution is to send the target path if the file is a reparse point.

- the function makes AutoIt do a fatal error (read in protected memory 0xC0000005) with, i.e. regedit.exe, but I assume with all system files that are redirected by WOW3264.
    -> a way to avoid this error is to de-activate the WOW3264 redirection. (Like in code example for trancexx and KaFu loop functions, with _WinAPI_Wow64EnableWow64FsRedirection()

edit: The pink text is still half-true, the thing is I was mistaken about the source of the problem: I got other file with this error after modifying the code: this 2nd system error was send because in trancexx code (and KaFu loop) there is a FileGetSize() to get the memory to allocate and which also requires to de-activate system redirections. So the problem was partly with allocation size, it seems.
 

Even if I found workarounds, I'd like to know if there are other ways to do:
- Is it possible to modify trancexx function so that it reads RP target instead of RP itself ?
- Is it possible to make it work with system protected files, like others methods do? In other words, is it possible to make the function works with WOW3264 redirection?

I think the global question would be: Why this function bypass the file system (NTFS) and Windows system (WOW3264) redirections?

- Is it possible to catch the real limitor of OP function, I mean what makes it return an error code if filesize is too big? What (memory?) variable should we check?

 

Anyway, thanks a lot for this function, it's unbelievably fast!

 

edit:

- added _WinAPI_Wow64EnableWow64FsRedirection() in posted code at filegetsize() levels, still only for trancexx and KaFu loop in order to compare what it gives with redirections.

- commented my post ^^

Edited December 18, 2013 by AvvA

[KaFu]

- Kafu loop with trancexx idea (post #13) is slower than OP function, but really faster than Ward and AutoIt function above 1MB but only up to, I guess, memory available, afterward it becomes the slowest.

 

Nope, wrong guess . That was the whole Idea of looping and re-using a 10MB buffer for every cycle, the function "should" process all files regardless of their size.

Edit:

You might want to give SMF a try, there I globally define a buffer and re-use it for all files... but I have to admit it's source is terrible to read ...

Edited December 18, 2013 by KaFu

[AvvA]

Yes, your loop allows to check any file size, but at a cost of a lot of time when original trancexx function doesn't work with this file size. It comes from the results I gave in the first, well, the second "code block". You can check it with file 1, 2 and 3. Of course, files of this size aren't common, but the results are here ^^.

The guess was a "memory available" guess (on DllCall("advapi32.dll", "none", "MD5Update",...), and I was talking about checksum calculation speed.

To resume it: If trancexx OP function fails, your trancexx-loop is slower than AutoIt and Ward functions (and a lot slower: +1/3).

I already planned to have a look at SMF (and its source). I keep it for when I'll have finished my program to compare the duplicates "find time", it shouldn't be very long from now.

Edited December 18, 2013 by AvvA

[Iczer]

i'm trying to use trancexx script from first post to get crc32 and md5 for files in my backup project... so 2 (maybe stupid ^_^' ) questions:

- it seems there are some cashing occuring in the first call (64bit):

first run:

! CRC32 1st call took 1673.84318398009 ms
! CRC32 2nd call took 577.403552686165 ms
! MD5 took 587.722741298126 ms

second run:

! CRC32 1st call took 612.193728532537 ms
! CRC32 2nd call took 577.421990783745 ms
! MD5 took 587.62887461954 ms

- is it possible to use that, already cashed file, and write it to the another folder directly (so i save time needed to read file again to be able to copy it to another folder after calculating crc32 and md5 ) ?

- is it possible to make calculations of crc32 and md5 in the same time on different cores ?

- it seems calculations of crc32 and md5 take pretty same time - is it right?

if calculations made in 32bit script? it take:

! CRC32 1st call took 544.092335757757 ms
! CRC32 2nd call took 543.410684877547 ms
! MD5 took 452.442419357768 ms

it somewhat faster... why?

Edited April 17, 2014 by Iczer