Searching for the *POSITION* from that on files are different.

[rudi]

Hi.

I have a problem with a certain Server. It happens sometimes, that large files copied to the server are not 100% copied. The resulting file on the server is "Filled with NULL" starting at the point of copy break down, which *CAN* be not noticed if certain conditions are met. (set NCP signature option = 0)

This code does find the position fine, but it's really slow: Giga link, and approxymately only 1MB/s are processed.

My approach is to binary read parts of 1MB to memory and to compare them. (it's enough to me if I just get the "trouble position" with an accuracy of +/- 1MB)

Howto improve?

expandcollapse popup

#cs
    Author: Rudolf Thilo
    Suche nach der Position, *AB* wo sich 2 Dateien unterscheiden.
#ce

$Orginal="X:\download\Original-file-600Meg.zip" ; local disk. 
$Kopie="H:\Restore\corrupted\Copy-of-Original-File-600Meg.zip" ; Network Drive
$MaxCompare=FileGetSize("File1.zip")

$Block = 1024 * 1024 ; Ein Happen = 1 Megabyte
$FullBlocks = Floor($MaxCompare / $Block)
$Rest = $MaxCompare - $FullBlocks * $Block
ConsoleWrite("full size         : " & $MaxCompare & @CRLF)
ConsoleWrite("# of 1 Meg blocs  : " & $FullBlocks & @CRLF)
ConsoleWrite("Size of all blocks: " &$FullBlocks * $Block & @CRLF)
ConsoleWrite("Rest after blocks : " & $Rest & @CRLF)
ConsoleWrite("calced together   : " & $FullBlocks * $Block + $Rest & @CRLF)


$OHandle = FileOpen($Orginal, 16) ; Lesen; Binary
$KHandle = FileOpen($Kopie, 16) ; Lesen; Binary

Dim $Compared=0

While 1
    ConsoleWrite("Block Count: " & $FullBlocks & ", current block: " & $Compared & @CRLF)
    $OrgHappen = FileRead($OHandle, $Block)
    if @error then 
        ConsoleWrite("Failing to read Orginal file: @Error = " & @error & @CRLF & $Orginal & @crlf)
        Exit
    EndIf
    
    $KopHappen = FileRead($KHandle, $Block)
    if @error then 
        ConsoleWrite("Failing to read Copied file  : @Error = " & @error & @CRLF & $Kopie & @CRLF)
        Exit
    EndIf
    
    If StringCompare($OrgHappen, $KopHappen, 1) <> 0 Then
        ; Unterschied gefunden.
        $DiffAt = $Compared * $Block
        ConsoleWrite("Differnce at: " & $DiffAt & @CRLF)
        Exit
    EndIf
    $Compared += 1
    If $Compared = $FullBlocks Then
        $OrgHappen = FileRead($OHandle, $Rest)
        $KopHappen = FileRead($KHandle, $Rest)
        If StringCompare($OrgHappen, $KopHappen, 1) <> 0 Then
            ; Unterschied gefunden.
            $DiffAt = $Compared * $Block
            ConsoleWrite("Difference at: " & $DiffAt & @CRLF)
            Exit
        EndIf
        ; ende erreicht
        ConsoleWrite("No differences found." & @CRLF)
        Exit
    EndIf
WEnd

Regards, Rudi.

Edit: Typo

Edit2: syntax nonsense, by translating to English .... : $DiffAt = $Compared & $Block > $DiffAt = $Compared * $Block

Edited November 3, 2008 by rudi

[rudi]

Bump.

I've used this code already successfully. Anyways, I'm interested how it could be improved to speed up.

Thanks, Rudi.

Hi.

I have a problem with a certain Server. It happens sometimes, that large files copied to the server are not 100% copied. The resulting file on the server is "Filled with NULL" starting at the point of copy break down, which *CAN* be not noticed if certain conditions are met. (set NCP signature option = 0)

This code does find the position fine, but it's really slow: Giga link, and approxymately only 1MB/s are processed.

My approach is to binary read parts of 1MB to memory and to compare them. (it's enough to me if I just get the "trouble position" with an accuracy of +/- 1MB)

Howto improve?

expandcollapse popup

#cs
    Author: Rudolf Thilo
    Suche nach der Position, *AB* wo sich 2 Dateien unterscheiden.
#ce

$Orginal="X:\download\Original-file-600Meg.zip" ; local disk. 
$Kopie="H:\Restore\corrupted\Copy-of-Original-File-600Meg.zip" ; Network Drive
$MaxCompare=FileGetSize("File1.zip")

$Block = 1024 * 1024 ; Ein Happen = 1 Megabyte
$FullBlocks = Floor($MaxCompare / $Block)
$Rest = $MaxCompare - $FullBlocks * $Block
ConsoleWrite("full size         : " & $MaxCompare & @CRLF)
ConsoleWrite("# of 1 Meg blocs  : " & $FullBlocks & @CRLF)
ConsoleWrite("Size of all blocks: " &$FullBlocks * $Block & @CRLF)
ConsoleWrite("Rest after blocks : " & $Rest & @CRLF)
ConsoleWrite("calced together   : " & $FullBlocks * $Block + $Rest & @CRLF)


$OHandle = FileOpen($Orginal, 16) ; Lesen; Binary
$KHandle = FileOpen($Kopie, 16) ; Lesen; Binary

Dim $Compared=0

While 1
    ConsoleWrite("Block Count: " & $FullBlocks & ", current block: " & $Compared & @CRLF)
    $OrgHappen = FileRead($OHandle, $Block)
    if @error then 
        ConsoleWrite("Failing to read Orginal file: @Error = " & @error & @CRLF & $Orginal & @crlf)
        Exit
    EndIf
    
    $KopHappen = FileRead($KHandle, $Block)
    if @error then 
        ConsoleWrite("Failing to read Copied file  : @Error = " & @error & @CRLF & $Kopie & @CRLF)
        Exit
    EndIf
    
    If StringCompare($OrgHappen, $KopHappen, 1) <> 0 Then
        ; Unterschied gefunden.
        $DiffAt = $Compared * $Block
        ConsoleWrite("Differnce at: " & $DiffAt & @CRLF)
        Exit
    EndIf
    $Compared += 1
    If $Compared = $FullBlocks Then
        $OrgHappen = FileRead($OHandle, $Rest)
        $KopHappen = FileRead($KHandle, $Rest)
        If StringCompare($OrgHappen, $KopHappen, 1) <> 0 Then
            ; Unterschied gefunden.
            $DiffAt = $Compared * $Block
            ConsoleWrite("Difference at: " & $DiffAt & @CRLF)
            Exit
        EndIf
        ; ende erreicht
        ConsoleWrite("No differences found." & @CRLF)
        Exit
    EndIf
WEnd

Regards, Rudi.

Edit: Typo

Edit2: syntax nonsense, by translating to English .... : $DiffAt = $Compared & $Block > $DiffAt = $Compared * $Block

[rover]

Hi.

I have a problem with a certain Server. It happens sometimes, that large files copied to the server are not 100% copied. The resulting file on the server is "Filled with NULL" starting at the point of copy break down, which *CAN* be not noticed if certain conditions are met. (set NCP signature option = 0)

This code does find the position fine, but it's really slow: Giga link, and approxymately only 1MB/s are processed.

My approach is to binary read parts of 1MB to memory and to compare them. (it's enough to me if I just get the "trouble position" with an accuracy of +/- 1MB)

Howto improve?

removed posted code, it obviously must be rubbish if it doesn't rate comment. Edited November 11, 2008 by rover

[kjcdude]

I've delt with MD5 comparing and it seems to be extremely slow in autoit.

In some instances i found it faster to copy all than md5 compare the files and copy the changed ones.

I've resorted to comparing filesize, modified date, and created date.

[rover]

I've delt with MD5 comparing and it seems to be extremely slow in autoit.

In some instances i found it faster to copy all than md5 compare the files and copy the changed ones.

I've resorted to comparing filesize, modified date, and created date.

Hi kjcdude

yes, very slow unless using plugins (external dll) or LazyCats run assembler code hack (very fast)

most practical would be an external file compare exe.

that's what I was thinking of instead of doing it in AutoIt.

OP seems to have file corruption problem with a server

and wants the file position (approximate +- 1mb) where errors begin.

Cheers

[rudi]

Hi Rover.

removed posted code, it obviously must be rubbish if it doesn't rate comment.

Pardon? Sorry, I absolutely miss what that shall tell me?

Can you say it using slang free English, pls?

Regards, Rudi.