BinaryLen() size differ on different computers when combining data

[Factfinder]

This could be a challenge. I have used the following code to debug a size issue when combining data to upload a file:

$fpath = @SystemDir & "\userinit.exe"
$hopen = FileOpen($fpath, 16)
$fcontent = FileRead($hopen)
$fcontent = BinaryToString($fcontent)
$filename = StringRegExpReplace($fpath, ".+\\", "")
$token = 'qpqpqpqpqp'
$boundary = "----158082776"

$data1 = '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="scanToken"'  & @CRLF & @CRLF & $token & @CRLF
$data1 &= '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="file"; filename="' & $filename & '"' &@CRLF
$data1 &= 'Content-Type: application/x-msdownload' & @CRLF & @CRLF
$data2 = @CRLF & '--' & $boundary & '--'
$data = $data1 & $fcontent & $data2

MsgBox(0, FileGetSize($fpath)+ BinaryLen($data1&$data2),  BinaryLen($data))

I run the script in x32 mode and get a smaller size for BinaryLen($data) on my PC  (40900 bytes) while the size of FileGetSize($fpath)+ BinaryLen($data1&$data2)  shows the expected size (41701 bytes).

I ran the same script on my laptop; both sizes are correct and the same. Both the machines have Windows 10 Home Edition. I ran it on another machine with Windows 11 and the sizes are correct and equal.

I need the exact size to be able to upload the file and it seems that on my main computer combining data can lead to distortion. To make sure I have made a clean install of Autoit on my PC but it didn't help.

I would appreciate it if anyone has seen this before or could advise me.

Thank you in advance.

Edited March 3 by Factfinder

[SmOke_N]

I'm just curious, why are you using BinaryLen() on a string?

Are the files on the x86 machine 32bit and 64bit on the x64 machines?

[Nine]

Probably redirection (since you are reading from the system folder).

try to put this at the start of your script : _WinAPI_Wow64EnableWow64FsRedirection(False)

[Factfinder]

1 hour ago, SmOke_N said:

I'm just curious, why are you using BinaryLen() on a string?

Are the files on the x86 machine 32bit and 64bit on the x64 machines?

I use BinaryLen() because I need the size in byte.

If you run the file script as 32bit on both x86 and x64 machines or you run a 64bit compiled script on the x64 machine, the result is the same. It is definitely not a redirection issue. The script for reading the file size and the file content would go to the same file.

[Factfinder]

1 hour ago, Nine said:

Probably redirection (since you are reading from the system folder).

try to put this at the start of your script : _WinAPI_Wow64EnableWow64FsRedirection(False)

It is not a redirection issue FileGetSize() and FileOpen() would work on the same file. Running the compiled x64 bit doesn't redirect either and the result for compiled x64 is the same.

It is probably something on my main system. On any other machine I have tried the script, it works as expected and the result is as it should be. Only on my main machine the result is inconsistent.

 

[Nine]

You are right it is not a redirection issue.  I should have thought of it a bit longer. FWIW, I tested your script on a few files and results are consistent on my side...

[argumentum]

1 hour ago, Factfinder said:

Only on my main machine the result is inconsistent.

OS version, language, BOM or not ?. All the same ?

[Factfinder]

1 hour ago, Nine said:

You are right it is not a redirection issue.  I should have thought of it a bit longer. FWIW, I tested your script on a few files and results are consistent on my side...

Yes, I have tested it too. Thanks for the confirmation.

[KaFu]

$fcontent = BinaryToString($fcontent, 4) gives exactly 40.900 bytes, what a coincidence 😉.

Maybe try to enforce the encoding with something like this?

$fcontent = BinaryToString($fcontent, 1)

 

[Factfinder]

1 hour ago, argumentum said:

OS version, language, BOM or not ?. All the same ?

 My own OS is English, but it works on another English language. It worked on a OS with a native language correctly.

The string is made as a string, I have used different files and the results are the same. Only on my main macine the results are inconsistent.

[Factfinder]

48 minutes ago, KaFu said:

$fcontent = BinaryToString($fcontent, 4) gives exactly 40.900 bytes, what a coincidence 😉.

Maybe try to enforce the encoding with something like this?

$fcontent = BinaryToString($fcontent, 1)

 

In fact, I'm already using $fcontent = BinaryToString($fcontent, 1) as when using $fcontent = BinaryToString($fcontent)  the default flag (1) is used. I have used BinaryToString($fcontent, 1) too but no change.

It should be related to my main system as it works on other systems.

Edited March 3 by Factfinder
Edited to add that the suggestion is tried.

[KaFu]

With BinaryToString($fcontent, 4) giving exactly 40.900 bytes, it definitely has something to do with encoding.

Give $hopen = FileOpen($fpath, 512) a try, maybe also try MsgBox(0,"",FileGetEncoding($fpath)) on both machines.

Also compare the initial lines of the file on both computers, are they really identical (both with BOM / without BOM)?

expandcollapse popup

$fpath = @SystemDir & "\userinit.exe"
$hopen = FileOpen($fpath, 512)
$fcontent = FileRead($hopen)
$filename = StringRegExpReplace($fpath, ".+\\", "")
$token = 'qpqpqpqpqp'
$boundary = "----158082776"

$data1 = '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="scanToken"' & @CRLF & @CRLF & $token & @CRLF
$data1 &= '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="file"; filename="' & $filename & '"' & @CRLF
$data1 &= 'Content-Type: application/x-msdownload' & @CRLF & @CRLF
$data2 = @CRLF & '--' & $boundary & '--'
$data = $data1 & $fcontent & $data2

MsgBox(0, FileGetSize($fpath) + BinaryLen($data1) + BinaryLen($data2), BinaryLen($data) & @CRLF & @CRLF & BinaryLen($fcontent))

MsgBox(0,"",_HexEncode($fcontent))

Func _HexEncode($bInput, $iFlags = 0x0000000b)
    ; CryptBinaryToString function (Windows)
    ; http://msdn.microsoft.com/en-us/library/windows/desktop/aa379887%28v=vs.85%29.aspx

    Local $tInput = DllStructCreate("byte[" & BinaryLen($bInput) & "]")
    DllStructSetData($tInput, 1, $bInput)
    Local $a_iCall = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "dword", $iFlags, _
            "ptr", 0, _
            "dword*", 0)

    If @error Or Not $a_iCall[0] Then
        Return SetError(1, 0, "")
    EndIf

    Local $iSize = $a_iCall[5]
    Local $tOut = DllStructCreate("char[" & $iSize & "]")

    $a_iCall = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "dword", $iFlags, _
            "ptr", DllStructGetPtr($tOut), _
            "dword*", $iSize)

    If @error Or Not $a_iCall[0] Then
        Return SetError(2, 0, "")
    EndIf

    Return SetError(0, 0, DllStructGetData($tOut, 1))

EndFunc   ;==>_HexEncode

 

Edited March 3 by KaFu

[argumentum]

5 hours ago, KaFu said:

Also compare the initial lines of the file on both computers, are they really identical (both with BOM / without BOM)?

9 hours ago, argumentum said:

BOM or not ?. All the same ?

That is my trend of thought too.

[Factfinder]

10 hours ago, KaFu said:

With BinaryToString($fcontent, 4) giving exactly 40.900 bytes, it definitely has something to do with encoding.

Give $hopen = FileOpen($fpath, 512) a try, maybe also try MsgBox(0,"",FileGetEncoding($fpath)) on both machines.

Also compare the initial lines of the file on both computers, are they really identical (both with BOM / without BOM)?

expandcollapse popup

$fpath = @SystemDir & "\userinit.exe"
$hopen = FileOpen($fpath, 512)
$fcontent = FileRead($hopen)
$filename = StringRegExpReplace($fpath, ".+\\", "")
$token = 'qpqpqpqpqp'
$boundary = "----158082776"

$data1 = '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="scanToken"' & @CRLF & @CRLF & $token & @CRLF
$data1 &= '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="file"; filename="' & $filename & '"' & @CRLF
$data1 &= 'Content-Type: application/x-msdownload' & @CRLF & @CRLF
$data2 = @CRLF & '--' & $boundary & '--'
$data = $data1 & $fcontent & $data2

MsgBox(0, FileGetSize($fpath) + BinaryLen($data1) + BinaryLen($data2), BinaryLen($data) & @CRLF & @CRLF & BinaryLen($fcontent))

MsgBox(0,"",_HexEncode($fcontent))

Func _HexEncode($bInput, $iFlags = 0x0000000b)
    ; CryptBinaryToString function (Windows)
    ; http://msdn.microsoft.com/en-us/library/windows/desktop/aa379887%28v=vs.85%29.aspx

    Local $tInput = DllStructCreate("byte[" & BinaryLen($bInput) & "]")
    DllStructSetData($tInput, 1, $bInput)
    Local $a_iCall = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "dword", $iFlags, _
            "ptr", 0, _
            "dword*", 0)

    If @error Or Not $a_iCall[0] Then
        Return SetError(1, 0, "")
    EndIf

    Local $iSize = $a_iCall[5]
    Local $tOut = DllStructCreate("char[" & $iSize & "]")

    $a_iCall = DllCall("Crypt32.dll", "int", "CryptBinaryToString", _
            "ptr", DllStructGetPtr($tInput), _
            "dword", DllStructGetSize($tInput), _
            "dword", $iFlags, _
            "ptr", DllStructGetPtr($tOut), _
            "dword*", $iSize)

    If @error Or Not $a_iCall[0] Then
        Return SetError(2, 0, "")
    EndIf

    Return SetError(0, 0, DllStructGetData($tOut, 1))

EndFunc   ;==>_HexEncode

 

Thank you for taking time.

I did another test to see if the issue was related to combining data.

$fpath = @SystemDir & "\userinit.exe"
$hopen = FileOpen($fpath, 16)
$fcontent = FileRead($hopen)
$fcontent = BinaryToString($fcontent, 1)
MsgBox(0, FileGetSize($fpath),  BinaryLen($fcontent))

The size on my system is not the same. So the file shrinks when I use BinaryToString($fcontent, 1).

FileGetEncoding($fpath) returns 1024 on both systems. So the file on Windows 10 is Unicode UTF16 Little Endian (without BOM) encoded.

I can see the content of the files on both systems with Notepad++. Also using your script makes it even more visible. The files have the same size and hash on both systems but the content differs a little.

 

 

 

[jchd]

54 minutes ago, Factfinder said:

FileGetEncoding($fpath) returns 1024 on both systems. So the file on Windows 10 is Unicode UTF16 Little Endian (without BOM) encoded.

No.

File encoding is only meaningful on text files, but you target an executable.

55 minutes ago, Factfinder said:

I can see the content of the files on both systems with Notepad++.

A text editor isn't the right tool for visualizing binary data.

56 minutes ago, Factfinder said:

The files have the same size and hash on both systems but the content differs a little.

Files having the same hash are equal, provided a strong hash function.

The only difference is due to using unsuitable tools or read modes for binary data.

[KaFu]

When the files differ a little and still have the same hash, then you use a bad hash algorithm. E.g. full MD5 can not be the same, when the file differs.

Please try FileOpen($fpath, 512) to enforce fileread as ANSI.

#include <Crypt.au3>

$fpath = @SystemDir & "\userinit.exe"

$hopen = FileOpen($fpath, 16)
$fcontent_Org = FileRead($hopen)
$fcontent_BinaryToString = BinaryToString($fcontent_Org)
FileClose($hopen)

MsgBox(0, FileGetSize($fpath), BinaryLen($fcontent_Org) & @CRLF & @CRLF & BinaryLen($fcontent_BinaryToString) & @crlf & @crlf & _Crypt_HashFile($fpath, $CALG_MD5))



$hopen = FileOpen($fpath, 512)
$fcontent_Org = FileRead($hopen)
$fcontent_BinaryToString = BinaryToString($fcontent_Org)
FileClose($hopen)

MsgBox(0, FileGetSize($fpath), BinaryLen($fcontent_Org) & @CRLF & @CRLF & BinaryLen($fcontent_BinaryToString) & @crlf & @crlf & _Crypt_HashFile($fpath, $CALG_MD5))

I guess this might be related to this topic:

Edited March 4 by KaFu

[Factfinder]

4 hours ago, jchd said:

No.

File encoding is only meaningful on text files, but you target an executable.

A text editor isn't the right tool for visualizing binary data.

Files having the same hash are equal, provided a strong hash function.

The only difference is due to using unsuitable tools or read modes for binary data.

Thank you. But I think the point was not clear. The files are the same and that is no issue. The issue is that when we look at the content, we don't directly look at the content of the files but what we see is the content that is provided by _HexEncode($fcontent). In fact, BinaryToString($fcontent, 1) works differently on two systems. On my main PC it is working like BinaryToString($fcontent, 4) while on another system with the same file, it is working as expected.

[SOLVE-SMART]

Hi @Factfinder,

did you already try what @KaFu was suggesting to you?

4 hours ago, KaFu said:

Please try FileOpen($fpath, 512) to enforce fileread as ANSI.

Maybe this will help us narrow down the problem further. Thanks.

Best regards
Sven

[Factfinder]

4 hours ago, KaFu said:

When the files differ a little and still have the same hash, then you use a bad hash algorithm. E.g. full MD5 can not be the same, when the file differs.

Please try FileOpen($fpath, 512) to enforce fileread as ANSI.

#include <Crypt.au3>

$fpath = @SystemDir & "\userinit.exe"

$hopen = FileOpen($fpath, 16)
$fcontent_Org = FileRead($hopen)
$fcontent_BinaryToString = BinaryToString($fcontent_Org)
FileClose($hopen)

MsgBox(0, FileGetSize($fpath), BinaryLen($fcontent_Org) & @CRLF & @CRLF & BinaryLen($fcontent_BinaryToString) & @crlf & @crlf & _Crypt_HashFile($fpath, $CALG_MD5))



$hopen = FileOpen($fpath, 512)
$fcontent_Org = FileRead($hopen)
$fcontent_BinaryToString = BinaryToString($fcontent_Org)
FileClose($hopen)

MsgBox(0, FileGetSize($fpath), BinaryLen($fcontent_Org) & @CRLF & @CRLF & BinaryLen($fcontent_BinaryToString) & @crlf & @crlf & _Crypt_HashFile($fpath, $CALG_MD5))

I guess this might be related to this topic:

I explained in the previous post that the files have no issues and the hashes are correct. The content of both files is opened directly, they are the same. It is the content of $fcontent string that differs on the two systems. It is all about the BinaryToString() function that causes the inconsistency on my main PC. I guess BinaryToString() is based on a MS function that for some reason is not working on my main system.

I have tried FileOpen() with various flags with no success.

Thank you for the link.

[KaFu]

Maybe it's related to this bug in the Binary() function?

https://www.autoitscript.com/trac/autoit/ticket/3731

Don't know what BinaryToString() does under the hood, maybe a dev can take a look and tell which functions are used.

Does this give different results?

MsgBox(0, "", FileGetVersion($fpath, "DefaultLangCodepage"))

Or skip the usage of BinaryToString() altogether and compare the binary size directly.

$fpath = @SystemDir & "\userinit.exe"
$hopen = FileOpen($fpath, 16)
$fcontent = FileRead($hopen)
$filename = StringRegExpReplace($fpath, ".+\\", "")
$token = 'qpqpqpqpqp'
$boundary = "----158082776"

$data1 = '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="scanToken"' & @CRLF & @CRLF & $token & @CRLF
$data1 &= '--' & $boundary & @CRLF & 'Content-Disposition: form-data; name="file"; filename="' & $filename & '"' & @CRLF
$data1 &= 'Content-Type: application/x-msdownload' & @CRLF & @CRLF
$data2 = @CRLF & '--' & $boundary & '--'

$data = StringToBinary($data1) & $fcontent & StringToBinary($data2) ; StringToBinary() required to concatenate the values correctly, otherwise binary and string will be concatenate to string and size will be off again

MsgBox(0, FileGetSize($fpath) + BinaryLen($data1 & $data2), BinaryLen($data) & @crlf & VarGetType($data))

 

Edited March 4 by KaFu