AutoIt Memory Allocation bug?

[TheDcoder]

Hello, I think I have stumbled upon a bug in AutoIt... Helpfile says that it can store 2,147,483,647 maximum bytes of binary data but I get a memory allocation error... Here is my script:

#include <FileConstants.au3>

Local $hFile = FileOpen(@ScriptDir & '\2 GB.bin', $FO_BINARY)

$dData = FileRead($hFile)

You can use Dummy File Creator to create a dummy file

Edited December 6, 2015 by TheDcoder

[JohnOne]

And you have over 2GB ram available?

[water]

IIRC there has been a discussion about the size limits you see in real life some time ago.
A user tried to create a string with maximum size of 2GB. But as the computers memory is always fragmented the OS couldn't get a contiguous piece of memory and crashed.

[TheDcoder]

@JohnOne 6 GB DDR3

[water]

I guess this is the total memory of your computer?

[TheDcoder]

[water]

I think you will hit the same limitation when trying to create and fill a 2GB string variable.

[TheDcoder]

@water ?

[jchd]

IIRC Windows used to limit process memory to 2Gb. Increasing that limit to 3Gb is possible using one more esoteric registry setting.

[TheDcoder]

@jchd Oh ! Thanks for the clarification

[water]

  On 12/6/2015 at 4:04 PM, TheDcoder said:

@water ?

Here you can find what I'm talking about.
Here Jon describes how it works with strings.

[TheDcoder]

@water Ok

[water]

Is this a real problem you try to solve or are you just curious how AutoIt works?

[TheDcoder]

@water Curious

[TheDcoder]

I discovered why this is not possible. Because AutoIt3.exe (the interpreter) already occupies 15 MB of the memory so 15 MB + 2 GB = >2GB .

 

P.S Sorry for bumping old thread, I wanted to post the answer.

[Iczer]

It seems AutoIt x64 also fail with Memory Allocation error in process of FileRead() over 2Gb file. System have 16Gb RAM. Can someone confirm this?

[jchd]

Do you FileRead binary of text? Don't forget that AutoIt stores text as UTF16 (UCS-2 exactly), which implies 16-bits per character. So reading a 2Gb text file means allocating a 4Gb memory storage, whatever text encoding the file uses.

[Iczer]

No, it's binary.  But FileRead() still uses 2x amount peak working set size, see:

#AutoIt3Wrapper_UseX64=Y

#include <FileConstants.au3>
#include <WinAPIProc.au3>



$h = FileOpen("!__TestSize.file", $FO_OVERWRITE + $FO_BINARY)
FileSetPos($h, 1.999999996*(1024*1024*1024), $FILE_BEGIN )
FileWrite($h,0)
FileClose($h)

MsgBox(64,"Create file",GetInfo())

$h = FileOpen("!__TestSize.file", $FO_BINARY)
FileRead($h)
FileClose($h)

MsgBox(64,"Read file",GetInfo())

;FileDelete("!__TestSize.file")

Func GetInfo($iPID = 0)
    $aInfo = _WinAPI_GetProcessMemoryInfo($iPID)
    Return 'Peak working set size: ' & Round($aInfo[1]/1024/1024,1) & ' Mb' & @CRLF & 'Current working set size: ' & Round($aInfo[2]/1024/1024,1) & ' Mb'
EndFunc

And any way why only 2/4Gb in x64 freshly booted system with 16Gb RAM? It too precise - it has to be some kind of limit, I mean I newer able to archive even 2,01 Gb on AutoIt FileRead while I can freely work with SQLite database well over 2Gb and set cache 2x amount of database size.

Edited December 19, 2016 by Iczer

[mLipok]

Think about this:
 

#include <FileConstants.au3>
_Example()
Func _Example()
    Local $iFileLen_asSstring = StringLen(FileRead(@ScriptFullPath))

    Local $hFile = FileOpen(@ScriptFullPath, $FO_BINARY)
    Local $iFileLen_asBinaryVariantAkaBLOB = StringLen(FileRead($hFile))
    FileClose($hFile)
    Local $sSummary = _
            $iFileLen_asSstring & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB / $iFileLen_asSstring & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB - (2 * $iFileLen_asSstring) & @CRLF
    ConsoleWrite($sSummary & @CRLF)
    MsgBox(0, 'Test', $sSummary)
EndFunc   ;==>_Example

Result:

  Quote

612
1232
2.01307189542484
8

Expand  

 

[jchd]

@mLipok,

You're converting the binary content read to string by using StringLen. That isn't fair

BinaryLen is more suited to the comparison you want to make.

Don't forget that nowadays, SciTE is saving .au3 files as UTF8 or some other encoding you have set, so you likely count the UTFx BOM if the default file encoding is not UTF8 w/o BOM.

This isn't the case here, as the byte size difference you see is 8. It corresponds to the two characters '0x' prepended to the string after implicit conversion forced by using StringLen. Those two characters are 16-bit each, so they account for 4 bytes. The 4 bytes left must be the BOM you use and only UTF16 ou UTF32 have a BOM using 2 characters (once converted to string, don't forget). I'm guessing your source file is UTF16-LE encoded.