TheDcoder

AutoIt Memory Allocation bug?

20 posts in this topic

#1 ·  Posted (edited)

Hello, I think I have stumbled upon a bug in AutoIt... Helpfile says that it can store 2,147,483,647 maximum bytes of binary data but I get a memory allocation error... Here is my script:

#include <FileConstants.au3>

Local $hFile = FileOpen(@ScriptDir & '\2 GB.bin', $FO_BINARY)

$dData = FileRead($hFile)

df084c9f7dd45e335e4340dfd3ac6534.png

You can use Dummy File Creator to create a dummy file :)

Edited by TheDcoder

AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites



IIRC there has been a discussion about the size limits you see in real life some time ago.
A user tried to create a string with maximum size of 2GB. But as the computers memory is always fragmented the OS couldn't get a contiguous piece of memory and crashed.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@JohnOne 6 GB DDR3 ;)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

I guess this is the total memory of your computer?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

195aa7549e4eb44fed0e6359bc90c15a.png

:yes:


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

I think you will hit the same limitation when trying to create and fill a 2GB string variable.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@water :huh:?


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

IIRC Windows used to limit process memory to 2Gb. Increasing that limit to 3Gb is possible using one more esoteric registry setting.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

@jchd Oh :o! Thanks for the clarification :)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

Here you can find what I'm talking about.
Here Jon describes how it works with strings.

1 person likes this

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@water Ok :)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

Is this a real problem you try to solve or are you just curious how AutoIt works?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@water Curious :)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

I discovered why this is not possible. Because AutoIt3.exe (the interpreter) already occupies 15 MB of the memory so 15 MB + 2 GB = >2GB :D.

 

P.S Sorry for bumping old thread, I wanted to post the answer.


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

It seems AutoIt x64 also fail with Memory Allocation error in process of FileRead() over 2Gb file. System have 16Gb RAM. Can someone confirm this?

Share this post


Link to post
Share on other sites

Do you FileRead binary of text? Don't forget that AutoIt stores text as UTF16 (UCS-2 exactly), which implies 16-bits per character. So reading a 2Gb text file means allocating a 4Gb memory storage, whatever text encoding the file uses.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

No, it's binary.  But FileRead() still uses 2x amount peak working set size, see:

#AutoIt3Wrapper_UseX64=Y

#include <FileConstants.au3>
#include <WinAPIProc.au3>



$h = FileOpen("!__TestSize.file", $FO_OVERWRITE + $FO_BINARY)
FileSetPos($h, 1.999999996*(1024*1024*1024), $FILE_BEGIN )
FileWrite($h,0)
FileClose($h)

MsgBox(64,"Create file",GetInfo())

$h = FileOpen("!__TestSize.file", $FO_BINARY)
FileRead($h)
FileClose($h)

MsgBox(64,"Read file",GetInfo())

;FileDelete("!__TestSize.file")

Func GetInfo($iPID = 0)
    $aInfo = _WinAPI_GetProcessMemoryInfo($iPID)
    Return 'Peak working set size: ' & Round($aInfo[1]/1024/1024,1) & ' Mb' & @CRLF & 'Current working set size: ' & Round($aInfo[2]/1024/1024,1) & ' Mb'
EndFunc

And any way why only 2/4Gb in x64 freshly booted system with 16Gb RAM? It too precise - it has to be some kind of limit, I mean I newer able to archive even 2,01 Gb on AutoIt FileRead while I can freely work with SQLite database well over 2Gb and set cache 2x amount of database size.

Edited by Iczer

Share this post


Link to post
Share on other sites

Think about this:
 

#include <FileConstants.au3>
_Example()
Func _Example()
    Local $iFileLen_asSstring = StringLen(FileRead(@ScriptFullPath))

    Local $hFile = FileOpen(@ScriptFullPath, $FO_BINARY)
    Local $iFileLen_asBinaryVariantAkaBLOB = StringLen(FileRead($hFile))
    FileClose($hFile)
    Local $sSummary = _
            $iFileLen_asSstring & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB / $iFileLen_asSstring & @CRLF & _
            $iFileLen_asBinaryVariantAkaBLOB - (2 * $iFileLen_asSstring) & @CRLF
    ConsoleWrite($sSummary & @CRLF)
    MsgBox(0, 'Test', $sSummary)
EndFunc   ;==>_Example

Result:

Quote

612
1232
2.01307189542484
8

 


Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

@mLipok,

You're converting the binary content read to string by using StringLen. That isn't fair :)

BinaryLen is more suited to the comparison you want to make.

Don't forget that nowadays, SciTE is saving .au3 files as UTF8 or some other encoding you have set, so you likely count the UTFx BOM if the default file encoding is not UTF8 w/o BOM.

This isn't the case here, as the byte size difference you see is 8. It corresponds to the two characters '0x' prepended to the string after implicit conversion forced by using StringLen. Those two characters are 16-bit each, so they account for 4 bytes. The 4 bytes left must be the BOM you use and only UTF16 ou UTF32 have a BOM using 2 characters (once converted to string, don't forget). I'm guessing your source file is UTF16-LE encoded.

 


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now