Jump to content

Bug fix #3789 FileRead() causes Memory Allocation Error when cannot access file


Recommended Posts

I'm seeing a periodic "Error Allocating Memory" error with FileRead().  I believe this error message is incorrect and has to do with file access, not memory.

I have a project where I run two applications in tandem, a C# application that is constantly re-writing a text file, and an autoit script that periodically reads that file.  After this bugfix was introduced, I started seeing random 'Error Allocating Memory' errors, presumably when autoit was unable to read the file due to it being locked by the other application.

I created a loop calling FileRead on this file 1,000,000 times as a test.  

On 3.3.14.5 it reads the file every time without throwing an error.  Once in awhile FileRead does return an empty string.  This must be the original behavior when a file cannot be accessed.  It seems that after the bug #3789 fix the Error Allocating Memory error is being thrown instead of just returning a blank screen or setting @error.

The empty string I can handle and re-read the file, but the Error Allocating Memory error is fatal and I don't believe that can be handled.

I believe the change causing this is from the line "Fixed #3789: FileRead() on big ANSI file (1Gb)." in the release notes because I've tested each version below, pointing to that change:

3.3.14.5              works as expected
3.3.15.3-beta     works as expected
3.3.15.4-beta     issue is present
3.3.16.0              issue is present
3.3.16.1-beta    issue is present

 

What can I do to help facilitate a change to the way FileRead behaves when a file is inaccessible, or how can I work around this?

 

Link to post
Share on other sites
42 minutes ago, cycotron said:

or how can I work around this?

_WinAPI_FileInUse could be a way to work around your problem.

Musashi-C64.png

"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move."

Link to post
Share on other sites
24 minutes ago, Musashi said:

_WinAPI_FileInUse could be a way to work around your problem.

I can try this in a loop waiting for the file to be accessible, however I'm a little worried that in the millisecond between checking if the File is in use and then reading the file I could still hit the same situation.  It'd be great If either FileRead or some other function could return an error instead of crashing so that I could handle that situation.

Link to post
Share on other sites

When a file is modified by one process while being read by another, weird things can happen. That's why file locks should be used to obviate the issue.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites
37 minutes ago, jchd said:

When a file is modified by one process while being read by another, weird things can happen. That's why file locks should be used to obviate the issue.

Forgive my ignorance and thanks for your reply!

The C# application is using File.WriteAllText() which I believe does lock the file.  So unless I'm extra confused, the problem is that AutoIt is ignoring the lock and trying to read while locked and crashing (while in the past it just returned an empty string instead)? 

Would I need to lock the file with AutoIt before reading it and then release the lock?

Link to post
Share on other sites

I highly doubt the C# app is actually using a lock. At least, MSDN doesn't say so.  Hence you should create a lock yourself, perform your I/O operation then release the lock.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites
6 hours ago, cycotron said:

I created a loop calling FileRead on this file 1,000,000 times as a test.  

what is the size of the file you are using? 

which type of type ANSI UTFxx BOM?

I need those information to reproduce as the fix was so little and I don't see the relation nothing with file locking.

thanks for the help.

Link to post
Share on other sites

I would actually expect that test to work as in my case the two applications are seperate processes running simultaneously.  I suspect it has something to do with app #1 writing the file and then app #2 (autoit) reading the file during the write.  This is why I had to test with a huge number of iterations, most of the time those two events don't overlap.

I am running with #AutoIt3Wrapper_UseX64=n on Windows 7 x64.  This script has a large number of references to external dlls and many UDFs so not sure if switching to 64 will work or not.  I can give that a whirl, especially at least for this specific test.

I'll get a C# project and autoit test script posted for you early next week.  On "vacation" right now :)

 

Link to post
Share on other sites

I created a c# project to write the file repeatedly and an autoit script to read the file repeatedly.  When running both, I can reliably reproduce the error.

 

I created these on Win10 x64 and also with #AutoIt3Wrapper_UseX64=Y and it persists, so OS and x86/x64 don't seem to matter.

 

Attached a zip with both applications.  If you don't want to open the C# project in Visual Studio, you can just run the console application from FileTest\FileWriteTest\FileWriteTest\bin\Debug\FileWriteTest.exe

Link to post
Share on other sites
Posted (edited)

I did some more testing on this.  I've updated the C# application that writes the file so you can choose between two modes. 

1- Write the file without locking using File.WriteAllText
2 -Write the file with locking using FileStream.Write

In the first mode, Autoit FileRead will get the Out of Memory exception.

In the second mode, Autoit FileRead will sometimes return a blank string if the file is locked (which is the behavior I had seen in the past)

 

I've attached the improved testing code.

FileTest.zip

Edited by cycotron
Accidentally submitted post before finished
Link to post
Share on other sites

If you have access to the C# program source as well (whether you wrote it or not is not immediately clear to me), you should try to do this via inter-process communication instead of writing files. It's kind of a waste to constantly write a large file over and over especially on an SSD.

In the past, I had success using SendMessage API with $WM_COPYDATA, but it wasn't in autoit. Based on my past research, this is the easiest way to send data between programs on windows (none of the methods are easy though). It actually just needs the window title and it will pipe the data into the window. There's no worrying about the data being locked and you can send at whatever rate you want. You'll have to look for examples cause it's not trivial.

If I were doing what you're doing, this is exactly how I would do it. It's more proper.

Link to post
Share on other sites
  • 2 months later...

@steipal, the answer I guess would be yes, it got fixed. When he claims something, is because is a done deal, historically speaking.
Now, the only way to know for sure, is to test it out under more environments and peer review any findings, that way we all participate in bettering the product ( AutoIt3 ).
So try it out and share your findings. They will be welcomed by us all. :)

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...