Bug fix #3789 FileRead() causes Memory Allocation Error when cannot access file

[cycotron]

I'm seeing a periodic "Error Allocating Memory" error with FileRead().  I believe this error message is incorrect and has to do with file access, not memory.

I have a project where I run two applications in tandem, a C# application that is constantly re-writing a text file, and an autoit script that periodically reads that file.  After this bugfix was introduced, I started seeing random 'Error Allocating Memory' errors, presumably when autoit was unable to read the file due to it being locked by the other application.

I created a loop calling FileRead on this file 1,000,000 times as a test.  

On 3.3.14.5 it reads the file every time without throwing an error.  Once in awhile FileRead does return an empty string.  This must be the original behavior when a file cannot be accessed.  It seems that after the bug #3789 fix the Error Allocating Memory error is being thrown instead of just returning a blank screen or setting @error.

The empty string I can handle and re-read the file, but the Error Allocating Memory error is fatal and I don't believe that can be handled.

I believe the change causing this is from the line "Fixed #3789: FileRead() on big ANSI file (1Gb)." in the release notes because I've tested each version below, pointing to that change:

3.3.14.5              works as expected
3.3.15.3-beta     works as expected
3.3.15.4-beta     issue is present
3.3.16.0              issue is present
3.3.16.1-beta    issue is present

 

What can I do to help facilitate a change to the way FileRead behaves when a file is inaccessible, or how can I work around this?

 

[Musashi]

42 minutes ago, cycotron said:

or how can I work around this?

_WinAPI_FileInUse could be a way to work around your problem.

[cycotron]

24 minutes ago, Musashi said:

_WinAPI_FileInUse could be a way to work around your problem.

I can try this in a loop waiting for the file to be accessible, however I'm a little worried that in the millisecond between checking if the File is in use and then reading the file I could still hit the same situation.  It'd be great If either FileRead or some other function could return an error instead of crashing so that I could handle that situation.

[jchd]

When a file is modified by one process while being read by another, weird things can happen. That's why file locks should be used to obviate the issue.

[cycotron]

37 minutes ago, jchd said:

When a file is modified by one process while being read by another, weird things can happen. That's why file locks should be used to obviate the issue.

Forgive my ignorance and thanks for your reply!

The C# application is using File.WriteAllText() which I believe does lock the file.  So unless I'm extra confused, the problem is that AutoIt is ignoring the lock and trying to read while locked and crashing (while in the past it just returned an empty string instead)? 

Would I need to lock the file with AutoIt before reading it and then release the lock?

[jchd]

I highly doubt the C# app is actually using a lock. At least, MSDN doesn't say so.  Hence you should create a lock yourself, perform your I/O operation then release the lock.

[jpm]

6 hours ago, cycotron said:

I created a loop calling FileRead on this file 1,000,000 times as a test.  

what is the size of the file you are using? 

which type of type ANSI UTFxx BOM?

I need those information to reproduce as the fix was so little and I don't see the relation nothing with file locking.

thanks for the help.

[cycotron]

Tanks for the reply jpm!

File size is super small, less than one KB.  ~490 bytes.  File is UTF-8-BOM.

Let me know if you need anything else!

[jpm]

Thanks can you post the file to be for sure?

are you running in X64 mode?

Cheers

PS my first check  I can't reproduce

#3789 FileRead 1000000 time.au3

[cycotron]

I would actually expect that test to work as in my case the two applications are seperate processes running simultaneously.  I suspect it has something to do with app #1 writing the file and then app #2 (autoit) reading the file during the write.  This is why I had to test with a huge number of iterations, most of the time those two events don't overlap.

I am running with #AutoIt3Wrapper_UseX64=n on Windows 7 x64.  This script has a large number of references to external dlls and many UDFs so not sure if switching to 64 will work or not.  I can give that a whirl, especially at least for this specific test.

I'll get a C# project and autoit test script posted for you early next week.  On "vacation" right now

 

[jpm]

Can you reproduce with a Windows 10?

[cycotron]

I created a c# project to write the file repeatedly and an autoit script to read the file repeatedly.  When running both, I can reliably reproduce the error.

 

I created these on Win10 x64 and also with #AutoIt3Wrapper_UseX64=Y and it persists, so OS and x86/x64 don't seem to matter.

 

Attached a zip with both applications.  If you don't want to open the C# project in Visual Studio, you can just run the console application from FileTest\FileWriteTest\FileWriteTest\bin\Debug\FileWriteTest.exe

[jpm]

you miss to include the zip🥵

[cycotron]

Wow, I'm amazing.  Let's try that again!

FileTest.zip

[cycotron]

I did some more testing on this.  I've updated the C# application that writes the file so you can choose between two modes. 

1- Write the file without locking using File.WriteAllText
2 -Write the file with locking using FileStream.Write

In the first mode, Autoit FileRead will get the Out of Memory exception.

In the second mode, Autoit FileRead will sometimes return a blank string if the file is locked (which is the behavior I had seen in the past)

 

I've attached the improved testing code.

FileTest.zip

Edited July 6, 2022 by cycotron
Accidentally submitted post before finished

[jpm]

Hi,

Thank you for your help,

I find the reason, during the retrieving of the characteristic of the file the BOM was detected and the size was 0 wwhich lead to a wrong calculation.

I fix it for the next Beta/release

Cheers

[cycotron]

That's great news and I'm glad to help!  I'll keep an eye out for the next beta/release.

[Synaps3]

If you have access to the C# program source as well (whether you wrote it or not is not immediately clear to me), you should try to do this via inter-process communication instead of writing files. It's kind of a waste to constantly write a large file over and over especially on an SSD.

In the past, I had success using SendMessage API with $WM_COPYDATA, but it wasn't in autoit. Based on my past research, this is the easiest way to send data between programs on windows (none of the methods are easy though). It actually just needs the window title and it will pipe the data into the window. There's no worrying about the data being locked and you can send at whatever rate you want. You'll have to look for examples cause it's not trivial.

If I were doing what you're doing, this is exactly how I would do it. It's more proper.

[steipal]

@jpm Has this issue been fixed in the new 3.3.16.1 release?

[argumentum]

@steipal, the answer I guess would be yes, it got fixed. When he claims something, is because is a done deal, historically speaking.
Now, the only way to know for sure, is to test it out under more environments and peer review any findings, that way we all participate in bettering the product ( AutoIt3 ).
So try it out and share your findings. They will be welcomed by us all.