Text File to Memory size correlation?

[BigDaddyO]

Is there a correlation between a text file size and the memory usage when loading that file with FileReadToArray?

Since that command seems to fully lock my script, not even Adlib works, I created a second helper script that just displays a popup showing % load by reading the Memory usage of the main script and calculating against the target file size but I’m having problems with figuring out the correct math to calculate a good %.

on one file where the lines are not very long but > 1million rows:  (Mem Size / (File size * 2)) * 100 was pretty close.

on another file that has really long lines, but only around 400k rows I actually needed to do ((Mem Size * 2) / File Size) * 100 to get close to the right %’s

 

Is there some other way that I could use to get a better % no matter the file?

Thanks,

Mike

[jchd]

The ratio will heavily depend on which encoding the text file uses and its actual content. AutoIt strings use 16-bit per character; add something for array overhead.

[orbs]

@BigDaddyO,

i know that is not what you asked, but...

5 hours ago, BigDaddyO said:

... that command seems to fully lock my script ...

if you need to handle large files, perhaps you should be reading and parsing them line-by-line, rather than reading the entire text and parsing it. no doubt that will decrease your script memory footprint.

[jchd]

If you really have to read a text file into an array while preserving AdLib functionality, prefer _FileReadToArray over FileReadToArray. The former is an UDF, hence an interruptible piece of AutoIt code, while the latter is a single built-in (uninterruptible) instruction.

[BrewManNH]

_FRTA mostly uses FRTA unless you're using the Delimiter parameter, or one of the flags used to return the array as an array-of-arrays or returning the count in the [0] element. So, it's only interruptable if you're using one of them.

[jchd]

Well spotted.

[BigDaddyO]

@orbs

I ended up doing it your way.  I'm parsing the file line by line as I'm validating.  Overall it added about 10 minutes to the 3 hour long process but as it gives the users a good progress bar, they seem to prefer it.  Though as the file is on a Network share, hopefully they don't loose connectivity while they are running else it will probably error out.

[orbs]

3 minutes ago, BigDaddyO said:

hopefully they don't loose connectivity while they are running else it will probably error out.

you ought to introduce proper error checking then. after each FileReadLine() check the @error status and the return value. you may want to record the file size before you start reading, and increment a counter as you read it line-by-line, so you can verify the entire file was parsed.

[BigDaddyO]

5 hours ago, orbs said:

you ought to introduce proper error checking then. after each FileReadLine() check the @error status and the return value. you may want to record the file size before you start reading, and increment a counter as you read it line-by-line, so you can verify the entire file was parsed.

I did use _FileCountLines() so I know the total row count, but I missed adding the error check on the FileReadLine()  I'll add that.  Thanks!

[orbs]

@BigDaddyO,

On 6/13/2016 at 9:32 PM, BigDaddyO said:

I did use _FileCountLines() ...

i was wondering how _FileCountLines() could determine the count of lines without reading the entire file. so i looked at the UDF. and guess what?

it does.

_FileCountLines() reads the entire file, and then counts the line breaks.

so you are actually still reading the entire file. and at the origin of this topic, you were actually reading the entire file twice - once at _FileCountLines(), and once in your main script! no wonder it took so long.

you should drop the _FileCountLines() and stick to file size indicator, which does not require reading the file at all.

 

[BigDaddyO]

I didn't have _FileCountLines() initially as I was just using the array size to know the count.  The _FileCountLines() may read the file, but it doesn't take long at all, about 15 seconds for the 1 million rows in my test file.

I did find out that _FileCountLines() does not seem to work for text files in Binary format as it only ever comes back with 1 for line count.  I tried updating the UDF to open the file in Binary mode but that didn't work.  So, for now I have split my script into 2 different files.  1 for the majority of the files, and the original one without much of a progress for the Binary file.

[MilesAhead]

Could the network be slowing you down?  Have you tried a simple copy to the local drive, then read from the HD?  If it is still slow you might try memory mapped files.  That is what the OS uses to map exe images into ram.  User programs can create memory mapped files to share the same data across processes.

 

Where it may help you is the call MapViewOfFile which lets you map a chunk of the file into a memory range.  With some experience, or if you can find a library for Memory Mapped Files, you may use that call to create a window into the file.  The position in the file used to fill the buffer is adjustable.  Also the size.

 

Edited June 17, 2016 by MilesAhead
fix typo

[water]

I wonder why it takes 3 hours to process 1 million lines of a text file. How do you process the lines? Do you write the result to Excel, Word ..?
Maybe your script could be enhanced to run much faster