Working with large files

[LostUser]

Hi all. I was wondering what possible ways there are to work with large files in AutoIt without bogging down the memory on a PC.

For example, how does regedit look at the huge amount of data in the registry without bogging down. If I export the registry to a file, the file is huge. I am guessing that the reason that the export is so huge is that the .dat registry is compressed and uses codes to indicate locations rather than the full text of the export. It also may have a table (like a directory) that it loads and looks up the table while only displaying that data that can be shown on screen.

However, what about regular text/data/exe/etc... files?

Can a large file be opened by small sections at a time in a way that you can scroll back and forth through its contents by constantly accessing just the portion of the file you are viewing?

Thanks

[LostUser]

Hi all. I was wondering what possible ways there are to work with large files in AutoIt without bogging down the memory on a PC.

For example, how does regedit look at the huge amount of data in the registry without bogging down. If I export the registry to a file, the file is huge. I am guessing that the reason that the export is so huge is that the .dat registry is compressed and uses codes to indicate locations rather than the full text of the export. It also may have a table (like a directory) that it loads and looks up the table while only displaying that data that can be shown on screen.

However, what about regular text/data/exe/etc... files?

Can a large file be opened by small sections at a time in a way that you can scroll back and forth through its contents by constantly accessing just the portion of the file you are viewing?

Thanks

Just posting to put this on top again to see if I could get a reply.

[JSThePatriot]

You can always just read so many characters into a text file, and the next pass read the next chunk. You can either do that or create some sort of indexing system for the large file like the Registry does do.

Edit: I dont know that reading just part of the file will actually allow you to save memory, but it should help with not having to use all that memory while you are in process of working on the string you returned.

IMO,

JS

Edited June 12, 2006 by JSThePatriot

[LostUser]

You can always just read so many characters into a text file, and the next pass read the next chunk. You can either do that or create some sort of indexing system for the large file like the Registry does do.

Edit: I dont know that reading just part of the file will actually allow you to save memory, but it should help with not having to use all that memory while you are in process of working on the string you returned.

IMO,

JS

Thanks JS.

You can always just read so many characters into a text file, and the next pass read the next chunk.

I haven't messed with the file open utilities much in AutoIt yet.

Are you saying that I could read parts of a file into a text file and view them that way?

I guess I'll just have to try some things out in AutoIT and ask more questions later.

[MHz]

If you use FileOpen to get the handle of a text file and use FileReadLine to read each line of text into a variable, then you are saving memory as the variable in memory is only storing 1 line at a time while you loop through each line.

Another method is to read the whole file into memory with FileRead, then use StringSplit to add each line into an array. At that point, your memory used would be twice the size of the file read. Then clear the variable used in the intial FileRead to release back some memory and process the array as needed. This method is perhaps about twice as fast in reading the file but has the penalty of high memory usage.

If you do not care for speed and want to save memory, then FileReadLine may do, otherwise the read into an array is efficent with speed of reading. Depends on the script at hand for choice of what method suits best.

Edit:

Have a look at _FileReadToArray Standard UDF for ease.

Edited June 16, 2006 by MHz