Search through very large .txt files?

[Waeolwen]

I've been working on some scripts to read through .txt files and pull specific information out. They work, however some of the text files are very large and the search will take a long time and use up a lot of the CPU power.

The way that works best is searching line-by-line, as my test with arrays can't hold all of the data and causes an error when I try to load a very large text file.

Would anyone happen to know a faster, less CPU intensive way to search through very large text files?

[weaponx]

AutoIt can handle arrays with up to 16 million elements, did you try to exceed this?

[Swift]

How exactly do you want to search? Open the file and read it to a array? Or check line by line, and display a variable that way?

[Waeolwen]

AutoIt can handle arrays with up to 16 million elements, did you try to exceed this?

I'm not quite sure, I attempted to run it on a text file of >300mb and it gave me an error saying "Windows failed to allocate memory"

But, even on a text file of around 8mb, the array process takes a very long time. The line-by-line version took around 15 mins on my 300mb one, but did finish.

How exactly do you want to search? Open the file and read it to a array? Or check line by line, and display a variable that way?

What I'm doing is running through a text file searching for entries that meet certain criteria and writing those lines to another text file.

[Siao]

Your best bet is Windows commandline utils find.exe and findstr.exe. Findstr even supports some kind of regular expressions. If you need more features, there are other tools of that kind on the internets.

You could write something like that yourself with bunch of DllCalls and stuff, but why bother? Autoit is best used to automate things, not write something from scratch.

Edited April 4, 2008 by Siao

[weaponx]

AutoIt chokes on variables sometime after 100 mb unless you are running a 64 bit OS.

#500355

See here for some proof of concept testing:

http://www.autoitscript.com/forum/index.php?showtopic=62354

Edited April 4, 2008 by weaponx

[Waeolwen]

Your best bet is Windows commandline utils find.exe and findstr.exe. Findstr even supports some kind of regular expressions. If you need more features, there are other tools of that kind on the internets.

You could write something like that yourself with bunch of DllCalls and stuff, but why bother? Autoit is best used to automate things, not write something from scratch.

I've figured out how to run the findstr command through .bat's and such, but can't seem to get it to work in AutoIT. I know it's something like:

RunWait(@ComSpec & " /c findstr ........................................... )

But I can't see to get it to work beyond just flashing on the screen for a moment (not sure if it's running findstr or not, but it is running the ComSpec) I figure it has something to do with my quotations and such, but not sure how to do it since I'll need quote around my search string and around both file locations...

Nevermind! *smacks forehead* Had my double and single quotes in the wrong order in my variables ><

Edited April 4, 2008 by Waeolwen