Jump to content
Sign in to follow this  
Waeolwen

Search through very large .txt files?

Recommended Posts

Waeolwen

I've been working on some scripts to read through .txt files and pull specific information out. They work, however some of the text files are very large and the search will take a long time and use up a lot of the CPU power.

The way that works best is searching line-by-line, as my test with arrays can't hold all of the data and causes an error when I try to load a very large text file.

Would anyone happen to know a faster, less CPU intensive way to search through very large text files? :)

Share this post


Link to post
Share on other sites
weaponx

AutoIt can handle arrays with up to 16 million elements, did you try to exceed this?

Share this post


Link to post
Share on other sites
Swift

How exactly do you want to search? Open the file and read it to a array? Or check line by line, and display a variable that way?

Share this post


Link to post
Share on other sites
Waeolwen

AutoIt can handle arrays with up to 16 million elements, did you try to exceed this?

I'm not quite sure, I attempted to run it on a text file of >300mb and it gave me an error saying "Windows failed to allocate memory"

But, even on a text file of around 8mb, the array process takes a very long time. The line-by-line version took around 15 mins on my 300mb one, but did finish.

How exactly do you want to search? Open the file and read it to a array? Or check line by line, and display a variable that way?

What I'm doing is running through a text file searching for entries that meet certain criteria and writing those lines to another text file.

Share this post


Link to post
Share on other sites
Siao

Your best bet is Windows commandline utils find.exe and findstr.exe. Findstr even supports some kind of regular expressions. If you need more features, there are other tools of that kind on the internets.

You could write something like that yourself with bunch of DllCalls and stuff, but why bother? Autoit is best used to automate things, not write something from scratch.

Edited by Siao

"be smart, drink your wine"

Share this post


Link to post
Share on other sites
Waeolwen

Your best bet is Windows commandline utils find.exe and findstr.exe. Findstr even supports some kind of regular expressions. If you need more features, there are other tools of that kind on the internets.

You could write something like that yourself with bunch of DllCalls and stuff, but why bother? Autoit is best used to automate things, not write something from scratch.

I've figured out how to run the findstr command through .bat's and such, but can't seem to get it to work in AutoIT. I know it's something like:

RunWait(@ComSpec & " /c findstr ........................................... )

But I can't see to get it to work beyond just flashing on the screen for a moment (not sure if it's running findstr or not, but it is running the ComSpec) I figure it has something to do with my quotations and such, but not sure how to do it since I'll need quote around my search string and around both file locations...

Nevermind! *smacks forehead* Had my double and single quotes in the wrong order in my variables ><

Edited by Waeolwen

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.