Sign in to follow this  
Followers 0
JackyChan

Very slow => large file

8 posts in this topic

Hi,

AutoIt can check string in a file with FileOpen and StringInString,

but very slow with large file.

Do we have other ways to check string in a large file?

Thank for read

Share this post


Link to post
Share on other sites



JackyChan,

I think this way too may be helpfull.

#include <Array.au3>
#include <File.au3>
;;;;;;;;;;;;;;;; Generetaing big file
For $i=0 to 1500
    FileWrite("test.txt",$i & @CRLF)
    If $i=1500 Then
;;;;;;;;;;;;;;;; END OF Generetaing big file
        ikinci()

        EndIf
    Next


Func ikinci()

Dim $avarray

_FileReadToArray("test.txt",$avarray)
for $i=0 To $avarray[0]
    If $avarray[$i]="1200" Then ;1200 here search string
        MsgBox(64,"Founded","String 1200 Founded!" & " " & $i)

    EndIf
Next
EndFunc

[size="5"] [/size]

Share this post


Link to post
Share on other sites

I would bet that WinAPI and RegEX would be faster.

WinApi: Probably, But RegEx will be a lot slower than the standard string functions.

Sh3llC043r, I think a single function would be faster than a large array...

Look in the scite directory at "findstr.au3" for a way to search a folder for a string in a file, there may be a way to search just the file.

Share this post


Link to post
Share on other sites

AutoIt can check string in a file with FileOpen and StringInString,

but very slow with large file.

Do we have other ways to check string in a large file?

There are various efficient solutions, the choice which best fits depends on your context.

So you need to check if a given string is present in a given large file.

Are you talking about text strings or binary?

Did you give grep and friends a try? There are really many implementations around.

OK, how large is large? I mean can it be always loaded in memory?

If you insist on doing it by yourself, can you show the code that you find slow?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

It's about 100-1000mb,

I thinked API will faster

Share this post


Link to post
Share on other sites

It's about 100-1000mb,

I thinked API will faster

What about the other answers?

Grep is still a good choice.

But which API do you have in mind and how do you intend to invoke it?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Hi,

if you only want to know if the pattern is a part of the string, the following is very fast! It is twice as fast as stringregexp()

stringreplace($string,$pattern,"",1,2)
if @extended then msgbox(0,0,"match")

If you need the position of the pattern in a (very long) string, i would prefer to take the FindBytes()-Function from the prospeed.dll. 

Share this post


Link to post
Share on other sites

if you only want to know if the pattern is a part of the string, the following is very fast! It is twice as fast as stringregexp()

That's a bad advice: why replace since the only information needed is presence?

Replace will need twice the memory and obviously will be slower _much_ than StringInStr.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0