Very slow => large file

[JackyChan]

Hi,

AutoIt can check string in a file with FileOpen and StringInString,

but very slow with large file.

Do we have other ways to check string in a large file?

Thank for read

[Fire]

JackyChan,

I think this way too may be helpfull.

#include <Array.au3>
#include <File.au3>
;;;;;;;;;;;;;;;; Generetaing big file
For $i=0 to 1500
    FileWrite("test.txt",$i & @CRLF)
    If $i=1500 Then
;;;;;;;;;;;;;;;; END OF Generetaing big file
        ikinci()

        EndIf
    Next


Func ikinci()

Dim $avarray

_FileReadToArray("test.txt",$avarray)
for $i=0 To $avarray[0]
    If $avarray[$i]="1200" Then ;1200 here search string
        MsgBox(64,"Founded","String 1200 Founded!" & " " & $i)

    EndIf
Next
EndFunc

[Mat]

  On 1/16/2010 at 12:02 AM, 'Thanubis said:

I would bet that WinAPI and RegEX would be faster.

WinApi: Probably, But RegEx will be a lot slower than the standard string functions.

Sh3llC043r, I think a single function would be faster than a large array...

Look in the scite directory at "findstr.au3" for a way to search a folder for a string in a file, there may be a way to search just the file.

[jchd]

  On 1/15/2010 at 11:47 PM, 'JackyChan said:

AutoIt can check string in a file with FileOpen and StringInString,

but very slow with large file.

Do we have other ways to check string in a large file?

There are various efficient solutions, the choice which best fits depends on your context.

So you need to check if a given string is present in a given large file.

Are you talking about text strings or binary?

Did you give grep and friends a try? There are really many implementations around.

OK, how large is large? I mean can it be always loaded in memory?

If you insist on doing it by yourself, can you show the code that you find slow?

[JackyChan]

It's about 100-1000mb,

I thinked API will faster

[jchd]

  On 1/16/2010 at 1:43 AM, 'JackyChan said:

It's about 100-1000mb,

I thinked API will faster

What about the other answers?

Grep is still a good choice.

But which API do you have in mind and how do you intend to invoke it?

[AndyG]

Hi,

if you only want to know if the pattern is a part of the string, the following is very fast! It is twice as fast as stringregexp()

stringreplace($string,$pattern,"",1,2)
if @extended then msgbox(0,0,"match")

If you need the position of the pattern in a (very long) string, i would prefer to take the FindBytes()-Function from the prospeed.dll. 

[jchd]

  On 1/16/2010 at 4:29 PM, 'AndyG said:

if you only want to know if the pattern is a part of the string, the following is very fast! It is twice as fast as stringregexp()

That's a bad advice: why replace since the only information needed is presence?

Replace will need twice the memory and obviously will be slower _much_ than StringInStr.