Faster way to pull lines matching string?

[xenoranger]

So, I am still a total newb when it comes to programming. I mostly write scripts.

I wrote the following applet to filter the GPS data for a single unit and write each line to a new file. The problem is that it cycles through each line. When my files have over 1,000,000 lines, this can take days. Is there a faster way to just pull the lines that match certain criteria and write those to the new file?

expandcollapse popup

$xnFile = InputBox("File???", "Which file do you want to review?","C:")
$xnUnit = Inputbox("Unit ID","What is the Unit ID you want?","Some Unit String")

$xnTargetFile = $xnFile & " - Unit - " & $xnUnit & " - Parsed.csv"
FileOpen($xnFile,0)
FileWrite($xnTargetFile,"")
FileOpen($xnTargetFile, 1)

$i = 1
ProgressOn("Creating CSV","Extracting matching data.","",0,0,16)
While $i > 0

$xnCurrentLine = FileReadLine($xnFile,$i)
;MsgBox(1,"",$xnCurrentLine)

if stringlen($xnCurrentLine) > 1 Then

     if StringInStr($xnCurrentLine,$xnUnit & " $GPRMC") > 40 Then
    
     $xnDate = stringmid($xnCurrentLine,stringinstr($xnCurrentLine,"Verbose: 0 : ") + 13,10)
    
     $xnTime = stringmid($xnCurrentLine,StringInStr($xnCurrentLine,"4G?`") - 12,8)
     $xnGeoData = stringmid($xnCurrentLine,StringInStr($xnCurrentLine,"$GPRMC"))
        
     $xnTargetLine = $xnDate & "," & $xnTime & "," & $xnGeoData & @CRLF
    
     FileWriteLine($xnTargetFile,$xnTargetLine)
    
    
     EndIf

     $i = $i + 1
    
     if round($i/1000000 * 100,1) < 90 Then
    
         ProgressSet(round($i/1000000 * 100,1),$i & " lines examined." & @cr & "Thank you for your patience.")
    
     Else
    
         ProgressSet(90,$i & " lines examined." & @cr & @cr & "Just a little longer... ..." & @CR & "Thank you for your patience.")
    
    
     EndIf
    
    
Else
     $i = 0

EndIf


WEnd
ProgressOff()

PS: in the ProgressSet line, the 1,000,000 comes from other people hating that it didn't have a progress bar that updates and the estimation of at least 1,000,000 lines in each log file.

Edited October 1, 2012 by xenoranger

[Melba23]

xenoranger,

Welcome to the AutoIt forum.

You might be able to read the file into an array and manipulate it - that is faster than reading each line. However, I have never tried to do that with a 1M line file. Perhaps you could give it a try - look at _FileReadToArray in the Help file. Let us know if it throws an error or crashes - but if it succeeds please do NOT try and display the result with _ArrayDisplay as that would take forever!

You might also be able to use a RegEx to extract the data directly if the required data had a suitable pattern to search for. Could you let us have an example (preferably less than 1M lines ) of a log file and indicate which lines you wish to extract. We can then see what might be possible.

M23

[guinness]

Also what is the filesize of that file?

[kylomas]

xenoranger,

You are reading a file line by line in a loop using the file name. Per the doc

If a filename is given rather than a file handle - the file will be opened and closed during the function call - for parsing large text files this will be much slower than using filehandles.

You are opening the file for read but not getting a file handle. Change

FileOpen($xnFile,0)

to

local $hfl = FileOpen($xnFile,0)

then use the handle for reading like

$xnCurrentLine = FileReadLine($hfl,$i)

Incidentally, you do not need to specify the line number if you are reading the file sequentially (again, in the help file!).

This change will speed up you script, however, processing the file as an array or string variable will obviously be much faster. The entire file can be read to a variable (either array or string) in one instruction.

Good Luck,

kylomas

[kylomas]

xenoranger,

One other thing that I noticed. You should create an output string within your search loop and write the output file from that string when you exit the loop. Filewriteline also does an open and close for each write. This may, or may not impact you depending on the volume of matches you find..

kylomas

[xenoranger]

Also what is the filesize of that file?

The file is 100,000 MB in size.

I've been moved to other projects, so I'll have to come back to this one. I appreciate your help!

[jchd]

Searching a 100GB flat text file doesn't sound anything linke reasonable to me, particularly using an interpreted language.

You GPS data almost certainly has some structure which you should use to build a database. Adequate indexing can then bring back queries results in seconds, not days.

You can use SQLite for that.

[xenoranger]

Sorry, jchd, I meant 100,000kb (100MB). Still, I do need to sort through super huge files.

I did find out that someone else wrote an application in C++ that would output the lines matching criteria. As a result, I haven't advanced this project much further.

Thank you guys for your help!!

Edited October 29, 2012 by xenoranger