[Resolved] Faster way to count lines in a text file

[SpookMeister]

I'm occasionally dealing with some seriously large (900+ MB) text files. So far I have been using FileOpen and FileReadLine to first get a count of the lines just so I can give some kind of progress on where in the file the "work portion" of my script is at. Just looping through the file and incrementing a counter in this method can take several minutes, and this is before I even start my search/data gather/whatever.

Has someone worked out a faster method to get a line count?

Edited January 21, 2009 by SpookMeister

[Paulie]

_FileCountLines()

[KaFu]

Guess that won't work/make it faster for +900MB files as the UDF contains this row

$sFileContent = StringStripWS(FileRead($hFile), 2)

Hmmm, and I fear there is no really superior method to count linebreaks, as they are just character occurences of @lf or chr(10)...

Cheers

[SpookMeister]

I did a little creative thinking and came up with a workable solution for my needs.

Because I am only needing the number of lines for progress information, and in my case this does not need to be exact, I grabbed a sample of the file and averaged out the number of characters per line in the sample. Then, by multiplying the current line by the average I am able to gauge where I am in the the file by comparing that to the file size.

Here is an example of searching a large file for a string of text.

expandcollapse popup

#include <array.au3>
#include <string.au3>
HotKeySet("{ESC}", "Terminate")
Dim $a_Results[1]
$a_Results[0] = 0

; Select file
$Path = FileOpenDialog("Select the file to search", @WorkingDir & "\", "All Files (*.*)")
If @error Then
    MsgBox(0, "Error", "Failed to locate file")
    Exit
EndIf

; Request search string from the user
$s_SearchString = InputBox("Search String", "Enter the string that you want to search for:")
If @error Then
    MsgBox(0, "Error", "No search string entered")
    Exit
EndIf

; Find the Average number of Characters (Bytes) Per Line from a sample of the file
$i_FileSize = FileGetSize($Path)
$h_file = FileOpen($Path, 0)
If $h_file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
$i_bytes = 0
For $i = 1 To 50000 ; the 50k is arbitrary, but I found it took less than half a second to complete
    $line = FileReadLine($h_file)
    If @error = -1 Then ExitLoop
    $i_bytes += StringLen($line)
Next
FileClose($h_file)
$n_ABPL = $i_bytes / $i ; Average Bytes Per Line

; Re-open the file and begin search
$h_file = FileOpen($Path, 0)
$i_LineCount = 0
$i_sub = 0
While 1
    $line = FileReadLine($h_file)
    If @error = -1 Then ExitLoop
    $i_LineCount += 1
    $i_sub += 1
    Select
        Case StringInStr($line, $s_SearchString) ; if string is found add it to the array
            _ArrayAdd($a_Results, $line)
            $a_Results[0] += 1
        Case $i_sub >= 5000 ; every 5k lines update the tooltip
            $n_Estimate = Int($i_LineCount * $n_ABPL)
            $prog = _StringAddThousandsSep($n_Estimate) & " / " & _StringAddThousandsSep($i_FileSize)
            $msg = "Searching: " & @LF & _
                    $Path & @LF & @LF & _
                    "For the string:" & @LF & _
                    $s_SearchString & @LF & @LF & _
                    "Estimated Progress: " & $prog
            ToolTip($msg)
            $i_sub = 0
    EndSelect
WEnd
ToolTip("")
_ArrayDisplay($a_Results)

Func Terminate()
    MsgBox(0, "Abort", "Search aborted by user")
    Exit
EndFunc   ;==>Terminate

Edited January 21, 2009 by SpookMeister

[wiela]

I did a little creative thinking and came up with a workable solution for my needs.

<...>

Well, in that/such cases (huge text files; various bulk text/data processing) you should/could consider using special utilities,

such as word count (wc) utility and the others from gnuwin32 coreutils

(i don't think, that portability is the issue; AutoIt is not efficient (if suitable at all) for such cases either...)

Local $starttime = _Timer_Init()
RunWait(@ComSpec & " /c wc -l C:\tmp\test.txt  | cut -d ' ' -f 1 > C:\tmp\line_cnt.txt")
Local $fh = FileOpen("C:\tmp\line_cnt.txt", 0)
ConsoleWrite(@CRLF & "File size (MB): " & FileGetSize("C:\tmp\test.txt")/1048576)
ConsoleWrite(@CRLF & "Line count: " & FileReadLine($fh))
ConsoleWrite(@CRLF & "Count took (ms): " & _Timer_Diff($starttime) & @CRLF)

Results:

File size (MB): 888.109111785889

Line count: 71214500

Count took (ms): 107967.818535596

Note line count and the fact that test had run on the old laptop with a slow disk...

Overall speed increase should/could be up to 100x if the number of lines is around a few millions

and up to 10x in the "extreme" cases like in this test.

Edited January 4, 2010 by wiela

[GEOSoft]

I did a little creative thinking and came up with a workable solution for my needs.

Because I am only needing the number of lines for progress information, and in my case this does not need to be exact, I grabbed a sample of the file and averaged out the number of characters per line in the sample. Then, by multiplying the current line by the average I am able to gauge where I am in the the file by comparing that to the file size.

Here is an example of searching a large file for a string of text.

expandcollapse popup

#include <array.au3>
#include <string.au3>
HotKeySet("{ESC}", "Terminate")
Dim $a_Results[1]
$a_Results[0] = 0

; Select file
$Path = FileOpenDialog("Select the file to search", @WorkingDir & "\", "All Files (*.*)")
If @error Then
    MsgBox(0, "Error", "Failed to locate file")
    Exit
EndIf

; Request search string from the user
$s_SearchString = InputBox("Search String", "Enter the string that you want to search for:")
If @error Then
    MsgBox(0, "Error", "No search string entered")
    Exit
EndIf

; Find the Average number of Characters (Bytes) Per Line from a sample of the file
$i_FileSize = FileGetSize($Path)
$h_file = FileOpen($Path, 0)
If $h_file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
$i_bytes = 0
For $i = 1 To 50000 ; the 50k is arbitrary, but I found it took less than half a second to complete
    $line = FileReadLine($h_file)
    If @error = -1 Then ExitLoop
    $i_bytes += StringLen($line)
Next
FileClose($h_file)
$n_ABPL = $i_bytes / $i ; Average Bytes Per Line

; Re-open the file and begin search
$h_file = FileOpen($Path, 0)
$i_LineCount = 0
$i_sub = 0
While 1
    $line = FileReadLine($h_file)
    If @error = -1 Then ExitLoop
    $i_LineCount += 1
    $i_sub += 1
    Select
        Case StringInStr($line, $s_SearchString) ; if string is found add it to the array
            _ArrayAdd($a_Results, $line)
            $a_Results[0] += 1
        Case $i_sub >= 5000 ; every 5k lines update the tooltip
            $n_Estimate = Int($i_LineCount * $n_ABPL)
            $prog = _StringAddThousandsSep($n_Estimate) & " / " & _StringAddThousandsSep($i_FileSize)
            $msg = "Searching: " & @LF & _
                    $Path & @LF & @LF & _
                    "For the string:" & @LF & _
                    $s_SearchString & @LF & @LF & _
                    "Estimated Progress: " & $prog
            ToolTip($msg)
            $i_sub = 0
    EndSelect
WEnd
ToolTip("")
_ArrayDisplay($a_Results)

Func Terminate()
    MsgBox(0, "Abort", "Search aborted by user")
    Exit
EndFunc   ;==>Terminate

That could have been done much easier using a Regular Expression.

Just writing it on the fly without testing even the SRE

#include<array.au3> ;; For _ArrayDisplay() only
$Path = FileOpenDialog("Select the file to search", @WorkingDir & "\", "All Files (*.*)")
If @error Then
    MsgBox(0, "Error", "Failed to locate file")
    Exit
EndIf
$s_SearchString = InputBox("Search String", "Enter the string that you want to search for:")
If @error Then
    MsgBox(0, "Error", "No search string entered")
    Exit
EndIf
$aFound = _BuildArray($s_SearchString)
If NOT @Error Then
    _ArrayDisplay($aFound)
Else
    MsgBox(0, "Ooops", "Something is amiss")
EndIf

Func _BuildArray($sFind, $icase = 0);; If not 0 then match is case-sensitive
    $sCase = "(?i)"
    If $iCase Then $sCase = ""
    $aResults = StringRegExp(FileRead($Path), $sCase & "(?m:^).*" & $sFind & ".*(?:\v|$)+", 3)
    If @Error Then Return SetError(1)
    Return $aResults
EndFunc