_FileLineCount Blank Lines

[me88]

Hi Everyone.

I tried searching the forums for a solution but can't seem to find anything on this (I thought it would've come up before). I am working on this fairly simple program that needs to count the number of lines in a text file and I'm using:

$LineCount = _FileCountLines("new.txt")

to give me the number of lines in the file. This works fine until I have a blank line at the end of the file which it won't count but I need it to. I was wondering if there was a simple way to get it to recognize the last line or lines of the file if they are blank..?

Thanks.

[jchd]

If your files aren't too large to fit in memory, you can use string functions to count precisely what you want.

For instance, the following will count the number of LineFeed characters in the file:

Local $s = FileRead('example.txt')
StringReplace($s, @LF, @LF, 0, 1)
Local $LFcount = @extended
ConsoleWrite($LFcount & @LF)

EDIT: btw, welcome to the forum!

Edited May 23, 2010 by jchd

[Malkey]

If your files aren't too large to fit in memory, you can use string functions to count precisely what you want.

For instance, the following will count the number of LineFeed characters in the file:

Local $s = FileRead('example.txt')
StringReplace($s, @LF, @LF, 0, 1)
Local $LFcount = @extended
ConsoleWrite($LFcount & @LF)

EDIT: btw, welcome to the forum!

The last line, blank or otherwise, of a file does not end with a @LF ("Line Feed"), a vertical white space . Because, if the line did end with a vertical white space, the line would not be the last line.

The line :-

Local $LFcount = @extended

should be

Local $LFcount = @extended + 1

in order to count the last line.

[jchd]

Correct. This was due to the posting time (=7:07 AM after a day + night of hard work)

[wolf9228]

Hi Everyone.

I tried searching the forums for a solution but can't seem to find anything on this (I thought it would've come up before). I am working on this fairly simple program that needs to count the number of lines in a text file and I'm using:

$LineCount = _FileCountLines("new.txt")

to give me the number of lines in the file. This works fine until I have a blank line at the end of the file which it won't count but I need it to. I was wondering if there was a simple way to get it to recognize the last line or lines of the file if they are blank..?

Thanks.

#Include <WinAPI.au3>

MsgBox(0,"",FileCountLines("new.txt"))

Func FileCountLines($FileName)
Local $CountLines = 1,$nBytes,$Size,$tBuffer,$hFile
$Size = FileGetSize($FileName)
If $Size = 0 Then Return 0
$hFile = _WinAPI_CreateFile($FileName, 2, 2)
$tBuffer = DllStructCreate("CHAR[1]")
While $Size
$Size -= 1
_WinAPI_SetFilePointer($hFile, $Size)
_WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), 1, $nBytes)
If DllStructGetData($tBuffer,1) == @LF Then 
$Size -= 1
_WinAPI_SetFilePointer($hFile, $Size)
_WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), 1, $nBytes)
If DllStructGetData($tBuffer,1) == @CR Then $CountLines += 1
EndIf
WEnd
_WinAPI_CloseHandle($hFile)
Return $CountLines 
EndFunc

Edited May 23, 2010 by wolf9228

[jchd]

Overcomplex: you'd rather use FileSetPos to do that.

[wolf9228]

Overcomplex: you'd rather use FileSetPos to do that.

#Include <WinAPI.au3>

MsgBox(0,"",FileCountLines("rr.txt"))

Func FileCountLines($FileName)
Local $CountLines = 1,$Size,$hFile
$Size = FileGetSize($FileName)
If $Size = 0 Then Return 0
$hFile = FileOpen($FileName, 0)
While $Size
$Size -= 1
FileSetPos($hFile,$Size, $FILE_BEGIN)
$chars = FileRead($hFile, 1)
If $chars == @LF Then 
$Size -= 1
FileSetPos($hFile,$Size, $FILE_BEGIN)
$chars = FileRead($hFile, 1)
If $chars == @CR Then $CountLines += 1
EndIf
WEnd
FileClose($hFile)
Return $CountLines 
EndFunc

Edited May 23, 2010 by wolf9228

[GEOSoft]

Or something like

StringRegExpReplace($sStr, "(?m:^|\n)(?:\v+|$)", "")
MsgBox(0, "Result", "There are " & @Extended & " blank lines.")

Edited May 23, 2010 by GEOSoft

[JailDoctor]

HI guys,

I borrowed stuff from here to count lines in extremelly large txt files. This works fine for files smaller than 1/2 GB. Larger files (1.1 GB) makes the script stops with a message "error allocating memory." I have some files as big as 40 GB that I will need to count their lines. Thanks.

Here is the code:

Local $message = "Please choose a file to count its lines."

Local $var = FileOpenDialog($message, @MyDocumentsDir & "\", "Text (*.txt;*.csv)", 1 + 4)

If @error Then
    MsgBox(64, "JailDoctor", "No File chosen",5)
    Exit
Else

    $var = StringReplace($var, "|", @CRLF)
    Local $CountLines = _FileCountLines($var)
EndIf
MsgBox(64,"Lines Total" , "There are " & $CountLines - 1 & " lines in file " & $var)

Edited June 3, 2013 by JailDoctor

[jchd]

For files of that size you would better use a canned tool like Windows port of wc. wc -l <file> will work faster than reading the file line by line in AutoIt.

[guinness]

I thought you would have used a regular expression jchd back 3 years ago?

[jchd]

Yes I remember that but for speed efficiency on files known to be of that size, I bet that the simple wc will easily outperform any AutoIt-based algorithm. To clarify, the wallclock difference will be large enough to make the former latter unbearable.

Years ago we started (IIRC) with a simple "read the whole lot and regex it" but derived to a more sliced approach for moderately larger files. Here we know we'll have to deal with huge files almost exclusively. Just 40Gib of I/O takes long time.

EDIT: strikeout

Edited June 3, 2013 by jchd

[JailDoctor]

Thak you guys. I will research what is Windows port of wc. wc -l <file>, learn to use it and will let you know.

[guinness]

I found one on sysinternals.com.

[Malkey]

The Powershell approach.

For Win 7 and newer, Powershell comes with the system.

For older than Win 7, Microsoft has free downloads of Powershell 1.0 for different platforms.

Using this example, my 60.9MB text file has 669867 lines and took around 2 secs to count.

Local $begin = TimerInit()
Local $sFileName = "C:\....\Documents\File60_9MB.txt" ; <--- Full path of file.

; Powershell command - Find number of lines in file.
Local $sCommand = '$count = 0' & @CRLF
$sCommand &= '$reader = New-Object IO.StreamReader "' & $sFileName & '"' & @CRLF
$sCommand &= 'while($reader.ReadLine() -ne $null){ $count++ }' & @CRLF
$sCommand &= '$reader.Close()' & @CRLF
$sCommand &= '$count' & @CRLF

MsgBox(0, "Line Count", "Number of lines in " & $sFileName & " is " & _PowerShell($sCommand, "") & @CRLF & @CRLF & _
        "Time taken: " & Round(TimerDiff($begin) / 1000, 3) & "secs" & @CRLF)

; Note: My 60.9MB text file has 669867 lines and took around 2 secs to count.


; ----- Check that powershell.exe path is correct. --------
; Note: Use single quotes to enclose the commands, and double quotes for internal quotes.
Func _PowerShell($sCmd, $sDir = "")
    $sCmd = StringReplace($sCmd, '"', '"""') ; Enables any internal quotes to be read in AutoIt's Run function which runs the powershell commands in powershell.
    Local $text = '', $Pid = Run('"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" ' & $sCmd, $sDir, @SW_HIDE, 2 + 4)
    While 1
        $text &= StdoutRead($Pid, False, False)
        If @error Then ExitLoop
        Sleep(10)
    WEnd
    While 1
        $text &= StderrRead($Pid)
        If @error Then ExitLoop
        ConsoleWrite(@LF & "!#------------------------# Error #---------------------------# " & @LF)
    WEnd
    Return $text
EndFunc   ;==>_PowerShell

[guinness]

First time I ran your code Malkey, it took 7 seconds on file with 33 lines, then running again it took 1 second. Perhaps because I don't use PowerShell that often.

[jchd]

Got stain in your harddisk, guinness ? Try WD40

[guinness]

Now it's fine. I will re-start my system today and test again. I will also try the WD40 jchd!

[JailDoctor]

It took 10 seconds to count 3,005,950 in a 1.1 GB file.

I stoped it after afte 3 minutes on my 40 GB file.

[guinness]

JailDoctor,

What text files are you working with that have a size of 40GB?