FileReadLine

[jezzzzy]

I have created a little app to read a log file from my FTP server into a listview so I can filter the results. However, I have an efficiency problem. I have the listview repopulate from the log file at a set interval. I need a way to only have it read the new entries since the last refresh instead of reading the file from the beginning. Not sure how to keep track of where it left off in the last loop.

The function ReadFile() is where the magic happens. Snippet here:

While 1

        $line = FileReadLine($file)
        If @error = -1 Then ExitLoop; end of file
        If @error = 1 Then
            MsgBox(0,"error","Cannot read line")
        EndIf
        
;Valid Line - Count it
;$totallines = $totallines + 1
        
        If $optFilterRead = 4 Then
            GUICtrlCreateListViewItem($line,$listview)

Wend

wFTPLogRead.au3

Edited December 6, 2005 by jezzzzy

[Zedna]

Remember count of lines in file

and then read from this line (use number of line as parameter in FileReadLine) ...

EDIT: typos

Edited December 6, 2005 by Zedna

[Valuater]

look at this.. compare two files..

then load your view from the difference file

http://www.autoitscript.com/forum/index.ph...opic=18819&hl=#

8)

[jezzzzy]

Just didn't know if this was a good idea because the help file specifically cautions agains reading a file line by line because it forces AutoIt to search from the beginning every time to get to the requested line. I will try it and see what happens. Thanks.

[PartyPooper]

Count the number of lines using _FileCountLines( ) because it's quicker than reading the entire file in and incrementing a counter.

As for reading a particular line (or lines), I'm not sure what is quicker - reading the entire file into an array and using your line count to read from that point, or, using FileReadLine () and specifying the individual line (or lines) to read using your line count value.

I will do some timing tests today to see which is quicker.

[PartyPooper]

Took a couple of hours to run, but using the code below here are my results:

expandcollapse popup

#include <File.au3>
Dim $File
Dim $WCTimes
Dim $WOTimes
$File = 'C:\Documents and Settings\Admin\Desktop\SVSpeedStats\Data\lchnl.dat'; large text file
SplashTextOn("Please wait...", "Reading file.", 100, 20, -1, -1, 20, "", 10)
GUISetCursor(15, 1)
For $i = 1 To 5; run tests 5 times to reduce inconsistancies in results
    WithCount()
    WithOutCount()
Next
GUISetCursor(2, 0)
SplashOff()
    MsgBox(0,"Results","With Count Times:" & @CRLF & $WCTimes & @CRLF & @CRLF & "WithOut Count Times:" & @CRLF & $WOTimes )
Func WithCount()
    Local $TotalLines
    Local $x
    Local $Line
    Local $Timer
    Local $TimeDiff
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    For $x = $TotalLines - 300 To $TotalLines; reads each of the last 300 lines into the variable - $Line
        $Line = FileReadLine($File, $x)
    Next
    $TimeDiff = TimerDiff($Timer)
    $WCTimes = $WCTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc
Func WithOutCount()
    Local $TotalLines
    Local $x
    Local $Line[500000]; file has approimately 500,000 lines
    Local $Timer
    Local $TimeDiff
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    For $x = 1 To $TotalLines; reads all lines of the text file into the array - $Line
        $Line[$x] = FileReadLine($File)
    Next
    $TimeDiff = TimerDiff($Timer)
    $WOTimes = $WOTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc

The last couple of times on the WithOut results are slightly high because I had to use the computer while the test was running. In real terms, it didn't make much difference because they would have been around the 1020 mark anyway. Looks like reading a specific line is about 4 times as quick than reading a whole file. In truth, I was expecting it to be the other way around.

BTW, the text file I was using is about 15,500 KB (15.5 MB) and has just under 500,000 lines.

Results:

Edit: Brainfart - disregard results.

Edited December 8, 2005 by PartyPooper

[greenmachine]

...

expandcollapse popup

#include <File.au3>
Dim $File
Dim $WCTimes
Dim $WOTimes
$File = 'C:\Documents and Settings\Admin\Desktop\SVSpeedStats\Data\lchnl.dat'; large text file
SplashTextOn("Please wait...", "Reading file.", 100, 20, -1, -1, 20, "", 10)
GUISetCursor(15, 1)
For $i = 1 To 5; run tests 5 times to reduce inconsistancies in results
    WithCount()
    WithOutCount()
Next
GUISetCursor(2, 0)
SplashOff()
    MsgBox(0,"Results","With Count Times:" & @CRLF & $WCTimes & @CRLF & @CRLF & "WithOut Count Times:" & @CRLF & $WOTimes )
Func WithCount()
    Local $TotalLines
    Local $x
    Local $Line
    Local $Timer
    Local $TimeDiff
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    For $x = $TotalLines - 300 To $TotalLines; reads each of the last 300 lines into the variable - $Line
        $Line = FileReadLine($File, $x)
    Next
    $TimeDiff = TimerDiff($Timer)
    $WCTimes = $WCTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc
Func WithOutCount()
    Local $TotalLines
    Local $x
    Local $Line[500000]; file has approimately 500,000 lines
    Local $Timer
    Local $TimeDiff
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    For $x = 1 To $TotalLines; reads all lines of the text file into the array - $Line
        $Line[$x] = FileReadLine($File)
    Next
    $TimeDiff = TimerDiff($Timer)
    $WOTimes = $WOTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc

The last couple of times on the WithOut results are slightly high because I had to use the computer while the test was running. In real terms, it didn't make much difference because they would have been around the 1020 mark anyway. Looks like reading a specific line is about 4 times as quick than reading a whole file. In truth, I was expecting it to be the other way around.

BTW, the text file I was using is about 15,500 KB (15.5 MB) and has just under 500,000 lines.

...

Might it be that your two functions are counting a different amount of lines?

Func WithCount()

.....

For $x = $TotalLines - 300 To $TotalLines; reads each of the last 300 lines into the variable - $Line

$Line = FileReadLine($File, $x)

Next

.....

EndFunc

(only 300 lines being read)

Func WithOutCount()

.....

For $x = 1 To $TotalLines; reads all lines of the text file into the array - $Line

$Line[$x] = FileReadLine($File)

Next

.....

EndFunc

(all lines read)

That could make a huge difference, especially if your file truly has 500,000 lines.

[PartyPooper]

Of course, that was the point to the test - I coded it that way. The reason I was surprised with the results was that in the Help File under FileReadLine () it states:

From a performance standpoint it is a bad idea to read line by line specifying "line" parameter whose value is incrementing by one. This forces AutoIt to reread the file from the beginning until it reach the specified line.

I had assumed using the line parameter of FileReadLine() would be slower than reading the entire file. Maybe I'm just misreading that statement...

Yes, the file truly does have close to 500,000 lines.

[PartyPooper]

OK, had a brainfart and forgot to open and close the file before reading. This has a substantial impact upon the results. I will redo the tests because I know using the line option is slower.

[PartyPooper]

Updated the code to overcome my brainfart and re-ran the test (only one interation though because it's slooow).

expandcollapse popup

#include <File.au3>
Dim $File
Dim $WCTimes
Dim $WOTimes
$File = 'C:\Documents and Settings\Admin\Desktop\SVSpeedStats\Data\lchnl.dat'; large text file
SplashTextOn("Please wait...", "Reading file.", 100, 20, -1, -1, 20, "", 10)
GUISetCursor(15, 1)
For $i = 1 To 1
    WithCount()
    WithOutCount()
Next
GUISetCursor(2, 0)
SplashOff()
    MsgBox(0,"Results","With Count Times:" & @CRLF & $WCTimes & @CRLF & @CRLF & "WithOut Count Times:" & @CRLF & $WOTimes )
Func WithCount()
    Local $TotalLines
    Local $x
    Local $Line
    Local $Timer
    Local $TimeDiff
    Local $OpenFile
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    $OpenFile = FileOpen($File, 0)
    For $x = $TotalLines - 300 To $TotalLines; reads each of the last 300 lines into the variable - $Line
        $Line = FileReadLine($OpenFile, $x)
    Next
    FileClose($OpenFile)
    $TimeDiff = TimerDiff($Timer)
    $WCTimes = $WCTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc
Func WithOutCount()
    Local $TotalLines
    Local $x
    Local $Line[500000]; file has approximately 500,000 lines
    Local $Timer
    Local $TimeDiff
    Local $OpenFile
    $Timer = TimerInit()
    $TotalLines = _FileCountLines($File); total number of lines in text file
    $OpenFile = FileOpen($File, 0)
    For $x = 1 To $TotalLines; reads all lines of the text file into the array - $Line
        $Line[$x] = FileReadLine($OpenFile)
    Next
    FileClose($OpenFile)
    $TimeDiff = TimerDiff($Timer)
    $WOTimes = $WOTimes & Round($TimeDiff/1000,3) & " secs" & @CRLF
EndFunc

As you can see (and IAW the help file), using FileReadLine() with the line parameter is significantly slower than using it without (see Result1).

Result1:

However, if you have a large file and only need to read the last couple of lines, then using FileReadLine() with the line parameter is quicker than using it without (see Result2). Here is shows it takes about 2 secs to read the last line of a 500,000 line file, while reading the entire file takes about 8.5 secs. I ran this test 5 times to be sure.

Result2:

Edit: forgot to say I ran the last test 5 times

Edited December 8, 2005 by PartyPooper