Trailing a log file, efficiently.

[jokke]

Hiya!

Looking for a way to efficiently trail a log file.

I used methods such as:

First one:

$file = "Logfile.txt"

Dim $line[2] ;[lines total,lines read]
$line[0] = _FileCountLines($file) ;Lines file contain.
$line[1] = _FileCountLines($file) ;I want to start trailing from current line.

While 1
    $line[0] = _FileCountLines($file)
    If $line[0] >= $line[1] Then
        ConsoleWrite("Data read: " & FileReadLine($file,$line[1]) & @CRLF)
        $line[1] +=1
    EndIf
WEndoÝ÷ Ù'¢whê¾*.²­+(èZ±¦âùZÂ)Üjǧ{ew¢yh+-ëè­Âíuë^rÛajÖ®¶­sbb33c¶fÆRÒgV÷C´ÆövfÆRçGBgV÷C° ¤FÒb33c¶ÆæU³%Òµ¶ÆæW2F÷FÂÆÆæW2&VEТb33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ¶FFöâFRÆ7BÆæRà¢b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ´vçBFò7F'BG&Æærg&öÒ7W'&VçBÆæRà ¥vÆR b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ bb33c¶ÆæU³ÒfÇC²fwC²b33c¶ÆæU³ÒFVà b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ¶æ66RÆ7BÆæR6ævW27F÷&RBBöæ6Rà 6öç6öÆUw&FRgV÷C´FF&VC¢gV÷C²fײb33c¶ÆæU³Òfײ5$Äb VæD`¥tVæ@

Both of these ways are ok to use if the log file only contains a few lines, but when the log file is 6Mb large and cotains more then 50,000 lines both are either slow or unreliable.

All help appreciated

Edited August 9, 2008 by jokke

[jokke]

Oh and btw i cannot use this method either as the log file will allways grow:

#include <file.au3>
$file = "Logfile.txt"

Dim $line[1] 
$line[0] = _FileCountLines($file) ;data on the last line.

While 1
    $line[0] = _FileCountLines($file)
    If $line[0] > 1 Then
        $line[1] = FileReadLine($file,1) ;incase last line changes i store it at once.
        ConsoleWrite("Data read: " & $line[1] & @CRLF)
        _FileWriteToLine($file,"",1)
    EndIf
WEnd

Edited August 9, 2008 by jokke

[davecollinsnz]

Are there other programs monitoring the logfile? If not, you could rename logfile.txt to logfile_data.txt and create an empty logfile.txt file and monitor that file. When it changes, read the lines and append to logfile_data.txt and wipe logfile.txt.

[jokke]

Well the logfile is written by another application, it will always return the whole file again if i edit or delete it. Problem is that the other application does not append its log data, it writes the whole file on "update".

Else i would use the example i posted above witch is very efficiant when its useable.

Edited August 9, 2008 by jokke

[davecollinsnz]

Instead of counting file lines, what about getting the file size? When it changes, count the lines and get the new ones.

[ChrisL]

You need to use Randallc's APITailRW functions http://www.autoitscript.com/forum/index.ph...mp;#entry336438

[jokke]

Instead of counting file lines, what about getting the file size? When it changes, count the lines and get the new ones.

I'we tried that, but the file is so large and it grows all the time, a few lines every second so doing that would be releasing the stress from reading the the file, to checking the file size or modified time.

You need to use Randallc's APITailRW functions http://www.autoitscript.com/forum/index.ph...mp;#entry336438

Thanks, ill look at it

[ChrisL]

I'we tried that, but the file is so large and it grows all the time, a few lines every second so doing that would be releasing the stress from reading the the file, to checking the file size or modified time.

Thanks, ill look at it

I would guess you need to do a $var1 = filegetSize , then periodically do $Var2 = FileGetSize again and compare it to the first value, if it is different (If $var2 <> $var1) then you should be able to work out how much of the file you need to read in with APITailRW using the difference of $var1 and $var2 then once read store $var1 = $var2 and then go back to checking $var2 = FileGetSize

[ChrisL]

I would guess you need to do a $var1 = filegetSize , then periodically do $Var2 = FileGetSize again and compare it to the first value, if it is different (If $var2 <> $var1) then you should be able to work out how much of the file you need to read in with APITailRW using the difference of $var1 and $var2 then once read store $var1 = $var2 and then go back to checking $var2 = FileGetSize

Quick example

#include "APITailRW.au3"
$log = "Testlog.txt"
$copyLog = "Testlog2.txt"
$var1 = fileGetSize($log)
ConsoleWrite($var1 & @crlf)
While 1
    $var2 = FileGetSize($log)
    If $var2 <> $Var1 then 
        ConsoleWrite("Read in " & $var2 - $var1 & " bytes" &  @crlf)
        ConsoleWrite("New data" & @crlf)
        $file =_FileOpenAPI($log)
        $read = _FileReadAPI($file, $var2 - $var1, $var1)
        _FileCloseAPI($file)
        ConsoleWrite($read & @crlf)
        FileWrite($copyLog,$read)
        $var1 = $var2
    EndIf
    Sleep(1000)
WEnd

Also this is Randallc's APITailRW modified with the newer dllcall syntax see attached

APITailRW.au3

[jokke]

Also this is Randallc's APITailRW modified with the newer dllcall syntax see attached

Great, thanks alot

[jokke]

Oh damn it seems like this function is depending on that the file is not open.

Cause when i try to tail the file i only get this:

1555745
Read in 544 bytes 
New data: 
Read in 450 bytes 
New data: 
Read in 236 bytes 
New data: 
Read in 634 bytes 
New data:

But if i tail a random file then write to it it returns the data iwe added.

[ChrisL]

Not sure if this will work or not but try replacing this function _FileOpenAPI($sFile) in the APITailRW.au3 file with this one below. According to MSDN the file is opened with exclusive access so it is likely failing becasue another process already has it open, I think this should enable it to share access

Func _FileOpenAPI($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
;$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", 0, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc  ;==>_FileOpenAPI

[jokke]

Not sure if this will work or not but try replacing this function _FileOpenAPI($sFile) in the APITailRW.au3 file with this one below. According to MSDN the file is opened with exclusive access so it is likely failing becasue another process already has it open, I think this should enable it to share access

Func _FileOpenAPI($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
;$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", 0, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc ;==>_FileOpenAPI

Again thanks, you'r code did the trick!