Sign in to follow this  
Followers 0
jokke

Trailing a log file, efficiently.

13 posts in this topic

#1 ·  Posted (edited)

Hiya!

Looking for a way to efficiently trail a log file.

I used methods such as:

First one:

$file = "Logfile.txt"

Dim $line[2] ;[lines total,lines read]
$line[0] = _FileCountLines($file) ;Lines file contain.
$line[1] = _FileCountLines($file) ;I want to start trailing from current line.

While 1
    $line[0] = _FileCountLines($file)
    If $line[0] >= $line[1] Then
        ConsoleWrite("Data read: " & FileReadLine($file,$line[1]) & @CRLF)
        $line[1] +=1
    EndIf
WEndoÝ÷ Ù'¢whê¾*.²­+(èZ±¦âùZÂ)Üjǧ{ew¢yh+-ëè­Âíuë^rÛajÖ®¶­sbb33c¶fÆRÒgV÷C´ÆövfÆRçGBgV÷C° ¤FÒb33c¶ÆæU³%Òµ¶ÆæW2F÷FÂÆÆæW2&VEТb33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ¶FFöâFRÆ7BÆæRà¢b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ´vçBFò7F'BG&Æærg&öÒ7W'&VçBÆæRà ¥vÆR b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ bb33c¶ÆæU³ÒfÇC²fwC²b33c¶ÆæU³ÒFVà b33c¶ÆæU³ÒÒfÆU&VDÆæRb33c¶fÆRÂÓ¶æ66RÆ7BÆæR6ævW27F÷&RBBöæ6Rà 6öç6öÆUw&FRgV÷C´FF&VC¢gV÷C²fײb33c¶ÆæU³Òfײ5$Äb VæD`¥tVæ@

Both of these ways are ok to use if the log file only contains a few lines, but when the log file is 6Mb large and cotains more then 50,000 lines both are either slow or unreliable.

All help appreciated :P

Edited by jokke

UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Oh and btw i cannot use this method either as the log file will allways grow:

#include <file.au3>
$file = "Logfile.txt"

Dim $line[1] 
$line[0] = _FileCountLines($file) ;data on the last line.

While 1
    $line[0] = _FileCountLines($file)
    If $line[0] > 1 Then
        $line[1] = FileReadLine($file,1) ;incase last line changes i store it at once.
        ConsoleWrite("Data read: " & $line[1] & @CRLF)
        _FileWriteToLine($file,"",1)
    EndIf
WEnd
Edited by jokke

UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

Are there other programs monitoring the logfile? If not, you could rename logfile.txt to logfile_data.txt and create an empty logfile.txt file and monitor that file. When it changes, read the lines and append to logfile_data.txt and wipe logfile.txt.


Trust me to make SkyNet with AutoIt...

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Well the logfile is written by another application, it will always return the whole file again if i edit or delete it. Problem is that the other application does not append its log data, it writes the whole file on "update".

Else i would use the example i posted above witch is very efficiant when its useable.

Edited by jokke

UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

Instead of counting file lines, what about getting the file size? When it changes, count the lines and get the new ones.


Trust me to make SkyNet with AutoIt...

Share this post


Link to post
Share on other sites

Instead of counting file lines, what about getting the file size? When it changes, count the lines and get the new ones.

I'we tried that, but the file is so large and it grows all the time, a few lines every second so doing that would be releasing the stress from reading the the file, to checking the file size or modified time.

You need to use Randallc's APITailRW functions http://www.autoitscript.com/forum/index.ph...mp;#entry336438

Thanks, ill look at it :P


UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

I'we tried that, but the file is so large and it grows all the time, a few lines every second so doing that would be releasing the stress from reading the the file, to checking the file size or modified time.

Thanks, ill look at it :P

I would guess you need to do a $var1 = filegetSize , then periodically do $Var2 = FileGetSize again and compare it to the first value, if it is different (If $var2 <> $var1) then you should be able to work out how much of the file you need to read in with APITailRW using the difference of $var1 and $var2 then once read store $var1 = $var2 and then go back to checking $var2 = FileGetSize

Share this post


Link to post
Share on other sites

I would guess you need to do a $var1 = filegetSize , then periodically do $Var2 = FileGetSize again and compare it to the first value, if it is different (If $var2 <> $var1) then you should be able to work out how much of the file you need to read in with APITailRW using the difference of $var1 and $var2 then once read store $var1 = $var2 and then go back to checking $var2 = FileGetSize

Quick example

#include "APITailRW.au3"
$log = "Testlog.txt"
$copyLog = "Testlog2.txt"
$var1 = fileGetSize($log)
ConsoleWrite($var1 & @crlf)
While 1
    $var2 = FileGetSize($log)
    If $var2 <> $Var1 then 
        ConsoleWrite("Read in " & $var2 - $var1 & " bytes" &  @crlf)
        ConsoleWrite("New data" & @crlf)
        $file =_FileOpenAPI($log)
        $read = _FileReadAPI($file, $var2 - $var1, $var1)
        _FileCloseAPI($file)
        ConsoleWrite($read & @crlf)
        FileWrite($copyLog,$read)
        $var1 = $var2
    EndIf
    Sleep(1000)
WEnd

Also this is Randallc's APITailRW modified with the newer dllcall syntax see attached

APITailRW.au3

Share this post


Link to post
Share on other sites

Also this is Randallc's APITailRW modified with the newer dllcall syntax see attached

Great, thanks alot :P


UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

Oh damn it seems like this function is depending on that the file is not open.

Cause when i try to tail the file i only get this:

1555745
Read in 544 bytes 
New data: 
Read in 450 bytes 
New data: 
Read in 236 bytes 
New data: 
Read in 634 bytes 
New data:

But if i tail a random file then write to it it returns the data iwe added.


UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

Not sure if this will work or not but try replacing this function _FileOpenAPI($sFile) in the APITailRW.au3 file with this one below. According to MSDN the file is opened with exclusive access so it is likely failing becasue another process already has it open, I think this should enable it to share access

Func _FileOpenAPI($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
;$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", 0, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc  ;==>_FileOpenAPI

Share this post


Link to post
Share on other sites

Not sure if this will work or not but try replacing this function _FileOpenAPI($sFile) in the APITailRW.au3 file with this one below. According to MSDN the file is opened with exclusive access so it is likely failing becasue another process already has it open, I think this should enable it to share access

Func _FileOpenAPI($sFile)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080, $FILE_SHARE_NONE = 0x00000000, $FILE_SHARE_READ = 0x00000001, $FILE_SHARE_WRITE = 0x00000002
    Local $AFO_h
;$AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", 0, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long",BitOR($GENERIC_READ, $GENERIC_WRITE), "long", BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE), "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc ;==>_FileOpenAPI
Again thanks, you'r code did the trick!

UDF:Crypter a file encrypt / decrypt tool with no need to remember a password again. Based on Caesar cipher using entire ASCII Table.Script's: PixelSearch Helper, quick and simple way to create a PixelSeach.Chatserver - simplified, not so complicated multi-socket server.AutoIT - Firewall, simple example on howto create a firewall with AutoIt.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0