Fast log file analyser

[SpuddyMcSpud]

Hi all,

I did some searching but couldn't find anything that looked promising, apologies if there's a post I missed...

I'm looking to improve a log file analyser that I wrote. It works, but takes a stupid amount of time to complete since the logfiles are reasonably big (10-20MB). The script compares the last line of the server log file to the last line it recorded, and if there's new lines it processes through them, strips out lines containing a few regexp's, and writes the output to a new log file which I then have another script process and display in a GUI.

Can anyone think of a faster way to process through the log and create the output? Any help appreciated!

CODE

Test()

Func Test()

$LogFile_Test = "C:\log.log"

$AlertFile_Test = "C:\output.log"

$RegistryLastLine_Test = RegRead("HKLM\Software\Alerter", "AlertProcessor_RecordedLastLine_Test")

FileOpen($LogFile_Test, 0)

$LogFileLastLine = _FileCountLines($LogFile_Test)

FileClose($LogFile_Test)

If $RegistryLastLine_Test < $LogFileLastLine Then ; If the log file contains new lines

FileOpen($LogFile_Test, 0)

$AlertText = ""

$Line = $RegistryLastLine_Test

Do

$LineText = FileReadLine($LogFile_Test, $Line)

$LineTextSearch = StringRegExp($LineText, "(\.gif|\.jpg|\.png|\.css|\.js)")

If $LineTextSearch = 0 Then

$AlertText = $AlertText&@CRLF&$LineText

EndIf

$Line = $Line + 1

;MsgBox(0,"",$Line)

$RegistryLastLine_Test = RegWrite("HKLM\Software\Alerter", "AlertProcessor_RecordedLastLine_Test", "REG_SZ", $Line)

Until $Line = $LogFileLastLine

FileClose($LogFile_Test)

FileOpen($AlertFile_Test, 9)

FileWrite($AlertFile_Test, $AlertText)

FileClose($AlertFile_Test)

Snooze()

Else

Snooze()

EndIf

EndFunc

Func Snooze()

Sleep(25000)

Run(@ScriptName)

Exit

EndFunc

[SmOke_N]

_FileReadToArray rather than making several calls over and over, you'll have it all in one array.

On that size file... not quite sure how good regexp may be...but you may want to look at that option as well.

[randallc]

Hi,

In my sig, look at "tailRW" and included files; search also for "tail", as someone has used this to write a server log/ chat thing which runs fast

The last line or any number of lines of the tail can be read for processing without reading the whole file.

best, randall

[SpuddyMcSpud]

Thanks smoke_n and randallc.

I think tailRW is what I'm after but it's probably a little beyond my scripting skills. I'll have a crack at it and see how I go.

cheers,

spud

[flyingboz]

Thanks smoke_n and randallc.

I think tailRW is what I'm after but it's probably a little beyond my scripting skills. I'll have a crack at it and see how I go.

cheers,

spud

lod3n wrote a _runreadstd() udf, that coupled with a windows binary implementation of tail is all you need. ; nothing against the other udf's out there, but I haven't needed to explore them further with the above mentioned capabilities in the toolbox.

[ptrex]

@SpuddyMcSpud

MS LogParser is made for this kind if jobs.

It is very fast since it uses an SQL engine.

MS LogParser in AU3

Regards

ptrex