Sign in to follow this  
Followers 0
SpuddyMcSpud

Fast log file analyser

6 posts in this topic

Hi all,

I did some searching but couldn't find anything that looked promising, apologies if there's a post I missed...

I'm looking to improve a log file analyser that I wrote. It works, but takes a stupid amount of time to complete since the logfiles are reasonably big (10-20MB). The script compares the last line of the server log file to the last line it recorded, and if there's new lines it processes through them, strips out lines containing a few regexp's, and writes the output to a new log file which I then have another script process and display in a GUI.

Can anyone think of a faster way to process through the log and create the output? Any help appreciated!

CODE
Test()

Func Test()

$LogFile_Test = "C:\log.log"

$AlertFile_Test = "C:\output.log"

$RegistryLastLine_Test = RegRead("HKLM\Software\Alerter", "AlertProcessor_RecordedLastLine_Test")

FileOpen($LogFile_Test, 0)

$LogFileLastLine = _FileCountLines($LogFile_Test)

FileClose($LogFile_Test)

If $RegistryLastLine_Test < $LogFileLastLine Then ; If the log file contains new lines

FileOpen($LogFile_Test, 0)

$AlertText = ""

$Line = $RegistryLastLine_Test

Do

$LineText = FileReadLine($LogFile_Test, $Line)

$LineTextSearch = StringRegExp($LineText, "(\.gif|\.jpg|\.png|\.css|\.js)")

If $LineTextSearch = 0 Then

$AlertText = $AlertText&@CRLF&$LineText

EndIf

$Line = $Line + 1

;MsgBox(0,"",$Line)

$RegistryLastLine_Test = RegWrite("HKLM\Software\Alerter", "AlertProcessor_RecordedLastLine_Test", "REG_SZ", $Line)

Until $Line = $LogFileLastLine

FileClose($LogFile_Test)

FileOpen($AlertFile_Test, 9)

FileWrite($AlertFile_Test, $AlertText)

FileClose($AlertFile_Test)

Snooze()

Else

Snooze()

EndIf

EndFunc

Func Snooze()

Sleep(25000)

Run(@ScriptName)

Exit

EndFunc

Share this post


Link to post
Share on other sites



_FileReadToArray rather than making several calls over and over, you'll have it all in one array.

On that size file... not quite sure how good regexp may be...but you may want to look at that option as well.


Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

Hi,

In my sig, look at "tailRW" and included files; search also for "tail", as someone has used this to write a server log/ chat thing which runs fast

The last line or any number of lines of the tail can be read for processing without reading the whole file.

best, randall

Share this post


Link to post
Share on other sites

Thanks smoke_n and randallc.

I think tailRW is what I'm after but it's probably a little beyond my scripting skills. I'll have a crack at it and see how I go.

cheers,

spud

Share this post


Link to post
Share on other sites

Thanks smoke_n and randallc.

I think tailRW is what I'm after but it's probably a little beyond my scripting skills. I'll have a crack at it and see how I go.

cheers,

spud

lod3n wrote a _runreadstd() udf, that coupled with a windows binary implementation of tail is all you need. ; nothing against the other udf's out there, but I haven't needed to explore them further with the above mentioned capabilities in the toolbox.


Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0