Log Parser

[Fade91]

Hello, first time poster here

I am working on a project that has to parse a log file in real time. The thing is I know it's hard for Autoit to attach itself to log files when they're already in use by other programs, at least in my experience.

I was taking a look at this thread because the log file is quite large and I think Autoit might be a little slow on it's own.

The thing is I don't know how to use this properly to extract all data out of a log file or is there a native way to do this using Autoit.

Basically , I just need a log parser that is able to read from a log that is 'already opened' and 'being written to'

 

Thanks!

[iamtheky]

I do not believe that AutoIt has any difficulties, the event log UDF reads from a log that is both already opened and being written to.   Since you have told us pretty much zero about the app or file lets begin:  when the app is running can you do a simple fileread("logfile.txt")?

[Fade91]

I actually haven't really written any code other than grab the File(Open,Read,Close) functions from the help file and changed the path to where my log is. I just tried it and the read does work when the application is running.

What I really would like is to be able to read each "new line" and do something and then move on to the next line.

I'm not quite sure how to start though.

[iamtheky]

there are a couple hundred ways you could do this, here is my thought:  Read to array, check the size, if it is bigger than before then read the last line.  If that works you can start getting sexy and return all the lines that are new, since you have both the previous and current size.  Something like this pseudo......

$oldnumberoflines = 0

While 1

$array = _FileReadToArray("logfile.log")

$newnumberoflines = ubound($array) - 1

If $newnumberoflines > $oldnumberoflines Then msgbox(0, '' , $array[ubound($array) - 1])

$newnumberoflines = $oldnumberoflines

sleep (5000) ; check every 5 seconds

wend

 

 

Edited August 23, 2015 by boththose

[Fade91]

#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

Global $Index = 0

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

While 1
    $Array = _FileReadToArray($Path, $Index, 0)

    $Recount_Log = UBound($Index) - 1

    If $Recount_Log > $Count_Log Then ToolTip($Index[UBound($Index) - 1], 0, 0)
    ;If $Recount_Log > $Count_Log Then MsgBox(0, '' , $Index[UBound($Index) - 1])

    $Recount_Log = $Count_Log
    Sleep(250)
WEnd

So far this works great! Thank you!

Edited August 23, 2015 by Fade91

[iamtheky]

If you need to do this on a log of any decent size, or if you really only want the last line:  you may want to consider filereadline($log , -1), as checking that one string will be much less of a penalty than the readtoarray and ubound check.  

 

Edited August 23, 2015 by boththose

[Fade91]

The file is written continuously and my script will perform actions on the most recent action it logs. Which means it will have to pause the script and continue on from where it left off afterwards. So reading just the last line wouldn't work all that well but may be if I were to get the line number and store it and continue on from there, that might just work.

Basically the script is taking this log information and projecting it on my monitor in the form of a GUI and when different toggles are on/off, it shows the user (me)

I'm not sure how well this is going to work out though because once it reads the last piece of the log it has to decode it and show it in the GUI and continue from where it left off, which is no big deal but the GUI is suppose to be active while its looping through the logs. So it's like I almost need two scripts communicating back and forth or something.

It seems like quite the project to be taking on but once I get the reading of the log finalized the GUI part is easier but its getting them to work at the same time that is going to be a bit challenging for me.

[iamtheky]

Sure, but it would still be a quicker first test in the current loop, because if that string matches then you could just sleep and check again.  and if it doesnt match then you can read the file to array without a need to test the ubound, and in the gui display all the lines between $Recount_log and ubound($array)  - 1. 

[Fade91]

I will experiment trying different things and see how far I get and if I need more help I will post here , I really appreciate your help!

Edited August 24, 2015 by Fade91

[kylomas]

Fade91,

The following is an example of how to return the unread portion of a file from scan to scan...

expandcollapse popup

; *** Start added by AutoIt3Wrapper ***
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
; *** End added by AutoIt3Wrapper ***

#include <array.au3>
#include <date.au3>
#include <File.au3>

Local $file_monitored   = @ScriptDir & '\log.log' ; file to monitor
Local $monitor_interval = .25 ; minute(s) between file check
Local $bMonitor         = False ;   switch to start monitor

#AutoIt3Wrapper_Add_Constants=n

Local $gui010 = GUICreate('Log Monitor')
Local $aSize  = WinGetClientSize($gui010)
Local $lbl010 = GUICtrlCreateLabel('Log', 10, 10, 50, 20)
Local $log010 = GUICtrlCreateEdit('', 10, 30, $aSize[0] - 20, $aSize[0] - 50, $es_readonly)
GUICtrlSetFont($log010, 7, 800, -1, 'lucinda console')
GUISetState()

; routine to start scan
AdlibRegister('_start_monitor', $monitor_interval * 60 * 1000)

While 1

    Switch GUIGetMsg()
        Case $gui_event_close
            Exit
    EndSwitch



    If $bMonitor = True Then
        _scan_file()
        $bMonitor = False
    EndIf



WEnd



; scan routine
Func _scan_file()

    Local $sTMP = ''
    Local Static $FilePos = 0

    ;read log file entries added since the last scan
    Local $fl = FileOpen($file_monitored)
    If $fl = -1 Then Exit MsgBox($MB_ICONERROR, '', 'File open failed')

    ; set file position to last position read
    If Not FileSetPos($fl, $FilePos, 0) Then _
            Exit MsgBox(0, 'ERROR', 'FileSetPos failed')

    ; read file from last position read to the end
    Local $sLogEntries = FileRead($fl)

    ; save file position for the next loop
    $FilePos = FileGetPos($fl)
    FileClose($fl)

    ; do whatever you want with the log entries...I am just writing them to the gui



    GUICtrlSetData($log010, 'Log Scan at ' & _Now() & @CRLF & $sLogEntries, 1)
    If StringRight(GUICtrlRead($log010), 2) <> @CRLF Then GUICtrlSetData($log010, @CRLF, 1)

EndFunc   ;==>_scan_file

; adlib to set switch that initiates scan
Func _start_monitor()
    $bMonitor = True
EndFunc   ;==>_start_monitor

Note - 1) If lines are deleted and subsequently populated that range will not be returned, although this could be coded for.

           2) The file position is only valid while the script is running.  You can probably externalize this value, if needed.

Cursory comments are included, please ask if you have questions.

kylomas

[Fade91]

#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

$Handle = FileOpen($Path)

While 1
    $Recount_Log = _FileCountLines($Path)
    If $Recount_Log > $Count_Log Then
        ConsoleWrite(FileReadLine($Path, _FileCountLines($Path))) ; last written log
        ; Do Stuff

_String(FileReadLine($Path, _FileCountLines($Path)))

    EndIf
    ;;; Probably won't use
    $Count_Log = $Recount_Log ; Would update to the very last written log
    ;;; Probably won't use
    ToolTip($Count_Log & @CRLF & $Recount_Log, 0, 0) ; log count compare (test only)
WEnd
FileClose($Handle)

 

 Now I need to somehow count where I left off and do something based on whats actually written and repeat this till I catch up to the 'last count'.

I was thinking something like...

_String(FileReadLine($Path, _FileCountLines($Path)))

and have the function figure out what to do with the info

Now with the useful information of the log im thinking about may be writing it to a seperate log or ini file and having the GUI as a seperate script and having it read the log/ini to show the needed information

The reason for this is to simply increase the speed of both scripts.

 

I think im quite happy may be going this route. I think it might work

Edited August 24, 2015 by Fade91

[jchd]

_String(FileReadLine($Path, _FileCountLines($Path)))

No, just use FileRead for the first time and store the byte size read off the file, which is set in @extended. Next runs, use FileSetPos to start reading up from the start of newly written data, then FileRead up to EOF. Of course this requires that nothing truncates the file in between, i.e. that the logger always appends data.