Fade91

Log Parser

12 posts in this topic

Hello, first time poster here :)

I am working on a project that has to parse a log file in real time. The thing is I know it's hard for Autoit to attach itself to log files when they're already in use by other programs, at least in my experience.

I was taking a look at this thread because the log file is quite large and I think Autoit might be a little slow on it's own.

The thing is I don't know how to use this properly to extract all data out of a log file or is there a native way to do this using Autoit.

Basically , I just need a log parser that is able to read from a log that is 'already opened' and 'being written to'

 

Thanks! :)

Share this post


Link to post
Share on other sites



I do not believe that AutoIt has any difficulties, the event log UDF reads from a log that is both already opened and being written to.   Since you have told us pretty much zero about the app or file lets begin:  when the app is running can you do a simple fileread("logfile.txt")?


,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

I actually haven't really written any code other than grab the File(Open,Read,Close) functions from the help file and changed the path to where my log is. I just tried it and the read does work when the application is running.

What I really would like is to be able to read each "new line" and do something and then move on to the next line.

I'm not quite sure how to start though.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

there are a couple hundred ways you could do this, here is my thought:  Read to array, check the size, if it is bigger than before then read the last line.  If that works you can start getting sexy and return all the lines that are new, since you have both the previous and current size.  Something like this pseudo......

$oldnumberoflines = 0

While 1

$array = _FileReadToArray("logfile.log")

$newnumberoflines = ubound($array) - 1

If $newnumberoflines > $oldnumberoflines Then msgbox(0, '' , $array[ubound($array) - 1])

$newnumberoflines = $oldnumberoflines

sleep (5000) ; check every 5 seconds

wend

 

 

Edited by boththose
1 person likes this

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

Global $Index = 0

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

While 1
    $Array = _FileReadToArray($Path, $Index, 0)

    $Recount_Log = UBound($Index) - 1

    If $Recount_Log > $Count_Log Then ToolTip($Index[UBound($Index) - 1], 0, 0)
    ;If $Recount_Log > $Count_Log Then MsgBox(0, '' , $Index[UBound($Index) - 1])

    $Recount_Log = $Count_Log
    Sleep(250)
WEnd

So far this works great! Thank you!

Edited by Fade91

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

If you need to do this on a log of any decent size, or if you really only want the last line:  you may want to consider filereadline($log , -1), as checking that one string will be much less of a penalty than the readtoarray and ubound check.  

 

Edited by boththose
1 person likes this

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

The file is written continuously and my script will perform actions on the most recent action it logs. Which means it will have to pause the script and continue on from where it left off afterwards. So reading just the last line wouldn't work all that well but may be if I were to get the line number and store it and continue on from there, that might just work.

Basically the script is taking this log information and projecting it on my monitor in the form of a GUI and when different toggles are on/off, it shows the user (me)

I'm not sure how well this is going to work out though because once it reads the last piece of the log it has to decode it and show it in the GUI and continue from where it left off, which is no big deal but the GUI is suppose to be active while its looping through the logs. So it's like I almost need two scripts communicating back and forth or something.

It seems like quite the project to be taking on but once I get the reading of the log finalized the GUI part is easier but its getting them to work at the same time that is going to be a bit challenging for me.

Share this post


Link to post
Share on other sites

Sure, but it would still be a quicker first test in the current loop, because if that string matches then you could just sleep and check again.  and if it doesnt match then you can read the file to array without a need to test the ubound, and in the gui display all the lines between $Recount_log and ubound($array)  - 1. 


,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

I will experiment trying different things and see how far I get and if I need more help I will post here :) , I really appreciate your help!

Edited by Fade91

Share this post


Link to post
Share on other sites

Fade91,

The following is an example of how to return the unread portion of a file from scan to scan...

; *** Start added by AutoIt3Wrapper ***
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
; *** End added by AutoIt3Wrapper ***

#include <array.au3>
#include <date.au3>
#include <File.au3>

Local $file_monitored   = @ScriptDir & '\log.log' ; file to monitor
Local $monitor_interval = .25 ; minute(s) between file check
Local $bMonitor         = False ;   switch to start monitor

#AutoIt3Wrapper_Add_Constants=n

Local $gui010 = GUICreate('Log Monitor')
Local $aSize  = WinGetClientSize($gui010)
Local $lbl010 = GUICtrlCreateLabel('Log', 10, 10, 50, 20)
Local $log010 = GUICtrlCreateEdit('', 10, 30, $aSize[0] - 20, $aSize[0] - 50, $es_readonly)
GUICtrlSetFont($log010, 7, 800, -1, 'lucinda console')
GUISetState()

; routine to start scan
AdlibRegister('_start_monitor', $monitor_interval * 60 * 1000)

While 1

    Switch GUIGetMsg()
        Case $gui_event_close
            Exit
    EndSwitch



    If $bMonitor = True Then
        _scan_file()
        $bMonitor = False
    EndIf



WEnd



; scan routine
Func _scan_file()

    Local $sTMP = ''
    Local Static $FilePos = 0

    ;read log file entries added since the last scan
    Local $fl = FileOpen($file_monitored)
    If $fl = -1 Then Exit MsgBox($MB_ICONERROR, '', 'File open failed')

    ; set file position to last position read
    If Not FileSetPos($fl, $FilePos, 0) Then _
            Exit MsgBox(0, 'ERROR', 'FileSetPos failed')

    ; read file from last position read to the end
    Local $sLogEntries = FileRead($fl)

    ; save file position for the next loop
    $FilePos = FileGetPos($fl)
    FileClose($fl)

    ; do whatever you want with the log entries...I am just writing them to the gui



    GUICtrlSetData($log010, 'Log Scan at ' & _Now() & @CRLF & $sLogEntries, 1)
    If StringRight(GUICtrlRead($log010), 2) <> @CRLF Then GUICtrlSetData($log010, @CRLF, 1)

EndFunc   ;==>_scan_file

; adlib to set switch that initiates scan
Func _start_monitor()
    $bMonitor = True
EndFunc   ;==>_start_monitor

Note - 1) If lines are deleted and subsequently populated that range will not be returned, although this could be coded for.

           2) The file position is only valid while the script is running.  You can probably externalize this value, if needed.

Cursory comments are included, please ask if you have questions.

kylomas

1 person likes this

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

$Handle = FileOpen($Path)

While 1
    $Recount_Log = _FileCountLines($Path)
    If $Recount_Log > $Count_Log Then
        ConsoleWrite(FileReadLine($Path, _FileCountLines($Path))) ; last written log
        ; Do Stuff

_String(FileReadLine($Path, _FileCountLines($Path)))

    EndIf
    ;;; Probably won't use
    $Count_Log = $Recount_Log ; Would update to the very last written log
    ;;; Probably won't use
    ToolTip($Count_Log & @CRLF & $Recount_Log, 0, 0) ; log count compare (test only)
WEnd
FileClose($Handle)

 

 Now I need to somehow count where I left off and do something based on whats actually written and repeat this till I catch up to the 'last count'.

I was thinking something like...

_String(FileReadLine($Path, _FileCountLines($Path)))

and have the function figure out what to do with the info

Now with the useful information of the log im thinking about may be writing it to a seperate log or ini file and having the GUI as a seperate script and having it read the log/ini to show the needed information

The reason for this is to simply increase the speed of both scripts.

 

I think im quite happy may be going this route. I think it might work :)

Edited by Fade91

Share this post


Link to post
Share on other sites

_String(FileReadLine($Path, _FileCountLines($Path)))

No, just use FileRead for the first time and store the byte size read off the file, which is set in @extended. Next runs, use FileSetPos to start reading up from the start of newly written data, then FileRead up to EOF. Of course this requires that nothing truncates the file in between, i.e. that the logger always appends data.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • Vivaed
      By Vivaed
      I have another AutoIT script making a Log file 

      Sample of Log file:
      2016/08/22 12:44:18 > Process: [RUNNING] [ACTIVE] 2016/08/22 12:48:35 > Process: [WAS NOT RUNNING] 2016/08/22 13:40:00 > Process: [FAILED] 2016/08/22 14:01:10 > Process: [WAS NOT RUNNING] I am looping through the Log file for  the word "FAILED"
      I then want to get all lines that have "FAILED" and get their TIME
       
      My Current code to get this far:
      If FileExists($fileLog) Then $contents = FileRead($fileLog) If @error Then MsgBox(0, 'File Error', $fileLog & ' could not be read.') Else For $i = 1 To _FileCountLines($fileLog) $result = StringInStr($contents,$search) If $result >= 1 Then $filteredLine = FileReadLine($fileLog,$i) If StringInStr($filteredLine,$search) Then ConsoleWrite($filteredLine & @CRLF) ; this gets me the results I want sans the time parse EndIf Else ConsoleWrite( $search & " not found!" & @CRLF) EndIf Next EndIf EndIf For this part:
      If StringInStr($filteredLine,$search) Then ConsoleWrite($filteredLine & @CRLF) ; this gets me the results I want sans the time parse EndIf OUTPUT: 2016/08/22 13:40:00 > Process: [FAILED] I dont understand how I read the time in that output?
       
      I have tried _DateTimeFormat - Dont think this applies 
      Tried _DateDiff - I dont have a the date yet so this doesnt work
       
      Would love if someone could tell me if I am thinking is the wrong direction and possibly lead me down the correct path to light side of the force  

       
    • mmoalem
      By mmoalem
      Hi there - quite a beginner to codding so please bare with me...
      i am writing an autoit code that will be running 24/7 executing various tasks every 10-20 minutes. it suppose to run on unattended server but i would like to occasionally remote log in and do stuff on the server without disturbing the autoit script (while the autoit script is in Sleep count). to achieve this I will need to know at what point in the commands run it is  and how long roughly until the next command (I use Sleep in between commands)...
       so, is there a way to create a monitor that shows me the last command executed and how long of the Sleep is left (like in a small window or in the windows task bar)?
      failing that any idea how can I utilize the FileWriteLog function to keep updating an always open text file ?
       
      thanks in advance
      michel
    • TheAutomator
      By TheAutomator
      Hi everyone
      Are there any people here that know how to use a parser generator that uses 'bnf' (not 'ebnf'!)
      i'm using the gold parser:
      http://goldparser.org/index.htm
      not possible in autoit:
      https://www.autoitscript.com/forum/topic/167663-activex-gold-parser-in-autoit-is-this-possible/
      What i'm looking for is the best way to parse a list:
      <List> ::= <Item> ',' <List> | <Item> <Item> ::= Number | String for example, the tree returned from:
      'test', 1, 2 is:

      since its not possible in AutoIT code I did it in vb script.
      this is what i did:
      ' in the parse loop: Case Rule_List_Comma set result = new list call result.input(.tokens(0).data,.tokens(2).data) ' the list class: class list private arg0 private arg1 public sub input(a,b) set arg0 = a set arg1 = b end sub private sub push(item,byref stack) redim preserve stack(ubound(stack) + 1) stack(ubound(stack)) = item end sub public function value value = array(arg0.value) value2 = arg1.value if isarray(value2) then for each thing in value2 push thing,value next else push value2,value end if end function end class  
      is there a better way to do this?
       
      regards,
      TheAutomator
    • rsmedude
      By rsmedude
      Hello everyone,
      I have been a member for a while but relatively new to posting to the message boards. I am also still relatively new to using AutoIt. I am working on a script that will help keep a program running on one of my file servers.
      Here is what I have accomplished so far looking at different threads on the site already.
      1. A check performed every 60 seconds to see if the application is running. (ultimately to be changed to something like once an hour)
      2. Write a log file that shows either the check was performed with no issue or the application had to be restarted.
      Both of those are working fine and I have to say that ever since I started using this script the program has not crashed!
      My next phase is to figure out how to make the script check and trim the log file. I am obviously not trying to make a log file that becomes bloated and chews up a large amount of hard drive space. The main purpose of the log file is so that I know if the application has been restarted. I am thinking have the script trim the file every 30 day maybe. Another words delete everything in the log file 30 days prior to the current day. So if someone can help me out here as I am not a scripting genius. Unfortunately I only know enough to be dangerous if you will.
      I also have another part of this script I want to write but first I want to get the log file working just the way I want. I am attaching a sample of what I have working so far.
      Thank you for any and all help folks.
      Respectfully,
      James
       
      #include <File.au3> While 1 ;Start Loop If ProcessExists("atcsmon.exe") Then ;~ Open the logfile in write mode. Local $hFile = FileOpen(@ScriptDir & "\ATCS_AutoIt.log", 1) _FileWriteLog($hFile, "ATCS check was completed and found running by the AutoIt3 Script") ; Write to the logfile passing the filehandle returned by FileOpen. FileClose($hFile) ; Close the filehandle to release the file. Else ;If Process does not exist Run("C:\atcs_monitor\atcsmon.exe", "", @SW_SHOWMINIMIZED) ;~ Open the logfile in write mode. Local $hFile = FileOpen(@ScriptDir & "\ATCS_AutoIt.log", 1) _FileWriteLog($hFile, "*ALERT* *ALERT* *ALERT* ATCS was restarted by the AutoIt3 Script") ; Write to the logfile passing the filehandle returned by FileOpen. FileClose($hFile) ; Close the filehandle to release the file. EndIf Sleep(60000) ;sleep 60 seconds WEnd ;Close Loop  
    • TheAutomator
      By TheAutomator
      hi everyone
      i have an interesting question about the gold parser:
      info:
      http://www.goldparser.org/index.htm
      download (regsvr32 to register):
      http://www.goldparser.org/engine/1/vb6/index.htm
      on the website it seems that this dll can be used as an activex object,
      does that mean that it can be used in autoit to?
      help for the activeX dll:
      http://www.goldparser.org/engine/1/vb6/doc/index.htm
      it gives me error code '4' if i try to use it... 
      Const $gpMsgAccept = 3 Const $gpMsgCommentBlockRead = 9 Const $gpMsgCommentError = 7 Const $gpMsgCommentLineRead = 10 Const $gpMsgInternalError = 8 Const $gpMsgLexicalError = 5 Const $gpMsgNotLoadedError = 4 Const $gpMsgReduction = 2 Const $gpMsgSyntaxError = 6 Const $gpMsgTokenRead = 1 $Parser = ObjCreate("goldparserengine.goldparser") $Parser.LoadCompiledGrammar("test_script.cgt") $Parser.OpenFile("Program.txt") $Response = $Parser.Parse() MsgBox(0,'test',$Response) If there are people interested in answering or helping feel free to reply and then i will upload the "test_script.cgt" somewhere if you want
      i know this question is a bit specific but you never know.. 
      Thanks for reading!
      TheAutomator.