Jump to content

Recommended Posts

Hello, first time poster here :)

I am working on a project that has to parse a log file in real time. The thing is I know it's hard for Autoit to attach itself to log files when they're already in use by other programs, at least in my experience.

I was taking a look at this thread because the log file is quite large and I think Autoit might be a little slow on it's own.

The thing is I don't know how to use this properly to extract all data out of a log file or is there a native way to do this using Autoit.

Basically , I just need a log parser that is able to read from a log that is 'already opened' and 'being written to'

 

Thanks! :)

Share this post


Link to post
Share on other sites

I do not believe that AutoIt has any difficulties, the event log UDF reads from a log that is both already opened and being written to.   Since you have told us pretty much zero about the app or file lets begin:  when the app is running can you do a simple fileread("logfile.txt")?


,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

I actually haven't really written any code other than grab the File(Open,Read,Close) functions from the help file and changed the path to where my log is. I just tried it and the read does work when the application is running.

What I really would like is to be able to read each "new line" and do something and then move on to the next line.

I'm not quite sure how to start though.

Share this post


Link to post
Share on other sites

there are a couple hundred ways you could do this, here is my thought:  Read to array, check the size, if it is bigger than before then read the last line.  If that works you can start getting sexy and return all the lines that are new, since you have both the previous and current size.  Something like this pseudo......

$oldnumberoflines = 0

While 1

$array = _FileReadToArray("logfile.log")

$newnumberoflines = ubound($array) - 1

If $newnumberoflines > $oldnumberoflines Then msgbox(0, '' , $array[ubound($array) - 1])

$newnumberoflines = $oldnumberoflines

sleep (5000) ; check every 5 seconds

wend

 

 

Edited by boththose

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites
#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

Global $Index = 0

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

While 1
    $Array = _FileReadToArray($Path, $Index, 0)

    $Recount_Log = UBound($Index) - 1

    If $Recount_Log > $Count_Log Then ToolTip($Index[UBound($Index) - 1], 0, 0)
    ;If $Recount_Log > $Count_Log Then MsgBox(0, '' , $Index[UBound($Index) - 1])

    $Recount_Log = $Count_Log
    Sleep(250)
WEnd

So far this works great! Thank you!

Edited by Fade91

Share this post


Link to post
Share on other sites

If you need to do this on a log of any decent size, or if you really only want the last line:  you may want to consider filereadline($log , -1), as checking that one string will be much less of a penalty than the readtoarray and ubound check.  

 

Edited by boththose

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

The file is written continuously and my script will perform actions on the most recent action it logs. Which means it will have to pause the script and continue on from where it left off afterwards. So reading just the last line wouldn't work all that well but may be if I were to get the line number and store it and continue on from there, that might just work.

Basically the script is taking this log information and projecting it on my monitor in the form of a GUI and when different toggles are on/off, it shows the user (me)

I'm not sure how well this is going to work out though because once it reads the last piece of the log it has to decode it and show it in the GUI and continue from where it left off, which is no big deal but the GUI is suppose to be active while its looping through the logs. So it's like I almost need two scripts communicating back and forth or something.

It seems like quite the project to be taking on but once I get the reading of the log finalized the GUI part is easier but its getting them to work at the same time that is going to be a bit challenging for me.

Share this post


Link to post
Share on other sites

Sure, but it would still be a quicker first test in the current loop, because if that string matches then you could just sleep and check again.  and if it doesnt match then you can read the file to array without a need to test the ubound, and in the gui display all the lines between $Recount_log and ubound($array)  - 1. 


,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

I will experiment trying different things and see how far I get and if I need more help I will post here :) , I really appreciate your help!

Edited by Fade91

Share this post


Link to post
Share on other sites

Fade91,

The following is an example of how to return the unread portion of a file from scan to scan...

; *** Start added by AutoIt3Wrapper ***
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
; *** End added by AutoIt3Wrapper ***

#include <array.au3>
#include <date.au3>
#include <File.au3>

Local $file_monitored   = @ScriptDir & '\log.log' ; file to monitor
Local $monitor_interval = .25 ; minute(s) between file check
Local $bMonitor         = False ;   switch to start monitor

#AutoIt3Wrapper_Add_Constants=n

Local $gui010 = GUICreate('Log Monitor')
Local $aSize  = WinGetClientSize($gui010)
Local $lbl010 = GUICtrlCreateLabel('Log', 10, 10, 50, 20)
Local $log010 = GUICtrlCreateEdit('', 10, 30, $aSize[0] - 20, $aSize[0] - 50, $es_readonly)
GUICtrlSetFont($log010, 7, 800, -1, 'lucinda console')
GUISetState()

; routine to start scan
AdlibRegister('_start_monitor', $monitor_interval * 60 * 1000)

While 1

    Switch GUIGetMsg()
        Case $gui_event_close
            Exit
    EndSwitch



    If $bMonitor = True Then
        _scan_file()
        $bMonitor = False
    EndIf



WEnd



; scan routine
Func _scan_file()

    Local $sTMP = ''
    Local Static $FilePos = 0

    ;read log file entries added since the last scan
    Local $fl = FileOpen($file_monitored)
    If $fl = -1 Then Exit MsgBox($MB_ICONERROR, '', 'File open failed')

    ; set file position to last position read
    If Not FileSetPos($fl, $FilePos, 0) Then _
            Exit MsgBox(0, 'ERROR', 'FileSetPos failed')

    ; read file from last position read to the end
    Local $sLogEntries = FileRead($fl)

    ; save file position for the next loop
    $FilePos = FileGetPos($fl)
    FileClose($fl)

    ; do whatever you want with the log entries...I am just writing them to the gui



    GUICtrlSetData($log010, 'Log Scan at ' & _Now() & @CRLF & $sLogEntries, 1)
    If StringRight(GUICtrlRead($log010), 2) <> @CRLF Then GUICtrlSetData($log010, @CRLF, 1)

EndFunc   ;==>_scan_file

; adlib to set switch that initiates scan
Func _start_monitor()
    $bMonitor = True
EndFunc   ;==>_start_monitor

Note - 1) If lines are deleted and subsequently populated that range will not be returned, although this could be coded for.

           2) The file position is only valid while the script is running.  You can probably externalize this value, if needed.

Cursory comments are included, please ask if you have questions.

kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites
#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>

$Path = "output_log.txt"
$Count_Log = _FileCountLines($Path)

$Handle = FileOpen($Path)

While 1
    $Recount_Log = _FileCountLines($Path)
    If $Recount_Log > $Count_Log Then
        ConsoleWrite(FileReadLine($Path, _FileCountLines($Path))) ; last written log
        ; Do Stuff

_String(FileReadLine($Path, _FileCountLines($Path)))

    EndIf
    ;;; Probably won't use
    $Count_Log = $Recount_Log ; Would update to the very last written log
    ;;; Probably won't use
    ToolTip($Count_Log & @CRLF & $Recount_Log, 0, 0) ; log count compare (test only)
WEnd
FileClose($Handle)

 

 Now I need to somehow count where I left off and do something based on whats actually written and repeat this till I catch up to the 'last count'.

I was thinking something like...

_String(FileReadLine($Path, _FileCountLines($Path)))

and have the function figure out what to do with the info

Now with the useful information of the log im thinking about may be writing it to a seperate log or ini file and having the GUI as a seperate script and having it read the log/ini to show the needed information

The reason for this is to simply increase the speed of both scripts.

 

I think im quite happy may be going this route. I think it might work :)

Edited by Fade91

Share this post


Link to post
Share on other sites

_String(FileReadLine($Path, _FileCountLines($Path)))

No, just use FileRead for the first time and store the byte size read off the file, which is set in @extended. Next runs, use FileSetPos to start reading up from the start of newly written data, then FileRead up to EOF. Of course this requires that nothing truncates the file in between, i.e. that the logger always appends data.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Colduction
      Hi again guys!, i had COVID-19 for twice and i couldn't check the forum since 3 or 4 months ago till now! i hope you will get better if you're fighting for beat COVID-19

      I have two question, first is about extracting all of the IP Address from an IP Ranges, for e.g: 192.168.1.1-192.255.255.255 (Start and End are variable and will be defined by the user) and for second one, i have a friend that he is Python programmer, he made a IP Parser that it can support large txt files (1TB) and it can parse all of them under 10min and it also supports low-end PCs that have 1 GB RAM!

      The list that his program parses are:
      #1765497 192.168.1.1 8082 #1765496 192.168.1.1 8084 #1965493 192.168.1.1 8089 #9565495 192.168.1.1 8086 #2565492 192.168.1.1 8081 and it converts very very fast to this:
      192.168.1.1:8082 192.168.1.1:8084 192.168.1.1:8089 192.168.1.1:8086 192.168.1.1:8081 I wonder how to do this via AutoIt, if you can help me in this way, i will be happy✌❤

      Thanks for your helps.

      FIRST_QUESTION_TEST.au3
    • By Alok_Arora
      Hello Sir, 
      While searching for the solution to my problem, I have just gone through some of your old post. 
      My name is Alok Arora 
      Email id is *snip*
      I am writing a code to store logs for each script I run. This I did by using FileWriteLog function and it is successfully storing logs in txt file. 
      Now, I have to work on to read the log file and if any script has been mistakenly clicked twice in a day script will pop up a message that task already done for the day by verifying entries in the log file. 
      I have the logic for it.. 
      I mean a variable will read the log file and will search for the entry and will perform action if entry found or not. 
      I believe for reading the file I can write a code like
      $i =fileread("logfile.txt")
      Now I am stuck on how to compare the log entry in log file. 
      Can you please help me in this? 
×
×
  • Create New...