Vivaed

Parse time from each line of log (text file)

21 posts in this topic

I have another AutoIT script making a Log file 

Sample of Log file:

2016/08/22 12:44:18 > Process: [RUNNING] [ACTIVE]
2016/08/22 12:48:35 > Process: [WAS NOT RUNNING]
2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 14:01:10 > Process: [WAS NOT RUNNING]

I am looping through the Log file for  the word "FAILED"

I then want to get all lines that have "FAILED" and get their TIME

 

My Current code to get this far:

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    ConsoleWrite($filteredLine & @CRLF)     ; this gets me the results I want sans the time parse
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
EndIf

For this part:

If StringInStr($filteredLine,$search) Then
  ConsoleWrite($filteredLine & @CRLF)   ; this gets me the results I want sans the time parse
EndIf

OUTPUT: 2016/08/22 13:40:00 > Process: [FAILED]

I dont understand how I read the time in that output?

 

I have tried _DateTimeFormat - Dont think this applies 

Tried _DateDiff - I dont have a the date yet so this doesnt work

 

Would love if someone could tell me if I am thinking is the wrong direction and possibly lead me down the correct path to light side of the force :) 

 

Share this post


Link to post
Share on other sites

@Vivaed welcome to the forum. Does the output need to be in date format, or will a string work for your needs? Something like this works, based on the Log file sample you provided (which I expanded a bit).

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        If StringInStr($sLine, "[FAILED]") Then
            $aTemp = StringSplit($sLine, ">")
            FileWriteLine(@DesktopDir & "\Output.txt", $aTemp[1] & " - Failed")
        EndIf
    Next

It returns the following:

2016/08/21 13:40:00  - Failed
2016/08/22 13:40:00  - Failed
2016/08/23 13:40:00  - Failed
2016/08/24 13:40:00  - Failed
2016/08/25 13:40:00  - Failed
2016/08/26 13:40:00  - Failed
2016/08/27 13:40:00  - Failed
2016/08/28 13:40:00  - Failed

 


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

Thanks for the Prompt response!!! @JLogan3o13

 

So I might not have explained it well lol

2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 13:41:00 > Process: [FAILED]
2016/08/22 13:48:00 > Process: [FAILED]
2016/08/22 13:51:00 > Process: [FAILED]
2016/08/22 14:00:00 > Process: [FAILED]

Above is my actual log file

 

Lets say my current time was  2016/08/22 14:01:00

I want to look at my log file and compare the log files time stamps to _NowCalc(), I will use _DateDiff() (But that comes later)

 

Essentially what I am trying to do:

If the log file has >= 1 FAILED log line(s) in the past 15 mins then send email with these errors inside


I dont know how I parse for the date in the log file, I have tried Googled all I know #StillLearning

Share this post


Link to post
Share on other sites

Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time:

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next

If you do a consolewrite, you will see that the output is:

2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

 


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
13 minutes ago, JLogan3o13 said:

Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time:

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next

If you do a consolewrite, you will see that the output is:

2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

 

Interesting, I will have to learn more about this StringSplit() 

 

Thanks for the insight, I hope it leads me down a better path lol

Share this post


Link to post
Share on other sites

If your log file is significantly large, you'd probably better use a database to store log entries. SQLite is a very good candidate and is simple and efficient to setup and use.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Vivaed,

Another way to do this...

#include <array.au3>
#include <date.au3>

Local $aFailed = StringRegExp(FileRead(@ScriptDir & '\log.test.txt'), '(?i).*?failed.*', 3)

For $i = 0 To UBound($aFailed) - 1
    If _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) <= 15 Then
        ;
        ; This is where you would do your email routine...I just print the entry to console...
        ;
        ConsoleWrite('! ' & $aFailed[$i] & ' ' & _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) & ' minutes ago...' & @CRLF)
        ;
    EndIf
Next

How large is your log file?

How is the file(s) managed? (hourly, daily, rolling, etc)

How often do you want to scan the file(s)?

kylomas

edit: file used for testing log.test.txt

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

@kylomas Thanks for the reply!

Another application checks to see if a process is running every minute and logs if [PASS] or [FAILED]

Then this app checks every 15 mins. It looks for if there were any fails in the last 15 mins, if so then email support team.

 

@jchd The log files are short for now, I could see them potentially getting long, but I would just delete them until another solution was created.

Share this post


Link to post
Share on other sites

Okay, I have the fist part working (Read log get FAILED lines and make new file with all FAILED lines):

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>


Global $file_output = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 1)
Global $file_output_wipe = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 2)
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

$search = "FAILED"

Global $aFile = FileReadToArray($fileOutput)
Global $sTemp = ''

ConsoleWrite(_FileCountLines($fileLog) & @CRLF)
ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents, $search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog, $i)
                If StringInStr($filteredLine, $search) Then
                    FileWrite($fileOutput,$filteredLine & @CRLF)
                EndIf
            Else
                ConsoleWrite($search & " not found!" & @CRLF)
                FileWrite($file_output_wipe, $search & " not found!" & @CRLF)
            EndIf
        Next
    EndIf

EndIf

;~ ----------------------------------------------BROKEN
;~ For $sLine In $aFile
;~  $aTemp = StringSplit($sLine, ">")
;~  $sTime = $aTemp[1]
;~  ConsoleWrite($sTime & @CRLF)
;~  FileWrite($file_output_wipe,$sTime & @CRLF)
;~  Local $theDiff = _DateDiff('n', $sTime, _NowCalc())
;~ Next
;~ If $theDiff <= 15 Then
;~  ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
;~  FileWrite($file_output,$filteredLine & @CRLF)
;~ Else
;~ EndIf
;~ ----------------------------------------------BROKEN

FileClose($file_output)
Exit


But as you can see the bottom FOR statement doesn't work ( when un-commented lol )

Console Output:

(65) : ==> Variable must be of type "Object".:
For $sLine In $aFile
For $sLine In $aFile^ ERROR
->09:45:23 AutoIt3.exe ended.rc:1
+>09:45:23 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 0.8435

I am missing something very obvious aren't I?

Share this post


Link to post
Share on other sites

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
1 minute ago, JLogan3o13 said:

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?

This?

Local $aFile = FileReadToArray($fileOutput)
ConsoleWrite(_ArrayDisplay($aFile))

Share this post


Link to post
Share on other sites

@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this:

Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)

Does it show you the array you're expecting to see?


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
1 minute ago, JLogan3o13 said:

@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this:

Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)

Does it show you the array you're expecting to see?

@JLogan3o13 No it does not, looks like it passes right over it

 

I will do more research on FileReadArray

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

So, that means it is not finding the file at that location, most likely. You could do something like this to confirm:

If FileExists("path to file") Then 
   FileReadToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

 

Edited by JLogan3o13

√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
3 minutes ago, JLogan3o13 said:

So, that means it is not finding the file at that location, most likely. You could do something like this to confirm:

If FileExists("path to file") Then 
   FileReadyToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

 

Its weird but I just put the snippet of code in the original IF statement:
 

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    For $sLine In $aFile
                        $aTemp = StringSplit($sLine, ">")
                        $sTime = $aTemp[1]
                        ConsoleWrite($sTime & @CRLF)
;~                      FileWrite($file_output,$filteredLine & @CRLF)

                        Local $theDiff =  _DateDiff('n', $sTime, _NowCalc())


                    Next
                        If $theDiff < 15 Then
                            ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
                        Else

                        EndIf
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
                FileWrite($file_output,$search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
FileClose($file_output)
EndIf

AND ITS OUTPUTTING????? (Kind of)

37
NOW CALC TIME 
2016/08/23 12:06:22
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
+>12:06:22 AutoIt3.exe ended.rc:0
+>12:06:22 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.8868

I think this needs to be after FileRead??? I dont really know...

But this gets me a step closer lol
 

Share this post


Link to post
Share on other sites
2 hours ago, JLogan3o13 said:

If FileExists("path to file") Then    FileReadyToArray("path to file")

Very cute  :)

Share this post


Link to post
Share on other sites

Simple mistype on my part; now fixed.


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

Of course it was... but it was a funny mistype   :)

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

Vivaed,

This seems to be what you are trying to do.  I would strongly encourage you to take JL's advice and understand what each function does by reading the Help file.

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>

; name of log file
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
; name of output file
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

; open output file for write..erase previous content
Global $file_output = FileOpen($fileOutput, 2)

; check that file opened
If $file_output = -1 Then Exit MsgBox(17, 'Output file failed to open', 'File = ' & $fileOutput)

$search = "FAILED"

; read log file to array
Global $aFile = FileReadToArray($fileLog)
If @error Then Exit MsgBox(17, 'Log file read to array failed', 'File = ' & $fileLog)

ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

; iterate through array of log entries lokkiing for a "failed" entry withini the last 15
; minutes.  If found writye it to the output file

For $i = 0 To UBound($aFile) - 1

    If StringInStr($aFile[$i], $search) > 0 Then
        If _DateDiff('n', StringLeft($aFile[$i], 19), _NowCalcDate()) <= 15 Then
            FileWrite($file_output, $aFile[$i])
            ConsoleWrite($aFile[$i] & @CRLF)
        EndIf
    EndIf

Next

FileClose($file_output)

kylomas

Edited by kylomas
2 people like this

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

@kylomas Thanks for the insight!

I have been reading ton about these functions.

This script is interesting, it outputs all of the FAILED found but doesn't filter them by only showing the ones from the last 15 mins. since you used _NowCalcDate(), but if I change it to just _NowCalc() I get the results I am looking for.

I am very grateful for the code comments, they help me understand what is what and why things are happening!!!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • Shedunn
      By Shedunn
      I've been looking through some different functions(not sure if that's the right word) to get the Date and time and I can't find one that doesn't include "/ " or ":" in what is returned.

      I'm trying to create a file name with this format "QuickConfig_MMDDYYYY_HHMM"  (doesn't matter if 24 or 12hr format)
      #include <Date.au3> Func _SaveConfig MouseClick('primary',155, 46, 1, 0) ;clicks save as WinWait('Save bluePRINT Configuration As...','', 4) MouseClick('primary',632, 47, 1, 0) ;clicks the file path bar $Documents = @MyDocumentsDir $FilePath = $Documents & '\BP3 Configs' Send($FilePath) Send('{ENTER}') MouseClick('primary',166, 580, 1, 0) ;clicks file name bar $FileName = 'QuickConfig_' & _NowDate & '_' & _NowTime Send($FileName) Send('{ENTER}') EndFunc  
       
      I'm wondering if there is a function somewhere to do this?
    • ur
      By ur
      I have date in the string format as "DD-MM-YYYY".
       
      I need to get yesterday's date from it.
      I tried converting this from _DateTimeFormat but not working.
      Is there any direct UDF available to get this.?
    • ur
      By ur
      In some systems the date is displayed in the Taskbar as
      10-01-2018
      And in some systems it is like, 1/10/2018
      And in some it is, 10-Jan-2018
      And in different formats.
       
      I wrote my code to convert 1/10/2018 to 10-01-2018
      Func TodaysDate() $NewDate = _DateTimeFormat(_NowCalcDate(),1) $Array = StringSplit( $NewDate , ',' ) _ArrayDelete($Array, 0) _ArrayDelete($Array, 0) $Array1 = StringSplit($Array[0],' ') RemoveEmptyArrayLines($Array1) ;Will return the present day's date with format dd-mmm-yyyy ;$Date = StringStripWS($Array1[2]&"-"&StringLeft($Array1[1], 3)&"-"&$Array[1], $STR_STRIPALL) $Date = StringStripWS($Array1[2]&"-"&changeDateformat(StringLeft($Array1[1], 3))&"-"&$Array[1], $STR_STRIPALL) return $Date EndFunc Func RemoveEmptyArrayLines(ByRef $arrLines) $intCount = 1 While $intCount < UBound($arrLines) $arrLines[$intCount] = StringStripWS($arrLines[$intCount],$STR_STRIPLEADING + $STR_STRIPTRAILING) If StringLen($arrLines[$intCount])=0 Then _ArrayDelete($arrLines, $intCount) $intCount = $intCount - 1 EndIf $intCount = $intCount + 1 WEnd EndFunc ;To convert mmm to mm format Func changeDateformat($sText) Local $sMsg = StringStripWS($sText, $STR_STRIPALL) Switch $sMsg Case "Jan" $sMsg = "01" Case "Feb" $sMsg = "02" Case "Mar" $sMsg = "03" Case "Apr" $sMsg = "04" Case "May" $sMsg = "05" Case "Jun" $sMsg = "06" Case "Jul" $sMsg = "07" Case "Aug" $sMsg = "08" Case "Sep" $sMsg = "09" Case "Oct" $sMsg = "10" Case "Nov" $sMsg = "11" Case "Dec" $sMsg = "12" EndSwitch return $sMsg EndFunc But again it will work on machines only with 1/10/2018.
      Is there any direct function which will give only in the format 10-01-2018, whatever the system settings is.?
      I tried all the arguments for _DateTimeFormat, but showing machine dependent settings only.
      Google given these two links in AutoIT but these are also for specific formats only,like mine.
       
      Please suggest.
    • nacerbaaziz
      By nacerbaaziz
      hello
      please i need to link a progress bar with a time can you help me?
      e.g
      i want to set a progress bar for 10 sec
      am waiting for your answers
      thank you.