Parse time from each line of log (text file)

Vivaed · August 22, 2016

I have another AutoIT script making a Log file

Sample of Log file:

2016/08/22 12:44:18 > Process: [RUNNING] [ACTIVE]
2016/08/22 12:48:35 > Process: [WAS NOT RUNNING]
2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 14:01:10 > Process: [WAS NOT RUNNING]

I am looping through the Log file for the word "FAILED"

I then want to get all lines that have "FAILED" and get their TIME

My Current code to get this far:

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    ConsoleWrite($filteredLine & @CRLF)     ; this gets me the results I want sans the time parse
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
EndIf

For this part:

If StringInStr($filteredLine,$search) Then
  ConsoleWrite($filteredLine & @CRLF)   ; this gets me the results I want sans the time parse
EndIf

OUTPUT: 2016/08/22 13:40:00 > Process: [FAILED]

I dont understand how I read the time in that output?

I have tried _DateTimeFormat - Dont think this applies

Tried _DateDiff - I dont have a the date yet so this doesnt work

Would love if someone could tell me if I am thinking is the wrong direction and possibly lead me down the correct path to light side of the force

JLogan3o13 · August 22, 2016

@Vivaed welcome to the forum. Does the output need to be in date format, or will a string work for your needs? Something like this works, based on the Log file sample you provided (which I expanded a bit).

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        If StringInStr($sLine, "[FAILED]") Then
            $aTemp = StringSplit($sLine, ">")
            FileWriteLine(@DesktopDir & "\Output.txt", $aTemp[1] & " - Failed")
        EndIf
    Next

It returns the following:

2016/08/21 13:40:00  - Failed
2016/08/22 13:40:00  - Failed
2016/08/23 13:40:00  - Failed
2016/08/24 13:40:00  - Failed
2016/08/25 13:40:00  - Failed
2016/08/26 13:40:00  - Failed
2016/08/27 13:40:00  - Failed
2016/08/28 13:40:00  - Failed

Vivaed · August 22, 2016

Thanks for the Prompt response!!! @JLogan3o13

So I might not have explained it well lol

2016/08/22 13:40:00 > Process: [FAILED]
2016/08/22 13:41:00 > Process: [FAILED]
2016/08/22 13:48:00 > Process: [FAILED]
2016/08/22 13:51:00 > Process: [FAILED]
2016/08/22 14:00:00 > Process: [FAILED]

Above is my actual log file

Lets say my current time was 2016/08/22 14:01:00

I want to look at my log file and compare the log files time stamps to _NowCalc(), I will use _DateDiff() (But that comes later)

Essentially what I am trying to do:

If the log file has >= 1 FAILED log line(s) in the past 15 mins then send email with these errors inside

I dont know how I parse for the date in the log file, I have tried Googled all I know #StillLearning

JLogan3o13 · August 22, 2016

Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time:

#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next

If you do a consolewrite, you will see that the output is:

2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

Vivaed · August 22, 2016

13 minutes ago, JLogan3o13 said:
Did you look at the code that I posted? Your first post was indeed a bit unclear, but the code should be roughly the same: you read the file to an array and parse through it. You then stringsplit the lines to give you your time:
#include <Array.au3>
#include <File.au3>

Local $aFile = FileReadToArray(@DesktopDir & "\Test.log")
Local $aTemp

    For $sLine In $aFile
        $aTemp = StringSplit($sLine, ">")
        $sTime = $aTemp[1]
    Next
If you do a consolewrite, you will see that the output is:
2016/08/22 13:40:00 
2016/08/22 13:41:00 
2016/08/22 13:48:00 
2016/08/22 13:51:00 
2016/08/22 14:00:00

Interesting, I will have to learn more about this StringSplit()

Thanks for the insight, I hope it leads me down a better path lol

jchd · August 22, 2016

If your log file is significantly large, you'd probably better use a database to store log entries. SQLite is a very good candidate and is simple and efficient to setup and use.

kylomas · August 23, 2016

Vivaed,

Another way to do this...

#include <array.au3>
#include <date.au3>

Local $aFailed = StringRegExp(FileRead(@ScriptDir & '\log.test.txt'), '(?i).*?failed.*', 3)

For $i = 0 To UBound($aFailed) - 1
    If _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) <= 15 Then
        ;
        ; This is where you would do your email routine...I just print the entry to console...
        ;
        ConsoleWrite('! ' & $aFailed[$i] & ' ' & _DateDiff('n', StringLeft($aFailed[$i], 19), _NowCalc()) & ' minutes ago...' & @CRLF)
        ;
    EndIf
Next

How large is your log file?

How is the file(s) managed? (hourly, daily, rolling, etc)

How often do you want to scan the file(s)?

kylomas

edit: file used for testing log.test.txt

Edited August 23, 2016 by kylomas

Vivaed · August 23, 2016

@kylomas Thanks for the reply!

Another application checks to see if a process is running every minute and logs if [PASS] or [FAILED]

Then this app checks every 15 mins. It looks for if there were any fails in the last 15 mins, if so then email support team.

@jchd The log files are short for now, I could see them potentially getting long, but I would just delete them until another solution was created.

Vivaed · August 23, 2016

Okay, I have the fist part working (Read log get FAILED lines and make new file with all FAILED lines):

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>


Global $file_output = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 1)
Global $file_output_wipe = FileOpen("C:\WB Resources\" & @ComputerName & "-FAILED-output.txt", 2)
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

$search = "FAILED"

Global $aFile = FileReadToArray($fileOutput)
Global $sTemp = ''

ConsoleWrite(_FileCountLines($fileLog) & @CRLF)
ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents, $search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog, $i)
                If StringInStr($filteredLine, $search) Then
                    FileWrite($fileOutput,$filteredLine & @CRLF)
                EndIf
            Else
                ConsoleWrite($search & " not found!" & @CRLF)
                FileWrite($file_output_wipe, $search & " not found!" & @CRLF)
            EndIf
        Next
    EndIf

EndIf

;~ ----------------------------------------------BROKEN
;~ For $sLine In $aFile
;~  $aTemp = StringSplit($sLine, ">")
;~  $sTime = $aTemp[1]
;~  ConsoleWrite($sTime & @CRLF)
;~  FileWrite($file_output_wipe,$sTime & @CRLF)
;~  Local $theDiff = _DateDiff('n', $sTime, _NowCalc())
;~ Next
;~ If $theDiff <= 15 Then
;~  ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
;~  FileWrite($file_output,$filteredLine & @CRLF)
;~ Else
;~ EndIf
;~ ----------------------------------------------BROKEN

FileClose($file_output)
Exit

But as you can see the bottom FOR statement doesn't work ( when un-commented lol )

Console Output:

(65) : ==> Variable must be of type "Object".:
For $sLine In $aFile
For $sLine In $aFile^ ERROR
->09:45:23 AutoIt3.exe ended.rc:1
+>09:45:23 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 0.8435

I am missing something very obvious aren't I?

JLogan3o13 · August 23, 2016

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?

Vivaed · August 23, 2016

1 minute ago, JLogan3o13 said:

After your FileReadToArray line, put an _ArrayDisplay($aFile). Does it show up as you would expect it to?

This?

Local $aFile = FileReadToArray($fileOutput)
ConsoleWrite(_ArrayDisplay($aFile))

JLogan3o13 · August 23, 2016

@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this:

Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)

Does it show you the array you're expecting to see?

Vivaed · August 23, 2016

1 minute ago, JLogan3o13 said:
@Vivaed you need to look these functions up in the help file so you know HOW things work. Try this:
Local $aFile = FileReadToArray($fileOutput)
_ArrayDisplay($aFile)
Does it show you the array you're expecting to see?

@JLogan3o13 No it does not, looks like it passes right over it

I will do more research on FileReadArray

JLogan3o13 · August 23, 2016

So, that means it is not finding the file at that location, most likely. You could do something like this to confirm:

If FileExists("path to file") Then 
   FileReadToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

Edited August 23, 2016 by JLogan3o13

Vivaed · August 23, 2016

3 minutes ago, JLogan3o13 said:
So, that means it is not finding the file at that location, most likely. You could do something like this to confirm:
If FileExists("path to file") Then 
   FileReadyToArray("path to file")
Else
   ConsoleWrite("Couldn't Find File" & @CRLF)
EndIf

Its weird but I just put the snippet of code in the original IF statement:

If FileExists($fileLog) Then
    $contents = FileRead($fileLog)
    If @error Then
        MsgBox(0, 'File Error', $fileLog & ' could not be read.')
    Else
        For $i = 1 To _FileCountLines($fileLog)
            $result = StringInStr($contents,$search)
            If $result >= 1 Then
                $filteredLine = FileReadLine($fileLog,$i)
                If StringInStr($filteredLine,$search) Then
                    For $sLine In $aFile
                        $aTemp = StringSplit($sLine, ">")
                        $sTime = $aTemp[1]
                        ConsoleWrite($sTime & @CRLF)
;~                      FileWrite($file_output,$filteredLine & @CRLF)

                        Local $theDiff =  _DateDiff('n', $sTime, _NowCalc())


                    Next
                        If $theDiff < 15 Then
                            ConsoleWrite($filteredLine & " " & $theDiff & " " & @CRLF)
                        Else

                        EndIf
                EndIf
            Else
                ConsoleWrite( $search & " not found!" & @CRLF)
                FileWrite($file_output,$search & " not found!" & @CRLF)
            EndIf
    Next
    EndIf
FileClose($file_output)
EndIf

AND ITS OUTPUTTING????? (Kind of)

37
NOW CALC TIME 
2016/08/23 12:06:22
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
2016/08/23 09:01:23 
2016/08/23 09:09:28 
2016/08/23 09:10:36 
2016/08/23 09:11:38 
2016/08/23 09:11:41 
2016/08/23 09:12:43 
2016/08/23 09:14:45 
2016/08/23 09:29:45 
2016/08/23 09:29:50 
2016/08/23 09:29:55 
2016/08/23 09:30:33 
+>12:06:22 AutoIt3.exe ended.rc:0
+>12:06:22 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.8868

I think this needs to be after FileRead??? I dont really know...

But this gets me a step closer lol

mikell · August 23, 2016

2 hours ago, JLogan3o13 said:

If FileExists("path to file") Then FileReadyToArray("path to file")

Very cute

JLogan3o13 · August 23, 2016

Simple mistype on my part; now fixed.

mikell · August 23, 2016

Of course it was... but it was a funny mistype

kylomas · August 23, 2016

Vivaed,

This seems to be what you are trying to do. I would strongly encourage you to take JL's advice and understand what each function does by reading the Help file.

#include <File.au3>
#include <Array.au3>
#NoTrayIcon
#include <AutoItConstants.au3>
#include <TrayConstants.au3>
#include <FileConstants.au3>
#include <Date.au3>
#include <WinAPIFiles.au3>

; name of log file
Global $fileLog = "C:\WB Resources\" & @ComputerName & "-WBLog.txt"
; name of output file
Global $fileOutput = "C:\WB Resources\" & @ComputerName & "-FAILED-output.txt"

; open output file for write..erase previous content
Global $file_output = FileOpen($fileOutput, 2)

; check that file opened
If $file_output = -1 Then Exit MsgBox(17, 'Output file failed to open', 'File = ' & $fileOutput)

$search = "FAILED"

; read log file to array
Global $aFile = FileReadToArray($fileLog)
If @error Then Exit MsgBox(17, 'Log file read to array failed', 'File = ' & $fileLog)

ConsoleWrite("NOW CALC TIME " & @CRLF & _NowCalc() & @CRLF)

; iterate through array of log entries lokkiing for a "failed" entry withini the last 15
; minutes.  If found writye it to the output file

For $i = 0 To UBound($aFile) - 1

    If StringInStr($aFile[$i], $search) > 0 Then
        If _DateDiff('n', StringLeft($aFile[$i], 19), _NowCalcDate()) <= 15 Then
            FileWrite($file_output, $aFile[$i])
            ConsoleWrite($aFile[$i] & @CRLF)
        EndIf
    EndIf

Next

FileClose($file_output)

kylomas

Edited August 23, 2016 by kylomas

Vivaed · August 23, 2016

@kylomas Thanks for the insight!

I have been reading ton about these functions.

This script is interesting, it outputs all of the FAILED found but doesn't filter them by only showing the ones from the last 15 mins. since you used _NowCalcDate(), but if I change it to just _NowCalc() I get the results I am looking for.

I am very grateful for the code comments, they help me understand what is what and why things are happening!!!

Sign In

Parse time from each line of log (text file)

Recommended Posts

Vivaed

JLogan3o13

Vivaed

JLogan3o13

Vivaed

jchd

kylomas

Vivaed

Vivaed

JLogan3o13

Vivaed

JLogan3o13

Vivaed

JLogan3o13

Vivaed

mikell

JLogan3o13

mikell

kylomas

Vivaed

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content

Free style DateTimeFormat

Get command line parameters Utility v1.0.3.0 - Retrieve Switches of command line Tools - Update of 2025-09-22

_GOLLOG UDF - LOGs made easy

Gollog UDF

ListView sorting on date

Browse

AutoIt Resources

Release

Beta