Detect almost duplicate log entries

[steelsking 0]

Hi,

I have the following log file.

2011-10-18 07:38:39 Local7.Notice 172.17.164.134 1671: Oct 18 07:45:40 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:40 Local7.Error 172.17.164.134 1672: Oct 18 07:45:40 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:41 Local7.Notice 172.17.164.134 1673: Oct 18 07:45:41 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:41 Local7.Error 172.17.164.134 1674: Oct 18 07:45:42 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:43 Local7.Notice 172.17.164.134 1675: Oct 18 07:45:44 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:47 Local7.Error 172.17.164.134 1676: Oct 18 07:45:48 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:48 Local7.Notice 172.17.164.134 1677: Oct 18 07:45:49 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:49 Local7.Error 172.17.164.134 1678: Oct 18 07:45:49 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:49 Local7.Notice 172.17.164.134 1679: Oct 18 07:45:50 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

What I need to do is to detect if any particular IP address is having an up and down status based on the logs files, within a stipulated time.

meaning to detect if within a space of 5 mins, if there are X numbers of ups and down, to sent an alert.

I have tried the following. Not sure if it is the tried easy logical method. I couldnt proceed further as I got stuck. Hope someone can guide me along.

Thanks.

expandcollapse popup

#include <file.au3>
#include <Array.au3>
#include <Date.au3>
Global $YYMMDD = StringMid(@YEAR & @MON & @MDAY, 3)
Global $inifile = @ScriptDir & "\settings.ini"
Global $Search = IniReadSection($inifile, "Search")
Global $Temp = @ScriptDir & "\tempdb" & "_" & $YYMMDD & ".txt"
Dim $aRecords, $aTemp
If Not _FileReadToArray("error.log", $aRecords) Then
MsgBox(4096, "Error", "Error reading log to Array error: " & @error)
Exit
EndIf
If Not FileExists(@ScriptDir & "\tempDB" & "_" & $YYMMDD & ".txt") Then
FileInstall("temp.txt", @ScriptDir & "\tempDB" & "_" & $YYMMDD & ".txt", 1)
IniWrite($inifile, "config", "LastEOF", "0")
;Exit
EndIf
For $j = 1 To $Search[0][0]
For $x = 1 To $aRecords[0]
  $aParts = StringSplit($aRecords[$x], @TAB)
  $result = StringInStr($aParts[4], $Search[$j][1])
  If $result = 0 Then
   $DT = StringReplace($aParts[1], "-", "/")
   $Combine = $DT & @TAB & $aParts[3]
   If Not _FileReadToArray($Temp, $aTemp) Then
    MsgBox(4096, "Error", "Error reading tempDB to Array error: " & @error)
    Exit
   EndIf
   If (_ArraySearch($aTemp, $Combine) = -1) Then
    FileWriteLine($Temp, $Combine)
   Else
    For $i = 1 To $aTemp[0] Step 1
    $aParts2 = StringSplit($aTemp[$i], @TAB)
    If IsArray($aParts2) And ($aParts2[2] = $aParts[3]) Then
     MsgBox(0, "", "Found.")
     FileWriteLine($Temp, $Combine)
    EndIf
    Next
   EndIf
  EndIf
Next
Next
Exit

ini file

[Config]

FileLocName="error.log"

[search]

1=changed state to up

2=changed state to down

[settings]

Frequency=10

Threshold=5

[Melba23 3,798]

steelsking,

I have made the assumption that the state changes are always the reverse of the previous - i.e the order is always up-down-up-down-up-down.... If that is the case then this seems to work:

expandcollapse popup

#include <File.au3>
#Include <Date.au3>
#include <Array.au3>
 
Global $sIni = "settings.ini"
Global $aLines
 
; Read log file
$sLog = IniRead($sIni, "Config", "FileLocName", "")
_FileReadToArray($sLog, $aLines)
 
; Create a suitably sized 2D array
Global $aSelection[$aLines[0] + 1][2] = [[$aLines[0], 0]]
$sStartDTG = ""
 
; Fill 2D array with time difference and actual DTG
For $i = 1 To $aSelection[0][0]
    If StringInStr($aLines[$i], "%LINEPROTO-5-UPDOWN") Then ; I have assumed that we ned only look for lines containing this text <<<<<<
        If $sStartDTG = "" Then
            $sStartDTG = StringLeft($aLines[$i], 19)
            $aSelection[$i][0] = StringFormat("%08i", 0)
            $aSelection[$i][1] = $sStartDTG
        Else
            $sDTG = StringLeft($aLines[$i], 19)
            $aSelection[$i][0] = StringFormat("%08i", _DateDiff("s", $sStartDTG, $sDTG))
            $aSelection[$i][1] = $sDTG
        EndIf
    EndIf
Next
 
; Remove all blank lines from the 2D array
For $i = $aSelection[0][0] To 1 Step -1
    If $aSelection[$i][0] = "" Then
        _ArrayDelete($aSelection, $i)
        $aSelection[0][0] -= 1
    EndIf
Next
 
; Read the required values from the ini file
$iThreshold = Number(IniRead($sIni, "Settings", "Threshold", 0)) ; assumed seconds for checking <<<<<<<<<<<<
$iEventFreq = Number(IniRead($sIni, "Settings", "Frequency", 0)) ; set to 2 for checking <<<<<<<<<<<<<<<<<<<< You need to alter your ini file
 
; Now run through the array checking the Frequency of events in every Threshold period
For $i = 1 To $aSelection[0][0]
    $iStartValue = $aSelection[$i][0]
    For $j = $i + 1 To $aSelection[0][0]
        ; If we reach the Threshold period
        If $aSelection[$j][0] - $aSelection[$i][0] > $iThreshold Then
            ; Count the events and see if they meet/exceed the required frequency
            $iFreq = $j - $i
            If $iFreq >= $iEventFreq Then
                ; If so then announce it
                ConsoleWrite($iFreq & " events between " & $aSelection[$i][1] & " and " & $aSelection[$j][1] & @CRLF)
            EndIf
            ExitLoop
        EndIf
    Next
Next

Note I have changed the Frequency value in the ini file to get sensible returns from the short log file section you posted. You would have to either multiply $iThreshold by 60 or change the _DateDiff Type parameter to get minutes.

Does it help?

M23

[steelsking 0]

hi melba23,

thanks alot!!! it works!!!

sorry for the delay, was testing it out.

Thanks!