Jump to content

Detect almost duplicate log entries


Recommended Posts

Hi,

I have the following log file.

2011-10-18 07:38:39 Local7.Notice 172.17.164.134 1671: Oct 18 07:45:40 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:40 Local7.Error 172.17.164.134 1672: Oct 18 07:45:40 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:41 Local7.Notice 172.17.164.134 1673: Oct 18 07:45:41 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:41 Local7.Error 172.17.164.134 1674: Oct 18 07:45:42 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:43 Local7.Notice 172.17.164.134 1675: Oct 18 07:45:44 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:47 Local7.Error 172.17.164.134 1676: Oct 18 07:45:48 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:48 Local7.Notice 172.17.164.134 1677: Oct 18 07:45:49 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

2011-10-18 07:38:49 Local7.Error 172.17.164.134 1678: Oct 18 07:45:49 SIN: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up

2011-10-18 07:38:49 Local7.Notice 172.17.164.134 1679: Oct 18 07:45:50 SIN: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

What I need to do is to detect if any particular IP address is having an up and down status based on the logs files, within a stipulated time.

meaning to detect if within a space of 5 mins, if there are X numbers of ups and down, to sent an alert.

I have tried the following. Not sure if it is the tried easy logical method. I couldnt proceed further as I got stuck. Hope someone can guide me along.

Thanks.

#include <file.au3>
#include <Array.au3>
#include <Date.au3>
Global $YYMMDD = StringMid(@YEAR & @MON & @MDAY, 3)
Global $inifile = @ScriptDir & "\settings.ini"
Global $Search = IniReadSection($inifile, "Search")
Global $Temp = @ScriptDir & "\tempdb" & "_" & $YYMMDD & ".txt"
Dim $aRecords, $aTemp
If Not _FileReadToArray("error.log", $aRecords) Then
MsgBox(4096, "Error", "Error reading log to Array error: " & @error)
Exit
EndIf
If Not FileExists(@ScriptDir & "\tempDB" & "_" & $YYMMDD & ".txt") Then
FileInstall("temp.txt", @ScriptDir & "\tempDB" & "_" & $YYMMDD & ".txt", 1)
IniWrite($inifile, "config", "LastEOF", "0")
;Exit
EndIf
For $j = 1 To $Search[0][0]
For $x = 1 To $aRecords[0]
  $aParts = StringSplit($aRecords[$x], @TAB)
  $result = StringInStr($aParts[4], $Search[$j][1])
  If $result = 0 Then
   $DT = StringReplace($aParts[1], "-", "/")
   $Combine = $DT & @TAB & $aParts[3]
   If Not _FileReadToArray($Temp, $aTemp) Then
    MsgBox(4096, "Error", "Error reading tempDB to Array error: " & @error)
    Exit
   EndIf
   If (_ArraySearch($aTemp, $Combine) = -1) Then
    FileWriteLine($Temp, $Combine)
   Else
    For $i = 1 To $aTemp[0] Step 1
    $aParts2 = StringSplit($aTemp[$i], @TAB)
    If IsArray($aParts2) And ($aParts2[2] = $aParts[3]) Then
     MsgBox(0, "", "Found.")
     FileWriteLine($Temp, $Combine)
    EndIf
    Next
   EndIf
  EndIf
Next
Next
Exit

ini file

[Config]

FileLocName="error.log"

[search]

1=changed state to up

2=changed state to down

[settings]

Frequency=10

Threshold=5

Link to comment
Share on other sites

  • Moderators

steelsking,

I have made the assumption that the state changes are always the reverse of the previous - i.e the order is always up-down-up-down-up-down.... If that is the case then this seems to work: ;)

#include <File.au3>
#Include <Date.au3>
#include <Array.au3>
 
Global $sIni = "settings.ini"
Global $aLines
 
; Read log file
$sLog = IniRead($sIni, "Config", "FileLocName", "")
_FileReadToArray($sLog, $aLines)
 
; Create a suitably sized 2D array
Global $aSelection[$aLines[0] + 1][2] = [[$aLines[0], 0]]
$sStartDTG = ""
 
; Fill 2D array with time difference and actual DTG
For $i = 1 To $aSelection[0][0]
    If StringInStr($aLines[$i], "%LINEPROTO-5-UPDOWN") Then ; I have assumed that we ned only look for lines containing this text <<<<<<
        If $sStartDTG = "" Then
            $sStartDTG = StringLeft($aLines[$i], 19)
            $aSelection[$i][0] = StringFormat("%08i", 0)
            $aSelection[$i][1] = $sStartDTG
        Else
            $sDTG = StringLeft($aLines[$i], 19)
            $aSelection[$i][0] = StringFormat("%08i", _DateDiff("s", $sStartDTG, $sDTG))
            $aSelection[$i][1] = $sDTG
        EndIf
    EndIf
Next
 
; Remove all blank lines from the 2D array
For $i = $aSelection[0][0] To 1 Step -1
    If $aSelection[$i][0] = "" Then
        _ArrayDelete($aSelection, $i)
        $aSelection[0][0] -= 1
    EndIf
Next
 
; Read the required values from the ini file
$iThreshold = Number(IniRead($sIni, "Settings", "Threshold", 0)) ; assumed seconds for checking <<<<<<<<<<<<
$iEventFreq = Number(IniRead($sIni, "Settings", "Frequency", 0)) ; set to 2 for checking <<<<<<<<<<<<<<<<<<<< You need to alter your ini file
 
; Now run through the array checking the Frequency of events in every Threshold period
For $i = 1 To $aSelection[0][0]
    $iStartValue = $aSelection[$i][0]
    For $j = $i + 1 To $aSelection[0][0]
        ; If we reach the Threshold period
        If $aSelection[$j][0] - $aSelection[$i][0] > $iThreshold Then
            ; Count the events and see if they meet/exceed the required frequency
            $iFreq = $j - $i
            If $iFreq >= $iEventFreq Then
                ; If so then announce it
                ConsoleWrite($iFreq & " events between " & $aSelection[$i][1] & " and " & $aSelection[$j][1] & @CRLF)
            EndIf
            ExitLoop
        EndIf
    Next
Next

Note I have changed the Frequency value in the ini file to get sensible returns from the short log file section you posted. You would have to either multiply $iThreshold by 60 or change the _DateDiff Type parameter to get minutes. :)

Does it help? :graduated:

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...