[Solved] How to record different string and count repeated string in a loop test?

[jackylee0908]

Hi @mikell,

Here I attach full logs and filtered logs which by your code and batch script for your reference.

cycle_log_448.zip

bmc_sel_counter_1.log

filter.log

Don't know why but your code has extra 1 count for some strings, below is the compared result between two filtered logs for your reference.

For example, for the string which be counted as 449 by your code, actually it should be counted as 448, and the "OEM System boot event | Asserted" should be counted with 15 but by your code it has been counted with 16, as well as "Memory #0xf9 | Correctable ECC | Asserted" string...etc..

And, I found that below string could not be listed into filter.log.

Quote

   a | Pre-Init Time-stamp   | Slot/Connector MateCONT PRSNT | Device Installed | Asserted

Also, is it possible to let code detect like below string which is with different format(without date and time) but still be listed in filter.log correctly, for example on below string, due to it does not have date and time recorded, so just list it with "OEM record e0 | cc30312e30394a4d4233000000" into filter.log(without the "2"), also for array search.

Quote

   2 | OEM record e0 | cc30312e30394a4d4233000000

BTW, now I understood how to sort the string, just by _ArraySort.

Thanks.

Edited July 2, 2019 by jackylee0908

[Malkey]

This method uses the Scripting.Dictionary object.

expandcollapse popup

#include <Array.au3>
#include <File.au3>

Local $Filteredlog = "Filtered.log"

;#cs ; If $Filteredlog file is to be continueously run, then remove the comment character, ";", from before the block comments, "#cs", and, "#ce".
; First run - $Filteredlog is created for reference to future runs.
If FileExists($Filteredlog) Then FileDelete($Filteredlog) ; Deleting $Filteredlog file restarts the $Filteredlog data accumulation.

Local $sFilename = "BMC_event_1.log"
Global $aFilteredArray = _FilteredLog($sFilename, $Filteredlog)
_ArrayDisplay($aFilteredArray, "$aFilteredArray 1st run")
ConsoleWrite("Filtered.log :- " & @CRLF & FileRead($Filteredlog) & @CRLF & "---------------------------------" & @CRLF & @CRLF)
;#ce

Local $sFilename2 = "BMC_event_2.log" ; <-- Change log file name if necessary.
$aFilteredArray = _FilteredLog($sFilename2, $Filteredlog)
_ArrayDisplay($aFilteredArray, "$aFilteredArray 2nd run")
ConsoleWrite("Filtered.log :- " & @CRLF & FileRead($Filteredlog) & @CRLF & "---------------------------------" & @CRLF & @CRLF)


Func _FilteredLog($sFilename, $sFilteredFileName)
    Local $aArr, $key, $oDict = ObjCreate("Scripting.Dictionary")

    ; -------- $sFilteredFileName file ------------
    If FileExists($sFilteredFileName) Then ; If exist, $Filteredlog contents to be added to current log file's ($sFilename) filtered data.
        _FileReadToArray($sFilteredFileName, $aArr, 0, @TAB)
        ;_ArrayDisplay($aArr)
        For $i = 0 To UBound($aArr) - 1
            $key = $aArr[$i][1]
            $oDict.Item($key) = $aArr[$i][0]
        Next
    EndIf

    ; -------- $sFilename file ------------
    Local $aArray = StringSplit(StringRegExpReplace(FileRead($sFilename), "(?m)^([^\|]+\|\h*){3}|\v*\z", ""), @LF, 3) ; $STR_ENTIRESPLIT (1) = entire delimiter string is needed to mark the split + $STR_NOCOUNT (2) = disable the return count in the first element - effectively makes the array 0-based (must use UBound() to get the size of the array in this case).
    For $i = 0 To UBound($aArray) - 1
        $key = $aArray[$i]
        If $oDict.Exists($key) Then
            $oDict.Item($key) = $oDict.Item($key) + 1
        Else
            $oDict.Add($key, 1)
        EndIf
    Next

    ; ------- Create array from Scripting Dictionary --------------
    Local $Count = $oDict.Count ; This tells how many keys are in $oDict
    Local $Array[$Count][2]
    Local $p = 0
    For $vKeys In $oDict
        $Array[$p][1] = $vKeys
        $Array[$p][0] = $oDict.item($vKeys)
        $p += 1
    Next

    ; --------- Save $sFilteredFileName file -----------
    Local $hFiltered = FileOpen($sFilteredFileName, 10)
    FileWrite($hFiltered, _ArrayToString($Array, @TAB, -1, -1, @CRLF))
    FileClose($hFiltered)
    Return $Array
EndFunc   ;==>_FilteredLog

 

[jackylee0908]

Thanks, @Malkey, I have tried your code to filter two kinds of logs, first one is with 448 logs totally and 1000 logs for 2nd one, both logs are attached.

1_cycle_log.zip

1_Filtered.log

2_event.zip

2_Filtered.log

 

I updated your code to loop filter mass log files.

For $c = 1 To 1000
    Local $sFilename = @scriptdir & "\" & "event" & "\" & "BMC_event_" & $c & ".log"
    ;Local $sFilename = @scriptdir & "\" & "cycle_log" & "\" & $c & "\" & "bmc_sel_1.log"
    Global $aFilteredArray = _FilteredLog($sFilename, $Filteredlog)
    _ArraySort($aFilteredArray, 0, 0, 0, 1)
    ;_ArrayDisplay($aFilteredArray, "$aFilteredArray 1st run")
    ;ConsoleWrite("Filtered.log :- " & @CRLF & FileRead($Filteredlog) & @CRLF & "---------------------------------" & @CRLF & @CRLF)
    ;#ce
Next

 

For the 448 logs filter I found the date and time for some strings are be included in Filtered.log.

 

And for the 1000 logs, I found the line alignment is not aligned if the count has 4 digit(1000).

 

It is perfectly if you could help to update your code for above two issues, thanks a lot!

Edited July 2, 2019 by jackylee0908

[Malkey]

This appears to work  with the given data.

expandcollapse popup

#include <Array.au3>
#include <File.au3>

Local $Filteredlog = "FilteredEvent.log"
If FileExists($Filteredlog) Then FileDelete($Filteredlog)

For $c = 1 To 1000
    Local $sFilename = @ScriptDir & "\event\BMC_event_" & $c & ".log"
    $aFilteredArray = _FilteredLog($sFilename, $Filteredlog)
Next
_ArrayDisplay($aFilteredArray, $Filteredlog)

Local $Filteredlog = "FilteredCycle.log"
If FileExists($Filteredlog) Then FileDelete($Filteredlog)
For $c = 1 To 448
    Local $sFilename = @ScriptDir & "\cycle_log\" & $c & "\bmc_sel_1.log"
    $aFilteredArray = _FilteredLog($sFilename, $Filteredlog)
Next
_ArrayDisplay($aFilteredArray, $Filteredlog)


Func _FilteredLog($sFilename, $sFilteredFileName)
    Local $aArr, $key, $oDict = ObjCreate("Scripting.Dictionary")

    ; -------- $sFilteredFileName file ------------
    If FileExists($sFilteredFileName) Then ; If exist, $Filteredlog contents to be added to current log file's ($sFilename) filtered data.
        _FileReadToArray($sFilteredFileName, $aArr, 0, @TAB)
        ;_ArrayDisplay($aArr)
        For $i = 0 To UBound($aArr) - 1
            $key = $aArr[$i][1]
            $oDict.Item($key) = $aArr[$i][0]
        Next
    EndIf

    ; -------- $sFilename file ------------
    ; Note: StringRegExpReplace allows each line in $sFilename having leading hex number, followed by optional date and/or optional time.
    Local $aArray = StringSplit(StringRegExpReplace(FileRead($sFilename), "(?im)^([a-f0-9\h]+\|\h*)([/:0-9\h]*\|?\h*){0,2}|\v*\z", ""), @LF, 3) ; $STR_ENTIRESPLIT (1) = entire delimiter string is needed to mark the split + $STR_NOCOUNT (2) = disable the return count in the first element - effectively makes the array 0-based (must use UBound() to get the size of the array in this case).
    For $i = 0 To UBound($aArray) - 1
        $key = $aArray[$i]
        If $oDict.Exists($key) Then
            $oDict.Item($key) = $oDict.Item($key) + 1
        Else
            $oDict.Add($key, 1)
        EndIf
    Next

    ; ------- Create array from Scripting Dictionary --------------
    Local $Count = $oDict.Count ; This tells how many keys are in $oDict
    Local $Array[$Count][2]
    Local $p = 0
    For $vKeys In $oDict
        $Array[$p][1] = $vKeys
        $Array[$p][0] = Number($oDict.item($vKeys))
        $p += 1
    Next
    _ArraySort($Array, 0, 0, 0, 0)

    ; --------- Save $sFilteredFileName file -----------
    Local $hFiltered = FileOpen($sFilteredFileName, 10)
    FileWrite($hFiltered, _ArrayToString($Array, @TAB, -1, -1, @CRLF))
    FileClose($hFiltered)
    Return $Array
EndFunc   ;==>_FilteredLog

 

[jackylee0908]

Hi @Malkey,

That's amazing, thanks!

But, a little thing still need your help, is it possible to get text aligned? As you can see below, the each string is not aligned, if the count over 1000, the string will be shift to right.

If possible, I'd like to let it aligned as below example, say that, reserve the space for the number of count which over 1000.

Thanks a lot!

Edited July 3, 2019 by jackylee0908

[Malkey]

In my above script replace this line, around #55,

$Array[$p][0] = Number($oDict.item($vKeys))

with this line,

$Array[$p][0] = StringFormat("%5s", $oDict.item($vKeys)) ; Right justified & 5 characters

This still allows _ArraySort  to sort the first column in numeric order.

[jackylee0908]

Hi @Malkey,

Great, it worked for me, thanks a lot!