log parsing

drtrann · August 9, 2012

hello folks,

let may start by saying that im sorry but i feel entirely overwhelmed with this small app I'm trying to build and have never experimented with this kind of coding, im kinda ashamed that after about 10 hours of research i still havn't got a clue where to start.

for testing purposes i am attempting to build a combat log parser for WOW. the log generates events like this

8/8 20:13:36.580  SWING_DAMAGE,0xF130EFD6000005D9,"Xin the Weaponmaster",0x10a48,0x0,0x02000000002F5E5E,"Maxximumpand",0x511,0x0,31130,-1,1,0,13342,0,nil,nil,nil

for my purposes i only need

8/8 20:13:36.580  SWING_DAMAGE,"Xin the Weaponmaster","Maxximumpand",31130,0,13342,0

the lines then should be broken down into arrays

time, type, caster, target, damage, damage done, damage resisted, damage blocked, damage absorbed

from there it should find lines with caster = "xin the weaponmaster" and a target of "maxximumpand"

and display them in work with just that information to give a read out of:

damage taken over the encounter, #of blocks, # of absorbs, total amount blocked, total amount absorbed, total mitigated.

now im not looking for someone to write this for me, but i am totally lost from where to start. i know _filereadtoarray fits in there but i have no idea on how to define what it keeps vs what it throws out, then how to sort that data into something that's understandable.

again i know idiots ask questions without researching a lot here, but i'm truly stumped on how to this as i have never worked with these functions before

thank you very much for your help

ps. i know there are several parsing tools out there, but none break down the information in this specific way, and im dealing with 50-60 logs with 100,000 lines of log each, so i'd rather work with something local that just gives me what i'm looking for.

Melba23 · August 9, 2012

Hi,

This thread was reported as "game-related", but while it is indeed that, it does not interact with the game in any way. The OP is asking how to parse a text file generated by an app that happens to be a game. In my view this thread is entirely legal and I encourage people to help. Particularly as the OP has previously shown himself to be quite aware of the rules.

drtrann,

I would start by using _FileReadToArray to get each line of the log into an element and then using StringSplit to break up the line into separate sections. Look at the Help file and see what you can work out for yourself - come back if you get stuck.

M23

Zedna · August 9, 2012

Here is something very simple for you to start with:

$lines = StringSplit(FileRead("file.log"), @CRLF, 1)
For $i = 1 To $lines[0]
    ConsoleWrite('line:' & $i & ' ')
    $columns = StringSplit($lines[$i], ",")
    For $j = 1 To $columns[0]
        ConsoleWrite('col' & $j & ':' & $columns[$j] & ' ')
    Next
    ConsoleWrite(@CRLF)
Next

StringSplit() is magic function for you ;-)

Zedna · August 9, 2012

And here it is with filtered only some columns on output:

$lines = StringSplit(FileRead("file.log"), @CRLF, 1)
For $i = 1 To $lines[0]
    If $lines[$i] = '' Then ContinueLoop
    ConsoleWrite('line:' & $i & ' ')
    $col = StringSplit($lines[$i], ",")
    $line_out = $col[1] & ',' & $col[3] & ',' & $col[7] & ',' & $col[10] & ',' & $col[13] & ',' & $col[14] & ',' & $col[15]
    ConsoleWrite($line_out & @CRLF)
Next

drtrann · August 10, 2012

you guys are awesome. thus far i believe i've gotten it to read and break apart the strings but it tossed in an error when i attempted to isolate the specific sections of text im looking for, the error led me to believe that its not properly splitting the strings. so i set it to write what the split string was supposed to look like and it seems somewhere in there i lost the array (export file comes out blank)

hopefully you guys could tell me why im a muppet

log file to demo parsing

8/7 18:35:11.877 SPELL_CAST_SUCCESS,0x02000000002F5E5E,"Maxximumpand",0x511,0x0,0xF130EFD6000066F8,"Xin the Weaponmaster",0x10a48,0x0,57755,"Heroic Throw",0x1
8/7 18:35:11.961 SPELL_CAST_FAILED,0x02000000002F5E5E,"Maxximumpand",0x511,0x0,0x0000000000000000,nil,0x80000000,0x80000000,57755,"Heroic Throw",0x1,"Not yet recovered"
8/7 18:35:12.200 SPELL_MISSED,0x02000000002F5E5E,"Maxximumpand",0x511,0x0,0xF130EFD6000066F8,"Xin the Weaponmaster",0x10a48,0x0,18498,"Silenced - Gag Order",0x1,IMMUNE
8/7 18:35:12.327 SPELL_CAST_FAILED,0x02000000002F5E5E,"Maxximumpand",0x511,0x0,0x0000000000000000,nil,0x80000000,0x80000000,57755,"Heroic Throw",0x1,"Not yet recovered"

code

#include <ButtonConstants.au3>
#include <ComboConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <GuiToolbar.au3>
#include <StaticConstants.au3>
#include <ToolbarConstants.au3>
#include <WindowsConstants.au3>
#include <file.au3>
#include <GuiConstants.au3>
;// vars
global $nmsg
global $file
local $aRawLog
global $lineout
Global $asplitLog
global $aModLog
;// parse tags
local $time
local $event
local $hideCaster
local $sourcGuid
local $sourceName
local $sourceFlags
local $sourceFlags2
local $destGUID
local $destName
local $destFlags
local $destFlags2
local $spellId
local $spellName
local $spellSchool
local $amount
local $misstype
local $overkill
local $school
local $resist
local $blocked
local $absorbed
local $critical
local $glancing
local $crushing
;// types of event
local $swingDamage
local $swingMissed
local $rangeDamage
local $spellDamgae
local $spellMissed
local $spellCastSuccess
main()
func main()
;// GUI
$Form1 = GUICreate("Form1", 619, 211, 193, 126)
;$ToolBar1 = _GUICtrlToolbar_Create($Form1, 0)
$logbtn = GUICtrlCreateButton("log", 0, 0, 75, 25)
$gobtn = GUICtrlCreateButton("process", 88, 0, 75, 25)
$savebtn = GUICtrlCreateButton("save", 176, 0, 75, 25)
$bossList = GUICtrlCreateCombo("bossList", 8, 56, 145, 25, BitOR($CBS_DROPDOWN,$CBS_AUTOHSCROLL))
GUICtrlSetData(4, "xin the weaponmaster|boss1|boss2", "bosslist")
$Label1 = GUICtrlCreateLabel("boss name", 8, 32, 55, 17)
$Label2 = GUICtrlCreateLabel("target name", 8, 88, 60, 17)
$tagetNameInput = GUICtrlCreateInput("Target Name", 8, 112, 145, 21)
$Label3 = GUICtrlCreateLabel("toal damage taken", 320, 64, 92, 17)
$Label4 = GUICtrlCreateLabel("# of blocks", 352, 88, 57, 17)
$Label5 = GUICtrlCreateLabel("#of absorbs", 352, 112, 60, 17)
$Label6 = GUICtrlCreateLabel("total amount blocked", 312, 136, 103, 17)
$Label7 = GUICtrlCreateLabel("total amount absorbed", 304, 160, 109, 17)
$Label8 = GUICtrlCreateLabel("total mitigated", 344, 184, 69, 17)
$Label9 = GUICtrlCreateLabel("fight length", 360, 40, 56, 17)
$fightLengthInput = GUICtrlCreateInput("fight length", 424, 32, 121, 21)
$totalDmgTakenInput = GUICtrlCreateInput("total Dmg Taken", 424, 56, 121, 21)
$numBlocksInput = GUICtrlCreateInput("num Blocks", 424, 80, 121, 21)
$numAbsorbsinput = GUICtrlCreateInput("num Absorbs", 424, 104, 121, 21)
$totalBlockedInput = GUICtrlCreateInput("total Blocked", 424, 128, 121, 21)
$totalAbsorbedInput = GUICtrlCreateInput("total Absorbed", 424, 152, 121, 21)
$totalMitInput = GUICtrlCreateInput("total Mitigated", 424, 176, 121, 21)
GUISetState()
While 1
$nMsg = GUIGetMsg()
Switch $nMsg
     Case $GUI_EVENT_CLOSE
         Exit
Case $logbtn
_logopen()
Case $savebtn
_save()
case $gobtn
_process()
EndSwitch
WEnd
EndFunc
Func _exit()
Exit
EndFunc
Func _logopen() ;// ignore fileopendialog wont work so hardcoded in process
$file = FileOpen("log.txt", 0)
MsgBox(0,0,$file)
EndFunc
Func _process()
_FileReadToArray("log.txt", $aRawLog)
MsgBox(0,"",$aRawLog[0])
$asplitLog = StringSplit($aRawLog, ",")
$aModLog = $asplitLog[1] ;& ', ' & $asplitLog[3] & ', ' & $asplitLog[7] & ', ' & $asplitLog[10]
EndFunc
Func _save()
$logExport = _FileWriteFromArray("logexport.txt", $aModLog, 1)
FileClose($logExport)
FileClose($file)
EndFunc

ignore the parse tags they were an experiment thats still ongoing.

also the time and event tags dont have the same delimmiter as the rest of the line so should i first stringsplit " " (double space) that happens between the time and the event tag, then stringsplit the event tag by "," for the rest of the tags?

also how would i go about looking for only the event tags that i want ie. if i only want to keep the lines that start with SPELL_CAST_SUCCESS

again thanks for the points. was a lot easier to get working. still fighting the damn interface though... cant get fileopendialog to actually show the window

Edited August 10, 2012 by drtrann

Melba23 · August 10, 2012

drtrann,

hopefully you guys could tell me why im a muppet

Since you ask so nicely.....!

- 1. There is no need to open the file at all - so function _logopen serves no purpose. If you want to use FileOpenDialog to choose the log to open then look at the script below.

- 2. In the _process function you are correctly reading the file into an array, but you have misunderstood how to use StringSplit. The function only works on single strings - not complete arrays, so you need to loop through the array and use it on each element in turn. The script below shows how to do it.

- 3.

should i first stringsplit " " (double space) that happens between the time and the event tag, then stringsplit the event tag by "," for the rest of the tags?

You see that you are not such a muppet after all as that is exactly what you need to do - except that you nee to split first on the commas and then the first section on spaces - otherwise you split on the spaces in the name.

Here is a short example script based on your example file which should demonstrate all the above:

#include <GUIConstantsEx.au3>
#include <File.au3>

#include <Array.au3> ; Just for display

main()

Func main()

    Local $sLogName

    Local $Form1 = GUICreate("Form1", 619, 211, 193, 126)

    Local $logbtn = GUICtrlCreateButton("log", 0, 0, 75, 25)
    Local $gobtn = GUICtrlCreateButton("process", 88, 0, 75, 25)
    Local $savebtn = GUICtrlCreateButton("save", 176, 0, 75, 25)

    GUISetState()

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                Exit
            Case $logbtn
                $sLogName = _logopen()
                ConsoleWrite($sLogName & @CRLF)
            Case $savebtn
                _save()
            Case $gobtn
                _process($sLogName)
        EndSwitch
    WEnd
EndFunc

Func _logopen()
    Return FileOpenDialog("Choose the log file", @Scriptdir, "Log files (*.txt)", 3)
EndFunc

Func _save()

EndFunc

Func _process($sLogName)
    ; Read file into an array
    Local $aLines
    _FileReadToArray($sLogName, $aLines)
    _ArrayDisplay($aLines, "Whole file") ; Just for display
    ; Now loop through the array and StringSplit each line
    For $i = 1 To $aLines[0]
        ; First we split on commas
        $aFirst_Split = StringSplit($aLines[$i], ",")
        _ArrayDisplay($aFirst_Split, "Split on commas") ; Just for display
        ; Now we split the first element on spaces
        $aSecond_Split = StringSplit($aFirst_Split[1], " ")
        _ArrayDisplay($aSecond_Split, "Split on spaces") ; Just for display
    Next
EndFunc

All you need to do now is extract the necessary elements from $aFirst_Split and $aSecond_Split.

Any questions?

M23

drtrann · August 10, 2012

SUCCESS! thanks

now all i need to do is identify each line based on their caster, in this case i only want to read lines from "xin the weaponmaster" which would be stored in $aFirstSplit[3] so just spit balling here but i would put something simple like an IF statement like this

for $i = 1 to $aLines[0]
;(the stuff you posted above)
if $aFirst_Split[3] = "xin the weaponmaster"
Filewriteline("logexport.txt", $aSecondSplit[1] & ", " $aSecondSplit[2] ) ; keep going till i have all the information i want
endif
next

i know there must be a better way of handling this then writing it to a file, but my simple mind likes to keep things easy where i can see where things go wrong :/ but in theory the above line should go through every line and pick out the lines that have xin casting or doing something, so i can sort out the 100,000s of lines that dont even pertain to what im looking for

*edit:

just woke up and was kinda bugged about how im going about this so i figured i would look for a way to store just the information in another array. i figured storing them all in a new array and adding a new column for each line i want to keep kind of makes sense(?) is there such a thing as having too many arrays or is this the proper way of going about it?

Edited August 11, 2012 by drtrann

Sign In

log parsing

Recommended Posts

drtrann

Link to comment

Share on other sites

Melba23

Link to comment

Share on other sites

Zedna

Link to comment

Share on other sites

Zedna

Link to comment

Share on other sites

drtrann

Link to comment

Share on other sites

Melba23

Link to comment

Share on other sites

drtrann

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta