Searching text files for specific text

ttleser · March 9, 2007

I've got a log file that's generated from a video game called Neverwinter Nights, which I host a server module for. The log file will contain many things in it. I need to search for specific text and report the results out to another text file.

Ok, here's an example of a text file to search through:

[Fri Feb 03 21:25:29]*** MODULE HEARTBEAT ***
[Fri Feb 03 21:25:39] BecDeCorbin Left as a Player (2 players left)
[Fri Feb 03 21:25:39] *** MODULE HEARTBEAT ***
[Fri Feb 03 21:25:50] *** MODULE HEARTBEAT ***
Connection Attempt made by BecDeCorbin (QC7MWWWY)
[Fri Feb 03 21:25:56] BecDeCorbin (QC7MWWWY) Joined as Player 3
[Fri Feb 03 21:26:00] *** MODULE HEARTBEAT ***
EVENT reported by BecDeCorbin in :
BecDeCorbin's faction reps being restored.
EVENT reported by BecDeCorbin in :
BecDeCorbin's faction reps being saved.
[Fri Feb 03 21:26:11] *** MODULE HEARTBEAT ***
EVENT reported by BecDeCorbin in CityDocks:
BecDeCorbin's faction reps being restored.
EVENT reported by BecDeCorbin in CityDocks:
BecDeCorbin entered CityDocks

Ok, I need to have a script search the log file for "Joined" and "Left" and output the whole line out to another text file so that I can easily search through the output file and determine how much the game is used by people. I'd like the output to look like this:

[Fri Feb 03 21:25:56] BecDeCorbin (QC7MWWWY) Joined as Player 3

[Fri Feb 03 21:25:39] BecDeCorbin Left as a Player (2 players left)

Here's the code I've got so far, which of course isn't working.

$file = FileOpen("test.txt", 0)
$log_file = FileOpen("results.txt", 1)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    If StringRegExp($line, 'Joined') = 1 then Joined()
    If StringRegExp($line, 'Left as a Player') = 1 then Left()
Wend

FileClose($file)
FileClose($log_file)

Func Joined()
; Check if file opened for writing OK
If $log_file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
FileWrite($log_file, $line & @CRLF)
EndFunc

Func Left()
; Check if file opened for writing OK
If $log_file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
FileWrite($log_file, $line & @CRLF)
EndFunc

What am I doing wrong?

SmOke_N · March 9, 2007

Might want to learn how to use StringRegExp() first... Or look at _FileReadToArray() + For/Next + StringInStr().

Edit:

~~Oh, and that was a really piss poor example of some type of effort.~~

Editing the edit... want to apologize for the comment, I'm seriously on pain killers... shouldn't even be in the forum... I do see you tried something.

Edit2:

My pain pill must be kicking in... This is pretty much all you need, but you'll need to loop through the arrays and write the files yourself (I'm assuming you're capable of that as long as you've been a member)

$sRead = FileRead('MyPath\MyFileToRead.Log')
$aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)
$aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)oÝ÷ ØGb·táÈ¶Æ¥Ø^wènëb¶+'ßÛlz«j×)¶nÞ²k¡¶)è·Z·*.çjëh×6$sRead = FileRead('MyPath\MyFileToRead.Log')
$aArray = StringRegExp($sRead, '\[.*?Joined.*?|\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)
_ArrayDisplay($aArray, '')

Edited March 9, 2007 by SmOke_N

_Kurt · March 9, 2007

Or, perhaps something like:

For $i = 1 To _FileCountLines("YourFilenameHere.txt")
    $line = FileReadLine("YourFilenameHere.txt", $i)
    If NOT StringInStr($line, "SearchString") = 0 Then Msgbox(0,"",$i & " line in the file contains the string.")
Next

Just tossing some ideas out there, Smoke_N's suggestion is probably better, but you might find this easier :whistle:

Kurt

ttleser · March 9, 2007

but you'll need to loop through the arrays and write the files yourself (I'm assuming you're capable of that as long as you've been a member)

ROFL @Smoke: Sorry bud, I'm still a newbie and haven't messed with arrays yet. I did look up the StringRegExp (that's how I came up with that crappy script :whistle:

) and I can't find in there anywhere about the code you've got in it: '\[.*?Joined.*?(?:\r|$)' Jesus! I can recognize Joined in there, but that the hell is the rest of that crap? I figure it's got to do with "I don't care what's before 'Joined' and I don't care with what's after it". Could you tell me what to search for in the help file that'll tell me where to find that?

@Kurt.. I'll look into that one.

On both scripts just so you guys know I'm not interested in a count of how many times Joined and Left comes up, I want the lines that contain those keywords to be output to another file.

SmOke_N · March 9, 2007

ROFL @Smoke: Sorry bud, I'm still a newbie and haven't messed with arrays yet. I did look up the StringRegExp (that's how I came up with that crappy script ) and I can't find in there anywhere about the code you've got in it: '\[.*?Joined.*?(?:\r|$)' Jesus! I can recognize Joined in there, but that the hell is the rest of that crap? I figure it's got to do with "I don't care what's before 'Joined' and I don't care with what's after it". Could you tell me what to search for in the help file that'll tell me where to find that?

I'm having a hard time understanding what you just said there... may not even be your fault. StringRegExp() is like a whole other language itself, here is a reference to maybe help you that I use: http://perldoc.perl.org/perlre.html#Regular-Expressions

If you run this exact example, replace the content in FileRead() with your exact path to the file to read... you may see the light.

#include <array.au3>
$sRead = FileRead('MyPath\MyFileToRead.Log')
$aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)
$aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)
_ArrayDisplay($aJoined, 'Joined')
_ArrayDisplay($aLeft, 'Left')

ttleser · March 9, 2007

Thanks for the reply Smoke we'll be playing tag on this post for a little while longer while you hopefully answer some of my questions.

First let me say about your remark, absolutely no offense taken. Hey, I'm just thankful for the replys as to what I did wrong. Like I said I'm still new to programming. :whistle:

Ok.

I see what your script did, it scanned through the file and displayed on the screen all the "Joined" entries then once that window closed it'll display the "Left" entries.

1. I looked in the help file under _array and couldn't find a function that'll write the output to a file, thoughts on what to search for? (You've lead me to water, but now let me drink as in you've got me started but let me finish the script and see if I can output it to a file).

2. I'd like to see it in Joined then Left format instead of all Joined then all Left. I'm guessing a While statement or a Do/Until or maybe a For/Next, but....

Sorry guys stupid questions but this is the only way I'll learn.

SmOke_N · March 9, 2007

The function to write to a file from an array I believe is _FileWriteFromArray().

Maybe someone will comment this for you, I wrote it quickly, ran it once, and it seemed to work

_DONT_KNOW_WHAT_TO_CALL_THIS_FUNCTION('MyPath\MyFileToRead.Log', 'MyPath\MyOutFile.Log')

Func _DONT_KNOW_WHAT_TO_CALL_THIS_FUNCTION($hFileIn, $hFileOut)
    Local $sRead = FileRead($hFileIn), $sHoldData, $aNames
    Local $aMatchNames, $bFound = False
    If $sRead = '' Then Return SetError(1, 0, 0)
    Local $aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)
    If IsArray($aJoined) = 0 Then Return SetError(2, 0, 0)
    Local $aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)
    If IsArray($aJoined) = 0 Then Return SetError(3, 0, 0)
    For $iCC = 0 To UBound($aJoined) - 1
        $aNames = StringRegExp($aJoined[$iCC], ']\s.*?\s', 1)
        If IsArray($aNames) = 0 Then Return SetError(4, 0, 0)
        For $xCC = 0 To UBound($aLeft) - 1
            $aMatchNames = StringRegExp($aJoined[$iCC], ']\s.*?\s', 1)
            If IsArray($aMatchNames) = 0 Then Return SetError(5, 0, 0)
            If $aMatchNames[0] = $aNames[0] Then
                $sHoldData &= '**************************************' & @CRLF & _
                    $aJoined[$iCC] & @CRLF & $aLeft[$iCC] & @CRLF & _
                    '**************************************' & @CRLF
                $bFound = True
            EndIf
        Next
        If $bFound = False Then
            $sHoldData &= '**************************************' & @CRLF & _
                $aJoined[$iCC] & @CRLF & _
                '**************************************' & @CRLF
        EndIf
        $bFound = False
    Next
    If $sHoldData Then FileWrite($hFileOut, $sHoldData)
    Return SetError(6, 0, 0)
EndFunc

Edited March 9, 2007 by SmOke_N

_Kurt · March 9, 2007

Sorry guys stupid questions but this is the only way I'll learn.

Don't worry! That's the only way any of us learn. Plus, you're not even being a "noob", a noob would ask for someone else to script something for them, provide absolutely no example or direct information about their problem. The small example I posted up there should work, but Smoke_N's solutions is probably the best way to go.

Kurt

ttleser · March 9, 2007

Was just gonna post that I found _FileWriteFromArray and added it to your script, but it shows all left (cause left overwrote the joined write), but it didn't write them Joined/Left/Joined/Left,etc...

#include <array.au3>

#include<File.au3>

$sRead = FileRead('test.txt')

$aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)

_FileWriteFromArray("results.txt", $aJoined,1)

$aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)

_FileWriteFromArray("results.txt", $aLeft,1)

I'll look through your and see if I can figure it out.

Edit:

Damn Smoke, maybe it's a losing battle for me to learn that's one complex script to me.

Tried your script and replaced your filenames with the appropriate names and got this error:

>"C:\Program Files\AutoIt3\beta\SciTE\..\autoit3.exe" /ErrorStdOut "C:\Documents and Settings\PC4\Desktop\junk2.au3"
C:\Documents and Settings\PC4\Desktop\junk2.au3 (18) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$sHoldData &= '**************************************' & @CRLF & $aJoined[$iCC]& @CRLF & $aLeft[$iCC] & @CRLF & '**************************************' & @CRLF
$sHoldData &= '**************************************' & @CRLF & $aJoined[$iCC] & @CRLF & ^ ERROR
>Exit code: 0 Time: 0.516

Edited March 9, 2007 by ttleser

SmOke_N · March 9, 2007

Was just gonna post that I found _FileWriteFromArray and added it to your script, but it shows all left (cause left overwrote the joined write), but it didn't write them Joined/Left/Joined/Left,etc...

#include <array.au3>
#include<File.au3>
$sRead = FileRead('test.txt')
$aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)
_FileWriteFromArray("results.txt", $aJoined,1)

$aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)
_FileWriteFromArray("results.txt", $aLeft,1)

I'll look through your and see if I can figure it out.

Edit:
Damn Smoke, maybe it's a losing battle for me to learn that's one complex script to me.

Tried your script and replaced your filenames with the appropriate names and got this error:

Yeah, my mistake (told you I was on pain meds) try this:

Func _DONT_KNOW_WHAT_TO_CALL_THIS_FUNCTION($hFileIn, $hFileOut)
    Local $sRead = FileRead($hFileIn), $sHoldData, $aNames
    Local $aMatchNames, $bFound = False
    If $sRead = '' Then Return SetError(1, 0, 0)
    Local $aJoined = StringRegExp($sRead, '\[.*?Joined.*?(?:\r|$)', 3)
    If IsArray($aJoined) = 0 Then Return SetError(2, 0, 0)
    Local $aLeft = StringRegExp($sRead, '\[.*?Left\sas\sa\sPlayer.*?(?:\r|$)', 3)
    If IsArray($aLeft) = 0 Then Return SetError(3, 0, 0)
    For $iCC = 0 To UBound($aJoined) - 1
        $aNames = StringRegExp($aJoined[$iCC], ']\s.*?\s', 1)
        If IsArray($aNames) = 0 Then Return SetError(4, 0, 0)
        For $xCC = 0 To UBound($aLeft) - 1
            $aMatchNames = StringRegExp($aLeft[$iCC], ']\s.*?\s', 1)
            If IsArray($aMatchNames) = 0 Then Return SetError(5, 0, 0)
            If $aMatchNames[0] = $aNames[0] Then
                $sHoldData &= '**************************************' & @CRLF & _
                    $aJoined[$iCC] & @CRLF & $aLeft[$xCC] & @CRLF & _
                    '**************************************' & @CRLF
                $bFound = True
                ExitLoop
            EndIf
        Next
        If $bFound = False Then
            $sHoldData &= '**************************************' & @CRLF & _
                $aJoined[$iCC] & @CRLF & _
                '**************************************' & @CRLF
        EndIf
        $bFound = False
    Next
    If $sHoldData Then FileWrite($hFileOut, $sHoldData)
    Return SetError(6, 0, 0)
EndFunc

Edit:

Dammit, try it now... :whistle:

Edited March 9, 2007 by SmOke_N

therks · March 9, 2007

Hate to ruin the fun, but unless I'm missing something it looks like you're both overcomplicating this just a little tiny... whole lot. :whistle:

I wrote this up from your initial code, does it work for you?

$hFileRead = FileOpen("test.txt", 0)
$hFileWrite = FileOpen("results.txt", 1)

; Check if either FileOpen failed.
If $hFileRead = -1 Then
    MsgBox(0, "Error", "Unable to open file for reading.")
    Exit
ElseIf $hFileWrite = -1 Then
    MsgBox(16, "Error", "Unable to open file for writing.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
While 1
    $sLine = FileReadLine($hFileRead)
    If @error = -1 Then ExitLoop
    ; Check if the line contains "Joined as Player" or contains "Left as a Player"
    ; It's case sensitive so it won't find lines with "left as a player"
    If StringInStr($sLine, 'Joined as Player', 1) OR StringInStr($sLine, 'Left as a Player', 1) Then
        FileWrite($hFileWrite, $sLine)
    EndIf
Wend

FileClose($hFileRead)
FileClose($hFileWrite)

Does that work?

SmOke_N · March 9, 2007

Hate to ruin the fun, but unless I'm missing something it looks like you're both overcomplicating this just a little tiny... whole lot. :whistle:

I wrote this up from your initial code, does it work for you?

$hFileRead = FileOpen("test.txt", 0)
$hFileWrite = FileOpen("results.txt", 1)

; Check if either FileOpen failed.
If $hFileRead = -1 Then
    MsgBox(0, "Error", "Unable to open file for reading.")
    Exit
ElseIf $hFileWrite = -1 Then
    MsgBox(16, "Error", "Unable to open file for writing.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
While 1
    $sLine = FileReadLine($hFileRead)
    If @error = -1 Then ExitLoop
    ; Check if the line contains "Joined as Player" or contains "Left as a Player"
    ; It's case sensitive so it won't find lines with "left as a player"
    If StringInStr($sLine, 'Joined as Player', 1) OR StringInStr($sLine, 'Left as a Player', 1) Then
        FileWrite($hFileWrite, $sLine)
    EndIf
Wend

FileClose($hFileRead)
FileClose($hFileWrite)

Does that work?

Mine isn't over complicated... I can walk you through the steps, and then you can see if yours does the same.

1. Find all the Joined Players

2. Find all the Left as Players

3. Find the name of the Joined Players

4. Find the name of the Left as Players

5. Match the Joined Name to the Left as Name

6. Group together in a Var

7. Write to File.

All yours does is find the Find/Left As, which the 2 lines I wrote of StringRegExp does. It doesn't group them like he wants.

ttleser · March 10, 2007

Thanks to both you guys. Sorry but this is the first time I got to get back online (too much crap going on).

I'll look over both scripts and see how you guys did it and see if I can learn from it.

Thanks

ttleser · March 10, 2007

Alrighty smoke, I tried your script and got this error:

>"C:\Program Files\AutoIt3\beta\SciTE\..\autoit3.exe" /ErrorStdOut "C:\Documents and Settings\PC4\Desktop\NWN_log_search.au3"
C:\Documents and Settings\PC4\Desktop\NWN_log_search.au3 (15) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$aMatchNames = StringRegExp($aLeft[$iCC], ']\s.*?\s', 1)
$aMatchNames = StringRegExp(^ ERROR
>Exit code: 0 Time: 70.652

Now, the text file is 176meg. It's pretty large. I'm assuming that the script couldn't handle something that large? I copy/pasted a smaller amount of the file into a newer file that's only 33meg now.

I ran the script and are a little confused by the output: Here's a page from the beginning of the output file:

**************************************
[sat Dec 02 01:54:42]Fr8lad (QVRTY7DY) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[sat Dec 02 01:55:15] Fr8lad (QVRTY7DY) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[sun Dec 03 10:44:00] itol (FTMMWM4C) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Mon Dec 04 06:34:46] Fink D'Eather (QG6LXD4E) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Tue Dec 05 16:39:26] saarrggg (VD7DV3QK) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Tue Dec 05 17:31:34] strider4 (QVRPVWJK) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Wed Dec 06 13:50:54] Cougar le beau (QC4GMLTJ) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Mon Dec 11 12:13:11] earyldor senocen (QC474Q7T) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Mon Dec 11 19:27:25] Azzie Elzabuad (FT6KEHTX) Joined as Player 1

**************************************
**************************************
[Mon Dec 11 19:39:36] coachclk (FTRQAQ9X) Joined as Player 2

**************************************
**************************************
[Tue Dec 12 07:51:35] Ka'Lon (QG6DW99C) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Tue Dec 12 13:05:51] Taramelion (QC4R9JDR) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************
**************************************
[Tue Dec 12 15:02:09] ceddadude (QGM99W66) Joined as Player 1

[sat Dec 02 01:54:43] Fr8lad Left as a Player (0 players left)

**************************************

I figured it should have gone line by line through the log file and found entries. It didn't do that. Checking Saundes script now.

ttleser · March 10, 2007

Saunders. I ran your script and it's what I was expecting mine to do. It's the output I was expecting. This will do.

Thanks Smoke, you added some fluff to the script like adding the **** breaks in it, but I really didn't need that. Just the raw data, paired would have been fine but not required. I will look at your script and hopefully learn from it as to what you were doing.

ttleser · March 10, 2007

Alrighty, update time. I modified Saunder's script, it created a results file that only showed lines that matched "joined" or "left".

$hFileRead = FileOpen("nwserverLog1.txt", 0)
$hFileWrite = FileOpen("results.txt", 1)

; Check if either FileOpen failed.
If $hFileRead = -1 Then
    MsgBox(0, "Error", "Unable to open file for reading.")
    Exit
ElseIf $hFileWrite = -1 Then
    MsgBox(16, "Error", "Unable to open file for writing.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Msgbox(0,"Search", "Scanning NWN log file", 3)
While 1
    $sLine = FileReadLine($hFileRead)
    If @error = -1 Then ExitLoop
    ; Check if the line contains "Joined as Player" or contains "Left as a Player"
    ; It's case sensitive so it won't find lines with "left as a player"
    If StringInStr($sLine, 'Joined as Player', 1) OR StringInStr($sLine, 'Left as a Player', 1) Then
        FileWrite($hFileWrite, $sLine&@CRLF)
    EndIf
Wend

FileClose($hFileRead)
FileClose($hFileWrite)

I'd like to modify the first script to match person A joined with person A left and extract the time on for each they are on, and figure out if that time fits into a certain timeframe, like:

0min - 5min

5min - 15min

15min - 1hr

1hr-24hr

Person A joined at 05:50

Person A left at 05:52

So Person A online for 52-50 = 2 min.

2min fits into the 0min-5min timeframe so 0min-5min variable increment by 1.

Rinse and repeat.

Unfortunately what makes this script more complicated is that Person B may join between with A joined and left, which makes the script have to keep track of multiple instances of joined to keep then matched up.

Can anyone post up thoughts of how about I should go about this? I'm not looking for scripts that anyone wrote, but ideas on how I should procede.

ttleser · March 23, 2007

Smoke, I tried your script again and got this error message:

C:\Documents and Settings\ttleser\Desktop\NWN_log_search.au3 (15) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$aMatchNames = StringRegExp($aLeft[$iCC], ']\s.*?\s', 1)
$aMatchNames = StringRegExp(^ ERROR

I know it worked for me once, but now it won't. Any thoughts?

ptrex · March 23, 2007

@ttleser

Maybe this can help you out. The LogParser was developed to do this kind of job.

MS LogParser

Look in the second Post for TXT files.

regards

ptrex

Searching text files for specific text

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members