Slow FileReadLine performance?

[sublimnl]

I'm writing a script to parse a directory of IIS log files to find every time a user accesses an Exchange 2000 mailbox other than their own. Right now the script is hardcoded for a specific file and the log file that I have been testing with is ~10600 lines and about 1.9MB. This script takes over 10 minutes to parse the file line by line. It seems to start out fast and then get slower as I continue to read each consecutive line. Any ideas why?

Thanks!

expandcollapse popup

#include <file.au3>
$file = "C:\ex041230.log"
$logfile = FileOpen ("C:\ex041230.log",0)
$resultfile = FileOpen ( "C:\eResult.log", 2)
$exchip = "172.16.4.3"

;
; This loop traverses files in the directory
; (not written yet)
; Do



;
; This loop reads the file
;

$linecount=1

ProgressOn("Parsing...", "","Current File: " & $file,-1,-1,18)
$totlines = _FileCountLines($file)

Do

    If StringRight($linecount,2) = "00" Then
    $fileprog = Round(($linecount/$totlines)*100)
    ProgressSet($fileprog)
    EndIf

    $logline = FileReadLine($logfile, $linecount)
    If @error = -1 Then ExitLoop

;
;Check current line and make sure it accesses a mailbox
; ex. "/exchange/username" instead of "/exchange/ - 200 Exchange-Server-Frontend-Proxy"
;
    If StringInStr($logline, $exchip & " 80 GET /exchange/") > 0 Then
    If StringInStr($logline, " 80 GET /exchange/ ") = 0 Then
    $charcount=0
    $charcount2=0
;
;Get location of first character of Login ID
;
    Do
      $charcount=$charcount+1
      $loginid=StringMid($logline,$charcount,1)
    Until StringIsAlpha($loginid)=1

;
;Get last letter of Login ID, trim trailing whitespace
;
    Do
      $charcount2=$charcount2+1
      $loginid=StringMid($logline,$charcount,$charcount2)
    Until StringRight($loginid,1) = " "
    $loginid=StringTrimRight($loginid,1)
;
;Write to results file is user is not accessing their own mailbox
;
    If StringInStr($logline,"GET /exchange/" & $loginid) = 0 Then
    FileWriteLine($resultfile, $logline)
    EndIf
    
    EndIf
    EndIf


    $linecount = $linecount + 1
Until $linecount = $totlines+1
ProgressOff()
MsgBox(4096,"Done!","Done!")
FileClose($logfile)
FileClose($resultfile)
Exit

Here is a sample of the logfile. The first line is a match.

2004-12-30 00:23:36 172.16.4.46 carlos.aya 172.16.4.3 80 GET /exchange/Jim.Chase/Inbox/Ocean+Service+SMDL+survey+update+(0600+hours,+December+29,+2004).EML Cmd=open 200 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:00:10 172.16.4.46 jim.chase 172.16.4.3 80 POLL /exchange/Jim.Chase/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:00:31 172.16.4.46 jan.coyle 172.16.4.3 80 POLL /exchange/Jan.Coyle/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+YPC+3.0.1;+.NET+CLR+1.1.4322)
2004-12-30 00:00:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:00:51 172.16.4.46 jason.taylor 172.16.4.3 80 POLL /exchange/Jason.Taylor/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:01:08 172.16.4.46 armin.tavassoli 172.16.4.3 80 POLL /exchange/Armin.Tavassoli/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461;+GIL+2;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322)
2004-12-30 00:01:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:02:01 172.16.4.46 ben.wheeler 172.16.4.3 80 POLL /exchange/Ben.Wheeler/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.1.4322)
2004-12-30 00:02:10 172.16.4.46 jim.chase 172.16.4.3 80 POLL /exchange/Jim.Chase/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:02:31 172.16.4.46 jan.coyle 172.16.4.3 80 POLL /exchange/Jan.Coyle/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+YPC+3.0.1;+.NET+CLR+1.1.4322)
2004-12-30 00:02:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:02:51 172.16.4.46 jason.taylor 172.16.4.3 80 POLL /exchange/Jason.Taylor/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:03:08 172.16.4.46 armin.tavassoli 172.16.4.3 80 POLL /exchange/Armin.Tavassoli/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461;+GIL+2;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322)
2004-12-30 00:03:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:04:02 172.16.4.46 ben.wheeler 172.16.4.3 80 POLL /exchange/Ben.Wheeler/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.1.4322)
2004-12-30 00:04:10 172.16.4.46 jim.chase 172.16.4.3 80 POLL /exchange/Jim.Chase/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:04:31 172.16.4.46 jan.coyle 172.16.4.3 80 POLL /exchange/Jan.Coyle/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+YPC+3.0.1;+.NET+CLR+1.1.4322)
2004-12-30 00:04:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:04:51 172.16.4.46 jason.taylor 172.16.4.3 80 POLL /exchange/Jason.Taylor/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)
2004-12-30 00:05:08 172.16.4.46 armin.tavassoli 172.16.4.3 80 POLL /exchange/Armin.Tavassoli/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461;+GIL+2;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322)
2004-12-30 00:05:33 172.16.4.43 - 172.16.4.3 80 HEAD / - 401 WhatsUp_Gold/7.0
2004-12-30 00:05:51 172.16.4.46 ben.wheeler 172.16.4.3 80 POLL /exchange/Ben.Wheeler/Calendar - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.1.4322)
2004-12-30 00:05:59 172.16.4.46 ben.wheeler 172.16.4.3 80 POLL /exchange/Ben.Wheeler/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.1.4322)
2004-12-30 00:06:11 172.16.4.46 jim.chase 172.16.4.3 80 POLL /exchange/Jim.Chase/Inbox - 207 Exchange-Server-Frontend-Proxy/6.0+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0)

[Insolence]

Well before you process everything, maybe you should use _FileReadToArray and work through that.

[sublimnl]

Thanks for the tips! I'll give it a go.

[SlimShady]

Now I'm positive I won't use FileReadLine ever again.

_FileReadToArray works much better, in every situation.

[sublimnl]

Now I'm positive I won't use FileReadLine ever again.

_FileReadToArray works much better, in every situation.

<{POST_SNAPBACK}>

AMEN TO THAT! What once took over 10 minutes now takes less than 10 seconds. Thanks!