Too Slow

[hankjrfan]

I have wrote a script that downloads a webpage and extracts just the info I need from it. But the problem is it is way too slow. The downloaded file ranges between 80kb and 200kb. It takes between 12 and 30 minutes to go through the file. I would really like to try to get this time down. About 5 minutes would be ideal. All comments and criticisms are welcome.

expandcollapse popup

Dim $world="Nova"
Dim $line
Dim $rows=20000
Dim $cols=5
Dim $name
Dim $level
Dim $vocation
Dim $array[$rows][$cols]

For $n=0 To $rows-1
    $array[$n][0]="0"
Next

While 1
    $begin = TimerInit()
    InetGet("http://www.tibia.com/statistics/?subtopic=whoisonline&world=" & $world,@ScriptDir & "\" & $world & ".txt")
    
    $file=FileOpen($world & ".txt", 0)
    
    While 1
        $line = $line & FileRead($file, 1)
        If @error = -1 Then ExitLoop
    Wend
    
    $string = "<TR><TD>Currently "
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp-1)
    $string = '<FORM ACTION="http://www.tibia.com/community/?subtopic=character" METHOD=post>'
    $temp = StringInStr($line,$string)
    $line = StringTrimRight($line,StringLen($line)-$temp+1)
    $line = StringReplace($line,"+"," ")
    $line = StringReplace($line,'<A NAME="'," ")
    $line = StringReplace($line,"%27"," ")
    
;Finds Number of Players Online
    $string = "<TR><TD>Currently "
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp+17)
    $string = " players are online.<BR>"
    $temp = StringInStr($line,$string)
    $online=StringLeft($line,$temp-1)
        
    $string = '<a target=_top HREF="http://www.tibia.com/community/?subtopic=character&name'
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp-1)
    
    For $n=1 To $online
        $found=0
    ;Finds Name
        $string = "name="
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+4)
        $string = '">'
        $temp = StringInStr($line,$string)
        $name=StringLeft($line,$temp-1)
        
    ;Finds Level
        $string = "10%>"
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+3)
        $string = "</TD>"
        $temp = StringInStr($line,$string)
        $level=Int(StringLeft($line,$temp-1))
        If $level<8 Then ContinueLoop
        
    ;Finds Vocation
        $string = "20%>"
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+3)
        $string = "</TD>"
        $temp = StringInStr($line,$string)
        $vocation=StringLeft($line,$temp-1)
        If $vocation="None" Then ContinueLoop
        
        For $i=0 To $rows-1
            If $array[$i][0]=$name Then
                $found=1
                $array[$i][3]=$array[$i][3]+1
                $array[$i][4]=@HOUR & ":" & @MIN
                $array[$i][1]=$level
                $array[$i][2]=$vocation
            EndIf
        Next
        
        If $found<>1 Then
            For $i=0 To $rows-1
                If $array[$i][0]="0" Then
                    $array[$i][0]=$name
                    $array[$i][1]=$level
                    $array[$i][2]=$vocation
                    $array[$i][3]=1
                    $array[$i][4]=@HOUR & ":" & @MIN
                    ExitLoop
                EndIf
            Next
        EndIf
        
    Next
    
    FileClose($file)
    FileMove($world & ".txt",$world & "bak.txt",1)
    FileMove($world & "bak.csv",$world & "bak2.csv",1)
    FileMove($world & ".csv",$world & "bak.csv",1)
    $file = FileOpen($world & ".csv",1)
    
    For $n=0 To $rows-1
        If $array[$n][0]="0" Then ExitLoop
        $name=$array[$n][0]
        $level=$array[$n][1]
        $vocation=$array[$n][2]
        $times=$array[$n][3]
        $time=$array[$n][4]
        $line=$name & "," & $level & "," & $vocation & "," & $times & "," & $time
        FileWriteLine($file,$line)
    Next
    FileClose($file)
    $dif = TimerDiff($begin)
    FileWriteLine("worldwatch.log","Minutes to run:  " & $dif/1000/60)
WEnd

[herewasplato]

try replacing:

While 1

$line = $line & FileRead($file, 1)

If @error = -1 Then ExitLoop

Wend

with

$line = FileRead($file,FileGetSize($file))

let me know......

[hankjrfan]

try replacing:

While 1

$line = $line & FileRead($file, 1)

If @error = -1 Then ExitLoop

Wend

with

$line = FileRead($file,FileGetSize($file))

let me know......

<{POST_SNAPBACK}>

It does not seem to work. I am not sure why it seems like it should. When I use your method the output file is empty.

[herewasplato]

It does not seem to work.  I am not sure why it seems like it should.  When I use your method the output file is empty.

<{POST_SNAPBACK}>

I just seemed to remember a one line solution for reading the entire contents of a file into a variable - I searched on FileRead and grabbed the first example that I noticed.

http://www.autoitscript.com/forum/index.ph...indpost&p=84379

It really should not save you that much time, but is better than reading the file one character at a time.

I'm not sure what else to suggest - good luck

[hankjrfan]

I just seemed to remember a one line solution for reading the entire contents of a file into a variable - I searched on FileRead and grabbed the first example that I noticed.

http://www.autoitscript.com/forum/index.ph...indpost&p=84379

It really should not save you that much time, but is better than reading the file one character at a time.

I'm not sure what else to suggest - good luck

<{POST_SNAPBACK}>

I got it!! The problem at first was that I was useing the file handle instead of the file name. Well anyway this method is much faster than the one I was useing.

Many Thanks