Sign in to follow this  
Followers 0
hankjrfan

Too Slow

5 posts in this topic

I have wrote a script that downloads a webpage and extracts just the info I need from it. But the problem is it is way too slow. The downloaded file ranges between 80kb and 200kb. It takes between 12 and 30 minutes to go through the file. I would really like to try to get this time down. About 5 minutes would be ideal. All comments and criticisms are welcome.

Dim $world="Nova"
Dim $line
Dim $rows=20000
Dim $cols=5
Dim $name
Dim $level
Dim $vocation
Dim $array[$rows][$cols]

For $n=0 To $rows-1
    $array[$n][0]="0"
Next

While 1
    $begin = TimerInit()
    InetGet("http://www.tibia.com/statistics/?subtopic=whoisonline&world=" & $world,@ScriptDir & "\" & $world & ".txt")
    
    $file=FileOpen($world & ".txt", 0)
    
    While 1
        $line = $line & FileRead($file, 1)
        If @error = -1 Then ExitLoop
    Wend
    
    $string = "<TR><TD>Currently "
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp-1)
    $string = '<FORM ACTION="http://www.tibia.com/community/?subtopic=character" METHOD=post>'
    $temp = StringInStr($line,$string)
    $line = StringTrimRight($line,StringLen($line)-$temp+1)
    $line = StringReplace($line,"+"," ")
    $line = StringReplace($line,'<A NAME="'," ")
    $line = StringReplace($line,"%27"," ")
    
;Finds Number of Players Online
    $string = "<TR><TD>Currently "
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp+17)
    $string = " players are online.<BR>"
    $temp = StringInStr($line,$string)
    $online=StringLeft($line,$temp-1)
        
    $string = '<a target=_top HREF="http://www.tibia.com/community/?subtopic=character&name'
    $temp = StringInStr($line,$string)
    $line = StringTrimLeft($line, $temp-1)
    
    For $n=1 To $online
        $found=0
    ;Finds Name
        $string = "name="
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+4)
        $string = '">'
        $temp = StringInStr($line,$string)
        $name=StringLeft($line,$temp-1)
        
    ;Finds Level
        $string = "10%>"
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+3)
        $string = "</TD>"
        $temp = StringInStr($line,$string)
        $level=Int(StringLeft($line,$temp-1))
        If $level<8 Then ContinueLoop
        
    ;Finds Vocation
        $string = "20%>"
        $temp = StringInStr($line,$string)
        $line = StringTrimLeft($line, $temp+3)
        $string = "</TD>"
        $temp = StringInStr($line,$string)
        $vocation=StringLeft($line,$temp-1)
        If $vocation="None" Then ContinueLoop
        
        For $i=0 To $rows-1
            If $array[$i][0]=$name Then
                $found=1
                $array[$i][3]=$array[$i][3]+1
                $array[$i][4]=@HOUR & ":" & @MIN
                $array[$i][1]=$level
                $array[$i][2]=$vocation
            EndIf
        Next
        
        If $found<>1 Then
            For $i=0 To $rows-1
                If $array[$i][0]="0" Then
                    $array[$i][0]=$name
                    $array[$i][1]=$level
                    $array[$i][2]=$vocation
                    $array[$i][3]=1
                    $array[$i][4]=@HOUR & ":" & @MIN
                    ExitLoop
                EndIf
            Next
        EndIf
        
    Next
    
    FileClose($file)
    FileMove($world & ".txt",$world & "bak.txt",1)
    FileMove($world & "bak.csv",$world & "bak2.csv",1)
    FileMove($world & ".csv",$world & "bak.csv",1)
    $file = FileOpen($world & ".csv",1)
    
    For $n=0 To $rows-1
        If $array[$n][0]="0" Then ExitLoop
        $name=$array[$n][0]
        $level=$array[$n][1]
        $vocation=$array[$n][2]
        $times=$array[$n][3]
        $time=$array[$n][4]
        $line=$name & "," & $level & "," & $vocation & "," & $times & "," & $time
        FileWriteLine($file,$line)
    Next
    FileClose($file)
    $dif = TimerDiff($begin)
    FileWriteLine("worldwatch.log","Minutes to run:  " & $dif/1000/60)
WEnd

Share this post


Link to post
Share on other sites



try replacing:

While 1

$line = $line & FileRead($file, 1)

If @error = -1 Then ExitLoop

Wend

with

$line = FileRead($file,FileGetSize($file))

let me know......


[size="1"][font="Arial"].[u].[/u][/font][/size]

Share this post


Link to post
Share on other sites

try replacing:

While 1

$line = $line & FileRead($file, 1)

If @error = -1 Then ExitLoop

Wend

with

$line = FileRead($file,FileGetSize($file))

let me know......

<{POST_SNAPBACK}>

It does not seem to work. I am not sure why it seems like it should. When I use your method the output file is empty.

Share this post


Link to post
Share on other sites

It does not seem to work.  I am not sure why it seems like it should.  When I use your method the output file is empty.

<{POST_SNAPBACK}>

I just seemed to remember a one line solution for reading the entire contents of a file into a variable - I searched on FileRead and grabbed the first example that I noticed.

http://www.autoitscript.com/forum/index.ph...indpost&p=84379

It really should not save you that much time, but is better than reading the file one character at a time.

I'm not sure what else to suggest - good luck


[size="1"][font="Arial"].[u].[/u][/font][/size]

Share this post


Link to post
Share on other sites

I just seemed to remember a one line solution for reading the entire contents of a file into a variable - I searched on FileRead and grabbed the first example that I noticed.

http://www.autoitscript.com/forum/index.ph...indpost&p=84379

It really should not save you that much time, but is better than reading the file one character at a time.

I'm not sure what else to suggest - good luck

<{POST_SNAPBACK}>

I got it!! The problem at first was that I was useing the file handle instead of the file name. Well anyway this method is much faster than the one I was useing.

Many Thanks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0