Script getting slower and slower

[steve8tch]

I was writing some code to show a problem I have with a script that uses 4 large arays - and the fact that each set of files that it was processing seem to take longer and longer. It turns out that the first part of this illustrative code exhibits the same issue.

If you run this code - (it take 10 seconds to complete) you will see that the first set of loops take approx 100ms to execute - but the 20th set take 600ms - 6x the time for the identical bit of code. I don't think memory use is an issue (autoit.exe memory use in Task manager hardly moves) and the resultant 40k text file is hardly huge. The CPU will run at 100% throughout.

I ran it again so that it executes 60 sets of loops - and the last loops are taking 1.8 seconds - this is 20x as long to exectute the same bit of code.

Here are the results of a couple of runs

20 loops - times in millisecond for each batch of 10 loops.

71 , 83 , 105 , 127 , 156 , 179 , 206 , 226 , 303 , 319 , 350 , 331 , 357 , 381 , 405 , 430 , 497 , 559 , 583 , 620

60 loops

65 , 82 , 105 , 126 , 155 , 178 , 206 , 231 , ,,,,,,,, 1636 , 1676 , 1674 , 1702 , 1749 , 1818 , 1860 , 1879 , 1887

The code - I have also attached the test script.

DirCreate(@Scriptdir & "\test files")

$x = FileOpen(@ScriptDir & "\test files\log.txt",1)

dim $str = ""

dim $time = ""

dim $counter = 0

$secs = TimerInit()

For $i = 1 to 200

For $j = 1 to 200

$str = $str & Chr(Random(97,122,1)) ; this bit just generates random lowercase charactors

Next

$str = $str & @CRLF

$counter = $counter + 1

If $counter = 10 Then ; lets get the time for each group of 10 loops

$time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "

$counter = 0

$secs = TimerInit()

EndIfNext

$x = FileOpen(@ScriptDir & "\test files\log.txt",1)

FileWriteLine($x,$str & @CRLF)

FileClose($x)

MsgBox(0,"Time in milliseconds for each group of 10 loops",$time)

Has anyone any thoughts.

Thanks for your help.

[Jos]

I was writing some code to show a problem I have with a script that uses 4 large arays - and the fact that each set of files that it was processing seem to take longer and longer. It turns out that the first part of this illustrative code exhibits the same issue.

It is this line causing the code to start running slower after each loop because the string is growing and takes longer to copy:

$str = $str & Chr(Random(97,122,1)) ; this bit just generates random lowercase charactors

Try replacing it with :

FileWrite($x,Chr(Random(97,122,1)))

and see the result...

[steve8tch]

Thankyou for your reply.

As you suggest - in this situation it is best to write to data direct to file. I thought that RAM was much faster than hard drive - and that constructing strings in "memory" would be much faster than continually writing out to hard drive.

Would you suggest that if we (AutoIt users) are writing scripts that "loop" round collecting information - we should avoid collecting data in $str and $array and write to temp files - and then read these files in when required ?

I'm slightly confused

[w0uter]

it wont probly help that much but "bla &= new" is faster then "bla = bla & new"

[Somerset]

here i just passed off the $str to another varible $grand if the length of $str gets beyond 5000 pass it onto another varible and then make $str equal nothing. i even raised the number of loops the times are pretty much the same. a simple ditch and switch. the times are all faster than what you were getting.

expandcollapse popup

DirCreate(@Scriptdir & "\test files")
$x = FileOpen(@ScriptDir & "\test files\log.txt",1)
dim $str = ""
dim $time = ""
dim $counter = 0
dim $grand = ""
$secs = TimerInit()
For $i = 1 to 400

    For $j = 1 to 200

        $str = $str & Chr(Random(97,122,1)); this bit just generates random lowercase charactors

            $len = StringLen($str)
            If $len > 5000 Then
            $grand = $grand & $str
            $str = ""
            endif
    Next
    $str = $str & @CRLF
    $counter = $counter + 1
    If $counter = 10 Then; lets get the time for each group of 10 loops

        $time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "
        $counter = 0
        $secs = TimerInit()


    EndIf


Next
$grand = $grand & $str
$str = $grand
$x = FileOpen(@ScriptDir & "\test files\log.txt",1)
FileWriteLine($x,$str & @CRLF)
FileClose($x)
MsgBox(0,"Time in milliseconds for each group of 10 loops",$time)

[steve8tch]

Thankyou for your replies.

As beerman suggested - if I alter the code to this

...
...
If $counter = 10 Then; lets get the time for each group of 10 loops
$strX = $strX & $str
$str = ""
$time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "
$counter = 0
$secs = TimerInit()
EndIf
...
...

I get the following timevalues (milliseconds)

149 , 149 , 156 , 146 , 149 , 152 , 149 , 150 , 148 , 149 , 151 , 151 , 150 , 150 , 151 , 149 , 150 , 151 , 150 , 151

What I am still slightly confused about is this:

I thought that writing to RAM was quick - I understand that AutoIt needs to manage its bit of memory - but are these times taken comparable with other script laguages ? I am not questioning the ability of the devs here - I am jst trying to understand what is happening in order to optimise some scripts.

Thanks for your help.

[Jos]

Try this script and see the difference in speed. my test shows that example 2 is 8 times faster.

I haven't looked in the source yet but assume that the first example will move the $var in temp mem space, then add $x at the end of it in temp mem space and then move the whole thing to $var. This is needed or else statements like :$var = $x & $var would fail......

The second example seems to find the "end" of the $var memspace and adds $x to it.

I am sure dev's will chime in when i am incorrect...

$Var = ""

$t = TimerInit()

For $x = 1 to 10000

$var = $var & $x

Next

ConsoleWrite('@@ Debug(6) : TimerDiff($t) = ' & TimerDiff($t) & @lf & '>Error code: ' & @error & @lf) ;### Debug Console

$Var = ""

$t = TimerInit()

For $x = 1 to 10000

$var &= $x

Next

ConsoleWrite('@@ Debug(6) : TimerDiff($t) = ' & TimerDiff($t) & @lf & '>Error code: ' & @error & @lf) ;### Debug Console

[steve8tch]

@w0uter and @JdeB

what a difference..

replacing

$str = $str & Chr(Random(97,122,1))

with

$str &= Chr(Random(97,122,1))

results in

208 , 113 , 113 , 123 , 114 , 113 , 113 , 118 , 113 , 117 , 112 , 114 , 116 , 112 , 116 , 114 , 113 , 115 , 116 , 115 ,

This must be the way to go.

Gents - thankyou