Sign in to follow this  
Followers 0
steve8tch

Script getting slower and slower

8 posts in this topic

I was writing some code to show a problem I have with a script that uses 4 large arays - and the fact that each set of files that it was processing seem to take longer and longer. It turns out that the first part of this illustrative code exhibits the same issue.

If you run this code - (it take 10 seconds to complete) you will see that the first set of loops take approx 100ms to execute - but the 20th set take 600ms - 6x the time for the identical bit of code. I don't think memory use is an issue (autoit.exe memory use in Task manager hardly moves) and the resultant 40k text file is hardly huge. The CPU will run at 100% throughout.

I ran it again so that it executes 60 sets of loops - and the last loops are taking 1.8 seconds - this is 20x as long to exectute the same bit of code.

Here are the results of a couple of runs

20 loops - times in millisecond for each batch of 10 loops.

71 , 83 , 105 , 127 , 156 , 179 , 206 , 226 , 303 , 319 , 350 , 331 , 357 , 381 , 405 , 430 , 497 , 559 , 583 , 620

60 loops

65 , 82 , 105 , 126 , 155 , 178 , 206 , 231 , ,,,,,,,, 1636 , 1676 , 1674 , 1702 , 1749 , 1818 , 1860 , 1879 , 1887

The code - I have also attached the test script.

DirCreate(@Scriptdir & "\test files")

$x = FileOpen(@ScriptDir & "\test files\log.txt",1)

dim $str = ""

dim $time = ""

dim $counter = 0

$secs = TimerInit()

For $i = 1 to 200

For $j = 1 to 200

$str = $str & Chr(Random(97,122,1)) ; this bit just generates random lowercase charactors

Next

$str = $str & @CRLF

$counter = $counter + 1

If $counter = 10 Then ; lets get the time for each group of 10 loops

$time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "

$counter = 0

$secs = TimerInit()

EndIfNext

$x = FileOpen(@ScriptDir & "\test files\log.txt",1)

FileWriteLine($x,$str & @CRLF)

FileClose($x)

MsgBox(0,"Time in milliseconds for each group of 10 loops",$time)

Has anyone any thoughts. ;)

Thanks for your help.

Share this post


Link to post
Share on other sites



I was writing some code to show a problem I have with a script that uses 4 large arays - and the fact that each set of files that it was processing seem to take longer and longer. It turns out that the first part of this illustrative code exhibits the same issue.

It is this line causing the code to start running slower after each loop because the string is growing and takes longer to copy:

$str = $str & Chr(Random(97,122,1)) ; this bit just generates random lowercase charactors

Try replacing it with :

FileWrite($x,Chr(Random(97,122,1)))

and see the result...


Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

Thankyou for your reply.

As you suggest - in this situation it is best to write to data direct to file. I thought that RAM was much faster than hard drive - and that constructing strings in "memory" would be much faster than continually writing out to hard drive.

Would you suggest that if we (AutoIt users) are writing scripts that "loop" round collecting information - we should avoid collecting data in $str and $array and write to temp files - and then read these files in when required ?

I'm slightly confused ;)

Share this post


Link to post
Share on other sites

here i just passed off the $str to another varible $grand if the length of $str gets beyond 5000 pass it onto another varible and then make $str equal nothing. i even raised the number of loops the times are pretty much the same. a simple ditch and switch. the times are all faster than what you were getting.

DirCreate(@Scriptdir & "\test files")
$x = FileOpen(@ScriptDir & "\test files\log.txt",1)
dim $str = ""
dim $time = ""
dim $counter = 0
dim $grand = ""
$secs = TimerInit()
For $i = 1 to 400

    For $j = 1 to 200

        $str = $str & Chr(Random(97,122,1)); this bit just generates random lowercase charactors

            $len = StringLen($str)
            If $len > 5000 Then
            $grand = $grand & $str
            $str = ""
            endif
    Next
    $str = $str & @CRLF
    $counter = $counter + 1
    If $counter = 10 Then; lets get the time for each group of 10 loops

        $time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "
        $counter = 0
        $secs = TimerInit()


    EndIf


Next
$grand = $grand & $str
$str = $grand
$x = FileOpen(@ScriptDir & "\test files\log.txt",1)
FileWriteLine($x,$str & @CRLF)
FileClose($x)
MsgBox(0,"Time in milliseconds for each group of 10 loops",$time)

 

Spoiler

shoot_zpsfd329d66.png
dontbelieveeverythingyouthink_zps0e1e900

Madness is the first step to understanding...

Share this post


Link to post
Share on other sites

Thankyou for your replies.

As beerman suggested - if I alter the code to this

...
...
If $counter = 10 Then; lets get the time for each group of 10 loops
$strX = $strX & $str
$str = ""
$time = $time & StringFormat("%.0f",TimerDiff($secs)) & " , "
$counter = 0
$secs = TimerInit()
EndIf
...
...

I get the following timevalues (milliseconds)

149 , 149 , 156 , 146 , 149 , 152 , 149 , 150 , 148 , 149 , 151 , 151 , 150 , 150 , 151 , 149 , 150 , 151 , 150 , 151

What I am still slightly confused about is this:

I thought that writing to RAM was quick - I understand that AutoIt needs to manage its bit of memory - but are these times taken comparable with other script laguages ? I am not questioning the ability of the devs here - I am jst trying to understand what is happening in order to optimise some scripts.

;)

Thanks for your help.

Share this post


Link to post
Share on other sites

Try this script and see the difference in speed. my test shows that example 2 is 8 times faster.

I haven't looked in the source yet but assume that the first example will move the $var in temp mem space, then add $x at the end of it in temp mem space and then move the whole thing to $var. This is needed or else statements like :$var = $x & $var would fail......

The second example seems to find the "end" of the $var memspace and adds $x to it.

I am sure dev's will chime in when i am incorrect... ;)

$Var = ""

$t = TimerInit()

For $x = 1 to 10000

$var = $var & $x

Next

ConsoleWrite('@@ Debug(6) : TimerDiff($t) = ' & TimerDiff($t) & @lf & '>Error code: ' & @error & @lf) ;### Debug Console

$Var = ""

$t = TimerInit()

For $x = 1 to 10000

$var &= $x

Next

ConsoleWrite('@@ Debug(6) : TimerDiff($t) = ' & TimerDiff($t) & @lf & '>Error code: ' & @error & @lf) ;### Debug Console


Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

@w0uter and @JdeB

what a difference.. ;):P

replacing

$str = $str & Chr(Random(97,122,1))

with

$str &= Chr(Random(97,122,1))

results in

208 , 113 , 113 , 123 , 114 , 113 , 113 , 118 , 113 , 117 , 112 , 114 , 116 , 112 , 116 , 114 , 113 , 115 , 116 , 115 ,

This must be the way to go.

Gents - thankyou

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0