Jump to content

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here. X
X


Photo

arrays with string elements


  • Please log in to reply
25 replies to this topic

#21 randallc

randallc

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 1,902 posts

Posted 01 December 2007 - 10:39 PM

As did I, but it wasn't as fast as I'd hoped, and I think I know why. Somewhere on the forum is a post from the Dev who wrote the initial Array code, including ReDim, and he says that ReDim makes things go real slow because of the way it has to work (internally copying all the data from the old array to a new array of the correct size, etc...). His suggestion was to only ReDim every so often, and in large chunks. I couldn't figure out how to do that in my functions, so I didn't worry so much about it. Since the ReDim is the bottleneck (as you two found), it just goes to show the problem with trying to use dynamically increasing arrays like this. I had another solution, but it didn't work on the project I was working on. Here that one is, for you to play with (based on the Wikipedia article for hash tables).

Hi,
You will see that i had eliminated redim in "dictionary2.au3", but it was still slow; I think it might be because you were using an array to hold an array too, and repeatedly refinig that; I don't think autoIt likes that for speed; see warning in Helpfile)
Best, randall







#22 randallc

randallc

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 1,902 posts

Posted 02 December 2007 - 02:43 AM

So i went with this idea and decided to check how it goes. It doesn't use ReDim at all. It simply uses _SpecialArrayUbound to know where to put next element

Good work!

For your precise problem, you have your answer.

If you already have written your file, it might be faster to use RegExp on the file, as you only need to read the file to a string, not to an array at all;
Then the file read is only about 80msecs,
The search is faster [unless, as with "1131", it is at the beginning of your array], usually about 30msecs wherever in the file.

Best, Randall
AutoIt         
; HashTest6.au3 #include<_ArrayRegExp.au3> Local $aArrayfile[17000], $sStringFile, $aArray1 ;= StringSplit($data, ":") $sFileCsv2 = @ScriptDir & "\arraytester2.csv" ;====================================================== ;~ For $i = 0 To 17000-1 ;~  $aArrayfile[$i]=",Key"&$i&",Value"&$i ;~ Next ;~ _FileWriteFromArray($sFileCsv2,$aArrayfile) ;====================================================== $timer = TimerInit() $fileRead = StringReplace(StringStripCR(FileRead($sFileCsv2)), ",", "|") ConsoleWrite("Read;" & Round(TimerDiff($timer), 2) & "msecs" & @LF) $timer2 = TimerInit() Local $sValue = _HashValue($fileRead, "Key11131", 1, 2); _HashValue(ByRef $sStringFile, $sKey, $KeyColumn, $ValColumn) ConsoleWrite("$sValue=" & $sValue & @LF) ConsoleWrite("Just Find ;" & Round(TimerDiff($timer2), 2) & "msecs" & @LF) Func _HashValue(ByRef $sStringFile, $sKey, $KeyColumn, $ValColumn)     Local $arStrings = _ArrayRegExp($sStringFile, $sKey, $KeyColumn);_FindLinesRegExp($s_FileF, $s_Searches, $i_Column = 0,  $i_Case = 0)     If Not IsArray($arStrings) Or Not StringInStr($arStrings[0], "|") Then Return 0     For $i = 0 To UBound($arStrings) - 1         $aArrayTemp = StringSplit($arStrings[$i], "|")         If $aArrayTemp[$KeyColumn+1] == $sKey Then Return $aArrayTemp[$ValColumn+1]     Next     Return 0 EndFunc   ;==>_HashValue

Edited by randallc, 02 December 2007 - 02:51 AM.


#23 SkinnyWhiteGuy

SkinnyWhiteGuy

    Feed the bunny!

  • Active Members
  • PipPipPipPipPipPip
  • 421 posts

Posted 02 December 2007 - 03:21 AM

Well, building off of what Valik said about avoiding ReDimming constantly, and randallc's point about Arrays in arrays, I have created the following. All function calls are the same, but the internals are a bit different, and the _InitDictionary() function now has an optional parameter that you can use to set the size of the array before using it (it defaults to 1000 currently), and if it needs to grow, it grows by 1.3. I'm still testing this (first test I ran in the application I'm mainly using it in now actually ran slower than my older one), so I'm not sure if all the logic errors are worked out, but they seem to be for the most part working just like earlier.

Attached Files



#24 Nutster

Nutster

    Developer at Large

  • Developers
  • 1,450 posts

Posted 06 December 2007 - 06:57 PM

Somewhere on the forum is a post from the Dev who wrote the initial Array code, including ReDim, and he says that ReDim makes things go really slow because of the way it has to work (internally copying all the data from the old array to a new array of the correct size, etc...). His suggestion was to only ReDim every so often, and in large chunks.

That Dev would be me. In any programming language, you want to minimize the number of the time you have to resize and reassign data structures; this is always a significant time waster.

As I said in my prior post, I did not test the code I posted, beyond Syntax Checker and Tidy not complaining. I should have time this weekend (maybe even tonight) to test it and build an example exerciser using the functions I posted and will post the whole thing in Example Scripts once it is working.

David Nuttall

Nuttall Computer Consulting

An Aquarius born during the Age of Aquarius
AutoIt allows me to re-invent the wheel so much faster.

I'm off to write a wizard, a wonderful wizard of odd...


#25 SkinnyWhiteGuy

SkinnyWhiteGuy

    Feed the bunny!

  • Active Members
  • PipPipPipPipPipPip
  • 421 posts

Posted 07 December 2007 - 08:08 AM

I thought it was you (avatar reminded me), but wasn't completely sure (someone else had that one at one time). I hope your tests go well as mine did.

#26 Nutster

Nutster

    Developer at Large

  • Developers
  • 1,450 posts

Posted 10 December 2007 - 05:49 PM

I should have time this weekend (maybe even tonight) to test it and build an example exerciser using the functions I posted and will post the whole thing in Example Scripts once it is working.

Okay, I posted the tested scripts in Example Scripts. Yeah, I missed a few key details in the version I posted before. Check out this post to see the corrected version.

David Nuttall

Nuttall Computer Consulting

An Aquarius born during the Age of Aquarius
AutoIt allows me to re-invent the wheel so much faster.

I'm off to write a wizard, a wonderful wizard of odd...





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users