Speeding Up Script

[mcfr1es]

I was wondering if there is a better method (faster) to do my script:

NOTE: The url below is of a forum

[Algorithm]

get URL

search url for 50 usernames that are contained within the page

usernames returned in a 50 element array

check to see if usernames in the array exist already in a text file

if not add them, if so discard

wash rinse repeat

[/Algorithm]

As you can probably guess, towards the early-mid part of the script, it begins running slowly as the usernames about to be added will be checked against the MANY usernames contained within the text file.

I was considering making an array with all of the usernames and adding it to the text file at the end but I am not sure if that will help much or if it will be equivilent to the method I am using now.

Any thoughts?

Edited August 15, 2004 by mcfr1es

[pekster]

It will speed the script up a lot if you load the current list from the file into an array, and then cycle through the array each time you want to test for an existing user.

If the user does exist, move on to the next potential user. If not, re-dimention the array to add another element on the end, and add in the proper info. When you are finished you can write the array back to the file.

When you keep the script it memory like this it will run a lot faster since it doesn't have to read/write to the hard drive for each user to check or add.

[mcfr1es]

ty for your quick reply (that I was both hoping for and dreading )

back to the drawing board

Edited August 15, 2004 by mcfr1es

[pekster]

Here's a small example for you. It does not deal with reading input from files, but it does demonstrate one potential method to testing if an item exists in an array, and adding in new elements as new items not in the list are created. Here you go:

MsgBox(0, "Sample array check script", "To exit the script, enter the "&_
 "number 0 into the input box." & @LF & "For all other numerical enteries "&_
 "the script will tell you if the number already has been entered or not."&_
 @LF & "Note: text counts as a 0, and will quit")

Global $array[1]

While 1
  $check = InputBox("Number Entry", "Enter a number ( 0 quits)", "1")
  $check = Number($check)
  If $check = 0 Then
    MsgBox(0, "Goodbye", "Now quitting.  Bye")
    Exit
  EndIf
  $exist = 0;reset the exist flag
  For $i = 0 To UBound($array) - 1
    If $check = $array[$i] Then
      MsgBox(0, "Already There!", "That number has already been entered.")
      $exist = 1;set an exist flag so we don't add it again
      ExitLoop;Get out of the for loop since it does exist
    EndIf
  Next
  If $exist Then ContinueLoop;don't add the number again if it existed
  ReDim $array[UBound($array) + 1];make room for another entry
  $array[UBound($array) - 1] = $check;add in the new number
  MsgBox(0, "New Number Added", "Your entry has been added")
WEnd

[mcfr1es]

thank you pekster but I had already finished my script before I saw your post and Did it Much faster this time ! I happy to say that my script looks almost identical to that except for the ContinueLoop Part (I used embedded loops) which I will go have a look at right now. Also, although the writing/reading part is much faster, the script is slowing down somewhere else resulting in this script going just as fast as my other. This is most likely a programming error on my part which will be hopefully fixed soon.

NOTE: I already fully ran my other script and got the results I wanted, (5000 unique Usernames) Muhahahahah

Anyway, this revision is just for educational purposes

Edited August 16, 2004 by mcfr1es

[pekster]

Also,  although the writing/reading part is much faster, the script is slowing down somewhere else resulting in this script going just as fast as my other. This is most likely a programming error on my part    which will be hopefully fixed soon.

<{POST_SNAPBACK}>

In any script that has to check a new entry against a growing list of current enteries, the script will become slower as execution time increases. This is simply due to the added looping that must be done to determine if an entry exists in the database that becomes larger with each addition.

[Nutster]

You could add a binary search and a sorting algorithum (an insertion sort works well here) to speed up searchs of the list.