Howto speedup Array cleaning up?

[rudi]

Hi.

Perhaps it's best to describe fist, what I want to do:

I have some files of GPS data representing LPG pumps in Europe.

File1: 24h open, EC-Cards accepted

File2: 24h open, cash (includes entries from file1)

File3: 7 days open, EC-Cards accepted (includes entries from File1 and File2)

File4: 7 days open, cash (includes entries from File1, 2, 3)

File5: Mo-Fr, Cards accepted (includes 1-4)

File6: Mo-Fr, cash (and so on... up to file8)

I wrote a working Autoit script to cleaning up file2 to file8, but it's ugly slow. What I do is this:

_FileReadToArray("File1.txt",$UpArray)

_FileReadToArray("File2.txt",$DownArray)

Then I compare the GPS coordinates for each entry ($x) of $UpArray with also each entry ($y) of $DownArray. When they equal, I use _ArrayDelete($DownArray,$y) for cleaning up.

Finally I write back the cleaned up array to File2.txt, then I continue with the next pair of files. (1 against 2 to 8, 2 against 3 to 8, ...)

The program is ugly slow. Is there a faster way to cleanup from a file2 all those lines, that start with the same substring found in any line of file1?

Thanks for any suggestions, Rudi.

PS: More background information:

www.Gas-Tankstellen.de offers the very kind service to download all known LPG petrol pumps from a always up-to-date database. These data can be loaded into several Car GPS systems, and I want to modify the offered data in a way that I will have different icons for "24h, EC-Card accepted". So the files will have to be uniquiified (each LPG station has to show up in only *ONE* file), and therefore I wrote this extremly slow program.

[weaponx]

I don't recommend using _ArrayDelete(). It is a very expensive function because it redimensions your array. Also people tend to misuse it in a loop and then wonder why they see:

"the array subscript or dimension has been exceeded"

Instead of _ArrayDelete maybe just set that element to NULL with "", then when you write the array back to a file just have it skip NULL elements, this will be much faster.

[andybiochem]

I wanted to know the answer to this ages ago, this is what I found...

Instead of using this:

(note the backward steps through the array)

for $i = (UBound($array) - 1) to 1 step -1
    
    if $array[$i] = "remove" Then
        _ArrayDelete($array,$i)
    EndIf

Next

Use this:

(note the forward steps through the array - much nicer!)

$string = ""

for $i = 1 to (UBound($array) - 1)
    
    if $array[$i] <> "remove" Then
        $string &= $array[$i] & "|"
    EndIf

Next

$array = StringSplit($string,"|")
_ArrayPop($array)

I didn't come up with this idea, I read it in another post but can't remember whose. Credit to them anyway; it's an incredibly fast way of removing unwanted elements....or rather, just keeping wanted ones.

[rudi]

I don't recommend using _ArrayDelete(). It is a very expensive function because it redimensions your array. Also people tend to misuse it in a loop and then wonder why they see:

"the array subscript or dimension has been exceeded"

That didn't give me trouble, as after each _ArrayDelete() I did a "MyArr[0]=ubound(MyArr) - 1", it's just about speed...

Instead of _ArrayDelete maybe just set that element to NULL with "", then when you write the array back to a file just have it skip NULL elements, this will be much faster.

That's a very good approach. What do you think ("Speed" point of view) of setting the value of entries to be cleaned up to "False", like this:

Dim $TempString

$MyArray = StringSplit("one,two,tree,four,five,six,seven,eight,nine,ten", ",")
$MyArray[3]=False
$MyArray[5]=False
$MyArray[8]=False
for $i = 1 to $MyArray[0]
    if $MyArray[$i] Then
        $TempString &= $MyArray[$i] & @CRLF ; each line, also the last one, "closed" with CRLF
    EndIf
Next
$hanlde=FileOpen("CleanedUp.txt",2)
FileWrite($hanlde,$TempString)
FileClose($hanlde)

Run("notepad CleanedUp.txt")

Thanks, Rudi.

[edit] How to set an array element to NULL? Array[$x]="" ?

Edited May 25, 2008 by rudi