_arraydelete is slow

[Xerix]

Hello

 

I'm looking for a function that would do the same as _arraydelete but faster.

 

The problem of _arraydelete is that it is very slow on large array.

 

Would you have this kind of function?

 

Thank you.

Edited January 16, 2014 by Xerix

[jchd]

No. Maybe there is a better approach to your specific context.

[jdelaney]

How big an array?

Maybe a database would be better...delete statements take fractions of a second.

[Xerix]

Not so great than that.

 

About 5000 lines (in 2D)

 

Maybe a database would be better...delete statements take fractions of a second.

 

 

Probably, but I don't know how to use databases

 

But I think that an array of 5000 line is not unmanageable.

[BrewManNH]

_ArrayDelete isn't optimized, but it does the work it was designed to do. If you want a faster array delete type function you'll probably have to create your own.

[AdamUL]

Are you using _ArrayDelete in a loop, and deleting a lot of lines?  

[Zedna]

Post your small reproducer script and people here can help you optimize it or rewrite it using database (SQLite) or Scripting.Dictionary ...

EDIT:

Maybe this topic will help you as start point

'?do=embed' frameborder='0' data-embedContent>>

Edited January 16, 2014 by Zedna

[Xerix]

Are you using _ArrayDelete in a loop, and deleting a lot of lines?  

Yes this is the case

[jdelaney]

Give us an example.  You can create a function that deletes ALL at one time, which would cut down on the looping to reset the array each time.

[jchd]

Is your data persistant?

[jdelaney]

I made a function that delete all the data in one loop...the results are huge time savings, but only works on a 1D array, for now

expandcollapse popup

#include <array.au3>
Local $aFixed[5000]
For $i = 0 To UBound($aFixed)-1
    $aFixed[$i]=$i+1
Next

$a_ArrayDelete=$aFixed
$a_ArrayDeleteMult=$aFixed

; Setup array to delete multiple items...sort descending, so you delete proper items
Local $aDeletes[50]
For $i = 0 To UBound($aDeletes)-1
    $aDeletes[$i] = $i*5
Next


; Time _arraydelete for 50 deletes
$iArrayDelete = TimerInit()
For $i = 0 To UBound($aDeletes)-1
    _ArrayDelete($a_ArrayDelete,$aDeletes[$i])
Next
ConsoleWrite("_arraydelete: " & TimerDiff($iArrayDelete) & @CRLF)

$iArrayDeleteMulti = TimerInit()
_ArrayDeleteMulti($a_ArrayDeleteMult,$aDeletes)
ConsoleWrite("_arraydeleteMulti: " & TimerDiff($iArrayDeleteMulti) & @CRLF)

;~ _ArrayDisplay($a_ArrayDelete)
;~ _ArrayDisplay($a_ArrayDeleteMult)

For $i = 0 To UBound($a_ArrayDeleteMult)-1
    If $a_ArrayDelete[$i] <> $a_ArrayDeleteMult[$i] Then ConsoleWrite("wrong" & @CRLF)
Next

Exit

Func _ArrayDeleteMulti(ByRef $avArray, $aDeletes)

    If Not IsArray($avArray) Then Return SetError(1, 0, 0)

    Local $iUBound = UBound($avArray, 1) - 1

    $iMoves = 0
    For $j = 0 To UBound($aDeletes)-1
        If $j = UBound($aDeletes)-1 Then
            $iMax = $iUBound-UBound($aDeletes)
        Else
            $iMax = $aDeletes[$j+1]
        EndIf
        For $i = $aDeletes[$j] To $iMax
            $avArray[$i] = $avArray[$i+$j+1]
        Next
    Next

    ReDim $avArray[UBound($avArray)-UBound($aDeletes)]

    Return True
EndFunc   ;==>_ArrayDelete

output:

_arraydelete: 568.492675738927
_arraydeleteMulti: 13.6767694278312

Edited January 16, 2014 by jdelaney

[Xerix]

Here's a simple example of what I would like to make

The idea is to remove any lines or appears the word doc in the 4th column.

Well here is a quick example but the idea is that the doc may be found in any line.

Not necessarily as here where the doc are all before the txt.

Thanks for your help

 

#include "array.au3"


Global $array[5000][6]


For $i = 0 To 4500
$array[$i][0] = $i
$array[$i][1] = "file" & $i
$array[$i][2] = "date" & $i
$array[$i][3] = "doc"
$array[$i][4] = "name" & $i
$array[$i][5] = "color" & $i
Next


For $i = 4501 To 4999
$array[$i][0] = $i
$array[$i][1] = "file" & $i
$array[$i][2] = "date" & $i
$array[$i][3] = "txt"
$array[$i][4] = "name" & $i
$array[$i][5] = "color" & $i
Next

_ArrayDisplay($array)

$ext = _ArrayFindAll($array, "doc", 0, 0, 0, 0, 3)
For $i = UBound($ext) - 1 To 0 Step -1
_ArrayDelete($array, $ext[$i])
Next

_ArrayDisplay($array)

[BrewManNH]

Try this, you'll see that it's much faster. Although it does presuppose you know the dimensions of the array to check.

expandcollapse popup

#include "array.au3"
Global $array[5000][6]

For $i = 0 To 4500
    $array[$i][0] = $i
    $array[$i][1] = "file" & $i
    $array[$i][2] = "date" & $i
    $array[$i][3] = "doc"
    $array[$i][4] = "name" & $i
    $array[$i][5] = "color" & $i
Next

For $i = 4501 To 4999
    $array[$i][0] = $i
    $array[$i][1] = "file" & $i
    $array[$i][2] = "date" & $i
    $array[$i][3] = "txt"
    $array[$i][4] = "name" & $i
    $array[$i][5] = "color" & $i
Next

_ArrayDisplay($array)

;~ $ext = _ArrayFindAll($array, "doc", 0, 0, 0, 0, 3)
;~ For $i = UBound($ext) - 1 To 0 Step -1
;~  _ArrayDelete($array, $ext[$i])
;~ Next
Local $aTemp[5000][6], $X = 0
For $i = 1 To UBound($array) - 1
    If $array[$i][3] <> "doc" Then
        For $j = 0 To 5
            $aTemp[$X][$j] = $array[$i][$j]
        Next
        $X += 1
    EndIf
Next
ReDim $aTemp[$X][6]
$array = $aTemp

_ArrayDisplay($array)

[Xerix]

Thank you BrewManNH

So the idea is to transfer all the txt file in an another temporary array then replace this temporary array values in the main array.

 

And indeed it is extremely faster

 

Can you explain in $X += 1 the +=

I think have already asked the question in another message but I don't know where.

 

Thank you

[Xerix]

Just a small correction.

To find the exact number should we not have

For $i = 0 To UBound($array) - 1

Rather than

For $i = 1 To UBound($array) - 1

And maybe

Local $aTemp[UBound($array)][6], $X = 0

Edited January 17, 2014 by Xerix

[MHz]

$X += 1
; is the same as
$X = $X + 1

Look at the 2nd parameter of UBound to set dimension to get the size. UBound($array, 1) 1st dimension (default), UBound($array, 2) 2nd dimension ...

[BrewManNH]

Just a small correction.

 

To find the exact number should we not have

For $i = 0 To UBound($array) - 1

Yes, you're right, small error on my part.

Rather than

For $i = 1 To UBound($array) - 1

And maybe

Local $aTemp[UBound($array)][6], $X = 0

As I had said, this would only work if you knew what size the incoming array would be. It was just an example to get the OP started in the right direction. _ArrayDelete is terrible when used in a loop with a lot of deletions, but that isn't exactly how it was intended to be used.

[Xerix]

OK, thank you all.

 

As we talked, what would be the code on this example with databases.

 

Thank you

[guinness]

The Array.au3 I feel should be only used for simple array manipulation if you're scared to get your feet wet with using arrays as they are.