Need for Speed (Array too Slow), because of ReDim (??)

[michaelslamet]

I have some issues with current array functions:

expandcollapse popup

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

$arr = StringSplit2D ($tdata, @LF, "::")

_ArrayDeleteCol($arr, 1)
_ArrayDeleteCol($arr, 2)
_ArrayDeleteCol($arr, 3)
_ArrayDeleteCol($arr, 4)
_ArrayDeleteCol($arr, 5)
_ArrayDeleteCol($arr, 6)
_ArrayDeleteCol($arr, 7)
ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)

_ArrayDisplay($arr)

Func StringSplit2D($string ,$delimiters1 ,$delimiters2)
    $Array1 = StringSplit ( $string, $delimiters1)
    Dim $MAXUBound = 0 , $OutArray2[1][1]

    For $I = 1 To $Array1[0] Step 1
      $OutArray =  StringSplit ( $Array1[$i], $delimiters2)

      If $MAXUBound < $OutArray[0] Then
         ReDim $OutArray2[$i][$OutArray[0]]
         $MAXUBound = $OutArray[0]
      Else
         ReDim $OutArray2[$i][UBound($OutArray2,2)]
      EndIf

      For $j = 1 To $OutArray[0] Step 1
         $OutArray2[$i - 1][$j - 1] = $OutArray[$j]
      Next
    Next
    Return $OutArray2
EndFunc

Func _ArrayDeleteCol(ByRef $avWork, $iCol)
    If Not IsArray($avWork) Then Return SetError(1, 0, 0); Not an array
    If UBound($avWork, 0) <> 2 Then Return SetError(1, 1, 0); Not a 2D array
    If ($iCol < 0) Or ($iCol > (UBound($avWork, 2) - 1)) Then Return SetError(1, 2, 0); $iCol out of range
    If $iCol < UBound($avWork, 2) - 1 Then
        For $c = $iCol To UBound($avWork, 2) - 2
            For $r = 0 To UBound($avWork) - 1
                $avWork[$r][$c] = $avWork[$r][$c + 1]
            Next
        Next
    EndIf
    ReDim $avWork[UBound($avWork)][UBound($avWork, 2) - 1]
    Return 1
EndFunc

1. It's too slow. Time to run 2000 records = 37.2815

I think that is because ReDim. At least that is what Opt("TrayIconDebug", 1) tell.

2. Result array from StringSplit2D function always have some blank columns, so I need to delete it with _ArrayDeleteCol

Those blank columns should not there at the first place

3. Result array from StringSplit2D function always have 1 blank row at the end.

That blank row/record should not there at the first place

Edit: typo on title

Edited January 3, 2014 by michaelslamet

[guinness]

Look in my signature for a discussion on ReDim. It's under _ReDim.

[Melba23]

michaelslamet,

There are lots of CSV-to-Array functions - which is essentially what you are doing - on the forum. Take a look at some of them and see how they have been constructed.

M23

[mikell]

This ?

expandcollapse popup

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

Local $arr
StringSplit2D ($tdata, $arr, @LF, "::")

ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)
;msgbox(0,"", stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000))

_ArrayDisplay($arr)


Func StringSplit2D($string, ByRef $aArray, $aDelim1 = @crlf, $aDelim2 = ";")
   $string = StringStripWS($string, 3)
   Local $aTemp, $aSplit, $iRows, $iCols
   $aTemp = StringSplit($string, $aDelim1, 3)
   $iRows = UBound($aTemp)
   $iCols = 1
   Dim $aArray[$iRows][$iCols]
   For $i = 0 to $iRows-1
       $aSplit = StringSplit($aTemp[$i], $aDelim2, 3)
       If $iCols < UBound($aSplit) Then   
             $iCols = UBound($aSplit)
             Redim $aArray[$iRows][$iCols]
       EndIf
       For $j = 0 to UBound($aSplit)-1
             $aArray[$i][$j] = $aSplit[$j]
       Next
   Next
EndFunc

Edit : added StringStripWS
 

Edited January 3, 2014 by mikell

[michaelslamet]

This ?

expandcollapse popup

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

Local $arr
StringSplit2D ($tdata, $arr, @LF, "::")

ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)
;msgbox(0,"", stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000))

_ArrayDisplay($arr)


Func StringSplit2D($string, ByRef $aArray, $aDelim1 = @crlf, $aDelim2 = ";")
   $string = StringStripWS($string, 3)
   Local $aTemp, $aSplit, $iRows, $iCols
   $aTemp = StringSplit($string, $aDelim1, 3)
   $iRows = UBound($aTemp)
   $iCols = 1
   Dim $aArray[$iRows][$iCols]
   For $i = 0 to $iRows-1
       $aSplit = StringSplit($aTemp[$i], $aDelim2, 3)
       If $iCols < UBound($aSplit) Then   
             $iCols = UBound($aSplit)
             Redim $aArray[$iRows][$iCols]
       EndIf
       For $j = 0 to UBound($aSplit)-1
             $aArray[$i][$j] = $aSplit[$j]
       Next
   Next
EndFunc

Edit : added StringStripWS

 

 

Wow, Time to run 2000 records = 0.0773 versus my previous function: Time to run 2000 records = 37.2815

That was a HUGE different!

Thanks alot, Mikell. Your modified function works great and very fast!

The ReDim statement only called once versus my previous one that called on every loop (that is why guinness suggest me to read about _ReDim)

Thanks too to guinness and M23.

Thanks again!!