Sign in to follow this  
Followers 0
michaelslamet

Need for Speed (Array too Slow), because of ReDim (??)

5 posts in this topic

#1 ·  Posted (edited)

I have some issues with current array functions:

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

$arr = StringSplit2D ($tdata, @LF, "::")

_ArrayDeleteCol($arr, 1)
_ArrayDeleteCol($arr, 2)
_ArrayDeleteCol($arr, 3)
_ArrayDeleteCol($arr, 4)
_ArrayDeleteCol($arr, 5)
_ArrayDeleteCol($arr, 6)
_ArrayDeleteCol($arr, 7)
ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)

_ArrayDisplay($arr)

Func StringSplit2D($string ,$delimiters1 ,$delimiters2)
    $Array1 = StringSplit ( $string, $delimiters1)
    Dim $MAXUBound = 0 , $OutArray2[1][1]

    For $I = 1 To $Array1[0] Step 1
      $OutArray =  StringSplit ( $Array1[$i], $delimiters2)

      If $MAXUBound < $OutArray[0] Then
         ReDim $OutArray2[$i][$OutArray[0]]
         $MAXUBound = $OutArray[0]
      Else
         ReDim $OutArray2[$i][UBound($OutArray2,2)]
      EndIf

      For $j = 1 To $OutArray[0] Step 1
         $OutArray2[$i - 1][$j - 1] = $OutArray[$j]
      Next
    Next
    Return $OutArray2
EndFunc

Func _ArrayDeleteCol(ByRef $avWork, $iCol)
    If Not IsArray($avWork) Then Return SetError(1, 0, 0); Not an array
    If UBound($avWork, 0) <> 2 Then Return SetError(1, 1, 0); Not a 2D array
    If ($iCol < 0) Or ($iCol > (UBound($avWork, 2) - 1)) Then Return SetError(1, 2, 0); $iCol out of range
    If $iCol < UBound($avWork, 2) - 1 Then
        For $c = $iCol To UBound($avWork, 2) - 2
            For $r = 0 To UBound($avWork) - 1
                $avWork[$r][$c] = $avWork[$r][$c + 1]
            Next
        Next
    EndIf
    ReDim $avWork[UBound($avWork)][UBound($avWork, 2) - 1]
    Return 1
EndFunc

1. It's too slow. Time to run 2000 records = 37.2815

I think that is because ReDim. At least that is what Opt("TrayIconDebug", 1) tell.

2. Result array from StringSplit2D function always have some blank columns, so I need to delete it with _ArrayDeleteCol

Those blank columns should not there at the first place

3. Result array from StringSplit2D function always have 1 blank row at the end.

That blank row/record should not there at the first place

Edit: typo on title

Edited by michaelslamet

Share this post


Link to post
Share on other sites



Look in my signature for a discussion on ReDim. It's under _ReDim.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

michaelslamet,

There are lots of CSV-to-Array functions - which is essentially what you are doing - on the forum. Take a look at some of them and see how they have been constructed. ;)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

This ?

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

Local $arr
StringSplit2D ($tdata, $arr, @LF, "::")

ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)
;msgbox(0,"", stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000))

_ArrayDisplay($arr)


Func StringSplit2D($string, ByRef $aArray, $aDelim1 = @crlf, $aDelim2 = ";")
   $string = StringStripWS($string, 3)
   Local $aTemp, $aSplit, $iRows, $iCols
   $aTemp = StringSplit($string, $aDelim1, 3)
   $iRows = UBound($aTemp)
   $iCols = 1
   Dim $aArray[$iRows][$iCols]
   For $i = 0 to $iRows-1
       $aSplit = StringSplit($aTemp[$i], $aDelim2, 3)
       If $iCols < UBound($aSplit) Then   
             $iCols = UBound($aSplit)
             Redim $aArray[$iRows][$iCols]
       EndIf
       For $j = 0 to UBound($aSplit)-1
             $aArray[$i][$j] = $aSplit[$j]
       Next
   Next
EndFunc

Edit : added StringStripWS
 

Edited by mikell
1 person likes this

Share this post


Link to post
Share on other sites

This ?

#include <array.au3>
#include <date.au3>

Opt("TrayIconDebug", 1)

Local $tdata, $arr

; generate data for testing
For $1 = 1 to 2000
    $tdata &= 'user' & stringformat('%04i',random(1,1000,1)) & ' :: ' & 'www.'
    for $2 = 1 to random(5,16,1)
        $tdata &= chr(random(97,122,1))
    Next
    $tdata &= '.com :: '
    $tdata &= _dateadd('D',random(-365,365,1),_NowCalcDate())
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1)
    $tdata &= ' :: ' & random(1000,100000,1) & @LF
Next
; ==> generate data for testing

local $st = timerinit()

Local $arr
StringSplit2D ($tdata, $arr, @LF, "::")

ConsoleWrite(stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000) & @lf)
;msgbox(0,"", stringformat('Time to run %03i records = %2.4f',$1-1,timerdiff($st)/1000))

_ArrayDisplay($arr)


Func StringSplit2D($string, ByRef $aArray, $aDelim1 = @crlf, $aDelim2 = ";")
   $string = StringStripWS($string, 3)
   Local $aTemp, $aSplit, $iRows, $iCols
   $aTemp = StringSplit($string, $aDelim1, 3)
   $iRows = UBound($aTemp)
   $iCols = 1
   Dim $aArray[$iRows][$iCols]
   For $i = 0 to $iRows-1
       $aSplit = StringSplit($aTemp[$i], $aDelim2, 3)
       If $iCols < UBound($aSplit) Then   
             $iCols = UBound($aSplit)
             Redim $aArray[$iRows][$iCols]
       EndIf
       For $j = 0 to UBound($aSplit)-1
             $aArray[$i][$j] = $aSplit[$j]
       Next
   Next
EndFunc

Edit : added StringStripWS

 

 

Wow, Time to run 2000 records = 0.0773 versus my previous function: Time to run 2000 records = 37.2815

That was a HUGE different!

Thanks alot, Mikell. Your modified function works great and very fast!

The ReDim statement only called once versus my previous one that called on every loop (that is why guinness suggest me to read about _ReDim)

Thanks too to guinness and M23.

Thanks again!!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0