Jump to content
Sign in to follow this  

Faster than nested For loops?

Recommended Posts



I'm trying to normalize data from 4 different multi-dimension arrays, but they are extremely large and I'm trying to learn a faster way of parsing data. Please note I am rather newb at programming.


The following script does exactly what I need, but its really slow. Is there a better way to parse data than nested For loops?


Basically the logic is:

1.       For All store/lanes in the Quarterly report, find any matches in Current day’s Lane issues 

2.       (if any), then if lane issues exist and they are in the quarterly report, find match in Revenue report

3.       (if any), then if lane issues exist and they are in the quarterly report, that have contract values from Revenue report, then perform calculations and output It


  I cannot provide the documents due to sensitive data.

  Quarterly Report array is about 80,000 Rows and 13 Columns

  Store/lane Issues array is about 25,000 Rows and 3 Columns

  Revenue array report is about 100 Rows and 13 Columns

  National Store/Retailer translation array is about 30,000 Rows and 3 Columns


  Current output takes roughly 0.5 seconds per cycle, so it would be about 11 hours for all 80,000 rows.


  Any help or advice would be greatly appreciated!!



$date = StringReplace(_NowCalcDate(), "/", "-")
$ROI_output = @ScriptDir & "\ROI_output_" & $date & ".txt"
$ROI_output1 = FileOpen($ROI_output, 1)
FileWrite($ROI_output1, "Retailer" & " | " & "Chain-Store" & " | " & "Store Value" & " | " &  "Lane" & " | " & "Lane Value"  & " | " & "Status"  & " | " & "StoreWeight" & " | " & "LaneWeight" & " | " & "RetailerValue " & @CRLF)

;Quarterly report
$lanestoreweights = "Z:\STAT\lanestoreweights.xlsx"
Local $oApplD2 = _Excel_Open(False);
Local $oExcelD2 = _Excel_BookOpen($oApplD2, $lanestoreweights)
Local $lanestoreweightsarray = _Excel_RangeRead($oExcelD2, Default, $oExcelD2.ActiveSheet.Usedrange.Columns("C:M"), 1, True)

;Current Day's lane issues
$storelanestatuses = "Z:\STAT\Store_Lane_values-" & $date & ".txt"
Local $storelanearray
_FileReadToArray($storelanestatuses, $storelanearray)

;Revenue Report
$rev =  "Z:\STAT\OI_Revenue_Report_PipeDelimited.txt"
Local $revarray
_FileReadToArray($rev, $revarray)

;Chain to Retailer translation
Local $nationallocal = "Z:\STAT\national.xls"
Local $oApplD1 = _Excel_Open(False)
Local $oExcelD1 = _Excel_BookOpen($oApplD1, $nationallocal)
Local $livestores = _Excel_RangeRead($oExcelD1, Default, $oExcelD1.ActiveSheet.Usedrange.Columns("A:C"), 1, True)

For $z = 2 To UBound($lanestoreweightsarray) -1
    $split2 = StringSplit($lanestoreweightsarray[$z][0], "-")
    If UBound($split2) > 3 Then
      $wtChain = StringFormat("%03i", $split2[1])
      $wtStore = StringFormat("%04i", $split2[2])
        $wtLane = StringFormat("%02i", $split2[3])

        $storeweight1 = ""
        $laneweight1 = ""
        $rev_retailer = ""
        $rev_val = ""
        For $x = 1 To UBound($storelanearray) -1
            $split = StringSplit($storelanearray[$x], " ")
            $chain = $split[3]
            $store = $split[4]
            $lane = $split[6]
            $statfilesplit = StringSplit($storelanearray[$x], "***")
            $lanestatus = $statfilesplit[4]

            If $chain = $wtChain  And $store = $wtStore And $lane = $wtLane Then
                $storeweight = $lanestoreweightsarray[$z][9]
                $storeweight1 = $storeweight * 100
                $laneweight = $lanestoreweightsarray[$z][10]
                $laneweight1 = $laneweight * 100
                For $y = 1 To UBound($livestores) -1
                    If $chain = StringFormat("%03i", $livestores[$y][2]) Then
                        If $livestores[$y][0] = "No Chain Group Assigned." Then
                            $retailer =  $livestores[$y][1]
                            $retailer =  $livestores[$y][0]

            For $r = 2 To UBound($revarray) -1
                $split3 = StringSplit($revarray[$r], "|")

                If $rev_retailer = "" Then
                    If $split3[1] = "" Then
                        ;ConsoleWrite($retailer & "---" & $split3[3] & @CRLF)
                        If $split3[3] = $retailer Then
                            $rev_retailer = $split3[3]
                            $rev_val = StringLeft($split3[13], (StringLen($split3[13]) - 2))

                        ;ConsoleWrite($retailer & "-" & $split3[1] & @CRLF)
                        If $split3[1] = $retailer Then

                            $rev_retailer = $split3[1]
                            $rev_val = StringLeft($split3[13], (StringLen($split3[13]) - 2))

            If $rev_val = "" Then
                $store_value = $rev_val * $storeweight1
                $printer_value = $store_value * $laneweight1
                ConsoleWrite($retailer & " | " & $chain & "-" & $store & " | " & "Store Value: " & $store_value & " | " &  "Lane: " & $lane & " | " & "Lane Value: " & $printer_value  & " | " & "Status: " & $lanestatus & " | " & "StoreWeight: " & $storeweight1 & " | " & "LaneWeight: " & $laneweight1 & " | " & "RetailerVal: " & $rev_val & @CRLF)
                FileWrite($ROI_output1, $retailer & " | " & $chain & "-" & $store & " | " & $store_value & " | " &  $lane & " | " & $printer_value  & " | " & $lanestatus & " | " &  $storeweight1 & " | " &  $laneweight1 & " | " & $rev_val & @CRLF)


Share this post

Link to post
Share on other sites

I haven't used SQLite myself.  But I would think it a more efficient tool than scanning arrays by hand.  Once you figure out how to import the excel data into the SQLite database you could run queries to generate the reports instead of rolling your own.

I would search for any SQLite examples esp. involving excel.


Edited by MilesAhead
  • Like 1

Share this post

Link to post
Share on other sites

Right on; I dont have much SQL experience but gotta learn some time. 


Share this post

Link to post
Share on other sites

Take a look at this post. The optimizations in this post are based on sorted arrays where you search the rows with a binary search algorithm. If your data can be sorted in the same way, so that it's possible to use binary search, your For-loops can be very optimized. If your arrays can be properly sorted it should be possible to reduce the 11 hours to a few minutes.

If it's possible for you to produce some test data with about 5% of the amount of rows, then I'll try to optimize the For-loops. It's important that the test data are relatively realistic in terms of sorting options.

  • Like 1

Share this post

Link to post
Share on other sites

Thank you LarsJ! Thats an excellent suggestion and great link. I'll provide some example files later today then try to practice sorting.


Thank you again!



Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • rm4453
      By rm4453
      I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.
      I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.
      Any help at all would be very much appreciated!
      *EDIT --> The array I am left with after parsing is $array[0-50000][16]
    • AnonymousX
      By AnonymousX
      So this may be more of a challenge of effective programming then specific to AutoIT but I want to solve this problem with AutoIT  so i'm putting it here. (If someone has a better language to solve with I'm all ears)
      So the task I'm trying to achieve is that I have multiple .CSV files that have: year, month, day, hour, value. I need to be able to sum up all the values that have the same date/time, then find which date and time has the maximum value.
      The problem is that each file may or may not have same amount of days/hours as the rest. So I need to devise a way to handle this. 
      File A   File B   File C 2018 1 1  1:00 10   2018 1 1 2:00 10   2018 1 1  1:00 10 2018 1 1  2:00 12   2018 1 1 3:00 12   2018 1 2 1:00 12 2018 1 1  3:00 14   2018 1 1 4:00 14   2018 2 1  1:00 16 2018 2 1  1:00 16   2018 2 1  1:00 16              
       Answer I want to be spit out is Feb 1st 2018 at 2:00 with value of 48
      So far I've got code to store all .CSV files to an array, then a loop to go through each csv, but not sure how to effectively manipulate the data. Keep in mind each file has over 7000 time entry points.
      If anyone can solve this that would be pretty awesome! 
      #include <Array.au3> #include <File.au3> #include <MsgBoxConstants.au3> RefineData() Func RefineData() Local $i, $filenum, $file, $csvArray, $FilePath = @ScriptDir $fileList = _FileListToArrayRec($FilePath, "*.csv", 1) ;Create and array of all .csv files within folder Local $chkArray[UBound($fileList)][2] ;=====Loop through the .csv files within the folder====== For $filenum = 1 To UBound($fileList) - 1 Step 1 $file = $fileList[$filenum] $sFilePath = $FilePath & "\" & $file ;=====Create array based on csv file===== _FileReadToArray($sFilePath, $csvArray, $FRTA_NOCOUNT, ",") ;#### Operations here ###### next msgbox(0,"", "Date: " & $date_of_max & "Value: " & $maxVal );display solution endfunc  
    • corz
      By corz
      Associative Array Functions
      I've seen a couple of UDFs for this on the forum. One of them I quite like. But it's still nearly not as good as this method, IMHO.
      I don't recall if I discovered the "Scripting.Dictionary" COM object myself or if I got the original base code from somewhere online. I have recently searched the web (and here) hard for any AutoIt references to this, other than my own over the years I've been using this (in ffe, etc..), and I can find nothing, so I dunno. If anyone does, I'd love to give credit where it's due; this is some cute stuff! It could actually be all my own work! lol
      At any rate, it's too useful to not have posted somewhere at autoitscript.com, so I've put together a wee demo.
      For those who haven't heard of the COM "Scripting.Dictionary".. 
      If you've ever coded in Perl or PHP (and many other languages), you know how useful associative arrays are. Basically, rather than having to iterate through an array to discover it's values, with an associative array you simply pluck values out by their key "names".
      I've added a few functions over the years, tweaked and tuned, and this now represent pretty much everything you need to easily work with associative arrays in AutoIt. En-joy!
      The main selling point of this approach is its simplicity and weight. I mean, look at how much code it takes to work with associative arrays! The demo is bigger than all the functions put together! The other selling point is that we are using Windows' built-in COM object functions which are at least theoretically, fast and robust.
      I've used it many times without issues, anyhow, here goes..
      ; Associative arrays in AutoIt? Hells yeah! ; Initialize your array ... global $oMyError = ObjEvent("AutoIt.Error", "AAError") ; Initialize a COM error handler ; first example, simple. global $simple AAInit($simple) AAAdd($simple, "John", "Baptist") AAAdd($simple, "Mary", "Lady Of The Night") AAAdd($simple, "Trump", "Silly Man-Child") AAList($simple) debug("It is said that Trump is a " & AAGetItem($simple, "Trump") & ".", @ScriptLineNumber);debug debug("") ; slightly more interesting.. $ini_path = "AA_Test.ini" ; Put this prefs section in your ini file.. ; [test] ; foo=foo value ; foo2=foo2 value ; bar=bar value ; bar2=bar2 value global $associative_array AAInit($associative_array) ; We are going to convert this 2D array into a cute associative array where we ; can access the values by simply using their respective key names.. $test_array = IniReadSection($ini_path, "test") for $z = 1 to 2 ; do it twice, to show that the items are *really* there! for $i = 1 to $test_array[0][0] $key_name = $test_array[$i][0] debug("Adding '" & $key_name & "'..");debug ; key already exists in "$associative_array", use the pre-determined value.. if AAExists($associative_array, $key_name) then $this_value = AAGetItem($associative_array, $key_name) debug("key_name ALREADY EXISTS! : =>" & $key_name & "<=" , @ScriptLineNumber);debug else $this_value = $test_array[$i][1] ; store left=right value pair in AA if $this_value then AAAdd($associative_array, $key_name, $this_value) endif endif next next debug(@CRLF & "Array Count: =>" & AACount($associative_array) & "<=" , @ScriptLineNumber);debug AAList($associative_array) debug(@CRLF & "Removing 'foo'..");debug AARemove($associative_array, "foo") debug(@CRLF & "Array Count: =>" & AACount($associative_array) & "<=" , @ScriptLineNumber);debug AAList($associative_array) debug(@CRLF & "Removing 'bar'..");debug AARemove($associative_array, "bar") debug(@CRLF & "Array Count: =>" & AACount($associative_array) & "<=" , @ScriptLineNumber);debug AAList($associative_array) quit() func quit() AAWipe($associative_array) AAWipe($simple) endfunc ;; Begin AA Functions func AAInit(ByRef $dict_obj) $dict_obj = ObjCreate("Scripting.Dictionary") endfunc ; Adds a key and item pair to a Dictionary object.. func AAAdd(ByRef $dict_obj, $key, $val) $dict_obj.Add($key, $val) If @error Then return SetError(1, 1, -1) endfunc ; Removes a key and item pair from a Dictionary object.. func AARemove(ByRef $dict_obj, $key) $dict_obj.Remove($key) If @error Then return SetError(1, 1, -1) endfunc ; Returns true if a specified key exists in the associative array, false if not.. func AAExists(ByRef $dict_obj, $key) return $dict_obj.Exists($key) endfunc ; Returns a value for a specified key name in the associative array.. func AAGetItem(ByRef $dict_obj, $key) return $dict_obj.Item($key) endfunc ; Returns the total number of keys in the array.. func AACount(ByRef $dict_obj) return $dict_obj.Count endfunc ; List all the "Key" > "Item" pairs in the array.. func AAList(ByRef $dict_obj) debug("AAList: =>", @ScriptLineNumber);debug local $k = $dict_obj.Keys ; Get the keys ; local $a = $dict_obj.Items ; Get the items for $i = 0 to AACount($dict_obj) -1 ; Iterate the array debug($k[$i] & " ==> " & AAGetItem($dict_obj, $k[$i])) next endfunc ; Wipe the array, obviously. func AAWipe(ByRef $dict_obj) $dict_obj.RemoveAll() endfunc ; Oh oh! func AAError() Local $err = $oMyError.number If $err = 0 Then $err = -1 SetError($err) ; to check for after this function returns endfunc ;; End AA Functions. ; debug() (trimmed-down version) ; ; provides quick debug report in your console.. func debug($d_string, $ln=false) local $pre ; For Jump-to-Line in Notepad++ if $ln then $pre = "(" & $ln & ") " & @Tab ConsoleWrite($pre & $d_string & @CRLF) endfunc  
      ;o) Cor
    • TheWizEd
      By TheWizEd
      How do I work with 2D arrays.  I've tried this but get errors.
      Local $aTest[4][4] = [[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]]
      ;$aTest[0][] = [10,11,12]  ; Error at []
      Local $sTest = ""
      For $i = 0 To UBound($aTest)-1
        Local $aExtract = _ArrayExtract($aTest,$i,$i)
        $sTest = $sTest & MyTest($aExtract)
      Func MyTest($aTemp)
        ; Error at    v $aTemp
        Return String($aTemp[0]) & " - " & String($aTemp[1]) & " - " & String($aTemp[2]) & @CRLF
    • Eli_jahbot
      By Eli_jahbot
      My esteemed Autoits I need your help once again.
      I'm trying to figure out how to create a loop that gets 1 value from an array and repeats until each value from the array has been used. I have never used arrays before and I know once I learn more things should get easier for me. 
      Here is what Im trying to do:
      -Have an array of values that determine what application i log into. ex: app1, app2, app3, app4 etc.
      -Have a loop that repeats a process sequentially using each value in the array to finish the process for each app1, app2, app 3 and so forth. I have 30 apps that I need to update on a regular basis and getting this sorted out is what I perceive to be the best way to do it.
      Here is my feeble attempt that obviously fails:
      #include <msgboxconstants.au3>
      #include <Constants.au3>
      #include <array.au3>
      Func Login()
          local $array[30] = ["10", "11", "12",etc etc]
          ;;Local $site = InputBox("ERx Site","What site do you want to login as?","","")
          Local $userid = InputBox("ERx Login", "What is your username?", "", "")
          Local $Passwd = InputBox("Security Check", "Enter your UAT password.", "", "*")
      for $1 = 1 to 30(I need to do the same steps in 30 different apps)
      Send (Sequential ARRAY VALUE HERE)
      WinClose("Home Page")
      your help is greatly appreciated.
      Thanks for your time