Jump to content
Sign in to follow this  
DrewSS

Faster than nested For loops?

Recommended Posts

Hello,

I'm trying to normalize data from 4 different multi-dimension arrays, but they are extremely large and I'm trying to learn a faster way of parsing data. Please note I am rather newb at programming.

 

The following script does exactly what I need, but its really slow. Is there a better way to parse data than nested For loops?

 

Basically the logic is:

1.       For All store/lanes in the Quarterly report, find any matches in Current day’s Lane issues 

2.       (if any), then if lane issues exist and they are in the quarterly report, find match in Revenue report

3.       (if any), then if lane issues exist and they are in the quarterly report, that have contract values from Revenue report, then perform calculations and output It

 

  I cannot provide the documents due to sensitive data.

  Quarterly Report array is about 80,000 Rows and 13 Columns

  Store/lane Issues array is about 25,000 Rows and 3 Columns

  Revenue array report is about 100 Rows and 13 Columns

  National Store/Retailer translation array is about 30,000 Rows and 3 Columns

 

  Current output takes roughly 0.5 seconds per cycle, so it would be about 11 hours for all 80,000 rows.

 

  Any help or advice would be greatly appreciated!!

 

 

$date = StringReplace(_NowCalcDate(), "/", "-")
$ROI_output = @ScriptDir & "\ROI_output_" & $date & ".txt"
$ROI_output1 = FileOpen($ROI_output, 1)
FileWrite($ROI_output1, "Retailer" & " | " & "Chain-Store" & " | " & "Store Value" & " | " &  "Lane" & " | " & "Lane Value"  & " | " & "Status"  & " | " & "StoreWeight" & " | " & "LaneWeight" & " | " & "RetailerValue " & @CRLF)

;Quarterly report
$lanestoreweights = "Z:\STAT\lanestoreweights.xlsx"
Local $oApplD2 = _Excel_Open(False);
Local $oExcelD2 = _Excel_BookOpen($oApplD2, $lanestoreweights)
Local $lanestoreweightsarray = _Excel_RangeRead($oExcelD2, Default, $oExcelD2.ActiveSheet.Usedrange.Columns("C:M"), 1, True)
_Excel_BookClose($oApplD2)

;Current Day's lane issues
$storelanestatuses = "Z:\STAT\Store_Lane_values-" & $date & ".txt"
Local $storelanearray
_FileReadToArray($storelanestatuses, $storelanearray)

;Revenue Report
$rev =  "Z:\STAT\OI_Revenue_Report_PipeDelimited.txt"
Local $revarray
_FileReadToArray($rev, $revarray)

;Chain to Retailer translation
Local $nationallocal = "Z:\STAT\national.xls"
Local $oApplD1 = _Excel_Open(False)
Local $oExcelD1 = _Excel_BookOpen($oApplD1, $nationallocal)
Local $livestores = _Excel_RangeRead($oExcelD1, Default, $oExcelD1.ActiveSheet.Usedrange.Columns("A:C"), 1, True)
_Excel_BookClose($oApplD1)



For $z = 2 To UBound($lanestoreweightsarray) -1
    $split2 = StringSplit($lanestoreweightsarray[$z][0], "-")
    If UBound($split2) > 3 Then
      $wtChain = StringFormat("%03i", $split2[1])
      $wtStore = StringFormat("%04i", $split2[2])
        $wtLane = StringFormat("%02i", $split2[3])
     EndIf

        $storeweight1 = ""
        $laneweight1 = ""
        $rev_retailer = ""
        $rev_val = ""
        For $x = 1 To UBound($storelanearray) -1
            $split = StringSplit($storelanearray[$x], " ")
            $chain = $split[3]
            $store = $split[4]
            $lane = $split[6]
            $statfilesplit = StringSplit($storelanearray[$x], "***")
            $lanestatus = $statfilesplit[4]

            If $chain = $wtChain  And $store = $wtStore And $lane = $wtLane Then
                $storeweight = $lanestoreweightsarray[$z][9]
                $storeweight1 = $storeweight * 100
                $laneweight = $lanestoreweightsarray[$z][10]
                $laneweight1 = $laneweight * 100
                
                For $y = 1 To UBound($livestores) -1
                    If $chain = StringFormat("%03i", $livestores[$y][2]) Then
                        If $livestores[$y][0] = "No Chain Group Assigned." Then
                            $retailer =  $livestores[$y][1]
                            ExitLoop
                        Else
                            $retailer =  $livestores[$y][0]
                            ExitLoop
                        EndIf
                    EndIf
                Next

            For $r = 2 To UBound($revarray) -1
                $split3 = StringSplit($revarray[$r], "|")

                If $rev_retailer = "" Then
                    If $split3[1] = "" Then
                        ;ConsoleWrite($retailer & "---" & $split3[3] & @CRLF)
                        If $split3[3] = $retailer Then
                            $rev_retailer = $split3[3]
                            $rev_val = StringLeft($split3[13], (StringLen($split3[13]) - 2))
                            ExitLoop
                        EndIf

                    Else
                        ;ConsoleWrite($retailer & "-" & $split3[1] & @CRLF)
                        If $split3[1] = $retailer Then

                            $rev_retailer = $split3[1]
                            $rev_val = StringLeft($split3[13], (StringLen($split3[13]) - 2))
                            ExitLoop
                        EndIf

                    EndIf
                EndIf
            Next
            
            If $rev_val = "" Then
                ContinueLoop
            Else
                $store_value = $rev_val * $storeweight1
                $printer_value = $store_value * $laneweight1
                ConsoleWrite($retailer & " | " & $chain & "-" & $store & " | " & "Store Value: " & $store_value & " | " &  "Lane: " & $lane & " | " & "Lane Value: " & $printer_value  & " | " & "Status: " & $lanestatus & " | " & "StoreWeight: " & $storeweight1 & " | " & "LaneWeight: " & $laneweight1 & " | " & "RetailerVal: " & $rev_val & @CRLF)
                FileWrite($ROI_output1, $retailer & " | " & $chain & "-" & $store & " | " & $store_value & " | " &  $lane & " | " & $printer_value  & " | " & $lanestatus & " | " &  $storeweight1 & " | " &  $laneweight1 & " | " & $rev_val & @CRLF)
            EndIf
    EndIf
    Next
Next

 

Share this post


Link to post
Share on other sites

I haven't used SQLite myself.  But I would think it a more efficient tool than scanning arrays by hand.  Once you figure out how to import the excel data into the SQLite database you could run queries to generate the reports instead of rolling your own.

I would search for any SQLite examples esp. involving excel.

 

Edited by MilesAhead

Share this post


Link to post
Share on other sites

Take a look at this post. The optimizations in this post are based on sorted arrays where you search the rows with a binary search algorithm. If your data can be sorted in the same way, so that it's possible to use binary search, your For-loops can be very optimized. If your arrays can be properly sorted it should be possible to reduce the 11 hours to a few minutes.

If it's possible for you to produce some test data with about 5% of the amount of rows, then I'll try to optimize the For-loops. It's important that the test data are relatively realistic in terms of sorting options.

Share this post


Link to post
Share on other sites

Thank you LarsJ! Thats an excellent suggestion and great link. I'll provide some example files later today then try to practice sorting.

 

Thank you again!

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • By uncommon
      So I have been reading on how to use OAuth2 Service accounts(https://developers.google.com/identity/protocols/OAuth2ServiceAccount) and ran into an issue with autoit I can not solve. To keep this short to communicate to the Google API I need to use a JSON Web signature containing
      {Base64url encoded header}.{Base64url encoded claim set}.

      The Header and claim set work fine but I do not know how to convert them to a byte array using autoit. I have looked around on the forums and website but have not found anything that seems to work. Here is my code.
      #include-once #include <Array.au3> #include <Constants.au3> #include <Debug.au3> #include <File.au3> #include "Json.au3" #include <Process.au3> #include <ProgressConstants.au3> #include <String.au3> #include <WindowsConstants.au3> #include "WinHttp.au3" #include <UnixTime.au3> #include <StringConstants.au3> Global $UnixTime = _TimeGetStamp() $sJWTheader = '{"alg":"RS256","typ":"JWT"}' $sJWTclaimset = '{"iss":"Removed@forprivacy.com","scope":"https://www.googleapis.com/auth/spreadsheets","aud":"https://www.googleapis.com/oauth2/v4/token","exp":' & $UnixTime + 3600 & ',"iat":' & $UnixTime & '}' Local $taiData = DllStructCreate("BYTE[256]") Local $sText = _base64($sJWTheader)&'.'&_base64($sJWTclaimset) DllStructSetData($taiData, 1, StringToBinary($sText, 4)) Local $bvResult = DllStructGetData($taiData, 1) $sJWTSigature = $bvResult $sJWT = _base64($sJWTheader)&'.'&_base64($sJWTclaimset)&'.'&_base64($sJWTSigature) $POSTHeader = "Content-Type: application/x-www-form-urlencoded" $hOpen = _WinHttpOpen() $hConnect = _WinHttpConnect($hOpen, "https://www.googleapis.com/") $sRead = _WinHttpSimpleSSLRequest($hConnect, "POST", "oauth2/v4/token", Default, "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=" & $sJWT) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) ConsoleWrite($sRead & @LF) The whole bytes thing is beyond with what I know about auotit. I can't tell what I need to fix.
       
       
    • By JackER4565
      Hi, first of all thanks to all the guys who always help people in the forums, I wouldn't be able to do anything if wasn't for your help, even if I don't ask it myself.
       
      I've created this code to get some info on a monitoring network on my work. It relays on _IETableGetCollection and _IETableWriteToArray.
      It works well, but take around 3:25 minutes to get the info from 28 pages (some of them are large and take longer to load, but most of them are small and fast).
      My question is if you see a way to get the program to go faster...
       
      I've tried to make it easy for you to understand and edited somethings with sensitive info.
      (Some of the pages doesn't have the black divider with MIRA in the end, so I need to search if it is there or not.)
       
      #include <IE.au3> #include <array.au3> Local $oIE = _IECreate("about:blank", 0, 0) Local $paginas[28] = [89, 90, 91, 92, 93, 96, 105, 113, 119, 125, 126, 129, 131, 133, 135, 137, 139, 140, 141, 144, 145, 146, 148, 149, 150, 151, 158, 159] Local $Datos_array[0][2] Local $oTable Local $tabla Local $aux_x = 1 Local $ar = 1 Local $Numtables_datos = 0 MsgBox(0, "asd", "asd") For $pag = 0 To UBound($paginas) - 1 Step 1 _IENavigate($oIE, "<WEBSITE URL>" & $paginas[$pag]) ; <<< the pages to load are always the same except for the last digits. _ArrayAdd($Datos_array, $paginas[$pag] & "|" & "Entrante", 0, "|") ; <<<<<<<<<<<<<<<< adds the page number toarray [0, 0] ;############################################ START counts amount of tables with traffic $oTable = _IETableGetCollection($oIE) Local $iNumTables = @extended For $i = 3 To $iNumTables - 2 Step 1 $oTable = _IETableGetCollection($oIE, $i) $nomb_tabla2 = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY. $string2 = StringStripWS($nomb_tabla2[1][0], 8) If $string2 <> "MIRA" Then $Numtables_datos = $Numtables_datos + 1 Next $tabla_End = $iNumTables - $Numtables_datos ;############################################ FIN $tabla_Start = 4 $tabla_trafico = 2 For $for = 1 To $Numtables_datos Step 1 $oTable = _IETableGetCollection($oIE, $tabla_Start - 1) ; <<<<<<<<<<< NAME OF THE TABLE; row2 = mira $nomb_tabla = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY ;########################################### ADDS the traffic number into the row $string = StringStripWS($nomb_tabla[1][0], 8) If $string == "MIRA" Then ;si o si pasa por aca 1 vez _ArrayAdd($Datos_array, $nomb_tabla[0][0]) $nomb_aux = $nomb_tabla[0][0] $aux_x = 1 $tabla_trafico = $tabla_trafico + 2 Else ;esto deberia ser por row _ArrayAdd($Datos_array, $nomb_aux & " " & $aux_x) $aux_x = $aux_x + 1 $tabla_trafico = $tabla_trafico + 1 EndIf $oTable = _IETableGetCollection($oIE, $tabla_trafico) Local $aTableData = _IETableWriteToArray($oTable) $bps = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 0, 0) $bps = StringRight($bps, 5) $bps = StringLeft($bps, 4) $trafico_actual = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 2, 2) If $bps == "Gbps" Then $trafico_actual = $trafico_actual * 1000 If $bps == "Kbps" Then $trafico_actual = $trafico_actual / 1000 $Datos_array[$ar][1] = $trafico_actual $ar = $ar + 1 If $string == "MIRA" Then $tabla_Start = $tabla_Start + 2 Else $tabla_Start = $tabla_Start + 1 EndIf Next $ar = $ar + 1 ;~ ############# CAÍDA ############ ;~ If $actual_entrante = 0 Then ;~ $xxx = 0 ;~ Do ;~ MsgBox(0, "Tráfico Caído", $paginas[$i], 5) ;~ $xxx = $xxx + 1 ;~ Until $xxx = 10 ;~ EndIf ;~ ############# CAÍDA ############. Local $Numtables_datos = 0 Next _ArrayDisplay($Datos_array, "Array display") _IEQuit($oIE) Thanks!! 


      monitoria.html
    • By supraaxdd
      Hello,
       
      I have recently tried experimenting with a new feature that I want to add to my program. I wanted to test packet loss, so I opted into a command prompt that displays the necessary data that I want. Now I want to sort out the data by using column sorting if possible by using arrays. My question is: Are you able to sort them out on CMD or is it only Excel spreadsheets?
      Below find the column I want to sort out:
       
      Pinging google.ie [2a00:1450:400b:c01::5e] with 32 bytes of data: Reply from 2a00:1450:400b:c01::5e: time=1033ms Reply from 2a00:1450:400b:c01::5e: time=309ms Reply from 2a00:1450:400b:c01::5e: time=37ms Reply from 2a00:1450:400b:c01::5e: time=732ms Ping statistics for 2a00:1450:400b:c01::5e: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 37ms, Maximum = 1033ms, Average = 527ms I want to extract the Packet loss; the average along with all the results in the "time" column.
      Looking forward for your response!
       
      Kind Regards,
      Supra
    • By Malkey
      The _ArraySortc() function
      _ArraySortC(ByRef $aArray, $iDescending = 0, $iStart = 0, $iEnd = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0)
      The _ArraySort() function from AutoIt's Array.au3 UDF include file is downwardly compatible with the _ArraySortc() function which is in the updated ArrayMultiSortCase.au3 UDF of post #1. In fact, the _ArraySortc() function is a modified version of the _ArraySort() function that gives access to optional numeric sorting and\or case sensitive sorting if needed.
      The _ArraySortInOrder() function
      _ArraySortInOrder(ByRef $array, $sSortingOrder, $iDescending = 0, $iStartRow = 0, $iEndRow = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0)
      The _ArraySortInOrder() function is an updated, better version from here.    The update fixes a bug when zero is sorted and is made similar to (and uses) the new _ArraySortc() function. The added $sSortingOrder parameter in _ArraySortInOrder() is the only additional difference when compared to the _ArraySortc() function's parameters.  The $sSortingOrder parameter contains the sorting string. The ascending order is the individual characters from left to right in the sorting order string.   Descending order is the $sSortingOrder string from right to left.
      The _ArrayMultiSort() function
      _ArrayMultiSort(ByRef $aArray, $sSort = '0a', $iStartRow = 0, $iEndRow = 0)
      _ArraySortc() and _ArraySortInOrder() are the two sorting functions used in the _ArrayMultiSort() function. 
      The $sSort parameter is a string that contains comma separated column descriptors.  The sorting order of the columns is the numeric value in each column descriptor from left to right. For other characters in each column descriptor see _ArrayMultiSort() function's header in the ArrayMultiSortCase.au3 UDF, and comments in the examples. 

       Here is the UDF containing the functions used to sort arrays with case sensitivity, forced numeric sorting, and specified ordered sorting.
      ArrayMultiSortCase.au3
      #include-once ; https://www.autoitscript.com/forum/topic/198148-_arraymultisort-sort-multiple-columns-with-case-sensitivity/ ; File Name: #include <ArrayMultiSortCase.au3> ;#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7 ; #INCLUDES# ========================================================================================================= #include <Array.au3> ; =============================================================================================================================== ; #FUNCTION# ==================================================================================================================== ; Name...........: _ArrayMultiSort ; Description ...: Sort a 1D or, sort a 2D array with the ability to specify which columns to sort and which direction to sort for each column. ; Syntax.........: _ArrayMultiSort(ByRef $avArray[, $sSort = '0a'[, $iStartRow = 0[, $iEndRow = 0]]]) ; Parameters ....: $avArray - Array to sort ; $sSort - [optional] The default setting only sorts the first column, column 0, in ascending order. Otherwise, $sSort is ; a string made up of comma separated column descriptors which contains a column number, sort direction, case ; sensitivity, and make string numbers numeric, for that column. Each column descriptor shows the sort order of ; columns of the array from left to right. ; These 4 fields can be in any order in the column descriptor:- ; 1/ "0" or "1" etc - The column number(0 1 2 ...) - Col#0 is the first column - zero-based; ; 2/ "a" - Ascending order (Without "a" or "d" default is "a"); or, ; "d" - Descending order; And, ; 3/ "c" - sort that column with case sensitivity - Ascending (0-9,A-Z,a-z,Ordered sorting) or, ; Without "c" present, that column is sorted with case insensitivity (default). ; Sorting with case insensitivity - Ascending (Ordered sorting,9-0,Aa-Zz) ; 4/ "n" - Force sorting all digits present in that column numerically. ; Without "n" and digits that are stored as a string will be sorted alphabetically. ; Digits stored as numbers will automatically be sorted numerically in that column. ; 5/ "(Characters used in sorting order)" - A string of characters contained within brckets that give the ; ascending sorting order ; Within the column descriptors, spaces are optional; and, ; Upper or lower case column descriptors, "CAD" or "cad", are recognized as the same - use either (except within the sorting order string). ; Examples of the $sSort variable:- ; "3 ac, ND2, 0n(bac), d1" -> First sort col#3 (case sensitive, ascending, digits as strings sorted alphabetically[default]). ; Secondly sort col#2 (case insensitive, descending, digits sorted numerically.). ; Thirdly sort col#0 (case insensitive[default], ascending[default], digits sorted numerically, ; sort the characters between the brackets, "bac" in this column in the order "b then "a" then "c"). ; Lastly sort col#1 (case insensitive[default], descending, digits as strings ; sorted alphabetically[default]). ; "0" or "" -> Sort col#0 (case insensitive[default], ascending[default], digits as strings ; sorted alphabetically[default]) ; See https://www.autoitscript.com/forum/topic/198148-_arraysort-_arraymultisort-with-case-sensitivity/?do=findComment&comment=1422656 ; $iStartRow - [optional] Row index of array to start sorting at. ; $iEndRow - [optional] Row index of array to stop sorting at. ; Return values .: Success - 1 ; Failure - 0, @error set to the same as returned from _ArraySort() function. (See _ArraySort in AutoIt help file) ; Author ........: Malkey ; Modified.......: ; Remarks .......: Updated version from https://www.autoitscript.com/forum/topic/166426-_filelisttoarray-sorting/?do=findComment&comment=1216234 ; This function requires the __TestExp function, and #Include <Array.au3> and the case sensitive versions of _ArraySort, ; __ArrayQuickSort1D, and __ArrayQuickSort2D. ; Related .......: ; Link ..........: ; Example .......: Yes ; =============================================================================================================================== Func _ArrayMultiSort(ByRef $aArray, $sSort = '0a', $iStartRow = 0, $iEndRow = 0) Local $iCol1, $Dir, $iLastRow, $iStart, $iRetV, $iEnd, $Err, $aSort2D[0][3] ;------------ $sSort (string) to $aSort2D (array) ----------- Local $aTempSort = StringSplit($sSort, ",", 2) ; $STR_NOCOUNT (2) Local $aSort2D[UBound($aTempSort)][5] For $i = 0 To UBound($aTempSort) - 1 $aSort2D[$i][4] = StringRegExp($aTempSort[$i], "\(.+?\)") ? StringRegExpReplace($aTempSort[$i], "^.*\(([^\)]+?)\).*$", "$1") : 0 ; Sorting to order $aTempSort[$i] = StringRegExpReplace($aTempSort[$i], "(\([^\)]+?\))", "") $aSort2D[$i][0] = StringInStr($aTempSort[$i], "c") ? 1 : 0 ; Case sensitivity $aSort2D[$i][1] = StringRegExp($aTempSort[$i], "\d+") ? StringRegExp($aTempSort[$i], "\d+", 1)[0] : 0 ; Column number $aSort2D[$i][2] = StringInStr($aTempSort[$i], "d") ? 1 : 0 ; Ascending/Decending sort order $aSort2D[$i][3] = StringInStr($aTempSort[$i], "n") ? 1 : 0 ; Numeric sorting Next ;_ArrayDisplay($aSort2D, "New method", "", 0, Default, "Case|Col#|Asc/Desc|Numeric|Sorting Order") ; ------------------------------------------------------------ If $iEndRow = 0 Or ($iEndRow > (UBound($aArray) - 1)) Then $iLastRow = UBound($aArray) - 1 $iStart = -1 $iEnd = -1 ; Sort the first column to be sorted. If IsString($aSort2D[0][4]) Then _ArraySortInOrder($aArray, $aSort2D[0][4], $aSort2D[0][2], $iStartRow, $iLastRow, Number($aSort2D[0][1]), 0, $aSort2D[0][0], $aSort2D[0][3]) Else _ArraySortC($aArray, $aSort2D[0][2], $iStartRow, $iLastRow, Number($aSort2D[0][1]), 0, $aSort2D[0][0], $aSort2D[0][3]) ; Sort on the first column (stored in $aSort2D[0][1]) EndIf ; Sort all remaining columns. For $j = 1 To UBound($aSort2D) - 1 ; For each group of equal values in the previous sorted column, sort the present column values on the same rows of each group. $iStart = -1 For $i = $iStartRow To $iLastRow Switch $i Case $iStartRow If $i <> $iLastRow Then If __TestExp($aArray, $aSort2D, $i, $j) Then $iStart = $i + 1 $iEnd = $i + 1 Else $iStart = $i $iEnd = $i + 1 EndIf EndIf Case $iLastRow $iEnd = $iLastRow If $iStart <> $iEnd Then If IsString($aSort2D[$j][4]) Then _ArraySortInOrder($aArray, $aSort2D[$j][4], $aSort2D[$j][2], $iStart, $iEnd, Number($aSort2D[$j][1]), 0, $aSort2D[$j][0], $aSort2D[$j][3]) Else _ArraySortC($aArray, $aSort2D[$j][2], $iStart, $iEnd, Number($aSort2D[$j][1]), 0, $aSort2D[$j][0], $aSort2D[$j][3]) EndIf EndIf Case Else If __TestExp($aArray, $aSort2D, $i, $j) Then $iEnd = $i If $iStart <> $iEnd Then If IsString($aSort2D[$j][4]) Then _ArraySortInOrder($aArray, $aSort2D[$j][4], $aSort2D[$j][2], $iStart, $iEnd, Number($aSort2D[$j][1]), 0, $aSort2D[$j][0], $aSort2D[$j][3]) Else _ArraySortC($aArray, $aSort2D[$j][2], $iStart, $iEnd, Number($aSort2D[$j][1]), 0, $aSort2D[$j][0], $aSort2D[$j][3]) EndIf EndIf $iStart = $i + 1 $iEnd = $iStart Else $iEnd = $i EndIf EndSwitch Next Next Return SetError($Err, 0, $iRetV) EndFunc ;==>_ArrayMultiSort ; #INTERNAL_USE_ONLY# =========================================================================================================== ; Name...........: __TestExp ; Description ...: Creates and tests a test expression. Internally used function by the _ArrayMultiSort() function ; Syntax.........: __TestExp($aA, $aC, $k, $m) ; Parameters ....: $aA - The 2D array to be sorted. ; $aC - A 2D array containing to column sorting order and each column's direction of sorting.(See Modified note) ; $k - The array's row index of the data being sorted. ; $m - The array's row index of the current column being sorted ; Return values .: 1 - When any of the created test expressions are true; or, ; 0 - When all of the created test expressions are false. ; Author ........: Malkey ; Modified.......: Malkey 2019/3/9 Array $aC is now [n][3] where [n][0] is case sensitivity; [n][1] is Column#; [n][2] is sort asc or desc. ; Remarks .......: This function is used internally by the _ArrayMultiSort function. The created expression tests the particular ; row, $k, of the test data, for each previously sorted column, $m -1, wheather the value in that column, $aA[$k][$aC[$w][1]], ; is not equal to the next value in that same column, $aA[$k + 1][$aC[$w][1]], where $aC[$w][1] contains the previously sorted column number. ; =============================================================================================================================== Func __TestExp(ByRef $aA, ByRef $aC, $k, $m) For $w = 0 To $m - 1 If ($aA[$k + 1][$aC[$w][1]]) <> ($aA[$k][$aC[$w][1]]) Then ; Like an "Or" join statements. If just one of the expressions in the For-Next loop is true, then function returns "1" (True). Return 1 EndIf Next Return 0 EndFunc ;==>__TestExp ; Description - This function sorts a 1D or 2D array, $array, in the order of the characters listed in a given string, $sSortingOrder. ; Parameters: ; $array - A 1D or 2D array to be sorted. ; $sSortingOrder - A string containing characters giving the ascending sorting order (left to right) of the same characters which appear in the array. ; $iDescending, $iStartRow, $iEndRow, $iSubItem, $iPivot - are the same parameters as in _ArraySort(), AutoIt's UDF function. ; $iCase - 0 (default) Upper and lower case are treated as the same. ; - 1 Upper and lower case are sorted separately. ; $iNumericalSort - 0 (default) If the number elements of an array are strings, these numbers are sorted alphabetically (lexicographically). ; Where one one, "11", is less than two, "2". ; - 1 Sorted in numerical order. Where two, 2, is less than eleven, 11. ; Requirement: #include <Array.au3> ; Returns: A sorted array. ; Remarks: Updated version from https://www.autoitscript.com/forum/topic/129312-an-array-or-string-sort-in-a-specific-order/ ; Func _ArraySortInOrder(ByRef $array, $sSortingOrder, $iDescending = 0, $iStartRow = 0, $iEndRow = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0) Local $aOrder = StringSplit($sSortingOrder, "", 2) If $iEndRow = 0 Then $iEndRow = UBound($array) - 1 Switch UBound($array, 0) Case 1 ; --- Convert all characters in each element of $array, into an padded index number of the same character of the $sSortingOrder array, $aOrder. --- For $i = $iStartRow To $iEndRow $aItem = StringSplit($array[$i], "", 2) For $j = 0 To UBound($aItem) - 1 $iIndex = StringInStr($sSortingOrder, $aItem[$j], 1) If $iIndex Then $array[$i] = StringRegExpReplace($array[$i], "(?<!~)\Q" & $aItem[$j] & "\E(?!#>)", "~" & StringRight("00" & ($iIndex - 1), 3) & "#>") Next Next ; ------- Sort modified array, $array. ---------- _ArraySortC($array, $iDescending, $iStartRow, $iEndRow, $iSubItem, $iPivot, $iCase, $iNumericalSort) ;_ArrayDisplay($array, "Sorted 1D Array") ; ------- Convert back to original characters -------- For $i = $iStartRow To $iEndRow $array[$i] = Execute('"' & StringRegExpReplace($array[$i], "~(\d{3})#>", '"&$aOrder[$1]&"') & '"') Next Case 2 ; --- Convert all characters in each element of the selected column of the $array, into an padded index number of the same character of the $sSortingOrder array, $aOrder. --- For $i = $iStartRow To $iEndRow $aItem = StringSplit($array[$i][$iSubItem], "", 2) For $j = 0 To UBound($aItem) - 1 $iIndex = StringInStr($sSortingOrder, $aItem[$j], 1) If $iIndex Then $array[$i][$iSubItem] = StringRegExpReplace($array[$i][$iSubItem], "(?<!~)\Q" & $aItem[$j] & "\E(?!\d{0,2}#>)", "~" & StringRight("00" & ($iIndex - 1), 3) & "#>") Next Next ; ------- Sort modified array, $array on selected column. ---------- _ArraySortC($array, $iDescending, $iStartRow, $iEndRow, $iSubItem, $iPivot, $iCase, $iNumericalSort) ;_ArrayDisplay($array, "Sorted 2D Array") ; ------- Convert back to original characters -------- For $i = $iStartRow To $iEndRow $array[$i][2] = Execute('"' & StringRegExpReplace($array[$i][2], "~(\d{3})#>", '"&$aOrder[$1]&"') & '"') Next EndSwitch EndFunc ;==>_ArraySortInOrder ; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ; =============================================================================================== ; The following three function, ; _ArraySortC, __ArrayQuickSort1DC, and __ArrayQuickSort2DC are modied copies of the functions, ; _ArraySort, __ArrayQuickSort1D, and __ArrayQuickSort2D from the Array.au3 include file. ; The modifications allow optiomal case sensitive sorting and optional numeric sorting of arrays. ; _ArraySortC is compatiable with _ArraySort when not using the optional parameters for case ; sensitivity and numeric sorting in the _ArraySortC function. ; =============================================================================================== ; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ; #FUNCTION# ==================================================================================================================== ; Author ........: Jos ; Modified.......: LazyCoder - added $iSubItem option; Tylo - implemented stable QuickSort algo; Jos - changed logic to correctly Sort arrays with mixed Values and Strings; Melba23 - implemented stable pivot algo; Malkey - added optional case sensitive & optional numeric sorting. ; =============================================================================================================================== Func _ArraySortC(ByRef $aArray, $iDescending = 0, $iStart = 0, $iEnd = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0) If $iDescending = Default Then $iDescending = 0 If $iStart = Default Then $iStart = 0 If $iEnd = Default Then $iEnd = 0 If $iSubItem = Default Then $iSubItem = 0 If $iPivot = Default Then $iPivot = 0 If $iCase = Default Then $iCase = 0 If Not IsArray($aArray) Then Return SetError(1, 0, 0) Local $iUBound = UBound($aArray) - 1 If $iUBound = -1 Then Return SetError(5, 0, 0) ; Bounds checking If $iEnd = Default Then $iEnd = 0 If $iEnd < 1 Or $iEnd > $iUBound Or $iEnd = Default Then $iEnd = $iUBound If $iStart < 0 Or $iStart = Default Then $iStart = 0 If $iStart > $iEnd Then Return SetError(2, 0, 0) ; Sort Switch UBound($aArray, $UBOUND_DIMENSIONS) Case 1 If $iNumericalSort Then ; <======== Optional numeric sorting 1D. For $i = $iStart To $iEnd If StringRegExp($aArray[$i], "^[+-]?\d+\.?\d*$") Then $aArray[$i] = Number($aArray[$i]) Next EndIf If $iPivot Then ; Switch algorithms as required __ArrayDualPivotSort($aArray, $iStart, $iEnd) Else __ArrayQuickSort1DC($aArray, $iStart, $iEnd, $iCase) EndIf If $iDescending Then _ArrayReverse($aArray, $iStart, $iEnd) Case 2 If $iNumericalSort Then ; <======== Optional numeric sorting 2D. For $i = $iStart To $iEnd If StringRegExp($aArray[$i][$iSubItem], "^[+-]?\d+\.?\d*$") Then $aArray[$i][$iSubItem] = Number($aArray[$i][$iSubItem]) Next EndIf If $iPivot Then Return SetError(6, 0, 0) ; Error if 2D array and $iPivot Local $iSubMax = UBound($aArray, 2) - 1 ; $UBOUND_COLUMNS (2) If $iSubItem > $iSubMax Then Return SetError(3, 0, 0) If $iDescending Then $iDescending = -1 Else $iDescending = 1 EndIf __ArrayQuickSort2DC($aArray, $iDescending, $iStart, $iEnd, $iSubItem, $iSubMax, $iCase) Case Else Return SetError(4, 0, 0) EndSwitch Return 1 EndFunc ;==>_ArraySortC ; #INTERNAL_USE_ONLY# =========================================================================================================== ; Name...........: __ArrayQuickSort1DC (Appended 'C' for case sensitivity) ; Description ...: Helper function for sorting 1D arrays ; Syntax.........: __ArrayQuickSort1D ( ByRef $aArray, ByRef $iStart, ByRef $iEnd ) ; Parameters ....: $aArray - Array to sort ; $iStart - Index of array to start sorting at ; $iEnd - Index of array to stop sorting at ; Return values .: None ; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima ; Modified.......: Malkey ; Remarks .......: For Internal Use Only. All StringCompare functions have $iCase parameter added for optional case sensitivity. ; Related .......: ; Link ..........: ; Example .......: ; =============================================================================================================================== Func __ArrayQuickSort1DC(ByRef $aArray, Const ByRef $iStart, Const ByRef $iEnd, Const ByRef $iCase) If $iEnd <= $iStart Then Return Local $vTmp ; InsertionSort (faster for smaller segments) If ($iEnd - $iStart) < 15 Then Local $vCur For $i = $iStart + 1 To $iEnd $vTmp = $aArray[$i] If IsNumber($vTmp) Then For $j = $i - 1 To $iStart Step -1 $vCur = $aArray[$j] ; If $vTmp >= $vCur Then ExitLoop If ($vTmp >= $vCur And IsNumber($vCur)) Or (Not IsNumber($vCur) And StringCompare($vTmp, $vCur, $iCase) >= 0) Then ExitLoop $aArray[$j + 1] = $vCur Next Else For $j = $i - 1 To $iStart Step -1 If (StringCompare($vTmp, $aArray[$j], $iCase) >= 0) Then ExitLoop $aArray[$j + 1] = $aArray[$j] Next EndIf $aArray[$j + 1] = $vTmp Next Return EndIf ; QuickSort Local $L = $iStart, $R = $iEnd, $vPivot = $aArray[Int(($iStart + $iEnd) / 2)], $bNum = IsNumber($vPivot) Do If $bNum Then ; While $aArray[$L] < $vPivot While ($aArray[$L] < $vPivot And IsNumber($aArray[$L])) Or (Not IsNumber($aArray[$L]) And StringCompare($aArray[$L], $vPivot, $iCase) < 0) $L += 1 WEnd ; While $aArray[$R] > $vPivot While ($aArray[$R] > $vPivot And IsNumber($aArray[$R])) Or (Not IsNumber($aArray[$R]) And StringCompare($aArray[$R], $vPivot, $iCase) > 0) $R -= 1 WEnd Else While (StringCompare($aArray[$L], $vPivot, $iCase) < 0) $L += 1 WEnd While (StringCompare($aArray[$R], $vPivot, $iCase) > 0) $R -= 1 WEnd EndIf ; Swap If $L <= $R Then $vTmp = $aArray[$L] $aArray[$L] = $aArray[$R] $aArray[$R] = $vTmp $L += 1 $R -= 1 EndIf Until $L > $R __ArrayQuickSort1DC($aArray, $iStart, $R, $iCase) __ArrayQuickSort1DC($aArray, $L, $iEnd, $iCase) EndFunc ;==>__ArrayQuickSort1DC ; #INTERNAL_USE_ONLY# =========================================================================================================== ; Name...........: __ArrayQuickSort2DC (Appended 'C' for case sensitivity) ; Description ...: Helper function for sorting 2D arrays ; Syntax.........: __ArrayQuickSort2D ( ByRef $aArray, ByRef $iStep, ByRef $iStart, ByRef $iEnd, ByRef $iSubItem, ByRef $iSubMax ) ; Parameters ....: $aArray - Array to sort ; $iStep - Step size (should be 1 to sort ascending, -1 to sort descending!) ; $iStart - Index of array to start sorting at ; $iEnd - Index of array to stop sorting at ; $iSubItem - Sub-index to sort on in 2D arrays ; $iSubMax - Maximum sub-index that array has ; $iCase - ; Return values .: None ; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima ; Modified.......: Malkey ; Remarks .......: For Internal Use Only. All StringCompare functions have $iCase parameter added for optional case sensitivity. ; Related .......: ; Link ..........: ; Example .......: ; =============================================================================================================================== Func __ArrayQuickSort2DC(ByRef $aArray, Const ByRef $iStep, Const ByRef $iStart, Const ByRef $iEnd, Const ByRef $iSubItem, Const ByRef $iSubMax, Const ByRef $iCase) If $iEnd <= $iStart Then Return ; QuickSort Local $vTmp, $L = $iStart, $R = $iEnd, $vPivot = $aArray[Int(($iStart + $iEnd) / 2)][$iSubItem], $bNum = IsNumber($vPivot) Do If $bNum Then ; While $aArray[$L][$iSubItem] < $vPivot While ($iStep * ($aArray[$L][$iSubItem] - $vPivot) < 0 And IsNumber($aArray[$L][$iSubItem])) Or (Not IsNumber($aArray[$L][$iSubItem]) And $iStep * StringCompare($aArray[$L][$iSubItem], $vPivot, $iCase) < 0) $L += 1 WEnd ; While $aArray[$R][$iSubItem] > $vPivot While ($iStep * ($aArray[$R][$iSubItem] - $vPivot) > 0 And IsNumber($aArray[$R][$iSubItem])) Or (Not IsNumber($aArray[$R][$iSubItem]) And $iStep * StringCompare($aArray[$R][$iSubItem], $vPivot, $iCase) > 0) $R -= 1 WEnd Else While ($iStep * StringCompare($aArray[$L][$iSubItem], $vPivot, $iCase) < 0) $L += 1 WEnd While ($iStep * StringCompare($aArray[$R][$iSubItem], $vPivot, $iCase) > 0) $R -= 1 WEnd EndIf ; Swap If $L <= $R Then For $i = 0 To $iSubMax $vTmp = $aArray[$L][$i] $aArray[$L][$i] = $aArray[$R][$i] $aArray[$R][$i] = $vTmp Next $L += 1 $R -= 1 EndIf Until $L > $R __ArrayQuickSort2DC($aArray, $iStep, $iStart, $R, $iSubItem, $iSubMax, $iCase) __ArrayQuickSort2DC($aArray, $iStep, $L, $iEnd, $iSubItem, $iSubMax, $iCase) EndFunc ;==>__ArrayQuickSort2DC Here are examples.
      #include <ArrayMultiSortCase.au3> ; --------------- Create test data -------------- Global $array[2000][5] = [["Col 0", "Col 1", "Col 2", "Col 3", "Col 4"]] ; For $i = 1 To UBound($array) - 1 $array[$i][0] = (Random(0, 1, 1) ? Chr(Random(97, 99, 1)) : Chr(Random(65, 67, 1))) ; Random lower case and upper case a,b, or, c. $array[$i][1] = (Random(0, 1, 1) ? Chr(Random(97, 99, 1)) : Chr(Random(65, 67, 1))) ; Random lower case and upper case a,b, or, c. $array[$i][2] = String(Random(0, 1, 1) ? Int(Random(0, 2, 1) & Random(0, 3, 1)) : (Random(0, 1, 1) ? Chr(Random(97, 99, 1)) : Chr(Random(65, 67, 1)))) ; Random lower case and upper case a,b, or, c. And random string numbers. $array[$i][3] = String(Random(0, 2, 1) & Random(1, 3, 1)) ; Random string numbers $array[$i][4] = Random(0, 20, 1) ; Random numeric numbers Next ; -----------> End of Create test data ----------- ; ---------------------------------------------------- Examples --------------------------------------------------------------- ; ======== _ArrayMultiSort() - 2D array ========= Local $aArray = $array ; Sort column 0(asc+case sensitive) then column 1(desc+case insensitive), then column 2(asc+case sensitive), ; then column 3(desc+case insensitive, numeric sorting), and lastly column 4(asc+case insensitive). ; _ArrayMultiSort(ByRef $aArray, $sSort = '0a', $iStartRow = 0, $iEndRow = 0) _ArrayMultiSort($aArray, "0a,c1d, 2 an(bacdefghijnmlosprqtuwxyzBA),3n,4d", 1) ; _ArrayDisplay($aArray, "Sorted 0a,1cd,2an(bacdefghijnmlosprqtuwxyzBA),3an,4d") #cs ; Note:- Colunm #0 - 1st sorted, ascending, case insenitive, digits not specifically numberfied. The string digits one one, "11", is smaller than the string digit "2" because of alphabetic sorting. See A's and a's mixed together because of case insenitivity. Colunm #1 - 2nd sorted, descending, case senitive, digits not specifically numberfied. See all c's are followed by all b's. c's and C's are separated because of case senitivity. Colunm #2 - 3rd sorted, ascending, incase insenitive. string digits numberfied. The number 2, is smaller than the number eleven, 11 when numerically sorted. All the characters in the array will be sorted in the ascending order (left to right) same as the characters between the brackets are listed. Colunm #3 - 4th sorted, ascending, case insenitive, numeric sorting of converted string numbers. Colunm #4 - Last sorted,descending, case insenitive, automatic numeric sorting of existing numeric values - No "n" necessary. In _ArrayDisplay, find the same two to four equal numbers in column #3. Note, in the corresponding rows in column #4, these two to four numbers are descending and numerically sorted. #ce ; ---------------------------------------------------------------------------------------------------------------------------- ; ======== _ArrayMultiSort() - 1D array ========= Local $aTest[] = [33, "Bmw", "Blue", 3.9, "brown", 11, "Clear", "green", 13, "Gray", -3.5, "gold", 23, "-3.1", "Gold", "3.7", "qqqqq", _ "0", "gray", "2", "ppppp", "3.3", "rrrrr", "3", "Royal", "112", "sssss", "42", "ace", "32", "White", "-112", "Yellow", "5"] ; _ArrayMultiSort(ByRef $aArray, $sSort = '0a', $iStartRow = 0, $iEndRow = 0) _ArrayMultiSort($aTest) _ArrayDisplay($aTest, "Sorted 0a") #cs ; Note:- Colunm #0 - Ascending - The numeric values are sorted numerically from smallest to largest. However, the string numbers are sorted alphabetically and interspersed between the numerically sorted values. And because of case insensitivity, the words starting with "G" and "g" are sorted as though the same letter. #ce ; ---------------------------------------------------------------------------------------------------------------------------- ; ======= _ArraySortInOrder() ========= Local $aTest[] = [33, "Bmw", "Blue", 3.9, "brown", 11, "Clear", "green", 13, "Gray", -3.5, "gold", 23, "-3.1", "Gold", "3.7", "qqqqq", _ "0", "gray", "2", "ppppp", "3.3", "rrrrr", "3", "Atlas", "112", "sssss", "42", "ace", "32", "White", "-112", "Yellow", "5", "kind"] ;_ArraySortInOrder(ByRef $array, $sSortingOrder, $iDescending = 0, $iStartRow = 0, $iEndRow = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0) _ArraySortInOrder($aTest, "bacdefghijnmlosprqtuwxyzBA", 0, 0, 0, 0, 0, 0, 1) _ArrayDisplay($aTest, "Ordered sorting & numeric sorting") #cs Note:- Sorting with case sensitivity - Ascending (0-9,A-Z,a-z,Ordered sorting) Sorting with case insensitivity - Ascending (Ordered sorting,9-0,Aa-Zz) Colunm #0 - Case insensitive, Ascending, and Numerically sorted, and Ordered sorting - All numbers are forced to numeric values and sorted numerically from smallest number to largest number. #ce ; ---------------------------------------------------------------------------------------------------------------------------- ; ======= _ArraySortc() - numeric sorting ========= Local $aTest[18][2] = [["Color", "#"], ["Black", "a"], ["Blue", "13"], ["white", "23"], ["Brown", "-4.2"], ["Clear", "33"], ["Gold", "43"], ["gold", "53"], _ ["green", "c"], ["Green", "73"], ["brown", "-4.1"], ["Purple", "b"], ["Red", "93"], ["Royal", "+5"], ["Silver", "2"], ["Sky", "12"], ["White", "3"], ["Yellow", "5"]] $aTest1 = $aTest ; _ArraySortC(ByRef $aArray, $iDescending = 0, $iStart = 0, $iEnd = 0, $iSubItem = 0, $iPivot = 0, $iCase = 0, $iNumericalSort = 0) _ArraySortc($aTest1, 0, 1, 0, 1, 0, 0, 1) _ArrayDisplay($aTest1, "_ArraySortc()") ; Or ; ======= _ArraySort() - numeric sorting ========= $aTest2 = $aTest ; _ArrayNumbertizeForSort(ByRef $aArray, $iSubItem = 0, $iStart = 0, $iEnd = 0) _ArrayNumbertizeForSort($aTest2, 1, 1, 0) ; For use with current _ArraySort() function to numeric sort. ; _ArraySort ( ByRef $aArray [, $iDescending = 0 [, $iStart = 0 [, $iEnd = 0 [, $iSubItem = 0 [, $iPivot = 0]]]]] ) _ArraySort($aTest2, 0, 1, 0, 1) _ArrayDisplay($aTest2, "_ArrayNumbertizeForSort()") ; _ArrayNumbertizeForSort() used only here, in this above example. (not in UDF) ; $iSubItem [optional] Sub-index to sort on in 2D arrays (default 0 = first column) Func _ArrayNumbertizeForSort(ByRef $aArray, $iSubItem = 0, $iStart = 0, $iEnd = 0) If $iEnd = 0 Then $iEnd = UBound($aArray) - 1 Switch UBound($aArray, 0) Case 1 ; 1D array For $i = $iStart To $iEnd If StringRegExp($aArray[$i], "^[+-]?\d+\.?\d*$") Then $aArray[$i] = Number($aArray[$i]) Next Case 2 ; 2D array For $i = $iStart To $iEnd If StringRegExp($aArray[$i][$iSubItem], "^[+-]?\d+\.?\d*$") Then $aArray[$i][$iSubItem] = Number($aArray[$i][$iSubItem]) Next EndSwitch EndFunc ;==>_ArrayNumbertizeForSort  
    • By TrashBoat
      Im trying to iterate through a 2d array created by 
      ProcessList() function that returns a 2d array containing names and pid's, i wanted to filter through the names and delete entries that have "svchost.exe" and this is how i did it
      Func _Sanitize($array) For $x = 1 To UBound($array) - 1 ConsoleWrite($array[$x][0] & @CRLF) If $array[$x][0] == "svchost.exe" Then _ArrayDelete($array, $x) EndIf Next EndFunc ;==>_Sanitize And i get this error message
      What is wrong?
×
×
  • Create New...