Jump to content

Fast Array Sorting and Management Functions UDF


LarsJ
 Share

Recommended Posts

FASudf is a sub-UDF of FAMudf, Fast Array Management Functions UDF. FAMudf is needed to use FASudf.

FASudf contains functions for sorting arrays and functions that are otherwise related to sorted arrays. In addition to sorting functions, this first version only contains functions to generate arrays of random data. Only 2D arrays are supported.

The sorting functions are implemented through index-based binary sorting algorithms. This means that an array is not sorted directly. Instead, an index is created that defines the sorting order of the rows. An array can be sorted by several columns by creating several sorting indexes. Functionality to handle duplicates is included.

The functions to generate random arrays does not create completely random data. Elements in a column are of same type: strings, integers or floats. Strings can either be a number of random characters from a source string or they may be row/column strings. Integers can be created as dates and times on the formats yymmdd and hhmmss. One or more empty columns can be included. Random arrays are used to test most of the functions in the sub-UDFs.

Both groups of functions are based on code in Using C# and VB Code in AutoIt.

 

Optimizations
Optimizations are based on techniques in this post about Optimizing C# and VB code. This is a summary of the techniques:

  • Use pure AutoIt code for small arrays
  • Use compiled code for medium arrays
  • Use multi-threaded code for large arrays
  • Consider data (strings take longer)
  • Limit the number of array transfers

Because there are several different factors to consider, code optimization is far from a trivial task. This means that it's not possible to implement all code in a single function. Along the way, it appears that there is a need for a fairly large number of functions. Eg. there are 4 different functions to create random arrays and 3 different functions to sort arrays.

 

Functions
This is a list of the most important functions copied from HelpFiles\Functions.txt:

; FAS_SortingInitVbSrc      Compile and load .NET code from source file.
; FAS_SortingInitVbDll      Load .NET code from .NET assembly DLL file.

; FAS_Random2DArray         Creates/returns a 2D array of random data.
; FAS_Random2DArrayCount    Returns number of rows to display progress.
; FAS_Random2DArrayGet      Returns the 2D array of random data from VB code.
; FAS_Random2DArrayWr       Splash text and progress bar for large arrays.

; FAS_Sort2DArray           Index based sorting of a 2D array.
; FAS_Sort2DArrayWr         Splash text and progress bar for large arrays.

; FAS_Random2DArrayGUI      GUI to create 2D arrays of random data.
; FAS_Sort2DArrayGUI        GUI to sort a 2D array.

FAS_Random2DArray and FAS_Random2DArrayWr are wrapper functions for 4 different functions to create random arrays. FAS_Sort2DArray and FAS_Sort2DArrayWr are wrapper functions for 3 different functions to sort arrays.

FAS_Random2DArrayWr and FAS_Sort2DArrayWr are the easy-to-use functions that automatically shows splash texts and progress bars for large arrays. Use FAS_Random2DArray and FAS_Sort2DArray if you want to create your own progress bars.

 

The code
Although there are many more functions and much more code, the ideas and methods are the same as in the small Delete UDF that is used as an example in Accessing AutoIt Arrays.

 

Examples
This version does not contain examples for all functions. Only for the functions in the list above. Examples for other functions will be added in later posts.

This is the example for FAS_Random2DArrayWr:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FASudf\Includes\SortingWr.au3"
FAS_SortingInitVbSrc( "..\..\..\FASudf\Sources\Sorting.vb" )
#include "..\..\..\Display\Display.au3"

Opt( "MustDeclareVars", 1 )

Example(      50 ) ; Small
Example(   50000 ) ; Medium
Example(  500000 ) ; Large
Example( 1000000 ) ; XL array
Example( 2000000 ) ; XXL array

Func Example( $iRows )
  Local $aArray, $aCols = "sifdtr"
  Local $sSource = "TheQuickBrownFoxJumpsOverTheLazyDog"
  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aCols )

  $aArray = FAS_Random2DArrayWr( $iRows, $aCols, $sSource, 1 ) ; $bRetArray = 1
  _ArrayDisplayEx( $aArray, $iRows & " rows", $aDisplay[1], 0, $aDisplay[0] )
EndFunc

 

Runtimes
Runtimes for random arrays:

Code executed as 32 bit code           Code executed as 64 bit code           
============================           ============================           

100 rows, 6 columns                    100 rows, 6 columns                    
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:            4.1606    Pure AutoIt code:            4.1559    
Compiled VB code:           17.8798    Compiled VB code:           25.5180    
Multi-threaded code:        71.1700    Multi-threaded code:        88.3572    

500 rows, 6 columns                    500 rows, 6 columns                    
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:           58.1467    Pure AutoIt code:           62.0461    
Compiled VB code:            2.7133    Compiled VB code:            6.1394    
Multi-threaded code:        58.8299    Multi-threaded code:        67.1078    

1,000 rows, 6 columns                  1,000 rows, 6 columns                  
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:           91.9293    Pure AutoIt code:           65.2808    
Compiled VB code:            4.9111    Compiled VB code:            5.0884    
Multi-threaded code:        60.2677    Multi-threaded code:         5.8344    

2,000 rows, 6 columns                  2,000 rows, 6 columns                  
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:          128.4206    Pure AutoIt code:           78.8515    
Compiled VB code:            8.5295    Compiled VB code:            9.0436    
Multi-threaded code:        60.2874    Multi-threaded code:        76.0251    

5,000 rows, 6 columns                  5,000 rows, 6 columns                  
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:          208.8182    Pure AutoIt code:          216.4811    
Compiled VB code:           23.7436    Compiled VB code:           22.5961    
Multi-threaded code:       106.6686    Multi-threaded code:        18.0873    

10,000 rows, 6 columns                 10,000 rows, 6 columns                 
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:          437.9269    Pure AutoIt code:          361.5627    
Compiled VB code:           47.4749    Compiled VB code:           42.8944    
Multi-threaded code:       125.4751    Multi-threaded code:       146.5746    

20,000 rows, 6 columns                 20,000 rows, 6 columns                 
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:          831.8726    Pure AutoIt code:          745.0602    
Compiled VB code:           96.0985    Compiled VB code:          102.5241    
Multi-threaded code:       174.3039    Multi-threaded code:       164.1153    

50,000 rows, 6 columns                 50,000 rows, 6 columns                 
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:         2026.3799    Pure AutoIt code:         1827.4788    
Compiled VB code:          272.8744    Compiled VB code:          227.1584    
Multi-threaded code:       232.7363    Multi-threaded code:       284.4890    

100,000 rows, 6 columns                100,000 rows, 6 columns                
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:         4093.4503    Pure AutoIt code:         3651.6002    
Compiled VB code:          523.6434    Compiled VB code:          432.8995    
Multi-threaded code:       494.8062    Multi-threaded code:       448.7902    

250,000 rows, 6 columns                250,000 rows, 6 columns                
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:        10148.1931    Pure AutoIt code:         9047.4262    
Compiled VB code:         1339.1534    Compiled VB code:         1159.0255    
Multi-threaded code:      1163.8851    Multi-threaded code:      1216.2328    

500,000 rows, 6 columns                500,000 rows, 6 columns                
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:        20467.1356    Pure AutoIt code:        18071.1545    
Compiled VB code:         2758.6908    Compiled VB code:         2558.2818    
Multi-threaded code:      2408.5988    Multi-threaded code:      2503.3011    

750,000 rows, 6 columns                750,000 rows, 6 columns                
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:        30603.0113    Pure AutoIt code:        27007.7100    
Compiled VB code:         4054.2868    Compiled VB code:         3890.3879    
Multi-threaded code:      3484.5627    Multi-threaded code:      3304.3183    

1,000,000 rows, 6 columns              1,000,000 rows, 6 columns              
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:        41063.6182    Pure AutoIt code:        36056.7854    
Compiled VB code:         5559.1431    Compiled VB code:         5490.3201    
Multi-threaded code:      4888.2765    Multi-threaded code:      4406.3113    

2,000,000 rows, 6 columns              2,000,000 rows, 6 columns              
Generate array with random data        Generate array with random data        
-----------------------------------    -----------------------------------    
Pure AutoIt code:        82056.5640    Pure AutoIt code:        71932.1727    
Compiled VB code:        10980.5317    Compiled VB code:        10904.8085    
Multi-threaded code:      9855.7511    Multi-threaded code:      9487.6442

 

Runtimes for sorting arrays. Includes runtimes for _ArraySort:

Code executed as 32 bit code             Code executed as 64 bit code             
============================             ============================             

100 rows, 6 columns                      100 rows, 6 columns                      
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:              3.7835    Pure AutoIt code:              3.0832    
Compiled VB code:             10.6970    Compiled VB code:             11.6062    
Multi-threaded code:          74.7990    Multi-threaded code:          80.2156    
_ArraySort, pure AutoIt:      21.4189    _ArraySort, pure AutoIt:      19.0863    

500 rows, 6 columns                      500 rows, 6 columns                      
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:             41.2535    Pure AutoIt code:             29.6888    
Compiled VB code:              2.9862    Compiled VB code:              2.2933    
Multi-threaded code:          64.8079    Multi-threaded code:          59.0133    
_ArraySort, pure AutoIt:      70.4841    _ArraySort, pure AutoIt:      56.6128    

1,000 rows, 6 columns                    1,000 rows, 6 columns                    
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:             43.8856    Pure AutoIt code:             64.8200    
Compiled VB code:              6.1015    Compiled VB code:              6.0453    
Multi-threaded code:          65.3971    Multi-threaded code:          71.4382    
_ArraySort, pure AutoIt:     110.7674    _ArraySort, pure AutoIt:      72.7744    

2,000 rows, 6 columns                    2,000 rows, 6 columns                    
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:             91.4105    Pure AutoIt code:             71.4255    
Compiled VB code:             11.6514    Compiled VB code:             11.3633    
Multi-threaded code:          75.2001    Multi-threaded code:          76.5462    
_ArraySort, pure AutoIt:     143.6646    _ArraySort, pure AutoIt:     135.8939    

5,000 rows, 6 columns                    5,000 rows, 6 columns                    
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:            246.0873    Pure AutoIt code:            198.8700    
Compiled VB code:             28.9558    Compiled VB code:             26.9486    
Multi-threaded code:          99.0564    Multi-threaded code:         107.8676    
_ArraySort, pure AutoIt:     331.9612    _ArraySort, pure AutoIt:     288.6266    

10,000 rows, 6 columns                   10,000 rows, 6 columns                   
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:            542.8726    Pure AutoIt code:            432.4651    
Compiled VB code:             75.7265    Compiled VB code:             59.6339    
Multi-threaded code:         150.3806    Multi-threaded code:         123.7613    
_ArraySort, pure AutoIt:     694.8603    _ArraySort, pure AutoIt:     602.0131    

20,000 rows, 6 columns                   20,000 rows, 6 columns                   
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:           1219.0863    Pure AutoIt code:            940.2462    
Compiled VB code:            171.3294    Compiled VB code:            159.0582    
Multi-threaded code:         216.6440    Multi-threaded code:         175.9055    
_ArraySort, pure AutoIt:    1460.1318    _ArraySort, pure AutoIt:    1256.1998    

50,000 rows, 6 columns                   50,000 rows, 6 columns                   
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:           3738.0328    Pure AutoIt code:           2661.5985    
Compiled VB code:            549.0830    Compiled VB code:            472.7786    
Multi-threaded code:         321.4487    Multi-threaded code:         310.9871    
_ArraySort, pure AutoIt:    3965.3475    _ArraySort, pure AutoIt:    3418.2730    

100,000 rows, 6 columns                  100,000 rows, 6 columns                  
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:           9359.8104    Pure AutoIt code:           5911.9112    
Compiled VB code:           1524.1232    Compiled VB code:           1161.3261    
Multi-threaded code:         809.8367    Multi-threaded code:         648.4087    
_ArraySort, pure AutoIt:    8379.1084    _ArraySort, pure AutoIt:    7261.7838    

250,000 rows, 6 columns                  250,000 rows, 6 columns                  
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:          35442.6676    Pure AutoIt code:          18072.7239    
Compiled VB code:           7033.7629    Compiled VB code:           4992.7779    
Multi-threaded code:        2169.2635    Multi-threaded code:        1714.5306    
_ArraySort, pure AutoIt:   22453.6015    _ArraySort, pure AutoIt:   19454.1581    

500,000 rows, 6 columns                  500,000 rows, 6 columns                  
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:         108605.2420    Pure AutoIt code:          44026.5583    
Compiled VB code:          24606.1316    Compiled VB code:          16528.1968    
Multi-threaded code:        5590.5327    Multi-threaded code:        4385.4038    
_ArraySort, pure AutoIt:   47146.5217    _ArraySort, pure AutoIt:   41167.9802    

750,000 rows, 6 columns                  750,000 rows, 6 columns                  
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:         219284.6336    Pure AutoIt code:          78915.0445    
Compiled VB code:          53395.0051    Compiled VB code:          34612.3167    
Multi-threaded code:       10383.0850    Multi-threaded code:        7705.0778    
_ArraySort, pure AutoIt:   72782.3310    _ArraySort, pure AutoIt:   64061.6790    

1,000,000 rows, 6 columns                1,000,000 rows, 6 columns                
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:         367550.2873    Pure AutoIt code:         123784.3113    
Compiled VB code:          94398.9554    Compiled VB code:          61403.6193    
Multi-threaded code:       16732.9943    Multi-threaded code:       11891.9866    
_ArraySort, pure AutoIt:   99281.2820    _ArraySort, pure AutoIt:   85971.6482    

2,000,000 rows, 6 columns                2,000,000 rows, 6 columns                
Sort array by column 0, strings          Sort array by column 0, strings          
-------------------------------------    -------------------------------------    
Pure AutoIt code:        1347477.8241    Pure AutoIt code:         415035.5839    
Compiled VB code:         402581.6253    Compiled VB code:         280823.8333    
Multi-threaded code:       60879.4978    Multi-threaded code:       49133.9863    
_ArraySort, pure AutoIt:  268274.5685    _ArraySort, pure AutoIt:  178115.1385

 

GUI-functions
The GUIs in "Examples\4) GUI functions" looks like this:


rnfjsAG.png

 

If you run RandomArrayGUI.au3 and clicks "Create array" and "Create code", code is created and saved in RandomArrayCode.au3. The code is immediately executable:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FASudf\Includes\SortingWr.au3" ; Path?
FAS_SortingInitVbSrc( "..\..\..\FASudf\Sources\Sorting.vb" ) ; Path?
#include "..\..\..\Display\Display.au3" ; Path?

Opt( "MustDeclareVars", 1 )

RandomArray( 100000 )
;Global $aArray = RandomArray( 100000 )

Func RandomArray( $iRows )

  ; --- Create array ---

  Local $sSource  = "TheQuickBrownFoxJumpsOverTheLazyDog"

  Local $aColumns = [ [ "Strings",    16,    32, 0 ], _
                      [ "Integers",    0, 99999, 0 ], _
                      [ "Floats",      0, 99999, 0 ], _
                      [ "Dates",    2010,  2020, 0 ], _
                      [ "Times",       0,    23, 0 ], _
                      [ "Row/Col"                  ] ]

  Local $aArray = FAS_Random2DArrayWr( $iRows, $aColumns, $sSource, 1 ) ; $bRetArray = 1

  ; --- Display array ---

  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aColumns )
  _ArrayDisplayEx( $aArray, "", $aDisplay[1], 0, $aDisplay[0] ) ; $sHeader, $aFeatures

  ; --- Del/Ret array ---

  If IsArray( $aArray ) Then $aArray = 0
  ;If IsArray( $aArray ) Then Return $aArray
EndFunc

 

If you run SortingArrayGUI.au3, deletes all rows (uncheck checkboxes) except "Dates" and "Times" and clicks "Sort array" and "Create code", code is created and saved in SortingArrayCode.au3. The file is copied and renamed to SortingArrayCode-a.au3. This code is not immediately executable:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FASudf\Includes\SortingWr.au3" ; Path?
FAS_SortingInitVbSrc( "..\..\..\FASudf\Sources\Sorting.vb" ) ; Path?
#include "..\..\..\Display\Display.au3" ; Path?

Opt( "MustDeclareVars", 1 )

;SortArray( $aArray )

Func SortArray( $aArray )

  ; --- Sort array ---

  Local $aCompare = [ [ 3, 0, 0 ], _
                      [ 4, 0, 0 ] ]

  Local $aSortIndex = FAS_Sort2DArrayWr( $aArray, $aCompare, 1, 1, 1 ) ; $bDelArray = 1, $bSortByIndex = 1, $bDuplicates = 1

  ; --- Display array ---

  Local $aColumns = [ "Strings", "Integers", "Floats", "Dates", "Times", "Row/Col" ]

  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aColumns )
  $aDisplay = Display_GetSort2DArrayFeaturesEx( $aDisplay, $aCompare, $aSortIndex, "Grey", UBound( $aColumns ) )
  _ArrayDisplayEx( $aArray, "", $aDisplay[1], 0, $aDisplay[0] ) ; $sHeader, $aFeatures

  ; --- Duplicates ---

  Local $iDupls = FAS_Sort2DDuplsCount()
  If $iDupls > 0 Then
    $aDisplay = Display_GetSort2DArrayDuplicates( $aDisplay, $iDupls )
    _ArrayDisplayEx( $aArray, $aDisplay[2], $aDisplay[1], 0, $aDisplay[0] ) ; $sTitle, $sHeader, $aFeatures
  EndIf

  ; --- Delete index ---

  If IsArray( $aSortIndex ) Or IsDllStruct( $aSortIndex ) Then $aSortIndex = 0
EndFunc

 

To make the code executable copy the contents of RandomArrayCode.au3 to the top of SortingArrayCode.au3 in this way:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FASudf\Includes\SortingWr.au3" ; Path?
FAS_SortingInitVbSrc( "..\..\..\FASudf\Sources\Sorting.vb" ) ; Path?
#include "..\..\..\Display\Display.au3" ; Path?

Opt( "MustDeclareVars", 1 )

;RandomArray( 100000 )
Global $aArray = RandomArray( 100000 )

Func RandomArray( $iRows )

  ; --- Create array ---

  Local $sSource  = "TheQuickBrownFoxJumpsOverTheLazyDog"

  Local $aColumns = [ [ "Strings",    16,    32, 0 ], _
                      [ "Integers",    0, 99999, 0 ], _
                      [ "Floats",      0, 99999, 0 ], _
                      [ "Dates",    2010,  2020, 0 ], _
                      [ "Times",       0,    23, 0 ], _
                      [ "Row/Col"                  ] ]

  Local $aArray = FAS_Random2DArrayWr( $iRows, $aColumns, $sSource, 1 ) ; $bRetArray = 1

  ; --- Display array ---

  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aColumns )
  _ArrayDisplayEx( $aArray, "", $aDisplay[1], 0, $aDisplay[0] ) ; $sHeader, $aFeatures

  ; --- Del/Ret array ---

  ;If IsArray( $aArray ) Then $aArray = 0
  If IsArray( $aArray ) Then Return $aArray
EndFunc


SortArray( $aArray )

Func SortArray( $aArray )

  ; --- Sort array ---

  Local $aCompare = [ [ 3, 0, 0 ], _
                      [ 4, 0, 0 ] ]

  Local $aSortIndex = FAS_Sort2DArrayWr( $aArray, $aCompare, 1, 1, 1 ) ; $bDelArray = 1, $bSortByIndex = 1, $bDuplicates = 1

  ; --- Display array ---

  Local $aColumns = [ "Strings", "Integers", "Floats", "Dates", "Times", "Row/Col" ]

  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aColumns )
  $aDisplay = Display_GetSort2DArrayFeaturesEx( $aDisplay, $aCompare, $aSortIndex, "Grey", UBound( $aColumns ) )
  _ArrayDisplayEx( $aArray, "", $aDisplay[1], 0, $aDisplay[0] ) ; $sHeader, $aFeatures

  ; --- Duplicates ---

  Local $iDupls = FAS_Sort2DDuplsCount()
  If $iDupls > 0 Then
    $aDisplay = Display_GetSort2DArrayDuplicates( $aDisplay, $iDupls )
    _ArrayDisplayEx( $aArray, $aDisplay[2], $aDisplay[1], 0, $aDisplay[0] ) ; $sTitle, $sHeader, $aFeatures
  EndIf

  ; --- Delete index ---

  If IsArray( $aSortIndex ) Or IsDllStruct( $aSortIndex ) Then $aSortIndex = 0
EndFunc

 

What's next
This first version of FASudf is not complete. The sorting functions were initially developed to support virtual listviews but can be used more generally in this version.

The sorting functions were also used to find out if the ideas and methods throughout the project works and are durable and stable. Sorting functions are sufficiently complicated to conclude that if these functions works, then other types of functions will probably also work.

The random arrays are used to test most of the functions in the entire FAMudf, Fast Array Management Functions UDF. But these two groups of functions are the only ones made so far.

Functions like FAS_InsertRows, FAS_DeleteRows, FAS_ModifyRows and FAS_SearchRows to manage sorted arrays are already planned.

Eg. to insert a row into a sorted array, it should be sufficient to add the row to the end of the array and then insert the row number at the correct position in the index (or indexes). To delete a row in a sorted array, it should be sufficient to delete the row in the index (or indexes). That way, it should be possible to make some fast functions to manage sorted arrays.

 

FASudf.7z
FAMudf.7z must be downloaded and unzipped to use FASudf. FASudf.7z must be unzipped in the same folder structure as FAMudf.7z. See Fast Array Management Functions UDF for details.

Start running some of the examples in FAMproj\FASudf\Examples\ (with F5 in SciTE) and look at the code in the examples.

You need AutoIt 3.3.10 or later. Tested on Windows 10 and Windows 7.

Comments are welcome. Let me know if there are any issues.

FASudf.7z

Edited by LarsJ
Minor updates
Link to comment
Share on other sites

FAMproj\FASudf\Examples\Example.au3

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\FASudf\Includes\SortingWr.au3"
FAS_SortingInitVbSrc( "..\..\FASudf\Sources\Sorting.vb" )
;FAS_SortingInitVbDll( "..\..\FASudf\DLLFiles\SortingVb.dll" )
If @error Then Exit ConsoleWrite( "FAS_SortingInitVbSrc/Dll: @error = " & @error & @CRLF )
#include "..\..\Display\Display.au3"

Opt( "MustDeclareVars", 1 )

Example(  100000 )
Example(  250000 )
Example(  500000 )
Example(  750000 )
Example( 1000000 )
;Example( 2000000 ) ; Try it

Func Example( $iRows )
  Local $s = StringRegExpReplace( $iRows, "(\d{1,3}(?=(\d{3})+\z)|\d{3}(?=\d))", "\1," ) & " rows, 6 columns, " _
           & StringRegExpReplace( $iRows * 6, "(\d{1,3}(?=(\d{3})+\z)|\d{3}(?=\d))", "\1," ) & " elements"
  ConsoleWrite( $s & @CRLF )

  ; Create random array
  Local $aCols = "sifdtr" ; Strings, ints, floats, dates, times, r/c
  Local $sSource = "TheQuickBrownFoxJumpsOverTheLazyDog"
  ConsoleWrite( "Create array with random data ..." & @CRLF )
  FAS_Random2DArrayWr( $iRows, $aCols, $sSource, 0 ) ; $bRetArray = 0
  ; The array is stored as global in compiled code

  ; Sort by dates and times
  Local $aCompare1 = [ 3, 4 ]
  ConsoleWrite( "Sort array by dates and times ..." & @CRLF )
  Local $aIndex1 = FAS_Sort2DArrayWr( 0, $aCompare1, 0 ) ; $aArray = 0, $bDelArray = 0
  ; Array is not passed to compiled code, not deleted in compiled code

  ; Sort also by strings
  Local $aCompare2 = [ [ 0, 1 ] ]
  ConsoleWrite( "Sort also array by strings ..." & @CRLF )
  Local $aIndex2 = FAS_Sort2DArrayWr( 0, $aCompare2, 0 ) ; $aArray = 0, $bDelArray = 0
  ; Array is not passed to compiled code, not deleted in compiled code

  ; Sort also by array index
  ConsoleWrite( "Sort also array by index ..." & @CRLF )
  Local $aIndex3[$iRows]
  For $i = 0 To $iRows - 1
    $aIndex3[$i] = $i
  Next

  ; Get array from compiled code
  ConsoleWrite( "Get array from VB code ..." & @CRLF )
  Local $aArray = FAS_Random2DArrayGetWr( $iRows, $aCols )
  ; Array is deleted in compiled code

  ; Display array
  ConsoleWrite( "Display array ..." & @CRLF & @CRLF )
  Local $aSortInfo = [ [ 3, $aIndex1, "" ], _ ; Column 3, dates
                       [ 0, $aIndex2, "" ], _ ; Column 0, strings
                       [ 5, $aIndex3, "" ] ]  ; Column 5, row/col
  Local $aDisplay = Display_GetSort2DArraySortInfo( Display_GetRandom2DArrayFeatures( $aCols ), $aSortInfo )
  _ArrayDisplayEx( $aArray, $s, $aDisplay[1], 0, $aDisplay[0] ) ; $sHeader, $aFeatures
EndFunc

If you haven't already run this example, you should try it. It's a nice example. A random array is created with a varying number of rows and sorted by 3 different columns: dates, strings and a row/col column. Duplicate dates are also sorted by a time column. The sorted array is displayed in a virtual listview. Click header items to switch sorting column.

Code optimization is performed with VB.NET code. At the top of first post there is a list of possible optimizations. All of these optimizations are used in the example. The number of array transfers associated with sorting is limited by storing the random array as a global variable in VB code. I'll guess that the code is about 20 times faster than pure AutoIt code.

Nevertheless, all .NET and VB code will be removed and replaced by C++ code in new versions of the UDF. The main reason for this is that it takes a really long time to pass arrays back and forth between AutoIt code and VB code. The problem of transferring arrays will be discussed in more detail in a later post.

The VB code in the example is much faster than the pure AutoIt code only because the number of array transfers are limited and the code to generate and sort arrays is sufficiently complicated to be of great benefit to the compiled VB code.

If the code had been simpler (eg. just moving around elements in an array without performing actual calculations as is the case in the column functions here) the VB code would not have been nearly as fast compared to the AutoIt code. The time saved by using compiled code is spent on transferring arrays.

The first version of the UDF contains .NET and VB code to save development time. The .NET and VB code was already implemented in Using C# and VB Code.

Edited by LarsJ
Minor updates
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...