Jump to content
LarsJ

Fast General Array Management Functions UDF

Recommended Posts

LarsJ
Posted (edited)

FGAudf is a sub-UDF of FAMudf, Fast Array Management Functions UDF. FAMudf is needed to use FGAudf. FASudf, Fast Array Sorting and Management Functions UDF is needed to run examples.

FGAudf that contains general array manipulation functions, is the first actual UDF that implements fast array management functions.

This is the list of functions as shown in Includes\General.au3:

; Functions\Initialization.au3
; ----------------------------
; FGA_GeneralInitDll      Load compiled code from C++ DLL file.
; FGA_GeneralInitVbSrc    Compile and load .NET code from source file.
; FGA_GeneralInitVbDll    Load .NET code from .NET assembly DLL file.

; Functions\Conversions.au3
; ----------------------------
; FGA_Array1DTo2D         Convert a 1D array to a 2D array with one column.
; FGA_Array2DTo1D         Convert a 2D array with one column to a 1D array.
; FGA_IntStructToArray    Convert a DllStruct of integers to a 1D array.

; Functions\ReadWrite.au3
; ----------------------------
; FGA_ArrayWrite          Write a 1D/2D array to disk as binary data. Optimized C++ code.
; FGA_ArrayWriteAu3       Write a 1D/2D array to disk as binary data. Pure AutoIt code.
; FGA_ArrayRead           Read a 1D/2D array from disk files. Optimized C++ code.
; FGA_ArrayReadAu3        Read a 1D/2D array from disk files. Pure AutoIt code.

; Functions\RowFuncs.au3
; ----------------------------
; FGA_RowsAdd             Add one or more new rows to the end of an existing 1D or 2D array.
; FGA_RowsAllocate        Allocate more space in a 1D or 2D array by inserting one or more empty rows
;                         at the specified (row) indices.
; FGA_RowsClear           Clear (blank) one or more rows at the specified (row) indices in a 1D or 2D array.
; FGA_RowsDelete          Delete one or more rows at the specified (row) indices in a 1D or 2D array.
; FGA_RowsExtract         Extract one or more rows at the specified (row) indices in a 1D or 2D array,
;                         and return the extracted rows as a new 1D or 2D array.
; FGA_RowsInsert          Insert one or more new rows at the specified (row) indices in a 1D or 2D array.
; FGA_RowsMove            Move one or more rows from one to another position in a 1D or 2D array.

; Functions\ColumnFuncs.au3
; ----------------------------
; FGA_ColumnsAdd          Add one or more new columns after last column in an existing 2D array.
; FGA_ColumnsAllocate     Allocate more space in a 2D array by inserting one or more empty columns
;                         at the specified (column) indices.
; FGA_ColumnsClear        Clear (blank) one or more columns at the specified (column) indices in a 2D array.
; FGA_ColumnsDelete       Delete one or more columns at the specified (column) indices in a 2D array.
; FGA_ColumnsExtract      Extract one or more columns at the specified (column) indices in a 2D array,
;                         and return the extracted columns as a new 2D array.
; FGA_ColumnsInsert       Insert one or more new columns at the specified (column) indices in a 2D array.
; FGA_ColumnsMove         Move one or more columns from one to another position in a 2D array.


#include-once
#include "Functions\Initialization.au3"
#include "Functions\Conversions.au3"
#include "Functions\ReadWrite.au3"
#include "Functions\RowFuncs.au3"
#include "Functions\ColumnFuncs.au3"

The functions are optimized with small pieces of C++ code. The last group of column functions is optimized with VB code. The ReadWrite functions retains the data type of the array elements as shown by VarGetType.

 

Examples
Examples\3) ReadWrite\FGA_ArrayWrite.au3:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FGAudf\Includes\General.au3"
FGA_GeneralInitDll( "..\..\..\FGAudf\DLLFiles" )
#include "..\..\..\FASudf\Includes\SortingWr.au3"
FAS_SortingInitVbSrc( "..\..\..\FASudf\Sources\Sorting.vb" )
#include "..\..\..\Display\Display.au3"

Opt( "MustDeclareVars", 1 )

Example( 100000 )

Func Example( $iRows )
  ; Create array
  Local $aCols = "sifdtr", $sSource = "TheQuickBrownFoxJumpsOverTheLazyDog"
  Local $aArray = FAS_Random2DArrayWr( $iRows, $aCols, $sSource )
  Local $aDisplay = Display_GetRandom2DArrayFeatures( $aCols )
  _ArrayDisplayEx( $aArray, "Test", $aDisplay[1], 0, $aDisplay[0] )

  ; Write array
  FGA_ArrayWrite( "Test", $aArray, 10, 1 ) ; 10 = $iStrBufSize, 1 = $bOverWrite
  If @error Then Return ConsoleWrite( "FGA_ArrayWrite: @error = " & @error & @CRLF )
EndFunc

Examples\3) ReadWrite\FGA_ArrayRead.au3:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "..\..\..\FGAudf\Includes\General.au3"
FGA_GeneralInitDll( "..\..\..\FGAudf\DLLFiles" )
#include "..\..\..\Display\Display.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  ; Read array
  Local $aArray = FGA_ArrayRead( "Test" )
  If @error Then Return ConsoleWrite( "FGA_ArrayRead: @error = " & @error & @CRLF )
  Local $aDisplay = Display_GetRandom2DArrayFeatures( "sifdtr" )
  _ArrayDisplayEx( $aArray, "Test", $aDisplay[1], 0, $aDisplay[0] )
EndFunc

 

Runtimes
FGA_IntStructToArray.au3:

Code executed as 32 bit code  Code executed as 64 bit code  
============================  ============================  

100 integers                  100 integers                  
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:        0.1341    AutoIt loop:        0.1089    
Optimized code:     0.5447    Optimized code:     0.6026    

500 integers                  500 integers                  
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:        3.2448    AutoIt loop:        6.4405    
Optimized code:     0.6029    Optimized code:     0.7472    

1,000 integers                1,000 integers                
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:        2.6441    AutoIt loop:        4.8112    
Optimized code:     0.3219    Optimized code:     0.6430    

2,000 integers                2,000 integers                
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:        4.8996    AutoIt loop:        4.1164    
Optimized code:     0.3784    Optimized code:     0.3294    

5,000 integers                5,000 integers                
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:       11.2276    AutoIt loop:       10.3660    
Optimized code:     0.3848    Optimized code:     0.5430    

10,000 integers               10,000 integers               
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:       15.7094    AutoIt loop:       20.5262    
Optimized code:     0.7261    Optimized code:     1.2802    

20,000 integers               20,000 integers               
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:       28.0012    AutoIt loop:       24.1433    
Optimized code:     1.2046    Optimized code:     1.2193    

50,000 integers               50,000 integers               
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:       73.8528    AutoIt loop:       56.2135    
Optimized code:     3.3196    Optimized code:     3.5119    

100,000 integers              100,000 integers              
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:      138.5275    AutoIt loop:      116.7239    
Optimized code:     4.9702    Optimized code:     5.3537    

250,000 integers              250,000 integers              
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:      336.8909    AutoIt loop:      287.8890    
Optimized code:    12.0601    Optimized code:    11.4880    

500,000 integers              500,000 integers              
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:      670.0658    AutoIt loop:      565.7541    
Optimized code:    23.7252    Optimized code:    23.4895    

750,000 integers              750,000 integers              
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:     1008.0166    AutoIt loop:      861.2616    
Optimized code:    42.6570    Optimized code:    36.7331    

1,000,000 integers            1,000,000 integers            
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:     1365.3218    AutoIt loop:     1132.6611    
Optimized code:    62.7739    Optimized code:    49.5879    

2,000,000 integers            2,000,000 integers            
Copy $tInts to $aArray        Copy $tInts to $aArray        
--------------------------    --------------------------    
AutoIt loop:     2692.6966    AutoIt loop:     2327.1560    
Optimized code:   103.7193    Optimized code:   103.0408

 

FGA_ArrayReadWrite.au3:

Code executed as 32 bit code                 Code executed as 64 bit code                 
============================                 ============================                 

100 rows, 6 columns                          100 rows, 6 columns                          
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                      4.7610    Create array:                      6.5227    
Write array                                  Write array                                  
    _FileWriteFromArray:           6.4917        _FileWriteFromArray:           5.6351    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:         29.8564            Pure AutoIt code:         28.5150    
        Optimized C++ code:        6.8341            Optimized C++ code:        5.6176    
Read array                                   Read array                                   
    _FileReadToArray:              3.2264        _FileReadToArray:              3.2267    
        Compare arrays:            2.7499            Compare arrays:            2.3853    
        Compare errors:               400            Compare errors:               400    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:         17.5141            Pure AutoIt code:         12.7714    
        Optimized C++ code:        0.8331            Optimized C++ code:        1.0852    
            Compare arrays:        2.2986                Compare arrays:        2.4859    
            Compare errors:             0                Compare errors:             0    

500 rows, 6 columns                          500 rows, 6 columns                          
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                     11.7793    Create array:                     15.4549    
Write array                                  Write array                                  
    _FileWriteFromArray:          27.1932        _FileWriteFromArray:          12.9956    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:         39.4211            Pure AutoIt code:         39.7807    
        Optimized C++ code:        3.9448            Optimized C++ code:        4.1642    
Read array                                   Read array                                   
    _FileReadToArray:              6.7294        _FileReadToArray:              7.5012    
        Compare arrays:            7.3084            Compare arrays:            6.0434    
        Compare errors:             2,000            Compare errors:             2,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:         35.5247            Pure AutoIt code:         31.2209    
        Optimized C++ code:        1.0486            Optimized C++ code:        0.8660    
            Compare arrays:        7.6694                Compare arrays:        6.6172    
            Compare errors:             0                Compare errors:             0    

1,000 rows, 6 columns                        1,000 rows, 6 columns                        
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                      4.0845    Create array:                      3.8559    
Write array                                  Write array                                  
    _FileWriteFromArray:          10.9981        _FileWriteFromArray:           8.3587    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:         55.5071            Pure AutoIt code:         52.7710    
        Optimized C++ code:        8.8200            Optimized C++ code:        5.6248    
Read array                                   Read array                                   
    _FileReadToArray:             15.0617        _FileReadToArray:             10.9449    
        Compare arrays:           14.1940            Compare arrays:           29.8340    
        Compare errors:             4,000            Compare errors:             4,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:         71.5238            Pure AutoIt code:         56.8183    
        Optimized C++ code:        2.0787            Optimized C++ code:        1.5273    
            Compare arrays:       16.5320                Compare arrays:       13.2324    
            Compare errors:             0                Compare errors:             0    

2,000 rows, 6 columns                        2,000 rows, 6 columns                        
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                     11.0394    Create array:                      7.7617    
Write array                                  Write array                                  
    _FileWriteFromArray:          30.5510        _FileWriteFromArray:          16.4486    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:         82.6690            Pure AutoIt code:        113.8084    
        Optimized C++ code:       16.6955            Optimized C++ code:       14.0420    
Read array                                   Read array                                   
    _FileReadToArray:             29.0516        _FileReadToArray:             25.0453    
        Compare arrays:           27.5484            Compare arrays:           30.7150    
        Compare errors:             8,000            Compare errors:             8,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:        137.6881            Pure AutoIt code:        115.6213    
        Optimized C++ code:        3.4159            Optimized C++ code:        2.6823    
            Compare arrays:       29.7154                Compare arrays:       26.6976    
            Compare errors:             0                Compare errors:             0    

5,000 rows, 6 columns                        5,000 rows, 6 columns                        
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                     24.0729    Create array:                     19.4468    
Write array                                  Write array                                  
    _FileWriteFromArray:          50.8904        _FileWriteFromArray:          42.6960    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:        212.7077            Pure AutoIt code:        266.9652    
        Optimized C++ code:       22.9378            Optimized C++ code:       20.4070    
Read array                                   Read array                                   
    _FileReadToArray:             64.6467        _FileReadToArray:             54.5238    
        Compare arrays:           72.3796            Compare arrays:           62.9066    
        Compare errors:            20,000            Compare errors:            20,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:        946.7204            Pure AutoIt code:        266.5835    
        Optimized C++ code:        8.5645            Optimized C++ code:        6.8610    
            Compare arrays:       76.3950                Compare arrays:       78.4471    
            Compare errors:             0                Compare errors:             0    

10,000 rows, 6 columns                       10,000 rows, 6 columns                       
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                     40.6110    Create array:                     38.1977    
Write array                                  Write array                                  
    _FileWriteFromArray:          99.6068        _FileWriteFromArray:          84.5119    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:        406.3738            Pure AutoIt code:        525.4707    
        Optimized C++ code:       50.0717            Optimized C++ code:       42.6650    
Read array                                   Read array                                   
    _FileReadToArray:            134.7227        _FileReadToArray:            114.4924    
        Compare arrays:          145.7609            Compare arrays:          122.7954    
        Compare errors:            40,000            Compare errors:            40,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:        670.0151            Pure AutoIt code:        539.2475    
        Optimized C++ code:       17.6031            Optimized C++ code:       14.5589    
            Compare arrays:      153.2677                Compare arrays:      137.4507    
            Compare errors:             0                Compare errors:             0    

20,000 rows, 6 columns                       20,000 rows, 6 columns                       
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                     94.7963    Create array:                     80.8854    
Write array                                  Write array                                  
    _FileWriteFromArray:         202.6457        _FileWriteFromArray:         166.4799    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:        794.6454            Pure AutoIt code:       1031.5769    
        Optimized C++ code:       96.9212            Optimized C++ code:       85.9356    
Read array                                   Read array                                   
    _FileReadToArray:            265.0509        _FileReadToArray:            231.8126    
        Compare arrays:          291.5814            Compare arrays:          249.0380    
        Compare errors:            80,000            Compare errors:            80,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:       1330.1447            Pure AutoIt code:       1073.0838    
        Optimized C++ code:       35.7195            Optimized C++ code:       30.1892    
            Compare arrays:      308.6617                Compare arrays:      264.7126    
            Compare errors:             0                Compare errors:             0    

50,000 rows, 6 columns                       50,000 rows, 6 columns                       
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                    251.3502    Create array:                    241.0808    
Write array                                  Write array                                  
    _FileWriteFromArray:         491.0023        _FileWriteFromArray:         397.4234    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:       1991.3867            Pure AutoIt code:       2578.5686    
        Optimized C++ code:      232.9565            Optimized C++ code:      222.1867    
Read array                                   Read array                                   
    _FileReadToArray:            648.6312        _FileReadToArray:            559.4462    
        Compare arrays:          699.0293            Compare arrays:          600.1184    
        Compare errors:           200,000            Compare errors:           200,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:       3310.3602            Pure AutoIt code:       2679.1066    
        Optimized C++ code:       96.9672            Optimized C++ code:       80.6161    
            Compare arrays:      749.4446                Compare arrays:      668.2451    
            Compare errors:             0                Compare errors:             0    

100,000 rows, 6 columns                      100,000 rows, 6 columns                      
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                    483.9308    Create array:                    396.8676    
Write array                                  Write array                                  
    _FileWriteFromArray:         972.0078        _FileWriteFromArray:         794.8432    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:       3965.4751            Pure AutoIt code:       5115.0041    
        Optimized C++ code:      462.2521            Optimized C++ code:      458.2148    
Read array                                   Read array                                   
    _FileReadToArray:           1312.4095        _FileReadToArray:           1143.8532    
        Compare arrays:         1385.8987            Compare arrays:         1195.7637    
        Compare errors:           400,000            Compare errors:           400,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:       6702.6805            Pure AutoIt code:       5420.8355    
        Optimized C++ code:      188.6855            Optimized C++ code:      163.0105    
            Compare arrays:     1492.8053                Compare arrays:     1328.1761    
            Compare errors:             0                Compare errors:             0    

250,000 rows, 6 columns                      250,000 rows, 6 columns                      
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                    992.7113    Create array:                   1055.9462    
Write array                                  Write array                                  
    _FileWriteFromArray:        2413.1376        _FileWriteFromArray:        1980.3471    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:      10042.4486            Pure AutoIt code:      12951.3586    
        Optimized C++ code:     1256.0259            Optimized C++ code:     1210.3004    
Read array                                   Read array                                   
    _FileReadToArray:           3296.9876        _FileReadToArray:           2837.6217    
        Compare arrays:         3459.7479            Compare arrays:         2990.7930    
        Compare errors:         1,000,000            Compare errors:         1,000,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:      17687.6209            Pure AutoIt code:      14008.8673    
        Optimized C++ code:      469.3641            Optimized C++ code:      411.3772    
            Compare arrays:     3743.6195                Compare arrays:     3397.4697    
            Compare errors:             0                Compare errors:             0    

500,000 rows, 6 columns                      500,000 rows, 6 columns                      
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                   2277.7159    Create array:                   2212.4201    
Write array                                  Write array                                  
    _FileWriteFromArray:        5055.9309        _FileWriteFromArray:        4108.9125    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:      20951.2827            Pure AutoIt code:      26377.1770    
        Optimized C++ code:     3186.9102            Optimized C++ code:     2735.0599    
Read array                                   Read array                                   
    _FileReadToArray:           6610.8270        _FileReadToArray:           5615.6630    
        Compare arrays:         6887.5677            Compare arrays:         6045.2805    
        Compare errors:         2,000,000            Compare errors:         2,000,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:      37089.5668            Pure AutoIt code:      30486.3904    
        Optimized C++ code:      931.2514            Optimized C++ code:      800.6115    
            Compare arrays:     7483.9338                Compare arrays:     6656.9123    
            Compare errors:             0                Compare errors:             0    

750,000 rows, 6 columns                      750,000 rows, 6 columns                      
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                   3254.4706    Create array:                   2896.7223    
Write array                                  Write array                                  
    _FileWriteFromArray:        7504.3306        _FileWriteFromArray:        6183.5898    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:      31334.5265            Pure AutoIt code:      40094.0106    
        Optimized C++ code:     5033.8636            Optimized C++ code:     4593.2607    
Read array                                   Read array                                   
    _FileReadToArray:           9898.2904        _FileReadToArray:           8410.9800    
        Compare arrays:        10398.6393            Compare arrays:         8950.6256    
        Compare errors:         3,000,000            Compare errors:         3,000,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:      62823.5177            Pure AutoIt code:      51804.0693    
        Optimized C++ code:     1407.3050            Optimized C++ code:     1211.1620    
            Compare arrays:    11273.4910                Compare arrays:    10455.2244    
            Compare errors:             0                Compare errors:             0    

1,000,000 rows, 6 columns                    1,000,000 rows, 6 columns                    
Write/read array to/from disk                Write/read array to/from disk                
-----------------------------------------    -----------------------------------------    
Create array:                   4565.6595    Create array:                   4257.2067    
Write array                                  Write array                                  
    _FileWriteFromArray:       10393.2012        _FileWriteFromArray:       10472.7202    
    FGA_ArrayWrite                               FGA_ArrayWrite                           
        Pure AutoIt code:      42031.4300            Pure AutoIt code:      60211.2321    
        Optimized C++ code:     6624.7612            Optimized C++ code:     6436.1374    
Read array                                   Read array                                   
    _FileReadToArray:          13164.0322        _FileReadToArray:          11280.1248    
        Compare arrays:        13851.5845            Compare arrays:        11944.7559    
        Compare errors:         4,000,000            Compare errors:         4,000,000    
    FGA_ArrayRead                                FGA_ArrayRead                            
        Pure AutoIt code:      87125.7872            Pure AutoIt code:      74393.8941    
        Optimized C++ code:     1884.6618            Optimized C++ code:     1610.3300    
            Compare arrays:    15078.5623                Compare arrays:    13405.1787    
            Compare errors:             0                Compare errors:             0

 

FGA_RowsInsert.au3:

Code executed as 32 bit code       Code executed as 64 bit code       
============================       ============================       

100 rows, 6 columns                100 rows, 6 columns                
Insert one row at index 25         Insert one row at index 25         
-------------------------------    -------------------------------    
_ArrayInsert:            0.7159    _ArrayInsert:            0.6087    
FGA_RowsInsert:          2.6422    FGA_RowsInsert:          3.0201    

500 rows, 6 columns                500 rows, 6 columns                
Insert one row at index 125        Insert one row at index 125        
-------------------------------    -------------------------------    
_ArrayInsert:            6.7059    _ArrayInsert:           13.4817    
FGA_RowsInsert:          3.5246    FGA_RowsInsert:          6.3012    

1,000 rows, 6 columns              1,000 rows, 6 columns              
Insert one row at index 250        Insert one row at index 250        
-------------------------------    -------------------------------    
_ArrayInsert:           11.9255    _ArrayInsert:           10.2926    
FGA_RowsInsert:          3.6268    FGA_RowsInsert:          5.0115    

2,000 rows, 6 columns              2,000 rows, 6 columns              
Insert one row at index 500        Insert one row at index 500        
-------------------------------    -------------------------------    
_ArrayInsert:           14.6414    _ArrayInsert:           10.9816    
FGA_RowsInsert:          8.5630    FGA_RowsInsert:          5.3611    

5,000 rows, 6 columns              5,000 rows, 6 columns              
Insert one row at index 1,250      Insert one row at index 1,250      
-------------------------------    -------------------------------    
_ArrayInsert:           32.8972    _ArrayInsert:           28.7703    
FGA_RowsInsert:         18.5431    FGA_RowsInsert:         14.3076    

10,000 rows, 6 columns             10,000 rows, 6 columns             
Insert one row at index 2,500      Insert one row at index 2,500      
-------------------------------    -------------------------------    
_ArrayInsert:           65.9608    _ArrayInsert:           56.4863    
FGA_RowsInsert:         37.4105    FGA_RowsInsert:         29.5826    

20,000 rows, 6 columns             20,000 rows, 6 columns             
Insert one row at index 5,000      Insert one row at index 5,000      
-------------------------------    -------------------------------    
_ArrayInsert:          134.4746    _ArrayInsert:          115.3622    
FGA_RowsInsert:         74.8673    FGA_RowsInsert:         55.7314    

50,000 rows, 6 columns             50,000 rows, 6 columns             
Insert one row at index 12,500     Insert one row at index 12,500     
-------------------------------    -------------------------------    
_ArrayInsert:          328.4011    _ArrayInsert:          283.3676    
FGA_RowsInsert:        187.4815    FGA_RowsInsert:        138.7198    

100,000 rows, 6 columns            100,000 rows, 6 columns            
Insert one row at index 25,000     Insert one row at index 25,000     
-------------------------------    -------------------------------    
_ArrayInsert:          636.1870    _ArrayInsert:          560.0946    
FGA_RowsInsert:        374.8816    FGA_RowsInsert:        278.7284    

250,000 rows, 6 columns            250,000 rows, 6 columns            
Insert one row at index 62,500     Insert one row at index 62,500     
-------------------------------    -------------------------------    
_ArrayInsert:         1619.4215    _ArrayInsert:         1389.8712    
FGA_RowsInsert:        926.4933    FGA_RowsInsert:        693.0172    

500,000 rows, 6 columns            500,000 rows, 6 columns            
Insert one row at index 125,000    Insert one row at index 125,000    
-------------------------------    -------------------------------    
_ArrayInsert:         3218.2437    _ArrayInsert:         2823.3835    
FGA_RowsInsert:       1857.2046    FGA_RowsInsert:       1411.2169    

750,000 rows, 6 columns            750,000 rows, 6 columns            
Insert one row at index 187,500    Insert one row at index 187,500    
-------------------------------    -------------------------------    
_ArrayInsert:         4816.4340    _ArrayInsert:         4111.5426    
FGA_RowsInsert:       2803.2133    FGA_RowsInsert:       2070.2246    

1,000,000 rows, 6 columns          1,000,000 rows, 6 columns          
Insert one row at index 250,000    Insert one row at index 250,000    
-------------------------------    -------------------------------    
_ArrayInsert:         6483.6767    _ArrayInsert:         5592.3917    
FGA_RowsInsert:       3754.6153    FGA_RowsInsert:       2764.8844    

2,000,000 rows, 6 columns          2,000,000 rows, 6 columns          
Insert one row at index 500,000    Insert one row at index 500,000    
-------------------------------    -------------------------------    
_ArrayInsert:        12841.9769    _ArrayInsert:        11450.4454    
FGA_RowsInsert:       7481.8667    FGA_RowsInsert:       5691.6660

 

FGA_ColumnsDelete.au3:

Code executed as 32 bit code       Code executed as 64 bit code       
============================       ============================       

100 rows, 12 columns               100 rows, 12 columns               
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:         6.4743    _ArrayColDelete:         8.3762    
FGA_ColumnsDelete:      12.5685    FGA_ColumnsDelete:      18.2236    

500 rows, 12 columns               500 rows, 12 columns               
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        12.6899    _ArrayColDelete:        10.8788    
FGA_ColumnsDelete:       7.8767    FGA_ColumnsDelete:       6.9910    

1,000 rows, 12 columns             1,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        15.0193    _ArrayColDelete:        11.2731    
FGA_ColumnsDelete:       9.4110    FGA_ColumnsDelete:       6.1250    

2,000 rows, 12 columns             2,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        27.5130    _ArrayColDelete:        23.8507    
FGA_ColumnsDelete:      17.1692    FGA_ColumnsDelete:      14.2918    

5,000 rows, 12 columns             5,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        67.8251    _ArrayColDelete:        56.3844    
FGA_ColumnsDelete:      48.3484    FGA_ColumnsDelete:      35.0711    

10,000 rows, 12 columns            10,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       135.4069    _ArrayColDelete:       112.9494    
FGA_ColumnsDelete:      90.1592    FGA_ColumnsDelete:      83.2508    

20,000 rows, 12 columns            20,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       263.8233    _ArrayColDelete:       221.9908    
FGA_ColumnsDelete:     174.9637    FGA_ColumnsDelete:     163.3729    

50,000 rows, 12 columns            50,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       678.3949    _ArrayColDelete:       577.2319    
FGA_ColumnsDelete:     435.9140    FGA_ColumnsDelete:     341.0131    

100,000 rows, 12 columns           100,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      1322.1306    _ArrayColDelete:      1105.7526    
FGA_ColumnsDelete:     933.8638    FGA_ColumnsDelete:     715.9550    

250,000 rows, 12 columns           250,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      3341.2247    _ArrayColDelete:      2839.3375    
FGA_ColumnsDelete:    2389.5969    FGA_ColumnsDelete:    1830.7820    

500,000 rows, 12 columns           500,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      6801.1776    _ArrayColDelete:      5568.0622    
FGA_ColumnsDelete:    4960.1362    FGA_ColumnsDelete:    3837.6910    

750,000 rows, 12 columns           750,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      9854.7078    _ArrayColDelete:      8530.9082    
FGA_ColumnsDelete:    7366.5377    FGA_ColumnsDelete:    5761.3387    

1,000,000 rows, 12 columns         1,000,000 rows, 12 columns         
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:     13090.2626    _ArrayColDelete:     11169.6369    
FGA_ColumnsDelete:   10153.7175    FGA_ColumnsDelete:    7796.6218

 

A look at the runtime measurements shows that FGA_IntStructToArray is 20 to 30 times faster than the corresponding conventional AutoIt code. That's great.

Conversely, at best, FGA_RowsInsert and FGA_ColumnsDelete are only twice as fast as the corresponding _ArrayInsert and _ArrayColDelete. These results are probably representative for both groups of row and column functions. That's pretty bad results. It's compiled code. It should be faster. What's the reason for these results and what can be done about it?

Runtime measurements are performed with arrays with 6/12 columns and a varying number of rows. The 6 columns are "sifdtr" columns. Ie. strings, integers, floats, dates, times and row/col columns. Dates and times are integers. The row/col column is strings. The 12 columns are "sifdtr" & "sifdtr" columns.

If the string columns are replaced by floats columns so that the arrays becomes pure arrays of numbers, the results looks better. Now the compiled code is at best four times faster:

 

FGA_RowsInsert-Numbers.au3:

Code executed as 32 bit code       Code executed as 64 bit code       
============================       ============================       

100 rows, 6 columns                100 rows, 6 columns                
Insert one row at index 25         Insert one row at index 25         
-------------------------------    -------------------------------    
_ArrayInsert:            1.2827    _ArrayInsert:            2.7468    
FGA_RowsInsert:          1.5758    FGA_RowsInsert:          2.3249    

500 rows, 6 columns                500 rows, 6 columns                
Insert one row at index 125        Insert one row at index 125        
-------------------------------    -------------------------------    
_ArrayInsert:           15.0842    _ArrayInsert:           14.2354    
FGA_RowsInsert:          3.9896    FGA_RowsInsert:          3.2785    

1,000 rows, 6 columns              1,000 rows, 6 columns              
Insert one row at index 250        Insert one row at index 250        
-------------------------------    -------------------------------    
_ArrayInsert:           14.3390    _ArrayInsert:           13.2319    
FGA_RowsInsert:          2.7953    FGA_RowsInsert:          2.1401    

2,000 rows, 6 columns              2,000 rows, 6 columns              
Insert one row at index 500        Insert one row at index 500        
-------------------------------    -------------------------------    
_ArrayInsert:           13.5841    _ArrayInsert:           14.7590    
FGA_RowsInsert:          3.1203    FGA_RowsInsert:          2.2363    

5,000 rows, 6 columns              5,000 rows, 6 columns              
Insert one row at index 1,250      Insert one row at index 1,250      
-------------------------------    -------------------------------    
_ArrayInsert:           30.6353    _ArrayInsert:           26.0359    
FGA_RowsInsert:          7.4227    FGA_RowsInsert:          5.7619    

10,000 rows, 6 columns             10,000 rows, 6 columns             
Insert one row at index 2,500      Insert one row at index 2,500      
-------------------------------    -------------------------------    
_ArrayInsert:           62.1590    _ArrayInsert:           56.1475    
FGA_RowsInsert:         15.6602    FGA_RowsInsert:         13.0668    

20,000 rows, 6 columns             20,000 rows, 6 columns             
Insert one row at index 5,000      Insert one row at index 5,000      
-------------------------------    -------------------------------    
_ArrayInsert:          123.7628    _ArrayInsert:          110.9456    
FGA_RowsInsert:         31.2173    FGA_RowsInsert:         23.0951    

50,000 rows, 6 columns             50,000 rows, 6 columns             
Insert one row at index 12,500     Insert one row at index 12,500     
-------------------------------    -------------------------------    
_ArrayInsert:          311.6687    _ArrayInsert:          268.6157    
FGA_RowsInsert:         74.9177    FGA_RowsInsert:         67.6460    

100,000 rows, 6 columns            100,000 rows, 6 columns            
Insert one row at index 25,000     Insert one row at index 25,000     
-------------------------------    -------------------------------    
_ArrayInsert:          608.1551    _ArrayInsert:          517.9057    
FGA_RowsInsert:        150.9420    FGA_RowsInsert:        120.8896    

250,000 rows, 6 columns            250,000 rows, 6 columns            
Insert one row at index 62,500     Insert one row at index 62,500     
-------------------------------    -------------------------------    
_ArrayInsert:         1515.0866    _ArrayInsert:         1318.7368    
FGA_RowsInsert:        380.4741    FGA_RowsInsert:        306.2980    

500,000 rows, 6 columns            500,000 rows, 6 columns            
Insert one row at index 125,000    Insert one row at index 125,000    
-------------------------------    -------------------------------    
_ArrayInsert:         3055.9588    _ArrayInsert:         2592.6736    
FGA_RowsInsert:        739.0425    FGA_RowsInsert:        603.7840    

750,000 rows, 6 columns            750,000 rows, 6 columns            
Insert one row at index 187,500    Insert one row at index 187,500    
-------------------------------    -------------------------------    
_ArrayInsert:         4579.8788    _ArrayInsert:         3906.9467    
FGA_RowsInsert:       1138.3392    FGA_RowsInsert:        912.8767    

1,000,000 rows, 6 columns          1,000,000 rows, 6 columns          
Insert one row at index 250,000    Insert one row at index 250,000    
-------------------------------    -------------------------------    
_ArrayInsert:         6158.5664    _ArrayInsert:         5282.8649    
FGA_RowsInsert:       1550.3611    FGA_RowsInsert:       1206.8620    

2,000,000 rows, 6 columns          2,000,000 rows, 6 columns          
Insert one row at index 500,000    Insert one row at index 500,000    
-------------------------------    -------------------------------    
_ArrayInsert:        12131.6853    _ArrayInsert:        10471.6424    
FGA_RowsInsert:       2976.3306    FGA_RowsInsert:       2425.4074

 

FGA_ColumnsDelete-Numbers.au3:

Code executed as 32 bit code       Code executed as 64 bit code       
============================       ============================       

100 rows, 12 columns               100 rows, 12 columns               
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:         6.5523    _ArrayColDelete:         7.3856    
FGA_ColumnsDelete:      11.4315    FGA_ColumnsDelete:      18.3203    

500 rows, 12 columns               500 rows, 12 columns               
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        25.8863    _ArrayColDelete:        14.6102    
FGA_ColumnsDelete:       4.2210    FGA_ColumnsDelete:       4.1808    

1,000 rows, 12 columns             1,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        18.8860    _ArrayColDelete:        14.3506    
FGA_ColumnsDelete:       4.0459    FGA_ColumnsDelete:       3.8863    

2,000 rows, 12 columns             2,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        26.4980    _ArrayColDelete:        21.2871    
FGA_ColumnsDelete:       8.1876    FGA_ColumnsDelete:       6.5035    

5,000 rows, 12 columns             5,000 rows, 12 columns             
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:        62.5781    _ArrayColDelete:        52.2130    
FGA_ColumnsDelete:      23.2558    FGA_ColumnsDelete:      20.7574    

10,000 rows, 12 columns            10,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       123.1954    _ArrayColDelete:       111.9750    
FGA_ColumnsDelete:      41.5448    FGA_ColumnsDelete:      38.3079    

20,000 rows, 12 columns            20,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       248.6938    _ArrayColDelete:       210.4331    
FGA_ColumnsDelete:      83.8376    FGA_ColumnsDelete:      70.0086    

50,000 rows, 12 columns            50,000 rows, 12 columns            
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:       616.1587    _ArrayColDelete:       541.6447    
FGA_ColumnsDelete:     221.0789    FGA_ColumnsDelete:     198.3267    

100,000 rows, 12 columns           100,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      1235.3286    _ArrayColDelete:      1048.3622    
FGA_ColumnsDelete:     536.6809    FGA_ColumnsDelete:     386.0548    

250,000 rows, 12 columns           250,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      3031.3522    _ArrayColDelete:      2622.3827    
FGA_ColumnsDelete:    1346.9687    FGA_ColumnsDelete:     896.6716    

500,000 rows, 12 columns           500,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      6039.6947    _ArrayColDelete:      5288.0697    
FGA_ColumnsDelete:    2659.5023    FGA_ColumnsDelete:    1928.5887    

750,000 rows, 12 columns           750,000 rows, 12 columns           
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:      9096.9044    _ArrayColDelete:      7954.2082    
FGA_ColumnsDelete:    4000.9829    FGA_ColumnsDelete:    2900.8610    

1,000,000 rows, 12 columns         1,000,000 rows, 12 columns         
Delete column at index 1           Delete column at index 1           
-------------------------------    -------------------------------    
_ArrayColDelete:     12468.9790    _ArrayColDelete:     10382.1064    
FGA_ColumnsDelete:    5308.2076    FGA_ColumnsDelete:    4039.3171

 

Maybe two sets of functions should have been used for arrays with and without strings? The column functions, which are the slowest, are VB code. Maybe it should have been C++ code?

 

What's next
Why are the row and column functions not faster? What can be done about it?

 

FGAudf.7z
FAMudf.7z must be downloaded and unzipped to use FGAudf. FGAudf.7z must be unzipped in the same folder structure as FAMudf.7z. See Fast Array Management Functions UDF for details. FASudf.7z is needed to run examples.

Start running some of the examples in FAMproj\FGAudf\Examples\ (with F5 in SciTE) and look at the code in the examples.

You need AutoIt 3.3.10 or later. Tested on Windows 10 and Windows 7.

Comments are welcome. Let me know if there are any issues.

FGAudf.7z

Edited by LarsJ
Minor updates
  • Like 1

Share this post


Link to post
Share on other sites
mLipok
Posted (edited)

It looks promising. I will look closer when I get some spare time. 

Thanks

Edited by mLipok
  • Like 1

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2018-03-17

Share this post


Link to post
Share on other sites
LarsJ
Posted (edited)

mLipok, Thank you. Let me know if you run into any issues.
 

In another thread there is a list of optimization options:

  • Use pure AutoIt code for small arrays
  • Use compiled code for medium arrays
  • Use multi-threaded code for large arrays
  • Consider data (strings take longer)
  • Limit the number of array transfers

In this thread, only the bold-typed optimization is implemented. So do not be too disappointed with the poor performance. It'll be improved in new versions.
 

To use the functions download the 3 7z-files (FAMudf.7z, FASudf.7z, FGAudf.7z (above)), move the files eg. to C:\Windows\Temp and unzip the files. This will create a FAMproj folder and 4 subfolders:

  • FAMproj\
    • Display\
    • FAMudf\
    • FASudf\
    • FGAudf\

Now you can test the examples in FASudf\Examples and FGAudf\Examples.

Edited by LarsJ

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×