Jump to content

Improving _ArrayDisplay speed


Recommended Posts

Hi everybody :)
when using _ArrayDisplay, I found a way to drastically reduce a listview load time (and later its sorting time when clicking on a column header) . We discussed of this a year ago with @jpm in this thread and I applied it to _ArrayDisplay just now.

1) The 1st script below uses the original functions of AutoIt
It works on a 2D array of 10.000 rows & 10 columns (you may change that if you want)
When you run this 1st script, the Console will indicate the number of seconds it takes to populate the listview. Please note this number somewhere so you'll compare it later with the 2nd script's number.

#include <Array.au3> ; original
#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Opt("MustDeclareVars", 1)
Global $hTimer

Example()

;==================================================
Func Example()

    _SplashOn("1 - Filling Array")
    Local $iRowMax = 10000, $iColMax = 10, $aArray[$iRowMax][$iColMax]
    For $i = 0 To $iRowMax - 1
        For $j = 0 To $iColMax - 1
            $aArray[$i][$j] = "Row " & $i & " / Col " & $j
        Next
    Next
    _ArrayShuffle($aArray)
    SplashOff()

    MsgBox($MB_TOPMOST, "Shuffled Array is ready", "Click to launch _ArrayDisplay original")

    _SplashOn("2 - Populating Listview")
    AdLibRegister(TimeToLoad)
    $hTimer = TimerInit()
    _ArrayDisplay($aArray, "Speed test ArrayDisplay Original")
EndFunc   ;==>Example

;==================================================
Func TimeToLoad() ; Adlib

    If WinActive("Speed test ArrayDisplay Original") Then
        ConsoleWrite(TimerDiff($hTimer) / 1000 & @lf)
        AdLibUnRegister(TimeToLoad)
        SplashOff()
    EndIf
EndFunc   ;==>Slider

;==================================================
Func _SplashOn($sFirstLine, $sSecondLine = "please wait...")

    SplashTextOn("", $sFirstLine & @CRLF & $sSecondLine, _
        250, 50, -1, -1, $DLG_NOTITLE + $DLG_TEXTVCENTER)
EndFunc   ;==>_SplashOn

2) The 2nd script below uses 2 "speed" files : ArrayDisplayInternals2.au3 and ArrayDisplayInternals_Fill.au3
You should notice that the listview appears much quicker and the Console should reflect it, maybe 2, 3 or 4 times faster.

; Array.au3 is not used during this speed test

#include "ArrayDisplayInternals2.au3" ; increased speed (listview fill & sort)
#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Opt("MustDeclareVars", 1)
Global $hTimer

Example()

;==================================================
Func Example()

    _SplashOn("1 - Filling Array")
    Local $iRowMax = 10000, $iColMax = 10, $aArray[$iRowMax][$iColMax]
    For $i = 0 To $iRowMax - 1
        For $j = 0 To $iColMax - 1
            $aArray[$i][$j] = "Row " & $i & " / Col " & $j
        Next
    Next
    _ArrayShuffle($aArray)
    SplashOff()

    MsgBox($MB_TOPMOST, "Shuffled Array is ready", "Click to launch _ArrayDisplay2")

    _SplashOn("2 - Populating Listview")
    AdLibRegister(TimeToLoad)
    $hTimer = TimerInit()
    _ArrayDisplay2($aArray, "Speed test ArrayDisplay2 Patched")
EndFunc   ;==>Example

;==================================================
Func TimeToLoad() ; Adlib

    If WinActive("Speed test ArrayDisplay2 Patched") Then
        ConsoleWrite(TimerDiff($hTimer) / 1000 & @lf)
        AdLibUnRegister(TimeToLoad)
        SplashOff()
    EndIf
EndFunc   ;==>Slider

;==================================================
Func _SplashOn($sFirstLine, $sSecondLine = "please wait...")

    SplashTextOn("", $sFirstLine & @CRLF & $sSecondLine, _
        250, 50, -1, -1, $DLG_NOTITLE + $DLG_TEXTVCENTER)
EndFunc   ;==>_SplashOn

; #FUNCTION# ====================================================================================================================
; Author ........: randallc, Ultima
; Modified.......: Gary Frost (gafrost), Ultima, Zedna, jpm, Melba23, AZJIO, UEZ - _ArrayDisplay() renamed _ArrayDisplay2() in this speed test
; ===============================================================================================================================
Func _ArrayDisplay2(Const ByRef $aArray, $sTitle = Default, $sArrayRange = Default, $iFlags = Default, $vUser_Separator = Default, $sHeader = Default, $iMax_ColWidth = Default)
    #forceref $vUser_Separator
    Local $iRet = __ArrayDisplay_Share($aArray, $sTitle, $sArrayRange, $iFlags, Default, $sHeader, $iMax_ColWidth, 0, False)
    Return SetError(@error, @extended, $iRet)
EndFunc   ;==>_ArrayDisplay2

; #FUNCTION# ====================================================================================================================
; Author ........: Melba23
; Modified.......: copied here because Array.au3 is not used during this speed test
; ===============================================================================================================================
Func _ArrayShuffle(ByRef $aArray, $iStart_Row = 0, $iEnd_Row = 0, $iCol = -1)

    ; Fisher–Yates algorithm

    If $iStart_Row = Default Then $iStart_Row = 0
    If $iEnd_Row = Default Then $iEnd_Row = 0
    If $iCol = Default Then $iCol = -1

    If Not IsArray($aArray) Then Return SetError(1, 0, -1)
    Local $iDim_1 = UBound($aArray, $UBOUND_ROWS)
    If $iEnd_Row = 0 Then $iEnd_Row = $iDim_1 - 1
    If $iStart_Row < 0 Or $iStart_Row > $iDim_1 - 1 Then Return SetError(3, 0, -1)
    If $iEnd_Row < 1 Or $iEnd_Row > $iDim_1 - 1 Then Return SetError(3, 0, -1)
    If $iStart_Row > $iEnd_Row Then Return SetError(4, 0, -1)

    Local $vTmp, $iRand
    Switch UBound($aArray, $UBOUND_DIMENSIONS)
        Case 1
            For $i = $iEnd_Row To $iStart_Row + 1 Step -1
                $iRand = Random($iStart_Row, $i, 1)
                $vTmp = $aArray[$i]
                $aArray[$i] = $aArray[$iRand]
                $aArray[$iRand] = $vTmp
            Next
            Return 1
        Case 2
            Local $iDim_2 = UBound($aArray, $UBOUND_COLUMNS)
            If $iCol < -1 Or $iCol > $iDim_2 - 1 Then Return SetError(5, 0, -1)
            Local $iCol_Start, $iCol_End
            If $iCol = -1 Then
                $iCol_Start = 0
                $iCol_End = $iDim_2 - 1
            Else
                $iCol_Start = $iCol
                $iCol_End = $iCol
            EndIf
            For $i = $iEnd_Row To $iStart_Row + 1 Step -1
                $iRand = Random($iStart_Row, $i, 1)
                For $j = $iCol_Start To $iCol_End
                    $vTmp = $aArray[$i][$j]
                    $aArray[$i][$j] = $aArray[$iRand][$j]
                    $aArray[$iRand][$j] = $vTmp
                Next
            Next
            Return 1
        Case Else
            Return SetError(2, 0, -1)
    EndSwitch

EndFunc   ;==>_ArrayShuffle

You may also notice that sorting on a column is much faster with the 2nd script, compared to the 1st script.

Thanks for readers who intend to test this. Just place all 4 files in a folder (i.e. the 2 preceding scripts + ArrayDisplayInternals2.au3 + ArrayDisplayInternals_Fill.au3) . There is nothing to change in AutoIt folders.

Please report if something went wrong or should be improved as I just ended this today. I already know that Global variable names which were added are perfectible, because they should follow the way ArrayDisplay "names" its variables, so they won't interfere with user's variables names.

For the curious ones, the differences between the original file ArrayDisplayInternals.au3 and the "speed" file ArrayDisplayInternals2.au3 are found in 5 sections, delimited like this :

;********** Speed patch 1 start *********
...
;**********  Speed patch 1 end  *********

Here are the 2 files to download :

ArrayDisplayInternals2.au3

ArrayDisplayInternals_Fill.au3

Link to comment
Share on other sites

Hi Jpm :)
Glad you found some interest in this ArrayDisplay speed process.

1 hour ago, jpm said:

Is it intended that the $ARRAYDISPLAY_TRANSPOSE is not speed up

It would have required 2 more additional functions and I wanted first to see where all this was going.
If these 2 functions are not scripted & added, then the (very ?) few people using transposed 2D arrays should continue to use the actual slow way to fill their LV

Something doesn't work for me with the file you uploaded a couple of hours ago (ArrayDisplayInternals2.au3)
* The filling of the LV is speeded, that's great.
* But the sort process (after a click on a column header) isn't speeded at all.

This is because patch #5 isn't anymore in the file :

;********** Speed patch 5 start *********
;~     Local $sVal1 = __ArrayDisplay_GetItemText($hWnd, $nItem1, $__g_aArrayDisplay_SortInfo[3])
    DllStructSetData($g_tItem, "SubItem", $__g_aArrayDisplay_SortInfo[3])
    GUICtrlSendMsg($g_idListView, $g_GetItemText, $nItem1, $g_pItem)
    Local $sVal1 = DllStructGetData($g_tBuffer, "Text")

;~     Local $sVal2 = __ArrayDisplay_GetItemText($hWnd, $nItem2, $__g_aArrayDisplay_SortInfo[3])
    DllStructSetData($g_tItem, "SubItem", $__g_aArrayDisplay_SortInfo[3])
    GUICtrlSendMsg($g_idListView, $g_GetItemText, $nItem2, $g_pItem)
    Local $sVal2 = DllStructGetData($g_tBuffer, "Text")
    ;**********  Speed patch 5 end  *********

I don't know if it was intentional, you'll tell us :)
In case you add it again, it requires 2 variables that should be renamed ($g_GetItemText and $g_idListView)

Thanks for your interest and the time it took you to rename all these variables !

Link to comment
Share on other sites

@jpm: I notice there are new variables in Beta 3.3.15.3, for example :

Global $g_sArrayDisplay_RowPrefix = "#"

So we probably should amend 3 lines in ArrayDisplayInternals2.au3, based on what is found in the beta script :

Line 357 (comment stays comment)
;~    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, "Row " & $i, 0)
becomes
;~    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, $g_sArrayDisplay_RowPrefix & " " & $i, 0)


Lines 658 AND 692 :
DllStructSetData($__g_tArrayDisplay_Buffer, "Text", "Row " & $i)
becomes
DllStructSetData($__g_tArrayDisplay_Buffer, "Text", $g_sArrayDisplay_RowPrefix & " " & $i)

Now both examples in help file (_ArrayDisplay & _DebugArrayDisplay) will correctly show the content of 1st col (# 0 instead of Row 0)

Link to comment
Share on other sites

@jpm @Melba23 : could you please explain us the reason why the While loop had to be changed in __ArrayDisplay_Share though there aren't 2 windows to deal with inside the function ?

Actual version (3.3.14.5)

While 1
    $iMsg = GUIGetMsg() ; (0)
    Switch $iMsg
        Case $_ARRAYCONSTANT_GUI_EVENT_CLOSE
            ExitLoop

        Case ...
             
    EndSwitch
Wend

Beta version (3.3.15.3)

While 1
    $aMsg = GUIGetMsg($_ARRAYCONSTANT_GUI_EVENT_ARRAY) ; (1)
    If $aMsg[1] = $hGUI Then
        Switch $aMsg[0]
            Case $_ARRAYCONSTANT_GUI_EVENT_CLOSE
                ExitLoop

            Case ...

        EndSwitch
    EndIf
Wend

Maybe there was some unwanted interaction between a main GUI and an ArrayDisplay GUI (closing the main GUI does in fact close the ArrayDisplay GUI with 3.3.14.5, but now with beta version closing the main GUI won't close anything until you close explicitly the ArrayDisplay GUI first)

#include <Array.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>

Opt("GUICloseOnESC", 0)

Example()

Func Example()
    Local $aSortTest[2][3] = [["1a", "1b", "1c"], ["2a", "2b", "2c"]]

    Local $hGUI = GUICreate("Test", 200, 100, -1, 80, -1, $WS_EX_TOPMOST)
    Local $idButton = GUICtrlCreateButton("Click me", 40, 40, 100, 30)

    GUISetState(@SW_SHOW)

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                ExitLoop

            Case $idButton
                GUICtrlSetData($idButton, "Close GUI  X")
                _ArrayDisplay($aSortTest)
                GUICtrlSetData($idButton, "Click me")

        EndSwitch
    WEnd
    GUIDelete($hGUI)
EndFunc   ;==>Example

Other possible explanations : no need to test any Case if a clicked control doesn't belong to the ArrayDisplay GUI. Or maybe you encountered a more serious issue ?

Thanks for your explanations :)

Link to comment
Share on other sites

  • Moderators

pixelsearch,

Quote

there was some unwanted interaction between a main GUI and an ArrayDisplay GUI

That is my vague recollection of why this happened - but I cannot find the precise thread in which it was discussed.

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

If you want to improve _ArrayDisplay loading speed, why not use a virtual listview. Since data isn't stored in a virtual listview, there's no loading time and the listview shows up instantly.

Here an example with 1,000,000 rows and 10 columns. Run the code in SciTE with F5.

#AutoIt3Wrapper_Au3Check_Parameters=-d -w- 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=Y

Opt( "MustDeclareVars", 1 )

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GuiListView.au3>

Global $iRows = 1000000, $iCols = 10, $aArray[$iRows][$iCols]
Global $hListView

Example()

Func Example()
  ; Create array
  Local $l, $hTimer = TimerInit()
  For $i = 0 To 9
    For $j = 0 To $iRows / 10 - 1
      $l = $i * $iRows / 10 + $j
      For $k = 0 To 9
        $aArray[$l][$k] = 10 * $l + $k
      Next
    Next
    ConsoleWrite( "Rows = " & ( $i + 1 ) * $j & @CRLF )
  Next
  ConsoleWrite( TimerDiff( $hTimer ) & @CRLF ) ; Approx. 8/11 seconds on Windows 7/10

  ; Create GUI
  Local $hGui = GUICreate( "Virtual ListView", 788+20, 788+20 )

  ; Create ListView
  Local $idListView = GUICtrlCreateListView( "", 10, 10, 788, 788, $LVS_OWNERDATA, $WS_EX_CLIENTEDGE )
  _GUICtrlListView_SetExtendedListViewStyle( $idListView, $LVS_EX_DOUBLEBUFFER + $LVS_EX_FULLROWSELECT )
  $hListView = GUICtrlGetHandle( $idListView )
  For $i = 0 To $iCols - 1
    _GUICtrlListView_AddColumn( $idListView, "Col" & $i, 75 )
  Next
  GUICtrlSendMsg( $idListView, $LVM_SETITEMCOUNT, $iRows, 0 )

  ; Register WM_NOTIFY message handler through subclassing
  Local $pNotifyHandler = DllCallbackGetPtr( DllCallbackRegister( "NotifyHandler", "lresult", "hwnd;uint;wparam;lparam;uint_ptr;dword_ptr" ) )
  DllCall( "comctl32.dll", "bool", "SetWindowSubclass", "hwnd", $hGui, "ptr", $pNotifyHandler, "uint_ptr", 0, "dword_ptr", 0 ) ; $iSubclassId = 0, $pData = 0

  ; Show GUI
  GUISetState( @SW_SHOW )

  ; Loop
  While 1
    Switch GUIGetMsg()
      Case $GUI_EVENT_CLOSE
        ExitLoop
    EndSwitch
  WEnd

  ; Cleanup
  DllCall( "comctl32.dll", "bool", "RemoveWindowSubclass", "hwnd", $hGui, "ptr", $pNotifyHandler, "uint_ptr", 0 ) ; $iSubclassId = 0
  GUIDelete( $hGui )
EndFunc

Func NotifyHandler( $hWnd, $iMsg, $wParam, $lParam, $iSubclassId, $pData )
  If $iMsg <> 0x004E Then Return DllCall( "comctl32.dll", "lresult", "DefSubclassProc", "hwnd", $hWnd, "uint", $iMsg, "wparam", $wParam, "lparam", $lParam )[0] ; 0x004E = $WM_NOTIFY
  Local $tNMHDR = DllStructCreate( $tagNMHDR, $lParam )
  Switch HWnd( DllStructGetData( $tNMHDR, "hWndFrom" ) )
    Case $hListView
      Switch DllStructGetData( $tNMHDR, "Code" )
        Case $LVN_GETDISPINFOW
          Local $tNMLVDISPINFO = DllStructCreate( $tagNMLVDISPINFO, $lParam )
          Local Static $tText = DllStructCreate( "wchar[50]" ), $pText = DllStructGetPtr( $tText )
          DllStructSetData( $tText, 1, $aArray[DllStructGetData($tNMLVDISPINFO,"Item")][DllStructGetData($tNMLVDISPINFO,"SubItem")] )
          DllStructSetData( $tNMLVDISPINFO, "Text", $pText )
          Return
      EndSwitch
  EndSwitch
  Return DllCall( "comctl32.dll", "lresult", "DefSubclassProc", "hwnd", $hWnd, "uint", $iMsg, "wparam", $wParam, "lparam", $lParam )[0]
  #forceref $iSubclassId, $pData
EndFunc

 

Link to comment
Share on other sites

Easy. You generate a sort index (1d array of integers) for each column you want sorted. These sort indexes should be calculated in advance just like the array itself. When you click a column header, the sorted array is displayed instantly. In my example, there is room for 40 rows in the listview. Showing 40 rows with or without an index makes only a very small difference. The sorted array will thus be displayed as fast as the unsorted array. There aren't really any good reasons not to use a virtual listview.

True argumentum. Of course. Sorting an array of many rows directly in the listview will necessarily be slow. Even in real compiled code.

Edited by LarsJ
Link to comment
Share on other sites

@jpm: I notice these 2 lines in ArrayDisplayInternals2.au3

Case "Map"
    __ArrayDisplay_AddSubItem($idListView, $iRowIndex, "{Map}", $iColFill)

These 2 lines are found in the Transpose part, but not in the "Regular" part. Should they be also there ?

Edited by pixelsearch
Link to comment
Share on other sites

@Melba23: thanks for having fixed the following fatal error, your patch is useful in many cases, for example :

#include <Array.au3>
Local $aArray[1] = [0]
_ArrayDisplay($aArray, "Title", "1:")

1) Actual version ArrayDisplayInternals.au3 (3.3.14.5)

"ArrayDisplayInternals.au3" (345) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:

$vTmp = $aArray[$i]
$vTmp = ^ ERROR

Line 138 creates this issue :

If $iItem_Start > $iItem_End Then

 

2) Beta version ArrayDisplayInternals.au3 (3.3.15.3) as found on AutoIt web site :
Line 155 fixed the issue :

If $iItem_Start > $iItem_End And $iItem_End > 0 Then

1583328010_betanoerror.png.6d448e3e79bd18ca3b6dff07f3a5e87f.png

Well done MB23 :thumbsup:

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...