listview with thousands of records - gui unresponsive when sorting

[Spiff59]

Encapsuling the update call with BeginUpdate and EndUpdate gives another speed boost.

_GUICtrlListView_BeginUpdate($iListView)
_GUICtrlListView_DeleteAllItems($iListView)
_LoadListView($iColCLicked, $iDescending)
_GUICtrlListView_EndUpdate($iListView)

I'd actually had a version like that last night and yanked the lines back out opting for shorter code versus an unnoticable speed difference. Well, when wrapping a timer around it this morning, what wasn't apparent to the naked eye, turns out to be fairly substantial. To be fair in testing, I moved the load of the first "index" out of the data load loop, bumped the data fields to have 5 significant digits (to allow for tests with over 9999 rows) and put in some timers. With a 5000x4 array I get a 3.2 second initialization and then only .5 sec per sort. At 10000x3 it climbs to 5.2 init and .75 per sort. At 40000x3 it becomes 20 seconds to intiialize the indexes and 5.2 seconds to sort, but also runs into some sort of system delay in painting/drawing the listview. I'd assume allowing listview to sort itself with that many rows/columns would still be much slower.

expandcollapse popup

#include <GuiConstants.au3>
#include <WindowsConstants.au3>
#include <GUIListView.au3>
;#include <Array.au3>

Global $hGUI = 0, $iListView = 0, $iColCLicked = 0, $iDescending = 0
Global $iDataRows = 5000, $iDataColumns = 4
Global $aArray[$iDataRows + 1][$iDataColumns * 2]

; build test array
For $x = 1 to $iDataRows
    For $y = 0 to $iDataColumns - 1
        Switch Mod($y, 3)
            Case 0 ; ascending values
             $aArray[$x][$y] = StringRight("0000" & $x, 5) & "_" & $y
         Case 1 ; descending values
             $aArray[$x][$y] = StringRight("0000" & $iDataRows + 1 - $x, 5) & "_" & $y
         Case 2 ; staggered values
             If Mod($x, 2) Then
                    $aArray[$x][$y] = StringRight("0000" & $iDataRows - $x, 5) & "_" & $y
             Else
                    $aArray[$x][$y] = StringRight("0000" & $x, 5) & "_" & $y
             EndIf
        EndSwitch
    Next
Next

$timer = TimerInit()
For $x = 1 to $iDataRows
    $aArray[$x][$iDataColumns] = $x ;index 1
Next
$aTemp = $aArray
For $x = 0 to $iDataColumns - 1 ; build remaining indexes
    _ArraySort($aTemp, 0, 0, 0, $x)
    For $y = 1 to $iDataRows
        $aArray[$y][$iDataColumns + $x] = $aTemp[$y][$iDataColumns]
    Next
Next
ConsoleWrite($iDataRows & "x" & $iDataColumns & " Initialization: " & Round(TimerDiff($timer) / 1000, 2) & " seconds" & @CRLF)
;_ArrayDisplay($aArray)

Build_GUI()
_LoadListView($iColCLicked)

While 1
    $msg = GUIGetMsg()
    Switch $msg
        Case $iListView
            $i = GUICtrlGetState($iListView)
            If $i = $iColCLicked Then
                $iDescending = (Not $iDescending)
            Else
                $iDescending = 0
         EndIf
            $iColCLicked = $i
            GUICtrlSetState($iListView, $GUI_DISABLE)
            _GUICtrlListView_BeginUpdate($iListView)
         _GUICtrlListView_DeleteAllItems($iListView)
         _LoadListView($iColCLicked, $iDescending)
         _GUICtrlListView_EndUpdate($iListView)
          GUICtrlSetState($iListView, $GUI_ENABLE)
        Case -3
            ExitLoop
    EndSwitch
WEnd
Exit

;===============================================================================
Func Build_GUI()
    Local $str
    $hGUI = GUICreate('', 300, 300)
    For $x = 0 to $iDataColumns - 1
     $str &= "Column " & $x & "|"
    Next
    $str = StringTrimRight($str, 1)
    $iListView = GUICtrlCreateListView($str, 0, 0, 300, 300, -1, BitOR($LVS_EX_FULLROWSELECT, $LVS_EX_GRIDLINES, $LVS_EX_DOUBLEBUFFER))
    GUISetState(@SW_SHOW, $hGUI)
EndFunc

Func _LoadListView($iCol, $iDesc = 0)
    $timer = TimerInit()
    $iCol += $iDataColumns
    Local $aLoopdata[2][3] = [[1, $iDataRows, 1],[$iDataRows, 1, -1]], $str
    For $x = $aLoopdata[$iDesc][0] to $aLoopdata[$iDesc][1] Step $aLoopdata[$iDesc][2]
        $str = ""
     For $y = 0 to $iDataColumns - 1
         $str &= $aArray[$aArray[$x][$iCol]][$y] & "|"
     Next
     $str = StringTrimRight($str, 1)
     GUICtrlCreateListViewItem($str, $iListView)
    Next
    ConsoleWrite($iDataRows & "x" & $iDataColumns & " Sort: " & Round(TimerDiff($timer) / 1000, 2) & " seconds" & @CRLF)
EndFunc

Edit: added GUI_DISABLE and GUI_ENABLE states to show "busy" status

Edited September 25, 2012 by Spiff59

[gcue]

thanks for testing spiff

[gcue]

Windows step in and set a window as "(Not Responding)" after the message pump hasn't worked for like five seconds. Realizing that, the solution is (relatively) simple.

In 3.3.8.1, unleash your inner hacker (maybe not so easy).

In the next release, set the new Volatile keyword to the callback (very easy).

Volatile Func __GUICtrlListView_Sort($nItem1, $nItem2, $hWnd)

i dont see anything in the change log for that version. how does it work? or where is it documented?

sounds promising

thanks

[Spiff59]

thanks for testing spiff

Internal timers don't seem to display accurately due to the odd paint/drawing issue.

But I went and ran the code from post #5 and latest code in post #21 with both set to 10000 rows and 5 columns.

Using the second hand in the Windows "Date and Time Properties" panel post #5 was averaging about 75-80 seconds to complete a sort. Post #21 averaged 9 seconds to initialize it's indices and then 3-4 seconds per sort. A very substantial difference.

[gcue]

nice very nice =)

thanks spiff!

[AdmiralAlkex]

i dont see anything in the change log for that version. how does it work? or where is it documented?

sounds promising

thanks

I used the word "next" as in the "next version coming". It doesn't exist yet.

"If callback then during execution of the function the main message pump is not blocked" <- developers word. It does interesting stuff with COM to. You'll see when it comes.

[gcue]

any eta?

cant wait! =)

[AdmiralAlkex]

No ETA. Your guess is as good as mine.

[Melba23]

gcue,

any eta?

The standard answer when this question is asked is always: "When it is ready".

M23