Load many files in listview faster

[Lupo73]

I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!

[martin]

I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!

The slow down is caused by windows trying to redraw the list view every time an item is added. You can stop windows doing that with

_GUICtrlListView_BeginUpdate

then use

_GUICtrlListView_EndUpdate

when you have addd all the items.

[Lupo73]

The slow down is caused by windows trying to redraw the list view every time an item is added. You can stop windows doing that with

_GUICtrlListView_BeginUpdate

then use

_GUICtrlListView_EndUpdate

when you have addd all the items.

It makes the process a few faster, but loading many items during the process listview is not correctly visible..

[torels]

the only thing I can think of is adding a "Loading" label and maybe a graphical element (an avi or GDI+ animation) to make the thing less boring

[ProgAndy]

you should also use _GUICtrlListView_SetItemCount to allocate the memory first

Edited November 16, 2008 by ProgAndy

[Lupo73]

I have tried with both _GUICtrlListView_BeginUpdate and _GUICtrlListView_SetItemCount methods, but the process gain only a few seconds (but the process for 350 files is more than 2 minutes long). It's strange because other software are able to load long lists really faster, so could be a way to do it. Can someone help me? Thanks!

[mikiutama]

........... post deleted..........

Edited January 10, 2009 by mikiutama

[torels]

Excuse me could you please post the code that's giving you problems ?

I have some problems downloading the zip file from sourceforge

anyway... I wrote a piece of code that scans ~1000 files, gets info on the artist, album and track title and sorts them in a treeview in about 2 or 3 minutes

I know it's still slow... but I think that what you need is faster.

I also believe that updating a listview is faster than updating a treeview

[HAL9000]

I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!

you host your app on sourgeforce.net

but where is the code?

[Lupo73]

Sorry for the late, I was away in this period.

@ HAL9000: source code is available on the page I linked (Download -> Browse all packages), in a separated zip file for each release.

@ torels: I use listview... the code is about 1500 lines, this is the function that load files and folders in the listview:

expandcollapse popup

Func Add($temp, $mainf)
    Local $search, $file, $size, $subtemp, $main, $dir, $dupli
    
    $i = 0
    _GUICtrlListView_BeginUpdate($hListViewH)
    If $mainf = 2 Then
    ; load only one file
        $main = StringTrimLeft($temp, StringLen(Cut($temp)))
        $size = FileGetSize($temp)
        $dir = StringTrimRight(Cut($temp), 1) & "\"
        $dupli = DuplicateCheck($dir, $main)
        $item = GUICtrlCreateListViewItem($main & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
        If $dupli = 1 Then
            GUICtrlSetBkColor($item, 0xFFBFBF)
            GUICtrlCreateListViewItem("0|1", $hListViewH)
        Else
            GUICtrlCreateListViewItem("0|0", $hListViewH)
        EndIf
    Else
    ; load main folder
        If $mainf = 0 Then
            $mainf = 1
            $main = StringTrimLeft($temp, StringLen(Cut($temp)))
            $size = DirGetSize($temp & "\" & $file)
            $dir = Cut($temp)
            $dupli = DuplicateCheck($dir, $main)
            $item = GUICtrlCreateListViewItem($main & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
            If $dupli = 1 Then
                GUICtrlSetBkColor($item, 0xFFBFBF)
                GUICtrlCreateListViewItem("1|1", $hListViewH)
            Else
                GUICtrlSetBkColor($item, 0xE5F7FF)
            ;GUICtrlSetBkColor($item, 0xC6F0FF)
                GUICtrlCreateListViewItem("1|0", $hListViewH)
            EndIf
        EndIf
    
    ; load files
        $search = FileFindFirstFile($temp & "\*.*")
        While 1
            $file = FileFindNextFile($search)
            If @error Then
                _GUICtrlListView_EndUpdate($hListViewH)
                ExitLoop
            EndIf
            $i = $i + 1
            $attrib = FileGetAttrib($temp & "\" & $file)
            If Not StringInStr($attrib, "D") Then
                $size = FileGetSize($temp & "\" & $file)
                $dir = $temp & "\"
                $dupli = DuplicateCheck($dir, $file)
                $item = GUICtrlCreateListViewItem($file & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
                If $dupli = 1 Then
                    GUICtrlSetBkColor($item, 0xFFBFBF)
                    GUICtrlCreateListViewItem("0|1", $hListViewH)
                Else
                    GUICtrlCreateListViewItem("0|0", $hListViewH)
                EndIf
            EndIf
        WEnd
        FileClose($search)
        
    ; load folders
        $search = FileFindFirstFile($temp & "\*.*")
        While 1
            $file = FileFindNextFile($search)
            If @error Then
                _GUICtrlListView_EndUpdate($hListViewH)
                ExitLoop
            EndIf
            $i = $i + 1
            $attrib = FileGetAttrib($temp & "\" & $file)
            If StringInStr($attrib, "D") Then
                $size = DirGetSize($temp & "\" & $file)
                $dir = $temp & "\"
                $dupli = DuplicateCheck($dir, $file)
                $item = GUICtrlCreateListViewItem($file & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
                If $dupli = 1 Then
                    GUICtrlSetBkColor($item, 0xFFBFBF)
                    GUICtrlCreateListViewItem("0|1", $hListViewH)
                Else
                    GUICtrlSetBkColor($item, 0xE5F7FF)
                    GUICtrlCreateListViewItem("0|0", $hListViewH)
                EndIf
                $subtemp = $temp & "\" & $file
                Add($subtemp, $mainf)
            EndIf
        WEnd
        _GUICtrlListView_EndUpdate($hListViewH)
        FileClose($search)
    EndIf
EndFunc

[Lupo73]

..I forgot: in the code I use two listview, one showed that include some info like file name, directory, size.. the other one hidden that include some info about files loaded.

Any advice?

[ResNullius]

Have you looked at _GUICtrlListView_AddArray() ?

It's very fast at adding items to a listview, but you'll need to make your file list an array first.

Possible that your array building might offset the speed of the _GuiCtrlListView_AddArray, but you never know.

[Lupo73]

thanks, I'll try it

[PsaltyDS]

thanks, I'll try it

You should put some debug points in there to collect the time it takes to do things. I'll bet you lunch that the real time-waster is DuplicateCheck().

[Lupo73]

Yes! Eureca ^^ ..it seem to be the big problem.

I have to run the duplicate check only at the end of the loading process, not each time a file is loaded. Can you help me to create a function that, given a listview, check if there are lines that are duplicate? thank you very much!

This is my idea (but need to be completed, or changed ^^):

Func DuplicateCheck()
    Local $nIt, $iI, $k, $h, $a, $b, $aa, $bb
; $a[3] = file name      $b[3] = directory
    
    $nIt = _GUICtrlListView_GetItemCount($hListView)
    For $k = 0 To $nIt
        $a = _GUICtrlListView_GetItem($hListView, $k, 0)
        $b = _GUICtrlListView_GetItem($hListView, $k, 1)
        For $h = 0 To $nIt
            If Not $h = $k Then
                $aa = _GUICtrlListView_GetItem($hListView, $h, 0)
                $bb = _GUICtrlListView_GetItem($hListView, $h, 1)
                If $bb[3] = $b[3] Then
                    If $aa[3] = $a[3] Then
                        
                        
                    ; here I need to add a code that change the background color of the corresponding line
                    ; and change a parameter in a second listview that has the same number of items

                        
                    EndIf
                EndIf
            EndIf
        Next
    Next
EndFunc

Edited January 29, 2009 by Lupo73

[PsaltyDS]

Yes! Eureca ^^ ..it seem to be the big problem.

I have to run the duplicate check only at the end of the loading process, not each time a file is loaded. Can you help me to create a function that, given a listview, check if there are lines that are duplicate? thank you very much!

First, you owe me lunch...

[Lupo73]

I have added a comment in the previous post (I didn't see your answer).. what I need (but I'm not able to create it) is a function that start at the end of the adding process, when the whole listview is loaded, and check the list only one time for duplicates. When the function find a duplicate item, it change the background of the item line and change also an item in a second listview. In particular this second listview is an hidden listview that contains parameters of files loaded in the main listview (so it has the same number and order of lines of the main one)... and I need that if a duplicate is found, in this second listview the corresponding item is changed from 0 to 1.

[PsaltyDS]

I have added a comment in the previous post (I didn't see your answer)...

I don't see where I get lunch out of this...

what I need (but I'm not able to create it) is a function that start at the end of the adding process, when the whole listview is loaded, and check the list only one time for duplicates. When the function find a duplicate item, it change the background of the item line and change also an item in a second listview. In particular this second listview is an hidden listview that contains parameters of files loaded in the main listview (so it has the same number and order of lines of the main one)... and I need that if a duplicate is found, in this second listview the corresponding item is changed from 0 to 1.

The nested For/Next loops are slowest possible way to do this. You can do it with only pass through both listviews: All you have to do is assemble a string of used values, with a delimiter like "|" and just use StringInStr() to test each value against that list for duplicates:

Func DuplicateCheck()
    Local $sSearch = "|"
   
    For $k = 0 To _GUICtrlListView_GetItemCount($hListView) - 1
        $sLVData = _GUICtrlListView_GetItemText($hListView, $k, 1)
        If StringInStr($sSearch, "|" & $sLVData & "|") Then
        ; Stuff to do if duplicate...
        EndIf
        $sSearch &= $sLVData & "|"
    Next
EndFunc

This method is quick enough on each item that it could be removed from the function and done in line with populating the listviews.

P.S. I note you didn't account for ListView item numbers being 0-based, while the sub-items are 1-based. You should check on that.

Edit: Typo, should be _GUICtrlListView_GetItemText()

Edited January 30, 2009 by PsaltyDS

[Lupo73]

I have made some changes to the code. I have removed the two listviews, now there is only one listview that for each element loaded use a line (4 columns) + a parameter (using this function _GUICtrlListView_SetItemParam).

So now the process of adding items is very fast, but at the end start the DuplicateCheck() that needs many time to do the checking.

I have created this function to do the check:

Func DuplicateCheck()
    Local $nIt, $iI, $k, $h, $param
    Local $a, $b, $aa, $bb
    
    $nIt = _GUICtrlListView_GetItemCount($hListView)
    For $k = 0 To $nIt - 1
        $a = _GUICtrlListView_GetItemText($hListView, $k, 0)
        $b = _GUICtrlListView_GetItemText($hListView, $k, 1)
        $param = _GUICtrlListView_GetItemParam($hListView, $k)
        If Not ($param = 2) Then
            For $h = 0 To $nIt - 1
                If Not ($h = $k) Then
                    $aa = _GUICtrlListView_GetItemText($hListView, $h, 0)
                    $bb = _GUICtrlListView_GetItemText($hListView, $h, 1)
                    If $bb = $b Then
                        If $aa = $a Then
                            _GUICtrlListView_SetItemParam($hListView, $h, 2)
                            
                        ;GUICtrlSetBkColor($iI, 0xFFBFBF)
                            
                        EndIf
                    EndIf
                EndIf
            Next
        EndIf
    Next
EndFunc

You said your function is really faster, can you explain me how to implement it? I didn't understand what it does.. I need to do something like the function I post (it receive the whole listview and have to detect if there are duplicate items in it..checking the file name, first element of each line, and the directory, second element of each line). Thanks!

[PsaltyDS]

I have made some changes to the code. I have removed the two listviews, now there is only one listview that for each element loaded use a line (4 columns) + a parameter (using this function _GUICtrlListView_SetItemParam).

So now the process of adding items is very fast, but at the end start the DuplicateCheck() that needs many time to do the checking.

I have created this function to do the check:

You said your function is really faster, can you explain me how to implement it? I didn't understand what it does.. I need to do something like the function I post (it receive the whole listview and have to detect if there are duplicate items in it..checking the file name, first element of each line, and the directory, second element of each line). Thanks!

It would be something like this, even simpler since there is only one listview now:

Func DuplicateCheck()
    Local $sSearch = "|"
  
    For $i = 0 To _GUICtrlListView_GetItemCount($hListView) - 1
        If StringInStr($sSearch, "|" & _GUICtrlListView_GetItemText($hListView, $i, 0) & "|") Then
            ; Stuff to do if duplicate...
                
        Else
            ; Not duplicate, add to list
            $sSearch &= $sLVData & "|"
        EndIf
    Next
EndFunc

But it's simple enough now that it doesn't need to be a function (unless you will be doing it from other places in the script). You can just put this check in with the same code that populates the listview, checking for duplicates as it adds them.