Sign in to follow this  
Followers 0
Lupo73

Load many files in listview faster

28 posts in this topic

I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites



I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!

The slow down is caused by windows trying to redraw the list view every time an item is added. You can stop windows doing that with

_GUICtrlListView_BeginUpdate

then use

_GUICtrlListView_EndUpdate

when you have addd all the items.


Serial port communications UDF Includes functions for binary transmission and reception.printing UDF Useful for graphs, forms, labels, reports etc.Add User Call Tips to SciTE for functions in UDFs not included with AutoIt and for your own scripts.Functions with parameters in OnEvent mode and for Hot Keys One function replaces GuiSetOnEvent, GuiCtrlSetOnEvent and HotKeySet.UDF IsConnected2 for notification of status of connected state of many urls or IPs, without slowing the script.

Share this post


Link to post
Share on other sites

The slow down is caused by windows trying to redraw the list view every time an item is added. You can stop windows doing that with

_GUICtrlListView_BeginUpdate

then use

_GUICtrlListView_EndUpdate

when you have addd all the items.

It makes the process a few faster, but loading many items during the process listview is not correctly visible..


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

the only thing I can think of is adding a "Loading" label and maybe a graphical element (an avi or GDI+ animation) to make the thing less boring :mellow:


Some Projects:[list][*]ZIP UDF using no external files[*]iPod Music Transfer [*]iTunes UDF - fully integrate iTunes with au3[*]iTunes info (taskbar player hover)[*]Instant Run - run scripts without saving them before :)[*]Get Tube - YouTube Downloader[*]Lyric Finder 2 - Find Lyrics to any of your song[*]DeskBox - A Desktop Extension Tool[/list]indifference will ruin the world, but in the end... WHO CARES :P---------------http://torels.altervista.org

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

you should also use _GUICtrlListView_SetItemCount to allocate the memory first :mellow:

Edited by ProgAndy

*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites

I have tried with both _GUICtrlListView_BeginUpdate and _GUICtrlListView_SetItemCount methods, but the process gain only a few seconds (but the process for 350 files is more than 2 minutes long). It's strange because other software are able to load long lists really faster, so could be a way to do it. Can someone help me? Thanks!


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

........... post deleted..........

Edited by mikiutama

Share this post


Link to post
Share on other sites

Excuse me could you please post the code that's giving you problems ?

I have some problems downloading the zip file from sourceforge

anyway... I wrote a piece of code that scans ~1000 files, gets info on the artist, album and track title and sorts them in a treeview in about 2 or 3 minutes

I know it's still slow... but I think that what you need is faster.

I also believe that updating a listview is faster than updating a treeview :)


Some Projects:[list][*]ZIP UDF using no external files[*]iPod Music Transfer [*]iTunes UDF - fully integrate iTunes with au3[*]iTunes info (taskbar player hover)[*]Instant Run - run scripts without saving them before :)[*]Get Tube - YouTube Downloader[*]Lyric Finder 2 - Find Lyrics to any of your song[*]DeskBox - A Desktop Extension Tool[/list]indifference will ruin the world, but in the end... WHO CARES :P---------------http://torels.altervista.org

Share this post


Link to post
Share on other sites

I'm working of this application: http://sourceforge.net/projects/arcthemall/

It's a multi-archiver in UPX, ZIP and 7z formats.

It works loading in a listview all files and folders dropped by the user and then process them at the end of the loading. I saw that if I drop a folder with many files (about 1000 for example), It loads first few files faster and then progressively slower (also 1 second per file). So it could need also some minutes to load all that files. Can I improve the code to make this process of loading faster? thanks!

you host your app on sourgeforce.net

but where is the code?

Share this post


Link to post
Share on other sites

Sorry for the late, I was away in this period.

@ HAL9000: source code is available on the page I linked (Download -> Browse all packages), in a separated zip file for each release.

@ torels: I use listview... the code is about 1500 lines, this is the function that load files and folders in the listview:

Func Add($temp, $mainf)
    Local $search, $file, $size, $subtemp, $main, $dir, $dupli
    
    $i = 0
    _GUICtrlListView_BeginUpdate($hListViewH)
    If $mainf = 2 Then
    ; load only one file
        $main = StringTrimLeft($temp, StringLen(Cut($temp)))
        $size = FileGetSize($temp)
        $dir = StringTrimRight(Cut($temp), 1) & "\"
        $dupli = DuplicateCheck($dir, $main)
        $item = GUICtrlCreateListViewItem($main & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
        If $dupli = 1 Then
            GUICtrlSetBkColor($item, 0xFFBFBF)
            GUICtrlCreateListViewItem("0|1", $hListViewH)
        Else
            GUICtrlCreateListViewItem("0|0", $hListViewH)
        EndIf
    Else
    ; load main folder
        If $mainf = 0 Then
            $mainf = 1
            $main = StringTrimLeft($temp, StringLen(Cut($temp)))
            $size = DirGetSize($temp & "\" & $file)
            $dir = Cut($temp)
            $dupli = DuplicateCheck($dir, $main)
            $item = GUICtrlCreateListViewItem($main & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
            If $dupli = 1 Then
                GUICtrlSetBkColor($item, 0xFFBFBF)
                GUICtrlCreateListViewItem("1|1", $hListViewH)
            Else
                GUICtrlSetBkColor($item, 0xE5F7FF)
            ;GUICtrlSetBkColor($item, 0xC6F0FF)
                GUICtrlCreateListViewItem("1|0", $hListViewH)
            EndIf
        EndIf
    
    ; load files
        $search = FileFindFirstFile($temp & "\*.*")
        While 1
            $file = FileFindNextFile($search)
            If @error Then
                _GUICtrlListView_EndUpdate($hListViewH)
                ExitLoop
            EndIf
            $i = $i + 1
            $attrib = FileGetAttrib($temp & "\" & $file)
            If Not StringInStr($attrib, "D") Then
                $size = FileGetSize($temp & "\" & $file)
                $dir = $temp & "\"
                $dupli = DuplicateCheck($dir, $file)
                $item = GUICtrlCreateListViewItem($file & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
                If $dupli = 1 Then
                    GUICtrlSetBkColor($item, 0xFFBFBF)
                    GUICtrlCreateListViewItem("0|1", $hListViewH)
                Else
                    GUICtrlCreateListViewItem("0|0", $hListViewH)
                EndIf
            EndIf
        WEnd
        FileClose($search)
        
    ; load folders
        $search = FileFindFirstFile($temp & "\*.*")
        While 1
            $file = FileFindNextFile($search)
            If @error Then
                _GUICtrlListView_EndUpdate($hListViewH)
                ExitLoop
            EndIf
            $i = $i + 1
            $attrib = FileGetAttrib($temp & "\" & $file)
            If StringInStr($attrib, "D") Then
                $size = DirGetSize($temp & "\" & $file)
                $dir = $temp & "\"
                $dupli = DuplicateCheck($dir, $file)
                $item = GUICtrlCreateListViewItem($file & "|" & $dir & "|" & ConvSize($size) & "| - ", $hListView)
                If $dupli = 1 Then
                    GUICtrlSetBkColor($item, 0xFFBFBF)
                    GUICtrlCreateListViewItem("0|1", $hListViewH)
                Else
                    GUICtrlSetBkColor($item, 0xE5F7FF)
                    GUICtrlCreateListViewItem("0|0", $hListViewH)
                EndIf
                $subtemp = $temp & "\" & $file
                Add($subtemp, $mainf)
            EndIf
        WEnd
        _GUICtrlListView_EndUpdate($hListViewH)
        FileClose($search)
    EndIf
EndFunc

SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

..I forgot: in the code I use two listview, one showed that include some info like file name, directory, size.. the other one hidden that include some info about files loaded.

Any advice? :)


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

Have you looked at _GUICtrlListView_AddArray() ?

It's very fast at adding items to a listview, but you'll need to make your file list an array first.

Possible that your array building might offset the speed of the _GuiCtrlListView_AddArray, but you never know.

Share this post


Link to post
Share on other sites

thanks, I'll try it :)

You should put some debug points in there to collect the time it takes to do things. I'll bet you lunch that the real time-waster is DuplicateCheck().

:lmao:


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

Yes! Eureca ^^ ..it seem to be the big problem.

I have to run the duplicate check only at the end of the loading process, not each time a file is loaded. Can you help me to create a function that, given a listview, check if there are lines that are duplicate? thank you very much!

This is my idea (but need to be completed, or changed ^^):

Func DuplicateCheck()
    Local $nIt, $iI, $k, $h, $a, $b, $aa, $bb
; $a[3] = file name      $b[3] = directory
    
    $nIt = _GUICtrlListView_GetItemCount($hListView)
    For $k = 0 To $nIt
        $a = _GUICtrlListView_GetItem($hListView, $k, 0)
        $b = _GUICtrlListView_GetItem($hListView, $k, 1)
        For $h = 0 To $nIt
            If Not $h = $k Then
                $aa = _GUICtrlListView_GetItem($hListView, $h, 0)
                $bb = _GUICtrlListView_GetItem($hListView, $h, 1)
                If $bb[3] = $b[3] Then
                    If $aa[3] = $a[3] Then
                        
                        
                    ; here I need to add a code that change the background color of the corresponding line
                    ; and change a parameter in a second listview that has the same number of items

                        
                    EndIf
                EndIf
            EndIf
        Next
    Next
EndFunc
Edited by Lupo73

SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

Yes! Eureca ^^ ..it seem to be the big problem.

I have to run the duplicate check only at the end of the loading process, not each time a file is loaded. Can you help me to create a function that, given a listview, check if there are lines that are duplicate? thank you very much!

First, you owe me lunch... :)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

I have added a comment in the previous post (I didn't see your answer).. what I need (but I'm not able to create it) is a function that start at the end of the adding process, when the whole listview is loaded, and check the list only one time for duplicates. When the function find a duplicate item, it change the background of the item line and change also an item in a second listview. In particular this second listview is an hidden listview that contains parameters of files loaded in the main listview (so it has the same number and order of lines of the main one)... and I need that if a duplicate is found, in this second listview the corresponding item is changed from 0 to 1.


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

I have added a comment in the previous post (I didn't see your answer)...

I don't see where I get lunch out of this... :lmao:

what I need (but I'm not able to create it) is a function that start at the end of the adding process, when the whole listview is loaded, and check the list only one time for duplicates. When the function find a duplicate item, it change the background of the item line and change also an item in a second listview. In particular this second listview is an hidden listview that contains parameters of files loaded in the main listview (so it has the same number and order of lines of the main one)... and I need that if a duplicate is found, in this second listview the corresponding item is changed from 0 to 1.

The nested For/Next loops are slowest possible way to do this. You can do it with only pass through both listviews: All you have to do is assemble a string of used values, with a delimiter like "|" and just use StringInStr() to test each value against that list for duplicates:
Func DuplicateCheck()
    Local $sSearch = "|"
   
    For $k = 0 To _GUICtrlListView_GetItemCount($hListView) - 1
        $sLVData = _GUICtrlListView_GetItemText($hListView, $k, 1)
        If StringInStr($sSearch, "|" & $sLVData & "|") Then
        ; Stuff to do if duplicate...
        EndIf
        $sSearch &= $sLVData & "|"
    Next
EndFunc

This method is quick enough on each item that it could be removed from the function and done in line with populating the listviews.

:)

P.S. I note you didn't account for ListView item numbers being 0-based, while the sub-items are 1-based. You should check on that.

Edit: Typo, should be _GUICtrlListView_GetItemText()

Edited by PsaltyDS

Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

I have made some changes to the code. I have removed the two listviews, now there is only one listview that for each element loaded use a line (4 columns) + a parameter (using this function _GUICtrlListView_SetItemParam).

So now the process of adding items is very fast, but at the end start the DuplicateCheck() that needs many time to do the checking.

I have created this function to do the check:

Func DuplicateCheck()
    Local $nIt, $iI, $k, $h, $param
    Local $a, $b, $aa, $bb
    
    $nIt = _GUICtrlListView_GetItemCount($hListView)
    For $k = 0 To $nIt - 1
        $a = _GUICtrlListView_GetItemText($hListView, $k, 0)
        $b = _GUICtrlListView_GetItemText($hListView, $k, 1)
        $param = _GUICtrlListView_GetItemParam($hListView, $k)
        If Not ($param = 2) Then
            For $h = 0 To $nIt - 1
                If Not ($h = $k) Then
                    $aa = _GUICtrlListView_GetItemText($hListView, $h, 0)
                    $bb = _GUICtrlListView_GetItemText($hListView, $h, 1)
                    If $bb = $b Then
                        If $aa = $a Then
                            _GUICtrlListView_SetItemParam($hListView, $h, 2)
                            
                        ;GUICtrlSetBkColor($iI, 0xFFBFBF)
                            
                        EndIf
                    EndIf
                EndIf
            Next
        EndIf
    Next
EndFunc

You said your function is really faster, can you explain me how to implement it? I didn't understand what it does.. I need to do something like the function I post (it receive the whole listview and have to detect if there are duplicate items in it..checking the file name, first element of each line, and the directory, second element of each line). Thanks!


SFTPEx, AutoCompleteInput_DateTimeStandard(), _ImageWriteResize()_GUIGraduallyHide(): some AutoIt functions.

Lupo PenSuite: all-in-one and completely free selection of portable programs and games.

DropIt: a personal assistant to automatically manage your files.

ArcThemALL!: application to multi-archive your files and folders.

Share this post


Link to post
Share on other sites

I have made some changes to the code. I have removed the two listviews, now there is only one listview that for each element loaded use a line (4 columns) + a parameter (using this function _GUICtrlListView_SetItemParam).

So now the process of adding items is very fast, but at the end start the DuplicateCheck() that needs many time to do the checking.

I have created this function to do the check:

You said your function is really faster, can you explain me how to implement it? I didn't understand what it does.. I need to do something like the function I post (it receive the whole listview and have to detect if there are duplicate items in it..checking the file name, first element of each line, and the directory, second element of each line). Thanks!

It would be something like this, even simpler since there is only one listview now:
Func DuplicateCheck()
    Local $sSearch = "|"
  
    For $i = 0 To _GUICtrlListView_GetItemCount($hListView) - 1
        If StringInStr($sSearch, "|" & _GUICtrlListView_GetItemText($hListView, $i, 0) & "|") Then
            ; Stuff to do if duplicate...
                
        Else
            ; Not duplicate, add to list
            $sSearch &= $sLVData & "|"
        EndIf
    Next
EndFunc

But it's simple enough now that it doesn't need to be a function (unless you will be doing it from other places in the script). You can just put this check in with the same code that populates the listview, checking for duplicates as it adds them.

:)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0