Sign in to follow this  
Followers 0
duckling78

A faster _FileListToArray

1 post in this topic

I had the task of parsing around 10,000 dxdiags. One of the bottlenecks was _FileListToArray taking a long time to run. I believe one of the bigger bottlenecks was the ReDim inside the While statement.

To improve the performance of this, I made the initial array really big, removed the ReDim from the While loop, and added one ReDim after the While loop ends. This seems to improve performance tremendously. Would something like this be able to make it into AutoIt3? I'm not that great of a scripter so someone smarter than me should probably do the actual writing of this :D

#include <GUIConstants.au3>
#include <File.au3>
#include <Array.au3>

Opt("GUIOnEventMode", 1)
#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("Duplicate Dxdiag Deleter", 445, 55, 193, 125)
GUISetOnEvent($GUI_EVENT_CLOSE, "Form1Close")
GUICtrlCreateLabel("Folder", 6, 8, 33, 17)
$inputFolder = GUICtrlCreateInput(@TempDir, 42, 4, 399, 21)
GUICtrlSetOnEvent(-1, "inputFolderChange")
$btnDeleteDuplicates = GUICtrlCreateButton("Delete Duplicates", 6, 28, 131, 25, 0)
GUICtrlSetOnEvent(-1, "btnDeleteDuplicatesClick")
$lblStatus = GUICtrlCreateLabel("Status: Idle", 141, 34, 300, 22)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

While 1
    Sleep(100)
WEnd

Func btnDeleteDuplicatesClick()
    Blah("Status: Before _FileListToArray")
    $fileList = _FileListToArray2(GUICtrlRead($inputFolder), "*", 1)
    Blah("Status: After _FileListToArray")
    Dim $fileExists[UBound($fileList, 1)][2]
    For $i = 0 To UBound($fileList, 1) - 1
        If Mod($i, 100) == 0 Then
            Blah("Status: " & $i & "/" & UBound($fileList, 1))
        EndIf
        $fileExists[$i][0] = $fileList[$i]
        If $i = 0 Then
            $fileExists[$i][1] = "FileExists"
        Else
            If FileExists(GUICtrlRead($inputFolder) & "\" & $fileList[$i]) Then
                $fileExists[$i][1] = "True"
            Else
                $fileExists[$i][1] = "False"
            EndIf
        EndIf
    Next
    Blah("Finished parsing " & $i & " files.")
    ;_ArrayDisplay($fileExists)
EndFunc
Func Form1Close()
    Exit
EndFunc
Func inputFolderChange()

EndFunc
Func Blah($text = "No text entered in 'Blah()' function")
    GUICtrlSetData($lblStatus, $text)
    ConsoleWrite(@CRLF & $text)
EndFunc

Func _FileListToArray2($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $asFileList[16777216]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    If (StringMid($sPath,StringLen($sPath),1) = "\") Then $sPath  = StringTrimRight($sPath,1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        $asFileList[0] = $asFileList[0] + 1
        $asFileList[UBound($asFileList) - 1] = $sFile
        If Mod($asFileList[0], 100) == 0 Then
            Blah("Status: " & $asFileList[0] & " files parsed")
        EndIf
    WEnd
    FileClose($hSearch)
    ReDim $asFileList[$asFileList[0]+1]
    Return $asFileList
EndFunc   ;==>_FileListToArray

Share this post


Link to post
Share on other sites



Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0