A faster _FileListToArray

[duckling78]

I had the task of parsing around 10,000 dxdiags. One of the bottlenecks was _FileListToArray taking a long time to run. I believe one of the bigger bottlenecks was the ReDim inside the While statement.

To improve the performance of this, I made the initial array really big, removed the ReDim from the While loop, and added one ReDim after the While loop ends. This seems to improve performance tremendously. Would something like this be able to make it into AutoIt3? I'm not that great of a scripter so someone smarter than me should probably do the actual writing of this

expandcollapse popup

#include <GUIConstants.au3>
#include <File.au3>
#include <Array.au3>

Opt("GUIOnEventMode", 1)
#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("Duplicate Dxdiag Deleter", 445, 55, 193, 125)
GUISetOnEvent($GUI_EVENT_CLOSE, "Form1Close")
GUICtrlCreateLabel("Folder", 6, 8, 33, 17)
$inputFolder = GUICtrlCreateInput(@TempDir, 42, 4, 399, 21)
GUICtrlSetOnEvent(-1, "inputFolderChange")
$btnDeleteDuplicates = GUICtrlCreateButton("Delete Duplicates", 6, 28, 131, 25, 0)
GUICtrlSetOnEvent(-1, "btnDeleteDuplicatesClick")
$lblStatus = GUICtrlCreateLabel("Status: Idle", 141, 34, 300, 22)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

While 1
    Sleep(100)
WEnd

Func btnDeleteDuplicatesClick()
    Blah("Status: Before _FileListToArray")
    $fileList = _FileListToArray2(GUICtrlRead($inputFolder), "*", 1)
    Blah("Status: After _FileListToArray")
    Dim $fileExists[UBound($fileList, 1)][2]
    For $i = 0 To UBound($fileList, 1) - 1
        If Mod($i, 100) == 0 Then
            Blah("Status: " & $i & "/" & UBound($fileList, 1))
        EndIf
        $fileExists[$i][0] = $fileList[$i]
        If $i = 0 Then
            $fileExists[$i][1] = "FileExists"
        Else
            If FileExists(GUICtrlRead($inputFolder) & "\" & $fileList[$i]) Then
                $fileExists[$i][1] = "True"
            Else
                $fileExists[$i][1] = "False"
            EndIf
        EndIf
    Next
    Blah("Finished parsing " & $i & " files.")
    ;_ArrayDisplay($fileExists)
EndFunc
Func Form1Close()
    Exit
EndFunc
Func inputFolderChange()

EndFunc
Func Blah($text = "No text entered in 'Blah()' function")
    GUICtrlSetData($lblStatus, $text)
    ConsoleWrite(@CRLF & $text)
EndFunc

Func _FileListToArray2($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $asFileList[16777216]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    If (StringMid($sPath,StringLen($sPath),1) = "\") Then $sPath  = StringTrimRight($sPath,1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        $asFileList[0] = $asFileList[0] + 1
        $asFileList[UBound($asFileList) - 1] = $sFile
        If Mod($asFileList[0], 100) == 0 Then
            Blah("Status: " & $asFileList[0] & " files parsed")
        EndIf
    WEnd
    FileClose($hSearch)
    ReDim $asFileList[$asFileList[0]+1]
    Return $asFileList
EndFunc   ;==>_FileListToArray