Jump to content
Sign in to follow this  
Tlem

Improvement of included _FileListToArray function.

Recommended Posts

In the case of anything that is using a disk read, consistant timing can be a bit difficult due to disk caching on the system. You will probably notice that the first test after system startup takes longer than subsequent tests.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

@spiff59

You are right about the filters not workings. There has to be a way to arrange this.. Im going to think about for a while.

Share this post


Link to post
Share on other sites

@spiff59

You are right about the filters not workings. There has to be a way to arrange this.. Im going to think about for a while.

A second loop is how post 175 ended up, something like:

#include<array.au3>
#include<file.au3>

$Path = @SystemDir
;$Path = "C:\Program Files\Autoit3\AutoitX"
$Filter = "*"
$RetItemType = 0; 0 = files/folder, 1 = files only, 2 = folders only
$RetPathType = 0; 0 = files/folder name only , 1 = relative path, 2 = full path
$Recursive = 0

$repeat = 100

$timer = TimerInit()
For $j = 1 to $repeat
    $x = _FLTA_NonRecursiveRecursion($Path, $Filter, $RetItemType, $RetPathType, $Recursive)
Next
$t1 = TimerDiff ($timer)
_ArrayDisplay($x)

$timer = TimerInit()
For $j = 1 to $repeat
    $x = _FileListToArray($Path, $Filter, $RetItemType)
Next
$t2 = TimerDiff ($timer)
_ArrayDisplay($x)

MsgBox (0, "", "$iRetItemType = " & $RetItemType & @CRLF &"Ver1:  " & Int($t1/10)/100 & @CRLF & "Ver2:  " & Int($t2/10)/100)


;===============================================================================
Func _FLTA_NonRecursiveRecursion($sPath, $sFilter= "*", $iRetItemType = 0, $iRetPathType = 0, $bRecursive = False)
    Local $sRet = "", $sRetPath = ""
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "")
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    $sPath &= "\|"
    $sOrigPathLen = StringLen($sPath) - 1
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|") - 1
        $sCurrPath = StringLeft($sPath, $sCurrPathLen)
        $search = FileFindFirstFile($sCurrPath & $sFilter)
        If @error Then
            $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
            ContinueLoop
        EndIf
        Switch $iRetPathType
            Case 1 ; relative path
                $sRetPath = StringTrimLeft($sCurrPath, $sOrigPathLen)
            Case 2 ; full path
                $sRetPath = $sCurrPath
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If ($iRetItemType + @extended = 2) Then ContinueLoop
            $sRet &= $sRetPath & $file & "|"
        WEnd
        FileClose($search)
        If $bRecursive Then
            $hSearch = FileFindFirstFile($sCurrPath & "*")
            While 1
                $file = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended Then $sPath &= $sCurrPath & $file & "\|"
            WEnd
            FileClose($hSearch)
        EndIf
        $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc  ;==>_FLTA_NonRecursiveRecursion

42 lines

Edited by Spiff59

Share this post


Link to post
Share on other sites

Spiff59, I can tell you right now, it's unlikely I will ever include any version of this function that contains multiple virtually identical while loops.

Share this post


Link to post
Share on other sites

Looking at the first time in those functions i also did not understood, why to use many loops on one parameter?

I would do it like this (also added FileGetAttrib method to check if the current path is dir, so the function will work on latest final release):

#include <Array.au3>
#include <File.au3>
;

;$Path = @SystemDir
$Path = @ProgramFilesDir & "\AutoIt3\AutoItX"
$Filter = "*"
$RetItemType = 1 ;0 = files/folder, 1 = files only, 2 = folders only
$RetPathType = 1 ;0 = files/folder name only , 1 = relative path, 2 = full path
$Recursive = 1

$x = _FLTA_NonRecursiveRecursion($Path, $Filter, $RetItemType, $RetPathType, $Recursive)

_ArrayDisplay($x)

;===============================================================================
Func _FLTA_NonRecursiveRecursion($sPath, $sFilter, $iRetItemType, $iRetPathType, $bRecursive)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "")
    
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    
    $sPath &= "\|"
    Local $sRet = "", $sRetPath = "", $sOrigPathLen = StringLen($sPath) - 1
    
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|") - 1
        $sCurrPath = StringLeft($sPath, $sCurrPathLen)
        $hSearch = FileFindFirstFile($sCurrPath & $sFilter)
        
        If @error Then
            $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
            ContinueLoop
        EndIf
        
        Switch $iRetPathType
            Case 1 ;Relative path
                $sRetPath = StringTrimLeft($sCurrPath, $sOrigPathLen)
            Case 2 ;Full path
                $sRetPath = $sCurrPath
        EndSwitch
        
        While 1
            $sNextFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            
            Switch $iRetItemType
                Case 1 ;Files only
                    If @extended = 0 And Not StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                        $sRet &= $sRetPath & $sNextFile & "|"
                Case 2 ;Folders only
                    If @extended Or StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                        $sRet &= $sRetPath & $sNextFile & "|"
                Case Else ;Files and Folders
                    $sRet &= $sRetPath & $sNextFile & "|"
            EndSwitch
        WEnd
        
        FileClose($hSearch)
        
        If $bRecursive Then
            $hSearch = FileFindFirstFile($sCurrPath & "*")
            
            While 1
                $sNextFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                
                If @extended Or StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                    $sPath &= $sCurrPath & $sNextFile & "\|"
            WEnd
            
            FileClose($hSearch)
        EndIf
        
        $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
    WEnd
    
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc ;==>_FLTA_NonRecursiveRecursion

 

Spoiler

Using OS: Win 7 Professional, Using AutoIt Ver(s): 3.3.6.1 / 3.3.8.1

AutoIt_Rus_Community.png AutoIt Russian Community

My Work...

Spoiler

AutoIt_Icon_small.pngProjects: ATT - Application Translate Tool {new}| BlockIt - Block files & folders {new}| SIP - Selected Image Preview {new}| SISCABMAN - SciTE Abbreviations Manager {new}| AutoIt Path Switcher | AutoIt Menu for Opera! | YouTube Download Center! | Desktop Icons Restorator | Math Tasks | KeyBoard & Mouse Cleaner | CaptureIt - Capture Images Utility | CheckFileSize Program

AutoIt_Icon_small.pngUDFs: OnAutoItErrorRegister - Handle AutoIt critical errors {new}| AutoIt Syntax Highlight {new}| Opera Library! | Winamp Library | GetFolderToMenu | Custom_InputBox()! | _FileRun UDF | _CheckInput() UDF | _GUIInputSetOnlyNumbers() UDF | _FileGetValidName() UDF | _GUICtrlCreateRadioCBox UDF | _GuiCreateGrid() | _PathSplitByRegExp() | _GUICtrlListView_MoveItems - UDF | GUICtrlSetOnHover_UDF! | _ControlTab UDF! | _MouseSetOnEvent() UDF! | _ProcessListEx - UDF | GUICtrl_SetResizing - UDF! | Mod. for _IniString UDFs | _StringStripChars UDF | _ColorIsDarkShade UDF | _ColorConvertValue UDF | _GUICtrlTab_CoverBackground | CUI_App_UDF | _IncludeScripts UDF | _AutoIt3ExecuteCode | _DragList UDF | Mod. for _ListView_Progress | _ListView_SysLink | _GenerateRandomNumbers | _BlockInputEx | _IsPressedEx | OnAutoItExit Handler | _GUICtrlCreateTFLabel UDF | WinControlSetEvent UDF | Mod. for _DirGetSizeEx UDF
 
AutoIt_Icon_small.pngExamples: 
ScreenSaver Demo - Matrix included | Gui Drag Without pause the script | _WinAttach()! | Turn Off/On Monitor | ComboBox Handler Example | Mod. for "Thinking Box" | Cool "About" Box | TasksBar Imitation Demo

Like the Projects/UDFs/Examples? Please rate the topic (up-right corner of the post header: Rating AutoIt_Rating.gif)

* === My topics === *

==================================================
My_Userbar.gif
==================================================

 

 

 

AutoIt is simple, subtle, elegant. © AutoIt Team

Share this post


Link to post
Share on other sites

Spiff59, I can tell you right now, it's unlikely I will ever include any version of this function that contains multiple virtually identical while loops.

Otay. Prior post updated. It would have been 56 lines with the one-liner to correct the production bug. It's a bit slower, but is down to 43 lines. It has optional folder recursion, a choice of what type of pathnames are returned (Zedna's BugTrac), the fix for the production version, is backwards compatible, and still much faster than the 3.3.1.1 version.

It is also, of course untested, and I've chosen to not edit the $iRetItemType and $iRetPathType variables and instead default invalid values. I'm not sure how that practice will be received.

Share this post


Link to post
Share on other sites

Need more testing, but this one is a combination of Wus information and bchris01 script.

; #FUNCTION# ================================================================================
; Name...........: _FileListToArrayEx
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = '*'[, $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = '', $iRetFormat = 1]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                  $sFilter - Optional: the filter to use, default is *, semicolon delimited. Search the Autoit3 helpfile for the word 'WildCards' For details.
;                             (Example: '*.exe;*.txt')
;                  $iFlag   - Optional: specifies whether to return files folders or both
;                    0 = (Default) Return both files and folders
;                    1 = Return files only
;                    2 = Return Folders only
;                  $iPathType
;                    0 = relative path
;                    1 = (Default) full path
;                  $iRecursive - Search files in specified directory and all subdirectories.
;                    0 = (Default) Search in specified directory.
;                    1 = Search in specified and all subdirectories.
;                  $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                    (Example: 'Unins*' will remove all files/folders that begin with 'Unins')
;                  $iRetFormat =  optional: return format
;                    0 = String ( '|' delimited)
;                    1 = (Default) one-dimensional array, 1-based
;                    2 = one-dimensional array, 0-based
;                  $sDelim - Delimiter for combined string.
; Return values .: @Error -
;                    1 = Path not found or invalid
;                    2 = Invalid $sFilter
;                    3 = Invalid $iFlag
;                    4 = No File(s) Found
;                    5 = Invalid $iPathType
;                    6 = Invalid $iRecursive
;                    7 = Invalid $iRetFormat
; Author ........:
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                    $array[0] = Number of Files\Folders returned
;                    $array[1] = 1st File\Folder
;                    $array[2] = 2nd File\Folder
;                    $array[3] = 3rd File\Folder
;                    $array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ===========================================================================================

Func _FileListToArrayEx($sPath, $sFilter = '*', $iFlag = 0, $iPathType = 1, $bRecursive = 0, $sExclude = '', $iRetFormat = 1, $sDelim = '|')
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist, $aFolderStack[2]

    $sPath = StringRegExpReplace($sPath, '[\\/]+\z', '')
    If Not FileExists($sPath) Then Return SetError(1, 1, '')
    If $sFilter = '' Or $sFilter = '-1' Or $sFilter = Default Then $sFilter = '*'
    If StringRegExp($sFilter, '[:\\</>\|]|(?s)\A\s*\z') Then Return SetError(2, 2, '')
    If Not StringRegExp($iFlag, '[012]') Then Return SetError(3, 3, '')
    If Not StringRegExp($iPathType, '[01]') Then Return SetError(5, 5, '')
    If Not StringRegExp($bRecursive, '[01]') Then Return SetError(6, 6, '')
    If Not StringRegExp($iRetFormat, '[012]') Then Return SetError(7, 7, '')
    $sFilter = StringReplace(StringStripWS(StringRegExpReplace($sFilter, '\s*;\s*', ';'), 3), ';', '|')
    $sMask = '(?i)^' & StringReplace(StringReplace(StringReplace($sFilter, '.', '\.'), '*', '.*'), '?', '.')
    $sExclude = StringStripWS(StringRegExpReplace($sExclude, '\s*;\s*', ';'), 3)
    $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1')
    $sExclude = '(?i)\A' & StringReplace(StringReplace(StringReplace($sExclude, '?', '.'), '*', '.*?'), ';', '|') & '\z'

    $aFolderStack[0] = 1
    $aFolderStack[1] = $sPath
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\*')
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Or StringInStr(FileGetAttrib($sWorkingdir & '\' & $sFile), 'D') Then
                If $bRecursive Then ; Increse the FolderStack
                    If $sExclude <> '' And StringRegExp(StringRegExpReplace($sWorkingdir & '\' & $sFile, '(.*?[\\/]+)*(.*?\z)', '\2'), $sExclude) Then ContinueLoop
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & '\' & $sFile
                EndIf
                If $iFlag <> 1 Then ; Add Folders
                    If Not $bRecursive And $sExclude <> '' And StringRegExp(StringRegExpReplace($sWorkingdir & '\' & $sFile, '(.*?[\\/]+)*(.*?\z)', '\2'), $sExclude) Then ContinueLoop
                    If Not StringRegExp($sFile, $sMask) Then ContinueLoop
                    $asFilelist &= $sWorkingdir & '\' & $sFile & $sDelim
                EndIf
            Else ; Add Files
                If $iFlag <> 2 Then
                    If Not StringRegExp($sFile, $sMask) Then ContinueLoop
                    If Not StringRegExp(StringRegExpReplace($sFilter, '^.*\.', ''), '[?*]') Then
                        If StringInStr($sFilter, StringRegExpReplace($sFile, '^.*\.', '')) = 0 Then ContinueLoop
                    EndIf
                    If $sExclude <> '' And StringRegExp(StringRegExpReplace($sWorkingdir & '\' & $sFile, '(.*?[\\/]+)*(.*?\z)', '\2'), $sExclude) Then ContinueLoop
                    $asFilelist &= $sWorkingdir & '\' & $sFile & $sDelim
                EndIf
            EndIf
        WEnd
        FileClose($hSearch)
    WEnd

    $asFilelist = StringTrimRight($asFilelist, 1)
    If StringLen($asFilelist) = 0 Then Return SetError(4, 4, '')
    If $iPathType = 0 Then $asFilelist = StringReplace($asFilelist, $sPath & '\', '')
    If $iRetFormat <> 0 Then $asFilelist = StringSplit($asFilelist, $sDelim, $iRetFormat)
    Return $asFilelist
EndFunc   ;==>_FileListToArrayEx

Note: Fixed error if try to do a nonrecursive or recursive search for only folders using some kind of filter.

Edited by Danny35d

AutoIt Scripts:NetPrinter - Network Printer UtilityRobocopyGUI - GUI interface for M$ robocopy command line

Share this post


Link to post
Share on other sites

why to use many loops on one parameter?

Well, my reasoning was that when a large directory, or multiple large directories, are searched that the function spends the majority of it's time in the innermost while loop. Splitting the while loop into 3 parts eliminated from 2 or 3 compares per file/folder read. It made, if I recall, up to an additional 13% difference in performance depending on the value of $iRetItemType ($iFlag) and $sPath.

Edit: But I've decided to be happy without that particular tweak :)

Edited by Spiff59

Share this post


Link to post
Share on other sites

Looking at the first time in those functions i also did not understood, why to use many loops on one parameter?

I would do it like this (also added FileGetAttrib method to check if the current path is dir, so the function will work on latest final release):

#include <Array.au3>
#include <File.au3>
;

;$Path = @SystemDir
$Path = @ProgramFilesDir & "\AutoIt3\AutoItX"
$Filter = "*"
$RetItemType = 1 ;0 = files/folder, 1 = files only, 2 = folders only
$RetPathType = 1 ;0 = files/folder name only , 1 = relative path, 2 = full path
$Recursive = 1

$x = _FLTA_NonRecursiveRecursion($Path, $Filter, $RetItemType, $RetPathType, $Recursive)

_ArrayDisplay($x)

;===============================================================================
Func _FLTA_NonRecursiveRecursion($sPath, $sFilter, $iRetItemType, $iRetPathType, $bRecursive)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "")
    
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    
    $sPath &= "\|"
    Local $sRet = "", $sRetPath = "", $sOrigPathLen = StringLen($sPath) - 1
    
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|") - 1
        $sCurrPath = StringLeft($sPath, $sCurrPathLen)
        $hSearch = FileFindFirstFile($sCurrPath & $sFilter)
        
        If @error Then
            $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
            ContinueLoop
        EndIf
        
        Switch $iRetPathType
            Case 1 ;Relative path
                $sRetPath = StringTrimLeft($sCurrPath, $sOrigPathLen)
            Case 2 ;Full path
                $sRetPath = $sCurrPath
        EndSwitch
        
        While 1
            $sNextFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            
            Switch $iRetItemType
                Case 1 ;Files only
                    If @extended = 0 And Not StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                        $sRet &= $sRetPath & $sNextFile & "|"
                Case 2 ;Folders only
                    If @extended Or StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                        $sRet &= $sRetPath & $sNextFile & "|"
                Case Else ;Files and Folders
                    $sRet &= $sRetPath & $sNextFile & "|"
            EndSwitch
        WEnd
        
        FileClose($hSearch)
        
        If $bRecursive Then
            $hSearch = FileFindFirstFile($sCurrPath & "*")
            
            While 1
                $sNextFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                
                If @extended Or StringInStr(FileGetAttrib($sCurrPath & $sNextFile), "D") Then _
                    $sPath &= $sCurrPath & $sNextFile & "\|"
            WEnd
            
            FileClose($hSearch)
        EndIf
        
        $sPath = StringTrimLeft($sPath, $sCurrPathLen + 1)
    WEnd
    
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc ;==>_FLTA_NonRecursiveRecursion

I don't think the filters are working on this script either..

Share this post


Link to post
Share on other sites

I don't think the filters are working on this script either..

I didn't trying to fix anything related to filters, i just showed how i would do that instead of using many loops.

 

Spoiler

Using OS: Win 7 Professional, Using AutoIt Ver(s): 3.3.6.1 / 3.3.8.1

AutoIt_Rus_Community.png AutoIt Russian Community

My Work...

Spoiler

AutoIt_Icon_small.pngProjects: ATT - Application Translate Tool {new}| BlockIt - Block files & folders {new}| SIP - Selected Image Preview {new}| SISCABMAN - SciTE Abbreviations Manager {new}| AutoIt Path Switcher | AutoIt Menu for Opera! | YouTube Download Center! | Desktop Icons Restorator | Math Tasks | KeyBoard & Mouse Cleaner | CaptureIt - Capture Images Utility | CheckFileSize Program

AutoIt_Icon_small.pngUDFs: OnAutoItErrorRegister - Handle AutoIt critical errors {new}| AutoIt Syntax Highlight {new}| Opera Library! | Winamp Library | GetFolderToMenu | Custom_InputBox()! | _FileRun UDF | _CheckInput() UDF | _GUIInputSetOnlyNumbers() UDF | _FileGetValidName() UDF | _GUICtrlCreateRadioCBox UDF | _GuiCreateGrid() | _PathSplitByRegExp() | _GUICtrlListView_MoveItems - UDF | GUICtrlSetOnHover_UDF! | _ControlTab UDF! | _MouseSetOnEvent() UDF! | _ProcessListEx - UDF | GUICtrl_SetResizing - UDF! | Mod. for _IniString UDFs | _StringStripChars UDF | _ColorIsDarkShade UDF | _ColorConvertValue UDF | _GUICtrlTab_CoverBackground | CUI_App_UDF | _IncludeScripts UDF | _AutoIt3ExecuteCode | _DragList UDF | Mod. for _ListView_Progress | _ListView_SysLink | _GenerateRandomNumbers | _BlockInputEx | _IsPressedEx | OnAutoItExit Handler | _GUICtrlCreateTFLabel UDF | WinControlSetEvent UDF | Mod. for _DirGetSizeEx UDF
 
AutoIt_Icon_small.pngExamples: 
ScreenSaver Demo - Matrix included | Gui Drag Without pause the script | _WinAttach()! | Turn Off/On Monitor | ComboBox Handler Example | Mod. for "Thinking Box" | Cool "About" Box | TasksBar Imitation Demo

Like the Projects/UDFs/Examples? Please rate the topic (up-right corner of the post header: Rating AutoIt_Rating.gif)

* === My topics === *

==================================================
My_Userbar.gif
==================================================

 

 

 

AutoIt is simple, subtle, elegant. © AutoIt Team

Share this post


Link to post
Share on other sites

Okay this one should not give any major problems (I hope). All filters should be working for recursive and non-recursive. I did quite a bit or testing. This one again is only for beta. (31 lines)

Func _FileListToArrayEx100000($sPath, $sFilter = "*", $iFlag = 0, $bRecursive = False)
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not StringRegExp($iFlag, '[012]') Then Return SetError(3, 3, "")
    If $sFilter <> "*" Then $sFilter = "(?i)\A(" & StringReplace(StringReplace(StringReplace(StringRegExpReplace(StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", ""), '[.$]', '\[\0\]'), "?", "."), "*", ".*?"), ";", "$|") & "$)"
    Local $aFolderStack[2] = [1, StringRegExpReplace($sPath, "[\\]+\z", "")]
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\*')
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then
                If $bRecursive Then
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & "\" & $sFile
                EndIf
                If $iFlag = 1 Or ($sFilter <> "*" And Not StringRegExp($sFile, $sFilter)) Then ContinueLoop
            Else
                If $iFlag = 2 Or ($sFilter <> "*" And Not StringRegExp($sFile, $sFilter)) Then ContinueLoop
            EndIf
            $asFilelist &= $sWorkingdir & "\" & $sFile & '|'
        WEnd
        FileClose($hSearch)
    WEnd
    If StringRight($asFilelist, 1) = '|' Then $asFilelist = StringTrimRight($asFilelist, 1)
    Return StringSplit($asFilelist, '|')
EndFunc   ;==>_FileListToArrayEx100000
Edited by bchris01

Share this post


Link to post
Share on other sites

I thought I'd try whipping the problem of having both an include parameter, and an exclude parameter. It really causes a lot of grief when those are implemented as the combinations between them (and also the "files and/or folders" flag, and the manner in which FileFindFirstFile behaves) make for a lot of confusion. Just one type of example would be, if you want to list all the '*.exe' files on your drive, but exclude the 'Windows' directory, a file named 'WindowsPlus.exe' (for example) would be excluded, even though it resided in a folder other than 'Windows'.

Separate include and exclude parameters for files, and for folders, is the route I've tried. I think the following works pretty well. It accepts semi-colon delimited lists for both the includelist, the excludelist, and the EXCLUDEfolderlist. It scans recursively in a single pass, never rereading the same directory. The Regular Expression seems to function better than others I've seen so far. Try out '*.exe', '*.exe*', '*.exe.*' or '*.*.*' on your @systemdir.

Func _FileListToArrayZ($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""

    ; Edit include path (strip trailing slashes, and append single slash)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")

    ; Edit exclude folders list
    If $sExcludeFolderList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeFolderList = StringRegExpReplace(StringRegExpReplace($sExcludeFolderList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeFolderList = StringRegExpReplace($sExcludeFolderList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeFolderList = StringReplace(StringReplace($sExcludeFolderList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeFolderList = "(?i)\A(?!" & StringReplace($sExcludeFolderList, ";", "$|")  & "$)"
    EndIf

    ; Edit include files list
    If $sIncludeList ="*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sIncludeList = "(?i)\A(" & StringReplace($sIncludeList, ";", "$|")  & "$)"
    EndIf

    ; Edit exclude files list
    If $sExcludeList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeList = StringRegExpReplace(StringRegExpReplace($sExcludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeList = StringRegExpReplace($sExcludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeList = StringReplace(StringReplace($sExcludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeList = "(?i)\A(?!" & StringReplace($sExcludeList, ";", "$|")  & "$)"
    EndIf

;   MsgBox(1,"Masks","File include: " & $sIncludeList & @CRLF & "File exclude: " & $ExcludeList & @CRLF _
;           & "Dir include : " & $FolderInclude & @CRLF & "Dir exclude : " & $ExcludeFolderList)

    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolderList And Not StringRegExp($file, $sExcludeFolderList) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & "\"
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            If $sExcludeList And Not StringRegExp($file, $sExcludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|"
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

Edit: changed the name of the routine

Edited by Spiff59

Share this post


Link to post
Share on other sites

Based on some new suggestions, I have developed a following version of _FileListToArrayNT7 --> _FileListToArrayXT

Some preliminary remarks:

This version is designed for AutoIt Version 3.3.1.1 or newer.

Otherwise you have to change all --> @extended <-- back to --> StringInStr(FileGetAttrib($sPath & $sFile), "D") <> 0 <-- (not tested)

In no circumstances a recommendation will be made to include it in a general UDF. ;-)

So use it or not , it is your decision.

According to a suggestion from Spiff59 I have changed the search strategy in some cases:

old _FileListToArrayNT7 (post #175): filtered search for each filter (strategy 1).

If recursion (search in subdirs) is necessary then additional unfiltered searches for subdirs are done.

new _FileListToArrayXT:

if no recursion: search like _FileListToArrayNT7 (strategy 1).

if recursion: do an unfiltered search for all items and filter out unwanted items (strategy 2).

To my surprise, this combination of search strategies gives an impressive speed improvement (up to 100%),

especially if more than on filter is used (ex: "*.avi;*,mpg;*.flv;*.mov")

There are also two branches of code in each case of recursion (if excluding is set or not) for speed improvement.

(and due to the new pseudo-recursion the unpractical "do not use" parameter $sWorkPath could be removed)

This results in noticeable more code lines (please don't flame therefore), but the operating profit is again a really good speed improvement

(compared to the already fast _FileListToArrayNT7) in most of the cases.

The functionality is still the same as in _FileListToArrayNT7 (multi-path, multi-filter, recursion, exclude and so on)

Two small changes:

Excluding now depends on $iRetItemType: If files only, excluding is only done for file names, otherwise also for folder names.

Search for "*.dat" returns only *.dat files (so no *.data will be returned as in the case of dir command)

Example:

Search for all video files on disk C: and D: , but no videos with "demo" or "test" in filename:

$aRetArray = _FileListToArrayXT("C:\;D:\", "*.avi;*.mpg;*.mpeg;*.mp4;*.mkv;*.flv;*.ogg;*.ogm;*.mov", 1, 2, True, "*demo*;*test*")

The first run is as usual somewhat slower (filling cache), but the next runs will be very fast (reading from cache).

Some test results (all done out of cache):

Test1: path: System32-Dir, filter="*.dll", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 7.70 , 1832 items

_FileListToArrayNT7: Sec = 4.61 , 1832 items

_FileListToArrayXT: Sec = 4.62 , 1832 items

Test2: path: System32-Dir, filter="*", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 12.80 , 3114 items

_FileListToArrayNT7: Sec = 7.30 , 3114 items

_FileListToArrayXT: Sec = 7.31 , 3114 items

Test3: path: System32-Dir, filter="*nothing*.xyz", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 0.53 , 0 items

_FileListToArrayNT7: Sec = 0.56 , 0 items

_FileListToArrayXT: Sec = 0.56 , 0 items

Test4: path: System32-Dir, filter="*.dll", return files/folders, full path, recursion, no exclusion, repeats 50

_FileListToArrayNT7: Sec = 7.52 , 2895 items

_FileListToArrayXT: Sec = 6.44 , 2895 items

Test5: path: System32-Dir, filter="*.dll;*.exe", return files/folders, full path, recursion, no exclusion, repeats 50

_FileListToArrayNT7: Sec = 9.14 , 3555 items

_FileListToArrayXT: Sec = 7.03 , 3555 items

Test6: path: System32-Dir, filter="*.dll;*.exe;*.sys;*.log", return files/folders, full path, recursion, no exclusion, repeats 50

_FileListToArrayNT7: Sec = 12.06 , 4180 items

_FileListToArrayXT: Sec = 7.88 , 4180 items

Test7: path: System32-Dir, filter="*", return files/folders, full path, recursion, exclusion: "*.dll;*.exe", repeats 50

_FileListToArrayNT7: Sec = 12.08 , 3669 items

_FileListToArrayXT: Sec = 8.16 , 3669 items

Test8: path: HDD C:, filter="*.avi;*.mpg;*.mpeg;*.mp4;*.mkv;*.flv;*.ogg;*.ogm;*.mov", return files, full path, recursion, exclusion: "*Demo*.exe", repeats 1

_FileListToArrayNT7: Sec = 16.16 , 116 items

_FileListToArrayXT: Sec = 8.72 , 116 items

Test9: path: Test-Dir (13608 files, 1092 folders), 6 filters, return files, full path, recursion, 1 exclusion, repeats 30

_FileListToArrayNT7: Sec = 36.94 , 10750 items

_FileListToArrayXT: Sec = 17.05 , 10750 items

As you can see, the performance isn't bad at all.

Please make your own tests to check stability and functionality and report if you find a bug.

_FileListToArrayXT (For Spiff59: _FileListToArraySuperTurbo5000ExtraPlusProWithMaximumUltraPowerboost)

; #FUNCTION# ===========================================================================================
; Name:             _FileListToArrayXT
; Description:      Lists files and\or folders in specified path(s) (Similar to using Dir with the /B Switch)
;                   additional features: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search
; Syntax:           _FileListToArrayXT([$sPath = @ScriptDir, [$sFilter = "*", [$iRetItemType, [$bRecursive = False, [$sExclude = "", [$iRetFormat = 1]]]]]])
; Parameter(s):     $sPath = optional: Search path(s), semicolon delimited (default: @ScriptDir)
;                            (Example: "C:\Tmp;D:\Temp")
;                   $sFilter = optional: Search filter(s), semicolon delimited . Wildcards allowed. (default: "*")
;                              (Example: "*.exe;*.txt")
;                   $iRetItemType = Include in search: 0 = Files and Folder, 1 = Files Only, 2 = Folders Only
;                   $iRetPathType = Returned element format: 0 = file/folder name only, 1 = relative path, 2 = full path
;                   $bRecursive = optional: True: recursive search including all subdirectories
;                                           False (default): search only in specified folder
;                   $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                               (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                   $iRetFormat =  optional: return format
;                                  0 = one-dimensional array, 0-based
;                                  1 = one-dimensional array, 1-based (default)
;                                  2 = String ( "|" delimited)
; Requirement(s):   AutoIt Version 3.3.1.1 or newer
; Return Value(s):  on success: 1-based or 0-based array or string (dependent on $iRetFormat)
;                   If no path is found, @error and @extended are set to 1, returns empty string
;                   If no filter is found, @error and @extended are set to 2, returns empty string
;                   If $iRetFormat is invalid, @error and @extended are set to 3, returns empty string
;                   If no data is found, @error and @extended are set to 4, returns empty string
; Author(s):        Half the AutoIt Community
; ====================================================================================================
Func _FileListToArrayXT($sPath = @ScriptDir, $sFilter = "*", $iRetItemType = 0, $iRetPathType = 0, $bRecursive = False, $sExclude = "", $iRetFormat = 1)
  Local $hSearchFile, $sFile, $sFileList, $sWorkPath, $sRetPath, $iRootPathLen, $iPCount, $iFCount, $fDirFlag

  ;[check and prepare parameters]
  ;---------------
  If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
  ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
  $sPath = StringRegExpReplace(StringRegExpReplace($sPath, "(\s*;\s*)+", ";"), "\A;|;\z", "")
  ;check that at least one path is set
  If $sPath = "" Then Return SetError(1, 1, "")
  ;-----
  If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
  ;prepare filter
  ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
  $sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", "")
  ;check for invalid chars or that at least one filter is set
  If StringRegExp($sFilter, "[\\/><:\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
  If $bRecursive Then
    ;Convert $sFilter for Regular Expression
    $sFilter = StringRegExpReplace($sFilter, '([\Q\.+[^]$(){}=!\E])', '\\$1')
    $sFilter = StringReplace($sFilter, "?", ".")
    $sFilter = StringReplace($sFilter, "*", ".*?")
    $sFilter = "(?i)\A(" & StringReplace($sFilter, ";", "$|") & "$)" ;case-insensitive, convert ';' to '|', match from first char, terminate strings
    ;$sFilter = "(?i)\A" & StringReplace($sFilter, ";", "|") & "\z"
  EndIf
  ;-----
  If $iRetItemType <> "1" And $iRetItemType <> "2" Then $iRetItemType = "0"
  ;-----
  If $iRetPathType <> "1" And $iRetPathType <> "2" Then $iRetPathType = "0"
  ;-----
  $bRecursive = ($bRecursive = "1")
  ;-----
  If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
  If $sExclude Then
    ;prepare $sExclude
    ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(\s*;\s*)+", ";"), "\A;|;\z", "")
    ;Convert $sExclude for Regular Expression
    $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1')
    $sExclude = StringReplace($sExclude, "?", ".")
    $sExclude = StringReplace($sExclude, "*", ".*?")
    $sExclude = "(?i)\A(" & StringReplace($sExclude, ";", "$|") & "$)" ;case-insensitive, convert ';' to '|', match from first char, terminate strings
    ;$sExclude = "(?i)\A" & StringReplace($sExclude, ";", "|") & "\z"
  EndIf
  ;-----
  ;If $iRetFormat <> "0" And $iRetFormat <> "2" Then $iRetFormat = "1"
  If Not ($iRetItemType = 0 Or $iRetItemType = 1 Or $iRetItemType = 2) Then Return SetError(3, 3, "")
  ;---------------
  ;[/check and prepare parameters]

  ;---------------

  Local $aPath = StringSplit($sPath, ';', 1) ;paths array
  Local $aFilter = StringSplit($sFilter, ';', 1) ;filters array

  ;---------------

  If $bRecursive Then ;different handling for recursion (strategy: unfiltered search for all items and filter unwanted)

    If $sExclude Then ;different handling dependent on $sExclude parameter is set or not

      For $iPCount = 1 To $aPath[0] ;Path loop
        $sPath = StringRegExpReplace($aPath[$iPCount], "[\\/]+\z", "") & "\" ;ensure exact one trailing slash
        If Not FileExists($sPath) Then ContinueLoop
        $iRootPathLen = StringLen($sPath) - 1

        Local $aPathStack[1024] = [1, $sPath]

        While $aPathStack[0] > 0
          $sWorkPath = $aPathStack[$aPathStack[0]]
          $aPathStack[0] -= 1
          ;-----
          $hSearchFile = FileFindFirstFile($sWorkPath & '*')
          If @error Then ContinueLoop
          ;-----
          Switch $iRetPathType
            Case 2 ;full path
              $sRetPath = $sWorkPath
            Case 1 ;relative path
              $sRetPath = StringTrimLeft($sWorkPath, $iRootPathLen + 1)
          EndSwitch
          ;-----
          Switch $iRetItemType
            Case 1
              While True ;Files only
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                $fDirFlag = @extended
                If $fDirFlag Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                  ContinueLoop
                EndIf
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                If StringRegExp($sFile, $sFilter) Then
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
            Case 2
              While True ;Folders only
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                $fDirFlag = @extended
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                If $fDirFlag Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                  If StringRegExp($sFile, $sFilter) Then
                    $sFileList &= $sRetPath & $sFile & "|"
                  EndIf
                EndIf
              WEnd
            Case Else
              While True ;Files and Folders
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                $fDirFlag = @extended
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                If $fDirFlag Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                EndIf
                If StringRegExp($sFile, $sFilter) Then
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
          EndSwitch
          ;-----
        WEnd

        FileClose($hSearchFile)

      Next ;$iPCount - next path

    Else ;If Not $sExclude

      For $iPCount = 1 To $aPath[0] ;Path loop
        $sPath = StringRegExpReplace($aPath[$iPCount], "[\\/]+\z", "") & "\" ;ensure exact one trailing slash
        If Not FileExists($sPath) Then ContinueLoop
        $iRootPathLen = StringLen($sPath) - 1

        Local $aPathStack[1024] = [1, $sPath]

        While $aPathStack[0] > 0
          $sWorkPath = $aPathStack[$aPathStack[0]]
          $aPathStack[0] -= 1
          ;-----
          $hSearchFile = FileFindFirstFile($sWorkPath & '*')
          If @error Then ContinueLoop
          ;-----
          Switch $iRetPathType
            Case 2 ;full path
              $sRetPath = $sWorkPath
            Case 1 ;relative path
              $sRetPath = StringTrimLeft($sWorkPath, $iRootPathLen + 1)
          EndSwitch
          ;-----
          Switch $iRetItemType
            Case 1
              While True ;Files only
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                  ContinueLoop
                EndIf
                If StringRegExp($sFile, $sFilter) Then
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
            Case 2
              While True ;Folders only
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                  If StringRegExp($sFile, $sFilter) Then
                    $sFileList &= $sRetPath & $sFile & "|"
                  EndIf
                EndIf
              WEnd
            Case Else
              While True ;Files and Folders
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then
                  $aPathStack[0] += 1
                  If UBound($aPathStack) <= $aPathStack[0] Then ReDim $aPathStack[UBound($aPathStack) * 2]
                  $aPathStack[$aPathStack[0]] = $sWorkPath & $sFile & "\"
                EndIf
                If StringRegExp($sFile, $sFilter) Then
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
          EndSwitch
          ;-----
        WEnd

        FileClose($hSearchFile)

      Next ;$iPCount - next path

    EndIf ;If $sExclude

  Else ;If Not $bRecursive (strategy: filtered search for items)

    If $sExclude Then ;different handling dependent on $sExclude parameter is set or not

      For $iPCount = 1 To $aPath[0] ;Path loop

        $sPath = StringRegExpReplace($aPath[$iPCount], "[\\/]+\z", "") & "\" ;ensure exact one trailing slash
        If Not FileExists($sPath) Then ContinueLoop
        ;-----
        Switch $iRetPathType
          Case 2 ;full path
            $sRetPath = $sPath
          Case 1 ;relative path
            $sRetPath = ""
        EndSwitch

        For $iFCount = 1 To $aFilter[0] ;filter loop
          ;-----
          $hSearchFile = FileFindFirstFile($sPath & $aFilter[$iFCount])
          If @error Then ContinueLoop
          ;-----
          Switch $iRetItemType
            Case 1 ;files Only
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then ContinueLoop ;bypass folder
                ;check for exclude files
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                $sFileList &= $sRetPath & $sFile & "|"
              WEnd
            Case 2 ;folders Only
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then ;bypass file
                  ;check for exclude folder
                  If StringRegExp($sFile, $sExclude) Then ContinueLoop
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
            Case Else ;files and folders
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                ;check for exclude files/folder
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                $sFileList &= $sRetPath & $sFile & "|"
              WEnd
          EndSwitch
          FileClose($hSearchFile)
        Next ;$iFCount - next filter

      Next ;$iPCount - next path

    Else ;If Not $sExclude

      For $iPCount = 1 To $aPath[0] ;Path loop

        $sPath = StringRegExpReplace($aPath[$iPCount], "[\\/]+\z", "") & "\" ;ensure exact one trailing slash
        If Not FileExists($sPath) Then ContinueLoop
        ;-----
        Switch $iRetPathType
          Case 2 ;full path
            $sRetPath = $sPath
          Case 1 ;relative path
            $sRetPath = ""
                EndSwitch

        For $iFCount = 1 To $aFilter[0] ;filter loop
          ;-----
          $hSearchFile = FileFindFirstFile($sPath & $aFilter[$iFCount])
          If @error Then ContinueLoop
          ;-----
          Switch $iRetItemType
            Case 1 ;files Only
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then ContinueLoop ;bypass folder
                $sFileList &= $sRetPath & $sFile & "|"
              WEnd
            Case 2 ;folders Only
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                If @extended Then ;bypass file
                  $sFileList &= $sRetPath & $sFile & "|"
                EndIf
              WEnd
            Case Else ;files and folders
              While True
                $sFile = FileFindNextFile($hSearchFile)
                If @error Then ExitLoop
                $sFileList &= $sRetPath & $sFile & "|"
              WEnd
          EndSwitch
          FileClose($hSearchFile)
        Next ;$iFCount - next filter

      Next ;$iPCount - next path

    EndIf ;If $sExclude

  EndIf ;If $bRecursive

  ;---------------

  ;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2 ;return a delimited string
        Return StringTrimRight($sFileList, 1)
      Case 0 ;return a 0-based array
        Return StringSplit(StringTrimRight($sFileList, 1), "|", 2)
      Case Else ;return a 1-based array
        Return StringSplit(StringTrimRight($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4, 4, "")
  EndIf

EndFunc   ;==>_FileListToArrayXT

Share this post


Link to post
Share on other sites

Some test results (all done out of cache):

Test1: path: System32-Dir, filter="*.dll", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 7.70 , 1832 items

_FileListToArrayNT7: Sec = 4.61 , 1832 items

_FileListToArrayXT: Sec = 4.62 , 1832 items

Test2: path: System32-Dir, filter="*", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 12.80 , 3114 items

_FileListToArrayNT7: Sec = 7.30 , 3114 items

_FileListToArrayXT: Sec = 7.31 , 3114 items

Test3: path: System32-Dir, filter="*nothing*.xyz", return files/folders, full path, no recursion, no exclusion, repeats 300

_FileListToArray: Sec = 0.53 , 0 items

_FileListToArrayNT7: Sec = 0.56 , 0 items

_FileListToArrayXT: Sec = 0.56 , 0 items

...

First, the more interesting statistic (in my opinion) is the reciprocal of the time, which is directly proportional with the speed. This can then be used to find a good metric of your % increase in speed.

Test1:

_FileListToArray: Sec = 7.70 , 1832 items, 0.130 1/s

_FileListToArrayXT: Sec = 4.62 , 1832 items, 0.216 1/s

* XT version is 66.2% faster

Test2:

_FileListToArray: Sec = 12.80 , 3114 items, 0.0781 1/s

_FileListToArrayXT: Sec = 7.31 , 3114 items, 0.137 1/s

* XT version is 75.1% faster

Thus you have made the function 60-80% faster. My question to you is, under what circumstances is a 80% increase in speed acceptable when a ~300% increase has been made to the source code? There are definitely situations where this would be warranted, but what is it about this situation that makes it worthwhile. I will admit that a significant amount of that bloat is probably from the added features, but still even an increase in source code commensurate with the increase in speed seems like a very poor choice in most cases.

I would say that the increase in function speed should be orders of magnitude larger than the source code bloat in order to make it worthwhile in most cases.

Edited by Wus

Share this post


Link to post
Share on other sites

I would say that the increase in function speed should be orders of magnitude larger than the source code bloat in order to make it worthwhile in most cases.

I would be curious what this means in KB for the compiled code... TB HDDs are standard and every simple games consumes 100's of MB to GB... do you really complain about 20kb?

Edit:

Compiled _FileListToArrayXT() is 3KB smaller than compiled _FileListToArray()... unstripped, talking of source-code bloating...

Edited by KaFu

Share this post


Link to post
Share on other sites

My use of the phrase 'source code bloat' references the number of lines in the source code, *not* the compiled size; I apologize if this is a non-standard use of the phrase. You are absolutely correct in that a few extra kilobytes of code isn't an issue. What *is* a problem is trying to debug, update, and manage an extra few hundred lines of code.

At this juncture I will stop posting in this thread. The important points have been made and at this point I feel that I am beating a dead horse. We clearly and irrevocably disagree. End of story.

Cheers.

Edited by Wus

Share this post


Link to post
Share on other sites

The extra options don't necessarily require much more code, Post 252 competes pretty well, and ended up at 63 lines. Twenty-four of those are one-time edits/formats of the input parameters.

Share this post


Link to post
Share on other sites

I haven't even begun to look into all the available options with this, but I wonder how this MS Word filesearch stacks up, performance-wise, versus the other recursive versions?

#include<array.au3>

$x = FLTA_MSWord("C:\", "*.txt")
_ArrayDisplay($x)

;-------------------------------------------------------------------------------
Func FLTA_MSWord($sPath, $sFilter)
    $objWord = ObjCreate("Word.Application")
    With $objWord.FileSearch
        .LookIn = $sPath
        .FileName = $sFilter
        .SearchSubfolders = True
        .Execute
    EndWith
    $str = ""
    For $objFile in $objWord.FileSearch.FoundFiles
        $str &= $objFile & "|"
    Next
    $objWord.Quit
    Return StringSplit($str, "|")
EndFunc

;   .NewSearch
;   .SearchSubFolders = True
;   .TextOrProperty = "San*"
;   .MatchTextExactly = True
;   .MatchAllWordForms = True
;   .FileType = msoFileTypeAllFiles

;   $objWord.FileSearch.FoundFiles.Count

Edit: And yes, I know this is not a valid FLTA replacement, as it relies upon the user having MS Office (or Word) installed.

Edited by Spiff59

Share this post


Link to post
Share on other sites

@BaKaMu

I use your function _FileListToArrayXT from a long time, and I never have any problem since today. :)

One of member of the French forum notice me on a strange behavior.

Create a directory structure with some directory and subdirectory.

Create directory and subdirectory with specific name.

Use _FileListToArrayXT to find these specifics directory and try to delete them.

The first time you lunch the script, directory can't be deleted, but you lunch the script a second time, it work !!! ;)

#include <Array.au3>

$SrcDir = "D:\test"
$KillDir = "TOTO"

$ar_Array = _FileListToArrayXT($SrcDir, $KillDir, 2, 2, 1)
If Not IsArray($ar_Array) Then Exit

For $i = $ar_Array[0] To 1 Step -1
    DirRemove($ar_Array[$i], 1)
Next

$ar_Array = _FileListToArrayXT($SrcDir, $KillDir, 2, 2, 1)
If Not IsArray($ar_Array) Then
    Exit
Else
    MsgBox(16, 'Error', 'Some folder can''t be deleted' & @CRLF & _
    'Push enter to see the folder list')
    _ArrayDisplay($ar_Array, 'Liste des dossiers')
EndIf

I try to add a FileClose after each While on each case, but doesn't work.

So perhaps have you an idea.

Just one additional thing: The _FileListToArrayNT7 work find on this case.

Share this post


Link to post
Share on other sites

@BaKaMu

...

@Tiem

unfortunately my tests have confirmed this behavior, but at the moment i have no idea why.

The result of _FileListToArrayXT is correct, but it seems, that the dir is locked.

_FileListToArrayXT only uses three file-operations: FileExists, FileFindFirstFile and FileFindNextFile.

All of these are normal internal autoit functions and besides there is nothing special in _FileListToArrayXT.

Maybe one of this functions is problematical (does not release file-handle or something else).

Maybe DirRemove is problematical.

Because these are internal AutoIt functions, i can not go deeper in source.

But it is good to know, that there is a problem.

Thank you for reporting this behavior, i will observe this further on.

bakamu

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...