Improvement of included _FileListToArray function.

Tlem · June 30, 2009

Like I said in #77, I think that we can keep the improved function _FileListToArray and add _FileListToArrayEx4 like an extra function.

You can name it : _FileListToArrayExt (Like extended ).

But like you said in #84 :

I think is the parameter are compatible with optional ones doing extra work we can stay with only one

As developer, the choice belongs to you.

BaKaMu · June 30, 2009

are we at the stage _FileListToArrayEx4 will replace _FileListToArray with extra functionality?

Slowly, slowly, not so fast :-)

Sorry, that's not the end of the road!

I have made some more tests with different variants of _FileListToArrayEx (some has not been shown on forum)

(all tests are done out of the cache, Test 1-4: 50 iterations, Test5: 100000 iterations)

_FileListToArrayEx3 = old _FileListToArrayEx3, little speed optimized

_FileListToArrayEx4 = old _FileListToArrayEx4, little speed optimized

_FileListToArrayEx5 = new _FileListToArrayEx4

_FileListToArrayEx6 = new _FileListToArrayEx4, little speed optimized

_FileListToArrayEx7 = old _FileListToArrayEx4, little speed optimized, functionality now like new _FileListToArrayEx4

_FileListToArrayOpt = old _FileListToArrayEx3, max speed optimized, reduced parameter checking, not for general use)

Test1: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, no exclusion

_FileListToArrayEx3: Sec = 27.68 , 12144 items

_FileListToArrayEx4: Sec = 28.48 , 12144 items

_FileListToArrayEx5: Sec = 30.24 , 12144 items

_FileListToArrayEx6: Sec = 30.13 , 12144 items

_FileListToArrayEx7: Sec = 28.71 , 12144 items

_FileListToArrayOpt: Sec = 25.49 , 12144 items

Test2: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, with one exclusion

_FileListToArrayEx3: Sec = 43.86 , 10632 items

_FileListToArrayEx4: Sec = 44.84 , 10632 items

_FileListToArrayEx5: Sec = 46.22 , 10632 items

_FileListToArrayEx6: Sec = 45.82 , 10632 items

_FileListToArrayEx7: Sec = 45.20 , 10632 items

_FileListToArrayOpt: Sec = 42.16 , 10632 items

Test3: one path with 10584 files, 1560 folders, filter="*", return files only, full path, recursion, no exclusion

_FileListToArrayEx3: Sec = 27.76 , 10584 items

_FileListToArrayEx4: Sec = 28.75 , 10584 items

_FileListToArrayEx5: Sec = 30.12 , 10584 items

_FileListToArrayEx6: Sec = 29.65 , 10584 items

_FileListToArrayEx7: Sec = 28.79 , 10584 items

_FileListToArrayOpt: Sec = 25.66 , 10584 items

Test4: one path with 10584 files, 1560 folders, filter="*", return folders only, full path, recursion, no exclusion

_FileListToArrayEx3: Sec = 25.77 , 1560 items

_FileListToArrayEx4: Sec = 26.43 , 1560 items

_FileListToArrayEx5: Sec = 26.69 , 1560 items

_FileListToArrayEx6: Sec = 26.40 , 1560 items

_FileListToArrayEx7: Sec = 26.41 , 1560 items

_FileListToArrayOpt: Sec = 23.47 , 1560 items

Test5: one root path with 14 files, 10 folders, filter="*", return files/folders, full path, no recursion, no exclusion

_FileListToArrayEx3: Sec = 43.04 , 24 items

_FileListToArrayEx4: Sec = 45.07 , 24 items

_FileListToArrayEx5: Sec = 47.52 , 24 items

_FileListToArrayEx6: Sec = 47.40 , 24 items

_FileListToArrayEx7: Sec = 45.10 , 24 items

_FileListToArrayOpt: Sec = 40.50 , 24 items

Result:

If i compare overall performance vs functionality and stability then _FileListToArrayEx7 will be the best.

So i have decided (just for me) to develop _FileListToArrayEx7 further on.

(it's a good combination of all suggestions and gives me an empty array (1- or 0-based) and the error if nothing has been found)

The only drawback is the last parameter $sWorkPath, that gives me a little pain.

A parameter in a function that should not be used is not the best programming technic.

But at the moment i don't know a better solution.

(but maybe this question is only academically, in practice it works)

The -1 or Default discussion:

In my understanding the "Default" value for function parameter gives not automatically the default value.

Help file: "For UDF it is the scripter responsabitly to check is the parameter has been set to Default keyword and to do the behavior he wants."

So the checks at the beginning of function has to be there.

(except for a max optimized function like _FileListToArrayOpt, but that's my privacy) :-) ;-)

@spiff59

I like the new header, a native English speaking person could explain this much better!

_FileListToArrayEx7

; #FUNCTION# ===========================================================================================
; Name:             _FileListToArrayEx7
; Description:      full compatible _FileListToArray replacement  (with greater performance and additional features)
;                   additional: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search
; Syntax:           _FileListToArrayEx7([$sPath = @ScriptDir, [$sFilter = "*", [$iSearchType, [$bRecursiv = False, [$sExclude = "", [$iRetFormat = 1]]]]]])
; Parameter(s):     $sPath = optional: Search path(s), semicolon delimited (default: @ScriptDir)
;                             (Example: "C:\Tmp;D:\Temp")
;                   $sFilter = optional: Search filter(s), semicolon delimited . Wildcards allowed. (default: "*")
;                              (Example: "*.exe;*.txt")
;                   $iSearchType = Include in search: 0 = Files and Folder, 1 = Files Only, 2 = Folders Only
;                   $iPathType = Returned element format: 0 = file/folder name only, 1 = relative path, 2 = full path
;                   $bRecursiv = optional: True: recursive search including all subdirectories
;                                          False (default): search only in specified folder
;                   $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                                (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                   $iRetFormat =  optional: return format
;                                  0 = one-dimensional array, 0-based
;                                  1 = one-dimensional array, 1-based (default)
;                                  2 = String ( "|" delimited)
;                   $sWorkPath = *** internal use only (do not use) ***
; Requirement(s):   none
; Return Value(s):  on success: 1-based or 0-based array or string (dependent on $iRetFormat)
;                   If no data is found, @error is set (to 4, for backwards compatibility)
; Author(s):        Half the AutoIt Community
; ====================================================================================================
Func _FileListToArrayEx7($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursiv = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTExclude, $sTWorkPath

  If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
  If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
  If $iSearchType = -1 Or $iSearchType = Default Then $iSearchType = 0
  If $iPathType = -1 Or $iPathType = Default Then $iPathType = 0
  If $bRecursiv = Default Then $bRecursiv = False
  If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
  If $iRetFormat = -1 Or $iRetFormat = Default Then $iRetFormat = 1

  Local $aPath = StringSplit($sPath, ';') ;paths array
  Local $aFilter = StringSplit($sFilter, ';') ;filters array

  If $sExclude Then

    ;prepare $sTExclude
    ;strip leading and trailing spaces and spaces between semi colon delimiter
    $sTExclude = StringStripWS(StringRegExpReplace($sExclude, "\s*;\s*", ";"), 3)
    ;convert $sExclude to fit StringRegExp (not perfect but useable)
    $sTExclude = StringRegExpReplace($sTExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1') ;thanks KaFu and Ascend4nt
    $sTExclude = StringReplace($sTExclude, "?", ".")
    $sTExclude = StringReplace($sTExclude, "*", ".*?")
    $sTExclude = "(?i)\A" & StringReplace($sTExclude, ";", "|") & "\z"

    For $iPCount = 1 To $aPath[0] ;Path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      If $iPathType = 2 Then $sDelim &= $sPathItem ;return full-path

      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              ;check for exclude files
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              ;check for exclude folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              ;check for exclude files/folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;check for exclude folder
            If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayEx7($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  Else ;If Not $sExclude

    For $iPCount = 1 To $aPath[0] ;path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      ;return full-path
      If $iPathType = 2 Then $sDelim &= $sPathItem

      ;perform the search
      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayEx7($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  EndIf ;If $sExclude

  ;---------------

  ;set according return value
  Switch $iRetFormat
    Case 2 ;return a delimited string
      If $sFileList Then
        Return $sFileList
      Else
        Return SetError(4, 4, "")
      EndIf
    Case 1 ;return a 1-based array
      If $sFileList Then
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", $iRetFormat)
      Else
        Local $aRet[1] = [0]
        Return SetError(4, 4, $aRet)
      EndIf
    Case 0 ;return a 0-based array
      If $sFileList Then
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", $iRetFormat)
      Else
        Local $aRet[0]
        Return SetError(4, 4, $aRet)
      EndIf
    Case Else ;return empty string if $iRetFormat is wrong
      Return SetError(3, 3, "")
  EndSwitch

EndFunc   ;==>_FileListToArrayEx7

Edited June 30, 2009 by BaKaMu

KaFu · June 30, 2009

How about adding the required min AU version to the functions header... just fought 10 minutes trying to figure out why _FileListToArrayEx7() does not search recursivly... until I realized that >3.3.1.0 is needed for @extended ...

Edit: What I wanted to figure out is, if an (optional?) increase of the priority with ProcessSetPriority() might result in a significant speed improvment, though most time will be used for IO...

Edited June 30, 2009 by KaFu

GEOSoft · June 30, 2009

Like I said in #77, I think that we can keep the improved function _FileListToArray and add _FileListToArrayEx4 like an extra function.
You can name it : _FileListToArrayExt (Like extended ).
But like you said in #84 :
As developer, the choice belongs to you.

Not to sure about using the _FileListToArrayEx() name here. There is a version already posted (by SmOke_N) using that name and many people are already using that in a separate UDF. I'm using a very fast, modified (unposted) version of it but still using the same name and if I get into a position where I have to include both the standard array.au3 and my custom file there will be a clash which is going to mean I have to start stripping the function from the standard library everytime there is an update.

Spiff59 · June 30, 2009

I'll jump on board with the Ex7 version, it's close enough to Ex4 for me.

It does need the StringSplit statements fixed to be a hardcoded 0 and 2 instead of $iRetFormat.

And if we've gotten rid of the other useless, non-occuring @error conditions, why not drop @error = 3 as well. Make "Case 1" into "Case Else" and it will default to 1-based.

I understood the helpfile differently, I thought it meant: If you place Default in the actual Func definition like : "Func Test($parm1=Default)" then the scripter was responsible to handle it. But the helpfile for Default isn't entirely clear to me. So, I still don't know if all the "If Default" tests are needed?

And then there's the confusion about filters and excludes and how they should behave. In Danny35d's (worthy) version, the (include) filter is only applied to filenames, and not folders. In Ex7, (include) filters affect both file and folder selection.

Exclude handling brings up similar questions. If you call Ex7 and include "*.exe" and exclude "Windows" you're pitching out both the Windows folder, and any files that begin with Windows, even though your intention may be to return all .exe's outside of the Windows folder. I hate to say it, but it's almost as if separate include-file and include-folder, and separate exclude-file and exclude-folder options are necessary for a user to really get back from the function what he is asking for. Maybe "file-filters" and "folder-filters" and if a "-" is placed in front it means exclude. I dunno...

BaKaMu · June 30, 2009

I'll jump on board with the Ex7 version, it's close enough to Ex4 for me.
It does need the StringSplit statements fixed to be a hardcoded 0 and 2 instead of $iRetFormat.

Fixed (oh yea, one bug is always on board)

And if we've gotten rid of the other useless, non-occuring @error conditions, why not drop @error = 3 as well. Make "Case 1" into "Case Else" and it will default to 1-based.

Done, good idea

I understood the helpfile differently, I thought it meant: If you place Default in the actual Func definition like : "Func Test($parm1=Default)" then the scripter was responsible to handle it. But the helpfile for Default isn't entirely clear to me. So, I still don't know if all the "If Default" tests are needed?

I have to test it, if the "If Default" tests are not needed i will be happy !!

And then there's the confusion about filters and excludes and how they should behave. In Danny35d's (worthy) version, the (include) filter is only applied to filenames, and not folders. In Ex7, (include) filters affect both file and folder selection.
Exclude handling brings up similar questions. If you call Ex7 and include "*.exe" and exclude "Windows" you're pitching out both the Windows folder, and any files that begin with Windows, even though your intention may be to return all .exe's outside of the Windows folder. I hate to say it, but it's almost as if separate include-file and include-folder, and separate exclude-file and exclude-folder options are necessary for a user to really get back from the function what he is asking for. Maybe "file-filters" and "folder-filters" and if a "-" is placed in front it means exclude. I dunno...

it is not difficult to program this, but isn't it more confusing?

(Maybe later on a version for special programmers) :-)

Corrected _FileListToArrayNT (formerly known as _FileListToArrayEx7) , thanks to GEOSoft

; #FUNCTION# ===========================================================================================
; Name:             _FileListToArrayNT
; Description:      full compatible _FileListToArray replacement  (with greater performance and additional features)
;                   additional: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search
; Syntax:           _FileListToArrayNT([$sPath = @ScriptDir, [$sFilter = "*", [$iSearchType, [$bRecursiv = False, [$sExclude = "", [$iRetFormat = 1]]]]]])
; Parameter(s):     $sPath = optional: Search path(s), semicolon delimited (default: @ScriptDir)
;                             (Example: "C:\Tmp;D:\Temp")
;                   $sFilter = optional: Search filter(s), semicolon delimited . Wildcards allowed. (default: "*")
;                              (Example: "*.exe;*.txt")
;                   $iSearchType = Include in search: 0 = Files and Folder, 1 = Files Only, 2 = Folders Only
;                   $iPathType = Returned element format: 0 = file/folder name only, 1 = relative path, 2 = full path
;                   $bRecursiv = optional: True: recursive search including all subdirectories
;                                          False (default): search only in specified folder
;                   $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                                (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                   $iRetFormat =  optional: return format
;                                  0 = one-dimensional array, 0-based
;                                  1 = one-dimensional array, 1-based (default)
;                                  2 = String ( "|" delimited)
;                   $sWorkPath = *** internal use only (do not use) ***
; Requirement(s):   none
; Return Value(s):  on success: 1-based or 0-based array or string (dependent on $iRetFormat)
;                   If no data is found, @error is set (to 4, for backwards compatibility)
; Author(s):        Half the AutoIt Community
; ====================================================================================================
Func _FileListToArrayNT($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursiv = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTExclude, $sTWorkPath

  If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
  If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
  If $iSearchType = -1 Or $iSearchType = Default Then $iSearchType = 0
  If $iPathType = -1 Or $iPathType = Default Then $iPathType = 0
  If $bRecursiv = Default Then $bRecursiv = False
  If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
  If $iRetFormat = -1 Or $iRetFormat = Default Then $iRetFormat = 1

  Local $aPath = StringSplit($sPath, ';') ;paths array
  Local $aFilter = StringSplit($sFilter, ';') ;filters array

  If $sExclude Then

    ;prepare $sTExclude
    ;strip leading and trailing spaces and spaces between semi colon delimiter
    $sTExclude = StringStripWS(StringRegExpReplace($sExclude, "\s*;\s*", ";"), 3)
    ;convert $sExclude to fit StringRegExp (not perfect but useable)
    $sTExclude = StringRegExpReplace($sTExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1') ;thanks KaFu and Ascend4nt
    $sTExclude = StringReplace($sTExclude, "?", ".")
    $sTExclude = StringReplace($sTExclude, "*", ".*?")
    $sTExclude = "(?i)\A" & StringReplace($sTExclude, ";", "|") & "\z"

    For $iPCount = 1 To $aPath[0] ;Path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      If $iPathType = 2 Then $sDelim &= $sPathItem ;return full-path

      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              ;check for exclude files
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              ;check for exclude folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              ;check for exclude files/folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;check for exclude folder
            If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayNT($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  Else ;If Not $sExclude

    For $iPCount = 1 To $aPath[0] ;path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      ;return full-path
      If $iPathType = 2 Then $sDelim &= $sPathItem

      ;perform the search
      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayNT($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  EndIf ;If $sExclude

  ;---------------

  ;set according return value
  Switch $iRetFormat
    Case 2 ;return a delimited string
      If $sFileList Then
        Return $sFileList
      Else
        Return SetError(4, 4, "")
      EndIf
    Case 0 ;return a 0-based array
      If $sFileList Then
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Else
        Local $aRet[0]
        Return SetError(4, 4, $aRet)
      EndIf
    Case Else ;return a 1-based array
      If $sFileList Then
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
      Else
        Local $aRet[1] = [0]
        Return SetError(4, 4, $aRet)
      EndIf
  EndSwitch

EndFunc   ;==>_FileListToArrayNT

BaKaMu · June 30, 2009

I have to test it, if the "If Default" tests are not needed i will be happy !!

Tested, but unfortunately the "Default" key as value for the function parameter only returns "Default".

So we have to stay with the parameter checking at the beginning. :-(

(Don't worry about things you can not change!)

Spiff59 · June 30, 2009

That's pretty screwy. Default doesn't do anything.

I assume it behaves differently with built-in fucntions.

$x = _FileListToArrayX(@SystemDir, Default, 2)
$x = _FileListToArrayX(@SystemDir, -1, 2)
$x = _FileListToArrayX(@SystemDir, "", 2)
$x = _FileListToArrayX(@SystemDir)
Exit

Func _FileListToArrayX($sPath, $sFilter = "*", $iSearchType = 0)
    MsgBox(1,"", "Parms received: " & @NumParams & @CRLF & @CRLF & $sPath & @CRLF & $sFilter & @CRLF & $iSearchType)
EndFunc

So what's the UDF standard? We have to handle Default? and -1?, a null/empty string "" ?

How many Or's are you supposed to park in those compound parameter-default statements?

SmOke_N · June 30, 2009

That's pretty screwy. Default doesn't do anything.
I assume it behaves differently with built-in fucntions.
$x = _FileListToArrayX(@SystemDir, Default, 2)
$x = _FileListToArrayX(@SystemDir, -1, 2)
$x = _FileListToArrayX(@SystemDir, "", 2)
$x = _FileListToArrayX(@SystemDir)
Exit

Func _FileListToArrayX($sPath, $sFilter = "*", $iSearchType = 0)
    MsgBox(1,"", "Parms received: " & @NumParams & @CRLF & @CRLF & $sPath & @CRLF & $sFilter & @CRLF & $iSearchType)
EndFunc
So what's the UDF standard? We have to handle Default? and -1?, a null/empty string "" ?
How many Or's are you supposed to park in those compound parameter-default statements?

You make the exception for it because users will undoubtedly use it as a param pass. And default does not equal -1. That's why you use it.

Edit:

You can even use something like:

Don't know if that would save time off of "Or" statements though.

Edited June 30, 2009 by SmOke_N

Tlem · June 30, 2009

@GEOSoft

Not to sure about using the _FileListToArrayEx() name here.

I do not propose that, I proposed _FileListToArrayExt()

There is a version already posted (by SmOke_N) using that name

Yes, and I know that. And this version is based on DIR cmd. But this choice is only reliable for English people because other countries can use special characters that DIR doesn't return correctly. Edited June 30, 2009 by Tlem

Spiff59 · June 30, 2009

Hadn't jpm indicated, as long as it was not a script-breaker, and the function had had the crap tested out of it, that he'd rather see a one-function solution. Meaning this would replace the existing slow and parm-limited _FileListToArray() ?

I do think we potentially have a script-breaker though...

If an existing script were coded something like:

$MyFileArray = _FileListToArray(@Scriptdir, "*")
If $MyFileArray = "" Then Msgbox(1,"","No files found!"

Then the third parameter of our SetError(4,4,0) statement would return a zero from the function, breaking the script.

How about this at the end?

If Not $sFileList Then Return SetError(4) 
  Switch $iRetFormat
    Case 2;return a delimited string
        Return $sFileList
    Case 0;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
    Case Else;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
  EndSwitch

Edit: I see the production version actually returns (4, 4, "") so maybe that's what is needed. Although I don't see the reason for the @extended flag.

Edited June 30, 2009 by Spiff59

KaFu · June 30, 2009

That's pretty screwy. Default doesn't do anything.
I assume it behaves differently with built-in functions.

You're right, expected a different behavior, sorry for confusing

jpm · July 1, 2009

Hadn't jpm indicated, as long as it was not a script-breaker, and the function had had the crap tested out of it, that he'd rather see a one-function solution. Meaning this would replace the existing slow and parm-limited _FileListToArray() ?

I do think we potentially have a script-breaker though...
If an existing script were coded something like:
$MyFileArray = _FileListToArray(@Scriptdir, "*")
If $MyFileArray = "" Then Msgbox(1,"","No files found!"
Then the third parameter of our SetError(4,4,0) statement would return a zero from the function, breaking the script.
How about this at the end?
If Not $sFileList Then Return SetError(4) 
  Switch $iRetFormat
    Case 2;return a delimited string
        Return $sFileList
    Case 0;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
    Case Else;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
  EndSwitch
Edit: I see the production version actually returns (4, 4, "") so maybe that's what is needed. Although I don't see the reason for the @extended flag.

if @error is set if both case I think that's not a script breaking. only @error=0 returning different result is a script breaking case.

I have no idea why @error was set to @extended in some UDF bt for me it is not needed. It can be a good way to differentiate the "same" type of error with additional info in @extended.

BaKaMu · July 1, 2009

If Not $sFileList Then Return SetError(4) 
  Switch $iRetFormat
    Case 2;return a delimited string
        Return $sFileList
    Case 0;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
    Case Else;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
  EndSwitch

Edit: I see the production version actually returns (4, 4, "") so maybe that's what is needed. Although I don't see the reason for the @extended flag.

Don't agree.

My intention is, if i set $iRetFormat to return a 1-based array, it should return a 1-based array, even if the search-result is empty.

If nothing found the return is a empty 1-based array (and setting error) with Array[0] = 0.

This makes programming easier (you don't have to check whether return is empty string or array, so this trap is removed).

Same ventilations for 0-based array.

jpm said it should being logical, so we shouldn't take over the disadvantages of _FileListToArray (from my point of view).

Danny35d · July 1, 2009

This is another version 50 lines of code.

To be able to make the script smaller I change the default value of $iPathType and $iRetFormat.

; #FUNCTION# ================================================================================
; Name...........: _FileListToArrayEx
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = "*"[, $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = "", $iRetFormat = 1]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                  $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;                  $iFlag   - Optional: specifies whether to return files folders or both
;                    0 =(Default) Return both files and folders
;                    1 = Return files only
;                    2 = Return Folders only
;                  $iPathType
;                    0 = relative path
;                    1 = (Default) full path
;                  $iRecursive - Search files in specified directory and all subdirectories.
;                    0 = (Default) Search in specified directory.
;                    1 = Search in specified and all subdirectories.
;                  $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                    (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                  $iRetFormat =  optional: return format
;                    0 = (Default) one-dimensional array, 1-based
;                    1 = String ( "|" delimited)
;                    2 = one-dimensional array, 0-based
;                  $iRunFirstTime = *** internal use only (do not use) ***
; Return values .: @Error -
;                    1 = Path not found or invalid
;                    2 = Invalid $sFilter
;                    3 = Invalid $iFlag
;                    4 = No File(s) Found
;                    5 = Invalid $iPathType
;                    6 = Invalid $iRecursive
;                    7 = Invalid $iRetFormat
;
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                    $array[0] = Number of Files\Folders returned
;                    $array[1] = 1st File\Folder
;                    $array[2] = 2nd File\Folder
;                    $array[3] = 3rd File\Folder
;                    $array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ===========================================================================================
;
;============================================================================================
Func _FileListToArrayEx($sPath, $sFilter = '*', $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = "", $iRetFormat = 0, $iRunFirstTime = 1)
    Local $aFileList = '', $aFolderList = '', $Tmp = '', $sMask

    $sPath = StringRegExpReplace($sPath, '\\+$', ''); Removed trailing backslash
    If Not FileExists($sPath) Then Return SetError(1, 1, 'Path Not Found Or Invalid.')
    If StringRegExp($sFilter, '[\\/ :> <\|]') Then Return SetError(2, 2, 'Invalid Mask('& $sFilter & ').')
    If Not StringRegExp($iFlag, '[012]') Then Return SetError(3, 3, 'Invalid Flag, Chooce (0,1 or 2).')
    If Not StringRegExp($iPathType, '[01]') Then Return SetError(5, 5, 'Invalid Path Type, Choose (0 or 1).')
    If Not StringRegExp($iRecursive, '[01]') Then Return SetError(6, 6, 'Invalid Recursive, Choose (0 or 1).')
    If Not StringRegExp($iRetFormat, '[012]') Then Return SetError(7, 7, 'Invalid RetFormat, Choose (0, 1 or 2).')
    
    $sMask = '(?i)^' & StringReplace(StringReplace(StringReplace($sFilter, '.', '\.'), '*', '.*'), '?', '.')
    If $sExclude <> '' And $iRunFirstTime Then ;convert $sExclude to fit StringRegExp (not perfect but useable)
        $sExclude = StringStripWS(StringRegExpReplace($sExclude, "\s*;\s*", ";"), 3) ;Strip leading and trailing spaces and spaces between semi colon delimiter
        $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1') ;thanks KaFu and Ascend4nt
        $sExclude = "(?i)\A" & StringReplace(StringReplace(StringReplace($sExclude, "?", "."), "*", ".*?"), ";", "|") & "\z"
    EndIf

    $iFirstFile = FileFindFirstFile($sPath & '\*')
    If @error Then Return
    While True
        $iNextFile = FileFindNextFile($iFirstFile)
        If @error Then ExitLoop
        $extended = @extended
        $sFullPath = $sPath & '\' & $iNextFile
        If $extended = 1 Then
            If $iFlag <> 1 And ($sFilter = '*' Or $sFilter = '*.*') Then $aFileList &= $sFullPath & '|'
            If $iRecursive Then
                If $sExclude <> '' And StringRegExp(StringRegExpReplace($sFullPath, "(.*?[\\/]+)*(.*?\z)", "\2"), $sExclude) Then ContinueLoop
                $Tmp = _FileListToArrayEx($sFullPath, $sFilter, $iFlag, $iPathType, $iRecursive, $sExclude, 1, 0)
                If $Tmp <> Chr(38) And $Tmp <> ChrW(38) Then $aFileList &= $Tmp
            EndIf
        Else
            If Not StringRegExp($iNextFile, $sMask) Then ContinueLoop
            If Not StringRegExp(StringRegExpReplace($sFilter, '^.*\.', ''), '[?*]') Then
                If StringRegExpReplace($sFilter, '^.*\.', '') <> StringRegExpReplace($iNextFile, '^.*\.', '') Then ContinueLoop
            EndIf
            If $sExclude <> '' And StringRegExp(StringRegExpReplace($sFullPath, "(.*?[\\/]+)*(.*?\z)", "\2"), $sExclude) Then ContinueLoop
            If $iFlag <> 2 Then $aFileList &= $sFullPath & '|'
        EndIf
    WEnd
    FileClose($iFirstFile)

    If $iRunFirstTime Then
        $aFileList = StringTrimRight($aFileList, 1)
        If StringLen($aFileList) = 0 Then Return SetError(4, 4, 'No File(s) Found.')
        If $iPathType = 0 Then $aFileList = StringReplace($aFileList, $sPath & '\', '')
        If $iRetFormat <> 1 Then $aFileList = StringSplit($aFileList, '|', $iRetFormat)
    EndIf
    Return ($aFileList)
EndFunc ;==>_FileListToArrayEx

Note: This line $sFullPath = $sPath & '' & $iNextFile should be $sFullPath = $sPath & '\' & $iNextFile for some reason when copy and paste into Scite it remove the back slash.

Edited July 1, 2009 by Danny35d

Spiff59 · July 1, 2009

Don't agree.
My intention is, if i set $iRetFormat to return a 1-based array, it should return a 1-based array, even if the search-result is empty.
If nothing found the return is a empty 1-based array (and setting error) with Array[0] = 0.
This makes programming easier (you don't have to check whether return is empty string or array, so this trap is removed).
Same ventilations for 0-based array.
jpm said it should being logical, so we shouldn't take over the disadvantages of _FileListToArray (from my point of view).

No, it won't be at all the same with 0-based arrays.

Local $aRet[0] is an improper statement.

The following causes a script execution error:

Dim $aRet[0]
If Isarray($aRet) Then Beep(800,200)

The test "If $aRet[0] = 0 Then" is not any easier than "If @error Then".

Logical would be being consistant, rather than having to explain in the helpfile that calls requesting an array be returned will in some cases return an array with one-element, and in other cases return nothing. Encouraging that "If @error Then" be the universal test for a not-find condition seems cleaner than encouraging "If @error Then", "If $aRet = '' Then" and "If $aRet[0] = 0".

(this route also removes a bunch of nested if's and array declarations and 'pretties-up' the exit paragraph considerably)

BaKaMu · July 1, 2009

Local $aRet[0] is an improper statement.
The following causes a script execution error:
Dim $aRet[0]
If Isarray($aRet) Then Beep(800,200)

You are right, i did not realised that until now.

Most of the time i'm working with 1-based arrays where i can write:

$aRet = _FileListToArrayx(...) : For $FCount = 1 to $aRet[0] do something Next

So no checking for nothing found is necessary.

The subscript error for Dim $aRet[0] is a little bit inconsequentially (probably only for me).

So in terms of this your suggestion is the best we can make.

(i'm now a little bit melancholy cause losing the $Array[0] = 0 term) ;-)

But as always i have made a little speed improvement ;-)

I think we can stop the development by now and make a extensively testing.

Return part:

;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2 ;return a delimited string
        Return $sFileList
      Case 0 ;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Case Else ;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4, 4, "")
  EndIf

complete _FileListToArrayNT (or whatever you want to name it)

; #FUNCTION# ===========================================================================================
; Name:             _FileListToArrayNT
; Description:      full compatible _FileListToArray replacement  (with greater performance and additional features)
;                   additional: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search
; Syntax:           _FileListToArrayNT([$sPath = @ScriptDir, [$sFilter = "*", [$iSearchType, [$bRecursiv = False, [$sExclude = "", [$iRetFormat = 1]]]]]])
; Parameter(s):     $sPath = optional: Search path(s), semicolon delimited (default: @ScriptDir)
;                             (Example: "C:\Tmp;D:\Temp")
;                   $sFilter = optional: Search filter(s), semicolon delimited . Wildcards allowed. (default: "*")
;                              (Example: "*.exe;*.txt")
;                   $iSearchType = Include in search: 0 = Files and Folder, 1 = Files Only, 2 = Folders Only
;                   $iPathType = Returned element format: 0 = file/folder name only, 1 = relative path, 2 = full path
;                   $bRecursiv = optional: True: recursive search including all subdirectories
;                                          False (default): search only in specified folder
;                   $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                                (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                   $iRetFormat =  optional: return format
;                                  0 = one-dimensional array, 0-based
;                                  1 = one-dimensional array, 1-based (default)
;                                  2 = String ( "|" delimited)
;                   $sWorkPath = *** internal use only (do not use) ***
; Requirement(s):   none
; Return Value(s):  on success: 1-based or 0-based array or string (dependent on $iRetFormat)
;                   If no data is found, @error is set (to 4, for backwards compatibility)
; Author(s):        Half the AutoIt Community
; ====================================================================================================
Func _FileListToArrayNT($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursiv = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTExclude, $sTWorkPath

  If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
  If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
  If $iSearchType = -1 Or $iSearchType = Default Then $iSearchType = 0
  If $iPathType = -1 Or $iPathType = Default Then $iPathType = 0
  If $bRecursiv = Default Then $bRecursiv = False
  If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
  If $iRetFormat = -1 Or $iRetFormat = Default Then $iRetFormat = 1

  Local $aPath = StringSplit($sPath, ';') ;paths array
  Local $aFilter = StringSplit($sFilter, ';') ;filters array

  If $sExclude Then

    ;prepare $sTExclude
    ;strip leading and trailing spaces and spaces between semi colon delimiter
    $sTExclude = StringStripWS(StringRegExpReplace($sExclude, "\s*;\s*", ";"), 3)
    ;convert $sExclude to fit StringRegExp (not perfect but useable)
    $sTExclude = StringRegExpReplace($sTExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1') ;thanks KaFu and Ascend4nt
    $sTExclude = StringReplace($sTExclude, "?", ".")
    $sTExclude = StringReplace($sTExclude, "*", ".*?")
    $sTExclude = "(?i)\A" & StringReplace($sTExclude, ";", "|") & "\z"

    For $iPCount = 1 To $aPath[0] ;Path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      If $iPathType = 2 Then $sDelim &= $sPathItem ;return full-path

      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              ;check for exclude files
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              ;check for exclude folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              ;check for exclude files/folder
              If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;check for exclude folder
            If StringRegExp(StringRegExpReplace($sFile, "(.*?[\\/]+)*(.*?\z)", "\2"), $sTExclude) Then ContinueLoop
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayNT($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  Else ;If Not $sExclude

    For $iPCount = 1 To $aPath[0] ;path loop
      Local $sPathItem = StringStripWS($aPath[$iPCount], 3) ;strip leading and trailing spaces
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($sPathItem, 1) <> "\" Then $sPathItem &= "\" ;ensure trailing slash

      ;return full-path
      If $iPathType = 2 Then $sDelim &= $sPathItem

      ;perform the search
      For $iFCount = 1 To $aFilter[0] ;Filter loop
        Local $FilterItem = StringStripWS($aFilter[$iFCount], 3) ;strip leading and trailing spaces
        If StringRegExp($FilterItem, "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($sPathItem & $FilterItem)
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended = 0 Then ContinueLoop ;bypass file
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursiv Then
        $hSearch = FileFindFirstFile($sPathItem & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended = 0 Then ContinueLoop ;bypass file (for Autoit versions > 3.3.0.0)
            ;call recursive search
            If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
            $sFileList &= _FileListToArrayNT($sPathItem & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursiv, $sExclude, 2, $sTWorkPath)
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  EndIf ;If $sExclude

  ;---------------

  ;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2 ;return a delimited string
        Return $sFileList
      Case 0 ;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Case Else ;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4, 4, "")
  EndIf

EndFunc   ;==>_FileListToArrayNT

Spiff59 · July 1, 2009

Actually, I do see your point, the most common usage for the routine will likely be returning 1-based arrays, and a "For $x = 1 to $aRet[0]" loop is a clean way to skip processing if nothing comes back. Hell, I'm wavering now lol. But... the other way is more consistant, a tad faster and simplifies documentation. But... but I'm also somewhat conflicted by giving up the "$aRet[0] = 0" option. But...

What's NT? New Technology? (are you Bill Gates???)

I'd thought of _FileListToArrayX() at one time ("X" for "experimental"), but who cares, isn't the intention for this is to go into production as _FileListToArray() replacing the existing function?

Regarding includes/excludes, we basically have a separate include for directories and files now, the multi-path parameter selects folders, and basically, the multi-filter is for files. I guess for now just make it clear in the hellfile that "exclude" removes all matching files, and all matching folders, and that all files within those folders (regardless if the filename matches or not) are removed? (I'm sure someone else can state that more clearly).

Danny35d: That is a nice functional routine. I find the filter processing interesting, and possibly superior. But... it looks like the code-du-jour (in this thread) is "speed over size". I'd proposed cutting this routine almost in half at the expense of about 3% of processing speed, and was shot down (in flames!). Size doesn't matter (today).

Edit: I'm fine with making the exception of returning a 1-element array containing zero only for $iRetFormat = 1. Either way.

Edit2: I'm beginning to lean, as far as the naming thing, towards the spiffy-sounding: _FileListToArrayTurbo3000Plus()

Edited July 1, 2009 by Spiff59

BaKaMu · July 1, 2009

What's NT? New Technology? (are you Bill Gates???)
I'd thought of _FileListToArrayX() at one time ("X" for "experimental"), but who cares, isn't the intention for this is to go into production as _FileListToArray() replacing the existing function?

Oh no, unfortunately i'm not Bill Gates. If so, i have to leave more bugs in the code (and make a monthly update). Oh that's bad. ;-)

_FileListToArrayX() or maybe _FileListToArray() or some other name is all OK for me. I prefer functionality. :-)

Regarding includes/excludes, we basically have a separate include for directories and files now, the multi-path parameter selects folders, and basically, the multi-filter is for files. I guess for now just make it clear in the hellfile that "exclude" removes all matching files, and all matching folders, and that all files within those folders (regardless if the filename matches or not) are removed? (I'm sure someone else can state that more clearly).

To your suggestion, i have a version with separated Filters/Excludes for files and folders in line. It's still under construction.

But there is an easy workaround:

First do a recursive search on folders and exclude what you want, then with the returned array make a non recursive search for files in each returned folder. Voila!

Danny35d: That is a nice functional routine. I find the filter processing interesting, and possibly superior. But... it looks like the code-du-jour (in this thread) is "speed over size". I'd proposed cutting this routine almost in half at the expense of about 3% of processing speed, and was shot down (in flames!). Size doesn't matter (today).

Oh sorry, i hope my comment was not a flame. If so, excuse me.

To Danny35d: Your code is interesting but somewhat slower (ca 25%) and does not return the right results if exclusion is used (the root excluded folder still remains).

ex: "Not StringRegExp" is slower then If ... Then (in this case). Spiff59 said it: nowadays size doesn't matter. (many decades ago I worked on PC's with 16K of memory, that was different)

Edit: I'm fine with making the exception of returning a 1-element array containing zero only for $iRetFormat = 1. Either way.

I would say: All or Nothing, being logically consistent we should follow your suggestion (if nothing found, return is an empty string)

Danny35d · July 1, 2009

To Danny35d: Your code is interesting but somewhat slower (ca 25%) and does not return the right results if exclusion is used (the root excluded folder still remains).
ex: "Not StringRegExp" is slower then If ... Then (in this case).

When copy and past for some reason it remove the back slash. This line $sFullPath = $sPath & '' & $iNextFile should be $sFullPath = $sPath & '\' & $iNextFile otherwise it will retun the wrong results. Also for $iPathType and $iRetFormat I use different values.

Danny35d: That is a nice functional routine. I find the filter processing interesting, and possibly superior. But... it looks like the code-du-jour (in this thread) is "speed over size".

Thanks!, that is what it didn't make sence to me. How come a script that is over 100 line longer can process faster? Thanks BaKaMu I over look "Not StringRegExp" is slower.

I Ran a test between _FileListToArrayNT() and my script using recursive and Flag 0, 1 and 2. Sort the array and create logs. After testing all three flags I compare the logs using WinMerge and they are identical for both functions. Tested with a network share 114 GB 138,600 files and 20,460 folders. The results show _FileListtoArrayNT() is about 12% faster.

Files/Folders 160148 Improve Flag = 0 Time: 268126

Files/Folders 160148 Danny35D Flag = 0 Time: 305225

Faster: Improve 12.15%

Files/Folders 139136 Improve Flag = 1 Time: 271598

Files/Folders 139136 Danny35D Flag = 1 Time: 304195

Faster: Improve 10.72%

Files/Folders 21012 Improve Flag = 2 Time: 263034

Files/Folders 21012 Danny35D Flag = 2 Time: 298344

Faster: Improve 11.84%

I took in consideration BaKaMu advise ("Not StringRegExp") and fix my script. The result show my script is about 46% faster.

Files/Folders 160148 Improve Flag = 0 Time: 265016

Files/Folders 160148 Danny35D Flag = 0 Time: 140058

Faster Danny35d 47.15%

Files/Folders 139136 Improve Flag = 1 Time: 259097

Files/Folders 139136 Danny35D Flag = 1 Time: 139562

Faster Danny35d 46.14%

Files/Folders 21012 Improve Flag = 2 Time: 257870

Files/Folders 21012 Danny35D Flag = 2 Time: 136742

Faster Danny35d 46.97%

This is the testing script.

#include <File.au3>
#include <Date.au3>
#include <Array.au3>

For $x = 0 To 2
    CalcTime('Start')
    $aFiles = _FileListToArrayNT('z:', '*', $x, 2, 1, '', 0)
    $error = @error
    $Time1 = Int(CalcTime('Finish'))
    If IsArray($aFiles) Then
        ConsoleWrite('Files/Folders ' & UBound($aFiles) & ' Improve Flag = ' & $x & ' Time: ' & $Time1 & @CRLF)
        ;_ArrayDisplay($aFiles, 'Improve Flag = ' & $x & ' Time: ' & $Time1)
        _ArraySort($aFiles, 0)
        FileWriteLine('H:\Testing\Improve' & $x & '.log',_ArrayToString($aFiles, @CRLF) & @CRLF)
    Else
        ConsoleWrite('Improve Error: ' & $error & ' Flag = ' & $x & ' Time: ' & $Time1 & '  ' & @CRLF)
        FileWriteLine('H:\Testing\Improve' & $x & '.log', StringReplace($aFiles, '|', @CRLF) & @CRLF)
    EndIf

    CalcTime('Start')
                               ;($sPath, $sFilter = '*', $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = "", $iRetFormat = 0)
    $aFiles = _FileListToArrayEx('z:', '*', $x, 1, 1, '', 2)
    $error = @error
    $Time2 = Int(CalcTime('Finish'))
    If IsArray($aFiles) Then
        ConsoleWrite('Files/Folders ' & UBound($aFiles) & ' Danny35D Flag = ' & $x & ' Time: ' & $Time2 & @CRLF)
        ;_ArrayDisplay($aFiles, 'Danny35D Flag = ' & $x & ' Time: ' & $Time2)
        _ArraySort($aFiles)
        FileWriteLine('H:\Testing\Danny35d' & $x & '.log', _ArrayToString($aFiles, @CRLF) & @CRLF)
    Else
        ConsoleWrite('Dan Error: ' & $error & ' Flag = ' & $x & ' Time: ' & $Time2 & @CRLF)
        FileWriteLine('H:\Testing\Danny35d' & $x & '.log', StringReplace($aFiles, '|', @CRLF) & @CRLF)
    EndIf
    If $Time1 < $Time2 Then
        $Percent = Round(100 - ($Time1/$Time2) * 100, 2)
        ConsoleWrite('Faster: Improve ' & $Percent & '%' & @CRLF & @CRLF)
    Else
        $Percent = Round(100 - ($Time2/$Time1) * 100, 2)
        ConsoleWrite('Faster Danny35d ' & $Percent & '%' & @CRLF & @CRLF)
    EndIf
Next

Func CalcTime($Action, $iMsgBox = 0)
    Local $CalcTimestamp_2[2], $Msecs, $Secs, $Mins, $Hour
    If $Action = "Start" Then
        Global $CalcTimestamp_1[2]
        $CalcTimestamp_1[0] = TimerInit()
    ElseIf $Action = "Finish" Then
        $CalcTimestamp_2[1] = TimerDiff($CalcTimestamp_1[0])
    EndIf
    Return ($CalcTimestamp_2[1])
EndFunc   ;==>CalcTime

The new script after adding BaKaMu advise.

; #FUNCTION# ================================================================================
; Name...........: _FileListToArrayEx
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = "*"[, $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = "", $iRetFormat = 1]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                  $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;                  $iFlag   - Optional: specifies whether to return files folders or both
;                    0 =(Default) Return both files and folders
;                    1 = Return files only
;                    2 = Return Folders only
;                  $iPathType
;                    0 = relative path
;                    1 = (Default) full path
;                  $iRecursive - Search files in specified directory and all subdirectories.
;                    0 = (Default) Search in specified directory.
;                    1 = Search in specified and all subdirectories.
;                  $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                    (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                  $iRetFormat =  optional: return format
;                    0 = (Default) one-dimensional array, 1-based
;                    1 = String ( "|" delimited)
;                    2 = one-dimensional array, 0-based
;                  $iRunFirstTime = *** internal use only (do not use) ***
; Return values .: @Error -
;                    1 = Path not found or invalid
;                    2 = Invalid $sFilter
;                    3 = Invalid $iFlag
;                    4 = No File(s) Found
;                    5 = Invalid $iPathType
;                    6 = Invalid $iRecursive
;                    7 = Invalid $iRetFormat
;
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                    $array[0] = Number of Files\Folders returned
;                    $array[1] = 1st File\Folder
;                    $array[2] = 2nd File\Folder
;                    $array[3] = 3rd File\Folder
;                    $array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ===========================================================================================
;
;============================================================================================
Func _FileListToArrayEx($sPath, $sFilter = '*', $iFlag = 0, $iPathType = 1, $iRecursive = 0, $sExclude = "", $iRetFormat = 0, $iRunFirstTime = 1)
    Local $aFileList = '', $aFolderList = '', $Tmp = '', $sMask

    If $iRunFirstTime Then
        $sPath = StringRegExpReplace($sPath, '\\+$', ''); Removed trailing backslash
        If Not FileExists($sPath) Then Return SetError(1, 1, 'Path Not Found Or Invalid.')
        If StringRegExp($sFilter, '[\\/ :> <\|]') Then Return SetError(2, 2, 'Invalid Mask('& $sFilter & ').')
        If Not StringRegExp($iFlag, '[012]') Then Return SetError(3, 3, 'Invalid Flag, Chooce (0,1 or 2).')
        If Not StringRegExp($iPathType, '[01]') Then Return SetError(5, 5, 'Invalid Path Type, Choose (0 or 1).')
        If Not StringRegExp($iRecursive, '[01]') Then Return SetError(6, 6, 'Invalid Recursive, Choose (0 or 1).')
        If Not StringRegExp($iRetFormat, '[012]') Then Return SetError(7, 7, 'Invalid RetFormat, Choose (0, 1 or 2).')
    EndIf

    $sMask = '(?i)^' & StringReplace(StringReplace(StringReplace($sFilter, '.', '\.'), '*', '.*'), '?', '.')
    If $sExclude <> '' And $iRunFirstTime Then ;convert $sExclude to fit StringRegExp (not perfect but useable)
        $sExclude = StringStripWS(StringRegExpReplace($sExclude, "\s*;\s*", ";"), 3) ;Strip leading and trailing spaces and spaces between semi colon delimiter
        $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1') ;thanks KaFu and Ascend4nt
        $sExclude = "(?i)\A" & StringReplace(StringReplace(StringReplace($sExclude, "?", "."), "*", ".*?"), ";", "|") & "\z"
    EndIf

    $iFirstFile = FileFindFirstFile($sPath & '\*')
    If @error Then Return
    While True
        $iNextFile = FileFindNextFile($iFirstFile)
        If @error Then ExitLoop
        $extended = @extended
        $sFullPath = $sPath & '\' & $iNextFile
        If $extended = 1 Then
            If $iFlag <> 1 And ($sFilter = '*' Or $sFilter = '*.*') Then $aFileList &= $sFullPath & '|'
            If $iRecursive Then
                If $sExclude <> '' And StringRegExp(StringRegExpReplace($sFullPath, "(.*?[\\/]+)*(.*?\z)", "\2"), $sExclude) Then ContinueLoop
                $Tmp = _FileListToArrayEx($sFullPath, $sFilter, $iFlag, $iPathType, $iRecursive, $sExclude, 1, 0)
                If $Tmp <> Chr(38) And $Tmp <> ChrW(38) Then $aFileList &= $Tmp
            EndIf
        Else
            If Not StringRegExp($iNextFile, $sMask) Then ContinueLoop
            If Not StringRegExp(StringRegExpReplace($sFilter, '^.*\.', ''), '[?*]') Then
                If StringRegExpReplace($sFilter, '^.*\.', '') <> StringRegExpReplace($iNextFile, '^.*\.', '') Then ContinueLoop
            EndIf
            If $sExclude <> '' And StringRegExp(StringRegExpReplace($sFullPath, "(.*?[\\/]+)*(.*?\z)", "\2"), $sExclude) Then ContinueLoop
            If $iFlag <> 2 Then $aFileList &= $sFullPath & '|'
        EndIf
    WEnd
    FileClose($iFirstFile)

    If $iRunFirstTime Then
        $aFileList = StringTrimRight($aFileList, 1)
        If StringLen($aFileList) = 0 Then Return SetError(4, 4, 'No File(s) Found.')
        If $iPathType = 0 Then $aFileList = StringReplace($aFileList, $sPath & '', '')
        If $iRetFormat <> 1 Then $aFileList = StringSplit($aFileList, '|', $iRetFormat)
    EndIf
    Return ($aFileList)
EndFunc ;==>_FileListToArrayEx

Improvement of included _FileListToArray function.

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members