Jump to content

Recursive(iterative) FileListToArray function comparaison and comments.


tip
 Share

Recommended Posts

Hi everyone;

I don't mean to create a new thread about a subject that has been already widely(wildly) discussed but after reading many many posts I still couldn't figure out which method(function) to prefer...

I gathered all the recursive FileListToArray functions I could find in the forums and made a small test package. If you could run the test script and post your results, I think we can create a modest comparison database on how these funcitons work on different Windows versions with different languages.

I'm not an experienced scripter and I don't entirely understand the methods/routines used in these function. But I also know that the fastest function is not always the best solution. So I would also like to hear what you have to say about these functions too... Reasons to avoid them or why you would prefer one over the other... For example, I've read that the dos dir command is better not to be used because of its non-english character shotcommings etc...

I think it would be really cool if we could find a definitive answer on this subject. :)

My results:

OS:WIN_XP Service Pack 3, OS Arch:X86, Language:0409, AutoIt: 3.3.8.1

Dos dir command returned 24296 items in 1.16 seconds.
FileListToArrayRecursive returned 24304 items in 0.9 seconds.
FileListToArray3 returned 24304 items in 1.52 seconds.
RecFileListToArray returned 24304 items in 1.32 seconds.
FileListToArrayEx returned 24304 items in 0.8 seconds.
FileListToArrayPlus returned 24304 items in 0.71 seconds.
FileListTreeToArray returned 24304 items in 2.83 seconds.

Sincerely,

Tip

FileListToArray Speed Test.zip

Edited by tip

[center]MsgBox_Tipped: Eye candy msgboxes/inputboxes/loginboxes. | CreateBlankBox: Semi-transparent layers with borders and rounded corners.[/center]

Link to comment
Share on other sites

Your _RecFileListToArray test is skewed in that you're using the sorting feature fo the function which will slow it down considerably, but you're not sorting with the other tests. Without the sorting these are the results that I get.

+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-> OS:WIN_7 Service Pack 1, OS Arch:X86, Language:0409

-> Dos command returned 27303 items in 1 seconds.

-> FileListToArrayRecursive returned 27322 items in 2.93 seconds.

-> FileListToArray3 returned 27322 items in 5.81 seconds.

-> RecFileListToArray returned 27322 items in 1 seconds.

-> FileListTreeToArray returned 27322 items in 2.36 seconds.

+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

Without a doubt RecFileListToArray is a very powerful UDF, compared to the rest I've seen.

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Link to comment
Share on other sites

Forgot to mention, as guiness stated, RecFileListToArray is also hands down more versatile because of it's search features and filters.

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

Surely this concludes the discussion then? :)

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Link to comment
Share on other sites

Your _RecFileListToArray test is skewed in that you're using the sorting feature fo the function which will slow it down considerably, but you're not sorting with the other tests.

I didn't notice that. Thank you for pointing it out. I've fixed the test file and updated my results...

Thank you both for quick replies. Is it normal that I have FileListToArrayRecursive function as the fastest with FileListToArray3 coming second and BrewManNH has those functions as the two slowest of all?

[center]MsgBox_Tipped: Eye candy msgboxes/inputboxes/loginboxes. | CreateBlankBox: Semi-transparent layers with borders and rounded corners.[/center]

Link to comment
Share on other sites

Without a doubt RecFileListToArray is a very powerful UDF, compared to the rest I've seen.

+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-> OS:WIN_8 , OS Arch:X64, Language:0409

-> Dos command returned 9416 items in 0.73 seconds.

-> FileListToArrayRecursive returned 9512 items in 0.44 seconds.

-> FileListToArray3 returned 9512 items in 1.46 seconds.

-> RecFileListToArray returned 9512 items in 1.97 seconds.

-> FileListTreeToArray returned 9512 items in 2.64 seconds.

-> _FileListToArrayEx returned 9512 items in 0.41 seconds.

+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Posted Image

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Link to comment
Share on other sites

I'm not familiar with the others but looking at the code the only one which has the option to choose the recursion depth is (you've guessed it) RecFileListToArray.

I wouldn't really pay much attention to the results because it's going to vary between systems.

-> Dos command returned 4475 items in 1.67 seconds.

-> FileListToArrayRecursive returned 4487 items in 0.21 seconds.

-> FileListToArray3 returned 4487 items in 1.88 seconds.

-> RecFileListToArray returned 4487 items in 0.63 seconds.

-> FileListTreeToArray returned 4487 items in 0.46 seconds.

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Link to comment
Share on other sites

Here's an another run of the test (without sorting) on a Win7 x64 laptop:

+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-> OS:WIN_7 Service Pack 1, OS Arch:X64, Language:0409

->

-> Dos command returned 13583 items in 1.75 seconds.

-> FileListToArrayRecursive returned 13590 items in 0.62 seconds.

-> FileListToArray3 returned 13590 items in 3.84 seconds.

-> RecFileListToArray returned 13590 items in 0.95 seconds.

-> FileListTreeToArray returned 13590 items in 3.46 seconds. +>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running on a 64 bit Win7, using 32bit AutoIt3 executable +>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-> OS:WIN_7 Service Pack 1, OS Arch:X64, Language:0409

->

-> Dos command returned 13583 items in 1.42 seconds.

-> FileListToArrayRecursive returned 13590 items in 0.73 seconds.

-> FileListToArray3 returned 13590 items in 3.89 seconds.

-> RecFileListToArray returned 13590 items in 1.06 seconds.

-> FileListTreeToArray returned 13590 items in 3.72 seconds. +>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I only use _RecFileListToArray just because I have so much control over what it does and what it can give me. EDIT: Formatting issues...again

Edited by BrewManNH

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

Hi guinness, First Sorry for my English

I answer you, as you know well autoit, and you understand argument, because I did not want to answer to some other because I will not go down to certain levels

I'm not familiar with the others

is not another, and that by default in autoit, just been updated

but looking at the code

and too simple to understand _FileListToArrayEx (uses only 39 line of code) use only 1 While, and so more than logical and more faster

he only one which has the option to choose the recursion depth is (you've guessed it) RecFileListToArray.

there is not anything that does not support _FileListToArrayEx?

I wouldn't really pay much attention to the results because it's going to vary between systems.

http://youtu.be/DvJZsZKaUWY

However, I want to clarify, this is a discourse without any importance, are not talking about who is better, God forbid (I with Melba23 in autoit us 1 with 10000 ehhhhhh, these my post, I'm not in any way disrespect about Melba23, also looking at its contribution to this community), I like me too much of the work Melba23 (and I was personally useful more than once), and also as a person, always present in the forum and always tries to help everyone without exception

at Melba23 as an example but the same goes for all other users

sorry again for my English

Ciao a tutti.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Link to comment
Share on other sites

Not so fast! Add another to your list!

I still consider the functions near the end of the June 2009 thread: probably the best examples around. Being buried in a 280-post thread they never seem to get much attention. There were 100 different function names used in the thread, such as "_FileListToArraySuperTurbo5000ExtraPlusProWithMaximumUltraPowerboost ", so I've never really known what best to call this, today it shall be _FileListToArrayPlus(). It has powerful options, the best speed I've seen, and comes in as a single backwards-compatible function of 64 lines:

; #FUNCTION# =====================================================================================================================
; _FileListToArrayPlus($sPath, $sInclude = "*", $iFlag = 0, $sExcludeFolder = "", $sExclude = "", $iPathType = 0, $bRecursive = False)
; Name...........:  _FileListToArray_Rec
; Parameters ....:  $sPath: Folder to search
;     $sInclude: String to match on (wildcards allowed, multiples delimited by ;)
;     $iFlag: Returned data type. 0 = Files and folders (default), 1 = Files only, 2 = Folders only
;     $sExcludeFolder: List of folders to exclude from search (wildcards allowed, multiples delimited by ;)
;     $sExclude: List of filenames to exclude from search (wildcards allowed, multiples delimited by ;)
;     $iPathType: Returned data format. 0 = Filename only (default), 1 = Path relative to $sPath, 2 = Full path/filename
;     $bRecursive: False = Search $sPath folder only (default), True = Search $sPath and all subfolders
; Author ........:  Half the Autoit Community (Forum thread #96952)
;===================================================================================================================================

Func _FileListToArrayPlus($sPath, $sInclude = "", $iFlag = 0, $sExcludeFolder = "", $sExclude = "", $iPathType = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""
    $sPath = StringRegExpReplace($sPath, "[/]+z", "") & "" ; ensure single trailing slash
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    ; Edit include files list
    If $sInclude = "*" Then $sInclude = ""
    If $sInclude Then
        If StringRegExp($sInclude, "[/ :><|]|(?s)As*z") Then Return SetError(2, 2, "") ; invalid characters test
        $sInclude = StringRegExpReplace(StringRegExpReplace($sInclude, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
        $sInclude = StringRegExpReplace($sInclude, "[][$.+^{}()]", "$0"); Ignore special characters
        $sInclude = StringReplace(StringReplace(StringReplace($sInclude, "?", "."), "*", ".*?"), ";", "$|") ; Convert ? to ., * to .*?, and ; to |
        $sInclude = "(?i)A(" & $sInclude & "$)"; case-insensitive, match from first char, terminate strings
    EndIf
    ; Edit exclude folders list
    If $sExcludeFolder Then
        $sExcludeFolder = StringRegExpReplace(StringRegExpReplace($sExcludeFolder, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
        $sExcludeFolder = StringRegExpReplace($sExcludeFolder, "[][$.+^{}()]", "$0"); Ignore special characters
        $sExcludeFolder = StringReplace(StringReplace(StringReplace($sExcludeFolder, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=.  *=.*?  ;=|
        $sExcludeFolder = "(?i)A(?!" & $sExcludeFolder & "$)"; case-insensitive, match from first char, terminate strings
    EndIf
    ; Edit exclude files list
    If $sExclude Then
        $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
        $sExclude = StringRegExpReplace($sExclude, "[][$.+^{}()]", "$0"); Ignore special characters
        $sExclude = StringReplace(StringReplace(StringReplace($sExclude, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=.  *=.*?  ;=|
        $sExclude = "(?i)A(?!" & $sExclude & "$)"; case-insensitive, match from first char, terminate strings
    EndIf
;   MsgBox(1,"Masks","File include: " & $sInclude & @CRLF & "File exclude: " & $sExclude & @CRLF & "Dir exclude : " & $sExcludeFolder)

    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iPathType
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolder And Not StringRegExp($file, $sExcludeFolder) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & ""
                EndIf
                If $iFlag = 1 Then ContinueLoop
             $sRet &= $sReturnFormat & $file & "|"
            Else ; File
                If $iFlag = 2 Then ContinueLoop
    If $sInclude And Not StringRegExp($file, $sInclude) Then ContinueLoop
    If $sExclude And Not StringRegExp($file, $sExclude) Then ContinueLoop
    $sRet &= $sReturnFormat & $file & "|"
            EndIf
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

-> Dos command returned 89370 items in 4.26 seconds.

-> FileListToArrayRecursive returned 89438 items in 2.89 seconds.

-> FileListToArray3 returned 89438 items in 5.06 seconds.

-> RecFileListToArray returned 89438 items in 5.15 seconds. (No sort)

-> FileListTreeToArray returned 89438 items in 11.1 seconds.

-> FileListToArrayPlus returned 89438 items in 2.16 seconds.

Sometimes one doesn't want to add 400+ lines and 16 helper functions for just for a simple recursive file search.

It's seems to me you can sort an array after it's built with one line of code?

edit: fix code tags, changed August to June

Edited by Spiff59
Link to comment
Share on other sites

-> Dos command returned 69405 items in 3.51 seconds.

-> FileListToArrayRecursive returned 69424 items in 2.27 seconds.

-> FileListToArray3 returned 69424 items in 9.69 seconds.

-> RecFileListToArray returned 69424 items in 3.08 seconds.

-> FileListTreeToArray returned 69424 items in 7.26 seconds.

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Link to comment
Share on other sites

This multitudes of tests is very attractive, but you would not should lose sight of the feature of each of these functions. ^^

1st : Certain functions have advanced options which can be useful only for one certain number of user.

2nd: Certain functions do not return basic search.

3rd: The gap of the run time of every function is not so important as that (It depends of the search you made).

I think that in this precise example, it is not reasonable to choose with regard to a duration of execution.

Just to give more explanation about that, if I compare _FileListToArrayEx, _RecFileListToArray, _FileListToArrayPlus (the best challengers in my point of view) :

If I use _RecFileListToArray and _FileListToArrayPlus in a recursive search for files and directory, I'v got the list of all directory !!!

I know that it is a request, but in real Windows search life, it doesn't exist anywhere. ^^

So if I want to search for files and directory that match a pattern, only _FileListToArrayEx can do the work correctly without additional treatment.

So finally, all these functions are great, but they do not still correspond to the need which we have of it.

Finaly, for me, I think that _FileListToArrayEx corresponds most to the Windows and AutoIt philosophy, the other functions being specialized in the other processings that those asked originally.

Best Regards.Thierry

Link to comment
Share on other sites

So if I want to search for files and directory that match a pattern, only _FileListToArrayEx can do the work correctly without additional treatment.

Not to come off as sounding like RFLTA needs defending (especially by me), but what exactly does this line mean? RFLTA can search for and/or exclude files and folders by filters so that what is returned can match anything you're looking for. What "additional treatement" do these other functions, other than the FLTAEx function, need that isn't needed in FLTAEx?

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

I have to agree with the points made by Tiem above. I personally use 3 different file listing functions that I wrote a few years ago.

The first is is a slightly modified version of AutoIts built in _FileListToArray which I use when I only need to search a single folder.

The second is a recursive version that only uses a simple match pattern e.g. *.txt

The third is similar to RecFileListToArray in that it allows very complex searches based on a number of parameters using positive patterns as the first match and then doing a second test against another set of parameters to check if it should be excluded.

The point is that there is not one solution and speed is not always the primary reason for choosing other things such as convenience are more important.

Another area where prior knowledge of your data can be useful is in choosing which sorting algorithm to use. Most people use the Quick sort algorithm all the time and where you have no prior knowled it's probaly the best choice. But for example if you are adding a few values to an already sorted list then an insertion sort is much quicker algorithm to resort the list.

Several posters have commented that some functions user fewer lines of code than another with the implication that fewer lines of code is better. This is a fallacy, it's using the correct logic that is important. Having a longer function with serperate sections of code to handle different situation rather than trying to cram all the logic into only a few lines is usually quicker.

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Link to comment
Share on other sites

Not so fast! Add another to your list!

I still consider the functions near the end of the June 2009 thread: probably the best examples around. Being buried in a 280-post thread they never seem to get much attention. There were 100 different function names used in the thread, such as "_FileListToArraySuperTurbo5000ExtraPlusProWithMaximumUltraPowerboost ", so I've never really known what best to call this, today it shall be _FileListToArrayPlus(). It has powerful options, the best speed I've seen, and comes in as a single backwards-compatible function of 64 lines:

Added to the test pack.

This multitudes of tests is very attractive, but you would not should lose sight of the feature of each of these functions.

...

Finaly, for me, I think that _FileListToArrayEx corresponds most to the Windows and AutoIt philosophy, the other functions being specialized in the other processings that those asked originally.

_FileListToArrayEx

Ciao a tutti.

Added to the test pack.

Thanks everyone for your input. I've just added the two mentioned UDFs and updated my results.

I personally think _RecFileListToArray is a very solid and sophisticated UDF and I must add that I don't think 27kb size of the UDF is a disadvantage since after upx compression it doesn't really matter much. But in my tests _FileListToArrayPlus seems to be the fastest. I know this not a real life test and applying include/exclude filters will change results but I believe, it will be very complicated to test for and track outcome of every scenerio. So I tried to keep it simple.

Sincerely,

Tip

[center]MsgBox_Tipped: Eye candy msgboxes/inputboxes/loginboxes. | CreateBlankBox: Semi-transparent layers with borders and rounded corners.[/center]

Link to comment
Share on other sites

@BrewManNH

This is an example script test :

If Not FileExists("C:MyTestMyTest.txt") Then
DirCreate("C:MyTest")
FileWrite("C:MyTestMyTest.txt", "My Test File")
EndIf
$Path = "C:"
$Filter = "MyTest*"
$Flag = 0
$Recursive = 1
$Begin = TimerInit()
$aArray1 = _FileListToArrayEx($Path, $Filter, $Flag, $Recursive)
ConsoleWrite("_FileListToArrayEx returned " & $aArray1[0] & " items in " & TimerDiff($begin)/1000 & " seconds " & @CRLF)
$Begin = TimerInit()
$aArray2 = _RecFileListToArray($Path, $Filter, $Flag, $Recursive)
ConsoleWrite("_RecFileListToArray returned " & $aArray2[0] & " items in " & TimerDiff($begin)/1000 & " seconds " & @CRLF)
DirRemove("C:MyTest", 1)

This is the result on my system :

>Running:(3.3.8.0):C:Program FilesAutoIt3autoit3.exe "D:FileListToArrayTestFileListToArrayTest.au3"  
_FileListToArrayEx returned 2 items in 4.18766156033798 seconds
_RecFileListToArray returned 13663 items in 5.11055455372121 seconds
+>23:57:10 AutoIT3.exe ended.rc:0
>Exit code: 0   Time: 11.557

My goal is to know witch file(s) and directory(s) have the searched pattern.

In a standard syntax, on my system RFLTA return me 13663 items instead of 2 and for me, this is not a request but an inconvenience.

Another example :

If Not FileExists("C:MyTestMyTestFile.txt") Then
    DirCreate("C:MyTestMyTestDir")
    DirCreate("C:MyTestAnotherDir")
    FileWrite("C:MyTestMyTest.txt", "My Test File")
EndIf

$Path = "C:MyTest"
$Filter = "MyTest*"
$Flag = 0
$Recursive = 1

$Begin = TimerInit()
$aArray1 = _FileListToArrayEx($Path, $Filter, $Flag, $Recursive)
ConsoleWrite("_FileListToArrayEx returned " & $aArray1[0] & " items in " & TimerDiff($begin)/1000 & " seconds " & @CRLF)

$Begin = TimerInit()
$aArray2 = _RecFileListToArray($Path, $Filter, $Flag, $Recursive)
ConsoleWrite("_RecFileListToArray returned " & $aArray2[0] & " items in " & TimerDiff($begin)/1000 & " seconds " & @CRLF)

DirRemove("C:MyTest", 1)

The result :

>Running:(3.3.8.0):C:Program FilesAutoIt3autoit3.exe "D:FileListToArrayTestFileListToArrayTest.au3" 
_FileListToArrayEx returned 2 items in 0.000978615997284571 seconds
_RecFileListToArray returned 3 items in 0.00127194936786659 seconds
+>00:34:16 AutoIT3.exe ended.rc:0
>Exit code: 0   Time: 2.002

Notice that RFLTA return 3 items instead of 2.

Now, it is good to know that some functions can make more work and give more results with more code in almost the same time. :)

Edited by Tlem

Best Regards.Thierry

Link to comment
Share on other sites

Different functions for different usages.

If you're going to do an iterative search in subfolders, and you know that what you're looking for is somewhere below "C:Program Files" but you aren't sure which folder it's in, in that folder, but you know it's going to be within the first to 3rd folder deep within that first folder, there's no point to search ALL of the folders under "C:Program Files". _FileListToArrayEX can't limit its search to just 3 folders deep, while _RecFileListToArray can. So, in this not so unusual case, i'm sure it will be a lot faster to return what I'm looking for because it's only looking in the folders I'm expecting it to be in, and not the 300 folders it's not going to be in. Not to mention, if I am looking for something and I know the things I DON'T want, RFLTA has a parameter to filter those things you don't want included, which FLTAEx also doesn't have.

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

Several posters have commented that some functions user fewer lines of code than another with the implication that fewer lines of code is better. ... Having a longer function with serperate sections of code to handle different situation rather than trying to cram all the logic into only a few lines is usually quicker.

I'd have to disagree with that general premise. I think the main goal is to produce well-structured code that is both easily understandable and easily maintained. Poor structure can result from a bloated single function trying to accomplish too much, one that needs to be broken up logically to make it more understandable. I don't see anything meeting that criteria here. Poor structure can also occur from over-fragmenting what would, if left intact, be much more palatable code. In more powerful languages corporate environments I think it's common to print out a source listing, and with marker in hand, study it, edit it, and return to the tube to make your changes. I find Autoit, being intended for less ambitious tasks, tends to be more conducive to (the generally bad practice of) "composing at the tube", where I deal with a monitor worth of code at a time and try to keep the entire program and logic flow in my head as I scroll from screen to screen. For me, that places additional emphasis on striving for a minimum number of simple, concise functions Edited by Spiff59
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...