Jump to content

Improvement of included _FileListToArray function.


Tlem
 Share

Recommended Posts

This is a modified version by Zedna and me to gain a few milliseconds and to return the full path of the files/folders (thanks Zedna).

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        If $iFlag > 3 Then $sFile = $sPath & "\" & $sFile
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc ;==>_FileListToArray

The gain of time isn't really significant (about 1,2 seconds for 260000 files) but if you use this function a lot, perhaps it can help you to gain few seconds. :D

Here is some tests results that I have done on my hardware.

Each test was doing 5 times (in a loop) and the result is the average of five values.

Here is the short description of the material that in served for the tests :

PC1 = Laptop IBM R60 C2D T5500 1.66 Ghz / 2.5Go Ram - Windows XP Pro SP3

PC2 = PC no name Quad Core Q6700 overcloked to 3Ghz / 4Go Ram - Windows Vista Pro SP1

Tests results :

PC1 test on 260000 files (stored on local drive) :

Original _FileListToArray = 5350 ms

Modified _FileListToArray = 4150 ms

PC1 test on 100000 files (stored on local drive) :

Original _FileListToArray = 2391 ms / 4075 ms (on battery)

Modified _FileListToArray = 1834 ms / 3150 ms (on battery)

PC1 test on 50000 files (stored on local drive) :

Original _FileListToArray = 1197 ms / 2002 ms (on battery)

Modified _FileListToArray = 924 ms / 1551 ms (on battery)

PC1 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24578 ms

Modified _FileListToArray = 23843 ms

PC2 test on 100000 files (stored on local drive) :

Original _FileListToArray = 1307 ms

Modified _FileListToArray = 1091 ms

PC2 test on 50000 files (stored on local drive) :

Original _FileListToArray = 681 ms

Modified _FileListToArray = 575 ms

PC2 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24635 ms

Modified _FileListToArray = 23447 ms

Edit :

There is a more great improvement on the last function, but it's for the newer version of AutoIt (Work time is divided by 4/5 for file search or folder search only).

I do an UDF with the too version but keep in mind that if you use only AutoIt > 3.3.1.0 then you can simplify to keep only the new version (that use @Extented from FileFindNextFile()).

; New version of FileListToArray (27/06/2009)
; #FUNCTION# ====================================================================================================


================
; Name...........: _FileListToArray
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArray($sPath[, $sFilter = "*"[, $iFlag = 0]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                 $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;                 $iFlag   - Optional: specifies whether to return files folders or both
;                 |$iFlag=0(Default) Return both files and folders
;                 |$iFlag=1 Return files only
;                 |$iFlag=2 Return Folders only
;                 |$iFlag=4 Return Folder/File names also with full path
; Return values .: @Error - 1 = Path not found or invalid
;                 |2 = Invalid $sFilter
;                 |3 = Invalid $iFlag
;                 |4 = No File(s) Found
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                               $array[0] = Number of Files\Folders returned
;                               $array[1] = 1st File\Folder
;                               $array[2] = 2nd File\Folder
;                               $array[3] = 3rd File\Folder
;                               $array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ====================================================================================================


===========================
;Special Thanks to Helge and Layer for help with the $iFlag update
;===============================================================================
Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    If @AutoItVersion > "3.3.1.0" Then
        _FileListToArrayNew($sPath, $sFilter, $iFlag)
    Else
        _FileListToArrayOld($sPath, $sFilter, $iFlag)
    EndIf
EndFunc ;==>_FileListToArray

; For versions of AutoIt > 3.3.1.0 (Use of @extented in FileFindNextFile()
Func _FileListToArrayNew($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"

    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

    If $iFlag > 3 Then
        $sDelim &= $sPath
        $iFlag -= 4
    EndIf

    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
        ; bypass folder
                If @extended Then ContinueLoop; This version for new beta
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
        ; bypass file
                If @extended = 0 Then ContinueLoop; This version for new beta
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch

    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayNew

; For versions of AutoIt < 3.3.1.1
Func _FileListToArrayOld($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"

    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

    If $iFlag > 3 Then
        $sDelim &= $sPath
        $iFlag -= 4
    EndIf

    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
        ; bypass folder
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
        ; bypass file
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch

    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayOld

New Edit :

A new version with recursive search, multiple search, exclusion of files/directory is in progress (see the last posts).

Thanks to BaKaMu.

Edited by Tlem

Best Regards.Thierry

Link to comment
Share on other sites

  • Replies 265
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Adding tests results on my first post.

That what I really want to know :D:D

edit just missing CPU speed for PC1 and PC2 I can't believe Vista is faster than XP ...

Edited by jpm
Link to comment
Share on other sites

If you're after speed, I would think you need only execute the BitAND and a portion of the string statement once, like below. Adding a Switch statement eliminates tons of compares. Between 520,000 and 780,000 less compares in your 260,000 file example, depending on the value of $iFlag.

#include<file.au3>
#include<array.au3>
$iflag = 0

; Beta Version -------------------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray(@SystemDir,"*",$iflag)
Next
$timer1 = TimerDiff ($timer)

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray3(@SystemDir,"*",$iflag)
Next
$timer2 = TimerDiff ($timer) 

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray4(@SystemDir,"*",$iflag)
Next
$timer3 = TimerDiff ($timer) 

MsgBox (0, "", $timer1 & @CRLF & $timer2 & @CRLF & $timer3)
;_ArrayDisplay($x)

; ------------------------------------------------
Func _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        If BitAND($iFlag, 4) Then $sFile = $sPath & "\" & $sFile
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc   ;==>_FileListToArray

; ------------------------------------------------
Func _FileListToArray4($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "") 
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    If $iFlag > 3 Then 
        $sDelim &= $sPath & "\" 
        $iflag -= 4
    EndIf
    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended Then ContinueLoop ; bypass folder
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended = 0 Then ContinueLoop  ; bypass file
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch
    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc   ;==>_FileListToArray

Edit: Oops, I should have read the referenced thread first <sheepish grin>

Edit2: Swapped out the exisitng BitAND statement for a simple "> 3" test. Same result, but easier to read.

Edit3: Removed unnecessary $asFileList[] variable.

Edited by Spiff59
Link to comment
Share on other sites

Sorry guy, I have modified my first post because I had forgotten something important.

Shame on me. :D

In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...

So I made the modification on the function.

Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files. :D

Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")

the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path. ;)

For jpm, I have added the speed of my hardware on test result.

Best Regards.Thierry

Link to comment
Share on other sites

Sorry guy, I have modified my first post because I had forgotten something important.

Shame on me. :D

In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...

So I made the modification on the function.

Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files. :D

Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")

the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path. ;)

For jpm, I have added the speed of my hardware on test result.

The example I'd posted works exactly the same as the first post. $iFlag must be between 0 and 6 or it returns @error=3. If it is >3 then the full path is returned. The only difference it that it is much faster than the other examples.
Link to comment
Share on other sites

Sorry, I haven't seen your example _FileListToArray4.

For the StringRegexp, it was what I want to do. :D

I have tested with StringRegexp and we won 10ms. :D

Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification. ;)

Edited by Tlem

Best Regards.Thierry

Link to comment
Share on other sites

Sorry, I haven't seen your example _FileListToArray4.

For the StringRegexp, it was what I want to do. :D

I have tested with StringRegexp and we won 10ms. :D

Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification. ;)

You are almost missing the point of Spiff59's modifications.

His function is by far the fastest in my testings.

And if he would replace:

If ... Then...
to:

If... Then
    ...
EndIf
it should be even faster.
Link to comment
Share on other sites

According to all suggestions, this is a new version :

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then
        Return SetError(1, 1, "")
    EndIf
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then
        Return SetError(2, 2, "")
    EndIf
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then
        Return SetError(3, 3, "")
    EndIf
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then
        $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    EndIf
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        Switch $iFlag
            Case 1
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf

            Case 2
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf

            Case 4
                $sFile = $sPath & "\" & $sFile

            Case 5
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile

            Case 6
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile
        EndSwitch
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc  ;==>_FileListToArray

This one seems to be more faster than the previous version.

On my test of 100000 files this is the result :

Original _FileListToArray = 2335ms

Modified _FileListToArray = 1591ms

I did not know that :

If... Then
    ...
EndIf
is faster than
If... Then
.

This evening, I would go to bed less stupid.

Thank you very much trancexx.

Edited by Tlem

Best Regards.Thierry

Link to comment
Share on other sites

Since the vaue of $iFlag does not change during a call to the routine, I don't see the point in testing it's value potentially hundreds of thousands of times. It ought to be outside the while loop, in my opinion. I believe that yeilds the fastest results.

And if he would replace:

CODEIf ... Then...to:

CODEIf... Then

...

EndIfit should be even faster.

That is surprising. One would think behind the scenes it would generate the same code. I'll see where I can make those changes... Edited by Spiff59
Link to comment
Share on other sites

@Tlem you miss the point that using @extended do the same as StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") in a much faster way

NbFiles $iflag Version Tlem Spiff59

281100 0 3.3.0.0 2845 19.09% 1944 44.73%

281100 0 3.3.1.1 2907 20.12% 1980 45.58%

281100 4 3.3.0.0 4284 0.00% 2172 49.31%

281100 4 3.3.1.1 4394 0.00% 2209 49.72%

the percentage are calculated against Std UDF, for $iflag=4 the percentage is against "Tlem" version
Link to comment
Share on other sites

Effectively, the Spiff59 function is quicker than my last proposition, but like Zedna said, we must have the last version.

I'll take a look tomorrow, because now it's party time. :D:D

Edited by Tlem

Best Regards.Thierry

Link to comment
Share on other sites

  • Moderators

KaFu,

Looks like Switch is a little slower - on my machine at least:

$fred = 0

$begin = TimerInit()
For $i = 1 To 1000000
    If $fred = 1 Then
        ConsoleWrite("fred" & @CRLF)
    EndIf
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000

    If $fred = 1 Then ConsoleWrite("fred" & @CRLF)

Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000
    Switch $fred
        Case 1
            ConsoleWrite("fred" & @CRLF)
    EndSwitch
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

; Results:
; 1156.37022303114
; 890.749103587188
; 992.363624427127

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...