Sign in to follow this  
Followers 0

Improvement of included _FileListToArray function.

266 posts in this topic

Posted (edited)

This is a modified version by Zedna and me to gain a few milliseconds and to return the full path of the files/folders (thanks Zedna).

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
	Local $hSearch, $sFile, $sFileList, $asFileList[1]
	If Not FileExists($sPath) Then Return SetError(1, 1, "")
	If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
	If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")
	If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
	$hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
	If $hSearch = -1 Then Return SetError(4, 4, "")
	While 1
		$sFile = FileFindNextFile($hSearch)
		If @error Then
			SetError(0)
			ExitLoop
		EndIf
		If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
		If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
		If $iFlag > 3 Then $sFile = $sPath & "\" & $sFile
		$sFileList &= $sFile & "|"
	WEnd
	FileClose($hSearch)
	$asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
	Return $asFileList
EndFunc ;==>_FileListToArray

The gain of time isn't really significant (about 1,2 seconds for 260000 files) but if you use this function a lot, perhaps it can help you to gain few seconds. :D

Here is some tests results that I have done on my hardware.

Each test was doing 5 times (in a loop) and the result is the average of five values.

Here is the short description of the material that in served for the tests :

PC1 = Laptop IBM R60 C2D T5500 1.66 Ghz / 2.5Go Ram - Windows XP Pro SP3

PC2 = PC no name Quad Core Q6700 overcloked to 3Ghz / 4Go Ram - Windows Vista Pro SP1

Tests results :

PC1 test on 260000 files (stored on local drive) :

Original _FileListToArray = 5350 ms

Modified _FileListToArray = 4150 ms

PC1 test on 100000 files (stored on local drive) :

Original _FileListToArray = 2391 ms / 4075 ms (on battery)

Modified _FileListToArray = 1834 ms / 3150 ms (on battery)

PC1 test on 50000 files (stored on local drive) :

Original _FileListToArray = 1197 ms / 2002 ms (on battery)

Modified _FileListToArray = 924 ms / 1551 ms (on battery)

PC1 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24578 ms

Modified _FileListToArray = 23843 ms

PC2 test on 100000 files (stored on local drive) :

Original _FileListToArray = 1307 ms

Modified _FileListToArray = 1091 ms

PC2 test on 50000 files (stored on local drive) :

Original _FileListToArray = 681 ms

Modified _FileListToArray = 575 ms

PC2 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24635 ms

Modified _FileListToArray = 23447 ms

Edit :

There is a more great improvement on the last function, but it's for the newer version of AutoIt (Work time is divided by 4/5 for file search or folder search only).

I do an UDF with the too version but keep in mind that if you use only AutoIt > 3.3.1.0 then you can simplify to keep only the new version (that use @Extented from FileFindNextFile()).

; New version of FileListToArray (27/06/2009)
; #FUNCTION# ====================================================================================================


================
; Name...........: _FileListToArray
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArray($sPath[, $sFilter = "*"[, $iFlag = 0]])
; Parameters ....: $sPath   - Path to generate filelist for.
;				  $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;				  $iFlag   - Optional: specifies whether to return files folders or both
;				  |$iFlag=0(Default) Return both files and folders
;				  |$iFlag=1 Return files only
;				  |$iFlag=2 Return Folders only
;				  |$iFlag=4 Return Folder/File names also with full path
; Return values .: @Error - 1 = Path not found or invalid
;				  |2 = Invalid $sFilter
;				  |3 = Invalid $iFlag
;				  |4 = No File(s) Found
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;								$array[0] = Number of Files\Folders returned
;								$array[1] = 1st File\Folder
;								$array[2] = 2nd File\Folder
;								$array[3] = 3rd File\Folder
;								$array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ====================================================================================================


===========================
;Special Thanks to Helge and Layer for help with the $iFlag update
;===============================================================================
Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
	If @AutoItVersion > "3.3.1.0" Then
		_FileListToArrayNew($sPath, $sFilter, $iFlag)
	Else
		_FileListToArrayOld($sPath, $sFilter, $iFlag)
	EndIf
EndFunc ;==>_FileListToArray

; For versions of AutoIt > 3.3.1.0 (Use of @extented in FileFindNextFile()
Func _FileListToArrayNew($sPath, $sFilter = "*", $iFlag = 0)
	Local $hSearch, $sFile, $sFileList, $sDelim = "|"

	If Not FileExists($sPath) Then Return SetError(1, 1, "")
	If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
	If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

	If $iFlag > 3 Then
		$sDelim &= $sPath
		$iFlag -= 4
	EndIf

	$hSearch = FileFindFirstFile($sPath & $sFilter)
	If $hSearch = -1 Then
		Return SetError(4, 4, "")
	EndIf

	Switch $iFlag
		Case 0; Files and Folders
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then
					ExitLoop
				EndIf
				$sFileList &= $sDelim & $sFile
			WEnd
		Case 1; Files Only
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then
					ExitLoop
				EndIf
		; bypass folder
				If @extended Then ContinueLoop; This version for new beta
				$sFileList &= $sDelim & $sFile
			WEnd
		Case 2; Folders Only
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then
					ExitLoop
				EndIf
		; bypass file
				If @extended = 0 Then ContinueLoop; This version for new beta
				$sFileList &= $sDelim & $sFile
			WEnd
		Case Else
			Return SetError(3, 3, "")
	EndSwitch

	FileClose($hSearch)
	Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayNew

; For versions of AutoIt < 3.3.1.1
Func _FileListToArrayOld($sPath, $sFilter = "*", $iFlag = 0)
	Local $hSearch, $sFile, $sFileList, $sDelim = "|"

	If Not FileExists($sPath) Then Return SetError(1, 1, "")
	If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
	If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

	If $iFlag > 3 Then
		$sDelim &= $sPath
		$iFlag -= 4
	EndIf

	$hSearch = FileFindFirstFile($sPath & $sFilter)
	If $hSearch = -1 Then
		Return SetError(4, 4, "")
	EndIf

	Switch $iFlag
		Case 0; Files and Folders
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then ExitLoop
				$sFileList &= $sDelim & $sFile
			WEnd
		Case 1; Files Only
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then ExitLoop
		; bypass folder
				If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
				$sFileList &= $sDelim & $sFile
			WEnd
		Case 2; Folders Only
			While 1
				$sFile = FileFindNextFile($hSearch)
				If @error Then ExitLoop
		; bypass file
				If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
				$sFileList &= $sDelim & $sFile
			WEnd
		Case Else
			Return SetError(3, 3, "")
	EndSwitch

	FileClose($hSearch)
	Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayOld

New Edit :

A new version with recursive search, multiple search, exclusion of files/directory is in progress (see the last posts).

Thanks to BaKaMu.

Edited by Tlem

Share this post


Link to post
Share on other sites



Posted

Good job old chaps(toi & Zedna) :D

You could have given the total execution before on the 260000 files so we can know what is the percentage gain.

1,2 sec on ???

Share this post


Link to post
Share on other sites

Posted

It's related to the entire thread, not specifically what you asked in your post. Just doing this for future reference. : )

Share this post


Link to post
Share on other sites

Posted

Adding tests results on my first post.

Share this post


Link to post
Share on other sites

Posted (edited)

Adding tests results on my first post.

That what I really want to know :D:D

edit just missing CPU speed for PC1 and PC2 I can't believe Vista is faster than XP ...

Edited by jpm

Share this post


Link to post
Share on other sites

Posted (edited)

If you're after speed, I would think you need only execute the BitAND and a portion of the string statement once, like below. Adding a Switch statement eliminates tons of compares. Between 520,000 and 780,000 less compares in your 260,000 file example, depending on the value of $iFlag.

#include<file.au3>
#include<array.au3>
$iflag = 0

; Beta Version -------------------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray(@SystemDir,"*",$iflag)
Next
$timer1 = TimerDiff ($timer)

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray3(@SystemDir,"*",$iflag)
Next
$timer2 = TimerDiff ($timer) 

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray4(@SystemDir,"*",$iflag)
Next
$timer3 = TimerDiff ($timer) 

MsgBox (0, "", $timer1 & @CRLF & $timer2 & @CRLF & $timer3)
;_ArrayDisplay($x)

; ------------------------------------------------
Func _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        If BitAND($iFlag, 4) Then $sFile = $sPath & "\" & $sFile
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc   ;==>_FileListToArray

; ------------------------------------------------
Func _FileListToArray4($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "") 
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    If $iFlag > 3 Then 
        $sDelim &= $sPath & "\" 
        $iflag -= 4
    EndIf
    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended Then ContinueLoop ; bypass folder
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended = 0 Then ContinueLoop  ; bypass file
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch
    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc   ;==>_FileListToArray

Edit: Oops, I should have read the referenced thread first <sheepish grin>

Edit2: Swapped out the exisitng BitAND statement for a simple "> 3" test. Same result, but easier to read.

Edit3: Removed unnecessary $asFileList[] variable.

Edited by Spiff59

Share this post


Link to post
Share on other sites

Posted

Sorry guy, I have modified my first post because I had forgotten something important.

Shame on me. :D

In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...

So I made the modification on the function.

Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files. :D

Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")

the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path. ;)

For jpm, I have added the speed of my hardware on test result.

Share this post


Link to post
Share on other sites

Posted

Sorry guy, I have modified my first post because I had forgotten something important.

Shame on me. :D

In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...

So I made the modification on the function.

Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files. :D

Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")

the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path. ;)

For jpm, I have added the speed of my hardware on test result.

The example I'd posted works exactly the same as the first post. $iFlag must be between 0 and 6 or it returns @error=3. If it is >3 then the full path is returned. The only difference it that it is much faster than the other examples.

Share this post


Link to post
Share on other sites

Posted (edited)

Sorry, I haven't seen your example _FileListToArray4.

For the StringRegexp, it was what I want to do. :D

I have tested with StringRegexp and we won 10ms. :D

Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification. ;)

Edited by Tlem

Share this post


Link to post
Share on other sites

Posted

Sorry, I haven't seen your example _FileListToArray4.

For the StringRegexp, it was what I want to do. :D

I have tested with StringRegexp and we won 10ms. :D

Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification. ;)

You are almost missing the point of Spiff59's modifications.

His function is by far the fastest in my testings.

And if he would replace:

If ... Then...
to:

If... Then
	...
EndIf
it should be even faster.

Share this post


Link to post
Share on other sites

Posted (edited)

According to all suggestions, this is a new version :

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then
        Return SetError(1, 1, "")
    EndIf
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then
        Return SetError(2, 2, "")
    EndIf
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then
        Return SetError(3, 3, "")
    EndIf
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then
        $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    EndIf
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        Switch $iFlag
            Case 1
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf

            Case 2
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf

            Case 4
                $sFile = $sPath & "\" & $sFile

            Case 5
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile

            Case 6
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile
        EndSwitch
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc  ;==>_FileListToArray

This one seems to be more faster than the previous version.

On my test of 100000 files this is the result :

Original _FileListToArray = 2335ms

Modified _FileListToArray = 1591ms

I did not know that :

If... Then
	...
EndIf
is faster than
If... Then
.

This evening, I would go to bed less stupid.

Thank you very much trancexx.

Edited by Tlem

Share this post


Link to post
Share on other sites

Posted (edited)

Since the vaue of $iFlag does not change during a call to the routine, I don't see the point in testing it's value potentially hundreds of thousands of times. It ought to be outside the while loop, in my opinion. I believe that yeilds the fastest results.

And if he would replace:

CODEIf ... Then...to:

CODEIf... Then

...

EndIfit should be even faster.

That is surprising. One would think behind the scenes it would generate the same code. I'll see where I can make those changes... Edited by Spiff59

Share this post


Link to post
Share on other sites

Posted

@Tlem you miss the point that using @extended do the same as StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") in a much faster way

NbFiles $iflag Version Tlem Spiff59

281100 0 3.3.0.0 2845 19.09% 1944 44.73%

281100 0 3.3.1.1 2907 20.12% 1980 45.58%

281100 4 3.3.0.0 4284 0.00% 2172 49.31%

281100 4 3.3.1.1 4394 0.00% 2209 49.72%

the percentage are calculated against Std UDF, for $iflag=4 the percentage is against "Tlem" version

Share this post


Link to post
Share on other sites

Posted

And if he would replace:

If ... Then...
to:

If... Then
    ...
EndIf
it should be even faster.
Are you sure about this?

I haven't heard about it and I see no reason what such behaviour.

Share this post


Link to post
Share on other sites

Posted

@Tlem you miss the point that using @extended do the same as StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") in a much faster way

But this is only in latest Beta version (not in release 3.3).

Share this post


Link to post
Share on other sites

Posted (edited)

Effectively, the Spiff59 function is quicker than my last proposition, but like Zedna said, we must have the last version.

I'll take a look tomorrow, because now it's party time. :D:D

Edited by Tlem

Share this post


Link to post
Share on other sites

Posted

Are you sure about this?

I've read somewhere that switch case is faster then if then...

Share this post


Link to post
Share on other sites

Posted

KaFu,

Looks like Switch is a little slower - on my machine at least:

$fred = 0

$begin = TimerInit()
For $i = 1 To 1000000
	If $fred = 1 Then
		ConsoleWrite("fred" & @CRLF)
	EndIf
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000

	If $fred = 1 Then ConsoleWrite("fred" & @CRLF)

Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000
	Switch $fred
		Case 1
			ConsoleWrite("fred" & @CRLF)
	EndSwitch
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

; Results:
; 1156.37022303114
; 890.749103587188
; 992.363624427127

M23

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0