_FileListToArray new proposal; comments?

[weaponx]

Hi,

Thanks foer all that effort!

I will try to complete it that way and incorporate it.

I am sorry it is all so unclear.

The change you have suggested about the exclude;

The conditional check you had to determine if the exclude string is blank is unnecessary.

(the check is actually on the $flag parameter, to make the exclude blank temporarily while making the folder list before the loop of each directory)

would exclude finding files in foldernames depending on the exclude when it might only be intended to "exclude" in the filename search..

So I think that part should not be changed.

Best, Randall

You're right. I read it wrong.

[randallc]

hi,

post #1 changed..

"** 1d version in post #1 fixes speed for "*" and "*.*", as it is twice as fast and is such a commonly used filter..." - thanks WeaponX

Best, Randall

[randallc]

Hi,

updated post #1;

** 1e fixes folders not returned if only 1 present, and @CR prob

Best, Randall

[MadBoy]

Hi,

updated post #1;

** 1e fixes folders not returned if only 1 present, and @CR prob

Best, Randall

I think something is wrong with newest version when listing only directories. I've just listed both, files only, dirs only and it seems like dirs only shows a lot less directories then it should:

#include <array.au3>
#include <C:\Projects\Project.AU3\Includes\FileListToArray3.au3>
$list_output = _FileListToArray3("C:\Programy", "*.*", 2, 1, 1)
_ArrayDisplay($list_output)oÝ÷ ØGb´ý½è(ëa®ì¨º·(uæ§v-ý½éÚ®º0yØ­v'¶­®)àEèÆáyÊ'²^¸­yû§rب
âµë½êò¶§«"¶ÈhºWm *'²^½êðéò¢è(Â+ajºÚÉج¦V²iû^­+kxz1¦+©àzÈ^jø¥Ê+wöÊ.¶­^¶¬éi³*.jÈz-ç!EèÆÚ«­¢+Ø
½¹Í½±]ɥѡ
I1µÀìÀÌØíÍ¥±MÑÉ¥¹Ä¤(1½°ÀÌØíÁÑÑɸôÌäì ý´¤¡x ý¤¤ÌäìµÀìÀÌØíÍ¥±ÑÈĵÀìÌäìÀÌØì¤Ìäìì°ÀÌØíÜô
½¹Í½±]É¥Ñ ÅÕ½ÐìÀÌØíÍ¥±ÑÈôÅÕ½ÐìµÀì1µÀìÀÌØíÍ¥±ÑÈĵÀì1¤°ÀÌØíÜô
½¹Í½±]É¥Ñ ÅÕ½ÐìÀÌØíÁÑÑɸôÅÕ½ÐìµÀì1µÀìÀÌØíÁÑÑɸµÀì1¤(ÀÌØíÍ1¥ÍÐôMÑÉ¥¹IáÀ ÀÌØíÍ¥±MÑÉ¥¹Ä°ÀÌØíÁÑÑɸ°Ì¤(}ÉÉå¥ÍÁ±ä ÀÌØíÍ1¥ÍФ

Edited February 25, 2008 by MadBoy

[MadBoy]

I've changed your code to fix this issue. Unfortunetly i don't know RegExp so i've used StringSplit. About line 150.

$sFileString1 = StringTrimRight(StringReplace($sFileString1, "*", @LF), 1)
        #cs ; OLD CODE
        ;Change the filters to RegExp filters
        $sFilter1 = StringReplace(StringReplace(StringReplace($sFilter1, ".", "\."), "*", ".*"), "?", ".")
        Local $pattern = '(?m)(^(?i)' & $sFilter1 & '$)' ;, $cw = ConsoleWrite("$sFilter    =" & @LF & $sFilter1 & @LF), $cw = ConsoleWrite("$pattern    =" & @LF & $pattern & @LF)
        $asList = StringRegExp($sFileString1, $pattern, 3)
        #ce
        $asList = StringSplit($sFileString1, @LF, 1)

[randallc]

I think something is wrong with newest version when listing only directories. I've just listed both, files only, dirs only and it seems like dirs only shows a lot less directories then it should:

Hi,

I think you are wrong; you have used a directory filter which requires a "." [DOT] in it, so you will have limited results; and I am not sure you will still get directories which all have a dot with your modification, will you?

I think, for directories, most people want the default, which is "*"?

What do you think?

Best, Randall

[MadBoy]

Ah, great. Didn't think about the filter being wrong Sorry for that.

[mnchartier]

I have implemented this into a script that backs up a few networked PDRs. I have not finished yet because I have noticed that when it retreives the files on one of them the file path is being added to the row number and not column 0 after the 3998th file is found. I have only seen this on one device and it is continuous everytime that I search that device, this device is the most used so there are 6000+ files on it, others are only around 2000-3000 so I have not noticed it anywhere else.

[weaponx]

I have implemented this into a script that backs up a few networked PDRs. I have not finished yet because I have noticed that when it retreives the files on one of them the file path is being added to the row number and not column 0 after the 3998th file is found. I have only seen this on one device and it is continuous everytime that I search that device, this device is the most used so there are 6000+ files on it, others are only around 2000-3000 so I have not noticed it anywhere else.

Listview limitation, manifests in _ArrayDisplay. Nothing is wrong with this function.

[jvnorris]

How do you implement exclusions??

I have the folowing code

$UserDirectories = _FileListToArray($BaseFolder, "*", 2); Build the list of users from the docs and settings folder

Now if I try to Add any of the other options any time I compile it craps out telling me ther are too many arguments for this function

Your code shows the Syntax as

Syntax:        _FileListToArray($s_Path, $s_Filter = "*", $i_Flag = 0)

But the Parameters as

; Parameter(s):     $s_Path = Path to generate filelist for
;                   $s_Filter = The filter to use. Search the Autoit3 manual for the word "WildCards" For details
;                  $i_Flag = determines whether to return file or folders or both
;                       $i_Flag=0(Default) Return both files and folders
;                      $i_Flag=1 Return files Only
;                       $i_Flag=2 Return Folders Only
;                   $i_Recurse = Indicate whether recursion to subfolders required
;                       $i_Recurse=0(Default) No recursion to subfolders
;                      $i_Recurse=1 recursion to subfolders
;                   $i_BaseDir = Indicate whether base directory name included in returned elements
;                       $i_BaseDir=0 base directory name not included
;                      $i_BaseDir=1 (Default) base directory name included
;                   $s_Exclude= The Exclude filter to use.  "WildCards" For details

How do we use these Parameters? I would like to put some of the other options to use to.

I am Using FileListToArray3.au3 BTW

THank you for your time.

[jvnorris]

How do you implement exclusions??

I have the folowing code

$UserDirectories = _FileListToArray($BaseFolder, "*", 2); Build the list of users from the docs and settings folder

Now if I try to Add any of the other options any time I compile it craps out telling me ther are too many arguments for this function

Your code shows the Syntax as

Syntax:        _FileListToArray($s_Path, $s_Filter = "*", $i_Flag = 0)

But the Parameters as

; Parameter(s):     $s_Path = Path to generate filelist for
;                   $s_Filter = The filter to use. Search the Autoit3 manual for the word "WildCards" For details
;                  $i_Flag = determines whether to return file or folders or both
;                       $i_Flag=0(Default) Return both files and folders
;                      $i_Flag=1 Return files Only
;                       $i_Flag=2 Return Folders Only
;                   $i_Recurse = Indicate whether recursion to subfolders required
;                       $i_Recurse=0(Default) No recursion to subfolders
;                      $i_Recurse=1 recursion to subfolders
;                   $i_BaseDir = Indicate whether base directory name included in returned elements
;                       $i_BaseDir=0 base directory name not included
;                      $i_BaseDir=1 (Default) base directory name included
;                   $s_Exclude= The Exclude filter to use.  "WildCards" For details

How do we use these Parameters? I would like to put some of the other options to use to.

I am Using FileListToArray3.au3 BTW

THank you for your time.

Never mind Found that the Header info and the actual function itself are different. Used the direct _FileListToArray3 Function and all is working now.

Very nice useful tool BTW nice work. Just need to clean up the documentation and comments in it a bit. Some basic scripters are used to juest reading that main comment and trusting it if it is there as the right commands.

Oh ANd where the heck is the Edit button I hate double posting

[Xenobiologist]

Hi,

I haven't read the whole thread, so sorry if this is already answered. I got a problem here.

@Error=4 No File(s) Found

If never get there cause I always get an array back where [0] is 1 even if a pass a filter like *.3243243214321432142343214

Mega

[SoulA]

bumping an old post but I need help... trying to get a comprehensive file listing of my computer but I want to exclude certain folders such as the windows folder and the recycle bin but when I put in this exclusion

$FileList = _FileListToArray3($drives[$i], "*", 0, 1, 1, "Windows", 1)

The windows file listing still comes up in my list. Any help?

[leuce]

I haven't read the entire thread either, and I'll certainly check out that function, but in the mean time, in case anyone wanders by and wants to know how to do this in MS DOS (Windows XP), here's how:

Recursive directory listing

List all files in current and subdirectories:

dir /a /s /b > dir.txt

Same as previous, but list only directories:

dir /a:d /s /b > dir.txt

List all *.txt files in current and subdirectories:

dir /a /s /b *.txt > dir.txt

Personally I then use either FileReadLine with a For...Next loop to put the paths in a variable, or I read the entire file in one go and use StringSplit with @CRLF as delimiter to create an array with the paths. Not elegant, but it works for now.

[SmOke_N]

I haven't read the entire thread either, and I'll certainly check out that function, but in the mean time, in case anyone wanders by and wants to know how to do this in MS DOS (Windows XP), here's how:

Recursive directory listing

List all files in current and subdirectories:

dir /a /s /b > dir.txt

Same as previous, but list only directories:

dir /a:d /s /b > dir.txt

List all *.txt files in current and subdirectories:

dir /a /s /b *.txt > dir.txt

Personally I then use either FileReadLine with a For...Next loop to put the paths in a variable, or I read the entire file in one go and use StringSplit with @CRLF as delimiter to create an array with the paths. Not elegant, but it works for now.

Yep, you haven't read the entire thread and or why they went this route. In the _FileListToArray referenced to me, I use 1 or more of the suggestions you've made. But they had an issue (which is true) with specific characters and such on non-english machines.

[leuce]

Yep, you haven't read the entire thread and or why they went this route. In the _FileListToArray referenced to me, I use 1 or more of the suggestions you've made. But they had an issue (which is true) with specific characters and such on non-english machines.

Well I did read some of it :-) and what you said about non-English machines etc did strike me as something that needs to be addressed. I just thought I post the above information here because I had looked for it and could not find it, and I thought it would be useful for anyone wandering along. And yes, I also work in an environment with non-English file names, so I know what you're on about.

[Mojo]

Do I understand it correctly, that with _FileListToArray(), also the original one (I'm still using AutoIt v3.2.12.0)

there's never a problem with non-english characters?

Those probs only occur with the DOS commands?

I first thought, that was the purpose of this thread, to create a FileListToArray function which doesn't have those character-problems,

but now I think it's only about speeding up the FileListToArray function, right? (AND adding more options, like recurse subfolders, etc.)

[spuri]

Hello,

$SubList = _FileListToArray3($drv & $client, "*", 2, 1, 1, "CS|DP", 1)

I need to filter directoris CD and DP and also parent directory of CS and DP, is it possible somehow?

Do you have any idea? Cpuld you please help?

thanks

s

I don't need the underlined dirs

example

d:\client1

d:\client1\cs

d:\client1\dp

d:\client2\

d:\client2\cs

d:\client2\dp

d:\specwork\

d:\specwork\client3

d:\specwork\client3\cs

d:\specwork\client3\dp

d:\specwork2

d:\specwork2\client4

d:\specwork2\client4\cs

d:\specwork2\client4\dp

Edited January 21, 2009 by spuri

[Spiff59]

This is notably faster than the existing _FileListToArray, and works?

If you're open to input, which you seem to be, I think there's room for improvement...

Glancing at the first 100 lines I see these changes would either eliminate a warning (Line 43), eliminate unnecessary code, or just increase speed...

Line 43:

Local $i_deleteduplicate = BitAND($i_Options, 1)

Line 54:

If StringRegexp($sFilter, "[\\/:<>]") Or (Not StringStripWS($sFilter, 8)) Then

Line 57:

Switch $iFlag
    Case 0,1,2
    Case Else
        Return SetError(3, 3, "")
EndSwitch

Line 61: comment is incorrect

Line 79:

ElseIf StringTrimRight($sFilter, 1) <> "*"  And StringTrimRight($sFilter, 1) <> "*.*"  Then

Line 92:

Else

I haven't spent enough time looking at it to offer more than line-at-a-time changes, but I could probably find some time if you're interested.

Edit: Just looked at the existing _FileListToArray, and notice both of these use FileFindFirstFile and FileFindNextFile. And it's gone from a mouse to an elephant... how are you realizing the speed increase? (I also see I sucked myself into an old thread someone resurrected)

Edited February 15, 2009 by Spiff59

[Spiff59]

I understand that almost all the processing time is spent in the While loop.

I wonder if the existing FLTA would benefit any from adding the variables $FilesOnly and $FoldersOnly?

Something like this:

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $asFileList[1], $FilesOnly, $FoldersOnly
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    Switch $iFlag
        Case 0
        Case 1
            $FilesOnly = 1
        Case 2
            $FoldersOnly = 1
        Case Else
            Return SetError(3, 3, "")
    EndSwitch
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $FilesOnly And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $FoldersOnly And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        $asFileList[0] += 1
        If UBound($asFileList) <= $asFileList[0] Then ReDim $asFileList[UBound($asFileList) * 2]
        $asFileList[$asFileList[0]] = $sFile
    WEnd
    FileClose($hSearch)
    ReDim $asFileList[$asFileList[0] + 1]; Trim unused slots
    Return $asFileList
EndFunc ;==>_FileListToArray

Update: I ran some tests, and the differences were negligible. I don't see how I'd realize a meaningful improvement without hardware changes; adding a fourth drive to my 3-drive striped array, or swapping out my 7200's for raptors.

Edited February 15, 2009 by Spiff59