255 replies to this topic

[Melba23]

Malkavian99,

The newest version is always in the first post of this thread. For UDFs that do not form part of the AutoIt install that is the standard we have adopted. The link in my sig will take you there.

M23

[Malkavian99]

I will head back to the first post and see if they match the one I have. Thanks.

[GhostProtocol]

When running this against user drive within my organization I am getting an autoit error:

Error allocating memory.

The number of files is roughly 8,000 and folders is 1,000 (all together size 3.41GB). I can run this UDF and just watch the RAM usage climb until it errors. I am guessing this UDF wont work for what I am trying, what other ideas are there to try? I thought listing the file to a DB or a flat file would do, are there any functions that do this?

$file = _RecFileListToArray($oldUserDrive, "*", 0, 1)

Basically my variable $file is getting to be bigger than 2Gb which is how much RAM is available.

Thanks!

[GhostProtocol]

I will head back to the first post and see if they match the one I have. Thanks.

I looked at my old UDF and looked at the date modified, it was August something, so I renamed it to .old and copied this new one from October in and then the date modified showed 10/18/2011

[Melba23]

GhostProtocol,

Welcome to the AutoIt forum.

Sorry to hear about your problem, but I have just run the UDF on my C: drive which found 27,301 folders and 181,320 files - and put them all into an _ArrayDisplay dialog without a murmur. I too have 2G of RAM. The UDF only stores the names of the files and folders - do you have ridiculously long path names?

So there must be another reason why it is not working for you. Please post all of the script you are using so we can take a look.

M23

[GhostProtocol]

GhostProtocol,

Welcome to the AutoIt forum.

Sorry to hear about your problem, but I have just run the UDF on my C: drive which found 27,301 folders and 181,320 files - and put them all into an _ArrayDisplay dialog without a murmur. I too have 2G of RAM. The UDF only stores the names of the files and folders - do you have ridiculously long path names?

So there must be another reason why it is not working for you. Please post all of the script you are using so we can take a look.

M23

I put a msgbox on each side of the function that calls your UDF I get the first msgbox and then the RAM climbs until I get the error allocating memory it never reaches the second msgbox. Please keep in mind that I have run this many times and I have never had an issue with it until recently, I'll try to post the code later, I'll have to change specific information to be more generic before I can post it.

[GhostProtocol]

Actually I just post this function cause I know the problem is in it.

AutoIt        expandcollapse  popup

Func CopyUserDrive()
$oldUserDrive = "domain.orguser" & @UserName & ""
$newUserDrive = @MyDocumentsDir
GUICtrlSetData($ProgressBar, 0)
GUICtrlSetData($ProgressText, "Gathering directory structure and files from old user drive (H:) for copy.")
WriteToLog("Gathering directory structure and files from old user drive (H:) for copy.")
$file = _RecFileListToArray($oldUserDrive, "*", 0, 1)
;~  _ArrayDisplay($file)
GUICtrlSetData($ProgressText, "Copying from: old user drive(H:) to: new user drive(My Documents).")
WriteToLog("Copying from: old user drive(H:) to: new user drive(My Documents).")
  For $index = 1 To $file[0]
                FileCopy($oldUserDrive & $file[$index],$newUserDrive & "" & $file[$index], 9); Flag = 1 + 8 (overwrite + create target directory structure)
      GUICtrlSetData($ProgressBar, $index*100/Number($file[0]))
      $iPercentage = StringInStr($index*100/Number($file[0]), ".") - 1 ;get the place of the dot and subtract 1 so dot isnt included
      $Percentage = StringLeft($index*100/Number($file[0]), $iPercentage) ; trim remainder from number to make the number a whole number
     ; $PercentageText = $Percentage & "% Complete" & @CRLF & "from: H:" & @CRLF & "to: My Documents"
     $PercentageText = $Percentage & "% Complete" & @CRLF & "From: H:" & StringReplace($file[$index], @CRLF, "") & @CRLF & "To: My Documents" & StringReplace($file[$index], @CRLF, "")
   WriteToLog("H:" & $file[$index] & " copied to My Documents" & $file[$index])
      ;GUICtrlSetData($ProgressText, $Percentage & " Percent Complete" & @CRLF & $oldUserDrive & "" & $file[$index] & @CRLF & " to:" & @CRLF & $newUserDrive & "" & $file[$index]) ;long version
      GUICtrlSetData($ProgressText, $PercentageText) ;short version
     Next
     WriteToLog(@CRLF)
EndFunc

The path names could be very long, there could be many nested directories, I have tested this on my own user drive and it works flawlessly of course I don't have as many directories and files as other people so that may have been why it worked. The _ArrayDisplay($file) never shows up I first get the error allocating memory. What can I do to easily see what path names might be to long without having to physically scour the drive?

Edited by GhostProtocol, 09 January 2012 - 01:49 PM.

[Melba23]

GhostProtocol,

First thought - why are you not using DirCopy to do this?

"DirCopy - Copies a directory and all sub-directories and files (Similar to xcopy)"

As to the UDF, I have just rerun the test on my C: drive - I got the memory usage up to only ~194MB with my 200,000 odd files and folders. So I can only wonder what size the folder tree must be on the drives you are trying to copy if the names only bulk out 2GB!

What I suggest to try out the possibility that it is the sheer size of the folder tree causing problems is to use the UDF to search only to a certain depth in the tree and see if the problem still exists. As you can see from the function header, setting the $iRecur parameter to a negative value limits the recursion level. So run the UDF on the drive (without the copying code) and gradually increase the recursion level to see how the RAM copes with the increased numbers of items found. If you find there is a problem at a certain level, we can look to see if the pathnames suddenly get extraordinarily long at that point.

Let me know how you get on - I am really intrigued by this.

M23

[z0iid]

GhostProtocol, First thought - why are you not using DirCopy to do this?

"DirCopy - Copies a directory and all sub-directories and files (Similar to xcopy)"

As to the UDF, I have just rerun the test on my C: drive - I got the memory usage up to only ~194MB with my 200,000 odd files and folders. So I can only wonder what size the folder tree must be on the drives you are trying to copy if the names only bulk out 2GB! What I suggest to try out the possibility that it is the sheer size of the folder tree causing problems is to use the UDF to search only to a certain depth in the tree and see if the problem still exists. As you can see from the function header, setting the $iRecur parameter to a negative value limits the recursion level. So run the UDF on the drive (without the copying code) and gradually increase the recursion level to see how the RAM copes with the increased numbers of items found. If you find there is a problem at a certain level, we can look to see if the pathnames suddenly get extraordinarily long at that point. Let me know how you get on - I am really intrigued by this. M23

Hi,

I work with TomCruise (GhostProtocol) - he is out for the day (he works nights).

Purpose of using this function over DirCopy:

need to log each file copy, so if user complains "I'm missing this file", we can refer to log.

Actual folder size and count:

59,125 files in 3,386 folder - 36.3GB

#include <_RecFileListToArray.au3>
#include<array.au3>
$oldUserDrive = "y:"
$file = _RecFileListToArray($oldUserDrive, "*", 0, -11)
_ArrayDisplay($file)

This creates an array with 78807 items (more than the total count of all files and folders combined - and any deeper, it will produce the error) - it takes 54 seconds.

My hardware: AMD 2378 2.4ghz Quad Core w/8gb of ram, win 7 x64.

See image for possible error - the array includes lines with .. - but as you can see, the actual path does not have any self references. We also noticed this with folders that contain a single quote in the name - ie, y:pictures'08-04-20-pictures - when stripping/renaming the folder w/out a single quote, the array did not produce lines with ..

The RegEx doesn't appear to handle single quotes - and I'm not proficient enough to tell if it handles paren's.

[Melba23]

z0iid,

Thanks for that - at least now we know what is going wrong. I am looking into it.

M23

[Melba23]

z0iid,

I am afraid I cannot reproduce the error with a similar folder structure. As I expected, the SRE copes with single quotes and parentheses without problem. Here is the script in SciTE, the folder tree in my file manager, and the resulting _ArrayDisplay - as you can see I get both the file with parentheses and the folder with a single quote:

Other than your tree being on a network drive, can you see anything else that is different?

M23

[z0iid]

The command window lists the "(N)EMERGENCY GENERATOR COOLING PIPING.TIF" file before the self reference and parent directory listings - could this have any impact? try putting some dummy files in the parent directories? (since you can see the array is referencing files in the parent directories)

4 actual files creates an array with 356 items..... not sure what else to look at.... I'll zip up that dir, maybe you can duplicate with just that small subset?

http://dl.dropbox.com/u/574904/pict.zip

Edited by z0iid, 09 January 2012 - 05:11 PM.

[Melba23]

z0iid,

The order of the listing is dependent on the algorithm used - MS and I obviously went different routes!

You appear to be picking up the . and .. folders in that particular branch and that is causing the UDF to run away. I wondered if it might have something to do with the folder name ("pict") but I do not get the same result when I run the UDF on my own folder tree - and I did try putting dummy files in the parent folders which ran perfectly.

What OS are you running?

And if you want to zip up the troublesome folder I would be happy to try it on my system.

M23

[z0iid]

see the link in the post above (pict.zip) - the parent directory is 100~mb - let me know if you want me to zip that up as well, but I'll have to make sure there isn't any proprietary information in it first.

I can try this on multiple OS's - but we originally noticed the problem on Windows XP 32bit - I've been doing the debugging on Windows 7 64bit since the behavior seems to be the same.

The behavior we noticed was on any folder that had a single quote or single paren - but when I created that folder structure on my local hard drive, it also behaved perfectly (not picking up . or ..) - we have a few user directories that are causing this behavior.

As a side note - those shares happen to reside on a Linux server running Netware with CIFS running. We are decommissioning Novell in our environment, and this function allows to to migrate their data with a log while maintaining ownership/permissions.

We have used it on hundreds of users without flaw - and now we've run into a few users with large directories. Obviously, restricting the level of recursiveness would "work" (not produce out of memory error) - but it would potentially error out when attempting to copy a file path with .. in it. Also, a function to acurately set how deep the folder structure is ahead of time could work, but it would slow things down. And wouldn't solve the ".." problem if it was referenced at a level higher than $iRecur was set.

A possible solution would be to add a regex that looks for the pattern "." or "..", and doesn't write to the array - but how do you prevent it from getting in a self referencing loop in the first place? (i'm sort of "musing aloud" - sorry if these are noobish or uneducated questions, i'm pretty much a beginner when it comes to anything deeper than basic procedural programming.)

[Melba23]

z0iid,

Try this version where I have specifically excluded folders named . and .. from being actioned:

AutoIt        expandcollapse  popup

#include-once

;#AutoIt3Wrapper_au3check_parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6

; #INDEX# =======================================================================================================================
; Title .........: _RecFileListToArray
; AutoIt Version : v3.3.1.1 or higher
; Language ......: English
; Description ...: Lists files andor folders in specified path with optional recursion to defined level and result sorting
; Note ..........:
; Author(s) .....: Melba23
; Remarks .......: - Modified Array.au3 functions - credit: Jos van der Zande, LazyCoder, Tylo, Ultima, SolidSnake and gcriaco
;                  - SRE patterns - credit: various forum members and Spiff59 in particular
;                  - Despite the name, this UDF is iterative, not recursive
; ===============================================================================================================================

; #CURRENT# =====================================================================================================================
; _RecFileListToArray: Lists files andor folders in a specified path with optional recursion to defined level and result sorting
; ===============================================================================================================================

; #INTERNAL_USE_ONLY#============================================================================================================
; _RFLTA_ListToMask ......; Convert include/exclude lists to SRE format
; _RFLTA_AddToList .......; Add element to list which is resized if necessary
; _RFLTA_AddFileLists ....; Add internal lists after resizing and optional sorting
; _RFLTA_FileListSearch ..; Search file match list for files associated with a folder
; _RFLTA_ArraySort .......; Wrapper for QuickSort function
; _RFLTA_QuickSort .......: Recursive array sort
; _RFLTA_ArrayConcatenate : Join 2 arrays
; ===============================================================================================================================

; #FUNCTION# ====================================================================================================================
; Name...........: _RecFileListToArray
; Description ...: Lists files andor folders in a specified path with optional recursion to defined level and result sorting
; Syntax.........: _RecFileListToArray($sPath[, $sInclude_List = "*"[, $iReturn = 0[, $iRecur = 0[, $iSort = 0[, $iReturnPath = 1[, $sExclude_List = ""[, $sExclude_List_Folder]]]]]]])
; Parameters ....: $sPath - Initial path used to generate filelist.  If path ends in  then folders will be returned with an ending 
;                  $sInclude_List - Optional: filter for included results (default "*"). Multiple filters must be separated by ";"
;                  $iReturn - Optional: specifies whether to return files, folders or both
;                  |$iReturn = 0 - Return both files and folders (Default)
;                                  If non-recursive Include/Exclude_List applies to files and folders
;                                  If recursive Include/Exclude_List applies to files only, all folders are searched unless excluded using $sExclude_List_Folder
;                  |$iReturn = 1 - Return files only    - Include/Exclude_List applies to files only, all folders searched if recursive
;                  |$iReturn = 2 - Return folders only  - Include/Exclude_List applies to folders only for searching and return
;                  $iRecur - Optional: specifies whether to search recursively in subfolders and to what level
;                  |$iRecur = 1 - Search in all subfolders (unlimited recursion)
;                  |$iRecur = 0 - Do not search in subfolders (Default)
;                  |$iRecur = Negative integer - Search in subfolders to specified depth
;                  $iSort - Optional: sort ordered in alphabetical and depth order
;                  |$iSort = 0 - Not sorted (Default)
;                  |$iSort = 1 - Sorted
;                  |$iSort = 2 - Sorted with faster algorithm (assumes files sorted within each folder - requires NTFS drive)
;                  $iReturnPath - Optional: specifies displayed path of results
;                  |$iReturnPath = 0 - File/folder name only
;                  |$iReturnPath = 1 - Relative to initial path (Default)
;                  |$iReturnPath = 2 - Full path included
;                  $sExclude_List - Optional: filter for excluded results (default ""). Multiple filters must be separated by ";"
;                  $sExclude_List_Folder - Optional: only used if $iReturn = 0 AND $iRecur = 1 to exclude folders matching the filter
; Requirement(s).: v3.3.1.1 or higher
; Return values .: Success: One-dimensional array made up as follows:
;                  |$array[0] = Number of FilesFolders returned
;                  |$array[1] = 1st FileFolder
;                  |$array[2] = 2nd FileFolder
;                  |...
;                  |$array[n] = nth FileFolder
;                   Failure: Null string and @error = 1 with @extended set as follows:
;                  |1 = Path not found or invalid
;                  |2 = Invalid $sInclude_List
;                  |3 = Invalid $iReturn
;                  |4 = Invalid $iRecur
;                  |5 = Invalid $iSort
;                  |6 = Invalid $iReturnPath
;                  |7 = Invalid $sExclude_List
;                  |8 = Invalid $sExclude_List_Folder
;                  |9 = No files/folders found
; Author ........: Melba23
; Remarks .......: Compatible with existing _FileListToArray syntax
; Related .......:
; Link ..........;
; Example .......; Yes
; ===============================================================================================================================
Func _RecFileListToArray($sInitialPath, $sInclude_List = "*", $iReturn = 0, $iRecur = 0, $iSort = 0, $iReturnPath = 1, $sExclude_List = "", $sExclude_List_Folder = "")

    Local $asReturnList[100] = [0], $asFileMatchList[100] = [0], $asRootFileMatchList[100] = [0], $asFolderMatchList[100] = [0], $asFolderSearchList[100] = [1]
    Local $sFolderSlash = "", $iMaxLevel, $sInclude_File_Mask, $sExclude_File_Mask, $sInclude_Folder_Mask = ".+", $sExclude_Folder_Mask = ":"
    Local $hSearch, $fFolder, $sRetPath = "", $sCurrentPath, $sName
    Local $asFolderFileSectionList[100][2] = [[0, 0]], $sFolderToFind, $iFileSectionStartIndex, $iFileSectionEndIndex

    ; Check for valid path
    If Not FileExists($sInitialPath) Then Return SetError(1, 1, "")
    ; Check if folders should have trailing  and ensure that initial path does have one
    If StringRight($sInitialPath, 1) = "" Then
        $sFolderSlash = ""
    Else
        $sInitialPath = $sInitialPath & ""
    EndIf
    ; Add path to folder search list
    $asFolderSearchList[1] = $sInitialPath

    ; Check for valid recur value
    If $iRecur > 1 Or Not IsInt($iRecur) Then Return SetError(1, 4, "")
    ; If required, determine  count for max recursive level setting
    If $iRecur < 0 Then
        StringReplace($sInitialPath, "", "", 2)
        $iMaxLevel = @extended - $iRecur
    EndIf

    ; Create Include mask for files
    If $sInclude_List = "*" Then
        $sInclude_File_Mask = ".+"
    Else
        If Not _RFLTA_ListToMask($sInclude_File_Mask, $sInclude_List) Then Return SetError(1, 2, "")
    EndIf
    ; Set Include mask for folders
    Switch $iReturn
        Case 0
            ; Folders affected by mask if not recursive
            Switch $iRecur
                Case 0
                    ; Folders match mask for compatibility
                    $sInclude_Folder_Mask = $sInclude_File_Mask
                ;Case Else
                    ; All folders match
            EndSwitch
        ;Case 1
            ; All folders match
        Case 2
            ; Folders affected by mask
            $sInclude_Folder_Mask = $sInclude_File_Mask
    EndSwitch

    ; Create Exclude List mask for files
    If $sExclude_List = "" Then
        $sExclude_File_Mask = ":" ; Set unmatchable mask
    Else
        If Not _RFLTA_ListToMask($sExclude_File_Mask, $sExclude_List) Then Return SetError(1, 7, "")
    EndIf
    ; Set Exclude mask for folders
    Switch $iReturn
        Case 0
            ; Folders affected by mask if not recursive
            Switch $iRecur
                Case 0
                    ; Folders match mask for compatibility
                    $sExclude_Folder_Mask = $sExclude_File_Mask
                Case Else
                    ; Exclude defined folders as set in extended
                    If $sExclude_List_Folder <> "" Then
                        If Not _RFLTA_ListToMask($sExclude_Folder_Mask, $sExclude_List_Folder) Then Return SetError(1, 8, "")
                    EndIf
            EndSwitch
        ;Case 1
            ; All folders match
        Case 2
            ; Folders affected by normal mask
            $sExclude_Folder_Mask = $sExclude_File_Mask
    EndSwitch

    ; Verify other parameters
    If Not ($iReturn = 0 Or $iReturn = 1 Or $iReturn = 2) Then Return SetError(1, 3, "")
    If Not ($iSort = 0 Or $iSort = 1 Or $iSort = 2) Then Return SetError(1, 5, "")
    If Not ($iReturnPath = 0 Or $iReturnPath = 1 Or $iReturnPath = 2) Then Return SetError(1, 6, "")

    ; Search within listed folders
    While $asFolderSearchList[0] > 0

        ; Set path to search
        $sCurrentPath = $asFolderSearchList[$asFolderSearchList[0]]

        ; Reduce folder search list count
        $asFolderSearchList[0] -= 1

        ; Determine return path to add to file/folder name
        Switch $iReturnPath
            ; Case 0 ; Name only
            ; Leave as ""
            Case 1 ;Relative to initial path
                $sRetPath = StringReplace($sCurrentPath, $sInitialPath, "")
            Case 2 ; Full path
                $sRetPath = $sCurrentPath
        EndSwitch

        ; Get search handle
        $hSearch = FileFindFirstFile($sCurrentPath & "*")
        ; If sorting files and folders with paths then store folder name and position of associated files in list
        If $iReturn = 0 And $iSort And $iReturnPath Then
            _RFLTA_AddToList($asFolderFileSectionList, $sRetPath, $asFileMatchList[0] + 1)
        EndIf
        ; If folder empty move to next in list
        If $hSearch = -1 Then
            ContinueLoop
        EndIf

        ; Search folder
        While 1
            $sName = FileFindNextFile($hSearch)
            ; Check for end of folder
            If @error Then
                ExitLoop
            EndIf
            ; Set subfolder flag - @extended set in 3.3.1.1 +
            $fFolder = @extended

            If $sName = "." Or $sName = ".." Then ContinueLoop ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

            ; If folder then check whether to add to search list
            If $fFolder Then
                Select
                    Case $iRecur < 0 ; Check recur depth
                        StringReplace($sCurrentPath, "", "", 0, 2)
                        If @extended < $iMaxLevel Then
                            ContinueCase ; Check if matched to masks
                        EndIf
                    Case $iRecur = 1 ; Full recur
                        If Not StringRegExp($sName, $sExclude_Folder_Mask) Then ; Add folder unless excluded
                                _RFLTA_AddToList($asFolderSearchList, $sCurrentPath & $sName & "")
                        EndIf
                    ; Case $iRecur = 0 ; Never add
                        ; Do nothing
                EndSelect
            EndIf

            If $iSort Then ; Save in relevant folders for later sorting

                If $fFolder Then
                    If StringRegExp($sName, $sInclude_Folder_Mask) And Not StringRegExp($sName, $sExclude_Folder_Mask) Then
                        _RFLTA_AddToList($asFolderMatchList, $sRetPath & $sName & $sFolderSlash)
                    EndIf
                Else
                    If StringRegExp($sName, $sInclude_File_Mask) And Not StringRegExp($sName, $sExclude_File_Mask) Then
                        ; Select required list for files
                        If $sCurrentPath = $sInitialPath Then
                            _RFLTA_AddToList($asRootFileMatchList, $sRetPath & $sName)
                        Else
                            _RFLTA_AddToList($asFileMatchList, $sRetPath & $sName)
                        EndIf
                    EndIf
                EndIf

            Else ; Save directly in return list
                If $fFolder Then
                    If $iReturn <> 1 And StringRegExp($sName, $sInclude_Folder_Mask) And Not StringRegExp($sName, $sExclude_Folder_Mask) Then
                        _RFLTA_AddToList($asReturnList, $sRetPath & $sName & $sFolderSlash)
                    EndIf
                Else
                    If $iReturn <> 2 And StringRegExp($sName, $sInclude_File_Mask) And Not StringRegExp($sName, $sExclude_File_Mask) Then
                        _RFLTA_AddToList($asReturnList, $sRetPath & $sName)
                    EndIf
                EndIf
            EndIf

        WEnd

        ; Close current search
        FileClose($hSearch)

    WEnd

    If $iSort Then

        ; Check if any file/folders have been added depending on required return
        Switch $iReturn
            Case 0 ; If no folders then number of files is immaterial
                If $asRootFileMatchList[0] = 0 And $asFolderMatchList[0] = 0 Then Return SetError(1, 9, "")
            Case 1
                If $asRootFileMatchList[0] = 0 And $asFileMatchList[0] = 0 Then Return SetError(1, 9, "")
            Case 2
                If $asFolderMatchList[0] = 0 Then Return SetError(1, 9, "")
        EndSwitch

        Switch $iReturn
            Case 2 ; Folders only
                ; Correctly size folder match list
                ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                ; Copy size folder match array
                $asReturnList = $asFolderMatchList
                ; Simple sort list
                _RFLTA_ArraySort($asReturnList)
            Case 1 ; Files only
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Simple sort combined file list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Combine sorted file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList, 1)
                EndIf
            Case 0 ; Both files and folders
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Set correct count for folder add
                    $asReturnList[0] += $asFolderMatchList[0]
                    ; Resize and add file match array
                    ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                    _RFLTA_ArrayConcatenate($asReturnList, $asFolderMatchList)
                    ; Simple sort final list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Size return list
                    Local $asReturnList[$asFileMatchList[0] + $asRootFileMatchList[0] + $asFolderMatchList[0] + 1]
                    $asReturnList[0] = $asFileMatchList[0] + $asRootFileMatchList[0] + $asFolderMatchList[0]
                    ; Sort root file list
                    _RFLTA_ArraySort($asRootFileMatchList, 1, $asRootFileMatchList[0])
                    ; Add the sorted root files at the top
                    For $i = 1 To $asRootFileMatchList[0]
                        $asReturnList[$i] = $asRootFileMatchList[$i]
                    Next
                    ; Set next insertion index
                    Local $iNextInsertionIndex = $asRootFileMatchList[0] + 1
                    ; Sort folder list
                    _RFLTA_ArraySort($asFolderMatchList, 1, $asFolderMatchList[0])
                    ; Work through folder list
                    For $i = 1 To $asFolderMatchList[0]
                        ; Format folder name for search
                        If $sFolderSlash Then
                            $sFolderToFind = $asFolderMatchList[$i]
                        Else
                            $sFolderToFind = $asFolderMatchList[$i] & ""
                        EndIf
                        ; Find folder in FolderFileSectionList
                        For $j = 1 To $asFolderFileSectionList[0][0]
                            If $sFolderToFind = $asFolderFileSectionList[$j][0] Then ExitLoop
                        Next
                        ; Set file list indexes
                        $iFileSectionStartIndex = $asFolderFileSectionList[$j][1]
                        If $j = $asFolderFileSectionList[0][0] Then
                            $iFileSectionEndIndex = $asFileMatchList[0]
                        Else
                            $iFileSectionEndIndex = $asFolderFileSectionList[$j + 1][1] - 1
                        EndIf
                        ; Sort files if required
                        If $iSort = 1 Then
                            _RFLTA_ArraySort($asFileMatchList, $iFileSectionStartIndex, $iFileSectionEndIndex)
                        EndIf
                        ; Add folder to return list
                        $asReturnList[$iNextInsertionIndex] = $asFolderMatchList[$i]
                        $iNextInsertionIndex += 1
                        ; Add files to return list
                        For $j = $iFileSectionStartIndex To $iFileSectionEndIndex
                            $asReturnList[$iNextInsertionIndex] = $asFileMatchList[$j]
                            $iNextInsertionIndex += 1
                        Next
                    Next
                EndIf
        EndSwitch

    Else ; No sort

        ; Check if any file/folders have been added
        If $asReturnList[0] = 0 Then Return SetError(1, 9, "")

        ; Remove any unused return list elements from last ReDim
        ReDim $asReturnList[$asReturnList[0] + 1]

    EndIf

    Return $asReturnList


EndFunc   ;==>_RecFileListToArray

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ListToMask
; Description ...: Convert include/exclude lists to SRE format
; Syntax ........: _RFLTA_ListToMask(ByRef $sMask, $sList)
; Parameters ....: $asMask - Include/Exclude mask to create
;                  $asList - Include/Exclude list to convert
; Return values .: Success: 1
;                  Failure: 0
; Author ........: SRE patterns developed from those posted by various forum members and Spiff59 in particular
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ListToMask(ByRef $sMask, $sList)

    ; Check for invalid characters within list
    If StringRegExp($sList, "|/|:|<|>||") Then Return 0
    ; Strip WS and insert | for ;
    $sList = StringReplace(StringStripWS(StringRegExpReplace($sList, "s*;s*", ";"), 3), ";", "|")
    ; Convert to SRE pattern
    $sList = StringReplace(StringReplace(StringRegExpReplace($sList, "[][$^.{}()+-]", "$0"), "?", "."), "*", ".*?")
    ; Add prefix and suffix
    $sMask =  "(?i)^(" & $sList & ")z"
    Return 1

EndFunc   ;==>_RFLTA_ListToMask

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddToList
; Description ...: Add element to [?] or [?][2] list which is resized if necessary
; Syntax ........: _RFLTA_AddToList(ByRef $asList, $vValue_0, [$vValue_1])
; Parameters ....: $aList - List to be added to
;                  $vValue_0 - Value to add (to [0] element in [?][2] array if $vValue_1 exists)
;                  $vValue_1 - Value to add to [1] element in [?][2] array (optional)
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddToList(ByRef $aList, $vValue_0, $vValue_1 = -1)

    If $vValue_1 = -1 Then ; [?] array
        ; Increase list count
        $aList[0] += 1
        ; Double list size if too small (fewer ReDim needed)
        If UBound($aList) <= $aList[0] Then ReDim $aList[UBound($aList) * 2]
        ; Add value
        $aList[$aList[0]] = $vValue_0
    Else ; [?][2] array
        $aList[0][0] += 1
        If UBound($aList) <= $aList[0][0] Then ReDim $aList[UBound($aList) * 2][2]
        $aList[$aList[0][0]][0] = $vValue_0
        $aList[$aList[0][0]][1] = $vValue_1
    EndIf

EndFunc   ;==>_RFLTA_AddToList

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddFileLists
; Description ...: Add internal lists after resizing and optional sorting
; Syntax ........: _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2[, $iSort = 0])
; Parameters ....: $asReturnList - Base list
;                  $asRootFileMatchList - First list to add
;                  $asFileMatchList - Second list to add
;                  $iSort - (Optional) Whether to sort lists before adding
;                  |$iSort = 0 (Default) No sort
;                  |$iSort = 1 Sort in descending alphabetical order
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2, $iSort = 0)

    ; Correctly size root file match array
    ReDim $asSource_1[$asSource_1[0] + 1]
    ; Simple sort root file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_1)
    ; Copy root file match array
    $asTarget = $asSource_1
    ; Add file match count
    $asTarget[0] += $asSource_2[0]
    ; Correctly size file match array
    ReDim $asSource_2[$asSource_2[0] + 1]
    ; Simple sort file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_2)
    ; Add file match array
    _RFLTA_ArrayConcatenate($asTarget, $asSource_2)

EndFunc   ;==>_RFLTA_AddFileLists

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_FileListSearch
; Description ...: Search file array for beginning and end indices of folder associated files
; Syntax ........: _RFLTA_FileListSearch(Const ByRef $avArray, $vValue)
; Parameters ....: $avArray - Array to search ($asFileMatchList)
;                  $vValue  - Value to search for (Folder name from $asFolderMatchList)
;                  $iIndex  - Index to begin search (search down from here - and then from here to top if not found)
;                  $sSlash  -  if folder names end in  - else empty string
; Return values .: Success: Array holding top and bottom indices of folder associated files
;                  Failure: Returns -1
; Author ........: Melba23
; Modified.......:
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_FileListSearch(Const ByRef $avArray, $vValue, $iIndex, $sSlash)

    Local $aRet[2]

    ; Add final  if required
    If Not $sSlash Then $vValue &= ""
    ; Start by getting top match - search down from start index
    For $i = $iIndex To $avArray[0]
        ; SRE gives path less filename
        If StringRegExpReplace($avArray[$i], "(^.*)(.*)", "1") = $vValue Then ExitLoop
    Next
    If $i > $avArray[0] Then
        ; No match found so look from start index upwards
        If $iIndex = $avArray[0] Then $iIndex -= 1
        For $i = $iIndex + 1 To 1 Step -1
            If StringRegExpReplace($avArray[$i], "(^.*)(.*)", "1") = $vValue Then ExitLoop
        Next
        ; If still no match - return " nothing found" for empty folder
        If $i = 0 Then Return SetError(1, 0, "")
        ; Set index of bottom file
        $aRet[1] = $i
        ; Now look for top match
        For $i = $aRet[1] To 1 Step -1
            If StringRegExpReplace($avArray[$i], "(^.*)(.*)", "1") <> $vValue Then ExitLoop
        Next
        ; Set top match
        $aRet[0] = $i + 1
    Else
        ; Set index of top associated file
        $aRet[0] = $i
        ; Now look for bottom match - find first file which does not match
        For $i = $aRet[0] To $avArray[0]
            If StringRegExpReplace($avArray[$i], "(^.*)(.*)", "1") <> $vValue Then ExitLoop
        Next
        ; Set bottom match
        $aRet[1] = $i - 1
    EndIf

    Return $aRet

EndFunc   ;==>_RFLTA_FileListSearch

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArraySort
; Description ...: Wrapper for QuickSort function
; Syntax ........: _RFLTA_ArraySort(ByRef $avArray)
; Parameters ....: $avArray - Array to sort
;                  $iStart  - Index to start sort
;                  $iEnd    - Index to end sort
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArraySort(ByRef $avArray, $iStart = 1, $iEnd = -99)

    If $iEnd = -99 Then $iEnd = UBound($avArray) - 1

    _RFLTA_QuickSort($avArray, $iStart, $iEnd)

EndFunc   ;==>_RFLTA_ArraySort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_QuickSort
; Description ...: Recursive array sort
; Syntax ........: _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)
; Parameters ....: $avArray - Array to sort in descending alphabetical order
;                  $iStart - Start index
;                  $iEnd - End index
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RFLTA_ArraySort
; ===============================================================================================================================
Func _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)

    Local $vTmp
    If ($iEnd - $iStart) < 15 Then
        Local $i, $j, $vCur
        For $i = $iStart + 1 To $iEnd
            $vTmp = $avArray[$i]
            If IsNumber($vTmp) Then
                For $j = $i - 1 To $iStart Step -1
                    $vCur = $avArray[$j]
                    If ($vTmp >= $vCur And IsNumber($vCur)) Or (Not IsNumber($vCur) And StringCompare($vTmp, $vCur) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $vCur
                Next
            Else
                For $j = $i - 1 To $iStart Step -1
                    If (StringCompare($vTmp, $avArray[$j]) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $avArray[$j]
                Next
            EndIf
            $avArray[$j + 1] = $vTmp
        Next
        Return
    EndIf
    Local $L = $iStart, $R = $iEnd, $vPivot = $avArray[Int(($iStart + $iEnd) / 2)], $fNum = IsNumber($vPivot)
    Do
        If $fNum Then
            While ($avArray[$L] < $vPivot And IsNumber($avArray[$L])) Or (Not IsNumber($avArray[$L]) And StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While ($avArray[$R] > $vPivot And IsNumber($avArray[$R])) Or (Not IsNumber($avArray[$R]) And StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        Else
            While (StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While (StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        EndIf
        If $L <= $R Then
            $vTmp = $avArray[$L]
            $avArray[$L] = $avArray[$R]
            $avArray[$R] = $vTmp
            $L += 1
            $R -= 1
        EndIf
    Until $L > $R
    _RFLTA_QuickSort($avArray, $iStart, $R)
    _RFLTA_QuickSort($avArray, $L, $iEnd)

EndFunc   ;==>_RFLTA_QuickSort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArrayConcatenate
; Description ...: Joins 2 arrays
; Syntax ........: _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)
; Parameters ....: $avArrayTarget - Base array
;                  $avArraySource - Array to add from element 1 onwards
; Return values .: None - array modified ByRef
; Author ........: Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)

    Local $iUBoundTarget = UBound($avArrayTarget) - 1, $iUBoundSource = UBound($avArraySource)
    ReDim $avArrayTarget[$iUBoundTarget + $iUBoundSource]
    For $i = 1 To $iUBoundSource - 1
        $avArrayTarget[$iUBoundTarget + $i] = $avArraySource[$i]
    Next

EndFunc   ;==>_RFLTA_ArrayConcatenate

If it works you at least have a workaround until we can get to bottom of the problem.

M23

[wraithdu]

On what OS a file system are the find functions return the . and .. folders? Does it matter if you have explorer set to hide/show hidden/system files?

[Melba23]

wraithdu,

I am using Vista x32. I show hidden files but hide protected operating system files.

M23

[z0iid]

the workaround - appears to work, no more loops. thanks!

during debugging - we had "show hidden files, including protected os files" - but on the end user computer, those settings were not enabled.

[Melba23]

z0iid,

Well at least you can get on with your transfer work now.

I would still like to know why you got those folders returned with the normal version though.

M23

[wraithdu]

Yeah, odd. I tried on my laptop (win7 x64) and did not get . or .. returned in a simple FileFindFirstFile/NextFile loop.