Help me with my logic here...

[SpookMeister]

I am using Larry's _FileSearch function in a little script a wrote, and I am having some concerns about the limitations it has. I tested it in a smaller subset of data, but now that I have it running on my real data, I'm having doubts about it working.

Essentially, the part I am concerned about is that I am searching a directory tree that contains a whole LOT of files... like... just under 80 million total, broken up into about 8000 subdirectories.

The path of each file looks like:

X:\XX\XXXX\xxxxxxxxxxxxx.xxx

I make that out to be 28 characters per file (full path)

Here's where I need your help...

First, do I correctly understand how Larry's little function works? I believe he basically makes a long string out of all the full paths of the files, separated by an *, then uses the StringSplit function to turn that string into an array of all the files found.

Second, if the above is correct, and the FAQ is accurate in that a "Maximum string length: 2,147,483,647 characters", won't I run into an error of some kind before it completes the search?

If (File length + Seperator) * number of files = total length of string

(28 + 1) * 80,000,000 = 2,320,000,000

Here's the version of Larry's function that I am using:

CODE

expandcollapse popup

; _FileSearch( <Path and Mask>, <Option (0 - normal, 1- recursive)>)
; Returns array. Either Array of files (full path) where...
; Array[0] is number of files.
; Array[0] = 0 if nothing found.
;
; EXAMPLE USAGE
;--------------------------------------------
$a = _FileSearch("C:\surf\*.exe",0)
If $a[0] > 0 Then
    For $i = 1 to $a[0]
         MsgBox(4096,"",$a[$i])
    Next
EndIf
;--------------------------------------------

Func _FileSearch($szMask,$nOption)
    $szRoot = ""
    $hFile = 0
    $szBuffer = ""
    $szReturn = ""
    $szPathList = "*"
    Dim $aNULL[1]

    If Not StringInStr($szMask,"\") Then
         $szRoot = @SCRIPTDIR & "\"
    Else
         While StringInStr($szMask,"\")
              $szRoot = $szRoot & StringLeft($szMask,StringInStr($szMask,"\"))
              $szMask = StringTrimLeft($szMask,StringInStr($szMask,"\"))
         Wend
    EndIf
    If $nOption = 0 Then
         _FileSearchUtil($szRoot, $szMask, $szReturn)
    Else
         While 1
              $hFile = FileFindFirstFile($szRoot & "*.*")
              If $hFile >= 0 Then
                   $szBuffer = FileFindNextFile($hFile)
                   While Not @ERROR
                        If $szBuffer <> "." And $szBuffer <> ".." And _
                             StringInStr(FileGetAttrib($szRoot & $szBuffer),"D") Then _
                             $szPathList = $szPathList & $szRoot & $szBuffer & "*"
                        $szBuffer = FileFindNextFile($hFile)
                   Wend
                   FileClose($hFile)
              EndIf
              _FileSearchUtil($szRoot, $szMask, $szReturn)
              If $szPathList == "*" Then ExitLoop
              $szPathList = StringTrimLeft($szPathList,1)
              $szRoot = StringLeft($szPathList,StringInStr($szPathList,"*")-1) & "\"
              $szPathList = StringTrimLeft($szPathList,StringInStr($szPathList,"*")-1)
         Wend
    EndIf
    If $szReturn = "" Then
         $aNULL[0] = 0
         Return $aNULL
    Else
         Return StringSplit(StringTrimRight($szReturn,1),"*")
    EndIf
EndFunc

Func _FileSearchUtil(ByRef $ROOT, ByRef $MASK, ByRef $RETURN)
    $hFile = FileFindFirstFile($ROOT & $MASK)
    If $hFile >= 0 Then
         $szBuffer = FileFindNextFile($hFile)
         While Not @ERROR
              If $szBuffer <> "." And $szBuffer <> ".." Then _
                   $RETURN = $RETURN & $ROOT & $szBuffer & "*"
              $szBuffer = FileFindNextFile($hFile)
         Wend
         FileClose($hFile)
    EndIf
EndFunc

So... if I am right about my concerns here.. how else might I go about my goal, which by the way is to perform a file by file size comparison to verify that I have made a good copy of the original files into another location (the files have been copied already, I'm just validating that the copy went well)

TIA

-Spook

Edited May 11, 2007 by SpookMeister

[Zedna]

Place your code directly into copy of _FileSearch().

So you will make your desired action with each file inside this function and needn't whole buffer/StringSplit/array of all files.