FileFindNextFile fails to detect final file in subdirectory

[RaySS]

1. The  FileFindNextFile statement does not return 1 when no more files are found to match the search criterion therefore, the inner While loop runs endlessly.

2. It takes 105 seconds to search for matches in a subdirectory that contains 10,000 files. I need to process as many as ten such subdirectories in every run. I am operating a lightly loaded 8-core processor 4GHz 64-bit Win7 Pro PC.

Here's the code:

expandcollapse popup

;For Clif: Select Subdirectories, search file name patterns, sort good subdirectory names 
#include <MsgBoxConstants.au3>
#include <File.au3>
#include <WinAPIFiles.au3>
#include <Array.au3>

Local $sTopDirectory = "E:\Clif\DoublestarSAVED\", $sSubdirectoryName, $sMaybeHit, $iHits = 0
Local $i0000Found
$aList = _FileListToArray($sTopDirectory, Default, $FLTA_FOLDERS) ;How many subdirectories in top directory?

;Allocate space for number of subdirectories.
Local $aConCat[$aList[0]], $aDateTime[$aList[0] + 1], $aFiles[$aList[0] + 1], $aComment[$aList[0] + 1]
If IsArray($aList) Then
    For $a = 1 To $aList[0]

        $sSubdirectoryName = $sTopDirectory & $aList[$a]
        ConsoleWrite(@CRLF & "Subdirectory Name: " & $sSubdirectoryName & @CRLF)
        ConsoleWrite("File pattern: " & $sSubdirectoryName & "\*.fit" & @CRLF)
        $sMaybeHit = "-1"
        $i0000Found = 0

        $hSearch = FileFindFirstFile($sSubdirectoryName & "\" & "*.fit")
        While 1
            $sFileName = FileFindNextFile($hSearch)
            While @error <> 1

                ;process until no more files in the subdirectory or if "for lucky" is in subdirectory name
                If @error Or StringInStr($aList[$a], "for lucky") Then ExitLoop
                $sMaybeHit = $aList[$a] & "\" & $sFileName
                ConsoleWrite($aList[$a] & "\" & $sFileName & " Non-Lucky holds at least one .fit file" & @CRLF)

                ConsoleWrite($aList[$a] & "   good hit" & @CRLF)
                Local $aSize = DirGetSize($sTopDirectory & $aList[$a], 1) ; extended mode, get file count
                $aFiles[$a] = $aSize[1]
                ConsoleWrite("Files: " & $aSize[1] & @CRLF) ; show number of files in directory
                Local $aTime = FileGetTime($sTopDirectory & $aList[$a], $FT_CREATED, $FT_STRING) ;get date&time of creation
                $aDateTime[$a] = $aTime
                ConsoleWrite($sTopDirectory & $aList[$a] & @CRLF)
                ConsoleWrite("  Date-Time: " & $aDateTime[$a] & @CRLF)
                $aConCat[$a - 1] = $aDateTime[$a] & "|" & $aList[$a] & "|" & $aFiles[$a]
                ConsoleWrite("Concatonated: " & $aConCat[$a - 1] & @CRLF)

                If StringInStr($sFileName, "_0000.fit") Then $i0000Found = 1

                Call("AppendToCandidates")
                $iHits = $iHits + 1

                ExitLoop
            WEnd
        WEnd
        FileClose($hSearch)
        If $i0000Found = 0 Then Call("AddComment", $sSubdirectoryName, " folder contains at least one .fit file, but no *_0000.fit file")

    Next
EndIf
_ArrayDisplay($aConCat, "BEFORE")
_ArraySort($aConCat, 0)
_ArrayDisplay($aConCat, "AFTER")
ConsoleWrite(@CRLF & "array size is: " & $aList[0] & " Hit count: " & $iHits & @CRLF)
$iQuant = $aList[0]

Func AppendToCandidates()
    ConsoleWrite("Appending to candidates" & @CRLF & @CRLF)
EndFunc   ;==>AppendToCandidates

Func AddComment($sSubdirectoryName, $aComment)
    ConsoleWrite($sSubdirectoryName & " " & $aComment & @CRLF)
EndFunc   ;==>AddComment

The console displays 10,000 groups of debug output corresponding to 10,000 .fit files in one subdirectory, but then it hangs in an endless loop.

The criteria for this job have changed since my posting in a different topic. It is now necessary to search for three file name patterns in each subdirectory. The code above searches for this file pattern: "*_0000.fit". Future coding will also search for "*_1999.fit" and "*-9999.fit". I was planning to search for all three patterns in a single pass on each subdirectory using AutoIT, but I noticed that the command dir *_9999.fit entered at the command prompt executes nearly instantaneously. Maybe the solution would be to make three separate CMD calls from AutoIT for each subdirectory. What's the coding for that and how do I detect successful hits on the dir operation?

Can anyone offer a faster way to search for these patterns in multiple subdirectories containing 10,000 files each? Please also correct the logic to eliminate the endless loop.

Thank you.

RaySS

 

[Melba23]

RaySS,

Why do you insist on using FindFirst/NextFile when we have perfectly good functions (_FilListToArray/_FileListToArrayRec) which make coding so much easier? The latter of those functions will even let you search for multiple patterns within a single pass.

This appears to do what you require when I test with dummy files in my system:

expandcollapse popup

#include <File.au3>
#include <Array.au3>

Local $sTopDirectory = "E:\Clif\DoublestarSAVED\"
Global $aFilesFound[1] = [0]

$aList = _FileListToArray($sTopDirectory, Default, $FLTA_FOLDERS) ;How many subdirectories in top directory?

;Allocate space for number of subdirectories.
Local $aConCat[$aList[0]], $aDateTime[$aList[0] + 1], $aFiles[$aList[0] + 1], $aComment[$aList[0] + 1]

If IsArray($aList) Then
    For $a = 1 To $aList[0]

        If StringInStr($aList[$a], "for lucky") Then ContinueLoop

        $sSubdirectoryName = $sTopDirectory & $aList[$a]

        ConsoleWrite(@CRLF & "Subdirectory Name: " & $sSubdirectoryName & @CRLF)

        $aFound = _FileListToArrayRec($sSubdirectoryName, "*_0000.fit;*_1999.fit;*-9999.fit", $FLTA_FILES)  ;

        ;_ArrayDisplay($aFound, $aList[$a], Default, 8) ; Should you want to check

        If IsArray($aFound) Then

            ; Increase size of array
            $aFilesFound[0] += 1
            ReDim $aFilesFound[$aFilesFound[0] + 1]

            ; Get time for each folder and add to array
            $aFilesFound[$aFilesFound[0]] = FileGetTime($sTopDirectory & $aList[$a], $FT_CREATED, $FT_STRING) & "|" & $aList[$a] & "|" & $aFound[0]

        EndIf

    Next

EndIf

_ArrayDisplay($aFilesFound, "Final", Default, 8)

M23

[RaySS]

Hi M23,

Why do you insist on using FindFirst/NextFile when we have perfectly good functions (_FilListToArray/_FileListToArrayRec) which make coding so much easier?

I followed the example given to me in my previous topic mainly because the searched-for file name pattern in FileFindNextFile was satisfied on the first hit. The second reason was that I was curious about the concept of file handles. I'd heard the term but never used or understood what they are.

The requirements of the job have changed since then. I now need to search for three separate file patterns. Your solution above does that nicely and it is super quick. I see no discernible difference in searching through 10,000 files in a subdirectory versus searching through 18 files  --  less than a second in both cases.

Your solution also introduced to me the ContinueLoop keyword. I had previously read through all the keywords in Help, but didn't remember that one. It certainly makes the logic easier for me to follow.

You have been a tremendous help to me. Thank you very much!

RaySS

 

[Melba23]

RaySS,

I was curious about the concept of file handles

These are a form of "shorthand" to identify files - think of them as the indices of an array in which Windows holds all the necessary information about a file when the file is "opened" so that it can all be very quickly retrieved when needed later.  If you carry out file operations within a loop, using a file name each time forces Windows to go through the whole process on each pass whereas using a handle makes the whole thing much faster - of course you need to "close" the handle when you have finished with it or the array can become full preventing access to further files. A very basic explanation, but one which I hope suffices for the moment.

Anyway, glad I could help - have a good Xmas.

M23

[RaySS]

Hi M23,

A very basic explanation, but one which I hope suffices for the moment.

That's definitely more helpful than the brief treatment of the subject at https://msdn.microsoft.com/en-us/library/windows/desktop/aa364225(v=vs.85).aspx

 

...have a good Xmas.

Our kids are home for Christmas. I'm old too, so their presence does make it a good Christmas.

Best regards,

RaySS