RaySS

FileFindNextFile fails to detect final file in subdirectory

5 posts in this topic

1. The  FileFindNextFile statement does not return 1 when no more files are found to match the search criterion therefore, the inner While loop runs endlessly.

2. It takes 105 seconds to search for matches in a subdirectory that contains 10,000 files. I need to process as many as ten such subdirectories in every run. I am operating a lightly loaded 8-core processor 4GHz 64-bit Win7 Pro PC.

Here's the code:

;For Clif: Select Subdirectories, search file name patterns, sort good subdirectory names 
#include <MsgBoxConstants.au3>
#include <File.au3>
#include <WinAPIFiles.au3>
#include <Array.au3>

Local $sTopDirectory = "E:\Clif\DoublestarSAVED\", $sSubdirectoryName, $sMaybeHit, $iHits = 0
Local $i0000Found
$aList = _FileListToArray($sTopDirectory, Default, $FLTA_FOLDERS) ;How many subdirectories in top directory?

;Allocate space for number of subdirectories.
Local $aConCat[$aList[0]], $aDateTime[$aList[0] + 1], $aFiles[$aList[0] + 1], $aComment[$aList[0] + 1]
If IsArray($aList) Then
    For $a = 1 To $aList[0]

        $sSubdirectoryName = $sTopDirectory & $aList[$a]
        ConsoleWrite(@CRLF & "Subdirectory Name: " & $sSubdirectoryName & @CRLF)
        ConsoleWrite("File pattern: " & $sSubdirectoryName & "\*.fit" & @CRLF)
        $sMaybeHit = "-1"
        $i0000Found = 0

        $hSearch = FileFindFirstFile($sSubdirectoryName & "\" & "*.fit")
        While 1
            $sFileName = FileFindNextFile($hSearch)
            While @error <> 1

                ;process until no more files in the subdirectory or if "for lucky" is in subdirectory name
                If @error Or StringInStr($aList[$a], "for lucky") Then ExitLoop
                $sMaybeHit = $aList[$a] & "\" & $sFileName
                ConsoleWrite($aList[$a] & "\" & $sFileName & " Non-Lucky holds at least one .fit file" & @CRLF)

                ConsoleWrite($aList[$a] & "   good hit" & @CRLF)
                Local $aSize = DirGetSize($sTopDirectory & $aList[$a], 1) ; extended mode, get file count
                $aFiles[$a] = $aSize[1]
                ConsoleWrite("Files: " & $aSize[1] & @CRLF) ; show number of files in directory
                Local $aTime = FileGetTime($sTopDirectory & $aList[$a], $FT_CREATED, $FT_STRING) ;get date&time of creation
                $aDateTime[$a] = $aTime
                ConsoleWrite($sTopDirectory & $aList[$a] & @CRLF)
                ConsoleWrite("  Date-Time: " & $aDateTime[$a] & @CRLF)
                $aConCat[$a - 1] = $aDateTime[$a] & "|" & $aList[$a] & "|" & $aFiles[$a]
                ConsoleWrite("Concatonated: " & $aConCat[$a - 1] & @CRLF)

                If StringInStr($sFileName, "_0000.fit") Then $i0000Found = 1

                Call("AppendToCandidates")
                $iHits = $iHits + 1

                ExitLoop
            WEnd
        WEnd
        FileClose($hSearch)
        If $i0000Found = 0 Then Call("AddComment", $sSubdirectoryName, " folder contains at least one .fit file, but no *_0000.fit file")

    Next
EndIf
_ArrayDisplay($aConCat, "BEFORE")
_ArraySort($aConCat, 0)
_ArrayDisplay($aConCat, "AFTER")
ConsoleWrite(@CRLF & "array size is: " & $aList[0] & " Hit count: " & $iHits & @CRLF)
$iQuant = $aList[0]

Func AppendToCandidates()
    ConsoleWrite("Appending to candidates" & @CRLF & @CRLF)
EndFunc   ;==>AppendToCandidates

Func AddComment($sSubdirectoryName, $aComment)
    ConsoleWrite($sSubdirectoryName & " " & $aComment & @CRLF)
EndFunc   ;==>AddComment

The console displays 10,000 groups of debug output corresponding to 10,000 .fit files in one subdirectory, but then it hangs in an endless loop.

The criteria for this job have changed since my posting in a different topic. It is now necessary to search for three file name patterns in each subdirectory. The code above searches for this file pattern: "*_0000.fit". Future coding will also search for "*_1999.fit" and "*-9999.fit". I was planning to search for all three patterns in a single pass on each subdirectory using AutoIT, but I noticed that the command dir *_9999.fit entered at the command prompt executes nearly instantaneously. Maybe the solution would be to make three separate CMD calls from AutoIT for each subdirectory. What's the coding for that and how do I detect successful hits on the dir operation?

Can anyone offer a faster way to search for these patterns in multiple subdirectories containing 10,000 files each? Please also correct the logic to eliminate the endless loop.

Thank you.

RaySS

 

Share this post


Link to post
Share on other sites



RaySS,

Why do you insist on using FindFirst/NextFile when we have perfectly good functions (_FilListToArray/_FileListToArrayRec) which make coding so much easier? The latter of those functions will even let you search for multiple patterns within a single pass.

This appears to do what you require when I test with dummy files in my system:

#include <File.au3>
#include <Array.au3>

Local $sTopDirectory = "E:\Clif\DoublestarSAVED\"
Global $aFilesFound[1] = [0]

$aList = _FileListToArray($sTopDirectory, Default, $FLTA_FOLDERS) ;How many subdirectories in top directory?

;Allocate space for number of subdirectories.
Local $aConCat[$aList[0]], $aDateTime[$aList[0] + 1], $aFiles[$aList[0] + 1], $aComment[$aList[0] + 1]

If IsArray($aList) Then
    For $a = 1 To $aList[0]

        If StringInStr($aList[$a], "for lucky") Then ContinueLoop

        $sSubdirectoryName = $sTopDirectory & $aList[$a]

        ConsoleWrite(@CRLF & "Subdirectory Name: " & $sSubdirectoryName & @CRLF)

        $aFound = _FileListToArrayRec($sSubdirectoryName, "*_0000.fit;*_1999.fit;*-9999.fit", $FLTA_FILES)  ;

        ;_ArrayDisplay($aFound, $aList[$a], Default, 8) ; Should you want to check

        If IsArray($aFound) Then

            ; Increase size of array
            $aFilesFound[0] += 1
            ReDim $aFilesFound[$aFilesFound[0] + 1]

            ; Get time for each folder and add to array
            $aFilesFound[$aFilesFound[0]] = FileGetTime($sTopDirectory & $aList[$a], $FT_CREATED, $FT_STRING) & "|" & $aList[$a] & "|" & $aFound[0]

        EndIf

    Next

EndIf

_ArrayDisplay($aFilesFound, "Final", Default, 8)

M23

1 person likes this

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Hi M23,

Why do you insist on using FindFirst/NextFile when we have perfectly good functions (_FilListToArray/_FileListToArrayRec) which make coding so much easier?

I followed the example given to me in my previous topic mainly because the searched-for file name pattern in FileFindNextFile was satisfied on the first hit. The second reason was that I was curious about the concept of file handles. I'd heard the term but never used or understood what they are.

The requirements of the job have changed since then. I now need to search for three separate file patterns. Your solution above does that nicely and it is super quick. I see no discernible difference in searching through 10,000 files in a subdirectory versus searching through 18 files  --  less than a second in both cases.

Your solution also introduced to me the ContinueLoop keyword. I had previously read through all the keywords in Help, but didn't remember that one. It certainly makes the logic easier for me to follow.

You have been a tremendous help to me. Thank you very much!

RaySS

 

Share this post


Link to post
Share on other sites

RaySS,

I was curious about the concept of file handles

These are a form of "shorthand" to identify files - think of them as the indices of an array in which Windows holds all the necessary information about a file when the file is "opened" so that it can all be very quickly retrieved when needed later.  If you carry out file operations within a loop, using a file name each time forces Windows to go through the whole process on each pass whereas using a handle makes the whole thing much faster - of course you need to "close" the handle when you have finished with it or the array can become full preventing access to further files. A very basic explanation, but one which I hope suffices for the moment.

Anyway, glad I could help - have a good Xmas.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now