Sign in to follow this  
Followers 0
grasshopper3

Split One Large Search into multiple Smaller Searches

5 posts in this topic

I am scanning a dir that has over 2000 sub-dir looking for files that have modified dates less than 24 hours old. Is there a way to cut this large search into multiple searches that run simultaneously? Like 20 searches of 100 dir each? I found the UDF ChildPro and its examples, but I don't follow what is going on. I don't understand how to make it do what I want. Can someone help me with this?

Thanks

Share this post


Link to post
Share on other sites



Your harddrive can only do one thing at a time right? Unless you've got a RAID setup.

Fair enough. Well is there a faster way to do a dir search than FindFirstFile and FindNextFile?

If I had a RAID how would I do it?

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Fair enough. Well is there a faster way to do a dir search than FindFirstFile and FindNextFile?

If I had a RAID how would I do it?

A striped (or striped and mirrored) RAID array still services requests sequentially, one at a time.

They can be considerably faster as far as I/O throughput, but there is no multitasking going on.

Basically, while one drive is reading a sector of a file, the next drive (or drives) can be positioning it's read head at the correct track to read the next portion of the file. This mechanical portion of the read process, having a servo position the read head, is the most time-consuming.

When I switched my C: drive to a RAID 0 array (three 120GB drives) a few years ago, the decrease in boot time and the time to load programs was easily noticable.

Edited by Spiff59

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

You can try modifying your algorithm to improve performance. Your situation may not be a case where splitting up the task is a benefit.

You could try posting the current file search code you are using and maybe someone can suggest how to improve it so it's faster.

This is one of the search scripts I use. It does fairly well, I think. You can just modify the code to incorporate the checking for the modified date.

#include <Array.au3>
Opt("MustDeclareVars", 1)

Global $start = TimerInit()
Global $result = _Search("d:\", "*.jpg", -1, False)
Global $time = TimerDiff($start)

_ArrayDisplay($result, "Time = " & _ConvertTime($time))

;### SEARCH FUNCTION ###
;Returns array containing full path of matched files, with [0] containing the count of returned elements.
;
;PARAMETERS:
;
;$path        = start location of search
;
;$filter      = expression to use for file matching (e.g. *.txt)
;
;$depth       = folder depth to limit search
;               Default set to -1
;
;               Values:
;                   0  -> current folder only
;                   n  -> search up to n folders deep
;                   -1 -> search in all subfolders
;
;$directories = set to true if directories are to be included in the search results
;               Default set to True
;
;$files       = set to true if files are to be included in the search results
;               Default set to True
;
;RETURN VALUE:
;
;Success:   Array of files and folders matching the search parameter.
;           [0] contains the count of matched elements.
;           matched folders end with "\"
;
;Failure:   Returns 0.
;           Sets @error to:
;               1   -> $path does not exist.
;               2   -> No matches found.
;
Func _Search($path, $filter = "*", $depth = -1, $directories = True, $files = True)
    ;check directory exists
    If FileExists($path) Then
        ;add "\" to end of path value if needed
        If StringCompare(StringRight($path, 1), "\") <> 0 Then
            $path &= "\"
        EndIf
        Local $result = ""
        ;conduct search
        _SearchUtil($result, $path, $filter, $depth, $directories, $files)
        ;create return array
        If StringCompare(StringRight($result, 1), "|") = 0 Then
            $result = StringTrimRight($result, 1)
            $result = StringSplit($result, "|")
        Else
            $result = 0
            SetError(2, 0, 0)
        EndIf
        Return $result
    Else
        SetError(1, 0, 0)
    EndIf
EndFunc   ;==>_Search
Func _SearchUtil(ByRef $result, $path, $filter, $depth, $directories, $files)
    Local $search = FileFindFirstFile($path & $filter)
    Local $fname
    If $search <> -1 Then
        While 1
            $fname = FileFindNextFile($search)
            If @error = 1 Then
                ExitLoop
            EndIf
            ;skip processing if directory/file is not included in search results
            If @extended = 1 Then
                If Not $directories Then
                    ContinueLoop
                Else
                    $fname &= "\"
                EndIf
            Else
                If Not $files Then
                    ContinueLoop
                EndIf
            EndIf
            ;add file to results
            $result &= $path & $fname & "|"
        WEnd
        FileClose($search)
    EndIf
    ;process subdirectories if within depth parameter
    If $depth <> 0 Then
        $search = FileFindFirstFile($path & "*")
        While 1
            $fname = FileFindNextFile($search)
            If @error = 1 Then
                ExitLoop
            EndIf
            ;search subdirectory
            If @extended = 1 Then
                _SearchUtil($result, $path & $fname & "\", $filter, $depth - 1, $directories, $files)
            EndIf
        WEnd
        FileClose($search)
    EndIf
EndFunc   ;==>_SearchUtil
;### END SEARCH FUNCTION ###

Func _ConvertTime($millis)
    Local $days = Int($millis / 86400000)
    $millis -= $days * 86400000
    Local $hours = Int($millis / 3600000)
    $millis -= $hours * 3600000
    Local $minutes = Int($millis / 60000)
    $millis -= $minutes * 60000
    Local $seconds = Int($millis / 1000)
    $millis -= $seconds * 1000
    Local $result = ""
    If $days > 0 Then
        $result &= $days & "d:"
    EndIf
    If $hours > 0 Then
        $result &= StringFormat("%.2d", $hours) & "h:"
    EndIf
    If $minutes > 0 Then
        $result &= StringFormat("%.2d", $minutes) & "m:"
    EndIf
    If $seconds > 0 Then
        $result &= StringFormat("%.2d", $seconds)
    Else
        $result &= "00"
    EndIf
    $result &= "." & StringFormat("%.3d", $millis) & "s"
    Return $result
EndFunc   ;==>_ConvertTime
Edited by omikron48

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0