Create Index.

[Alek]

well, i needed this function for a script of mine but as i got it working properly i found it to be very fast,

so i hooked it up with a small script and it was much faster for searching for files then any other programs i have tryed.

it created a index of my whole hardrive (250gb) in less then a 30 sec.

but it was very cpu intesiv at first, so i added a option to make it it sleep for a ms once to reduce the cpu usage of it.

it still works great but uses alot more time creating the index.(used 5-10min on my harddrive)

Edit:

-Added Fileopen in write mode and now it takes about 5-10sec to index my whole harddrive

was kinda silly of me not adding that in the first script

-Added a better way of check if its a file or dir.

still it comes around with la little less then 30sec but the index file is 3 times larger, damn windows files , maybe make it skip the windows dir??

btw:

anyone know a better way of checking if a unknown is a path or file.

i use stringinstr and look for . and thats just stupid because you can use . in a folername.

expandcollapse popup

Func _Create_Index($S_File_Path, $S_Index_File = -1)
    
    ;check and see if the user has a custom index file
    If $S_Index_File = -1 Then
        
        ;if not the then set the index file to scriptdir\index.dat
        $S_Index_File = @ScriptDir & "\Index.dat"
    EndIf
    
    ;Delete Old Index file (If there are any)
    FileDelete($S_Index_File)
    
    ;Set the File path to the a new variable that we need.
    $S_Paths = $S_File_Path
    
    ;open the index file to speed up stuff.
    $File = FileOpen($S_Index_File,1)
    
    ;Enter a loop while $S_Paths is not empty
    ;if $S_Paths is empty then the script is done.
    While  $S_Paths
        
        ;Set the file paths to search in
        $S_File_Path = StringSplit($S_Paths,"|")
        
        ;Set $S_Paths to 1
        $S_Paths = ""
        
        ;ente a For loop to search in the diffrent filepaths
        For $x = 1 To $S_File_Path[0]
            
            ;see if the filepath is other then nothing.
            ;Could use:
            ;If Fileexists($S_FilePath[$x]) then
            ;but i the way it is atm is faster.
            If $S_File_Path[$x] <> "" Then
                
                ;Find the first file.
                $S_First_File = FileFindFirstFile($S_File_Path[$x] & "\*.*")
                
                ;Enter a loop to get all the files in the current file path
                While 1
                    ;Find the next File.
                    $S_Files = FileFindNextFile($S_First_File)
                    
                    ;If @error then we have found all the files or there arent any here.
                    If @error Then ExitLoop
                    
                    ;Check and see if its a file or a folder.
                    If StringInStr(FileGetAttrib($S_File_Path[$x] & "\" & $S_Files), "D") Then
                        
                        ;Else added the foler's Path into the path list.
                        $S_Paths &= $S_File_Path[$x] & "\" & $S_Files & "|"
                    Else
                        
                        ;if it is a file then write it down in the index.
                        FileWrite($File,$S_File_Path[$x] & "\" & $S_Files & "|")
                    EndIf
                WEnd
            EndIf
        Next
    Wend
EndFunc

Edited December 17, 2007 by Alek

[Michel Claveau]

Hi!

A very fast way, for index the content of some directories is... BATCH !

@echo off
del index.dat
echo BatchIndexed>index.dat
FOR %%i IN (%*) DO dir %%i /b /s >>index.dat

Save like "index.bat"

Usage: index dir1 dir2 dir3 ...

Examples:

index  C:\data\sources
index  C:\data  D:\sauve\data
index  "C:\Program Files"  D:\data  "D:\my backups"

Try it ; it's very faster!

Edited December 17, 2007 by Michel Claveau

[randallc]

Hi,

Try "_fileListToArrayNew" link in my signature for accurate lists.

Yours is faster if you know for certain that you have no folders which include dots; You have made me think I could add this as an option to my UDF for such circumstances......

If StringInStr($S_Files, ".") Then

However, my understanding is that this sort of listing is not for general use; a lot of people do use such directories with dots.

Best, randall

[randallc]

Hi!

A very fast way, for index the content of some directories is... BATCH !

@echo off
del index.dat
echo BatchIndexed>index.dat
FOR %%i IN (%*) DO dir %%i /b /s >>index.dat

Save like "index.bat"

Usage: index dir1 dir2 dir3 ...

Examples:

index  C:\data\sources
index  C:\data  D:\sauve\data
index  "C:\Program Files"  D:\data  "D:\my backups"

Try it ; it's very faster!

Hi,

I know you do good scripting, but I can't immediately get this to work!

1. Can you put it as a UDF in an autoit wrapper for ease of testing, as well as AutoIt Example script?

2. Does it pick up files with extended characters (I know the DOS redirect tends to garble them usually, but I am not sure from batch..))

Best, randall

[Alek]

Hi,

Try "_fileListToArrayNew" link in my signature for accurate lists.

Yours is faster if you know for certain that you have no folders which include dots; You have made me think I could add this as an option to my UDF for such circumstances......

If StringInStr($S_Files, ".") Then

However, my understanding is that this sort of listing is not for general use; a lot of people do use such directories with dots.

Best, randall

thanks for the replay as i added right befor you posting this,

stringinstr was a stupid way of doing it, im changin it to.

If StringInStr(FileGetAttrib($S_Files), "D") as in your script, never noticed the FileGetAtrib function

[jokke]

Just a hint, using FileWrite every time you find files would consume alot search time, add to an array then after your search has finished then write to file.

Heres some code i wrote a few days ago when i wanted to create a mp3 index tool, note this one might use alot time since it has file size verifier:

Usage:

#include <myscript.au3>
#include <array.au3>
$t = TimerInit()
$array = Search("C:","*.*",99999999,0)
ConsoleWrite("Searched C: for files inn: "&TimerDiff($t)&"ms."&@CRLF)
_arraydisplay($array,"files") ;This function may use alot time when alot files are found.

expandcollapse popup

#cs ----------------------------------------------------------------------------

 AutoIt Version: 3.2.10.0
 Author:         jokke

 Script Function:
    Folder searching / indexing.

#ce ----------------------------------------------------------------------------

#include <File.au3>

Func Search($root = "c:\" ,$search = "*.mp3", $max = 10000 , $min = 1000)
    Dim $files[1][2]; [x][path,file]
    Dim $filestmp
    Dim $folders[1][2]; [x][path,searched 0/1]
    Dim $folderstmp
    Dim $timer[2] = [" ",10000]; [timerint,timeout(ms)]
    Dim $f = 0
    Dim $s = 0
    Dim $searching = 1
    Dim $maxsize = $max * 1024
    Dim $minsize = $min * 1024

    While $searching
        
        $filestmp = _FileListToArray ( $root ,$search, 1 )
        If UBound($filestmp) > 0 Then
            $size = False
            For $x = 1 To $filestmp[0]
                $size = FileGetSize($root&"\"&$filestmp[$x])
                If $size > $minsize And $size < $maxsize Then
                    ReDim $files[$s+1][2]
                    $files[$s][0] = $root
                    $files[$s][1] = $filestmp[$x]
                    $s +=1
                EndIf
                Next
        EndIf
        
        $folderstmp = _FileListToArray ( $root ,"*", 2 )
        If UBound($folderstmp) > 0 Then
            For $x = 1 To $folderstmp[0]
                ReDim $folders[$f+1][2]
                $folders[$f][0] = $root&"\"&$folderstmp[$x]
                $folders[$f][1] = 0
                $f +=1
            Next
        EndIf
        
        If UBound($folders,1) > 0 Then
            $root = False
            $stop = False
            For $x = 0 To UBound($folders,1) -1
                If $folders[$x][1] = 0 And $stop = False Then
                    $root = $folders[$x][0]
                    $folders[$x][1]  = 1
                    $stop = True
                EndIf
            Next
            If $root = False Then
                $searching = 0
            EndIf
        EndIf

    WEnd

    Return $files
EndFunc

Edited December 18, 2007 by jokke

[WolfWorld]

I did make one it call indexSource right now it Not open source but later maybe

http://www.autoitscript.com/forum/index.php?showtopic=59557

[Michel Claveau]

Hi!

Sorry for long time to answer (I'am very very busy...)

Sorry, also, for my bad english.

Can you put it as a UDF in an autoit wrapper for ease of testing, as well as AutoIt Example script?

Perhaps this next code can be agree by you?

$listdir='C:\Temp "C:\Program Files" D:\data "D:\my backups" '
$r=cmdr("cmd /cecho off && FOR %i IN ("& $listdir &") DO dir %i /b /s")
MsgBox(0,"Résultat :",$r)

Func cmdr($cde)
    $result=""
    $pid = Run($cde,"",0,6)
    While 1
        $result &= StdoutRead($pid)
        If @error Then ExitLoop
    Wend
    return $result
EndFunc

If you can get an array for result, you can split $r.

And, with the same function, you can use this code, more readable, but less generic:

$r=cmdr('cmd /cdir "C:\Doc\MVP" D:\data "D:\my backups" C:\Temp /b /s')
MsgBox(0,"Résultat :",$r)

Edited December 18, 2007 by Michel Claveau

[randallc]

Hi,

Yes so DOS again is quicker, but gives garbled output on WinXP if folders/ or files have extended characters; her's a comparison of outputs;

[0]|30

[1]|C:\Programs\SearchEngine\91 50~Extra $arrItems1

[2]|C:\Programs\SearchEngine\91 50~Extra $arrItems1

[3]|C:\Programs\SearchEngine\91 50\ExIcons7.au3~Extra $arrItems1

[4]|C:\Programs\SearchEngine\91 50\searchMine.au3~Extra $arrItems1

[5]|C:\Programs\SearchEngine\91 50\Table.xls~Extra $arrItems1

[6]|C:\Programs\SearchEngine\91 50\_GUICtrlListView.au3~Extra $arrItems1

[7]|C:\Programs\SearchEngine\91 50\_GUILVSort7.au3~Extra $arrItems1

[8]|C:\Programs\SearchEngine\91 50\ a¡d123aEhKNRuzo??é???æ????????????????????????.shs~Extra $arrItems1

[9]|C:\Programs\SearchEngine\91 50\searchMine.au3~Extra $arrItems1

[10]|C:\Programs\SearchEngine\91 50\Table.xls~Extra $arrItems1

[11]|C:\Programs\SearchEngine\91 50\_GUICtrlListView.au3~Extra $arrItems1

[12]|C:\Programs\SearchEngine\BackUp\_2DArrayPracsoftFindvalid_old1.au3~Extra $arrItems1

[13]|~Extra $arrItems1

[14]|C:\Programs\SearchEngine\91 é50~Extra $arrItems2

[15]|C:\Programs\SearchEngine\91 é50\ExIcons7.au3~Extra $arrItems2

[16]|C:\Programs\SearchEngine\91 é50\searchMine.au3~Extra $arrItems2

[17]|C:\Programs\SearchEngine\91 é50\Table.xls~Extra $arrItems2

[18]|C:\Programs\SearchEngine\91 é50\_GUICtrlListView.au3~Extra $arrItems2

[19]|C:\Programs\SearchEngine\91 é50\_GUILVSort7.au3~Extra $arrItems2

[20]|C:\Programs\SearchEngine\91 é50\áâãäåæçèéêëìíîïðò123ăĔĥĶŇŘũźǒə΅ΘΪΫλμύБТгфіҜө׀ם׳حكٔٯٰځڒڣٴڴۅۖۧ.shs~Extra $arrItems2

[21]|C:\Programs\SearchEngine\91 éê50~Extra $arrItems2

[22]|C:\Programs\SearchEngine\91 éê50\searchMine.au3~Extra $arrItems2

[23]|C:\Programs\SearchEngine\91 éê50\Table.xls~Extra $arrItems2

[24]|C:\Programs\SearchEngine\91 éê50\_GUICtrlListView.au3~Extra $arrItems2

[25]|C:\Programs\SearchEngine\BackUp\BackUp\~$copy.txt~Extra $arrItems2

[26]|C:\Programs\SearchEngine\BackUp\_2DArrayPracsoftFindvalid_old1é.au3~Extra $arrItems2

[27]|C:\Programs\SearchEngine\BackUp\~$y mydocNew.doc~Extra $arrItems2

[28]|C:\Programs\SearchEngine\SearchPathPlugin.suo~Extra $arrItems2

[29]|C:\Programs\SearchEngine\~$test.doc~Extra $arrItems2

[30]|C:\Programs\SearchEngine\~$y mydoc.doc~Extra $arrItems2

Best, Randall

[PS How would you modify that command to get files only?...]

[Alek]

Hi!

Sorry for long time to answer (I'am very very busy...)

Sorry, also, for my bad english.

Perhaps this next code can be agree by you?

...code

hmm, its faster but on my computer (vista) it dosent all ways get the files, sometimes it gets dir's only even if there are files in that dir.