Loading data from a file.

[photonbuddy]

Hi All,

I am using this bit of code in one of my programs. Currently testing it on a datafile with ~100,000 records, and it's pretty quick (~7 seconds), but my PC is fairly fast compared to what a lot of people have, so any speed improvements would be great. Each record consists of 3 fields, a filename, a boolean option, and a number.

Func loaddata()
    Local $inline, $splits,$lc
    dim $aArray
    $dbhandle = FileOpen($dbname, 0)
    $aArray = StringSplit( StringStripCR(FileRead($dbhandle, FileGetSize($dbname))), @LF)
    FileClose($dbhandle)
    $count=$aArray[0]-2
    Dim $mfile[$count+1][3],$favs[500],$favcount=-1
    for $lc=0 to $count
        $splitst = StringSplit($aArray[$lc+1],"|")
        If $splitst[0] = 3 Then;==> Removing this test saved around .8 of a second.
            If $lc/2500 = Int($lc/2500) Then
                GUICtrlSetData($info1, "Loading Database ... " & $lc & " Files");==> Update the display every 2500 records
            EndIf
            $mfile[$lc][0] = $splitst[1]
            $mfile[$lc][1] = $splitst[2]
            If $mfile[$lc][1]="1" and $favcount<500 Then
                $favcount=$favcount+1
                $favs[$favcount]=$mfile[$lc][0]
            EndIf
            $mfile[$lc][2] = $splitst[3]
        EndIf
    Next
    dim $aArray=0
EndFunc  ;==>loaddata

Any help much appreciated.

[randallc]

Hi,

This is nearly twice as fast, but I note your favcount takes about 0.8 secs too? [Array2D.au3 in my sig uses vbs too]

Best, randall

expandcollapse popup

#include<array2D.au3>
Local $dbname1 = @ScriptDir & "\testdata.txt", $TotalTimerIntCheck
$timerstamp1 = TimerInit()
$ar_Answer1 = loaddata($dbname1)
ConsoleWrite("Time " & Round(TimerDiff($timerstamp1)) & " ms" & @LF)
$ar_Answer3 = $ar_Answer1
ReDim $ar_Answer3[399][UBound($ar_Answer3, 2) - 1]
;~ _ArrayDisplay("UBound($ar_Answer1)"&UBound($ar_Answer1)& ";Here are first 400; Time " & Round(TimerDiff($timerstamp1)) & " ms", 0)
$timerstamp1 = TimerInit()
$ar_Answer = loaddata2($dbname1)
ConsoleWrite("Time " & Round(TimerDiff($timerstamp1)) & " ms" & @LF)
ConsoleWrite("Time $TotalTimerIntCheck" & Round($TotalTimerIntCheck) & " ms" & @LF)
$ar_Answer2 = $ar_Answer
ReDim $ar_Answer2[399][UBound($ar_Answer2, 2) - 1]
_ArrayDisplay($ar_Answer2,"UBound($ar_Answer)"&UBound($ar_Answer)& ";Here are first 400; Time " & Round(TimerDiff($timerstamp1)) & " ms", 0)
Func loaddata2($dbname)
    local $aArray2D[1],$favs[500],$TotalTimerIntCheck ,$favcount = -1,$aArray = StringSplit(FileRead($dbname), @crLF,1)
    $aArray2D = _Array2DCreateFromArraySt ($aArray, 2);_Array2DCreateFromArraySt($ar1_Array_Strings, $i_Base = 0, $i_Transpose = 0, $s_Separator = "|")
    For $lc = 0 To UBound($aArray2D) - 1
        If $aArray2D[$lc][1] = "1" And $favcount < 500 Then
            $favcount += 1
            $favs[$favcount] = $aArray2D[$lc][0]
        EndIf
    Next
    Return $aArray2D
EndFunc   ;==>loaddata2
Func loaddata($dbname)
    Local $inline, $splits, $lc,$aArray
    $dbhandle = FileOpen($dbname, 0)
    $aArray = StringSplit(FileRead($dbname), @crLF,1)
;~  $aArray = StringSplit(StringStripCR(FileRead($dbhandle, FileGetSize($dbname))), @LF)
    FileClose($dbhandle)
    $count = $aArray[0] - 2
    Dim $mfile[$count + 1][3], $favs[500], $favcount = -1
    For $lc = 0 To $count
        $splitst = StringSplit($aArray[$lc + 1], "|")
        If $splitst[0] = 3 Then
;~          $timerstamp1 = TimerInit()
;~             If $lc/2500 = Int($lc/2500) Then;==> Removing this test saved around .8 of a second.
;~                 GUICtrlSetData($info1, "Loading Database ... " & $lc & " Files");==> Update the display every 2500 records
;~             EndIf
;~          $TotalTimerIntCheck+=TimerDiff($timerstamp1)
            $mfile[$lc][0] = $splitst[1]
            $mfile[$lc][1] = $splitst[2]
            If $mfile[$lc][1] = "1" And $favcount < 500 Then
                $favcount += 1
                $favs[$favcount] = $mfile[$lc][0]
            EndIf
            $mfile[$lc][2] = $splitst[3]
        EndIf
    Next
    Dim $aArray = 0
    Return $mfile
EndFunc   ;==>loaddata

;~ local $ar_CreateFile[100001],$dbname1=@ScriptDir&"\testdata.txt"
;~ for $a= 1 to 100000 ;create database
;~  $ar_CreateFile[$a]="c:\programs\searchengine\test.au3|0|"&$a
;~ Next
;~ _FileWriteFromArray($dbname1,$ar_CreateFile)oÝ÷ Øô¦z+æÍ÷jëh×6$aArray = StringSplit(FileRead($dbname), @crLF,1)
            $favcount += 1

Edited May 30, 2007 by randallc

[photonbuddy]

Changing to the Array2D functions certainly does speed things up. I think that's probably getting to a point where if it's taking too long, you need to buy a faster PC.

Thanks so much for being very helpful.