Sign in to follow this  
Followers 0
photonbuddy

Loading data from a file.

3 posts in this topic

Hi All,

I am using this bit of code in one of my programs. Currently testing it on a datafile with ~100,000 records, and it's pretty quick (~7 seconds), but my PC is fairly fast compared to what a lot of people have, so any speed improvements would be great. Each record consists of 3 fields, a filename, a boolean option, and a number.

Func loaddata()
    Local $inline, $splits,$lc
    dim $aArray
    $dbhandle = FileOpen($dbname, 0)
    $aArray = StringSplit( StringStripCR(FileRead($dbhandle, FileGetSize($dbname))), @LF)
    FileClose($dbhandle)
    $count=$aArray[0]-2
    Dim $mfile[$count+1][3],$favs[500],$favcount=-1
    for $lc=0 to $count
        $splitst = StringSplit($aArray[$lc+1],"|")
        If $splitst[0] = 3 Then;==> Removing this test saved around .8 of a second.
            If $lc/2500 = Int($lc/2500) Then
                GUICtrlSetData($info1, "Loading Database ... " & $lc & " Files");==> Update the display every 2500 records
            EndIf
            $mfile[$lc][0] = $splitst[1]
            $mfile[$lc][1] = $splitst[2]
            If $mfile[$lc][1]="1" and $favcount<500 Then
                $favcount=$favcount+1
                $favs[$favcount]=$mfile[$lc][0]
            EndIf
            $mfile[$lc][2] = $splitst[3]
        EndIf
    Next
    dim $aArray=0
EndFunc  ;==>loaddata

Any help much appreciated.

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Hi,

This is nearly twice as fast, but I note your favcount takes about 0.8 secs too? [Array2D.au3 in my sig uses vbs too]

Best, randall

#include<array2D.au3>
Local $dbname1 = @ScriptDir & "\testdata.txt", $TotalTimerIntCheck
$timerstamp1 = TimerInit()
$ar_Answer1 = loaddata($dbname1)
ConsoleWrite("Time " & Round(TimerDiff($timerstamp1)) & " ms" & @LF)
$ar_Answer3 = $ar_Answer1
ReDim $ar_Answer3[399][UBound($ar_Answer3, 2) - 1]
;~ _ArrayDisplay("UBound($ar_Answer1)"&UBound($ar_Answer1)& ";Here are first 400; Time " & Round(TimerDiff($timerstamp1)) & " ms", 0)
$timerstamp1 = TimerInit()
$ar_Answer = loaddata2($dbname1)
ConsoleWrite("Time " & Round(TimerDiff($timerstamp1)) & " ms" & @LF)
ConsoleWrite("Time $TotalTimerIntCheck" & Round($TotalTimerIntCheck) & " ms" & @LF)
$ar_Answer2 = $ar_Answer
ReDim $ar_Answer2[399][UBound($ar_Answer2, 2) - 1]
_ArrayDisplay($ar_Answer2,"UBound($ar_Answer)"&UBound($ar_Answer)& ";Here are first 400; Time " & Round(TimerDiff($timerstamp1)) & " ms", 0)
Func loaddata2($dbname)
    local $aArray2D[1],$favs[500],$TotalTimerIntCheck ,$favcount = -1,$aArray = StringSplit(FileRead($dbname), @crLF,1)
    $aArray2D = _Array2DCreateFromArraySt ($aArray, 2);_Array2DCreateFromArraySt($ar1_Array_Strings, $i_Base = 0, $i_Transpose = 0, $s_Separator = "|")
    For $lc = 0 To UBound($aArray2D) - 1
        If $aArray2D[$lc][1] = "1" And $favcount < 500 Then
            $favcount += 1
            $favs[$favcount] = $aArray2D[$lc][0]
        EndIf
    Next
    Return $aArray2D
EndFunc   ;==>loaddata2
Func loaddata($dbname)
    Local $inline, $splits, $lc,$aArray
    $dbhandle = FileOpen($dbname, 0)
    $aArray = StringSplit(FileRead($dbname), @crLF,1)
;~  $aArray = StringSplit(StringStripCR(FileRead($dbhandle, FileGetSize($dbname))), @LF)
    FileClose($dbhandle)
    $count = $aArray[0] - 2
    Dim $mfile[$count + 1][3], $favs[500], $favcount = -1
    For $lc = 0 To $count
        $splitst = StringSplit($aArray[$lc + 1], "|")
        If $splitst[0] = 3 Then
;~          $timerstamp1 = TimerInit()
;~             If $lc/2500 = Int($lc/2500) Then;==> Removing this test saved around .8 of a second.
;~                 GUICtrlSetData($info1, "Loading Database ... " & $lc & " Files");==> Update the display every 2500 records
;~             EndIf
;~          $TotalTimerIntCheck+=TimerDiff($timerstamp1)
            $mfile[$lc][0] = $splitst[1]
            $mfile[$lc][1] = $splitst[2]
            If $mfile[$lc][1] = "1" And $favcount < 500 Then
                $favcount += 1
                $favs[$favcount] = $mfile[$lc][0]
            EndIf
            $mfile[$lc][2] = $splitst[3]
        EndIf
    Next
    Dim $aArray = 0
    Return $mfile
EndFunc   ;==>loaddata

;~ local $ar_CreateFile[100001],$dbname1=@ScriptDir&"\testdata.txt"
;~ for $a= 1 to 100000 ;create database
;~  $ar_CreateFile[$a]="c:\programs\searchengine\test.au3|0|"&$a
;~ Next
;~ _FileWriteFromArray($dbname1,$ar_CreateFile)oÝ÷ Øô¦z+æÍ÷jëh×6$aArray = StringSplit(FileRead($dbname), @crLF,1)
            $favcount += 1
Edited by randallc

Share this post


Link to post
Share on other sites

Changing to the Array2D functions certainly does speed things up. I think that's probably getting to a point where if it's taking too long, you need to buy a faster PC.

Thanks so much for being very helpful.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0