Sign in to follow this  
Followers 0
mohdumar

speed up INIREAD

9 posts in this topic

My script reads values from 50000 keys from a .ini file (in a loop) and compares it with a user-entered value, then displays the value of the matching key to the user.

Func READINI()
Local $startpos = 0
Local $loop = 50000
For $i = $startpos To $loop Step 1
Local $var = IniRead("database.ini", "db", $i, "Not Found")
If $var = $some_user_entered_value Then  ; equals it
MsgBox(4096, "Found", $var & " was found at " & $i) ;was found!
ExitLoop 1 ;no use for further compare
EndIf
GUICtrlSetData($progress_bar, $i / $loop * 100) ; set progress bar value
Next
GUICtrlSetData($progress_bar, 0)  ; set it back to zero
EndFunc

Is there anyway to speed it up? does high process priority work?

I also tried from this forum but they were slower than builtin disk I/O functions........

Share this post


Link to post
Share on other sites



I don't think ini is very practical for speed with 50,000 records, I suggest you think about using a dadabase

or maybe even xml.

I'm not very clever at all with either, so cant help much other to say, they will both improve with regards

to speed you get now with ini files.

1 person likes this

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

Switch to an SQLite database, it's a perfect fit for such simple application. Start with SQLite doc found in helpfile then chime in again if you're stuck.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

The fact that you use an .ini file leads me to believe you want a string key for a value. In that case, if 50000 is really the outer limit, you may find Scripting.Dictionary is sufficient. Not as robust as a database but for quick and dirty it's a lot easier to use. The data file could be a flat file alternating key value lines, if the keys and data can be expressed as a single line of text.

You could read the entire data file with _FileReadToArray(). Test the number of items read > 0 and not odd( with BitAND($array[0],1) ). Since these are key/data pairs an odd number of lines would indicate and error.

50000 may be stretching it a bit. I'm not sure if you can find any docs how Scripting.Dictionary relates to system memory or whatnot. MS seems to be kind of mum about it. I just ran a test on a 2 GB system, inside SciTE, loading 50000 lines of dummy data with a stringized number as key and it didn't balk.

Depends on what you are writing. If it's for money transfer between banks then you are better off using an engine. But if it's for a quickie user utility AssocArray may be enough.

These 2 functions are all you need. Also when getting an item using the key make sure to test with Exists method. As in If $array.Exists("key") then blah blah or else if you read with a key that's not there, the dictionary will create one with empty string as data. My AssocArray uses string key with case insensitive compare. But of course you can just write your own wrapper with a different CompareMode setting.

;use Scripting.Dictionary object for simple associative arrays
Func _AssocArray()
    Local $aArray = ObjCreate("Scripting.Dictionary")
    If @error Then
        Return SetError(1, 0, 0)
    EndIf
    $aArray.CompareMode = 1
    Return $aArray
EndFunc   ;==>_AssocArray

Func _AssocArrayDestroy(ByRef $aArray)
    If Not IsObj($aArray) Then
        Return False
    EndIf
    $aArray.RemoveAll()
    $aArray = 0
    Return True
EndFunc   ;==>_AssocArrayDestroy

Here's a simple example for creating an array of colors with the names as key.

Func _ColorList()
    Local $colorList = _AssocArray()
    If @error Then
        Return SetError(1, 0, 0)
    EndIf
    $colorList("AntiqueWhite") = 0xFAEBD7
    $colorList("Black") = 0x000000
    $colorList("Blue") = 0x0000FF
    $colorList("Brown") = 0xA52A2A
    $colorList("CadetBlue") = 0x5F9EA0
    $colorList("Chocolate") = 0xD2691E
    $colorList("Coral") = 0xFF7F50
    $colorList("CornflowerBlue") = 0x6495ED
    $colorList("DarkBlue") = 0x00008B
    $colorList("DarkCyan") = 0x008B8B
    $colorList("DodgerBlue") = 0x1E90FF
    $colorList("ForestGreen") = 0x228B22
    $colorList("Gold") = 0xFFD700
    $colorList("Gray") = 0x808080
    $colorList("Green") = 0x008000
    $colorList("HotPink") = 0xFF69B4
    $colorList("LightGreen") = 0x90EE90
    $colorList("LightPink") = 0xFFB6C1
    $colorList("LightSeaGreen") = 0x20B2AA
    $colorList("Lime") = 0x00FF00
    $colorList("Magenta") = 0xFF00FF
    $colorList("Maroon") = 0xB03060
    $colorList("MediumTurquoise") = 0x48D1CC
    $colorList("MediumVioletRed") = 0xC71585
    $colorList("MistyRose") = 0xFFE4E1
    $colorList("Navy") = 0x000080
    $colorList("Olive") = 0x808000
    $colorList("Orchid") = 0xDA70D6
    $colorList("PaleGreen") = 0x98FB98
    $colorList("PaleVioletRed") = 0xDB7093
    $colorList("Pink") = 0xFFC0CB
    $colorList("Plum") = 0xDDA0DD
    $colorList("Purple") = 0x800080
    $colorList("Red") = 0xFF0000
    $colorList("RoyalBlue") = 0x4169E1
    $colorList("Sienna") = 0x00A0522D
    $colorList("Silver") = 0xC0C0C0
    $colorList("SkyBlue") = 0x87CEEB
    $colorList("SteelBlue") = 0x004682B4
    $colorList("Tan") = 0xD2B48C
    $colorList("Teal") = 0x008080
    $colorList("Violet") = 0xEE82EE
    $colorList("Wheat") = 0xF5DEB3
    $colorList("White") = 0xFFFFFF
    $colorList("Yellow") = 0xFFFF00
    Return $colorList
EndFunc   ;==>_ColorList

Func _ColorListDestroy(ByRef $cList)
    Return _AssocArrayDestroy($cList)
EndFunc   ;==>_ColorListDestroy
Edited by MilesAhead

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Here's an "old-fashioned" way of loading the data.

You could do an _ArraySearch() and limit the start and end lines by using values from the $aSectionIndex[] array.

It has the advantage of being simple with low overhead and infinitely faster than 50,000 separate I/O opens, reads, and closes (via IniRead). It would not be as fast as other suggestions that have been made.

#include <Array.au3>
#include <File.au3>

Global $aRawfile, $iSections = 0, $x = 0
_FileReadToArray(@ScriptDir & "test.ini", $aRawfile)
Global $aSectionIndex[$aRawfile[0] + 1][3] ; section name, start line, end line
Global $aFile[$aRawfile[0] + 1][2] ; section or key name, key value (optional)

For $y = 1 to $aRawfile[0]
  If $aRawfile[$y] Then ; ignore blanks
    $x += 1
    If StringLeft($aRawfile[$y], 1) = "[" Then ; section header
     If $iSections > 0 Then $aSectionIndex[$iSections][2] = $x - 1
       $aFile[$x][0] = $aRawfile[$y]
       $iSections += 1
       $aSectionIndex[$iSections][0] = $aRawfile[$y]
       $aSectionIndex[$iSections][1] = $x + 1
    Else
       $temp = StringSplit($aRawfile[$y], "=")
       $aFile[$x][0] = $temp[1]
       $aFile[$x][1] = $temp[2]
    EndIf
  EndIf
Next
$aSectionIndex[$iSections][2] = $x
ReDim $aSectionIndex[$iSections + 1][3]
$aSectionIndex[0][0] = $iSections
ReDim $aFile[$x + 1][2]
$aFile[0][0] = $x

_ArrayDisplay($aRawfile, "Rawfile")
_ArrayDisplay($aSectionIndex, "Section Index")
_ArrayDisplay($aFile)

This assumes a "clean" ini file and has no edits to discard comments, strip unwanted whitespace, or handle keys with no value.

typo

Edited by Spiff59

Share this post


Link to post
Share on other sites

@MilesAhead

Just for information. I've used scriptingDictionary without any problems with 4 million key data pairs.


"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

If it will definitely be read-only data, then you may perhaps use a scripting dictionary, provided your application copes with it. But if ever you have to update it in a multi-process use case, I cast strong doubts about robustness. Also a DB will allow for partial search key, etc.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

@MilesAhead

Just for information. I've used scriptingDictionary without any problems with 4 million key data pairs.

Thanks for the info. I don't know what the OP's app is exactly. But if it's just key/data pairs I don't see the need for an engine unless there's some low latency requirement. Processing 200 records a minute is one thing. If a user is typing in a string and hitting Enter to get some value, then simpler is often better. After all C++ has map STL. I used it for flat file key/data lookup for programs I wrote in 2001 and I never got user feedback about the program being unable to fetch the data. And that was for a shell extension. Never hung up Explorer that I ever heard. :)

The nice thing about the Dictionary is as long as you check with Exists, if you make a mistake it's usually a typo. Nice and neat.

$val = $array("Key")

simple.

Edited by MilesAhead

Share this post


Link to post
Share on other sites

I also think that an SQLite database would be better suited for this situation. If you have to use the INI file though, I would go with a combination of FileRead and StringRegExp instead of the INI functions.

If you pass the path to your INI file and the search value into this function, it will return the key that corresponds to the value. The search value is case sensitive.

ConsoleWrite(_ReadINI(@ScriptDir & "Test.ini", "C") & @CRLF)

Func _ReadINI($s_IniPath, $s_Value)
    Local $a_RegEx = StringRegExp(StringRegExpReplace(FileRead($s_IniPath), "(rn|r)", @LF), "n(.*?)=" & $s_Value, 3)
    If Not IsArray($a_RegEx) Then Return SetError(1,0,0)
    Return $a_RegEx[0]
EndFunc

This was my Test.ini.

[Test]
Test1=A
Test2=B
Test3=C
Test4=D
Test5=E
Test6=F
Test7=G
Test8=H
Test9=I

this solution is really fast and easy.. by the way my script is really simple ..thx

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0