Stack - speed for dynamic data

[bluelamp]

If you want to add values to an array you have to redim it - that costs valuable cpu-cycles.

If you rely on speed you might consider using this stack implementation.

Here is a little example(this is also a speed test):

expandcollapse popup

#include <Stack.au3>
#include <Array.au3>

Const $NUM_VALUES = 5000
Local $i
Local $stack
Local $array[1] = [0]
Local $time

#region array test1
ReDim $array[1]
$time = TimerInit()
For $i = 0 To $NUM_VALUES Step 1
    _ArrayInsert($array, $i, $i)
Next
ConsoleWrite("_ArrayInsert=" & TimerDiff($time) & @CRLF)
#endregion array test1

#region array test2
ReDim $array[1]
$time = TimerInit()
For $i = 0 To $NUM_VALUES Step 1
    _ArrayPush($array, $i)
Next
ConsoleWrite("_ArrayPush=" & TimerDiff($time) & @CRLF)
#endregion array test2

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Stack_Plugin_Startup()
#region stack test1

$stack = Stack_Init()

$time = TimerInit()
For $i = 0 To $NUM_VALUES Step 1
    Stack_SetAt($stack, $i, $i)
Next
ConsoleWrite("Stack_SetAt=" & TimerDiff($time) & @CRLF)

Stack_Free($stack)
#endregion stack test1

#region stack test2
$stack = Stack_Init()

$time = TimerInit()
For $i = 0 To $NUM_VALUES Step 1
    Stack_Push($stack, $i)
Next
ConsoleWrite("Stack_Push=" & TimerDiff($time) & @CRLF)

Stack_Free($stack)
#endregion stack test2

Stack_Plugin_Shutdown()

results of the speed test:

_ArrayInsert=7682.87918240707

_ArrayPush=86.1614862445099

Stack_SetAt=17.2797508805599

Stack_Push=16.6188761977596

This zip contains the c sources for the stack plugin, some examples and of course the autoit include file

au3stack.zip

Edited March 2, 2011 by bluelamp

[ivan]

I get undefined function errors:

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(52,22) : ERROR: Stack_Init(): undefined function.

$stack = Stack_Init()

~~~~~~~~~~~~~~~~~~~~^

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(55,32) : ERROR: Stack_Push(): undefined function.

Stack_Push($stack, $array[$i])

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(71,27) : ERROR: Stack_Count(): undefined function.

$len = Stack_Count($stack)

~~~~~~~~~~~~~~~~~~~~~~~~~^

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(77,33) : ERROR: Stack_Iter(): undefined function.

Local $iter = Stack_Iter($stack)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(79,25) : ERROR: Stack_Get(): undefined function.

$val = Stack_Get($iter)

~~~~~~~~~~~~~~~~~~~~~~^

F:\Autoit_scripts\au3stack\au3stack\bin\Stack.au3(81,23) : ERROR: Stack_Free(): undefined function.

Stack_Free($iter, 0)

~~~~~~~~~~~~~~~~~~~^

[MrCreatoR]

If you want to add values to an array you have to redim it

Not neccessarily. You can dim the array at the begining, then add values by counter ($avArray[0] += 1), and redim the array to the counter at the end.

[bluelamp]

I get undefined function errors

You can ignore these.

It is because the functions are defined in a plugin.

If you want to have them to not appear put the following lines into C:\Program Files\AutoIt3\Au3Check.dat

or wherever you have your Au3Check.dat file:

!Stack_Init 0 0
!Stack_Iter 1 1
!Stack_Push 2 2
!Stack_GetAt 2 2
!Stack_SetAt 3 3
!Stack_Find 2 2
!Stack_Get 1 1
!Stack_Pop 1 1
!Stack_Free 1 2
!Stack_Count 1 1
!Stack_Sort 1 2

Not neccessarily. You can dim the array at the begining, then add values by counter ($avArray[0] += 1), and redim the array to the counter at the end.

I meant that you can't add a value to a fully filled array without redimming

Some code to visualize the problem(this produces an error in line 4):

Dim $a[5] = [1,2,3,4,5]

For $i = 0 To 9 Step 1
    $a[$i] = $i ; here comes the error
Next
ReDim $a[10]

For $i = 0 To 9 Step 1
    ConsoleWrite($i & "->" & $a[$i] & @CRLF)
Next

[ivan]

Now that makes sense.

[ivan]

Not neccessarily. You can dim the array at the begining, then add values by counter ($avArray[0] += 1), and redim the array to the counter at the end.

Wouldn't that imply 2 iterations, 1 for counting the other for assigning values to the added elements of the array?

[MrCreatoR]

Some code to visualize the problem(this produces an error in line 4):

How is that relevant to the speed issue?

Here is what i meant:

Dim $avArray[10000]

For $i = 1 To 100
    $avArray[0] += 1
    $avArray[$avArray[0]] = $i
Next

ReDim $avArray[$avArray[0]+1]

Wouldn't that imply 2 iterations, 1 for counting the other for assigning values to the added elements of the array?

No, see the example above.

[MrCreatoR]

If you want to have them to not appear put the following lines into C:\Program Files\AutoIt3\Au3Check.dat

You can also use the #AutoIt3Wrapper_Plugin_Funcs directive:

#AutoIt3Wrapper_Plugin_Funcs=Stack_Init,Stack_Iter,Stack_Push,Stack_GetAt,Stack_SetAt,Stack_Find,Stack_Get,Stack_Pop,Stack_Free,Stack_Count,Stack_Sort

[Melba23]

Hi,

Apologies in advance for the thread drift.

Another good trick, particularly useful if you have no idea how big an array is likely to get, is to double the size of the array each time you fill it. You need a count and a final ReDim as MrCreatoR has pointed out, but with a reasonable sized initial array, you get big arrays very quickly.

The code looks something like this - pulled straight from one of my scripts:

; Increase folder count
$aFolderList[0] += 1
; Double array size if full
If UBound($aFolderList) <= $aFolderList[0] Then ReDim $aFolderList[UBound($aFolderList) * 2]
; Add folder
$aFolderList[$aFolderList[0]] = $sSearchPath & $sFile & "\"

So an initial array of 100 becomes a final 10000 with only 8 ReDims:

100 - 200 - 400 - 800 - 1600 - 3200 - 6400 - 12800 - 10000

Just another way of avoiding the dreaded ReDim slowdown when filling large arrays.

M23

[ivan]

How is that relevant to the speed issue?

Here is what i meant:

Dim $avArray[10000]

For $i = 1 To 100
    $avArray[0] += 1
    $avArray[$avArray[0]] = $i
Next

ReDim $avArray[$avArray[0]+1]

No, see the example above.

I usually deal with dyinamic arrays with something of this sort if I desire a 1 based array:

Dim $avArray[1]
For $i = 1 To 100
    ReDim $avArray[Ubound($avArray)+1] 
        $avArray[Ubound($avArray)-1]=$i
Next
$avArray[0] = Ubound($avArray)

Although I usually keep my arrays as zero based, and use Ubound to get the count. This is a matter of preference but many of the Autoit funcs will return arrays with element 0 containing the count of elements, so I often find juggling between the 2.

Plugins are experimental and it is uncertain if they are going to remain in future versions of Autoit. Like with Static, I would refrain from using these until I am certain that they will be kept in future versions. This reduces the risk of making scripts obsolete when a new version is out.

[MrCreatoR]

I usually deal with dyinamic arrays with something of this sort if I desire a 1 based array:

This is the slowest way possible of doing it

[bluelamp]

How is that relevant to the speed issue?

Here is what i meant:

Dim $avArray[10000]

For $i = 1 To 100
    $avArray[0] += 1
    $avArray[$avArray[0]] = $i
Next

ReDim $avArray[$avArray[0]+1]

No, see the example above.

This would beat the stack in a speed test no question. But it uses 100 times more memory than needed.

Another problem would occur if we don't know the size of elements that shall go into the array. Maybe we want to have every line of a file in

a dynamic data structure.

This program would crash if the bounds of the buffer $avArray are exceeded.

You can also use the #AutoIt3Wrapper_Plugin_Funcs directive:

#AutoIt3Wrapper_Plugin_Funcs=Stack_Init,Stack_Iter,Stack_Push,Stack_GetAt,Stack_SetAt,Stack_Find,Stack_Get,Stack_Pop,Stack_Free,Stack_Count,Stack_Sort

Thanks for the hint

Hi,

Apologies in advance for the thread drift.

Another good trick, particularly useful if you have no idea how big an array is likely to get, is to double the size of the array each time you fill it. You need a count and a final ReDim as MrCreatoR has pointed out, but with a reasonable sized initial array, you get big arrays very quickly.

The code looks something like this - pulled straight from one of my scripts:

; Increase folder count
$aFolderList[0] += 1
; Double array size if full
If UBound($aFolderList) <= $aFolderList[0] Then ReDim $aFolderList[UBound($aFolderList) * 2]
; Add folder
$aFolderList[$aFolderList[0]] = $sSearchPath & $sFile & "\"

So an initial array of 100 becomes a final 10000 with only 8 ReDims:

100 - 200 - 400 - 800 - 1600 - 3200 - 6400 - 12800 - 10000

Just another way of avoiding the dreaded ReDim slowdown when filling large arrays.

M23

This is a very good compromise between speed and memory consumption.

Plugins are experimental and it is uncertain if they are going to remain in future versions of Autoit. Like with Static, I would refrain from using these until I am certain that they will be kept in future versions. This reduces the risk of making scripts obsolete when a new version is out.

I don't think the AutoIt devs would throw out such a wonderful feature. Plugins are a great tool for interfacing with C/C++ API's. DllCall is the alternative but well... it is ugly. Edited March 2, 2011 by bluelamp

[MrCreatoR]

But it uses 100 times more memory than needed.

It's released after ReDim.

Another problem would occur if we don't know the size of elements that shall go into the array

Then we check the counter, if it's grater than initial Dim, we ReDim the array again:

$iInitDim = 1000
Dim $avArray[$iInitDim]

For $i = 1 To 10000
    $avArray[0] += 1
    
    If $avArray[0] >= $iInitDim Then
        $iInitDim *= 2
        ReDim $avArray[$iInitDim]
    EndIf
    
    $avArray[$avArray[0]] = $i
Next

ReDim $avArray[$avArray[0]+1]

[twitchyliquid64]

My systems that need high speed memory data (if i cant code a file system or somefin) pre-emptifly resize the array. So instead of resizing the Array every new entry, I would resize to 100+ or even 1000+ if its a web server. thus the slowdown is not an issue anymore unless your in a tight loop, in which case you shouldn't be resizing on the fly anyway.

This is my database system.

It loads a encrypted database key-catagory -> value into database.

It is optimized for reads; it uses indexing and a sorting algorithm. I cant find the version that over-resizes to improve write speed, its on my web server thou.

expandcollapse popup

;=================================================================================================
;-----------------------------DATABASE MANAGEMENT SYSTEM------------------------------------------
;Author: Hyperzap
;Credits: ITS Project for the idea

;This system is based off the good database idea from Autoit ITS.
;Basically there are two parts to the database; the files and the arrays.
;Upon startup, the databases are loaded and decrypted from file, then each item is loaded into arrays.
;As data changes, it is both changed in the array and (later) commited to file.
;When parsed to files or to arrays, the data is ordered based on catagory for speedy access.
;
;This has a couple of purposes. Firstly, data can be accessed quickly through the array system -
;Alot faster than a file system alone. It is also rather scalable for larger databases.
;Secondly, the database is kept encrypted on file at all times, and the only unencrypted copy is stored
;in program memory itself. This means that even if the server is hacked, it is pretty hard to access the
;Database - (assuming there are no vunerabilities) you would have to be a pretty good hacker.
;
;The only thing I would not recommend using this for is for file storage and large blocks of text.
;This system is designed to hold user information and basic interactions - As everything is loaded
;into memory holding anything else would be inefficient and be slower in comparision to traditional database
;mechanisms. If you plan on storing files I would suggest storing the files as files on your server and using
;the database merely as a pointer to the file.

;Now you know how it works, its remarkably easy to use.

;Just remember, data is broken down into catagories, then to keys. Bit like an INI file.
;-----------------------------FUNCTIONS-----------------------------------------------------------
;Database_INIT, Database_WRITE, Database_READ,  Database_COMMIT, Database_CLOSE, Database_OPTIMIZE
;=================================================================================================

;HOW TO USE




#include<string.au3>
#Include <Crypt.au3>
#Include <file.au3>
#Include <array.au3>

global $DATABASE_register[2][3]
;[0] = database file path, [1] = encryption key, [2] = entry array size.

global $DATA_ENTRIES[1][2][3]
;[0] = catagory, [1] = key, [2] = Data
global $Max_current_entries = 0 ;This does NOT mean you can only hold * entries, rather, it holds the number of known entries currently mounted.

global $CATAGORIES[2][2][3]
;[0] = Catagory Name, [1] = First key, [2] = Last Key
global $Max_current_catagories = 0;This does NOT mean you can only hold * catagories, rather, it holds the number of known catagories currently mounted.


Func Database_CLOSE( $database)
    Database_COMMIT($database)
EndFunc


Func Database_INIT( $database_path, $access_code)
    if not FileExists( $database_path) then Create_database_file($database_path, $access_code)
    $databasekey = Ubound( $DATA_ENTRIES)
    $hKey=_Crypt_DeriveKey($access_code,$CALG_RC4)
    local $handle = FileOpen( $database_path)
    if $handle = -1 or $handle = 0 Then
        SetError(3) ;File Error!
        return 0
    EndIf
    dim $Entries
    _FileReadToArray( $database_path, $Entries)
    For $x = 1 to $Entries[0]
        $Entries[$x] = String(_HexToString(_Crypt_DecryptData($Entries[$x],$hKey,$CALG_USERKEY)))
    Next

    ReDim $DATABASE_register[$databasekey+1][3]
    $DATABASE_register[$databasekey][0] = $database_path
    $DATABASE_register[$databasekey][1] = $access_code
    $DATABASE_register[$databasekey][2] = $entries[0]-2

    if $Entries[0] < 2 then
        SetError(1) ;Bad database!
        return 0
    ElseIf $Entries[1] <> "ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789!@#$%^&*()-_=+;:[{]}\|" Then
        SetError(2) ;Bad Access Code!
        return 0
    EndIf
    local $knowncatagories = ""
    local $knowncount = 0
    For $a = 3 to $entries[0] step 1 ;Detect the Catagories.
        local $spl = StringSplit( $Entries[$a], "TYAA#!#", 1)
        if not StringInStr( $knowncatagories, $spl[1]) > 0 and $spl[1] <> "" then ;If we haven't seen it before
            if $knowncount+1 > $Max_current_catagories then $Max_current_catagories = $knowncount+1
            ReDim $CATAGORIES[$databasekey+1][$Max_current_catagories][3]
            $CATAGORIES[$databasekey][$knowncount][0] = $spl[1]
            $CATAGORIES[$databasekey][$knowncount][1] = ""
            $CATAGORIES[$databasekey][$knowncount][2] = ""
            $knowncatagories &= $spl[1]&"E@#!#"
            $knowncount += 1
        EndIf
    Next
    $knownentries = 0
    For $a = 0 to UBound($CATAGORIES, 2)-1 step 1 ;Loop through the catagories and add the entries, grouping those similar.
        local $found_first = False
        For $x = 3 to $entries[0] step 1 ;Detect the entries
            local $spl = StringSplit( $Entries[$x], "TYAA#!#", 1)
            if $spl[1] = $CATAGORIES[$databasekey][$a][0] Then
                if not $found_first Then
                    $found_first = True
                    $CATAGORIES[$databasekey][$a][1] = $knownentries
                EndIf
                if $knownentries+1 > $Max_current_entries then
                    $Max_current_entries = $knownentries+1
                    ReDim $DATA_ENTRIES[$databasekey+1][$Max_current_entries][3]
                EndIf
                $DATA_ENTRIES[$databasekey][$knownentries][0] = $CATAGORIES[$databasekey][$a][0]
                if Ubound($spl) > 2 then $DATA_ENTRIES[$databasekey][$knownentries][1] = $spl[2]
                if Ubound($spl) = 4 then $DATA_ENTRIES[$databasekey][$knownentries][2] = $spl[3]
                $CATAGORIES[$databasekey][$a][2] = $knownentries
                $knownentries += 1
            EndIf
        Next
    Next
    $DATABASE_register[$databasekey][2] = $knownentries
    ConsoleWrite(@CRLF & "DATA: _INIT Completed - " & $knowncount & " catagories, " & $knownentries & " entries.")
    FileClose($handle)
return $databasekey
EndFunc











Func Database_OPTIMIZE( $database = 0)
    ;$database += 1 ;Old Bug fix. Stop Whining. Whining is about as pointless as nailing jelly to a tree.
    local $old_array = $DATA_ENTRIES
    local $knowncatagories = ""
    local $knowncount = 0
    $timer = TimerInit()
    For $a = 0 to UBound($old_array, 2)-1 step 1 ;Detect the Catagories.
        if not StringInStr( $knowncatagories, $old_array[$database][$a][0]) > 0 and $old_array[$database][$a][0] <> "" then ;If we haven't seen it before
            if $knowncount+1 > $Max_current_catagories then $Max_current_catagories = $knowncount+1
            ReDim $CATAGORIES[UBound($CATAGORIES)+1][$Max_current_catagories][3]
            $CATAGORIES[$database][$knowncount][0] = $old_array[$database][$a][0]
            $knowncatagories &= $old_array[$database][$a][0]&"E@#!#"
            $knowncount += 1
        EndIf
    Next
    $knownentries = 0
    For $a = 0 to UBound($CATAGORIES, 2)-1 step 1 ;Loop through the catagories and add the entries, grouping those similar.
        local $found_first = False
        For $x = 0 to UBound($old_array, 2)-1 step 1 ;Detect the entries
            if $old_array[$database][$x][0] = $CATAGORIES[$database][$a][0] and $old_array[$database][$x][1] <> "" Then
                if not $found_first Then
                    $found_first = True
                    $CATAGORIES[$database][$a][1] = $knownentries
                EndIf
                if $knownentries+1 > $Max_current_entries then
                    $Max_current_entries = $knownentries+1
                    ReDim $DATA_ENTRIES[UBound($DATA_ENTRIES)+1][$Max_current_entries][3]
                EndIf

                $DATA_ENTRIES[$database][$knownentries][0] = $old_array[$database][$x][0]
                $DATA_ENTRIES[$database][$knownentries][1] = $old_array[$database][$x][1]
                $DATA_ENTRIES[$database][$knownentries][2] = $old_array[$database][$x][2]
                $CATAGORIES[$database][$a][2] = $knownentries
                $knownentries += 1
            EndIf
        Next
    Next
    ConsoleWrite(@CRLF & "DATA: _OPTIMIZE Completed - " & $knowncount & " catagories, " & $knownentries & " entries. Took: " & Round(TimerDiff( $timer), 2) & "ms." & @CRLF)
    ;DEBUG
    ;For $x = 0 to UBound( $DATA_ENTRIES, 2)-1 step 1
    ;   ConsoleWrite( $DATA_ENTRIES[1][$x][0] & "::" & $DATA_ENTRIES[1][$x][1] & "  " & $DATA_ENTRIES[1][$x][2] & @CRLF)
    ;Next
EndFunc












Func Database_WRITE( $database, $catagory, $keyname, $DATA)
    $entries = $DATABASE_register[$database][2]
    $DATABASE_register[$database][2] += 1
    ConsoleWrite(@CRLF & "DATA: _WRITE: ADD TO -> " & $catagory & ": CURRENT DATABASE ENTRIES: " & $entries)
    if $entries+3 > $Max_current_entries then
        $Max_current_entries = $entries+3
        ReDim $DATA_ENTRIES[UBound($DATA_ENTRIES)+1][$Max_current_entries][3]
    EndIf
    $callout = False
    $catagorycount = 0
    For $e = 0 to UBound( $CATAGORIES)-1 step 1
        if $catagory = $CATAGORIES[$database][$e][0] then
            $callout = True
            ExitLoop
        EndIf
        $catagorycount += 1
    Next
    if $callout = False and $catagorycount+1 > $Max_current_catagories then
        $Max_current_catagories = $catagorycount+1
        ReDim $CATAGORIES[UBound($CATAGORIES)+1][$Max_current_catagories][3]
        $CATAGORIES[$database][$catagorycount-1][0] = $catagory
        $CATAGORIES[$database][$catagorycount-1][1] = ""
        $CATAGORIES[$database][$catagorycount-1][2] = ""
    EndIf
    $DATA_ENTRIES[$database][$entries][0] = $catagory
    $DATA_ENTRIES[$database][$entries][1] = $keyname
    $DATA_ENTRIES[$database][$entries][2] = $DATA
EndFunc













FUNC Database_READ( $database, $catagory, $key)
local $timer = TimerInit()
local $last_known_indexed_entry = ""
    For $e = 0 to UBound( $CATAGORIES, 2)-1 step 1
        if $catagory = $CATAGORIES[$database][$e][0] then ;First check the grouped entries for the catagory, and see if in there.
        if $CATAGORIES[$database][$e][2] = "" then ContinueLoop ;If no index for the catagory then continue.
        ConsoleWrite( @CRLF & "DATA: _READ: Matching Indexed catagory found - beginning search in index." & @CRLF)
            for $q = $CATAGORIES[$database][$e][1] to $CATAGORIES[$database][$e][2] step 1 ;Search for key in index.
                if $key = $DATA_ENTRIES[$database][$q][1] Then
                    ConsoleWrite(TimerDiff( $timer))
                    Return $DATA_ENTRIES[$database][$q][2]
                EndIf
            Next
        EndIf
        if $CATAGORIES[$database][$e][2] <> "" then $last_known_indexed_entry = $CATAGORIES[$database][$e][2]
    Next
    ConsoleWrite( @CRLF & "DATA: _READ: Unoptimized or non-existant key - Scanning unindexed subscripts: ")
    if $last_known_indexed_entry = "" Then ;If no catagory is indexed, scan entire entry array.
        For $x = 0 to $Max_current_entries-1 and $DATA_ENTRIES[$database][$x][0] = $catagory step 1
            if $key = $DATA_ENTRIES[$database][$x][1] Then
                    ConsoleWrite(TimerDiff( $timer) & @CRLF & @CRLF)
                Return $DATA_ENTRIES[$database][$x][2]
            EndIf
        Next
    Else ;If there are indexes, search the unindexed portion.
        For $x = $last_known_indexed_entry to $Max_current_entries-1 step 1
            if $key = $DATA_ENTRIES[$database][$x][1] and $DATA_ENTRIES[$database][$x][0] = $catagory Then
                    ConsoleWrite(TimerDiff( $timer) & @CRLF & @CRLF)
                Return $DATA_ENTRIES[$database][$x][2]
            EndIf
        Next
    EndIf
    ConsoleWrite(TimerDiff( $timer) & @CRLF & @CRLF)
    SetError(1)
EndFunc


Func Database_COMMIT( $database)
    $databasepath = $DATABASE_register[$database][0]
    $key = $DATABASE_register[$database][1]
    if not Test_code( $key, $databasepath) then Return -1
    $hKey=_Crypt_DeriveKey($key,$CALG_RC4)
    $hnd =  FileOpen( $databasepath, 2)
    if $hnd = -1 then
        ConsoleWrite(@CRLF & @CRLF & "DATA: _COMMIT: Fatal File Error!"&@CRLF)
        return -1
    EndIf
    FileWrite( $hnd, _Crypt_EncryptData("ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789!@#$%^&*()-_=+;:[{]}\|",$hKey,$CALG_USERKEY) & @CRLF)
    FileWrite( $hnd, _Crypt_EncryptData("<DOCSTREAM-NORMAL>",$hKey,$CALG_USERKEY) & @CRLF)
    for $x = 0 to UBound($DATA_ENTRIES, 2)-1 step 1
        if $DATA_ENTRIES[$database][$x][1] <> "" Then
            FileWrite( $hnd, _Crypt_EncryptData($DATA_ENTRIES[$database][$x][0] & "TYAA#!#" & $DATA_ENTRIES[$database][$x][1] & "TYAA#!#" & $DATA_ENTRIES[$database][$x][2],$hKey,$CALG_USERKEY) & @CRLF)
        EndIf
    Next
    FileClose($hnd)
EndFunc

;==============================================================================
;PRIVATE CLASS (INTERNAL) FUNCTIONS - DONT FUCKING TOUCH
;==============================================================================

FUNC Write_Entry( $path, $code, $catagory, $data) ;DEBUG Function only - its inefficient so dont use!
    if Test_code( $code, $path) = false Then
        SetError(1) ;Bad Access Code!
        return 0
    EndIf
    $hnd = FileOpen( $path, 1)
    if $hnd = -1 or $hnd = 0 Then
        SetError(3) ;File Error!
        return 0
    EndIf
    $hKey=_Crypt_DeriveKey($code,$CALG_RC4)
    FileWrite( $hnd, _Crypt_EncryptData($catagory & "TYAA#!#" & $data,$hKey,$CALG_USERKEY) & @CRLF)
    FileClose( $hnd)
EndFunc


Func Test_code( $code, $path, $hnd = "", $hkey = "")
    if $hnd = "" then $hnd = FileOpen( $path)
    if $hnd = -1 then Return False
    $testdata = FileReadLine( $hnd, 1)
    FileClose( $hnd)
    if $testdata = "" then return False
    if $hKey = "" then $hKey=_Crypt_DeriveKey($code,$CALG_RC4)
    $bEncrypted=_Crypt_DecryptData($testdata,$hKey,$CALG_USERKEY)
    if $bEncrypted = "ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789!@#$%^&*()-_=+;:[{]}\|" then return True
    return False
EndFunc

Func Create_database_file($path, $code, $extended = "<DOCSTREAM-NORMAL>")
    if not FileExists( $path) Then
        _FileCreate( $path)
        $hKey=_Crypt_DeriveKey($code,$CALG_RC4)
        $bEncrypted=_Crypt_EncryptData("ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789!@#$%^&*()-_=+;:[{]}\|",$hKey,$CALG_USERKEY)
        FileWrite( $path, $bEncrypted & @CRLF & _Crypt_EncryptData($extended,$hKey,$CALG_USERKEY) & @CRLF)
    EndIf
EndFunc


Func Decrypt_Debug( $path, $key)
    ConsoleWrite( "Key Test: " & Test_code( $key, $path) & @CRLF)
    $key=_Crypt_DeriveKey($key,$CALG_RC4)
    Dim $aRecords
    _FileReadToArray($path, $aRecords)
    if Not FileExists( $path) Then
        ConsoleWrite( "File doesnt exist!" & @CRLF)
        return 0
    EndIf
    if not IsArray( $aRecords) then return 0
    For $x=1 to $aRecords[0] step 1
        ConsoleWrite( "DATA: " & _HexToString(_Crypt_DecryptData($aRecords[$x],$key,$CALG_USERKEY)) & @CRLF)
    Next
EndFunc

Edited March 5, 2011 by hyperzap