Compare Array to SQLite DB Contents?

[buymeapc]

Hi all,

I have a large array with data and a SQLite database with similar data. I'm trying to see what the differences are between the two and output an array of the differences.

I have a small reproducer that works, but the 'loop check' takes upwards of a minute to output the differences. Is there a better way to do this?

Thanks for the help!

expandcollapse popup

#include <SQLite.au3>
#include <SQLite.dll.au3>

Local $hQuery, $aRow
Local $sDBPath = @ScriptDir & "\Temp.db"
Local $aArray[10000]
_SQLite_Startup()

If FileExists($sDBPath) = 0 Then
    $iTimer = TimerInit()
    _SQLite_Open($sDBPath)
    _SQLite_Exec(-1, 'create table tblTest (a text, b text, c text);')

    ; Add random information to the db to query against
    _SQLite_Exec(-1, "begin immediate;")
    For $i = 0 To 10000
        _SQLite_Exec(-1, 'insert into tblTest (a, b, c) values ("' & Chr(Random(Asc("A"), Asc("Z"), 1)) & '", "' & Chr(Random(Asc("a"), Asc("z"), 1)) & '", "' & Random(0, 9) & '");')
    Next
    _SQLite_Exec(-1, "commit;")
    ConsoleWrite("!> DB Creation Timer: " & Round((TimerDiff($iTimer) / 1000), 2) & @CRLF)
Else
    _SQLite_Open($sDBPath)
EndIf

ConsoleWrite("Creating array..." & @CRLF)
For $i = 0 To UBound($aArray) - 1
    $aArray[$i] = Chr(Random(Asc("A"), Asc("Z"), 1))
Next

ConsoleWrite("Beginning loop check..." & @CRLF)
$iTimer = TimerInit()
_SQLite_Query(-1, 'select a from tblTest;', $hQuery)
While _SQLite_FetchData($hQuery, $aRow) = $SQLITE_OK
    Local $iItem = _ArraySearch($aArray, $aRow[0])
    If $iItem > 0 Then
        _ArrayDelete($aArray, $iItem)
    EndIf
WEnd
ConsoleWrite("!> Loop Check Timer: " & Round((TimerDiff($iTimer) / 1000), 2) & @CRLF)

_ArrayDisplay($aArray)

_SQLite_Close()
_SQLite_Shutdown()

[mikell]

The best way is to put the array into a memory db and then compare the 2 db using something like this (schematic example)

_SQLite_GetTable2d(-1, "SELECT * FROM table1 WHERE items1 NOT IN (SELECT items2 FROM table2) ;", $array, $iRows, $iColumns)

Edited June 5, 2014 by mikell

[buymeapc]

Ok, that makes sense, but I am not sure how to compare the 2 databases when the _SQLite_* commands can only point to a single database at a time.

Here's what I tried (which didn't work)

expandcollapse popup

#include <SQLite.au3>
#include <SQLite.dll.au3>

Local $hQuery, $aRow
Local $sDBPath = @ScriptDir & "\Temp.db"
Local $aArray[10000]
_SQLite_Startup()

If FileExists($sDBPath) = 0 Then
    $iTimer = TimerInit()
    _SQLite_Open($sDBPath)
    _SQLite_Exec(-1, 'create table tblTest (a text, b text, c text);')

    ; Add random information to the db to query against
    _SQLite_Exec(-1, "begin immediate;")
    For $i = 0 To 10000
        _SQLite_Exec(-1, 'insert into tblTest (a, b, c) values ("' & Chr(Random(Asc("A"), Asc("Z"), 1)) & '", "' & Chr(Random(Asc("a"), Asc("z"), 1)) & '", "' & Random(0, 9) & '");')
    Next
    _SQLite_Exec(-1, "commit;")
    ConsoleWrite("!> DB Creation Timer: " & Round((TimerDiff($iTimer) / 1000), 2) & @CRLF)
Else
    _SQLite_Open($sDBPath)
EndIf

ConsoleWrite("Creating array..." & @CRLF)
For $i = 0 To UBound($aArray) - 1
    $aArray[$i] = Chr(Random(Asc("A"), Asc("Z"), 1))
Next

ConsoleWrite("Creating memory DB..." & @CRLF)
$hMemDB = _SQLite_Open()
_SQLite_Exec(-1, 'create table tblTestMem (a text);')
_SQLite_Exec(-1, "begin immediate;")
For $i = 0 To UBound($aArray) - 1
    _SQLite_Exec($hMemDB, 'insert into tblTestMem (a) values ("' & $aArray[$i] & '");')
Next
_SQLite_Exec(-1, "commit;")

ConsoleWrite("Beginning 1st loop check..." & @CRLF)
$iTimer = TimerInit()
Dim $aResults, $iRows, $iColumns
_SQLite_GetTable2d($hMemDB, 'select * from tblTestMem where a NOT IN (select a from tblTest) ;', $aResults, $iRows, $iColumns)
ConsoleWrite("!> Loop Check Timer: " & Round((TimerDiff($iTimer) / 1000), 2) & @CRLF)

_ArrayDisplay($aResults)

;~ ConsoleWrite("Beginning loop check..." & @CRLF)
;~ $iTimer = TimerInit()
;~ _SQLite_Query(-1, 'select a from tblTest;', $hQuery)
;~ While _SQLite_FetchData($hQuery, $aRow) = $SQLITE_OK
;~  Local $iItem = _ArraySearch($aArray, $aRow[0])
;~  If $iItem > 0 Then
;~      _ArrayDelete($aArray, $iItem)
;~  EndIf
;~ WEnd
;~ ConsoleWrite("!> Loop Check Timer: " & Round((TimerDiff($iTimer) / 1000), 2) & @CRLF)

;~ _ArrayDisplay($aArray)

_SQLite_Close()
_SQLite_Shutdown()

[BrewManNH]

What makes you think that the _SQLite commands can only work with one DB at a time? Use the handles returned by the _SQLite_Open functions to use to refer to them, that's what they're for.

[jchd]

Two equally easy ways:

A] load your array into a memory DB making use of a large transaction and possibly multi-valued inserts. Then use ATTACH to attach the disk DB to the memory DB and run a single query. Details follow.

B] load your array into a new table (still using of a large transaction and possibly multi-valued inserts) of your disk DB and run the almost-same comparison as above.

SQL skeletons:

A]

1) open a memory DB

2) create table copy (columns with types)

3) begin

4) insert into copy (optional column list) values (values for row #1), (values for row #2), ... (values for row N)

Don't try to insert 100000 rows at a time, say 5 to 100 depending on the size of your data since SQL statements are limited in size).

5) commit

6) attach 'mydiskDBFilename' as disk

Depending on which side (or both) you want differences and on which criterions, your SQL may need change

7) select * from copy except select * from disk.myDiskTable

or

7) select * from disk.myDiskTable except select * from copy

or something more complex to highlight differences both sides in the same run (that can easily be done as well)

8) detach disk

B]

1) create table copy (columns with types)

2) begin

3) insert into copy (optional column list) values (values for row #1), (values for row #2), ... (values for row N)

Don't try to insert 100000 rows at a time, say 5 to 100 depending on the size of your data since SQL statements are limited in size).

4) commit

Depending on which side (or both) you want differences and on which criterions, your SQL may need change

5) select * from copy except select * from diskDBtable

or

5) select * from diskDBtable  except select * from copy

or something more complex to highlight differences both sides in the same run (that can easily be done as well)

Edited June 5, 2014 by jchd

[buymeapc]

Thanks jchd. I've followed your advice, but I'm stumped on how to tell _SQLite_Exec() or _SQLite_GetTable2d() to point to two different databases. I'm just so accustomed to setting $hDB as either -1 or the database handle that I've been using. How can I do this?

[mikell]

OK seems working 

expandcollapse popup

#include <SQLite.au3>
#include <SQLite.dll.au3>

Dim $aResults, $iRows, $iColumns
Local $hQuery, $aRow
Local $sDBPath = @ScriptDir & "\Temp.db"
_SQLite_Startup()

$sDB = _SQLite_Open($sDBPath)
_SQLite_Exec(-1, 'create table if not exists tblTest (a text, b text, c text);')
_SQLite_Exec(-1, "begin;")
For $i = 0 To 10000
     _SQLite_Exec(-1, 'insert into tblTest (a, b, c) values ("' & Chr(Random(Asc("C"), Asc("Z"), 1)) & '", "' & Chr(Random(Asc("a"), Asc("z"), 1)) & '", "' & Random(0, 9) & '");')
Next
_SQLite_Exec(-1, "commit;")


;_SQLite_GetTable2d(-1, 'select * from tblTest ;', $aResults, $iRows, $iColumns)
; _ArrayDisplay($aResults, "$sDB")

;======================================
Local $aArray[10000]
For $i = 0 To UBound($aArray) - 1
    $aArray[$i] = Chr(Random(Asc("A"), Asc("Z"), 1))
Next
; _ArrayDisplay($aArray, "$aArray")

_SQLite_Exec(-1, 'create table tblTestarray (a text);')
_SQLite_Exec(-1, "begin;")
For $i = 0 To UBound($aArray) - 1
    _SQLite_Exec(-1, 'insert into tblTestarray (a) values ("' & $aArray[$i] & '");')
Next
_SQLite_Exec(-1, "commit;")

_SQLite_GetTable2d(-1, 'select * from tblTestarray ;', $aResults, $iRows, $iColumns)
_ArrayDisplay($aResults, "tblTestarray")
;=======================================

_SQLite_GetTable2d(-1, 'select * from tblTestarray where a NOT IN (select a from tblTest) ;', $aResults, $iRows, $iColumns)
_ArrayDisplay($aResults, "check")

_SQLite_Exec(-1, 'drop table tblTestarray;')

_SQLite_Close()
_SQLite_Shutdown()

[buymeapc]

Oh, thank you mikell! I've been sitting here scratching my head thinking about how to compare two physical databases when it's so easy to simply create a temporary table in the currently working database and dump it when all the checking is done. 

Thanks!!

[jchd]

Ahem, it was my option B]

[mikell]

Of course it was ... with a little from post #2

A brain and brawn nice example 

[jchd]

My remark was directed to the OP.

BTW, generally avoid ... where a NOT IN (select ... and replace by the EXCEPT construct when dealing with the full table.

Of course building an index (maybe even temporary) on tabletest.a here will greatly help.

[buymeapc]

Thank you for your help, jchd and mikell.

You know, I kept focusing on your option A] and trying to figure out the code with your details, that I never even looked at option B]. Sorry about that!  I'm just starting my journey into the land of SQL, so I appreciate all the help.

[jchd]

No hurt done, it's all OK.