Update SQlite database and run file in memory.

[Belini]

I have a database with more than 100,000 rows and I need to replace data in it all when I send any number that has anywhere in this database, I did tests and I have no error return, but the exchange is not done in the database either, can anyone help?

expandcollapse popup

#include <SQLite.au3>
#include <array.au3>

Global $hQuery, $aRow, $sMsg, $sSql, $dados_linha
Global $BASE_DB, $cod_cartela, $sorteado, $retarr

_SQLite_Startup(@ScriptDir & "\sqlite3.dll")

If Not FileExists("database.db") Then
    $dbn = _SQLite_Open("database.db")
    _SQLite_Exec($dbn, "CREATE TABLE Dados (Numeros);")
    For $i = 1 To 20
        $dados = "02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q" & $i
        $sql = "INSERT INTO Dados VALUES('" & $dados & "');"
        _SQLite_Exec($dbn, $sql)
    Next
Else
    $dbn = _SQLite_Open("database.db")
EndIf

$sorteado = "06 "
$trocar = "| "
_SQLite_Exec($dbn, "UPDATE Dados set Numeros= " & _SQLite_FastEscape($sorteado) & " where Numeros= " & _SQLite_FastEscape($trocar)); exchange all numbers for "| "
$aResult = _SearchDatabase($trocar); Looking for line that has "| "
If UBound($aResult) > 1 Then
    MsgBox(4096, 'OK', 'All numbers found have been replaced by "| "', 2)
    _ArrayDisplay($aResult)
Else
    MsgBox(4096, 'ERROR', $sorteado & ' It was not exchanged for "| "')
    $aResult = _SearchDatabase($sorteado)
    _ArrayDisplay($aResult)
EndIf

_SQLite_Close()
_SQLite_Shutdown()

Func _SearchDatabase($sString)
    Local $sql, $sSql, $aTables, $aResult, $iState
    Local $tmp1, $tmp2
    $sql = "SELECT Numeros, ""Dados"" AS ""table"", rowid FROM  ""Dados"""
    $sSql = "SELECT * FROM (" & $sql & ") WHERE Numeros LIKE '%" & $sString & "%';"
    $iState = _SQLite_GetTable2d($dbn, $sSql, $aResult, $tmp1, $tmp2)
    If $iState <> $SQLITE_OK Then
        Return SetError(@error, 0, $iState)
    Else
        Return $aResult
    EndIf
EndFunc   ;==>_SearchDatabase

 

Edited January 20, 2020 by Belini

[jchd]

The DB rows in your example look like this:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q2'
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q3'
...
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q20'
 

Your SQL update statement is:
UPDATE Dados set Numeros= '06 ' where Numeros= '| '

No surprise no row is updated since no row equals '| '

If you want to update rows containing the string '06 '  and replace this part of the string by '| ', then you should do:

UPDATE Dados set Numeros= replace(numeros, '06 ', '| ') where Numeros like '%06 %'

[Belini]

It worked @jchd, thanks for the help.

[jchd]

Beware that in 100k rows you don't have '06 ' substrings in other places in the strings, else:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 106 PM-Q12345'

will give:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 1| PM-Q12345'

As always, know your own data!

[Belini]

This really happened and now i'm researching with space before and after the number, now it works perfectly.

Edited January 20, 2020 by Belini

[Belini]

Exchanging data and searching using the hard drive's file is taking too long, how could I open this file from the disk in memory and make all the changes while it is open without writing to the disk?

EDITED: I managed to create a database copy that is on the hard drive and run it in memory.

_SQLite_Startup()
Local $dbn = _SQLite_Open()
_SQLite_Exec($dbn, "ATTACH DATABASE '" & @ScriptDir & "\database.db' as Original;")
_SQLite_Exec($dbn, "CREATE TABLE Dados AS SELECT * FROM Original.Dados;")

 

Edited January 20, 2020 by Belini

[Belini]

Now I need to save what is in memory to the hard drive and I always have an error, can someone give a tip?

Local $dbdisk = _SQLite_Open(@ScriptDir & "\database.db")
_SQLite_Exec($dbdisk, "ATTACH DATABASE ':memory:' as Original;")
_SQLite_Exec($dbdisk, "CREATE TABLE Dados AS SELECT * FROM Original.Dados;")

I always get an error that the table was not found but in memory the table exists because it was created and populated.

Quote

!   SQLite.au3 Error
--> Function: _SQLite_Exec
--> Query:    CREATE TABLE Dados AS SELECT * FROM Original.Dados;
--> Error:    no such table: Original.Dados

 

[jchd]

I don't know how the two code parts posted above articulate. If they are just separated by code to perform the replacement in memory, then you do this backwards.

What you can try:

-) open the disk DB.
-) attach a :memory: db, name it MEM and copy the disk table to it
-) create a new table in MEM defined as: create table mem.changes (id integer primary key, numeros char);
-) perform the search/replace on MEM this way: select rows containing the searched value with a CTE and insert them in mem.changes after replacing the value; don't forget to set mem.id = rowid of the mem.dados table
-) rewrite only changes to the disk db: use a CTE on mem.changes to update dados set numeros = <cte table data> where rowid = <cte id value>

But I suspect there may be a much more efficient way to store the data.  Can you tell what the numeric values in numeros mean and which range they cover? It looks like the data is some kind of array, something SQL isn't particularly good at handling.

What I have in mind is along the following example, based on the data in the first post:
Instead of
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
it could be advantageous to store instead:
rowid, '00100110100000010000100100101010000110100001010100001011100000000010100100119', 'PM-Q1'
because then an index can then be used on the column numeros.

Explanations in the code below:

expandcollapse popup

#include <Array.au3>

Local $s = '02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
Local $a = StringSplit($s, ' ', 3)
_ArrayDisplay($a)
; the text part is last and there are ubound($a) entries
; create a string representing the values: 0 means the index isn't present, 1 means it is present, 2 means a | and 9 means end of string
; assuming the numeric values are in range [0..75] here
; assuming the values can't be too large (the string length derives from the largest value)
; also assuming the values are unique in a given entry
; values may appear in any order, since they are used as indices in the resulting string

; TO DO: if an original value is | then it can't be used as an index and the most logical way to denote that would be to store 2 (= |) in all positions between previous and next values
Local $data[$a[UBound($a) - 2] + 2]
For $i = 0 To UBound($data) - 2
    $data[$i] = '0'
Next
$data[UBound($data) - 1] = '9'
_ArrayDisplay($data)
For $i = 0 To UBound($a) - 2
    $data[$a[$i]] = '1'
Next
_ArrayDisplay($data)
Local $b = _ArrayToString($data, '')
Local $name = $a[UBound($a) - 1]
; show what the resulting data is:
ConsoleWrite($b & @TAB & $name & @LF)

Exit

#cs
; we're ready to store that entry to a DB
; the table should be declared this way:
"create table dados (id integer primary key, numeros char, name char);"
"create index num on dados (numeros);"                                      ; make it easy to search on a given value in numeros
"create index nam on dados (name);"                                         ; optional to find rows on name part
"insert into dados (numeros, name) value ('" & $b & "', '" & $name & "');"

; The table can then be searched more efficiently using the index
; for instance, to find rows with value '06', we do:
"select * from dados where numeros like '0000001%';"
; to update rows with '06' to '|' we do a CTE with the above select and perform the replace in string:
"update dados set numeros = substr(<numeros from cte>, <>, <>) || '2' || substr(<numeros from cte>, <>, <>) where id = <id from cte>;"
; this can also be done with a single printf()

; converting the string of 0/1 to binary would use less storage but wouldn't help searching as blobs can't be indexed
#ce

 

[Belini]

@jhcd I do need to copy a database that is on disk to memory and this I have already managed to do, after it is in memory I need to add more information in this database and then save a new file on the hard disk, I want to load it into memory to make the process faster and I’ll save it to disk only when I’ve finalized all the changes.

Quote

Can you tell what the numeric values in numeros mean and which range they cover?

These are numbers from a drawing that I need to check automatically each time a number with a space before and after it is typed " 30 " in every database I change " 30 " to " | "

Edited January 21, 2020 by Belini

[jchd]

OK, then you can just use the backup function back and forth.

 

[Belini]

I had already tried this but I get error messages when I run SQLitebackupTest

Quote

! ***************************************************************************************************************
! * Input file is UTF8 without BOM encoded, Obfuscator do not support UNICODE and will be skipped.      *
! * The file SHOULD BE encoded as UTF8 with BOM to continue processing by AutoIT3Wrapper.                       *
! *    #####################################################################################################    *
! * ##### AutoIt3Wrapper will not show a GUI or update the script to avoid any damage to your scriptfile. ##### *
! *    #####################################################################################################    *
! * When your file isn't a UTF8 file without BOM then please report this to me for review.                      *
! ***************************************************************************************************************

 

[jchd]

Strange. Just save the file as UTF8 + BOM then.

[Belini]

I was unable to change to UTF8 + BOM in version 3.3.9.4 of Autoit that I use, there is no simpler way to just save the database that is in memory to the hard drive after having made all the changes to this file that is being executed in the memory?

[jchd]

You should upgrade to the latest AutoIt version!

You can as well do what you say by dropping the DB table and recreating it from the memory table.

[Belini]

I would really like to be able to continue using version 3.3.9.4 because it is the last one that still obfuscates the script after compiled, I know that this does not solve for those with advanced knowledge but it keeps away curious people who decompile scripts, any other way to save the bank of data I have in memory for the hard drive would be welcome but I cannot change the version I use Autoit to do this.

[jchd]

Why not just read the UDF and save it as ASCII?

[Belini]

@jhcd i will do tests with what you passed me, thanks.

 

[Belini]

I was able to write the database that was open in memory to the hard disk thanks to @Manimal that gave me the tip to use the command "VACUUM INTO"

_SQLite_Exec($hBD_Mem, "VACUUM INTO '" & @ScriptDir & "\Bd_Copy.db'")

 

[jchd]

Ah yes, that's a pretty recent feature I never had a use for.

[Belini]

I found a very complete tutorial that talks about this and several other resources:  https://www.tutorialspoint.com/sqlite/sqlite_vacuum.htm