Jump to content
Belini

Update SQlite database and run file in memory.

Recommended Posts

I have a database with more than 100,000 rows and I need to replace data in it all when I send any number that has anywhere in this database, I did tests and I have no error return, but the exchange is not done in the database either, can anyone help?

#include <SQLite.au3>
#include <array.au3>

Global $hQuery, $aRow, $sMsg, $sSql, $dados_linha
Global $BASE_DB, $cod_cartela, $sorteado, $retarr

_SQLite_Startup(@ScriptDir & "\sqlite3.dll")

If Not FileExists("database.db") Then
    $dbn = _SQLite_Open("database.db")
    _SQLite_Exec($dbn, "CREATE TABLE Dados (Numeros);")
    For $i = 1 To 20
        $dados = "02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q" & $i
        $sql = "INSERT INTO Dados VALUES('" & $dados & "');"
        _SQLite_Exec($dbn, $sql)
    Next
Else
    $dbn = _SQLite_Open("database.db")
EndIf

$sorteado = "06 "
$trocar = "| "
_SQLite_Exec($dbn, "UPDATE Dados set Numeros= " & _SQLite_FastEscape($sorteado) & " where Numeros= " & _SQLite_FastEscape($trocar)); exchange all numbers for "| "
$aResult = _SearchDatabase($trocar); Looking for line that has "| "
If UBound($aResult) > 1 Then
    MsgBox(4096, 'OK', 'All numbers found have been replaced by "| "', 2)
    _ArrayDisplay($aResult)
Else
    MsgBox(4096, 'ERROR', $sorteado & ' It was not exchanged for "| "')
    $aResult = _SearchDatabase($sorteado)
    _ArrayDisplay($aResult)
EndIf

_SQLite_Close()
_SQLite_Shutdown()

Func _SearchDatabase($sString)
    Local $sql, $sSql, $aTables, $aResult, $iState
    Local $tmp1, $tmp2
    $sql = "SELECT Numeros, ""Dados"" AS ""table"", rowid FROM  ""Dados"""
    $sSql = "SELECT * FROM (" & $sql & ") WHERE Numeros LIKE '%" & $sString & "%';"
    $iState = _SQLite_GetTable2d($dbn, $sSql, $aResult, $tmp1, $tmp2)
    If $iState <> $SQLITE_OK Then
        Return SetError(@error, 0, $iState)
    Else
        Return $aResult
    EndIf
EndFunc   ;==>_SearchDatabase

 

Edited by Belini

Share this post


Link to post
Share on other sites

The DB rows in your example look like this:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q2'
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q3'
...
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q20'
 

Your SQL update statement is:
UPDATE Dados set Numeros= '06 ' where Numeros= '| '

No surprise no row is updated since no row equals '| '

If you want to update rows containing the string '06 '  and replace this part of the string by '| ', then you should do:

UPDATE Dados set Numeros= replace(numeros, '06 ', '| ') where Numeros like '%06 %'


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Beware that in 100k rows you don't have '06 ' substrings in other places in the strings, else:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 106 PM-Q12345'

will give:
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 1| PM-Q12345'

As always, know your own data!


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
This really happened and now i'm researching with space before and after the number, now it works perfectly.
Edited by Belini

Share this post


Link to post
Share on other sites

Exchanging data and searching using the hard drive's file is taking too long, how could I open this file from the disk in memory and make all the changes while it is open without writing to the disk?

EDITED: I managed to create a database copy that is on the hard drive and run it in memory.

_SQLite_Startup()
Local $dbn = _SQLite_Open()
_SQLite_Exec($dbn, "ATTACH DATABASE '" & @ScriptDir & "\database.db' as Original;")
_SQLite_Exec($dbn, "CREATE TABLE Dados AS SELECT * FROM Original.Dados;")

 

Edited by Belini

Share this post


Link to post
Share on other sites

Now I need to save what is in memory to the hard drive and I always have an error, can someone give a tip?

Local $dbdisk = _SQLite_Open(@ScriptDir & "\database.db")
_SQLite_Exec($dbdisk, "ATTACH DATABASE ':memory:' as Original;")
_SQLite_Exec($dbdisk, "CREATE TABLE Dados AS SELECT * FROM Original.Dados;")

I always get an error that the table was not found but in memory the table exists because it was created and populated.

Quote

!   SQLite.au3 Error
--> Function: _SQLite_Exec
--> Query:    CREATE TABLE Dados AS SELECT * FROM Original.Dados;
--> Error:    no such table: Original.Dados

 

Share this post


Link to post
Share on other sites

I don't know how the two code parts posted above articulate. If they are just separated by code to perform the replacement in memory, then you do this backwards.

What you can try:

-) open the disk DB.
-) attach a :memory: db, name it MEM and copy the disk table to it
-) create a new table in MEM defined as: create table mem.changes (id integer primary key, numeros char);
-) perform the search/replace on MEM
this way: select rows containing the searched value with a CTE and insert them in mem.changes after replacing the value; don't forget to set mem.id = rowid of the mem.dados table
-) rewrite only changes to the disk db: use a CTE on mem.changes to update dados set numeros = <cte table data> where rowid = <cte id value>

But I suspect there may be a much more efficient way to store the data.  Can you tell what the numeric values in numeros mean and which range they cover? It looks like the data is some kind of array, something SQL isn't particularly good at handling.

What I have in mind is along the following example, based on the data in the first post:
Instead of
'02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
it could be advantageous to store instead:
rowid, '00100110100000010000100100101010000110100001010100001011100000000010100100119', 'PM-Q1'
because then an index can then be used on the column numeros.

Explanations in the code below:

#include <Array.au3>

Local $s = '02 05 06 08 15 20 23 26 28 30 35 36 38 43 45 47 52 54 55 56 66 68 71 74 75 PM-Q1'
Local $a = StringSplit($s, ' ', 3)
_ArrayDisplay($a)
; the text part is last and there are ubound($a) entries
; create a string representing the values: 0 means the index isn't present, 1 means it is present, 2 means a | and 9 means end of string
; assuming the numeric values are in range [0..75] here
; assuming the values can't be too large (the string length derives from the largest value)
; also assuming the values are unique in a given entry
; values may appear in any order, since they are used as indices in the resulting string

; TO DO: if an original value is | then it can't be used as an index and the most logical way to denote that would be to store 2 (= |) in all positions between previous and next values
Local $data[$a[UBound($a) - 2] + 2]
For $i = 0 To UBound($data) - 2
    $data[$i] = '0'
Next
$data[UBound($data) - 1] = '9'
_ArrayDisplay($data)
For $i = 0 To UBound($a) - 2
    $data[$a[$i]] = '1'
Next
_ArrayDisplay($data)
Local $b = _ArrayToString($data, '')
Local $name = $a[UBound($a) - 1]
; show what the resulting data is:
ConsoleWrite($b & @TAB & $name & @LF)

Exit

#cs
; we're ready to store that entry to a DB
; the table should be declared this way:
"create table dados (id integer primary key, numeros char, name char);"
"create index num on dados (numeros);"                                      ; make it easy to search on a given value in numeros
"create index nam on dados (name);"                                         ; optional to find rows on name part
"insert into dados (numeros, name) value ('" & $b & "', '" & $name & "');"

; The table can then be searched more efficiently using the index
; for instance, to find rows with value '06', we do:
"select * from dados where numeros like '0000001%';"
; to update rows with '06' to '|' we do a CTE with the above select and perform the replace in string:
"update dados set numeros = substr(<numeros from cte>, <>, <>) || '2' || substr(<numeros from cte>, <>, <>) where id = <id from cte>;"
; this can also be done with a single printf()

; converting the string of 0/1 to binary would use less storage but wouldn't help searching as blobs can't be indexed
#ce

 


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

@jhcd I do need to copy a database that is on disk to memory and this I have already managed to do, after it is in memory I need to add more information in this database and then save a new file on the hard disk, I want to load it into memory to make the process faster and I’ll save it to disk only when I’ve finalized all the changes.

Quote

Can you tell what the numeric values in numeros mean and which range they cover?

These are numbers from a drawing that I need to check automatically each time a number with a space before and after it is typed " 30 " in every database I change " 30 " to " | "

Edited by Belini

Share this post


Link to post
Share on other sites

OK, then you can just use the backup function back and forth.

 


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I had already tried this but I get error messages when I run SQLitebackupTest

Quote

! ***************************************************************************************************************
! * Input file is UTF8 without BOM encoded, Obfuscator do not support UNICODE and will be skipped.      *
! * The file SHOULD BE encoded as UTF8 with BOM to continue processing by AutoIT3Wrapper.                       *
! *    #####################################################################################################    *
! * ##### AutoIt3Wrapper will not show a GUI or update the script to avoid any damage to your scriptfile. ##### *
! *    #####################################################################################################    *
! * When your file isn't a UTF8 file without BOM then please report this to me for review.                      *
! ***************************************************************************************************************

 

Share this post


Link to post
Share on other sites

Strange. Just save the file as UTF8 + BOM then.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I was unable to change to UTF8 + BOM in version 3.3.9.4 of Autoit that I use, there is no simpler way to just save the database that is in memory to the hard drive after having made all the changes to this file that is being executed in the memory?

Share this post


Link to post
Share on other sites

You should upgrade to the latest AutoIt version!

You can as well do what you say by dropping the DB table and recreating it from the memory table.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I would really like to be able to continue using version 3.3.9.4 because it is the last one that still obfuscates the script after compiled, I know that this does not solve for those with advanced knowledge but it keeps away curious people who decompile scripts, any other way to save the bank of data I have in memory for the hard drive would be welcome but I cannot change the version I use Autoit to do this.

Share this post


Link to post
Share on other sites

Why not just read the UDF and save it as ASCII?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I was able to write the database that was open in memory to the hard disk thanks to @Manimal that gave me the tip to use the command "VACUUM INTO"

_SQLite_Exec($hBD_Mem, "VACUUM INTO '" & @ScriptDir & "\Bd_Copy.db'")

 

Share this post


Link to post
Share on other sites

Ah yes, that's a pretty recent feature I never had a use for.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...