Jump to content
argumentum

SQLite returning query as JSON

Recommended Posts

#include <SQLite.au3>

;-- When SQLite is compiled with the JSON1 extensions it provides builtin tools
;-- for manipulating JSON data stored in the database.
;-- This is a gist showing SQLite return query as a JSON object.
;-- https://www.sqlite.org/json1.html

Example()
Func Example()
    _SQLite_Startup() ; "<your path>\sqlite3.dll", False, 1) ; https://www.autoitscript.com/autoit3/docs/libfunctions/_SQLite_Startup.htm
    _SQLite_Open() ;  ...if you can not run this due to errors, get the latest DLL from https://www.sqlite.org/
    If _SQLite_Exec(-1, "CREATE TABLE users (id INTEGER PRIMARY KEY NOT NULL, full_name TEXT NOT NULL, email TEXT NOT NULL, created DATE NOT NULL );") Then Return 4
    If _SQLite_Exec(-1, 'INSERT INTO users VALUES ' & _
            '(1, "Bob McFett", "bmcfett@hunters.com", "32-01-01"),' & _
            '(2, "Angus O''Vader","angus.o@destroyers.com", "02-03-04"),' & _
            '(3, "Imperator Colin", "c@c.c", "01-01-01");') Then Return 5

;   -- Get query data as a JSON object using the
;   -- json_group_object() [1] and json_object() [2] functions.
    _SQLite_GetTable2d_ArrayToConsole("SELECT" & _
            " json_group_object(" & _
            "   email," & _
            "   json_object('full_name', full_name, 'created', created)" & _
            " ) AS json_result" & _
            " FROM (SELECT * FROM users WHERE created > ""02-01-01"");")
    ; {"bmcfett@hunters.com":{"full_name":"Bob McFett","created":"32-01-01"},"angus.o@destroyers.com":{"full_name":"Angus O'Vader","created":"02-03-04"}}


;   -- Get query data as a JSON object using the
;   -- json_group_array() function to maintain order.
    _SQLite_GetTable2d_ArrayToConsole("SELECT" & _
            " json_group_array(" & _
            "   json_object('full_name', full_name, 'created', created)" & _
            " ) AS my_json_result_OrAnythingReally" & _
            " FROM (SELECT * FROM users ORDER BY created);")
    ; [{"full_name":"Imperator Colin","created":"01-01-01"},{"full_name":"Angus O'Vader","created":"02-03-04"},{"full_name":"Bob McFett","created":"32-01-01"}]

;-- Links
;-- [1] https://www.sqlite.org/json1.html#jgroupobject
;-- [2] https://www.sqlite.org/json1.html#jobj
; example found at https://gist.github.com/akehrer/481a38477dd0518ec0086ac66e38e0e2
EndFunc   ;==>Example

Func _SQLite_GetTable2d_ArrayToConsole($sSQL, $hDB = -1)
    Local $aResult, $iRows, $iColumns
    If _SQLite_GetTable2d($hDB, $sSQL, $aResult, $iRows, $iColumns) Then
        ConsoleWrite("! SQLite Error: " & _SQLite_ErrCode($hDB) & @CRLF & "! " & _SQLite_ErrMsg($hDB) & @CRLF)
    Else
        _SQLite_Display2DResult($aResult)
    EndIf
    ConsoleWrite(@CRLF)
EndFunc   ;==>_SQLite_GetTable2d_ArrayToConsole

Based on this example, you can build your own query. 
The code has all the explanations.
Enjoy :) 

Share this post


Link to post
Share on other sites

Thank you for the examples; I battled to wrap my head around JSON group objects and ended up doing it the looong way :sweating:

Share this post


Link to post
Share on other sites

Nice, but as I said in another thread, you have to build your own formatting depending on each query, just to put column names in the resulting json.

An ideal solution would be a function like this:

select jsonify(select ...) whatever the inner select is.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
Posted (edited)

...same here @dmob, but I've got lucky. :P 
 

1 hour ago, jchd said:

An ideal solution would be a function like this: select jsonify(select ...) whatever the inner select is.

yes. And that is what I did  something along the idea is what I did for my testing tho, is not a general purpose func. , just a hack to get it out of my way:

Func JsonIze_Query($s, $sQuery)
    Local $s_json_object_Str = "", $sSQL = "SELECT json_group_array( json_object("
    Local $a_json_object_Str = StringSplit($s, ",")
    For $n = 1 To $a_json_object_Str[0]
        $s_json_object_Str &= ",'" & $a_json_object_Str[$n] & "'," & $a_json_object_Str[$n]
    Next
    $sSQL &= StringTrimLeft($s_json_object_Str, 1) & ")) AS json_result FROM (" & $sQuery & ");"
    ConsoleWrite($sSQL & @CRLF)
    Return $sSQL
EndFunc

 

Edited by argumentum
clarify a bit

Share this post


Link to post
Share on other sites

I'm not trying to belittle your good code, not at all. What I find unpractical but inevitable is that you still have to setup both $s and $sQuery in paired terms for every distinct query. Either this way, or merge both when typing the query as in the other thread.

Deriving $s from the analysis of $sQuery automagically is way beyond doable for the general case (SQL grammar isn't simple!).

Passing the result array to an extension function in plain C in order to do the json formatting quickly is hard due to AutoIt using Variant type.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

..basically this is just "SELECT json_group_array( json_object(" & "'name', name, 'etc', etc" & ")) AS json_result FROM (" & $sSQL & ");"
Coding a table_names_generator() can get cumbersome and mainly, slow, just to get the "'name', name, 'etc', etc" part. And that string will be used for every call to the same query.
I love automagical functions but given the time cost of AutoIt executions, in the aim of speed, coding such would defeat the purpose of a speedy script.
The loop I made to take an array and generate a json string was practical but slow, so one has to choose: hard code or magical function. What is faster.

Finding a query that makes the resultant faster, that in itself I'd call "magical query" ( as I did not come up with it solely on my own ), so, I saved around 200 ms. ( in my query ) by having SQLite formatting the return value.

And I feel I know you better than believing that you would aim to belittle anyone. Moreover, you strive to enlighten everyone in every posting, so, thanks for taking the time to evaluate possibilities.

Now, if an "ASM god" comes up with a _SQLite_GetTable2d_array_to_json(), now that would be vary much welcomed :) 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By argumentum
      #include <SQLite.au3> ;~ #include <SQLite.dll.au3> Local $hQuery, $aRow, $aNames _SQLite_Startup() ConsoleWrite("_SQLite_LibVersion=" & _SQLite_LibVersion() & @CRLF) _SQLite_Open() ; open :memory: Database _SQLite_Exec(-1, "CREATE TABLE aTest (A,B int not null unique ,C text);") _SQLite_Exec(-1, "INSERT INTO aTest(a,b,c) VALUES ('c','2','World');") _SQLite_Exec(-1, "INSERT INTO aTest(a,b,c) VALUES ('b','3',' ');") _SQLite_Exec(-1, "INSERT INTO aTest(a,b,c) VALUES ('a','1','Hello');") _SQLite_Query(-1, "SELECT _ROWID_,* FROM aTest ORDER BY a;", $hQuery) _SQLite_FetchTypes($hQuery, $aNames) ; Read out Column Types ConsoleWrite(StringFormat(" %-10s %-10s %-10s %-10s ", $aNames[0], $aNames[1], $aNames[2], $aNames[3]) & @CRLF) _SQLite_FetchNames($hQuery, $aNames) ; Read out Column Names ConsoleWrite(StringFormat(" %-10s %-10s %-10s %-10s ", $aNames[0], $aNames[1], $aNames[2], $aNames[3]) & @CRLF) While _SQLite_FetchData($hQuery, $aRow) = $SQLITE_OK ConsoleWrite(StringFormat(" %-10s %-10s %-10s %-10s ", $aRow[0], $aRow[1], $aRow[2], $aRow[3]) & @CRLF) WEnd _SQLite_Exec(-1, "DROP TABLE aTest;") _SQLite_Close() _SQLite_Shutdown() ; Output: ; INTEGER int text ; rowid A B C ; 3 a 1 Hello ; 2 b 3 ; 1 c 2 World Func _SQLite_FetchTypes($hQuery, ByRef $aTypes) Dim $aTypes[1] If __SQLite_hChk($hQuery, 3, False) Then Return SetError(@error, 0, $SQLITE_MISUSE) Local $avDataCnt = DllCall($__g_hDll_SQLite, "int:cdecl", "sqlite3_column_count", "ptr", $hQuery) If @error Then Return SetError(1, @error, $SQLITE_MISUSE) ; DllCall error If $avDataCnt[0] <= 0 Then Return SetError(-1, 0, $SQLITE_DONE) ReDim $aTypes[$avDataCnt[0]] Local $avColName For $iCnt = 0 To $avDataCnt[0] - 1 $avColName = DllCall($__g_hDll_SQLite, "wstr:cdecl", "sqlite3_column_decltype16", "ptr", $hQuery, "int", $iCnt) If @error Then Return SetError(2, @error, $SQLITE_MISUSE) ; DllCall error $aTypes[$iCnt] = $avColName[0] Next Return $SQLITE_OK EndFunc ;==>_SQLite_FetchTypes  If you wanna build a proper JSON string, you may want to know if is {"int":123} or {"text":"123"}
      and for that, this can help, obviously only when declared in the SQLite table.
    • By argumentum
      I was thinking but I don't have the experience, so you may have the experience.
      I was thinking to chop a DB with 100.000 ( 20 columns ) in 10.000 DB chunks ( 10 DBs ) and query concurrently all of them to speed up a search, then add the results. Is that a sound idea ? Or will I run in trouble down the road. Should a DB better be keep in one piece ?
      The DB is now in MySQL. I wanna do all this chopping and use SQLite. 
      Thanks
      [solved]
    • By matwachich
      Hi AutoIters!
      Here is my new UDF about GUIs: it's an enhanced mixture of Advanced InputBox (deprecated) and KODA Parser (deprecated), with additional functions.
      What you can do with it:
      Parse KODA files and directly create GUIs (_GUIUtils_CreateFromKODA) Parse a simple JSON form definition to simply create advanced InputBoxes with any amount/type of input controls (_GUIUtils_CreateFromJSON) Created GUIs are returned as Scripting.Dictionary objects, and you have helper functions to access GUIs controls by their names (_GUIUtils_HWnd, _GUIUtils_CtrlID, _GUIUtils_HCtrl ...) A function that can make a GUI created from KODA/JSON (defined by it's $oForm object) and make it a modal InputDialogBox (just like InputBox, but returns all entered data as Scripting.Dictionary object) Functions are documented, and there are some examples.
      Consider this UDF as beta, but since I'm currently using it in a small commercial project, it should become production ready in near future.
      To always get the latest code version, get it from Github (more up to date than this topic).
      Update 04/02/2020:
      New simple application example Fixed focused control handling in _GUIUtils_InputDialog Reset input values before returning from _GUIUtils_InputDialog Fixed CloseOnEsc in _GUIUtils_InputDialog Fixed Input not reset Fixed all ListBox items are selected Project now on GitHub https://github.com/matwachich/autoit-guiutils/
      Update 05/02/2020:
      Support for nodate for Date and Time input controls (pass null value) New handling of focused control _GUIUtils_InputDialog: Now, you can specify the focused control in $oInitialData by setting "controlName:focus" = True Updated documentation of _GUIUtils_InputDialog Update 07/02/2020:
      Bug corrected when setting Data input control Added: abillity to read a single input control New functions (_GUIUtils_GetInputs, _GUIUtils_WriteInputs), documentation completion. Readme file and screenshots on Github Page  
      GUIUtils.zip
    • By Roy_
      Ciao,
      This example reproduces the problem I encounter when sending a post request to an echo server.
      Instead of being sent as text utf8, the data is transmitted base64 encoded.
      Example:
      - json String: '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'
      - server response: "data:application/octet-stream;base64,AAAAALgE6QNYAIAAAgAAANQCAAAAAAAAWF8AAAAAAAD0VQAAAAAAAD9APQAAAAAAAwAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAABBAAAAAAABRUAAAAmIZwcAbRt8/LMa9UAAAAAAAAAAGFzIGlkIHRlbXB1cyBtaS4gTW9yYmkgaXBzdW0gZW5pbSwgdWx0cmljaWVzIGFjIGF1Z3VlIHNpdCBhbWV0LCB1bGxhbWNvcnBlciBmaW5pYnVzIGV4LiBWZXN0aWJ1bHVtIHZlbCBwb3N1ZXJlIG5pYmgsIG5lYyBmYXVjaWJ1cyBlcm9zLiBOYW0gbWFsZXN1YWRhIG5vbiBsYWN1cyBhIHN1c2NpcGl0LiBOdWxsYSByaG9uY3VzIHRlbXB1cyBtaSBxdWlzIHBsYWNlcmF0LiBDdXJhYml0dXIgY29tbW9kbyB0aW5jaWR1bnQganVzdG8gcXVpcyBzb2xsaWNpdHVkaW4uIn0="
       
      How can I go about transmitting data in text format utf8?
      Thanks in advance for the help.
      In the zip file:
      - CurlJsonPost.au3: this script
      - Curl.au3: UDF by Ward (thank you!)
      - data.json: json srting for command line test
       
      Note: using Curl.exe with json string saved in a file (utf8 encoded) named data.json, works perfectly
      To try with the command line tool:
      - save data.json in curl\bin directory
      - open cmd.exe and cd to curl\bin directory
      - Enter the following command:
      curl -H "Content-Type: application/json" --data @data.json https://httpbin.org/post
       
      #Include "Curl.au3" Global $_cURL_OutputBuffer Local $sJson = '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}' Local $iRetCode, $sServerResponse ConsoleWrite("=== Json post test ===" & @LF) ConsoleWrite(StringFormat("Retcode: %s - %s", $iRetCode, Curl_Easy_strerror($iRetCode)) & @LF) ConsoleWrite("Data returned from server" & @LF & @LF) $sServerResponse = JsonPost_Test($sJson, $iRetCode) ConsoleWrite($sServerResponse & @LF) Func JsonPost_Test($sJson, ByRef $iRetCode) ; Init Easy Curl Interface e set url (echo service) Local $oCurl = Curl_Easy_Init() curl_easy_setopt($oCurl, $CURLOPT_URL, "https://httpbin.org/post") ; Set content type header Local $headers = curl_slist_append(0, "Content-Type: application/json") curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers) ; Post fields & size curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson) curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDSIZE, StringLen($sJson)) ; Set callbac function to get server response back (see global var $_cURL_OutputBuffer) $hWriteFunc = DllCallbackRegister("WriteFunc_CallBack", "uint:cdecl", "ptr;uint;uint;ptr") curl_easy_setopt($oCurl, $CURLOPT_WRITEFUNCTION, DllCallbackGetPtr($hWriteFunc)) ; Ignore ssl certificates check curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYPEER, 0) curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYHOST, 0) ; Execute the post request $iRetCode = curl_easy_perform($oCurl) ; Set return trasfer & clear output buffer global var Local $sReturnTransfer = $_cURL_OutputBuffer $_cURL_OutputBuffer = "" Return SetError(0, 0, $sReturnTransfer) EndFunc Func WriteFunc_CallBack($ptr,$nSize,$nMemb,$pStream) Local $vData = DllStructCreate ("byte[" & $nSize*$nMemb & "]",$ptr) $_cURL_OutputBuffer &= BinaryToString(DllStructGetData($vData,1)) Return $nSize*$nMemb EndFunc  
      curlJsonPost.zip
    • By D2thunder
      Hi all, I wasnt active in this Forum forum for Years. 😅
      Since I need something with Json in AutoIt i write a little UDF to save a variable to file to read it with another language - and back. After a while until my other script grows i thought - lets integrate all variables wich is possible, because i want have arrays to.
      This is my first public UDF in Autoit - normaly my AutoIt scripts are quick an dirty.... - like the examples for this udf. - someone want to write nice examples?
      I tried to write this UDF properly, cared for processing speed, low mem consumption as possible, a short code, and a high usability without beeing very familiar with arrays.
      What is it for? - convert nearly all types of variables to a json string for saving it in a file or to an ini. Read it back in to a same formated variable and preserv variable types. All array types are supported with nested arrays, the only restriction is ram and array/stringsize - and time if the array is very big. You can use Filters for variable types and some other options.
      If anyone found a bug or any suggestions, please post it.
       
      get Json.au3 from here Json.au3
       
       
      JsonVar 2019.01.25.1.zip
×
×
  • Create New...