Jump to content
Sign in to follow this  

Best way to handle large JSON data? MAP still supported?

Recommended Posts

Hello.  I have a program that has used ADO database connection to return a query and then subsequently put the query results into an array using getrows.  See snippet below:

$constrim="DRIVER={SQL Server};SERVER=xxx-xxxxx\CSC;DATABASE=xxxxxxxxx;uid=xxxxxxxxxx;pwd=xxxxxxxxxxxx;"
$adCN = ObjCreate ("ADODB.Connection") ; <== Create SQL connection
$adCN.Open ($constrim) ;
local $sQuery = "select * from tbl_Apps"                     ; get all applications in the database
local $oAppRecordSet = $adCN.Execute($sQuery)
local $aAppsInDB = $oAppRecordSet.Getrows(5000)

With the code above I can perform array operations very efficiently. 

Now I need to get this data via JSON which is working (using Ward's JSON UDF)  but the data set returned is large (approx 4MB) and I'm wondering what is the most efficient way to get this data into an array.  

Dim $obj = ObjCreate ("WinHttp.WinHttpRequest.5.1")
$obj.Open("GET", $URL, false)
$obj.SetRequestHeader("Content-Type", "application/json")
$json = JSON_decode( $obj.ResponseText )

Any help would be appreciated!!

Share this post

Link to post
Share on other sites

So what's your problem with Ward's UDF?
It seem to work for you but you search a faster solution - right?

Maybe my JSON-UDF is a good try for you.
This UDF convert a JSON-string into nested AutoIt-datastructures (strings, numbers, bools, arrays, dictionaries).

But it shouldn't be faster than Ward's because it's written in native AutoIt. Ward's based on a professional external JSON-parser.

Anyway - here's an example of how to use:

#include <JSON.au3>
Global $s_String = '[{"id":"4434156","url":"https://legacy.sky.com/v2/schedules/4434156","title":"468_CORE_1_R.4 Schedule","time_zone":"London","start_at":"2017/08/10 19:00:00 +0100","end_at":null,"notify_user":false,"delete_at_end":false,"executions":[],"recurring_days":[],"actions":[{"type":"run","offset":0}],"next_action_name":"run","next_action_time":"2017/08/10 14:00:00 -0400","user":{"id":"9604","url":"https://legacy.sky.com/v2/users/9604","login_name":"robin@ltree.com","first_name":"Robin","last_name":"John","email":"robin@ltree.com","role":"admin","deleted":false},"region":"EMEA","can_edit":true,"vm_ids":null,"configuration_id":"19019196","configuration_url":"https://legacy.sky.com/v2/configurations/19019196","configuration_name":"468_CORE_1_R.4"},{"id":"4444568","url":"https://legacy.sky.com/v2/schedules/4444568","title":"468_CORE_1_R.4 Schedule","time_zone":"London","start_at":"2017/08/11 12:00:00 +0100","end_at":null,"notify_user":false,"delete_at_end":false,"executions":[],"recurring_days":[],"actions":[{"type":"suspend","offset":0}],"next_action_name":"suspend","next_action_time":"2017/08/11 07:00:00 -0400","user":{"id":"9604","url":"https://legacy.sky.com/v2/users/9604","login_name":"robin@ltree.com","first_name":"Robin","last_name":"John","email":"robin@ltree.com","role":"admin","deleted":false},"region":"EMEA","can_edit":true,"vm_ids":null,"configuration_id":"19019196","configuration_url":"https://legacy.sky.com/v2/configurations/19019196","configuration_name":"468_CORE_1_R.4"}]'

; ================= parse JSON-string into AutoIt-datatypes (+Scripting.Dictionary) ==============
$o_Object = _JSON_Parse($s_String)

; ================= queries directly in AutoIt-syntax ================================
$s_Type = $o_Object[1].Item("actions")[0].Item("type")
ConsoleWrite("type: " & $s_Type & @CRLF)

; ================= queries with function JSON_Get (safer) ================================
$s_Type = _JSON_Get($o_Object, "[1].actions[0].type")
ConsoleWrite("type: " & $s_Type & @CRLF)

; ================= convert nestest AutoIt-datastructure into a JSON-string ==============
ConsoleWrite(_JSON_Generate($o_Object) & @CRLF & @CRLF)
; compact JSON:
ConsoleWrite(_JSON_Generate($o_Object, "", "", "", "", "", "") & @CRLF & @CRLF)










Edited by AspirinJunkie
wrong file attached / now faster on parsing arrays / even faster array processing

Share this post

Link to post
Share on other sites

Maybe Chilkat free activex component and my UDF ?



Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest beginning - communication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2019-10-01

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By argumentum
      #include <SQLite.au3> ;-- When SQLite is compiled with the JSON1 extensions it provides builtin tools ;-- for manipulating JSON data stored in the database. ;-- This is a gist showing SQLite return query as a JSON object. ;-- https://www.sqlite.org/json1.html Example() Func Example() _SQLite_Startup() ; "<your path>\sqlite3.dll", False, 1) ; https://www.autoitscript.com/autoit3/docs/libfunctions/_SQLite_Startup.htm _SQLite_Open() ; ...if you can not run this due to errors, get the latest DLL from https://www.sqlite.org/ If _SQLite_Exec(-1, "CREATE TABLE users (id INTEGER PRIMARY KEY NOT NULL, full_name TEXT NOT NULL, email TEXT NOT NULL, created DATE NOT NULL );") Then Return 4 If _SQLite_Exec(-1, 'INSERT INTO users VALUES ' & _ '(1, "Bob McFett", "bmcfett@hunters.com", "32-01-01"),' & _ '(2, "Angus O''Vader","angus.o@destroyers.com", "02-03-04"),' & _ '(3, "Imperator Colin", "c@c.c", "01-01-01");') Then Return 5 ; -- Get query data as a JSON object using the ; -- json_group_object() [1] and json_object() [2] functions. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_object(" & _ " email," & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS json_result" & _ " FROM (SELECT * FROM users WHERE created > ""02-01-01"");") ; {"bmcfett@hunters.com":{"full_name":"Bob McFett","created":"32-01-01"},"angus.o@destroyers.com":{"full_name":"Angus O'Vader","created":"02-03-04"}} ; -- Get query data as a JSON object using the ; -- json_group_array() function to maintain order. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_array(" & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS my_json_result_OrAnythingReally" & _ " FROM (SELECT * FROM users ORDER BY created);") ; [{"full_name":"Imperator Colin","created":"01-01-01"},{"full_name":"Angus O'Vader","created":"02-03-04"},{"full_name":"Bob McFett","created":"32-01-01"}] ;-- Links ;-- [1] https://www.sqlite.org/json1.html#jgroupobject ;-- [2] https://www.sqlite.org/json1.html#jobj ; example found at https://gist.github.com/akehrer/481a38477dd0518ec0086ac66e38e0e2 EndFunc ;==>Example Func _SQLite_GetTable2d_ArrayToConsole($sSQL, $hDB = -1) Local $aResult, $iRows, $iColumns If _SQLite_GetTable2d($hDB, $sSQL, $aResult, $iRows, $iColumns) Then ConsoleWrite("! SQLite Error: " & _SQLite_ErrCode($hDB) & @CRLF & "! " & _SQLite_ErrMsg($hDB) & @CRLF) Else _SQLite_Display2DResult($aResult) EndIf ConsoleWrite(@CRLF) EndFunc ;==>_SQLite_GetTable2d_ArrayToConsole Based on this example, you can build your own query. 
      The code has all the explanations.
    • By matwachich
      Hi AutoIters!
      Here is my new UDF about GUIs: it's an enhanced mixture of Advanced InputBox (deprecated) and KODA Parser (deprecated), with additional functions.
      What you can do with it:
      Parse KODA files and directly create GUIs (_GUIUtils_CreateFromKODA) Parse a simple JSON form definition to simply create advanced InputBoxes with any amount/type of input controls (_GUIUtils_CreateFromJSON) Created GUIs are returned as Scripting.Dictionary objects, and you have helper functions to access GUIs controls by their names (_GUIUtils_HWnd, _GUIUtils_CtrlID, _GUIUtils_HCtrl ...) A function that can make a GUI created from KODA/JSON (defined by it's $oForm object) and make it a modal InputDialogBox (just like InputBox, but returns all entered data as Scripting.Dictionary object) Functions are documented, and there are some examples.
      Consider this UDF as beta, but since I'm currently using it in a small commercial project, it should become production ready in near future.
      To always get the latest code version, get it from Github (more up to date than this topic).
      Update 04/02/2020:
      New simple application example Fixed focused control handling in _GUIUtils_InputDialog Reset input values before returning from _GUIUtils_InputDialog Fixed CloseOnEsc in _GUIUtils_InputDialog Fixed Input not reset Fixed all ListBox items are selected Project now on GitHub https://github.com/matwachich/autoit-guiutils/
      Update 05/02/2020:
      Support for nodate for Date and Time input controls (pass null value) New handling of focused control _GUIUtils_InputDialog: Now, you can specify the focused control in $oInitialData by setting "controlName:focus" = True Updated documentation of _GUIUtils_InputDialog Update 07/02/2020:
      Bug corrected when setting Data input control Added: abillity to read a single input control New functions (_GUIUtils_GetInputs, _GUIUtils_WriteInputs), documentation completion. Readme file and screenshots on Github Page  
    • By Roy_
      This example reproduces the problem I encounter when sending a post request to an echo server.
      Instead of being sent as text utf8, the data is transmitted base64 encoded.
      - json String: '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'
      How can I go about transmitting data in text format utf8?
      Thanks in advance for the help.
      In the zip file:
      - CurlJsonPost.au3: this script
      - Curl.au3: UDF by Ward (thank you!)
      - data.json: json srting for command line test
      Note: using Curl.exe with json string saved in a file (utf8 encoded) named data.json, works perfectly
      To try with the command line tool:
      - save data.json in curl\bin directory
      - open cmd.exe and cd to curl\bin directory
      - Enter the following command:
      curl -H "Content-Type: application/json" --data @data.json https://httpbin.org/post
      #Include "Curl.au3" Global $_cURL_OutputBuffer Local $sJson = '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}' Local $iRetCode, $sServerResponse ConsoleWrite("=== Json post test ===" & @LF) ConsoleWrite(StringFormat("Retcode: %s - %s", $iRetCode, Curl_Easy_strerror($iRetCode)) & @LF) ConsoleWrite("Data returned from server" & @LF & @LF) $sServerResponse = JsonPost_Test($sJson, $iRetCode) ConsoleWrite($sServerResponse & @LF) Func JsonPost_Test($sJson, ByRef $iRetCode) ; Init Easy Curl Interface e set url (echo service) Local $oCurl = Curl_Easy_Init() curl_easy_setopt($oCurl, $CURLOPT_URL, "https://httpbin.org/post") ; Set content type header Local $headers = curl_slist_append(0, "Content-Type: application/json") curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers) ; Post fields & size curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson) curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDSIZE, StringLen($sJson)) ; Set callbac function to get server response back (see global var $_cURL_OutputBuffer) $hWriteFunc = DllCallbackRegister("WriteFunc_CallBack", "uint:cdecl", "ptr;uint;uint;ptr") curl_easy_setopt($oCurl, $CURLOPT_WRITEFUNCTION, DllCallbackGetPtr($hWriteFunc)) ; Ignore ssl certificates check curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYPEER, 0) curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYHOST, 0) ; Execute the post request $iRetCode = curl_easy_perform($oCurl) ; Set return trasfer & clear output buffer global var Local $sReturnTransfer = $_cURL_OutputBuffer $_cURL_OutputBuffer = "" Return SetError(0, 0, $sReturnTransfer) EndFunc Func WriteFunc_CallBack($ptr,$nSize,$nMemb,$pStream) Local $vData = DllStructCreate ("byte[" & $nSize*$nMemb & "]",$ptr) $_cURL_OutputBuffer &= BinaryToString(DllStructGetData($vData,1)) Return $nSize*$nMemb EndFunc  
    • By D2thunder
      Hi all, I wasnt active in this Forum forum for Years. 😅
      Since I need something with Json in AutoIt i write a little UDF to save a variable to file to read it with another language - and back. After a while until my other script grows i thought - lets integrate all variables wich is possible, because i want have arrays to.
      This is my first public UDF in Autoit - normaly my AutoIt scripts are quick an dirty.... - like the examples for this udf. - someone want to write nice examples?
      I tried to write this UDF properly, cared for processing speed, low mem consumption as possible, a short code, and a high usability without beeing very familiar with arrays.
      What is it for? - convert nearly all types of variables to a json string for saving it in a file or to an ini. Read it back in to a same formated variable and preserv variable types. All array types are supported with nested arrays, the only restriction is ram and array/stringsize - and time if the array is very big. You can use Filters for variable types and some other options.
      If anyone found a bug or any suggestions, please post it.
      get Json.au3 from here Json.au3
      JsonVar 2019.01.25.1.zip
  • Create New...