Jump to content

Best way to handle large JSON data? MAP still supported?

Recommended Posts


Hello.  I have a program that has used ADO database connection to return a query and then subsequently put the query results into an array using getrows.  See snippet below:

$constrim="DRIVER={SQL Server};SERVER=xxx-xxxxx\CSC;DATABASE=xxxxxxxxx;uid=xxxxxxxxxx;pwd=xxxxxxxxxxxx;"
$adCN = ObjCreate ("ADODB.Connection") ; <== Create SQL connection
$adCN.Open ($constrim) ;
local $sQuery = "select * from tbl_Apps"                     ; get all applications in the database
local $oAppRecordSet = $adCN.Execute($sQuery)
local $aAppsInDB = $oAppRecordSet.Getrows(5000)

With the code above I can perform array operations very efficiently. 

Now I need to get this data via JSON which is working (using Ward's JSON UDF)  but the data set returned is large (approx 4MB) and I'm wondering what is the most efficient way to get this data into an array.  

Dim $obj = ObjCreate ("WinHttp.WinHttpRequest.5.1")
$obj.Open("GET", $URL, false)
$obj.SetRequestHeader("Content-Type", "application/json")
$json = JSON_decode( $obj.ResponseText )

Any help would be appreciated!!

Share this post

Link to post
Share on other sites

So what's your problem with Ward's UDF?
It seem to work for you but you search a faster solution - right?

Maybe my JSON-UDF is a good try for you.
This UDF convert a JSON-string into nested AutoIt-datastructures (strings, numbers, bools, arrays, dictionaries).

But it shouldn't be faster than Ward's because it's written in native AutoIt. Ward's based on a professional external JSON-parser.

Anyway - here's an example of how to use:

#include <JSON.au3>
Global $s_String = '[{"id":"4434156","url":"https://legacy.sky.com/v2/schedules/4434156","title":"468_CORE_1_R.4 Schedule","time_zone":"London","start_at":"2017/08/10 19:00:00 +0100","end_at":null,"notify_user":false,"delete_at_end":false,"executions":[],"recurring_days":[],"actions":[{"type":"run","offset":0}],"next_action_name":"run","next_action_time":"2017/08/10 14:00:00 -0400","user":{"id":"9604","url":"https://legacy.sky.com/v2/users/9604","login_name":"robin@ltree.com","first_name":"Robin","last_name":"John","email":"robin@ltree.com","role":"admin","deleted":false},"region":"EMEA","can_edit":true,"vm_ids":null,"configuration_id":"19019196","configuration_url":"https://legacy.sky.com/v2/configurations/19019196","configuration_name":"468_CORE_1_R.4"},{"id":"4444568","url":"https://legacy.sky.com/v2/schedules/4444568","title":"468_CORE_1_R.4 Schedule","time_zone":"London","start_at":"2017/08/11 12:00:00 +0100","end_at":null,"notify_user":false,"delete_at_end":false,"executions":[],"recurring_days":[],"actions":[{"type":"suspend","offset":0}],"next_action_name":"suspend","next_action_time":"2017/08/11 07:00:00 -0400","user":{"id":"9604","url":"https://legacy.sky.com/v2/users/9604","login_name":"robin@ltree.com","first_name":"Robin","last_name":"John","email":"robin@ltree.com","role":"admin","deleted":false},"region":"EMEA","can_edit":true,"vm_ids":null,"configuration_id":"19019196","configuration_url":"https://legacy.sky.com/v2/configurations/19019196","configuration_name":"468_CORE_1_R.4"}]'

; ================= parse JSON-string into AutoIt-datatypes (+Scripting.Dictionary) ==============
$o_Object = _JSON_Parse($s_String)

; ================= queries directly in AutoIt-syntax ================================
$s_Type = $o_Object[1].Item("actions")[0].Item("type")
ConsoleWrite("type: " & $s_Type & @CRLF)

; ================= queries with function JSON_Get (safer) ================================
$s_Type = _JSON_Get($o_Object, "[1].actions[0].type")
ConsoleWrite("type: " & $s_Type & @CRLF)

; ================= convert nestest AutoIt-datastructure into a JSON-string ==============
ConsoleWrite(_JSON_Generate($o_Object) & @CRLF & @CRLF)
; compact JSON:
ConsoleWrite(_JSON_Generate($o_Object, "", "", "", "", "", "") & @CRLF & @CRLF)









Edited by AspirinJunkie
wrong file attached / now faster on parsing arrays / even faster array processing

Share this post

Link to post
Share on other sites

Maybe Chilkat free activex component and my UDF ?



Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2018-03-17

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • Cobra528
      By Cobra528
      Map feature would be really a nice thing in AutoIt as seen in the beta version.
      Will this functionality be taken up in a stable future release ?
      I do not ask when. I know the response ;-).
    • Ward
      By Ward
      JSON (Javascript Object Notation) is a popular data-interchange format and supported by a lot of script languages. On AutoIt, there is already a >JSON UDF written by Gabriel Boehme. It is good but too slow, and not supports unicode and control characters very well. So I write a new one (and of course, fast one as usual).

      I use a machine code version of JSON parser called "jsmn". jsmn not only supports standard JSON, but also accepts some non-strict JSON string. See below for example.
      Important Update!!
      I rename the library from jsmn.au3 to json.au3. All function names are changed, too.
      Decoding Function
      Json_Decode($Json) $Json can be a standard or non-standard JSON string. For example, it accepts:
      { server: example.com port: 80 message: "this looks like a config file" } The most JSON data type will be decoded into corresponding AutoIt variable, including 1D array, string, number, true, false, and null. JSON object will be decoded into "Windows Scripting Dictionary Object" retuned from ObjCreate("Scripting.Dictionary"). AutoIt build-in functions like IsArray, IsBool, etc. can be used to check the returned data type. But for Object and Null, Json_IsObject() and Json_IsNull() should be used.

      If the input JSON string is invalid, @Error will be set to $JSMN_ERROR_INVAL. And if the input JSON string is not finish (maybe read from stream?), @Error will be set to $JSMN_ERROR_PART.

      Encoding Function
      Json_Encode($Data, $Option = 0, $Indent = "\t", $ArraySep = ",\r\n", $ObjectSep = ",\r\n", $ColonSep = ": ") $Data can be a string, number, bool, keyword(default or null), 1D arrry, or "Scripting.Dictionary" COM object. Ptr will be converted to number, Binary will be converted to string in UTF8 encoding. Other unsupported types like 2D array, dllstruct or object will be encoded into null.

      $Option is bitmask consisting following constant:
      $JSON_UNESCAPED_ASCII ; Don't escape ascii charcters between chr(1) ~ chr(0x1f) $JSON_UNESCAPED_UNICODE ; Encode multibyte Unicode characters literally $JSON_UNESCAPED_SLASHES ; Don't escape / $JSON_HEX_TAG ; All < and > are converted to \u003C and \u003E $JSON_HEX_AMP ; All &amp;amp;amp;amp;s are converted to \u0026 $JSON_HEX_APOS ; All ' are converted to \u0027 $JSON_HEX_QUOT ; All " are converted to \u0022 $JSON_PRETTY_PRINT ; Use whitespace in returned data to format it $JSON_STRICT_PRINT ; Make sure returned JSON string is RFC4627 compliant $JSON_UNQUOTED_STRING ; Output unquoted string if possible (conflicting with $JSMN_STRICT_PRINT) Most encoding option have the same means like PHP's json_enocde() function. When $JSON_PRETTY_PRINT is set, output format can be change by other 4 parameters ($Indent, $ArraySep, $ObjectSep, and $ColonSep). Because these 4 output format parameters will be checked inside Jsmn_Encode() function, returned string will be always accepted by Jsmn_Decode(). $JSON_UNQUOTED_STRING can be used to output unquoted string that also accetped by Jsmn_Decode().

      $JSON_STRICT_PRINT is used to check output format setting and avoid non-standard JSON output. So this option is conflicting with $JSON_UNQUOTED_STRING.

      Get and Put Functions
      Json_Put(ByRef $Var, $Notation, $Data, $CheckExists = False) Json_Get(ByRef $Var, $Notation) These functions helps user to access object or array more easily. Both dot notation and square bracket notation can be supported. Json_Put() by default will create non-exists objects and arrays. For example:
      Local $Obj Json_Put($Obj, ".foo", "foo") Json_Put($Obj, ".bar[0]", "bar") Json_Put($Obj, ".test[1].foo.bar[2].foo.bar", "Test") Local $Test = Json_Get($Obj, '["test"][1]["foo"]["bar"][2]["foo"]["bar"]') ; "Test" Object Help Functions
      Json_ObjCreate() Json_ObjPut(ByRef $Object, $Key, $Value) Json_ObjGet(ByRef $Object, $Key) Json_ObjDelete(ByRef $Object, $Key) Json_ObjExists(ByRef $Object, $Key) Json_ObjGetCount(ByRef $Object) Json_ObjGetKeys(ByRef $Object) Json_ObjClear(ByRef $Object) These functions are just warps of "Scripting.Dictionary" COM object. You can use these functions if you are not already familiar with it.
      == Update 2013/05/19 ==
      * Add Jsmn_Encode() option "$JSMN_UNESCAPED_ASCII". Now the default output of Json_Encode() is exactly the same as PHP's json_encode() function (for example, chr(1) will be encoded into u0001).
      $JSON_UNESCAPED_ASCII ; Don't escape ascii charcters between chr(1) ~ chr(0x1f)  
      == Update 2015/01/08 ==
      * Rename the library from jsmn.au3 to json.au3. All function names are changed, too.
      * Add Json_Put() and Json_Get()
      * Add Null support
      * Using BinaryCall.au3 to loading the machine code.
      == Update 2018/01/13=   Jos
      * Add JsonDump() to list all Json Keys and their values to easily figure out what they are.

    • jesus40
      By jesus40
      Hello friends, I need help getting data from json. I read json from url into a variable.
      Is possible to get volume_USD value etc.  where product_id= "ETH-BTC", ? So I want get "37421.69109185" as variable.
      I think is better keeping it in the memory as variable. Or easier to convert it to db or csv?
      #include <Inet.au3> #include <json.au3> $URL="https://api.abucoins.com/products/stats" $data = _INetGetSource($URL) MsgBox("","",$data) $object=json_decode($data) local $i=0 while 1 $message=json_get($object,'[product_id]');???????????????????????????????? MsgBox("","volume_USD",$message) $i=$i+1 WEnd  
      the json structure is here:
      [ { "product_id": "ETH-BTC", "last": "0.05202015", "open": "0.05009375", "high": "0.05250000", "low": "0.04872420", "volume": "47.77440456", "volume_BTC": 2.48523169, "volume_USD": 37421.69109185, "volume_7d": "364.40240341", "volume_30d": "3555.83377072", "change": "3.85" }, { "product_id": "LTC-BTC", "last": "0.01737490", "open": "0.01744636", "high": "0.01779000", "low": "0.01658091", "volume": "67.62762830", "volume_BTC": 1.17502328, "volume_USD": 17941.95920676, "volume_7d": "611.57893068", "volume_30d": "3881.57766044", "change": "-0.41" }, { "product_id": "ETC-BTC", "last": "0.00191846", "open": "0.00189001", "high": "0.00198000", "low": "0.00162741", "volume": "66.48327742", "volume_BTC": 0.12754551, "volume_USD": 1947.54976323, "volume_7d": "738.94495089", "volume_30d": "4090.74712156", "change": "1.51" }, { "product_id": "ZEC-BTC", "last": "0.03466163", "open": "0.03365452", "high": "0.03466163", "low": "0.03295811", "volume": "33.40101560", "volume_BTC": 1.15773364, "volume_USD": 17677.95599574, "volume_7d": "191.15953471", "volume_30d": "887.70158297", "change": "2.99" } ]  
    • satanico64
      By satanico64
      hi guys !
      how are you ? fine ? yeahhh
      Here is my problem:
      I have 2 autoit applications, a.exe and b.exe
      In a.exe, i call b.exe with a json as parameter:
      FileWrite("N:\+++ Dev\ClientServeur\FolderMonitor.nca", "lejson in A:" & Json_Encode($json_tmp)& @CRLF) Run(@ScriptDir & "\FolderMonitor.exe " & Json_Encode($json_tmp) )                        
              In program B:
      ;Firstline of program: FileWrite("N:\+++ Dev\ClientServeur\FolderMonitor.nca", @CRLF & "json in B:" & $CmdLine[1] & @CRLF)        
              lejson in A:{"DOSSIERS":{"Dossier_Hidden":"N:\\+++ Dev\\ClientServeur\\","Dossier_du_script":"N:\\+++ Dev\\ClientServeur\\","Fichier_Historique":"N:\\+++ Dev\\ClientServeur\\Historique.log","Fichier_DEBUG":"N:\\+++ Dev\\ClientServeur\\FolderMonitor.nca"}}
              json in B  :{DOSSIERS:{Dossier_Hidden:N:\\+++ Dev\\ClientServeur\\,Dossier_du_script:N:\\+++ Dev\\ClientServeur\\,Fichier_Historique:N:\\+++ Dev\\ClientServeur\\Historique.log,Fichier_DEBUG:N:\\+++ Dev\\ClientServeur\\FolderMonitor.nca}}
      As you see, i lost my quote in my json so it's au problem to parse my json.
      I use THIS library
      I tried different type of encoding (ex JSON_STRICT_PRINT), but i don't think that it's the problem. Maybe i'm wrong
    • jandings
      By jandings
      Hello there,
      since I spent some time to access the REST-API of the web shop system Shopware, I'd like to share a few lines to make life easier for others.
      With this you can access your Shopware database, reading and writing all kind of data. 
      Replies are JSON style.
      To work with JSON I usually depend on either just own AutoIt string operations or this AutoIt library:
      ; File        : Json.au3 (2015.01.08)
      ; Purpose    : A Non-Strict JavaScript Object Notation (JSON) Parser UDF
      ; Author    : Ward
      $UserName="xxxx" ;Shopware credentials of Shopware user who has the API checkbox ticked $PassWord="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" ;API-Key of this user, NOT the Password $URL="www.XXX.com/api/" $Command="orders/64682?useNumberAsId=true" ;A list of commands and options is available through shopware REST-API help; Here we read order number 64682 $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", "https://" & $URL & $Command , False) $oHTTP.SetCredentials($UserName, $PassWord, 0) $oHTTP.Send() $response = $oHTTP.ResponseText ConsoleWrite ($response & @CRLF & @CRLF)  

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.