Jump to content
Ward

A Non-Strict JSON UDF (JSMN)

Recommended Posts

I guess that should be:

ConsoleWrite('output1: ' & Json_Get( $json2, '.mother.sosa') &  @LF )
ConsoleWrite('output: ' & Json_Get( $json2, ".lastname") &  @LF )

Jos

JSON blijft lastig :)


SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

 

19 hours ago, Jos said:

I guess that should be:

ConsoleWrite('output1: ' & Json_Get( $json2, '.mother.sosa') &  @LF )
ConsoleWrite('output: ' & Json_Get( $json2, ".lastname") &  @LF )

Jos

JSON blijft lastig :)

That worked, sorry that i missed that. But what is the dividends between those functions Json_Get and Json_ObjGet?  

Share this post


Link to post
Share on other sites

@Jos

Now i'm using the Packege UDF to store some txt files and read it, it's around 2 thousand files... to read one of the data it takes around 2secs. If i change it to json, it will be faster? I'm trying to create a database and i don't know what is the best option.

Share this post


Link to post
Share on other sites

I have no idea as your provided scenario is very explicit. :)
It probably all depends what kind of lexing you need to do with those text files when reading processing them.

Just give it a try and simply test it would be the best advice as that will give you a conclusive answer.

Jos


SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
On 1/13/2018 at 10:06 AM, Jos said:

Added JsonDump() and updated/attached a new Zipfile in first post

 

Jos

In _Json_TokenDump, could you put a Local in front of $cObjPath so it doesn't cause a warning when we require that variables be declared?

Func _Json_TokenDump(ByRef $Json, $Ptr, ByRef $Next, $ObjPath = "")
    If $Next = -1 Then Return Null

    Local $Token = DllStructCreate("int;int;int;int", $Ptr + ($Next * 20))
    Local $Type = DllStructGetData($Token, 1)
    Local $Start = DllStructGetData($Token, 2)
    Local $End = DllStructGetData($Token, 3)
    Local $Size = DllStructGetData($Token, 4)
    $Next += 1

    If $Type = 0 And $Start = 0 And $End = 0 And $Size = 0 Then ; Null Item
        $Next = -1
        Return Null
    EndIf

    Switch $Type
        Case 0 ; Json_PRIMITIVE
            Local $Primitive = StringMid($Json, $Start + 1, $End - $Start)
            Switch $Primitive
                Case "true"
                    Return True
                Case "false"
                    Return False
                Case "null"
                    Return Null
                Case Else
                    If StringRegExp($Primitive, "^[+\-0-9]") Then
                        Return Number($Primitive)
                    Else
                        Return Json_StringDecode($Primitive)
                    EndIf
            EndSwitch

        Case 1 ; Json_OBJECT
            For $i = 0 To $Size - 1 Step 2
                Local $Key = _Json_TokenDump($Json, $Ptr, $Next)
                Local $cObjPath = $ObjPath & "." & $Key                                     ;<--add Local here
                Local $Value = _Json_TokenDump($Json, $Ptr, $Next, $ObjPath & "." & $Key)
                If Not ($Value = False) Then
                    If Not IsString($Key) Then
                        $Key = Json_Encode($Key)
                    EndIf
                    ; show the key and its value
                    ConsoleWrite("+-> " & $cObjPath & '  =' & $Value & @CRLF)
                EndIf
            Next
            Return False
        Case 2 ; Json_ARRAY
            Local $sObjPath = $ObjPath
            Local $Array[$Size]
            For $i = 0 To $Size - 1
                $sObjPath = $ObjPath & "[" & $i & "]"
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $sObjPath)
                If $Value <> False Then
                    ; show the key and its value
                    ConsoleWrite("+=> " & $sObjPath & "=>" & $Value & @CRLF)
                EndIf
            Next
            $ObjPath = $sObjPath
            Return False

        Case 3 ; Json_STRING
            Local $LastKey = Json_StringDecode(StringMid($Json, $Start + 1, $End - $Start))
            Return $LastKey
    EndSwitch
EndFunc   ;==>_Json_TokenDump

 

Share this post


Link to post
Share on other sites

First, thank you for creating this UDF.. Hopefully, it will work for what I need it for. Second.. I know nothing about json files.. Having to learn for a project. I have a json with data similar to the following.

{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}

What I need to do is read the ID# and then take the largest # and add 1, then add a new section, with in this case id: 104.. name: Example 4 etc. Lastly write the json back to disk. Could I do this with this UDF?


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

That is why I made the json_dump() udf to help you with the logic of how the retrieve the appropriate field value:

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
ConsoleWrite("!== Json_TokenDump ============================================================================================================" & @CRLF)
Json_Dump($data)

Start there and post in case you still have questions. :)

Jos


SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

Yes, I played with the

Json_Dump($data)

and it produces and an output, but cannot figure out how to correlate that into variables I can manipulate.  Maybe I'm just blind? I mean I can single out items with this.

ConsoleWrite(Json_Get($Data1, '["items"][0]["id"]') & @LF)

but i'm looking for more of maybe $target = $Data1[items][$x][id] kinda thing so I can loop $x from 0 to 10 and get each id#? if that makes any sense? or is all the data from the decoded json stored in array? that then I can just push data into and convert the array back the json data.

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

Shouldn't be too hard, just look closely at the produced output in the dump! 
something like this you mean?  :) 

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
Local $i = 0
While 1
    $id = json_get($object, '.items[' & $i & '].id')
    If @error Then ExitLoop
    ConsoleWrite('id:' & $id & @CRLF)
    $i += 1
WEnd

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

That's it.. Thank you.. Helps make that data clearer.. I was in the process of attempting something similar with do/until but this works...

Side request. is the Json_put only designed to do one element at a time, or is there a preferred method to add a section?  Such as needing to add this..

{
      "id",104
      "name","Example 4"
      "hash",null
      "folder",null
      "download",null
      "version",null
      "website",null
      "files",[]
    }

 

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

Answered my own question from above..

 

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

Is parsing json data normally very slow? loading the file in...

Local $file_json_raw = FileRead($ini_file_path & "\data.json")
Local $file_json_data = Json_Decode($file_json_raw)

takes about 6 seconds.

Parsing the data into an array,

While 1 ; list of all servers
        $server_id = json_get($file_json_data, '.servers[' & $a & '].id')
        If ($server_id == "") Then ExitLoop
        $server_name = json_get($file_json_data, '.servers[' & $a & '].name')
        $server_name_override = json_get($file_json_data, '.servers[' & $a & '].nameoverride')
        If ($server_name_override <> "") Then $server_name = $server_name_override
    WEnd

takes about 30 seconds.

The json file is about 900mb, the server section consist of about 2100 entries.  Is there a quicker way to be handing the data?


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites
6 minutes ago, zone97 said:

Is parsing json data normally very slow?

Fact or opinion?  ;)

Parsing a 900Mb JSON encoded file doesn't sound as a common thing to me as I am used to using JSON in REST calls used for API's, which in general aren't that big.
Maybe you need you see whether you need to change your approach and use RegEx?

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
7 minutes ago, TheDcoder said:

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

I'm very new to using this UDF.. how do they differ? does json_objGet, create a 2d array of the values of a section? (because that is ultimately what the goal is for me.)


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites

@zone97 No, it is a wrapper for the Scripting.Dictionary object or simply a dictionary variable. You can access the values inside a dictionary by using the Json_ObjGet and Json_ObjPut functions, you should see the examples and read the code for those functions so you could understand better :)


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

I have found a couple of small issues with the Json_Dump() logic.  One issue is related to comparing different data types.  The other issue is probably just an oversight.

If you run the following example with the latest version of the json UDF, you will get the following console output:

#include "json.original.au3" ; <== Modify this line to point to your copy of json.udf
;~ #include "json.modified.au3" ; <== Modify this line to point to your copy of json.udf


example()

Func example()

    #cs - PrettyPrint of JSON const below
    {
        "null_item": null,
        "zero_primitive_item": 0,
        "false_item": false,
        "true_item": true,
        "string_item": "This is a string",
        "array_item": [0,1,2]
    }
    #ce
    
    Const $JSON_STRING = '{"null_item": null, "zero_primitive_item": 0,"false_item": false,"true_item": true,"string_item": "This is a string","array_item": [0,1,2]}'
    
    Json_Dump($JSON_STRING)

EndFunc
+-> .null_item  =
+-> .true_item  =True
+-> .string_item  =This is a string
+=> .array_item[1]=>1
+=> .array_item[2]=>2

As you can see, items that have a primitive value of 0 or false, do not get written out at all.  This was because of comparing different data types.  Also notice that values that are null do not display "null", they simply display no value.  Since it is a dump, I would assume that you would want to see all values. 

I have made changes to 5 lines of code in the UDF.  When run with my modifications, you get the following output.  All lines get displayed with their correct literal values.

+-> .null_item  =null
+-> .zero_primative_item  =0
+-> .false_item  =false
+-> .true_item  =true
+-> .string_item  =This is a string
+=> .array_item[0]=>0
+=> .array_item[1]=>1
+=> .array_item[2]=>2

I tried to leave the existing code as close to original as possible.  In other words, instead of rewriting the functions, I made the smallest modifications possible in order to get the correct result. 

I have attached the original and the modified versions of the UDF so that you can easily do a DIFF to see the changes. Each of the modified lines is also denoted by a comment that says ";XC - Changed line" to make them easy to FIND.

 

I just notice that I did make a few more changes to the UDF.  I added an extra function, "Json_ObjGetItems(ByRef $Object)".  Because I use very strict directives, I made sure that all variables were declared before being used so I would stop getting warnings during compilation.

 

Hopefully my modifications pass muster and can be added to the UDF.

 

Json.modified.au3

Json.original.au3

Edited by TheXman
Added that I made a few additional changes to the UDF and why.

Share this post


Link to post
Share on other sites
On 9/12/2018 at 8:00 AM, Jos said:

Will have a look at what you have done in a week or 2. Remind me when you don't have a reply by the end of the month. ;)

@Jos

As requested, this is an end of the month reminder. 

Because I have posted a version that fixes the issues that I identified, this reminder is more for those who don't keep up with this topic.

Being a Chief Technology Officer, I can empathize with the demands on one's time.  Any attention towards making this UDF as robust and reliable as possible, I'm certain, will be greatly appreciated by the many users of this very useful UDF, including me.

Thanks

:ILA2:

Share this post


Link to post
Share on other sites

Thanks for the reminder! It was needed. Will have a look at it tomorrow to see what you have done and let you know.

Stay tuned,
Jos


SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By TheXman
      This UDF brings the power and flexibility of jq to AutoIt scripts.  jq is an open-source, powerful, and flexible command-line based JSON processor.  As it says on their website, jq is like 'sed' for JSON.  jq can be used for the simplest of tasks like retrieving JSON objects and values (parsing), to very advanced JSON processing using its numerous built-in functions and conditional processing.  Its built-in functions can handle math, selection, conditional processing, mapping, object and array manipulation, flattening, reduction, grouping, and much more.  You can even create your own jq functions.  You can learn more about jq and even play with it in real-time, using jq's online jq playground, all on their website.
      Here and some helpful links to get you more familiar with jq, what can be done with it, its built-in functions, and its syntax.
      jq Website: https://stedolan.github.io/jq/ jq Manual: https://stedolan.github.io/jq/manual/ jqWiki (FAQ, Cookbook, Advanced Topics) https://stedolan.github.io/jq/manual/ jq Online Testing Playground: https://jqplay.org/ jq is a single 32 or 64 bit executable that has no other dependencies.  Just like using the SQLite UDF, the only requirement to use this UDF is that the jq executable reside in a location in which the UDF can execute it.  The latest win32 & win64 versions have been included in the UDF download.  You can always get newer versions from the jq website.
      I don't consider this UDF as a replacement for some of the other JSON UDFs like the one based on JSMN.  If speed and simple JSON parsing are your primary goals, then other UDFs may be a better choice.  However, if you like having the power to do just about anything related to JSON processing/manipulation, using a single UDF, then jq may be worth checking out.  It can be used by novices and experts alike.  Below, is a brief explanation of how jq works, how to use the jq UDF, and a few examples of how to do some simple tasks.  If you want to learn more about jq and what it can do, I would highly suggest checking out the jq website and some of the other resources listed above.
      jq at a high level
      Like 'sed', jq reads JSON in, either through STDIN or one or more files, processes it thru one or more "filters", and outputs the results.  You can, optionally, supply "options" that affect how it reads the input, where it gets its "filters", and how it writes its output.  It looks a little like this:
      JSON ---> jq processor (using supplied filters and options) ---> Output
      So in jq lingo, you basically use "Filters" to tell jq what you want it to do.  So in the UDF file, that is why the main functions ( _jqExec() and _jqExecFile() ) refer to filters and options.  Please make note that jq works with relatively strict JSON.  This means that all JSON read must be conform to the standard.  Luckily, jq is pretty good at identifying where a format error exists in non standard JSON.
      The jq UDF
      There are 2 main funtions in the UDF file, _jqExec and jqExecFile.  With these 2 functions, you can pretty much do anything that jq can do.  The only difference between to two functions is whether the JSON is supplied by a string or a file.  The 2 primary functions simply call the jq executable with the supplied information, after properly formatting the parameters.  There are additional functions in the UDF to easily pretty-print your json, compact-print your json, dump the json data with its associated paths, and see if specific JSON keys exist, but they all just execute the _jqExec or _jqExecFile function with the proper filter.  There are also a couple of extra functions to display what version of the UDF and jq executable you are currently using.  There are also a couple of functions to enable and disable logging of jq information for debugging purposes.  Most of the jq UDF file functions return an @error if unsuccessful.  Some also include @extended info.  Please see the actual function headers for more information on their usage and return values.
      The 2 primary functions below just format your jq request and pass it on the jq executable.  The functions will also properly escape double quotes (") that are used in the filter.  For most simple tasks, you just need to supply the JSON source and a filter.
      _jqExec($sJson, $sFilter, $sOptions = Default, $sWorkingDir = Default) Or _jqExecFile($sJsonFile, $sFilter, $sOptions = Default, $sWorkingDir = Default) Using jq in your script
      As stated earlier, the jq executable must reside somewhere where the script can locate and execute it.  The _jqInit() function always has to be executed before any jq processing occurs.  _jqInit() merely locates the executable or uses the supplied path.  It also clears any previous debug log.  The jq UDF folder contains a jq example script that has several examples to how to do some of the most common JSON processing tasks.  Here are a few examples to get you started:

      How to pretty-print some JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"fruits":[{"Apple":{"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqPrettyPrintJson($sJson) ConsoleWrite(@CRLF & "Pretty-Print JSON" & @CRLF & $sCmdOutput & @CRLF) How to compact-print some JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{ "fruits" : [{"Apple" : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqCompactPrintJson($sJson) ConsoleWrite(@CRLF & "Compact-Print JSON" & @CRLF & $sCmdOutput & @CRLF) Dump JSON data (paths and values)
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{ "fruits" : [{"Apple" : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqDump($sJson) ConsoleWrite(@CRLF & "Dump JSON paths and values" & @CRLF & $sCmdOutput & @CRLF) How to GET JSON values
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple" : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana.color' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Get color of banana" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) or
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple" : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = 'getpath(["Banana", "color"])' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Get color of banana" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF)  
      Check for the existence of a key
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana | has("color")' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Check for existence of color key within Banana object" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) Count of how many Items in an object
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana | length' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("How many items in the Banana object" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) How to PUT/Create/Modify JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sInput = "" $sFilter = 'setpath(["Apple","color"];"Red") | setpath(["Banana","color"];"Yellow") | setpath(["Banana","season"];"Summer")' $sOptions = '-n' ;required if no input supplied $sCmdOutput = _jqExec($sInput, $sFilter, $sOptions) ConsoleWrite("Update/Create JSON" & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF) List all of the fruits (top-level keys)
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = 'keys | .[]' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("List all top-level keys (fruits)" & @CRLF) ConsoleWrite("Input : " & $sJson & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF) Calculate the sum of all of the objects' price * qty
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '[{"id":1,"price":20.00,"qty":10},{"id":2,"price":15.00,"qty":20.25},{"id":3,"price":10.50,"qty":30}]' $sFilter = 'map(.price * .qty) | add' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Calculate the sum of all of the objects' price * qty" & @CRLF) ConsoleWrite("Input : " & $sJson & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF)
      The examples above, and the ones in the example files, merely scratch the surface of what jq can do.  It may look intimidating at first but it really isn't that bad once you start playing with it.
      If you have any questions regarding the UDF, or how to perform a certain task using jq, I'll try my best to answer them.  Since jq has been around for a while now, there's also several jq-related questions and answers on StackOverflow.
      If you work with JSON, I hope you find this UDF useful as I do.
      >>> Download in the Files Section <<<
       
    • By TheXman
      This UDF brings the power and flexibility of jq to AutoIt scripts.  jq is an open-source, powerful, and flexible command-line based JSON processor.  As it says on their website, jq is like 'sed' for JSON.  jq can be used for the simplest of tasks like retrieving JSON objects and values (parsing), to very advanced JSON processing using its numerous built-in functions and conditional processing.  Its built-in functions can handle math, selection, conditional processing, mapping, object and array manipulation, flattening, reduction, grouping, and much more.  You can even create your own jq functions.  You can learn more about jq and even play with it in real-time, using jq's online jq playground, all on their website.
      Here and some helpful links to get you more familiar with jq, what can be done with it, its built-in functions, and its syntax.
      jq Website: https://stedolan.github.io/jq/ jq Manual: https://stedolan.github.io/jq/manual/ jqWiki (FAQ, Cookbook, Advanced Topics) https://stedolan.github.io/jq/manual/ jq Online Testing Playground: https://jqplay.org/ jq is a single 32 or 64 bit executable that has no other dependencies.  Just like using the SQLite UDF, the only requirement to use this UDF is that the jq executable reside in a location in which the UDF can execute it.  The latest win32 & win64 versions have been included in the UDF download.  You can always get newer versions from the jq website.
      I don't consider this UDF as a replacement for some of the other JSON UDFs like the one based on JSMN.  If speed and simple JSON parsing are your primary goals, then other UDFs may be a better choice.  However, if you like having the power to do just about anything related to JSON processing/manipulation, using a single UDF, then jq may be worth checking out.  It can be used by novices and experts alike.  Below, is a brief explanation of how jq works, how to use the jq UDF, and a few examples of how to do some simple tasks.  If you want to learn more about jq and what it can do, I would highly suggest checking out the jq website and some of the other resources listed above.
      jq at a high level
      Like 'sed', jq reads JSON in, either through STDIN or one or more files, processes it thru one or more "filters", and outputs the results.  You can, optionally, supply "options" that affect how it reads the input, where it gets its "filters", and how it writes its output.  It looks a little like this:
      JSON ---> jq processor (using supplied filters and options) ---> Output
      So in jq lingo, you basically use "Filters" to tell jq what you want it to do.  So in the UDF file, that is why the main functions ( _jqExec() and _jqExecFile() ) refer to filters and options.  Please make note that jq works with relatively strict JSON.  This means that all JSON read must be conform to the standard.  Luckily, jq is pretty good at identifying where a format error exists in non standard JSON.
      The jq UDF
      There are 2 main funtions in the UDF file, _jqExec and jqExecFile.  With these 2 functions, you can pretty much do anything that jq can do.  The only difference between to two functions is whether the JSON is supplied by a string or a file.  The 2 primary functions simply call the jq executable with the supplied information, after properly formatting the parameters.  There are additional functions in the UDF to easily pretty-print your json, compact-print your json, dump the json data with its associated paths, and see if specific JSON keys exist, but they all just execute the _jqExec or _jqExecFile function with the proper filter.  There are also a couple of extra functions to display what version of the UDF and jq executable you are currently using.  There are also a couple of functions to enable and disable logging of jq information for debugging purposes.  Most of the jq UDF file functions return an @error if unsuccessful.  Some also include @extended info.  Please see the actual function headers for more information on their usage and return values.
      The 2 primary functions below just format your jq request and pass it on the jq executable.  The functions will also properly escape double quotes (") that are used in the filter.  For most simple tasks, you just need to supply the JSON source and a filter.
      _jqExec($sJson, $sFilter, $sOptions = Default, $sWorkingDir = Default) Or _jqExecFile($sJsonFile, $sFilter, $sOptions = Default, $sWorkingDir = Default) Using jq in your script
      As stated earlier, the jq executable must reside somewhere where the script can locate and execute it.  The _jqInit() function always has to be executed before any jq processing occurs.  _jqInit() merely locates the executable or uses the supplied path.  It also clears any previous debug log.  The jq UDF folder contains a jq example script that has several examples to how to do some of the most common JSON processing tasks.  Here are a few examples to get you started:

      How to pretty-print some JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"fruits":[{"Apple":{"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqPrettyPrintJson($sJson) ConsoleWrite(@CRLF & "Pretty-Print JSON" & @CRLF & $sCmdOutput & @CRLF) How to compact-print some JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{ "fruits" : [{"Apple" : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqCompactPrintJson($sJson) ConsoleWrite(@CRLF & "Compact-Print JSON" & @CRLF & $sCmdOutput & @CRLF) Dump JSON data (paths and values)
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{ "fruits" : [{"Apple" : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}' $sCmdOutput = _jqDump($sJson) ConsoleWrite(@CRLF & "Dump JSON paths and values" & @CRLF & $sCmdOutput & @CRLF) How to GET JSON values
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple" : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana.color' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Get color of banana" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) or
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple" : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = 'getpath(["Banana", "color"])' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Get color of banana" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF)  
      Check for the existence of a key
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana | has("color")' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Check for existence of color key within Banana object" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) Count of how many Items in an object
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = '.Banana | length' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("How many items in the Banana object" & @CRLF) ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF) How to PUT/Create/Modify JSON
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sInput = "" $sFilter = 'setpath(["Apple","color"];"Red") | setpath(["Banana","color"];"Yellow") | setpath(["Banana","season"];"Summer")' $sOptions = '-n' ;required if no input supplied $sCmdOutput = _jqExec($sInput, $sFilter, $sOptions) ConsoleWrite("Update/Create JSON" & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF) List all of the fruits (top-level keys)
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}' $sFilter = 'keys | .[]' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("List all top-level keys (fruits)" & @CRLF) ConsoleWrite("Input : " & $sJson & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF) Calculate the sum of all of the objects' price * qty
      #include "jq.au3" _jqInit() If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF) $sJson = '[{"id":1,"price":20.00,"qty":10},{"id":2,"price":15.00,"qty":20.25},{"id":3,"price":10.50,"qty":30}]' $sFilter = 'map(.price * .qty) | add' $sCmdOutput = _jqExec($sJson, $sFilter) ConsoleWrite("Calculate the sum of all of the objects' price * qty" & @CRLF) ConsoleWrite("Input : " & $sJson & @CRLF) ConsoleWrite("Filter : " & $sFilter & @CRLF) ConsoleWrite("Output : " & $sCmdOutput & @CRLF)
      The examples above, and the ones in the example files, merely scratch the surface of what jq can do.  It may look intimidating at first but it really isn't that bad once you start playing with it.
      If you have any questions regarding the UDF, or how to perform a certain task using jq, I'll try my best to answer them.  Since jq has been around for a while now, there's also several jq-related questions and answers on StackOverflow.
      If you work with JSON, I hope you find this UDF useful as I do.
    • By argumentum
      #include <SQLite.au3> ;-- When SQLite is compiled with the JSON1 extensions it provides builtin tools ;-- for manipulating JSON data stored in the database. ;-- This is a gist showing SQLite return query as a JSON object. ;-- https://www.sqlite.org/json1.html Example() Func Example() _SQLite_Startup() ; "<your path>\sqlite3.dll", False, 1) ; https://www.autoitscript.com/autoit3/docs/libfunctions/_SQLite_Startup.htm _SQLite_Open() ; ...if you can not run this due to errors, get the latest DLL from https://www.sqlite.org/ If _SQLite_Exec(-1, "CREATE TABLE users (id INTEGER PRIMARY KEY NOT NULL, full_name TEXT NOT NULL, email TEXT NOT NULL, created DATE NOT NULL );") Then Return 4 If _SQLite_Exec(-1, 'INSERT INTO users VALUES ' & _ '(1, "Bob McFett", "bmcfett@hunters.com", "32-01-01"),' & _ '(2, "Angus O''Vader","angus.o@destroyers.com", "02-03-04"),' & _ '(3, "Imperator Colin", "c@c.c", "01-01-01");') Then Return 5 ; -- Get query data as a JSON object using the ; -- json_group_object() [1] and json_object() [2] functions. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_object(" & _ " email," & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS json_result" & _ " FROM (SELECT * FROM users WHERE created > ""02-01-01"");") ; {"bmcfett@hunters.com":{"full_name":"Bob McFett","created":"32-01-01"},"angus.o@destroyers.com":{"full_name":"Angus O'Vader","created":"02-03-04"}} ; -- Get query data as a JSON object using the ; -- json_group_array() function to maintain order. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_array(" & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS my_json_result_OrAnythingReally" & _ " FROM (SELECT * FROM users ORDER BY created);") ; [{"full_name":"Imperator Colin","created":"01-01-01"},{"full_name":"Angus O'Vader","created":"02-03-04"},{"full_name":"Bob McFett","created":"32-01-01"}] ;-- Links ;-- [1] https://www.sqlite.org/json1.html#jgroupobject ;-- [2] https://www.sqlite.org/json1.html#jobj ; example found at https://gist.github.com/akehrer/481a38477dd0518ec0086ac66e38e0e2 EndFunc ;==>Example Func _SQLite_GetTable2d_ArrayToConsole($sSQL, $hDB = -1) Local $aResult, $iRows, $iColumns If _SQLite_GetTable2d($hDB, $sSQL, $aResult, $iRows, $iColumns) Then ConsoleWrite("! SQLite Error: " & _SQLite_ErrCode($hDB) & @CRLF & "! " & _SQLite_ErrMsg($hDB) & @CRLF) Else _SQLite_Display2DResult($aResult) EndIf ConsoleWrite(@CRLF) EndFunc ;==>_SQLite_GetTable2d_ArrayToConsole Based on this example, you can build your own query. 
      The code has all the explanations.
      Enjoy  
    • By matwachich
      Hi AutoIters!
      Here is my new UDF about GUIs: it's an enhanced mixture of Advanced InputBox (deprecated) and KODA Parser (deprecated), with additional functions.
      What you can do with it:
      Parse KODA files and directly create GUIs (_GUIUtils_CreateFromKODA) Parse a simple JSON form definition to simply create advanced InputBoxes with any amount/type of input controls (_GUIUtils_CreateFromJSON) Created GUIs are returned as Scripting.Dictionary objects, and you have helper functions to access GUIs controls by their names (_GUIUtils_HWnd, _GUIUtils_CtrlID, _GUIUtils_HCtrl ...) A function that can make a GUI created from KODA/JSON (defined by it's $oForm object) and make it a modal InputDialogBox (just like InputBox, but returns all entered data as Scripting.Dictionary object) Functions are documented, and there are some examples.
      Consider this UDF as beta, but since I'm currently using it in a small commercial project, it should become production ready in near future.
      To always get the latest code version, get it from Github (more up to date than this topic).
      Update 04/02/2020:
      New simple application example Fixed focused control handling in _GUIUtils_InputDialog Reset input values before returning from _GUIUtils_InputDialog Fixed CloseOnEsc in _GUIUtils_InputDialog Fixed Input not reset Fixed all ListBox items are selected Project now on GitHub https://github.com/matwachich/autoit-guiutils/
      Update 05/02/2020:
      Support for nodate for Date and Time input controls (pass null value) New handling of focused control _GUIUtils_InputDialog: Now, you can specify the focused control in $oInitialData by setting "controlName:focus" = True Updated documentation of _GUIUtils_InputDialog Update 07/02/2020:
      Bug corrected when setting Data input control Added: abillity to read a single input control New functions (_GUIUtils_GetInputs, _GUIUtils_WriteInputs), documentation completion. Readme file and screenshots on Github Page  
      GUIUtils.zip
    • By Roy_
      Ciao,
      This example reproduces the problem I encounter when sending a post request to an echo server.
      Instead of being sent as text utf8, the data is transmitted base64 encoded.
      Example:
      - json String: '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'
      - server response: "data:application/octet-stream;base64,AAAAALgE6QNYAIAAAgAAANQCAAAAAAAAWF8AAAAAAAD0VQAAAAAAAD9APQAAAAAAAwAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAABBAAAAAAABRUAAAAmIZwcAbRt8/LMa9UAAAAAAAAAAGFzIGlkIHRlbXB1cyBtaS4gTW9yYmkgaXBzdW0gZW5pbSwgdWx0cmljaWVzIGFjIGF1Z3VlIHNpdCBhbWV0LCB1bGxhbWNvcnBlciBmaW5pYnVzIGV4LiBWZXN0aWJ1bHVtIHZlbCBwb3N1ZXJlIG5pYmgsIG5lYyBmYXVjaWJ1cyBlcm9zLiBOYW0gbWFsZXN1YWRhIG5vbiBsYWN1cyBhIHN1c2NpcGl0LiBOdWxsYSByaG9uY3VzIHRlbXB1cyBtaSBxdWlzIHBsYWNlcmF0LiBDdXJhYml0dXIgY29tbW9kbyB0aW5jaWR1bnQganVzdG8gcXVpcyBzb2xsaWNpdHVkaW4uIn0="
       
      How can I go about transmitting data in text format utf8?
      Thanks in advance for the help.
      In the zip file:
      - CurlJsonPost.au3: this script
      - Curl.au3: UDF by Ward (thank you!)
      - data.json: json srting for command line test
       
      Note: using Curl.exe with json string saved in a file (utf8 encoded) named data.json, works perfectly
      To try with the command line tool:
      - save data.json in curl\bin directory
      - open cmd.exe and cd to curl\bin directory
      - Enter the following command:
      curl -H "Content-Type: application/json" --data @data.json https://httpbin.org/post
       
      #Include "Curl.au3" Global $_cURL_OutputBuffer Local $sJson = '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}' Local $iRetCode, $sServerResponse ConsoleWrite("=== Json post test ===" & @LF) ConsoleWrite(StringFormat("Retcode: %s - %s", $iRetCode, Curl_Easy_strerror($iRetCode)) & @LF) ConsoleWrite("Data returned from server" & @LF & @LF) $sServerResponse = JsonPost_Test($sJson, $iRetCode) ConsoleWrite($sServerResponse & @LF) Func JsonPost_Test($sJson, ByRef $iRetCode) ; Init Easy Curl Interface e set url (echo service) Local $oCurl = Curl_Easy_Init() curl_easy_setopt($oCurl, $CURLOPT_URL, "https://httpbin.org/post") ; Set content type header Local $headers = curl_slist_append(0, "Content-Type: application/json") curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers) ; Post fields & size curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson) curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDSIZE, StringLen($sJson)) ; Set callbac function to get server response back (see global var $_cURL_OutputBuffer) $hWriteFunc = DllCallbackRegister("WriteFunc_CallBack", "uint:cdecl", "ptr;uint;uint;ptr") curl_easy_setopt($oCurl, $CURLOPT_WRITEFUNCTION, DllCallbackGetPtr($hWriteFunc)) ; Ignore ssl certificates check curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYPEER, 0) curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYHOST, 0) ; Execute the post request $iRetCode = curl_easy_perform($oCurl) ; Set return trasfer & clear output buffer global var Local $sReturnTransfer = $_cURL_OutputBuffer $_cURL_OutputBuffer = "" Return SetError(0, 0, $sReturnTransfer) EndFunc Func WriteFunc_CallBack($ptr,$nSize,$nMemb,$pStream) Local $vData = DllStructCreate ("byte[" & $nSize*$nMemb & "]",$ptr) $_cURL_OutputBuffer &= BinaryToString(DllStructGetData($vData,1)) Return $nSize*$nMemb EndFunc  
      curlJsonPost.zip
×
×
  • Create New...