Jump to content

A Non-Strict JSON UDF (JSMN)


Ward
 Share

Recommended Posts

  • Developers

I guess that should be:

ConsoleWrite('output1: ' & Json_Get( $json2, '.mother.sosa') &  @LF )
ConsoleWrite('output: ' & Json_Get( $json2, ".lastname") &  @LF )

Jos

JSON blijft lastig :)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • 4 weeks later...

@Jos

Now i'm using the Packege UDF to store some txt files and read it, it's around 2 thousand files... to read one of the data it takes around 2secs. If i change it to json, it will be faster? I'm trying to create a database and i don't know what is the best option.

Link to comment
Share on other sites

  • Developers

I have no idea as your provided scenario is very explicit. :)
It probably all depends what kind of lexing you need to do with those text files when reading processing them.

Just give it a try and simply test it would be the best advice as that will give you a conclusive answer.

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • 4 weeks later...
On 1/13/2018 at 10:06 AM, Jos said:

Added JsonDump() and updated/attached a new Zipfile in first post

 

Jos

In _Json_TokenDump, could you put a Local in front of $cObjPath so it doesn't cause a warning when we require that variables be declared?

Func _Json_TokenDump(ByRef $Json, $Ptr, ByRef $Next, $ObjPath = "")
    If $Next = -1 Then Return Null

    Local $Token = DllStructCreate("int;int;int;int", $Ptr + ($Next * 20))
    Local $Type = DllStructGetData($Token, 1)
    Local $Start = DllStructGetData($Token, 2)
    Local $End = DllStructGetData($Token, 3)
    Local $Size = DllStructGetData($Token, 4)
    $Next += 1

    If $Type = 0 And $Start = 0 And $End = 0 And $Size = 0 Then ; Null Item
        $Next = -1
        Return Null
    EndIf

    Switch $Type
        Case 0 ; Json_PRIMITIVE
            Local $Primitive = StringMid($Json, $Start + 1, $End - $Start)
            Switch $Primitive
                Case "true"
                    Return True
                Case "false"
                    Return False
                Case "null"
                    Return Null
                Case Else
                    If StringRegExp($Primitive, "^[+\-0-9]") Then
                        Return Number($Primitive)
                    Else
                        Return Json_StringDecode($Primitive)
                    EndIf
            EndSwitch

        Case 1 ; Json_OBJECT
            For $i = 0 To $Size - 1 Step 2
                Local $Key = _Json_TokenDump($Json, $Ptr, $Next)
                Local $cObjPath = $ObjPath & "." & $Key                                     ;<--add Local here
                Local $Value = _Json_TokenDump($Json, $Ptr, $Next, $ObjPath & "." & $Key)
                If Not ($Value = False) Then
                    If Not IsString($Key) Then
                        $Key = Json_Encode($Key)
                    EndIf
                    ; show the key and its value
                    ConsoleWrite("+-> " & $cObjPath & '  =' & $Value & @CRLF)
                EndIf
            Next
            Return False
        Case 2 ; Json_ARRAY
            Local $sObjPath = $ObjPath
            Local $Array[$Size]
            For $i = 0 To $Size - 1
                $sObjPath = $ObjPath & "[" & $i & "]"
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $sObjPath)
                If $Value <> False Then
                    ; show the key and its value
                    ConsoleWrite("+=> " & $sObjPath & "=>" & $Value & @CRLF)
                EndIf
            Next
            $ObjPath = $sObjPath
            Return False

        Case 3 ; Json_STRING
            Local $LastKey = Json_StringDecode(StringMid($Json, $Start + 1, $End - $Start))
            Return $LastKey
    EndSwitch
EndFunc   ;==>_Json_TokenDump

 

Link to comment
Share on other sites

  • 3 weeks later...

First, thank you for creating this UDF.. Hopefully, it will work for what I need it for. Second.. I know nothing about json files.. Having to learn for a project. I have a json with data similar to the following.

{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}

What I need to do is read the ID# and then take the largest # and add 1, then add a new section, with in this case id: 104.. name: Example 4 etc. Lastly write the json back to disk. Could I do this with this UDF?

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Link to comment
Share on other sites

  • Developers

That is why I made the json_dump() udf to help you with the logic of how the retrieve the appropriate field value:

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
ConsoleWrite("!== Json_TokenDump ============================================================================================================" & @CRLF)
Json_Dump($data)

Start there and post in case you still have questions. :)

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Yes, I played with the

Json_Dump($data)

and it produces and an output, but cannot figure out how to correlate that into variables I can manipulate.  Maybe I'm just blind? I mean I can single out items with this.

ConsoleWrite(Json_Get($Data1, '["items"][0]["id"]') & @LF)

but i'm looking for more of maybe $target = $Data1[items][$x][id] kinda thing so I can loop $x from 0 to 10 and get each id#? if that makes any sense? or is all the data from the decoded json stored in array? that then I can just push data into and convert the array back the json data.

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Link to comment
Share on other sites

  • Developers

Shouldn't be too hard, just look closely at the produced output in the dump! 
something like this you mean?  :) 

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
Local $i = 0
While 1
    $id = json_get($object, '.items[' & $i & '].id')
    If @error Then ExitLoop
    ConsoleWrite('id:' & $id & @CRLF)
    $i += 1
WEnd

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

That's it.. Thank you.. Helps make that data clearer.. I was in the process of attempting something similar with do/until but this works...

Side request. is the Json_put only designed to do one element at a time, or is there a preferred method to add a section?  Such as needing to add this..

{
      "id",104
      "name","Example 4"
      "hash",null
      "folder",null
      "download",null
      "version",null
      "website",null
      "files",[]
    }

 

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Link to comment
Share on other sites

Is parsing json data normally very slow? loading the file in...

Local $file_json_raw = FileRead($ini_file_path & "\data.json")
Local $file_json_data = Json_Decode($file_json_raw)

takes about 6 seconds.

Parsing the data into an array,

While 1 ; list of all servers
        $server_id = json_get($file_json_data, '.servers[' & $a & '].id')
        If ($server_id == "") Then ExitLoop
        $server_name = json_get($file_json_data, '.servers[' & $a & '].name')
        $server_name_override = json_get($file_json_data, '.servers[' & $a & '].nameoverride')
        If ($server_name_override <> "") Then $server_name = $server_name_override
    WEnd

takes about 30 seconds.

The json file is about 900mb, the server section consist of about 2100 entries.  Is there a quicker way to be handing the data?

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Link to comment
Share on other sites

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

  • Developers
6 minutes ago, zone97 said:

Is parsing json data normally very slow?

Fact or opinion?  ;)

Parsing a 900Mb JSON encoded file doesn't sound as a common thing to me as I am used to using JSON in REST calls used for API's, which in general aren't that big.
Maybe you need you see whether you need to change your approach and use RegEx?

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

7 minutes ago, TheDcoder said:

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

I'm very new to using this UDF.. how do they differ? does json_objGet, create a 2d array of the values of a section? (because that is ultimately what the goal is for me.)

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Link to comment
Share on other sites

@zone97 No, it is a wrapper for the Scripting.Dictionary object or simply a dictionary variable. You can access the values inside a dictionary by using the Json_ObjGet and Json_ObjPut functions, you should see the examples and read the code for those functions so you could understand better :)

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

  • 4 months later...

I have found a couple of small issues with the Json_Dump() logic.  One issue is related to comparing different data types.  The other issue is probably just an oversight.

If you run the following example with the latest version of the json UDF, you will get the following console output:

#include "json.original.au3" ; <== Modify this line to point to your copy of json.udf
;~ #include "json.modified.au3" ; <== Modify this line to point to your copy of json.udf


example()

Func example()

    #cs - PrettyPrint of JSON const below
    {
        "null_item": null,
        "zero_primitive_item": 0,
        "false_item": false,
        "true_item": true,
        "string_item": "This is a string",
        "array_item": [0,1,2]
    }
    #ce
    
    Const $JSON_STRING = '{"null_item": null, "zero_primitive_item": 0,"false_item": false,"true_item": true,"string_item": "This is a string","array_item": [0,1,2]}'
    
    Json_Dump($JSON_STRING)

EndFunc
+-> .null_item  =
+-> .true_item  =True
+-> .string_item  =This is a string
+=> .array_item[1]=>1
+=> .array_item[2]=>2

As you can see, items that have a primitive value of 0 or false, do not get written out at all.  This was because of comparing different data types.  Also notice that values that are null do not display "null", they simply display no value.  Since it is a dump, I would assume that you would want to see all values. 

I have made changes to 5 lines of code in the UDF.  When run with my modifications, you get the following output.  All lines get displayed with their correct literal values.

+-> .null_item  =null
+-> .zero_primative_item  =0
+-> .false_item  =false
+-> .true_item  =true
+-> .string_item  =This is a string
+=> .array_item[0]=>0
+=> .array_item[1]=>1
+=> .array_item[2]=>2

I tried to leave the existing code as close to original as possible.  In other words, instead of rewriting the functions, I made the smallest modifications possible in order to get the correct result. 

I have attached the original and the modified versions of the UDF so that you can easily do a DIFF to see the changes. Each of the modified lines is also denoted by a comment that says ";XC - Changed line" to make them easy to FIND.

 

I just notice that I did make a few more changes to the UDF.  I added an extra function, "Json_ObjGetItems(ByRef $Object)".  Because I use very strict directives, I made sure that all variables were declared before being used so I would stop getting warnings during compilation.

 

Hopefully my modifications pass muster and can be added to the UDF.

 

Json.modified.au3

Json.original.au3

Edited by TheXman
Added that I made a few additional changes to the UDF and why.
Link to comment
Share on other sites

  • Developers

Will have a look at what you have done in a week or 2. Remind me when you don't have a reply by the end of the month. ;)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • 3 weeks later...
On 9/12/2018 at 8:00 AM, Jos said:

Will have a look at what you have done in a week or 2. Remind me when you don't have a reply by the end of the month. ;)

@Jos

As requested, this is an end of the month reminder. 

Because I have posted a version that fixes the issues that I identified, this reminder is more for those who don't keep up with this topic.

Being a Chief Technology Officer, I can empathize with the demands on one's time.  Any attention towards making this UDF as robust and reliable as possible, I'm certain, will be greatly appreciated by the many users of this very useful UDF, including me.

Thanks

:ILA2:

Link to comment
Share on other sites

  • Developers

Thanks for the reminder! It was needed. Will have a look at it tomorrow to see what you have done and let you know.

Stay tuned,
Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...