Jump to content
Ward

A Non-Strict JSON UDF (JSMN)

Recommended Posts

Jos

I guess that should be:

ConsoleWrite('output1: ' & Json_Get( $json2, '.mother.sosa') &  @LF )
ConsoleWrite('output: ' & Json_Get( $json2, ".lastname") &  @LF )

Jos

JSON blijft lastig :)


Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
hmobron

 

19 hours ago, Jos said:

I guess that should be:

ConsoleWrite('output1: ' & Json_Get( $json2, '.mother.sosa') &  @LF )
ConsoleWrite('output: ' & Json_Get( $json2, ".lastname") &  @LF )

Jos

JSON blijft lastig :)

That worked, sorry that i missed that. But what is the dividends between those functions Json_Get and Json_ObjGet?  

Share this post


Link to post
Share on other sites
x_bennY

@Jos

Now i'm using the Packege UDF to store some txt files and read it, it's around 2 thousand files... to read one of the data it takes around 2secs. If i change it to json, it will be faster? I'm trying to create a database and i don't know what is the best option.

Share this post


Link to post
Share on other sites
Jos

I have no idea as your provided scenario is very explicit. :)
It probably all depends what kind of lexing you need to do with those text files when reading processing them.

Just give it a try and simply test it would be the best advice as that will give you a conclusive answer.

Jos

  • Like 1

Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
Decibel
On 1/13/2018 at 10:06 AM, Jos said:

Added JsonDump() and updated/attached a new Zipfile in first post

 

Jos

In _Json_TokenDump, could you put a Local in front of $cObjPath so it doesn't cause a warning when we require that variables be declared?

Func _Json_TokenDump(ByRef $Json, $Ptr, ByRef $Next, $ObjPath = "")
    If $Next = -1 Then Return Null

    Local $Token = DllStructCreate("int;int;int;int", $Ptr + ($Next * 20))
    Local $Type = DllStructGetData($Token, 1)
    Local $Start = DllStructGetData($Token, 2)
    Local $End = DllStructGetData($Token, 3)
    Local $Size = DllStructGetData($Token, 4)
    $Next += 1

    If $Type = 0 And $Start = 0 And $End = 0 And $Size = 0 Then ; Null Item
        $Next = -1
        Return Null
    EndIf

    Switch $Type
        Case 0 ; Json_PRIMITIVE
            Local $Primitive = StringMid($Json, $Start + 1, $End - $Start)
            Switch $Primitive
                Case "true"
                    Return True
                Case "false"
                    Return False
                Case "null"
                    Return Null
                Case Else
                    If StringRegExp($Primitive, "^[+\-0-9]") Then
                        Return Number($Primitive)
                    Else
                        Return Json_StringDecode($Primitive)
                    EndIf
            EndSwitch

        Case 1 ; Json_OBJECT
            For $i = 0 To $Size - 1 Step 2
                Local $Key = _Json_TokenDump($Json, $Ptr, $Next)
                Local $cObjPath = $ObjPath & "." & $Key                                     ;<--add Local here
                Local $Value = _Json_TokenDump($Json, $Ptr, $Next, $ObjPath & "." & $Key)
                If Not ($Value = False) Then
                    If Not IsString($Key) Then
                        $Key = Json_Encode($Key)
                    EndIf
                    ; show the key and its value
                    ConsoleWrite("+-> " & $cObjPath & '  =' & $Value & @CRLF)
                EndIf
            Next
            Return False
        Case 2 ; Json_ARRAY
            Local $sObjPath = $ObjPath
            Local $Array[$Size]
            For $i = 0 To $Size - 1
                $sObjPath = $ObjPath & "[" & $i & "]"
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $sObjPath)
                If $Value <> False Then
                    ; show the key and its value
                    ConsoleWrite("+=> " & $sObjPath & "=>" & $Value & @CRLF)
                EndIf
            Next
            $ObjPath = $sObjPath
            Return False

        Case 3 ; Json_STRING
            Local $LastKey = Json_StringDecode(StringMid($Json, $Start + 1, $End - $Start))
            Return $LastKey
    EndSwitch
EndFunc   ;==>_Json_TokenDump

 

Share this post


Link to post
Share on other sites
zone97

First, thank you for creating this UDF.. Hopefully, it will work for what I need it for. Second.. I know nothing about json files.. Having to learn for a project. I have a json with data similar to the following.

{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}

What I need to do is read the ID# and then take the largest # and add 1, then add a new section, with in this case id: 104.. name: Example 4 etc. Lastly write the json back to disk. Could I do this with this UDF?


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
Jos

That is why I made the json_dump() udf to help you with the logic of how the retrieve the appropriate field value:

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
ConsoleWrite("!== Json_TokenDump ============================================================================================================" & @CRLF)
Json_Dump($data)

Start there and post in case you still have questions. :)

Jos


Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
zone97
Posted (edited)

Yes, I played with the

Json_Dump($data)

and it produces and an output, but cannot figure out how to correlate that into variables I can manipulate.  Maybe I'm just blind? I mean I can single out items with this.

ConsoleWrite(Json_Get($Data1, '["items"][0]["id"]') & @LF)

but i'm looking for more of maybe $target = $Data1[items][$x][id] kinda thing so I can loop $x from 0 to 10 and get each id#? if that makes any sense? or is all the data from the decoded json stored in array? that then I can just push data into and convert the array back the json data.

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
Jos
Posted (edited)

Shouldn't be too hard, just look closely at the produced output in the dump! 
something like this you mean?  :) 

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
Local $i = 0
While 1
    $id = json_get($object, '.items[' & $i & '].id')
    If @error Then ExitLoop
    ConsoleWrite('id:' & $id & @CRLF)
    $i += 1
WEnd

Jos

Edited by Jos

Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
zone97
Posted (edited)

That's it.. Thank you.. Helps make that data clearer.. I was in the process of attempting something similar with do/until but this works...

Side request. is the Json_put only designed to do one element at a time, or is there a preferred method to add a section?  Such as needing to add this..

{
      "id",104
      "name","Example 4"
      "hash",null
      "folder",null
      "download",null
      "version",null
      "website",null
      "files",[]
    }

 

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
zone97
Posted (edited)

Answered my own question from above..

 

Edited by zone97

 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
zone97

Is parsing json data normally very slow? loading the file in...

Local $file_json_raw = FileRead($ini_file_path & "\data.json")
Local $file_json_data = Json_Decode($file_json_raw)

takes about 6 seconds.

Parsing the data into an array,

While 1 ; list of all servers
        $server_id = json_get($file_json_data, '.servers[' & $a & '].id')
        If ($server_id == "") Then ExitLoop
        $server_name = json_get($file_json_data, '.servers[' & $a & '].name')
        $server_name_override = json_get($file_json_data, '.servers[' & $a & '].nameoverride')
        If ($server_name_override <> "") Then $server_name = $server_name_override
    WEnd

takes about 30 seconds.

The json file is about 900mb, the server section consist of about 2100 entries.  Is there a quicker way to be handing the data?


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
TheDcoder

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) mean to do that to anybody!!!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites
Jos
Posted (edited)
6 minutes ago, zone97 said:

Is parsing json data normally very slow?

Fact or opinion?  ;)

Parsing a 900Mb JSON encoded file doesn't sound as a common thing to me as I am used to using JSON in REST calls used for API's, which in general aren't that big.
Maybe you need you see whether you need to change your approach and use RegEx?

Jos

Edited by Jos

Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
zone97
7 minutes ago, TheDcoder said:

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

I'm very new to using this UDF.. how do they differ? does json_objGet, create a 2d array of the values of a section? (because that is ultimately what the goal is for me.)


 

Spoiler

WinSizer 2.1 (01/04/2017) - Download - [ Windows Layout Manager ]
Folder+Program (12/23/2016) - Download - [ USB Shortcut Creator ]

 

Share this post


Link to post
Share on other sites
TheDcoder

@zone97 No, it is a wrapper for the Scripting.Dictionary object or simply a dictionary variable. You can access the values inside a dictionary by using the Json_ObjGet and Json_ObjPut functions, you should see the examples and read the code for those functions so you could understand better :)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) mean to do that to anybody!!!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • jesus40
      By jesus40
      Hello friends, I need help getting data from json. I read json from url into a variable.
      Is possible to get volume_USD value etc.  where product_id= "ETH-BTC", ? So I want get "37421.69109185" as variable.
      I think is better keeping it in the memory as variable. Or easier to convert it to db or csv?
       
       
      #include <Inet.au3> #include <json.au3> $URL="https://api.abucoins.com/products/stats" $data = _INetGetSource($URL) MsgBox("","",$data) $object=json_decode($data) local $i=0 while 1 $message=json_get($object,'[product_id]');???????????????????????????????? MsgBox("","volume_USD",$message) $i=$i+1 WEnd  
      the json structure is here:
      https://api.abucoins.com/products/stats
      [ { "product_id": "ETH-BTC", "last": "0.05202015", "open": "0.05009375", "high": "0.05250000", "low": "0.04872420", "volume": "47.77440456", "volume_BTC": 2.48523169, "volume_USD": 37421.69109185, "volume_7d": "364.40240341", "volume_30d": "3555.83377072", "change": "3.85" }, { "product_id": "LTC-BTC", "last": "0.01737490", "open": "0.01744636", "high": "0.01779000", "low": "0.01658091", "volume": "67.62762830", "volume_BTC": 1.17502328, "volume_USD": 17941.95920676, "volume_7d": "611.57893068", "volume_30d": "3881.57766044", "change": "-0.41" }, { "product_id": "ETC-BTC", "last": "0.00191846", "open": "0.00189001", "high": "0.00198000", "low": "0.00162741", "volume": "66.48327742", "volume_BTC": 0.12754551, "volume_USD": 1947.54976323, "volume_7d": "738.94495089", "volume_30d": "4090.74712156", "change": "1.51" }, { "product_id": "ZEC-BTC", "last": "0.03466163", "open": "0.03365452", "high": "0.03466163", "low": "0.03295811", "volume": "33.40101560", "volume_BTC": 1.15773364, "volume_USD": 17677.95599574, "volume_7d": "191.15953471", "volume_30d": "887.70158297", "change": "2.99" } ]  
       
    • satanico64
      By satanico64
      hi guys !
      how are you ? fine ? yeahhh
      Here is my problem:
      I have 2 autoit applications, a.exe and b.exe
      In a.exe, i call b.exe with a json as parameter:
      FileWrite("N:\+++ Dev\ClientServeur\FolderMonitor.nca", "lejson in A:" & Json_Encode($json_tmp)& @CRLF) Run(@ScriptDir & "\FolderMonitor.exe " & Json_Encode($json_tmp) )                        
              In program B:
      ;Firstline of program: FileWrite("N:\+++ Dev\ClientServeur\FolderMonitor.nca", @CRLF & "json in B:" & $CmdLine[1] & @CRLF)        
              Output:
                     
              lejson in A:{"DOSSIERS":{"Dossier_Hidden":"N:\\+++ Dev\\ClientServeur\\","Dossier_du_script":"N:\\+++ Dev\\ClientServeur\\","Fichier_Historique":"N:\\+++ Dev\\ClientServeur\\Historique.log","Fichier_DEBUG":"N:\\+++ Dev\\ClientServeur\\FolderMonitor.nca"}}
              json in B  :{DOSSIERS:{Dossier_Hidden:N:\\+++ Dev\\ClientServeur\\,Dossier_du_script:N:\\+++ Dev\\ClientServeur\\,Fichier_Historique:N:\\+++ Dev\\ClientServeur\\Historique.log,Fichier_DEBUG:N:\\+++ Dev\\ClientServeur\\FolderMonitor.nca}}
      As you see, i lost my quote in my json so it's au problem to parse my json.
      I use THIS library
      I tried different type of encoding (ex JSON_STRICT_PRINT), but i don't think that it's the problem. Maybe i'm wrong
       
      thanks
      Nicolas.
    • jandings
      By jandings
      Hello there,
      since I spent some time to access the REST-API of the web shop system Shopware, I'd like to share a few lines to make life easier for others.
      With this you can access your Shopware database, reading and writing all kind of data. 
      Replies are JSON style.
      To work with JSON I usually depend on either just own AutoIt string operations or this AutoIt library:
      ; File        : Json.au3 (2015.01.08)
      ; Purpose    : A Non-Strict JavaScript Object Notation (JSON) Parser UDF
      ; Author    : Ward
      $UserName="xxxx" ;Shopware credentials of Shopware user who has the API checkbox ticked $PassWord="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" ;API-Key of this user, NOT the Password $URL="www.XXX.com/api/" $Command="orders/64682?useNumberAsId=true" ;A list of commands and options is available through shopware REST-API help; Here we read order number 64682 $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", "https://" & $URL & $Command , False) $oHTTP.SetCredentials($UserName, $PassWord, 0) $oHTTP.Send() $response = $oHTTP.ResponseText ConsoleWrite ($response & @CRLF & @CRLF)  
    • BJP
      By BJP
      Hello.  I have a program that has used ADO database connection to return a query and then subsequently put the query results into an array using getrows.  See snippet below:
      $constrim="DRIVER={SQL Server};SERVER=xxx-xxxxx\CSC;DATABASE=xxxxxxxxx;uid=xxxxxxxxxx;pwd=xxxxxxxxxxxx;" $adCN = ObjCreate ("ADODB.Connection") ; <== Create SQL connection $adCN.Open ($constrim) ; local $sQuery = "select * from tbl_Apps"                     ; get all applications in the database local $oAppRecordSet = $adCN.Execute($sQuery) local $aAppsInDB = $oAppRecordSet.Getrows(5000) With the code above I can perform array operations very efficiently. 
      Now I need to get this data via JSON which is working (using Ward's JSON UDF)  but the data set returned is large (approx 4MB) and I'm wondering what is the most efficient way to get this data into an array.  
      Dim $obj = ObjCreate ("WinHttp.WinHttpRequest.5.1") $obj.Open("GET", $URL, false) $obj.SetRequestHeader("Content-Type", "application/json") $obj.Send() $json = JSON_decode( $obj.ResponseText ) Any help would be appreciated!!
    • baolo073
      By baolo073
      [ [ [ "Kinh Oanh,\r\n", "Dear Oanh,\r\n", null, null, 3 ], [ "C\u1ea3m \u01a1n b\u1ea1n r\u1ea5t nhi\u1ec1u v졢\u1ee9c th\u01b0 c\u1ee7a b\u1ea1n \u0111\u1ebfn m\u1ed9t v᩠ngṠtr\u01b0\u1edbc. ", "Thank you very much for your letter which arrived a few days ago.", null, null, 3 ], [ "Th\u1eadt \u0111⯧ y뵠khi nghe t\u1eeb b\u1ea1n.\r\n", "It was lovely to hear from you.\r\n", null, null, 3 ], [ "b\u1ea1n \u0111i \u0111㵠v\u1eady.\r\n", "where do you go.\r\n", null, null, 1 ], [ "T\u1ea1m bi\u1ec7t!", "Goodbye!", null, null, 1 ] ], null, "en", null, null, null, 0.91366601, null, [ [ "en" ], null, [ 0.91366601 ], [ "en" ] ] ] How to parse array to json?
×