Jump to content

A Non-Strict JSON UDF (JSMN)


Ward
 Share

Recommended Posts

@jugador You seem offended, sorry. But I wasn't suggesting that your UDF is malicious, I was simply pointing out the fact that it is not secure.

As per my understanding, you use Microsoft's JScript engine to parse JSON by passing it as input to your own function, but the input is not sanitized to protect against arbitrary code, am I right?

A malicious JSON payload might contain code within the JSON or even outside it which is executed without the user's or script author's knowledge. It is important that the JSON input is sanitized because it is directly being injected as code which is run.

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

  • Developers

@TheDcoder,

Isn't the same true for Compiled script by definition with the encoded script being in the PE header

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

@Jos Are we talking about compiled AutoIt scripts? If so I don't think I understand, please explain :sweating:

Edited by TheDcoder

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

  • Developers
4 hours ago, TheDcoder said:

Are we talking about compiled AutoIt scripts? If so I don't think I understand, please explain

Yes ....  and no, not up for further discussion in the open forum. My point is that this talk about code injection is somewhat far-fetched. ;) 

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

5 minutes ago, Jos said:

My point is that this talk about code injection is somewhat far-fetched. ;) 

How so? It only takes one malicious JSON payload to exploit. And security by obscurity is never a good idea :)

7 minutes ago, Jos said:

Yes ....  and no, not up for further discussion in the open forum.

I don't see the attack vector in the context of an compiled AutoIt script. Unless you mean the script itself can be malicious? That's not what I was focusing on. Even a genuine script can be used by a malicious 3rd-party who just has the ability to pass on JSON to the script, such as a remote web server.

I took the liberty to clarify my points a bit in this post as I was going to reply anyway, hope you don't mind :)

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

On 7/1/2022 at 5:39 PM, Mateocedillo said:

Can you speed up the processing speed on long JSONs that contains much items?

The short answer is "most likely yes".  Without more information, there's really no better answer that doesn't include a bunch of speculation and assumptions.  If you want better answers, then learn how to ask better questions.  The more detailed and specific your questions and/or requests are, the more accurate the responses will most likely be.

You appear to be asking whether there is a way to speed up the code that you have written.  Seriously, how do you expect anyone to be able to answer that question without seeing your script or a script that accurately shows the logic that you are using, some sample JSON data, and a detailed description of the expected results you are trying to gather from that sample data?

JSMN, which is the JSON parsing engine that the json.au3 UDF lib is built around, is one of the fastest parsing engines around.  So I seriously doubt that any performance issues with your script are related to the functions in the UDF lib.  The most likely cause of your performance issues is the logic you are using to get the data or information that you want from the JSON.  As you've read, there are multiple tools & UDF's available for parsing (and processing) JSON.  If you are primarily wanting to pluck (parse) out data values, then I'm sure that the json.au3 UDF can do it as fast and efficiently as most any other JSON parsing engine.  If you need to gather more than just data, meaning you need to gather information (like sums, averages, counts/stats, etc.), add/delete/modify JSON values or objects, or reformat/splice/extract JSON structures, then I would suggest a tool like jq.  Some might even suggest parsing JSON using regular expressions, which may be very fast but (in my opinion) not as reliable, unless you have full control over the JSON being used as input (the order of the JSON data AND and its formatting).

 

On 7/1/2022 at 5:39 PM, Mateocedillo said:

I have the latest dependency (2021.11.06)

I'm not sure what you meant by this.  If you are referring to the latest version of the json.au3 UDF library, then it is 2021.11.20 not 2021.11.06.  You can find it near the bottom of the 1st post in this topic.

Edited by TheXman
Link to comment
Share on other sites

1 hour ago, qinylj said:

There is an exception in the _Json_token function that is not judged, if the key value is an empty array, there will be a problem。Corrected as follows:

case 2;Json_ARRAY
    If $Size = 0 Then
      Local $Array = null
    Else
        Origin Code Here
   Endif

 

Please show complete reproducer.

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Codefor other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 *

 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2023-04-24

Link to comment
Share on other sites

Hello, thanks for your answers.
I'll try the UDFs you provided and let me know how it goes, though we'll most likely have to rewrite parts of a JSON-based UDF.
About JSMN, the danger and the code injection, the truth would be needing a little more context...

Quote

JSMN, which is the JSON parsing engine that the json.au3 UDF lib is built around, is one of the fastest parsing engines around.  So I seriously doubt that any performance issues with your script are related to the functions in the UDF lib.  The most likely cause of your performance issues is the logic you are using to get the data or information that you want from the JSON.  As you've read, there are multiple tools & UDF's available for parsing (and processing) JSON.  If you are primarily wanting to pluck (parse) out data values, then I'm sure that the json.au3 UDF can do it as fast and efficiently as most any other JSON parsing engine.  If you need to gather more than just data, meaning you need to gather information (like sums, averages, counts/stats, etc.), add/delete/modify JSON values or objects, or reformat/splice/extract JSON structures, then I would suggest a tool like jq.  Some might even suggest parsing JSON using regular expressions, which may be very fast but (in my opinion) not as reliable, unless you have full control over the JSON being used as input (the order of the JSON data AND and its formatting).

Well, I'll explain and I'm sorry in advance if I didn't make myself understood. Yes, that is what I meant. This UDF uses JSON.au3, and the contents of the JSON that take time to process are arrays that contain information, and what we do is extract that data from the JSON array to convert it into an array for the UDF to manipulate and examine it.

 

Link to comment
Share on other sites

Well, in case more is needed, a sample of this is a UDF that I have been making in the company of @Danyfirex. Especially, in the FakeYou.au3 UDF, functions _FakeYouGetVoicesList and _FakeYouGetCategoriesList (in the latter it is acceptable but in the previous function it takes time to process the data) This is the GitHub: UDF repository

BTW, I'm sorry for my english... :(

Link to comment
Share on other sites

@Mateocedillo

I took a quick look at your _FakeYouGetVoicesList() function.  I'm not sure how long your functions takes on your test PC's but, I have written a function that creates the voices array in a little over half a second on my PC. My array includes all of the values in each model, just like your function.

Output:

Time to create array: 0.518 seconds
Number of rows:       1941
Number of cols:       15

My script uses jq to create the array because I think it is better suited for this particular task.  Since I think discussing alternative solutions in detail, in someone else's topic is rude, I will not go into detail on how I did it here.  If you want to see suggestions on how it could be done using jq, then I would suggest that you create a new topic with any questions that you may have related to using jq.

If you want to stick with the json.au3 UDF, then there are several ways to speed up your function.  Since none of these suggestions have anything to do with the json.au3 UDF itself, and the fact that your issues have nothing to do with the json.au3 functionality, I'm not sure that this is the right topic for a detailed discussion on how it could be done.  So without going into detail, I will just list a few items/questions for you to think about in  terms of speeding up that function:

  1. Do you really need an array at all?  Basically you are moving all of the data from one structured format (json/dictionary) that can be queried to another one (variant array), that seems totally unnecessary and time-consuming.
  2. Do you really need to have all of those values in your array?  If you don't need or use values like create_at/update_at/etc, why take the time to move them to a separate array?
  3. Working with arrays and strings are 2 of the slowest processes you can do in AutoIt.  The fewer execution of array and string functions you need to create your array, the faster that process will be.
Edited by TheXman
Link to comment
Share on other sites

1 hour ago, TheXman said:

@Mateocedillo

I took a quick look at your _FakeYouGetVoicesList() function.  I'm not sure how long your functions takes on your test PC's but, I have written a function that creates the voices array in a little over have a second on my PC. My array includes all of the values in each model, just like your function.

Output:

Time to create array: 0.518 seconds
Number of rows:       1941
Number of cols:       15

My script uses jq to create the array because I think it is better suited for this particular task.  Since I think discussing alternative solutions in detail, in someone else's topic is rude, I will not go into detail on how I did it here.  If you want to see suggestions on how it could be done using jq, then I would suggest that you create a new topic with any questions that you may have related to using jq.

If you want to stick with the json.au3 UDF, then there are several ways to speed up your function.  Since none of these suggestions have anything to do with the json.au3 UDF itself, and the fact that your issues have nothing to do with the json.au3 functionality, I'm not sure that this is the right topic for a detailed discussion on how it could be done.  So without going into detail, I will just list a few items/questions for you to think about in  terms of speeding up that function:

  1. Do you really need an array at all?  Basically you are moving all of the data from one structured format (json/dictionary) that can be queried to another one (variant array), that seems totally unnecessary and time-consuming.
  2. Do you really need to have all of those values in your array?  If you don't need or use values like create_at/update_at/etc, why take the time to move them to a separate array?
  3. Working with arrays and strings are 2 of the slowest processes you can do in AutoIt.  The fewer execution of array and string functions you need to create your array, the faster that process will be.

I understand. Well, anyway feel free to send pull requests and so on, although I know I have to do my part and I take it into account. It takes me about 20-30 seconds to finish. And yes, I think the last items (created at and updated at) may not be necessary for now, so I'll think about doing something to speed up the process, as well as alternatives. Thank you anyway...

Link to comment
Share on other sites

On 7/4/2022 at 11:32 AM, Mateocedillo said:

Well, anyway feel free to send pull requests...Thank you anyway...

Unfortunately, without a compelling reason, I have no intention of doing any pull requests or otherwise doing any of your scripting for you.  You're welcome anyway... 

If you are interested in learning more about jq and how to use it, I may be willing to help you with that.

 

On 7/4/2022 at 11:32 AM, Mateocedillo said:

It takes me about 20-30 seconds to finish

If it takes your script 20-30 seconds to build your array using json.au3 functions and it only takes 0.5 seconds to build the same array using jq, then it might be worth taking the time to learn how to use jq.  JSMN (json.au3) is a great JSON parser.  jq is a great JSON processor

Generating a tab-separated-value list of all of the data needed to build the array is done in a single function call to jq, which returns its result in well under 0.5 seconds (it took an additional 0.1 seconds to convert the TSV list to an array which came to a total elapsed time of just over 0.5 seconds).  Your current way is to loop through the 1900+ array objects, with each object containing 15 key/value pairs, doing over 29,000 json_get() calls in order to parse and put each individual value into your array.  I think it is pretty clear why your way takes considerably longer (20-30 seconds) to generate its result.

This was an excellent example of the difference between JSON parsing and JSON processing.  It was also an excellent example of how using the right tool(s) for the job can make a HUGE difference!

Edited by TheXman
Link to comment
Share on other sites

  • 2 months later...


Ok, so sorry if this is covered or if I do not include all the correct information, I'm no seasoned programmer just trying to learn what I need to get by. 

I have managed to learn over the last few weeks how to decompile a msgpack string into json and then use this UDF to get/put data and recompile back to msgpack. 

I am stuck on what seems a simple problem that no suggestions in this thread have worked and I see this has been noted as an issue but the latest UDF is not allowing me to progress. I am using the UDF by Ward, 2021.11.20

The issue is when a period is within the key itself so I have this sample json here I am trying to read the VST.magic and VST3.uid values: 
 

{
  "VST.magic","Test"
  "VST3.uid",[
    2229853791
    2455052834
    2533024787
    1016289816
  ]
  "pluginName","Omnisphere"
  "pluginVendor","Spectrasonics"
}

I have this example code: 

$sJsonString =  FileRead(@ScriptDir & "\Test_PLID_Data.json")
Json_Dump($sJsonString)
$oJsonString = Json_Decode($sJsonString)
;$aKey = $oJsonString.keys

$data = Json_Get($oJsonString, '.VST.magic')
ConsoleWrite("RESULT = " & $data & @CRLF)

I cannot get any result  trying to read these keys, just blank.

This works fine: 
$data = Json_Get($oJsonString, '.pluginName')

And the above will work if I remove the period from the key and try read it again so it is specific to the period in the key.

I saw this as an issue some time ago and some workaround proposed do not work for me. I have tried: 

$data = Json_Get($oJsonString, '."VST"."magic"')

$data = Json_Get($oJsonString, '.VST."magic"')

$data = Json_Get($oJsonString, '[VST][magic]')

$data = Json_Get($oJsonString, '[VST]["magic"]')

And probably every variation of all the above attempts but cannot get anything to work. 

EDIT: My current method to work around this since these are the only 2 keys that have a period is to StringReplace these with _ to allow my code to do what needs to be done then substitute the period back again when writing the string out again. Not perfect but works.

Edited by jestermgee
Link to comment
Share on other sites

On 9/19/2022 at 5:35 AM, jestermgee said:


Ok, so sorry if this is covered or if I do not include all the correct information, I'm no seasoned programmer just trying to learn what I need to get by. 

I have managed to learn over the last few weeks how to decompile a msgpack string into json and then use this UDF to get/put data and recompile back to msgpack. 

I am stuck on what seems a simple problem that no suggestions in this thread have worked and I see this has been noted as an issue but the latest UDF is not allowing me to progress. I am using the UDF by Ward, 2021.11.20

The issue is when a period is within the key itself so I have this sample json here I am trying to read the VST.magic and VST3.uid values: 
 




{
  "VST.magic","Test"
  "VST3.uid",[
    2229853791
    2455052834
    2533024787
    1016289816
  ]
  "pluginName","Omnisphere"
  "pluginVendor","Spectrasonics"
}

I have this example code: 

$sJsonString =  FileRead(@ScriptDir & "\Test_PLID_Data.json")
Json_Dump($sJsonString)
$oJsonString = Json_Decode($sJsonString)
;$aKey = $oJsonString.keys

$data = Json_Get($oJsonString, '.VST.magic')
ConsoleWrite("RESULT = " & $data & @CRLF)

I cannot get any result  trying to read these keys, just blank.

This works fine: 
$data = Json_Get($oJsonString, '.pluginName')

And the above will work if I remove the period from the key and try read it again so it is specific to the period in the key.

I saw this as an issue some time ago and some workaround proposed do not work for me. I have tried: 

$data = Json_Get($oJsonString, '."VST"."magic"')

$data = Json_Get($oJsonString, '.VST."magic"')

$data = Json_Get($oJsonString, '[VST][magic]')

$data = Json_Get($oJsonString, '[VST]["magic"]')

And probably every variation of all the above attempts but cannot get anything to work. 

EDIT: My current method to work around this since these are the only 2 keys that have a period is to StringReplace these with _ to allow my code to do what needs to be done then substitute the period back again when writing the string out again. Not perfect but works.

 

 

You might want to try something like this:
 

$sJsonString =  FileRead(@ScriptDir & "\Test_PLID_Data.json")
$JsonObj = Json_Decode($sJsonString)

If Json_IsObject($JsonObj) Then
    Local $Keys = Json_ObjGetKeys($JsonObj)
    Local $Index = _ArraySearch($Keys,"VST.Magic")
    Local $data[1] = [$JsonObj.Items[$Index]]
    ConsoleWrite("RESULT1 = " & $data[0] & @CRLF)
EndIf

Also you can get direct array results from the object like this:
If $JsonObj.Keys[0]=='VST.Magic' Then.....
If $JsonObj.Items[0]=='Test' Then.....

See also:
https://learn.microsoft.com/en-us/office/vba/language/reference/user-interface-help/items-method

The UDF can only do so much things, in some cases you will have to supply your own routines to handle the objects.

 

 

Edited by lbsl
Link to comment
Share on other sites

In case anyone desires to have the dump-function output to a 2D array rather than a string, replace the _json_tokenDump with this one:

Func _Json_TokenDump(ByRef $Json, $Ptr, ByRef $Next, $ObjPath = "")
    If $Next = -1 Then Return Null

    Local $Token = DllStructCreate("int;int;int;int", $Ptr + ($Next * 20))
    Local $Type = DllStructGetData($Token, 1)
    Local $Start = DllStructGetData($Token, 2)
    Local $End = DllStructGetData($Token, 3)
    Local $Size = DllStructGetData($Token, 4)
    Local $Value
    $Next += 1

    If $Type = 0 And $Start = 0 And $End = 0 And $Size = 0 Then ; Null Item
        $Next = -1
        Return Null
    EndIf

    Switch $Type
        Case 0 ; Json_PRIMITIVE
            Local $Primitive = StringMid($Json, $Start + 1, $End - $Start)
            Switch $Primitive
                Case "true"
                    Return "True"
                Case "false"
                    Return "False"
                Case "null"
                    Return "Null"
                Case Else
                    If StringRegExp($Primitive, "^[+\-0-9]") Then
                        Return Number($Primitive)
                    Else
                        Return Json_StringDecode($Primitive)
                    EndIf
            EndSwitch

        Case 1 ; Json_OBJECT
            For $i = 0 To $Size - 1 Step 2
                Local $Key = _Json_TokenDump($Json, $Ptr, $Next)
                Local $cObjPath = $ObjPath & "." & $Key
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $ObjPath & "." & $Key)
                If Not (IsBool($Value) And $Value = False) Then
                    If Not IsString($Key) Then
                        $Key = Json_Encode($Key)
                    EndIf
                    If Not IsArray($Total_JSON_DUMP_Output) Then
                        Dim $Total_JSON_DUMP_Output[1][2]
                    Else
                        ReDim $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) + 1][2]
                    EndIf                    ; show the key and its value
                    ConsoleWrite("+-> " & $cObjPath & '  =' & $Value & @CRLF)
                    $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) - 1][0] = $cObjPath
                    $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) - 1][1] = $Value
                    ;$Total_JSON_DUMP_Output &= "+-> " & $cObjPath & '  =' & $Value & @CRLF
                EndIf
            Next
            Return False
        Case 2 ; Json_ARRAY
            Local $sObjPath = $ObjPath
            For $i = 0 To $Size - 1
                $sObjPath = $ObjPath & "[" & $i & "]"
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $sObjPath)
                If Not (IsBool($Value) And $Value = False) Then ;XC - Changed line
                    ; show the key and its value
                    If Not IsArray($Total_JSON_DUMP_Output) Then
                        Dim $Total_JSON_DUMP_Output[1][2]
                    Else
                        ReDim $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) + 1][2]
                    EndIf
                    ConsoleWrite("+=> " & $sObjPath & "=>" & $Value & @CRLF)
                    $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) - 1][0] = $sObjPath
                    $Total_JSON_DUMP_Output[UBound($Total_JSON_DUMP_Output) - 1][1] = $Value
                    ;$Total_JSON_DUMP_Output &= "+=> " & $sObjPath & "=>" & $Value & @CRLF
                EndIf
            Next
            $ObjPath = $sObjPath
            Return False

        Case 3 ; Json_STRING
            Local $LastKey = Json_StringDecode(StringMid($Json, $Start + 1, $End - $Start))
            Return $LastKey
    EndSwitch
EndFunc   ;==>_Json_TokenDump

 

Link to comment
Share on other sites

  • 2 weeks later...

Here is a function that will allow you to get a key that contains a period such as a website address.

It will only find a direct child of the specified Json object so you need to find the path just before it using the normal Json_Get.  example is in the function description.

 

;Used to get a direct child item under the specified object.
;  written specifically to obtain those items which contain periods . in the item names that the normal functions will attempt to break out into seperate items
;  Ex:  $oUserList = Json_Get($oJson, '.dual_engine.user_list_data_1')          Find the parent of the website key
;       $oWebsite = Json_ObjGetChild($oUserList, "https://www.yahoo.com/")      Find the website key
;       $sAddedOn = Json_Get($oWebsite, ".date_added")                          Use the returned Json object to get a value under the website key
;By BigDaddyO
Func Json_ObjGetChild(ByRef $Object, $Key)
    If Not IsObj($Object) Then Return SetError(1)

    Local $DynObject = $Object
    If $DynObject.Exists($Key) Then
        Return $DynObject.Item($Key)
    EndIf

    Return SetError(2, 0, '')
EndFunc

 

Link to comment
Share on other sites

Back when the issue regarding not being able to access keys that have periods in them was posted HERE on 9/18, I took a look at the issue and fixed my local copy.  Then, I saw the post HERE on 9/28 that had the same issue.  After mulling it over a bit, I decided to share my fix (attached below).  Even though there are other JSON UDF's available that do not have this issue, I decided that it was still important to offer a fix since this UDF lib is widely used and very fast when doing simple parsing.

Of course one can access keys with periods in them by directly accessing the underlying dictionary, where the key/value pairs are actually stored.  However, that doesn't address being able to access & manipulate such keys, using dot-notation and bracket-notation, with the Json_Get() and Json_Put() functions.

Only 4 lines in the most current version needed to be modified, 2 lines in Json_Get() and 2 lines in Json_Put().  I could go into a detailed analysis of the root cause(s) of the issue and why I chose to fix it the way I did, but I'm sure that most don't care.  All anyone really cares about is that it works.  😉  I left the Json_ObjGet() and Json_ObjExists() helper functions as-is.  Although those 2 helper functions are related to the issue, they are not needed for Json_Get() and Json_Put() to correctly traverse a mutli-level JSON notation that has special characters like slashes, spaces, brackets, and periods.

The example script below shows keys with periods, spaces, brackets, and slashes, being created, accessed, and modified, using Json_Put() and Json_Get():

#include <Constants.au3>
;~ #include "Json.au3"
#include "Json(2022-09-18)_TheXman.au3" ;<== Modified json.au3 (only 4 lines updated)

Const $JSON = _
    '{'                                                   & _
    '  "dual_engine": {'                                  & _
    '    "user_list_data_1": {'                           & _
    '       "https://www.yahoo.com/": {'                  & _
    '          "date_added": "13308840098237395",'        & _
    '          "engine": 2,'                              & _
    '          "visits_after_expiration": 0'              & _
    '       }'                                            & _
    '    }'                                               & _
    '  },'                                                & _
    '  "VST.magic": "Test",'                              & _
    '  "VST3.uid": [2229853791, 2455052834, 2533024787],' & _
    '  "pluginName": "Omnisphere",'                       & _
    '  "pluginVendor": "Spectrasonics"'                   & _
    '}'

json_example()

Func json_example()
    Local $oJson

    ;Decode JSON into a dictionary object
    $oJson = Json_Decode($JSON)
    If @error Then Exit MsgBox($MB_ICONERROR + $MB_TOPMOST, "JSON DECODE ERROR", "@error = " & @error)


    ;Pretty-print the original JSON
    log_line("===  Pretty-Printed JSON (original)  ===")
    log_line(Json_Encode($oJson, $JSON_PRETTY_PRINT + $JSON_UNESCAPED_SLASHES, "   ") & @CRLF)


    ;Update an existing key / Add some valid keys with spaces, slashes, and periods
    Json_Put($oJson, '.dual_engine.user_list_data_1."https://www.yahoo.com/".visits_after_expiration', 3) ;Update key's value to 3

    Json_Put($oJson, '."//a.b.c//".". [.] ."', "Value of ridiculous but valid key (. [.] .)")      ;Add key
    Json_Put($oJson, '."//a.b.c//"."/ /"'    , "Value of another ridiculous but valid key (/ /)")  ;Add key
    Json_Put($oJson, '."key with spaces"'    , "Value of key with spaces")                         ;Add key

    Json_Put($oJson, '."lvl1.key"."lvl2.key1"."lvl3.key1"'   , "Level 3 Key1")  ;Add key using dot-notation
    Json_Put($oJson, '["lvl1.key"]["lvl2.key1"]["lvl3.key2"]', "Level 3 Key2")  ;Add key using bracket-notation


    ;Pretty-print the updated JSON
    log_line("===  Pretty-Printed JSON (with keys updated/added)  ===")
    log_line(Json_Encode($oJson, $JSON_PRETTY_PRINT + $JSON_UNESCAPED_SLASHES, "   ") & @CRLF)


    ;Display some of the JSON values
    log_line('===  Display some JSON keys & values  ===')
    log_line('."//a.b.c//".". [.] ." => ' & Json_Get($oJson, '."//a.b.c//".". [.] ."'))
    log_line('."//a.b.c//"."/ /"     => ' & Json_Get($oJson, '."//a.b.c//"."/ /"'))
    log_line('."key with spaces"     => ' & Json_Get($oJson, '."key with spaces"'))
    log_line()
    log_line('.dual_engine.user_list_data_1."https://www.yahoo.com/".date_added              => ' & _
             Json_Get($oJson, '.dual_engine.user_list_data_1."https://www.yahoo.com/".date_added'))
    log_line('[dual_engine][user_list_data_1]["https://www.yahoo.com/"][engine]              => ' & _
             Json_Get($oJson, '[dual_engine][user_list_data_1]["https://www.yahoo.com/"][engine]'))
    log_line('.dual_engine.user_list_data_1."https://www.yahoo.com/".visits_after_expiration => ' & _
             Json_Get($oJson, '.dual_engine.user_list_data_1."https://www.yahoo.com/".visits_after_expiration'))
    log_line()
    log_line('."VST.magic"   => ' & Json_Get($oJson, '."VST.magic"'))
    log_line('."VST3.uid"[0] => ' & Json_Get($oJson, '."VST3.uid"[0]'))
    log_line()
    log_line('."lvl1.key"."lvl2.key1"."lvl3.key1"    => ' & Json_Get($oJson, '."lvl1.key"."lvl2.key1"."lvl3.key1"'))
    log_line('["lvl1.key"]["lvl2.key1"]["lvl3.key1"] => ' & Json_Get($oJson, '["lvl1.key"]["lvl2.key1"]["lvl3.key1"]'))
EndFunc

Func log_line($sMsg = "")
    ConsoleWrite($sMsg & @CRLF)
EndFunc

Console Output:

===  Pretty-Printed JSON (original)  ===
{
   "dual_engine": {
      "user_list_data_1": {
         "https://www.yahoo.com/": {
            "date_added": "13308840098237395",
            "engine": 2,
            "visits_after_expiration": 0
         }
      }
   },
   "VST.magic": "Test",
   "VST3.uid": [
      2229853791,
      2455052834,
      2533024787
   ],
   "pluginName": "Omnisphere",
   "pluginVendor": "Spectrasonics"
}

===  Pretty-Printed JSON (with keys updated/added)  ===
{
   "VST.magic": "Test",
   "VST3.uid": [
      2229853791,
      2455052834,
      2533024787
   ],
   "pluginName": "Omnisphere",
   "pluginVendor": "Spectrasonics",
   "dual_engine": {
      "user_list_data_1": {
         "https://www.yahoo.com/": {
            "date_added": "13308840098237395",
            "engine": 2,
            "visits_after_expiration": 3
         }
      }
   },
   "//a.b.c//": {
      ". [.] .": "Value of ridiculous but valid key (. [.] .)",
      "/ /": "Value of another ridiculous but valid key (/ /)"
   },
   "key with spaces": "Value of key with spaces",
   "lvl1.key": {
      "lvl2.key1": {
         "lvl3.key1": "Level 3 Key1",
         "lvl3.key2": "Level 3 Key2"
      }
   }
}

===  Display some JSON keys & values  ===
."//a.b.c//".". [.] ." => Value of ridiculous but valid key (. [.] .)
."//a.b.c//"."/ /"     => Value of another ridiculous but valid key (/ /)
."key with spaces"     => Value of key with spaces

.dual_engine.user_list_data_1."https://www.yahoo.com/".date_added              => 13308840098237395
[dual_engine][user_list_data_1]["https://www.yahoo.com/"][engine]              => 2
.dual_engine.user_list_data_1."https://www.yahoo.com/".visits_after_expiration => 3

."VST.magic"   => Test
."VST3.uid"[0] => 2229853791

."lvl1.key"."lvl2.key1"."lvl3.key1"    => Level 3 Key1
["lvl1.key"]["lvl2.key1"]["lvl3.key1"] => Level 3 Key1

 

 

Json(2022-09-18)_TheXman.au3

Edited by TheXman
Link to comment
Share on other sites

  • 3 months later...
  • Developers
8 minutes ago, antai said:

Where is the json.au3 download link?

Initial topic post link not working?

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...