Jump to content
Roy_

**SOLVED - Curl Json Post send base64 encoded instead of utf8 text

Recommended Posts

Ciao,

This example reproduces the problem I encounter when sending a post request to an echo server.
Instead of being sent as text utf8, the data is transmitted base64 encoded.

Example:
- json String: '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'

- server response: "data:application/octet-stream;base64,AAAAALgE6QNYAIAAAgAAANQCAAAAAAAAWF8AAAAAAAD0VQAAAAAAAD9APQAAAAAAAwAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAABBAAAAAAABRUAAAAmIZwcAbRt8/LMa9UAAAAAAAAAAGFzIGlkIHRlbXB1cyBtaS4gTW9yYmkgaXBzdW0gZW5pbSwgdWx0cmljaWVzIGFjIGF1Z3VlIHNpdCBhbWV0LCB1bGxhbWNvcnBlciBmaW5pYnVzIGV4LiBWZXN0aWJ1bHVtIHZlbCBwb3N1ZXJlIG5pYmgsIG5lYyBmYXVjaWJ1cyBlcm9zLiBOYW0gbWFsZXN1YWRhIG5vbiBsYWN1cyBhIHN1c2NpcGl0LiBOdWxsYSByaG9uY3VzIHRlbXB1cyBtaSBxdWlzIHBsYWNlcmF0LiBDdXJhYml0dXIgY29tbW9kbyB0aW5jaWR1bnQganVzdG8gcXVpcyBzb2xsaWNpdHVkaW4uIn0="

 

How can I go about transmitting data in text format utf8?
Thanks in advance for the help.

In the zip file:
- CurlJsonPost.au3: this script
- Curl.au3: UDF by Ward (thank you!)
- data.json: json srting for command line test

 

Note: using Curl.exe with json string saved in a file (utf8 encoded) named data.json, works perfectly
To try with the command line tool:
- save data.json in curl\bin directory
- open cmd.exe and cd to curl\bin directory
- Enter the following command:
curl -H "Content-Type: application/json" --data @data.json https://httpbin.org/post

 

#Include "Curl.au3"

Global $_cURL_OutputBuffer

Local $sJson  = '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'
Local $iRetCode, $sServerResponse

ConsoleWrite("=== Json post test ===" & @LF)
ConsoleWrite(StringFormat("Retcode: %s - %s", $iRetCode, Curl_Easy_strerror($iRetCode)) & @LF)
ConsoleWrite("Data returned from server" & @LF & @LF)
$sServerResponse = JsonPost_Test($sJson, $iRetCode)
ConsoleWrite($sServerResponse & @LF)


Func JsonPost_Test($sJson, ByRef $iRetCode)

    ; Init Easy Curl Interface e set url (echo service)
    Local $oCurl = Curl_Easy_Init()
    curl_easy_setopt($oCurl, $CURLOPT_URL, "https://httpbin.org/post")

    ; Set content type header
    Local $headers = curl_slist_append(0, "Content-Type: application/json")
    curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers)

    ; Post fields & size
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDSIZE, StringLen($sJson))

    ; Set callbac function to get server response back (see global var $_cURL_OutputBuffer)
    $hWriteFunc = DllCallbackRegister("WriteFunc_CallBack", "uint:cdecl", "ptr;uint;uint;ptr")
    curl_easy_setopt($oCurl, $CURLOPT_WRITEFUNCTION, DllCallbackGetPtr($hWriteFunc))

    ; Ignore ssl certificates check
    curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYPEER, 0)
    curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYHOST, 0)

    ; Execute the post request
    $iRetCode = curl_easy_perform($oCurl)

    ; Set return trasfer & clear output buffer global var
    Local $sReturnTransfer = $_cURL_OutputBuffer
    $_cURL_OutputBuffer = ""

    Return SetError(0, 0, $sReturnTransfer)
EndFunc

Func WriteFunc_CallBack($ptr,$nSize,$nMemb,$pStream)
    Local $vData = DllStructCreate ("byte[" & $nSize*$nMemb & "]",$ptr)
    $_cURL_OutputBuffer &= BinaryToString(DllStructGetData($vData,1))
    Return $nSize*$nMemb
EndFunc

 

curlJsonPost.zip

Edited by Roy_

Share this post


Link to post
Share on other sites

I have just found the solution and I want to share it with you

The secret lies in passing the json data as a pointer instead of as a string.

So changing these lines ...

; Post fields & size
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson)

with these:

; Post fields & size
    Local $stString = DllStructCreate("char v[" & StringLen($sJson) & "]")
    DllStructSetData($stString, 1, $sJson)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, DllStructGetPtr($stString))

and it works perfectly, returning this response:

=== Json post test ===
Retcode:  - No error
Data returned from server

{
  "args": {}, 
  "data": "{\"firstName\":\"Jonathan\",\"lastName\":\"Freeman\",\"loginCount\":4,\"active\": \"yes\",\"text\":\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin.\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "437", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-5e3a9a12-f59424ee2b283d6ca7874c48"
  }, 
  "json": {
    "active": "yes", 
    "firstName": "Jonathan", 
    "lastName": "Freeman", 
    "loginCount": 4, 
    "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."
  }, 
  "origin": "87.8.171.14", 
  "url": "https://httpbin.org/post"
}

Attached correct version.

Roy

CurlJsonPostOk.au3

Share this post


Link to post
Share on other sites

... and finally if you have to send utf-8 characters like àèéìòù and so on:

1) specify this header

; Set content type header & specify charset
    Local $headers = curl_slist_append(0, "Content-Type: application/json;charset=UTF-8")
    curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers)

2) convert your json to utf-8 charset like this

; Post fields & size
    Local $sUtf8 = _WinAPI_WideCharToMultiByte($sJson, 65001)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sUtf8)

that's all.

Enjoy curl :)

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By argumentum
      #include <SQLite.au3> ;-- When SQLite is compiled with the JSON1 extensions it provides builtin tools ;-- for manipulating JSON data stored in the database. ;-- This is a gist showing SQLite return query as a JSON object. ;-- https://www.sqlite.org/json1.html Example() Func Example() _SQLite_Startup() ; "<your path>\sqlite3.dll", False, 1) ; https://www.autoitscript.com/autoit3/docs/libfunctions/_SQLite_Startup.htm _SQLite_Open() ; ...if you can not run this due to errors, get the latest DLL from https://www.sqlite.org/ If _SQLite_Exec(-1, "CREATE TABLE users (id INTEGER PRIMARY KEY NOT NULL, full_name TEXT NOT NULL, email TEXT NOT NULL, created DATE NOT NULL );") Then Return 4 If _SQLite_Exec(-1, 'INSERT INTO users VALUES ' & _ '(1, "Bob McFett", "bmcfett@hunters.com", "32-01-01"),' & _ '(2, "Angus O''Vader","angus.o@destroyers.com", "02-03-04"),' & _ '(3, "Imperator Colin", "c@c.c", "01-01-01");') Then Return 5 ; -- Get query data as a JSON object using the ; -- json_group_object() [1] and json_object() [2] functions. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_object(" & _ " email," & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS json_result" & _ " FROM (SELECT * FROM users WHERE created > ""02-01-01"");") ; {"bmcfett@hunters.com":{"full_name":"Bob McFett","created":"32-01-01"},"angus.o@destroyers.com":{"full_name":"Angus O'Vader","created":"02-03-04"}} ; -- Get query data as a JSON object using the ; -- json_group_array() function to maintain order. _SQLite_GetTable2d_ArrayToConsole("SELECT" & _ " json_group_array(" & _ " json_object('full_name', full_name, 'created', created)" & _ " ) AS my_json_result_OrAnythingReally" & _ " FROM (SELECT * FROM users ORDER BY created);") ; [{"full_name":"Imperator Colin","created":"01-01-01"},{"full_name":"Angus O'Vader","created":"02-03-04"},{"full_name":"Bob McFett","created":"32-01-01"}] ;-- Links ;-- [1] https://www.sqlite.org/json1.html#jgroupobject ;-- [2] https://www.sqlite.org/json1.html#jobj ; example found at https://gist.github.com/akehrer/481a38477dd0518ec0086ac66e38e0e2 EndFunc ;==>Example Func _SQLite_GetTable2d_ArrayToConsole($sSQL, $hDB = -1) Local $aResult, $iRows, $iColumns If _SQLite_GetTable2d($hDB, $sSQL, $aResult, $iRows, $iColumns) Then ConsoleWrite("! SQLite Error: " & _SQLite_ErrCode($hDB) & @CRLF & "! " & _SQLite_ErrMsg($hDB) & @CRLF) Else _SQLite_Display2DResult($aResult) EndIf ConsoleWrite(@CRLF) EndFunc ;==>_SQLite_GetTable2d_ArrayToConsole Based on this example, you can build your own query. 
      The code has all the explanations.
      Enjoy  
    • By matwachich
      Hi AutoIters!
      Here is my new UDF about GUIs: it's an enhanced mixture of Advanced InputBox (deprecated) and KODA Parser (deprecated), with additional functions.
      What you can do with it:
      Parse KODA files and directly create GUIs (_GUIUtils_CreateFromKODA) Parse a simple JSON form definition to simply create advanced InputBoxes with any amount/type of input controls (_GUIUtils_CreateFromJSON) Created GUIs are returned as Scripting.Dictionary objects, and you have helper functions to access GUIs controls by their names (_GUIUtils_HWnd, _GUIUtils_CtrlID, _GUIUtils_HCtrl ...) A function that can make a GUI created from KODA/JSON (defined by it's $oForm object) and make it a modal InputDialogBox (just like InputBox, but returns all entered data as Scripting.Dictionary object) Functions are documented, and there are some examples.
      Consider this UDF as beta, but since I'm currently using it in a small commercial project, it should become production ready in near future.
      To always get the latest code version, get it from Github (more up to date than this topic).
      Update 04/02/2020:
      New simple application example Fixed focused control handling in _GUIUtils_InputDialog Reset input values before returning from _GUIUtils_InputDialog Fixed CloseOnEsc in _GUIUtils_InputDialog Fixed Input not reset Fixed all ListBox items are selected Project now on GitHub https://github.com/matwachich/autoit-guiutils/
      Update 05/02/2020:
      Support for nodate for Date and Time input controls (pass null value) New handling of focused control _GUIUtils_InputDialog: Now, you can specify the focused control in $oInitialData by setting "controlName:focus" = True Updated documentation of _GUIUtils_InputDialog Update 07/02/2020:
      Bug corrected when setting Data input control Added: abillity to read a single input control New functions (_GUIUtils_GetInputs, _GUIUtils_WriteInputs), documentation completion. Readme file and screenshots on Github Page  
      GUIUtils.zip
    • By D2thunder
      Hi all, I wasnt active in this Forum forum for Years. 😅
      Since I need something with Json in AutoIt i write a little UDF to save a variable to file to read it with another language - and back. After a while until my other script grows i thought - lets integrate all variables wich is possible, because i want have arrays to.
      This is my first public UDF in Autoit - normaly my AutoIt scripts are quick an dirty.... - like the examples for this udf. - someone want to write nice examples?
      I tried to write this UDF properly, cared for processing speed, low mem consumption as possible, a short code, and a high usability without beeing very familiar with arrays.
      What is it for? - convert nearly all types of variables to a json string for saving it in a file or to an ini. Read it back in to a same formated variable and preserv variable types. All array types are supported with nested arrays, the only restriction is ram and array/stringsize - and time if the array is very big. You can use Filters for variable types and some other options.
      If anyone found a bug or any suggestions, please post it.
       
      get Json.au3 from here Json.au3
       
       
      JsonVar 2019.01.25.1.zip
    • By JoeBar
      Hi, i'm trying to implement Imgur API functions in an AutoIt program.
      I successfully coded the oAuth2 Authentication and i'm testing some Imgur functions.
       
      Example : https://api.imgur.com/3/account/me/images
      The string received is like :
      {"data":[{"id":"fd6f54s","title":null,"description":null,"datetime":1574502473,"type":"image\/png","animated":false,"width":1147,"height":715,"size":53516,"views":18,"bandwidth":963288,"vote":null,"favorite":false,"nsfw":null,"section":null,"account_url":"ACCOUNT","account_id":ACCOUNTID,"is_ad":false,"in_most_viral":false,"has_sound":false,"tags":[],"ad_type":0,"ad_url":"","edited":"0","in_gallery":false,"deletehash":"d6f5sd4fsf","name":"image","link":"https:\/\/i.imgur.com\/fd6f54s.png"} I'm using JSON.au3 UDF and i don't manage to make it working :
      Local $Obj = Json_Decode($hRequestSSL) Local $Links = Json_Get($Obj, '["data"]["link"]') The Json_Decode sends in console :
      "E:\Portable\Dev\AutoIt3\IncludeP\JSON.au3" (374) : ==> Variable must be of type "Object".: Return $Object.Keys() Return $Object^ ERROR It says it's not an object or i have seen in some UDF that Json_Decode takes a string as parameter.
      I don't know what's wrong.
    • By Ward
      Introduction
      JSON (Javascript Object Notation) is a popular data-interchange format and supported by a lot of script languages. On AutoIt, there is already a >JSON UDF written by Gabriel Boehme. It is good but too slow, and not supports unicode and control characters very well. So I write a new one (and of course, fast one as usual).

      I use a machine code version of JSON parser called "jsmn". jsmn not only supports standard JSON, but also accepts some non-strict JSON string. See below for example.
       
      Important Update!!
      I rename the library from jsmn.au3 to json.au3. All function names are changed, too.
       
      Decoding Function
      Json_Decode($Json) $Json can be a standard or non-standard JSON string. For example, it accepts:
      { server: example.com port: 80 message: "this looks like a config file" } The most JSON data type will be decoded into corresponding AutoIt variable, including 1D array, string, number, true, false, and null. JSON object will be decoded into "Windows Scripting Dictionary Object" retuned from ObjCreate("Scripting.Dictionary"). AutoIt build-in functions like IsArray, IsBool, etc. can be used to check the returned data type. But for Object and Null, Json_IsObject() and Json_IsNull() should be used.

      If the input JSON string is invalid, @Error will be set to $JSMN_ERROR_INVAL. And if the input JSON string is not finish (maybe read from stream?), @Error will be set to $JSMN_ERROR_PART.

      Encoding Function
      Json_Encode($Data, $Option = 0, $Indent = "\t", $ArraySep = ",\r\n", $ObjectSep = ",\r\n", $ColonSep = ": ") $Data can be a string, number, bool, keyword(default or null), 1D arrry, or "Scripting.Dictionary" COM object. Ptr will be converted to number, Binary will be converted to string in UTF8 encoding. Other unsupported types like 2D array, dllstruct or object will be encoded into null.

      $Option is bitmask consisting following constant:
      $JSON_UNESCAPED_ASCII ; Don't escape ascii charcters between chr(1) ~ chr(0x1f) $JSON_UNESCAPED_UNICODE ; Encode multibyte Unicode characters literally $JSON_UNESCAPED_SLASHES ; Don't escape / $JSON_HEX_TAG ; All < and > are converted to \u003C and \u003E $JSON_HEX_AMP ; All &amp;amp;amp;amp;s are converted to \u0026 $JSON_HEX_APOS ; All ' are converted to \u0027 $JSON_HEX_QUOT ; All " are converted to \u0022 $JSON_PRETTY_PRINT ; Use whitespace in returned data to format it $JSON_STRICT_PRINT ; Make sure returned JSON string is RFC4627 compliant $JSON_UNQUOTED_STRING ; Output unquoted string if possible (conflicting with $JSMN_STRICT_PRINT) Most encoding option have the same means like PHP's json_enocde() function. When $JSON_PRETTY_PRINT is set, output format can be change by other 4 parameters ($Indent, $ArraySep, $ObjectSep, and $ColonSep). Because these 4 output format parameters will be checked inside Jsmn_Encode() function, returned string will be always accepted by Jsmn_Decode(). $JSON_UNQUOTED_STRING can be used to output unquoted string that also accetped by Jsmn_Decode().

      $JSON_STRICT_PRINT is used to check output format setting and avoid non-standard JSON output. So this option is conflicting with $JSON_UNQUOTED_STRING.

      Get and Put Functions
      Json_Put(ByRef $Var, $Notation, $Data, $CheckExists = False) Json_Get(ByRef $Var, $Notation) These functions helps user to access object or array more easily. Both dot notation and square bracket notation can be supported. Json_Put() by default will create non-exists objects and arrays. For example:
      Local $Obj Json_Put($Obj, ".foo", "foo") Json_Put($Obj, ".bar[0]", "bar") Json_Put($Obj, ".test[1].foo.bar[2].foo.bar", "Test") Local $Test = Json_Get($Obj, '["test"][1]["foo"]["bar"][2]["foo"]["bar"]') ; "Test" Object Help Functions
      Json_ObjCreate() Json_ObjPut(ByRef $Object, $Key, $Value) Json_ObjGet(ByRef $Object, $Key) Json_ObjDelete(ByRef $Object, $Key) Json_ObjExists(ByRef $Object, $Key) Json_ObjGetCount(ByRef $Object) Json_ObjGetKeys(ByRef $Object) Json_ObjClear(ByRef $Object) These functions are just warps of "Scripting.Dictionary" COM object. You can use these functions if you are not already familiar with it.
       
      == Update 2013/05/19 ==
       
      * Add Jsmn_Encode() option "$JSMN_UNESCAPED_ASCII". Now the default output of Json_Encode() is exactly the same as PHP's json_encode() function (for example, chr(1) will be encoded into u0001).
      $JSON_UNESCAPED_ASCII ; Don't escape ascii charcters between chr(1) ~ chr(0x1f)  
      == Update 2015/01/08 ==
      * Rename the library from jsmn.au3 to json.au3. All function names are changed, too.
      * Add Json_Put() and Json_Get()
      * Add Null support
      * Using BinaryCall.au3 to loading the machine code.
       
      == Update 2018/01/13==   (Jos)
      * Add JsonDump() to list all Json Keys and their values to easily figure out what they are.
      == Update 2018/10/01==  (Jos)
      * Fixed JsonDump() some fields and values were not showing as discussed here  - tnx @TheXman .
      == Update 2018/10/01b==   (Jos)
      * Added Json_ObjGetItems, Tidied source and fixed au3check warnings - tnx @TheXman .
      == Update 2018/10/28==   (Jos)
      * Added declaration for $value to avoid au3check warning - tnx @DerPensionist
      == Update 2018/12/16==   (Jos)
      * Added another declaration for $value to avoid au3check warning  and updated the version at the top - tnx @maniootek
      == Update 2018/12/29==   (Jos)
      * Changed Json_ObjGet() and Json_ObjExists() to allow for multilevel object in string.
      == Update 2019/01/17==   (Jos)
      * Added support for DOT notation in JSON functions.
      _Json(2019.01.17).zip   reverted to this version as the last version has some regression issues that is being worked on.
      == Update 2019/07/15==   (Jos)
      *  Added support for reading keys with a dot inside when using a dot as separator (updated)
       
×
×
  • Create New...