phamhoangthi

[resolved] Charset problem with POST method

23 posts in this topic

#1 ·  Posted (edited)

Hello, everybody, I want to use Post method, something like Curl in Php, to post the data to the server and then, receive the respond to do something.
But the respond is not in Charset utf-8, so I can't read it.
Here is my code (after seeing some example sample codes):

func POST($URL, $DATA)

        Local $oHTTP = ObjCreate("WinHttp.WinHttpRequest.5.1")
        $oHTTP.Open("POST", $URL, False)
        If (@error) Then Return SetError(1, 0, 0)
        $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
        $oHTTP.SetRequestHeader("Accept-Charset", "UTF-8,*")
        $oHTTP.Send($Data)
        If (@error) Then Return SetError(2, 0, 0)
        If ($oHTTP.Status <> 200) Then Return SetError(3, 0, 0)
        Return SetError(0, 0, $oHTTP.ResponseText) //Resolved: Change this to ResponseBody and everything's ok  
EndFunc


Global $TEXT = POST("http://mylink.com/ba.php","var1=13567567290")
  FileWriteLine("result.html",$TEXT)

and the result file, I can't read anything.

ODackMj.png

How can I get passed it?
Please help me.

Edited by phamhoangthi

Share this post


Link to post
Share on other sites



Since you asked for UTF8, you got UTF8, which is exactly what you display. Convert $TEXT as received from UTF8 to native AutoIt string (UTF16-LE) and you'll be fine.

$TEXT = BinaryToString(StringToBinary($TEXT, <correct parameter #1>), <correct parameter #2>)

Your mission -if you accept it- consists in reading the help to find the missing bits.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Since you asked for UTF8, you got UTF8, which is exactly what you display. Convert $TEXT as received from UTF8 to native AutoIt string (UTF16-LE) and you'll be fine.

$TEXT = BinaryToString(StringToBinary($TEXT, <correct parameter #1>), <correct parameter #2>)

Your mission -if you accept it- consists in reading the help to find the missing bits.

Thanks for helping.

I tried to convert my text to ANSI, and then, UTF8,
But "số điện thoại" became "
S��? �?i��?n thoại" after converting

Share this post


Link to post
Share on other sites

Do you display the string with ConsoleWrite or MsgBox? Conversion to ANSI turns non-ANSI characters into question marks; don't do that.

The former displays strings converted to Windows charset (so-called ANSI) by default while the second displays UTF16 Unicode strings directly.

To make SciTE display UTF8 you have to put the following in User Properties file (see Options):

code.page=65001
output.code.page=65001

Then you can use ConsoleWrite directly to display UTF8 data. To display native AutoIt strings (UTF16) you can use this:

Local $str = "số điện thoại"

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 4), 1)
EndFunc

ConsoleWrite(_StrToUtf8($str) & @LF)

NOTE: I justy checked and it seems that the latest release of SciTE doesn't display UTF8 correctly regardless of code.page setting. So use MsgBox with plain UTF16 strings (AutoIt native) for now until this is fixed.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Do you display the string with ConsoleWrite or MsgBox? Conversion to ANSI turns non-ANSI characters into question marks; don't do that.

The former displays strings converted to Windows charset (so-called ANSI) by default while the second displays UTF16 Unicode strings directly.

To make SciTE display UTF8 you have to put the following in User Properties file (see Options):

code.page=65001
output.code.page=65001

Then you can use ConsoleWrite directly to display UTF8 data. To display native AutoIt strings (UTF16) you can use this:

Local $str = "số điện thoại"

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 4), 1)
EndFunc

ConsoleWrite(_StrToUtf8($str) & @LF)

NOTE: I justy checked and it seems that the latest release of SciTE doesn't display UTF8 correctly regardless of code.page setting. So use MsgBox with plain UTF16 strings (AutoIt native) for now until this is fixed.

Sorry, My SciTE editor can't display unicode anymore after adding code.page=65001, it shows: "s? di?n tho?i" instead of "số điện thoại" and I don't know how to use UTF16 string.
I am originally a web coding and I am still new at this...
In html document, I just need to put a <meta charset='utf8' to solve my problem

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

That was the point of my bottom note in bold. This is being investigated and will be fixed shortly.

BUT, if it displays what you show, that means that you don't feed UTF8 but a conversion to ANSI. Show a short sample of failing code.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

That was the point of my bottom note in bold. This is being investigated and will be fixed shortly.

BUT, if it displays what you show, that means that you don't feed UTF8 but a conversion to ANSI. Show a short sample of failing code.

Yes, I did see your notice and I just didn't understand why.
This is my simple example, these written both in a file and in the console

 

$TEXT = _StrToUtf8("Ä�ã Ä�ến thÆ° viá»�n") ;This is result of POST method from a webpage. This originally is: "đã đến thư viện" that I can read in my language.

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 1), 4)
EndFunc
ConsoleWrite($TEXT)
FileDelete("Google.html")
  FileWriteLine("Google.html",$TEXT)

 

 

Console:

--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
??ã ???n thu vi??n+>17:31:57 AutoIt3.exe ended.rc:0
+>17:31:57 AutoIt3Wrapper Finished.

Output file:

?ã �?ến thư vi�?n

But I need

đã đến thư viện

 

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

Yes, I did see your notice and I just didn't understand why.

Most likely a bug in the SciTE settings.

Your script is mixing things up. The original string "đã đến thư viện" has the following UTF8 representation (in hex):

0xC491C3A320C491E1BABF6E207468C6B0207669E1BB876E

The string in the firrst line of your example is already UTF8 but gets loaded as a series of individual UTF16 characters in AutoIt. So you shouldn't convert that into UTF8, that won't work.

; say $UTF8 is the result you get from the server
Local $UTF8 = Binary("0xC491C3A320C491E1BABF6E207468C6B0207669E1BB876E")
MsgBox(0, "", BinaryToString($UTF8, 4))

 

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

My first line, "�ã �ến thư vi�n" is not only like this in SciTE editor consolewrite but also in output.html file.
I'm sorry that nothing is new, With 

BinaryToString($UTF8, 4)

in html output file, I see this:

"Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!"
meaning:

"Thank you for coming to library, see next time!"

"Cảm ơn bạn and hẹn gặp bạn lần sau! " is great, but not "�?ã �?ến thư vi�?n,"
 I think  those codes can't convert these characters: "Đ, ệ etc..."

 

 

 

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

What do you use to "see" that? UTF8 text has to be displayed accordingly to this encoding. When you "see" "Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!" that means that the text is somewhere converted to ANSI, where non-ANSI characters show up as �

 I think  those codes can't convert these characters: "Đ, ệ etc..."

Of course yes! Correct display only need to handle conversions correctly. The code in the first post results in an english page for me (coded in UTF8).


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

What do you use to "see" that? UTF8 text has to be displayed accordingly to this encoding. When you "see" "Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!" that means that the text is somewhere converted to ANSI, where non-ANSI characters show up as �

Of course yes! Correct display only need to handle conversions correctly. The code in the first post results in an english page for me (coded in UTF8).

 

Yeah! I view the output by using Notepad++, Chrome browser, same result.
And now I have to use Stringreplace or regurlar expression or something like that to fix some error characters? 
I don't think this is a good idea, why I can't receive the respond from a server as Utf8?
Why in Php, The result always looks good?
I am still new in this and I have many questions to answer.

Share this post


Link to post
Share on other sites

Can you post a working example of a website which returns a vietnamese UTF8 answer (I wild guess it's what you target)?

There's no need to StringReplace or regexp anything.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

yeah, I use a Php class to do that, (curl)
Of course this is only an example that I won't  stop at method process()  but anyway this method will return the source of the page after sending POST request. Example: The search result.

class test{
                private $ch; 
                private $page; 
                private $name; 
                private $domain;
                private $parse;
                
            function __construct($var, $studentid){  //declare the curl
                $parse= parse_url($var); //parse_url
                $param = array(
                'var1'=> '123456',  
                'var2' => '654321');                
                $this->domain = $parse['host'];
                $this->ch = curl_init($var);
                curl_setopt($this->ch, CURLOPT_RETURNTRANSFER,true);                
                curl_setopt($this->ch, CURLOPT_POST, count($param));  
                curl_setopt($this->ch,CURLOPT_POSTFIELDS, $param);
    
            }
            
            function process(){ //exec the curl
                
                return $this->page = curl_exec($this->ch); //This variable is result from POST METHOD, and Ofcourse I just need to set a Meta charset html tag to display to browser. No need converting.        
                curl_close($this->ch); //close curl
            }
        
        }
        
        
            $test = new test('http://link.com',$_REQUEST['input']); //Start 'curling'
            $test->process();

 

There's no need to StringReplace or regexp anything.
so how to fix the problem?

 

 

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

Still nothing AutoIt reproducible here.

BTW, you can download the new beta AutoIt3Wrapper just produced for fixing this issue with displaying UTF8.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

Still nothing AutoIt reproducible here.

BTW, you can download the new beta AutoIt3Wrapper just produced for fixing this issue with displaying UTF8.

I can't find new version while the latest version is still in 20-9-2015 
Do you have any solutions for my problem about receiving respond from POST request?

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

does your browser show the correct output? I'm sure is server side issue.

Also you can use utf8_encode/utf8_decode in server side.

Regards

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

does your browser show the correct output? I'm sure is server side issue.

Also you can use utf8_encode/utf8_decode in server side.

Regards

I am sorry that I can't touch to  the server side.
There is no way else?
Update: thank you for your answer, 
it's probably the cause of the problem.
If I post directly to an online website, the result, as far as jchd said, is already in Utf8, no need any converting funtion, it shows well.
But If I download the page (saving from browser) and put to localhost, the problem starts happening.

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

Can you post the output verbatim, or a reproducer script? There may be a simple way to restore the content.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I can't find new version while the latest version is still in 20-9-2015 Do you have any solutions for my problem about receiving respond from POST request?

http://www.autoitscript.com/autoit3/scite/download/beta_SciTE4AutoIt3/

 

1 person likes this

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

#20 ·  Posted (edited)

Yes, Here I have 2 files, source and result,
I send POST request to source file (on my localhost) and receive respond as result file.
There is only one paragraph in each file.

result.html

source.html

Edited by phamhoangthi

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now