UTF8 conversion problem

[FreeBeing]

Hi,

I have a problem when I read my external IP from a simple external PHP script (with InetRead), result is provided as UTF8 from my web server.

When I convert it into decimal, few characters are added at the beginning, meaning it's not the right charset.

Into Hex, there is that header (UTF8) : 0xEFBBBF

So, I use BinaryToString($external_ip, 4) to convert from UTF8 into decimal.

With that, I am near of my goal : I obtain a full IP address, but it stays just a little "?" char before.

In Hex, I see that strange header : 0x3F

In theory, UTF8 header must disappear with the first BinaryToString conversion, no ?

For now, I can't use my script because conversion seems to be partial, or maybe there is another solution ?

Full example :

Binary resultat from the PHP script :

0xEFBBBF38372E3233312E32372E323335

If converted with BinaryToString($external_ip, 4) and converted again into Binary :

0x3F38372E3233312E32372E323335

Decimal obtained : ?87.231.27.235

Expected result :

0x38372E3233312E32372E323335

Decimal expected : 87.231.27.235

 

Thank you

[MikahS]

Why not just use StringReplace to replace the '?' with '' so the string will read how you'd like?

[BrewManNH]

Is it possible that the ox3F is the BOM of the file you're reading from?

[Zedna]

I noticed exactly the same problem in one of my project.

I use brutal workaround:

If Asc(StringLeft($orig,1)) = 63 Then $orig = StringMid($orig,2) ; skip ? (3F BOM)

[Gianni]

>a post with a similar issue was on this forum few days ago,

don't know if the same issue has the same cause here, anyway you can try.
in short it seems the problem was due to a not well recognized "user agent string" that cause a wrong encoded string be returned to the InetRead() function from the server.
You could try to send this command just before the InetRead()

HttpSetUserAgent('Mozilla/5.0')

look >here for a better explanation

edit:

Changed InetGet with InetRead in text

Edited October 23, 2014 by Chimp

[FreeBeing]

Ok, so if I understand, I just have to skip or delete the 3F BOM, without any bug ?

Edit : I tried to change User Agent to Firefox 33, but it don't change anything.

UA : Mozilla/5.0 (Windows NT 6.2; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0

Edited October 23, 2014 by FreeBeing

[Zedna]

Ok, so if I understand, I just have to skip or delete the 3F BOM, without any bug ?

 

Yes.

[FreeBeing]

Perfect.

And for my own knowledge, it's an AutoIt bug, or a missing feature ?

Thank you

[jchd]

None of the above. These 3 bytes are the UTF8 "BOM", an initial sequence of bytes which represent a specific (invalid per se) Unicode character and the endianness. It indicates which encoding is being used. BOMs exist for the other UTF encodings.

A BOM is often used in text files albeit not mandatory. AutoIt has mecanism and options to handle a BOM when opening a file see help for FileOpen).

On the contrary, data passed "over the air" is most often using an implicit or fixed Unicode encoding, hence doesn't include a BOM. That's why you just have to skip, remove and ignore an initial BOM.

[FreeBeing]

Thank you for your explainations, very accurate.

[Gianni]

... hey, jchd is right! .....

$hTemp = FileOpen(".\Temp.txt", 18) ; overwrite & binary mode
FileWrite($hTemp, "0xEFBBBF38372E3233312E32372E323335")
FileClose($hTemp)
$hTemp = FileOpen(".\Temp.txt") ; read mode
MsgBox(0, "", FileRead($hTemp))
FileClose($hTemp)
FileDelete(".\Temp.txt")

edit:

maybe you could use InetGet() instead of InetRead() then

Edited October 24, 2014 by Chimp