String encodings problem

[jchd]

Hi,

After some discussion on he help forum, I still have very hard time without various string encoding issues.

I'll talk about string litterals here and keep other questions about strings elsewhere.

To demonstrate the problem, here's the code I'd like you to save and run.

Save it under various encodings (but make sure the "magic" characters are preserved to their original codepoint) and run it.

Also don't forget that SCite needs you modify the cource (e.g. add space, remove it) to enforce encoding change.

CODE

;;€ IMPORTANT: the first three characters must be semicolomn, semicolomn, Euro

Global $s = "éà€" ; must be &esharp; à €

; html (Windows compose numeric keypad) ANSI UTF-8

; &esharp; Alt 0 2 3 3 0xE9 0xC3 0xA9

; à Alt 0 2 2 4 0xE0 0xC3 0xA0

; € Alt 0 1 2 8 0x80 0xE2 0x82 0xAC

Global $b = Binary($s)

Global $hdl, $bom, $encoding, $shouldbe

$hdl = FileOpen(@ScriptFullPath, 16) ; read this script file in binary mode

$bom = FileRead($hdl, 3) ; read the first 3 bytes

FileClose($hdl)

ConsoleWrite("The string is " & $s & @LF)

MsgBox(0, "String display", $s)

Switch Binary($bom)

Case "0x3B3B80"

$encoding = "ANSI"

$shouldbe = "0xE9E080"

Case "0x3B3BE2"

$encoding = "UTF-8"

$shouldbe = "0xC3A9C3A0E282AC" ; AutoIt isn't aware of UTF-8 without BOM and reads ANSI instead

Case "0xEFBBBF"

$encoding = "UTF-8 with BOM"

$shouldbe = "0xC3A9C3A0E282AC"

Case "0xFEFF00"

$encoding = "UTF-16be (with BOM)"

$shouldbe = "0x00E900E020AC" ; this supposes AutoIt leaves literals unchanged

Case "0xFFFE3B"

$encoding = "UTF-16le (with BOM)"

$shouldbe = "0xE900E000AC20" ; this supposes AutoIt leaves literals unchanged

Case Else

$encoding = "Unknown"

$shouldbe = "I have no idea!"

EndSwitch

MsgBox(0, "Encoding dependancy", _

"Script is encoded as " & $encoding & @LF & _

"The reference string is " & $s & @LF & _

"This string contents is " & $b & @LF & _

"It should be equal to " & $shouldbe & @LF & _

"which is " & ($b = $shouldbe))

; Results: (the codebox was destroying formatting)

;

; File encoding........ConsoleWrite.....MsgBox........Binary

; ANSI Latin1..........incorrect..........correct.......correct

; UTF-8 without BOM......correct........incorrect.......correct

; UTF-8 with BOM.......incorrect..........correct.....incorrect

; UTF-16le with BOM....incorrect..........correct.....incorrect

; UTF-16be with BOM....incorrect..........correct.....incorrect

Edited April 22, 2009 by jchd

[Richard Robertson]

I'm not surprised that the console has trouble displaying unicode characters. I doubt it's an easily solved problem though.

[BrettF]

What about this?

ConsoleWrite ("Writing Chr() representation of 0xE0, 0xE9, 0x80" & @CRLF & Chr(0xE0) & @CRLF & Chr(0xE9) & @CRLF& Chr(0x80) & @CRLF)
MsgBox (0, "Test", Chr(0xE0) & @CRLF & Chr(0xE9) & @CRLF& Chr(0x80))

[jchd]

I'm not surprised that the console has trouble displaying unicode characters. I doubt it's an easily solved problem though.

OK it's understandable (the docs even warns against that somewhere) that the console would have problems displaying Unicode chars.

But it simply is not the truth!

It doesn't display ANSI correctly and it only displays Unicode when AutoIt beleaves it is ANSI.

Another more serious problem I see is that the various UTF-* litterals are stored as ANSI (that's what Binary() shows) even when the BOM warns AutoIt that the script contents is Unicode.

There's also a mystery: it's said that AutoIt parses the script as ANSI if it doesn't find a BOM, which I understand well. Indeed file encoding guessing is slow and not 100% fail-safe. But then, with a true ANSI file, the console should display ANSI chars, no? It doesn't!

With a UTF-8 without BOM (which AutoIt is supposed to read as ANSI, but in fact contains UTF-8 sequences), the console displays Unicode chars correctly !

It really looks like there's a serious problem with string encodings in AutoIt. This calls for clarification and deeper tests.

[jchd]

What about this?

ConsoleWrite ("Writing Chr() representation of 0xE0, 0xE9, 0x80" & @CRLF & Chr(0xE0) & @CRLF & Chr(0xE9) & @CRLF& Chr(0x80) & @CRLF)
MsgBox (0, "Test", Chr(0xE0) & @CRLF & Chr(0xE9) & @CRLF& Chr(0x80))

This would tend to demonstrate that both ConsoleWrite and MsgBox work correctly with ANSI strings only when some internal AutoIt function like Chr() forces a inner flag on string data being (correctly) ANSI.

But the issues remain.

[Valik]

You are making the assumption that the file encoding matters. It does not. The file encoding is irrelevant. AutoIt is a UNICODE application so strings are stored internally as UNICODE regardless of the file's encoding. I do not experience any of the problems you demonstrate. ConsoleWrite() works correctly for me no matter what encoding I use *if I ensure the data is not corrupted*. Same for MsgBox(). The binary representation doesn't match your expectations but your expectations are wrong.

I don't see any problems here.

Edit: Three addendum:

• Changing the encoding for me would sometimes corrupt the data. I could visibly see the characters change. Thus I knew they were corrupted so I would re-paste the contents and re-save the file to ensure no corruption.
• ConsoleWrite() writes ANSI data. All strings are converted to ANSI before being written. As long as the data isn't corrupt and has an ANSI representation it will dispaly.
• Binary() returns ANSI data. I just checked, it's hard-code to do that. This could probably be made more clear in the documentation.

To sum up once again, there are no problems here. I suspect some of what you are seeing is corruption due to changing the encoding. The other problems you see are due to your incorrect expectations as to how AutoIt and Binary() work. Edited April 22, 2009 by Valik

[jchd]

You are making the assumption that the file encoding matters. It does not. The file encoding is irrelevant. AutoIt is a UNICODE application so strings are stored internally as UNICODE regardless of the file's encoding. I do not experience any of the problems you demonstrate. ConsoleWrite() works correctly for me no matter what encoding I use *if I ensure the data is not corrupted*. Same for MsgBox(). The binary representation doesn't match your expectations but your expectations are wrong.

I don't see any problems here.

Edit: Three addendum:

• Changing the encoding for me would sometimes corrupt the data. I could visibly see the characters change. Thus I knew they were corrupted so I would re-paste the contents and re-save the file to ensure no corruption.
• ConsoleWrite() writes ANSI data. All strings are converted to ANSI before being written. As long as the data isn't corrupt and has an ANSI representation it will dispaly.
• Binary() returns ANSI data. I just checked, it's hard-code to do that. This could probably be made more clear in the documentation.

To sum up once again, there are no problems here. I suspect some of what you are seeing is corruption due to changing the encoding. The other problems you see are due to your incorrect expectations as to how AutoIt and Binary() work.

Thank you for your answer Valik.

But you say AutoIt is a Unicode program. There are no such things as "Unicode" programs. To the best it can handle some UTF (aka UCS) encodings and a subset of current Unicode set (possibly the whole set) all defined by the Unicode Consortium. Today, Unicode encompasses more than 240 000 codepoints. Since AutoIt offers string compare and casing functions as well as encoding conversion routines, it _must_ use the full ICU library to do so and claim compatibility to Unicode. I hardly believe it is so. ICU is about 3Mb and is slow as hell (lots to do): this would show!

You're sure ConsoleWrite displays only ANSI data? Let's check with the following setup: XP SP3 with Latin1 languages, vanilla AutoIt3Scite install, Scite fonts set to a good Unicode font (DejaVu Sans Mono or equivalent) encoding corrctly set to UTF-8 without BOM:

Global $s = "ЏАБВГДЕЖЗИЙКЛ"
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $s = ' & $s & @crlf & '>Error code: ' & @error & @crlf);### Debug Console

Output is:

@@ Debug(2) : $s = ЏАБВГДЕЖЗИЙКЛ

>Error code: 0

So sorry Valik you're wrong on this. ConsoleWrite doesn't convert to ANSI before display and it can display Unicode.

You say Binary silently converts to ANSI? That's very surprising for a function that is supposed to dump data in hex!! Lets check again:

Global $s = "€"
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : Binary(' & $s & ') = ' & Binary($s) & @crlf & '>Error code: ' & @error & @crlf);### Debug Console

Output is:

@@ Debug(2) : Binary(€) = 0xE282AC

>Error code: 0

Please note that 0xE2 0x82 0xAC is precisely the UTF-8 sequence for the Euro symbol.

Ouce again, saying that Binary translate data to ANSI before hexing it is simply not true. Using the current Latin1 codepage (CP_ACP) it would have translated this into ANSI Euro and its output would have need 0x80.

Sorry but the facts do contradict what you're saying.

Edited April 22, 2009 by jchd

[Valik]

But you say AutoIt is a Unicode program. There are no such things as "Unicode" programs.

Sure, whatever.

You're sure ConsoleWrite displays only ANSI data?

I'm positive - I wrote the code.

So sorry Valik you're wrong on this. ConsoleWrite doesn't convert to ANSI before display and it can display Unicode.

Oh really? That's not what the code I'm looking at says. The code I wrote. But what do I know. Maybe I just don't know how to read my own code?

You say Binary silently converts to ANSI? That's very surprising for a function that is supposed to dump data in hex!! Lets check again:

Ouce again, saying that Binary translate data to ANSI before hexing it is simply not true. Using the current Latin1 codepage (CP_ACP) it would have translated this into ANSI Euro and its output would have need 0x80.

You're right, it's not like I have the code sitting right here to look at or anything.

Sorry but the facts do contradict what you're saying.

The facts are I have the source code and you do not. I wrote one of the functions you are talking about. It's absolutely laughable to think that you would sit there and argue what a function does or does not do when I have the source code to the functions and can see what they are doing. Let's summarize why you are not worth listening to:

• Your first post here on this subject was full of unsubstantiated assumptions.
• You started off the quoted post spouting off some non-sense about UNICODE applications.
• You not only argue with a developer, you flat out state they are wrong when they have the source code and personally wrote some of the code in question.
• You argue a developer not about behavior but about implementation details. Telling me that I'm wrong rather than stating that maybe the function is wrong is just about trolling given that, as mentioned several times now, I have the source code.
• Your little example script that prints out the ASCII value of the Euro symbol prints 80 for me. Let me tell you what you've done to corrupt the data (despite my previous warning that you are doing this):
• You had SciTE configured to use the UTF-8 with BOM encoding.
• You wrote the example script.
• You changed SciTE's encoding to Code Page Property.
• When the encoding changed SciTE replaced the Euro symbol with the string "â¬".
• When you run AutoIt it sees a string containing three characters, not a string containing a single Euro symbol.
• You are given the ASCII values of those three characters but are not paying attention you have corrupted the data.

Edit: Added #5 to explain why the OP's attempt to disprove how Binary() works is just flat out wrong. Edited April 22, 2009 by Valik

[jchd]

Look Valik,

I know you have the source code in view but this doesn't allow you to dismiss everyone.

Sorry but as several users have already noticed in the help forum, the behavior I tell you about actually shows, not only here, without any encoding-change-mistake. Jos advised me to post here, which I did with simple code showing what the problems are.

You can consider users are not worth listening, but you'd rather rely on facts than on how-dumb-users-are assomptions. Your inflated ego may utterly suffer from looking at the reality but it won't change the reality.

Please substantiate what non-sense I've written about Unicode. You really don't seem to know what you're talking about when you talk UNICODE. Even if strings are strored as what Microsoft calls Unicode with "wide char" type, then it's not obviously Unicode compliant. To be so, it has to follow, say, UTF-16 norm, which allows for double-word characters and even much longer sequences as per latest version. This just isn't nonsense.

When AutoIt reads a litteral and converts it to Microsoft Unicode, it would typically invoke MultiByteToWideChar. But to do so, it has to give which encoding the input string uses, the codepage parameter. Then it matters, contrary to what you pretend, and things are not the same if you pass CP_ACP or CP_UTF8 as codepage. Of course you can keep on saying that I don't have the code in front of me, but it nonetheless have to invoke such function and the codepage isn't irrelevant.

Can we agree on that? Then which codepage do you convert from?

Also you say several user functions have their data converted to ANSI before display, but then how can you explain it doesn't show in practice?

You keep answering "It can be so just because my code says otherwise", but simple AutoIt code shows it doesn't _behave_ like you say. It would certainly be a more positive attitude to either dig further if needed or give a clear clarification about the issue.

To be Unicode compliant, an application must treat for instance casing in a compliant way. This brings in a lot of code and large tables to do so properly. Ignoring this by handwaving is just plain wrong, sorry for that.

Global $s = "françois"
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : StringUpper(' & $s & ') = ' & StringUpper($s) & @crlf & '>Error code: ' & @error & @crlf);### Debug Console

Output:

@@ Debug(2) : StringUpper(françois) = FRANçOIS

>Error code: 0

I should see:

@@ Debug(2) : StringUpper(françois) = FRANÇOIS

You see the difference?

The same issue would show when comparing strings containing non-ASCII letters. Unicode ordering isn't a piece of cake!

Now, I believe that every user of AutoIt (I for one) is willing to accept compromises, just because a fully compliant version of this program would be very large and very slow. Now it would be a real bonus for users worldwide if we knew precisely which compromise were made.

Please Valik, I sincerely trust you are a good programmer, there is no question. But you really should be a little less aggressive and more factual to avoid wasting time and energy of both yourself and your users. I never use wording meant to imply you were a moron, nor that your code (nor any other's code in AutoIt) is bad code. I've done my best to be polite and respectful as much as I can, english not being my mothertongue. I've tried to be as factual as possible, but I've recieved from you only misconsideration and handwaving arguments.

I have no problem at all when someone proves me wrong. I've written a lot of code also mind you, and sometimes made silly errors. I've even found mistakes in some of my own formal models. I always examine facts before going ballistic about other's complains/questions/remarks. This way I learn and move forward.

I would be extremely grateful if you would agree to start replying to my initial code snippet and actually look at the 12 cases correct/incorrect and actually discuss things from there with an ounce of serenity.

Edited April 22, 2009 by jchd

[Valik]

I know you have the source code in view but this doesn't allow you to dismiss everyone.

It does when you try to tell me the code does something I can clearly look at the code and see is not true.

You can consider users are not worth listening, but you'd rather rely on facts than on how-dumb-users-are assomptions. Your inflated ego may utterly suffer from looking at the reality but it won't change the reality.

I'm not the one so egotistical he thinks he can correct a developer. You flat out stated I was wrong about certain things when I was looking at the code. That makes whatever perceived ego I have look like nothing.

Please substantiate what non-sense I've written about Unicode.

You're taking a very hyper-literal interpretation. AutoIt is (or at least tries to be) as UNICODE compliant as any other Windows program. Are you happy now? Did I word that in such a way that you'll get off my back and spare me the long diatribes about shit I don't care about?

Also you say several user functions have their data converted to ANSI before display, but then how can you explain it doesn't show in practice?

Because you are incompetent and continue to corrupt the data thus causing unexpected results to be displayed.

You keep answering "It can be so just because my code says otherwise", but simple AutoIt code shows it doesn't _behave_ like you say.

Stop corrupting the characters, then it will work. I can't do anything about your inability to perform tests correctly.

It would certainly be a more positive attitude to either dig further if needed or give a clear clarification about the issue.

I don't know how to make it any more clear than to say, YOU are the problem because YOU keep corrupting the data in your tests.

But you really should be a little less aggressive and more factual to avoid wasting time and energy of both yourself and your users.

How do you suggest I get more factual than telling you what the code is doing? I'm not speculating, I'm not guessing, I'm not doing corrupted tests. I'm looking at the source code and saying "This is what happens". If that is not fact enough for you then it is beyond any being's ability to answer you sufficiently.

I would be extremely grateful if you would agree to start replying to my initial code snippet and actually look at the 12 cases correct/incorrect and actually discuss things from there with an ounce of senerity.

What exactly do you want me to tell you beyond what I already have? Either your expectations are wrong (in the case what Binary() returns) or you have caused SciTE to corrupt your data by changing the encoding.

I'll try and make this perfectly clear: AutoIt is working as expected in every scenario you have presented.

[jchd]

Ok, I won't reply to your endless useless ad-hominem diatribes.

You see, I've found that AutoIt is a globally very useful piece of software. That mostly why I selected it. It looked widespread enough and well maintained enough to be a stable platform in my eyes.

The fact that such support for brilliant software is in the hand on people like you that just don't want to look at what actually happens in _reality_ and keep their eyes fixed at the lines of brilliant code they have written cast more than a serious doubt on the pertinence of selecting AutoIt for any professional use. This is a bad service your offering to a software that I believe means something to you.

I've writen many million lines of code myself in too many languages for me to remember them all. I've switched at some time to formal method for _proving_ highly complex specifications correct (ATMs, cars, trains, planes, rockets, etc.) and in some case deriving _proven_ correct code.

I've no intention at my age and given my experience to be insulted like you do or otherwise be run over by youngsters on mopets believing their are Kings or the World just because one has given them access to a medium-size application code.

You don't know what your talking about when it comes to Unicode, are not willing to listen to anyone and learn anything outside MSDN and are "blind enough" to avoid running a handful lines of code that proves you wrong. Talk about being hypocritical!

I've developped a non-negligible "glue" application relying on AutoIt, but I today realize it's impossible to rely on this tool if your needs don't strictly match what AutoIt secretly does. Which you don't want to disclose. For you, the user (as pejorative at it may be in your view) is always the culprit.

I've now gone too far to switch over and start again with some other tool. What I'll do is have my code spend 70% of the CPU spending its time converting (correctly) back and forth between ANSI and UTF with my own routines for 7200 fields in one database and much more the other side. This is simply ridiculous, just because people like you refuse to _look_ and stick to what they _suppose_ their code do (without ever trying).

Really, you are giving bad service to people who founded AutoIt, which I respect wholehearthy.

Plonk.

[Valik]

jchd, you want to talk about hypocritical. You spout nonsense about me not listening but it is you who refuses to listen! I have run your code. It does not behave how you describe unless I intentionally make changes in SciTE to corrupt the data! Why aren't you getting that? Every single example you have provided works for me exactly how I expect it to work. Every single example you have provided works how you have describe only when I corrupt the data. This is not Valik being an arrogant prick. This is Valik running your code and telling you that it does not behave how you claim. This is Valik telling you how AutoIt works - only to be contested by you with statements like "you are flat out wrong". I'm not giving you the answers you want to hear so you're using my past to try to paint me out as some sort of bad-guy. The fact is, you're flat-out dead wrong about your assertions on AutoIt's behavior. The moment you stop doing exactly what you are accusing me of and start listening to what I'm saying you'll realize just how foolish you look.

This thread is closed. I'm not going to continue to argue with some self-proclaimed expert who wishes to prattle on and on about doing it right yet can't be bothered to do a simple test right without corrupting data. I have explained how the data is getting corrupted and not a single shred of evidence has been offered that indicates it is not exactly as I have described. Instead all I get is a bunch of know-it-all bullshit and assertions on the behavior of AutoIt.

No thanks.