Jump to content
Sign in to follow this  
dpryan

Problem Compiling with Unicode Support

Recommended Posts

dpryan

I'm not sure if this is a AutoIt problem or a AutoIt3Wrapper problem, but I'm having trouble getting a unicode file to compile properly.

I use the "Omega" symbol in my code. In ANSI, this looks like "Ω" (0xCEA9) and in Unicode it looks like "Ω".

If I run the script from SciTE (F5/"Go") everything works properly.

However, when I compile (Ctrl+F7) something converts the "Ω" or "Ω" to the letter "O". This affects both the source au3 file as well as the output exe. Also, if the source file is UTF8-BOM, then the file gets converted back to Code Page Property when you compile.

Although it affects the file that I'm compiling, it does not affect the #include files.

So as a workaround, I created a wrapper for my main au3 file that just has #AutoIt3Wrapper stuff and a #include of my main file. This fixed the problem. But it would be nice if I didn't have to do that.

File 1: unicodeproblem.au3

MsgBox(0,"Unicode Test","Ω")

File 2: unicodewrapper.au3

#include "unicodeproblem.au3"

Try running (F5) and compiling (Ctrl+F7) these two files and see if you run into the same problem.

Edited by dpryan

Share this post


Link to post
Share on other sites
jchd

Also, if the source file is UTF8-BOM, then the file gets converted back to Code Page Property when you compile.

Definitely not.

Your source is _not_ UTF8+BOM is that case. You need to change the encoding and _then_ to apply any change then save it.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
dpryan

Definitely not.

Your source is _not_ UTF8+BOM is that case. You need to change the encoding and _then_ to apply any change then save it.

Have you tried it?

At first I thought you got me on that one, because it turns out that changing the Encoding in SciTe doesn't qualify as a change to the file (or put an "asterisk" in the title bar), so then saving the "changed" file didn't actually save it.

But then I actually saved it, closed the file, reopened it (and it showed UTF+BOM). Then when I compiled the file to an exe it reverted.

Share this post


Link to post
Share on other sites
jchd

If I never had tried it, I'd never be that positive!

You must be using some compiler option. There is no reason the wrapper would rewrite the source by itself.

Download the files below to see by yourself that using F7 (Build) does work as expected.

>"C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /prod /in "D:\XLequit\AutoMAT\Test\UnicodeTest.au3" /autoit3dir "C:\Program Files\AutoIt3"

+>09:45:31 Starting AutoIt3Wrapper v.2.0.1.39 Environment(Language:040C Keyboard:0000040C OS:WIN_XP/Service Pack 3 CPU:X64 OS:X86)

! ***************************************************************************************************

! * Input file is UTF8 encoded, Au3Check/Tidy/Obfuscator do no support UNICODE and will be skipped.*

! ***************************************************************************************************

>Running:(3.3.6.1):C:\Program Files\AutoIt3\aut2exe\aut2exe.exe /in "D:\XLequit\AutoMAT\Test\UnicodeTest.au3" /out "D:\XLequit\AutoMAT\Test\UnicodeTest.exe" /nopack /comp 2

+>09:45:32 Aut2exe.exe ended.rc:0

>Running:(3.3.0.0):C:\Program Files\AutoIt3\aut2exe\upx.exe" --best --compress-icons=0 -qq "D:\XLequit\AutoMAT\Test\UnicodeTest.exe"

646969 -> 301369 46.58% win32/pe UnicodeTest.exe

+>09:45:32 UPX Ended: rc:0

+>09:45:32 Created program:D:\XLequit\AutoMAT\Test\UnicodeTest.exe

->09:45:32 Warning: This is an Unicode compiled script and will not run on Win9x/ME.

>Exit code: 0 Time: 2.309


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
dpryan

I just tried it. Run (F5) and Build (F7) both work properly, but Compile (Ctrl-F7) with default settings turns the Encoding into Code Page Property.

As a result, the source code looks like this:

Local $str[10] = [ _
  "Sant Julià de Lòria", _
  "Skrýchov u Oparan", _
  "Žíšov", _
  "??????? ???????", _
  "??????-???", _
  "??????", _
  "????", _
  " ?????? ?????", _
  "??? ???????? ?? ???? ?? ???????? ?? ????? ??? ??????? ??????????? ?? ?????? ??????? ??? ?????? ?????? ?? ?????????? ?? ??? ?? ?? ?????? ?????? ??????? ?? ??? ?? ?????? ???? ???????", _
  "????????????????????????" _
]

For $s In $str
    MsgBox(0, "Unicode test", $s)
Next

The compiled exe shows the question marks as well.

Share this post


Link to post
Share on other sites
jchd

It doesn't do that on my plain vanilla standard install with plain vanilla standard compile settings.

Can you tell us which compile option you use? It would greatly help resolution.

Do you use Increment file version or similar option?

Cut & paste the compile report as I did.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
dpryan

I'm using vanilla install with vanilla settings as well. I guess I am also using the AutoIt Script Editor (customized SciTe editor).

For the example file that you posted, I didn't use any compiler options. No #AutoIt3Wrapper directives get added when I compile.

Here is my log file...

>"C:\Program Files\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /ShowGui /in "C:\Users\x\Desktop\UnicodeTest.au3"

+>12:58:01 Starting AutoIt3Wrapper v.2.0.1.24 Environment(Language:0409 Keyboard:00000409 OS:WIN_VISTA/Service Pack 2 CPU:X64 OS:X86)

! ***************************************************************************************************

! * Input file is UTF8 encoded with BOM, Au3Check does not support UNICODE and will be skipped. *

! ***************************************************************************************************

-> 1 Change(s) made.

>Running:(3.3.6.1):C:\Program Files\AutoIt3\aut2exe\aut2exe.exe /in "C:\Users\x\Desktop\UnicodeTest.au3" /out "C:\Users\x\Desktop\UnicodeTest.exe" /nopack /comp 2

+>12:58:04 Aut2exe.exe ended.rc:0

>Running:(3.3.0.0):C:\Program Files\AutoIt3\aut2exe\upx.exe" --best --compress-icons=0 -qq "C:\Users\x\Desktop\UnicodeTest.exe"

646739 -> 301139 46.56% win32/pe UnicodeTest.exe

+>12:58:05 UPX Ended: rc:0

+>12:58:05 Created program:C:\Users\x\Desktop\UnicodeTest.exe

->12:58:05 Warning: This is an Unicode compiled script and will not run on Win9x/ME.

>Exit code: 0 Time: 5.041

Edited by dpryan

Share this post


Link to post
Share on other sites
jchd

There are two important differences between my compile and yours:

+>09:45:31 Starting AutoIt3Wrapper v.2.0.1.39 Environment(Language:040C Keyboard:0000040C OS:WIN_XP/Service Pack 3 CPU:X64 OS:X86)

! ***************************************************************************************************

! * Input file is UTF8 encoded, Au3Check/Tidy/Obfuscator do no support UNICODE and will be skipped.*

! ***************************************************************************************************

>Running:(3.3.6.1):C:\Program Files\AutoIt3\aut2exe\aut2exe.exe /in "D:\XLequit\AutoMAT\Test\UnicodeTest.au3" /out "D:\XLequit\AutoMAT\Test\UnicodeTest.exe" /nopack /comp 2

+>12:58:01 Starting AutoIt3Wrapper v.2.0.1.24 Environment(Language:0409 Keyboard:00000409 OS:WIN_VISTA/Service Pack 2 CPU:X64 OS:X86)

! ***************************************************************************************************

! * Input file is UTF8 encoded with BOM, Au3Check does not support UNICODE and will be skipped. *

! ***************************************************************************************************

-> 1 Change(s) made.

>Running:(3.3.6.1):C:\Program Files\AutoIt3\aut2exe\aut2exe.exe /in "C:\Users\x\Desktop\UnicodeTest.au3" /out "C:\Users\x\Desktop\UnicodeTest.exe" /nopack /comp 2

The "1 Change(s) made." line shows that the wrapper has been instructed to perform a change in the source file. This is most probably the "Increment file version" feature being actuve.

I had this very same issue for this reason and we worked out cyhanges in the wrapper to take care of original source file encoding in this case. That's the reason for the difference in wrapper version.

Either check "Don't change" in the "File Version" group and remove other source-updating option(s) or use the latest wrapper build. You can probably use the one bundled in the current beta version.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
dpryan

The increment file version was not enabled. However, updating my SciTe4AutoIt seemed to fix the problem.

I'm now running AutoIt3Wrapper v.2.0.3.0.

Thanks!

Share this post


Link to post
Share on other sites
jchd

I can't figure out of the blue which option was being set that caused the "1 change(s) made" but it's that one which made the wrapper rewrite the source, ignoring the original file encoding.

Jos would easily point out which option cause this rewrite outside the File Version one. I don't have enough Wrapper-Fu myself to guide you there.

Anyway, since upgrading solved the issue I wish you best of luck in your work.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
Jos

I can't figure out of the blue which option was being set that caused the "1 change(s) made" but it's that one which made the wrapper rewrite the source, ignoring the original file encoding.

Jos would easily point out which option cause this rewrite outside the File Version one. I don't have enough Wrapper-Fu myself to guide you there.

Anyway, since upgrading solved the issue I wish you best of luck in your work.

The old version of AutOit3Wrapper indeed had this issue with giving "1 Change(s) made" in certain conditions, plus it did screw up encoding in that case.

The current version should fix all of this.

r/Jos


Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
iCode

@Jos - the "current" version being 2.0.3.0, correct?

when running beta autoit with au3check 1.54.19.0, it's still reverting the script encoding back to ANSI (code page default) and giving "1 change made" if set as UTF8 with BOM (verified with NP++)

this happens even with "code.page=65001" in global ops

if just UTF8 it doesn't do this, though i'm not sure how au3check is running at all since it's not supposed to support unicode,is it?

another thing i'm seeing here is, again, even with "code.page=65001" set, new documents created from within scite are ANSI encoded

Edited by iCode

FUNCTIONS: WinDock (dock window to screen edge) | EditCtrl_ToggleLineWrap (line/word wrap for AU3 edit control) | SendEX (yet another alternative to Send( ) ) | Spell Checker (Hunspell wrapper) | SentenceCase (capitalize first letter of sentences)

CODE SNIPPITS: Dynamic tab width (set tab control width according to window width)

Share this post


Link to post
Share on other sites
Jos

PM me a file that I can test with and the exact steps you go through.

Assume you do Ctrl+F7 in SciTE showing the Options menu first?

Also am not sure what the relation is to au3check..

Jos


Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
iCode

i can do that, but i'm not sure it will be helpful - i'd be sending a blank .au3 script - i think you may just need the steps to reproduce (but i'll gladly send a test file if you want)

i'm using a fresh install of autoit beta, wrapper 2.0.3.0, scite 2.27 with a default config (i set the code.page back to default)

  • create a new, empty .au3 (ctrl+n)
  • change encoding to UTF8 - no BOM
  • add a char or space so you can save
  • ctrl+F7 > uncheck option to run au3check > click "save only"
now check encoding again... should change from UTF8 to default - i can verify this with NP++ which will detect ANSI

i'm not sure it's au3check that's changing the encoding - i can run au3check on the [empty] script and it doesn't change the encoding

it seems that selecting/deselecting the option to run au3check triggers somehting

here's the console output after unchecking option to run au3check and saving...

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /ShowGui /in "D:\TEMP\USER\eee.au3"
+>11:10:57 Starting AutoIt3Wrapper v.2.0.3.0    Environment(Language:0409  Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64)
-> 1 Change(s) made.
>Exit code: 0    Time: 5.145

note also that when i installed autoit beta, i chose all defaults (x86 tools, not x64)

Edited by iCode

FUNCTIONS: WinDock (dock window to screen edge) | EditCtrl_ToggleLineWrap (line/word wrap for AU3 edit control) | SendEX (yet another alternative to Send( ) ) | Spell Checker (Hunspell wrapper) | SentenceCase (capitalize first letter of sentences)

CODE SNIPPITS: Dynamic tab width (set tab control width according to window width)

Share this post


Link to post
Share on other sites
Jos

Ok, You said before UTF8 With BOM.

There is no easy way to detect a UTF8 without BOM and I don't know of a simple way to build support for that.

Jos


Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
iCode

ah, yeah i did and now i'm not completely sure if i was actually setting it with BOM - i can't reproduce it now (with BOM)


FUNCTIONS: WinDock (dock window to screen edge) | EditCtrl_ToggleLineWrap (line/word wrap for AU3 edit control) | SendEX (yet another alternative to Send( ) ) | Spell Checker (Hunspell wrapper) | SentenceCase (capitalize first letter of sentences)

CODE SNIPPITS: Dynamic tab width (set tab control width according to window width)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.