Jump to content

Forcing UTF8+BOM encoding for every file in SciTE?


jchd
 Share

Recommended Posts

I can't remember how to do that and where it should be.

Is it at all possible to do so that it survives installation of new versions of SciTE4AutoIt3?

TIA

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Use these properties in your SciTEUser.properties file, hopefully.

code.page
output.code.page

To support a DBCS language such as Japanese, a code page can be set here. This ensures that double byte characters are always treated as a unit so the caret is never located between the two bytes of a double byte character.
 

Code page Value

Default (single byte character set) 0

UTF-8 65001

Japanese Shift-JIS 932

Simplified Chinese GBK 936

Korean Wansung 949

Traditional Chinese Big5 950

Korean Johab 1361

Setting code.page to 65001 starts Unicode mode and the document is treated as a sequence of characters expressed as UTF-8. Display is performed by converting to the platform's normal Unicode encoding first so characters from any language will be displayed. Correct glyphs may only be displayed if fonts are chosen that contain the appropriate glyphs. The Tahoma font contains a wide range of glyphs so may be a good choice. 
This property can not set a single byte character set.
If output.code.page is set then it is used for the output pane which otherwise matches the edit pane.

http://www.scintilla.org/SciTEDoc.html

Edited by wraithdu
Link to comment
Share on other sites

  • Developers

Quick check told me there is no setting for it but could be done with a lua script.

Woud be happy to add that into the standard installer and have a properties keyword set this to the correct value.

You could then add that into the sciteuser.properties to survive any upgrades.

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Sorry for being lazy: I meant "every new file created by SciTE", not every file in the wild...

code.page=65001

did work up to some point to force the console into using UTF8, but it doesn't do anything in the last few releases of SciTE4AutoIt3 (W7x64 if that matters). That contradicts the SciTE doc.

I didn't know about output.code.page=65001 but it's NOP as well (for forcing the console into UTF8). Until recently as exposed above, I had the console display Unicode without this setting, just the previous one.

I know for sure that Dos "CHCP 65001" works, but not more beyond in terms of Windows consoles.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

^^ SciTE has one particular property called unicodeMode. It's enum value defined like this:

enum UniMode {
    uni8Bit = 0, uni16BE = 1, uni16LE = 2, uniUTF8 = 3,
    uniCookie = 4
};

When unicodeMode is set to uniUTF8 the BOM you want is generated.
Considering SciTE doesn't have user-settable property named unicode.mode the only way to force BOM genaration is by using menu option (or executing some script).

Edited by trancexx

♡♡♡

.

eMyvnE

Link to comment
Share on other sites

I see, well I believe I see. This is in contrast with NotePad++ which has such a user setting.

I'm completely Lua- and SciTE-illiterate, so forgive me if I'm thick asking whether a silent script (or some other way) is possible at launch time?

I don't ask for my own needs (in fact I currently don't need Scite, nor AutoIt at all). I see or get PMed scripts in various languages encoded in their own ANSI codepage. Those have litterals and comments destroyed in my system and it's a pain to recode them after fixing. If everyone used UTF8 scripts, exchanges would be painless and unambiguous regardless of the region they live in.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Adding something like that to SciTE directly is trivial. However, the SciTE-for-AutoIt policy so far was that any customizations on SciTE core level should be kept at minimum because of maintainence issues with upgrades.

What's current policy is beyond me, ask developers.

♡♡♡

.

eMyvnE

Link to comment
Share on other sites

  • Developers

Sorry for being lazy: I meant "every new file created by SciTE", not every file in the wild...

 

That is also what I meant. :)

I will add the LUA script I found on this topic and make that dependent on a keyword in the properties files.

More to follow.

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

I didn't plan to push the idea as a firm request but if you find it can be done someday and it's useful for users at large, then it's fine.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

@Jos,

BTW neither code.page=65001 or output.code.page=65001 actually work to display Unicode correctly in the SciTE console. I don't test systematically every version/beta so I can't say when it stopped working but it's something relatively recent, I'd say with the current beta series.

Is this something that can be a consequence of recent changes in SciTE4AutoIt3 or should I look somewhere else? Apart from regular Win7 updates, I don't change this system much.

What I find strange is that I can paste Unicode e.g. 한국어 텍스트의 예 טקסט עברית ירושלים to the output console and it shows fine. ConsoleWrite is certainly not the culprit since is doesn't work either when launched with F5 (3.3.8.1), albeit it used to work fine with previous Scite4AutoIt3.

@Others,

Do you have any success with these settings?

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

I generally use the function BinaryToString() if I want to use non-ascii characters in my script. This just seems a safer alternative than inputting raw characters. It does require more messing about to do this, but I doubt I'll change the habit. Even so it would be nice to be able to use the real characters in comments.

Edited by czardas
Link to comment
Share on other sites

  • Developers

@Jos,

 

BTW neither code.page=65001 or output.code.page=65001 actually work to display Unicode correctly in the SciTE console. I don't test systematically every version/beta so I can't say when it stopped working but it's something relatively recent, I'd say with the current beta series.

 

Is this something that can be a consequence of recent changes in SciTE4AutoIt3 or should I look somewhere else? Apart from regular Win7 updates, I don't change this system much.

 

What I find strange is that I can paste Unicode e.g. 한국어 텍스트의 예 טקסט עברית ירושלים to the output console and it shows fine. ConsoleWrite is certainly not the culprit since is doesn't work either when launched with F5 (3.3.8.1), albeit it used to work fine with previous Scite4AutoIt3.

I am not going to be able to check this in the coming days, but could you PM me a simple replicator script in a zipped format so I can test next week when i am back home?

Thanks

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

Jos,

Yes of course, enjoy some good time (at least I hope it is), this isn't going to kill anyone in the meantime!

czardas,

BinaryToString? Really? That seems utterly impractical to me :  or to be used once a year!

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

BinaryToString? Really? That seems utterly impractical to me :  or to be used once a year!

 

It is impractical, but for someone who knows so little about the differences between the methods employed in various kinds of software (and the consequences which may occur), I feel I have finally gained some control over how my code will perform worldwide. I can also grab the binary for purposes beyond my personal AutoIt projects - post my code on the web etc...

Edited by czardas
Link to comment
Share on other sites

But isn't precisely why thousands of experts painfully specified Unicode?

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

You don't need to. Just copy Unicode from charmap.exe in advanced display mode into an UTF-enabled editor and you're done: worldwide fame!

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Yeah but you'd have to find commonly spread/available font(s) covering more. I now realize you're likely using musical symbols, so no charmap is impotent.

But then how do you handle encoding with surrogates?

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...