Jump to content
careca

Yet another Encoding topic ANSI->UTF8

Recommended Posts

careca
Posted (edited)

Hello, trying to read from the console, and ive seen some posts where people mention changes in registry and in scite properties file, which i did, but none worked.

So now im stuck with this, as you can see, in the console the words are faithfully represented, but when autoit reads it, its lacking, in the help it says its a limitation that may be addressed in the future, but is there anything i can do, now to fix this?

 

Thanks.

 

consoler2.PNG

consoler1.PNG

Oh, the change in registry was this

[HKEY_CURRENT_USER\Console]
"CodePage"=dword:0000fde9
"FaceName"="Lucida Console"

Edited by careca
  • Haha 1

Spoiler

Paster - Main function is to paste text, but has more functions. (No longer mantained, switched to String Trigger)

Renamer - Rename files and folders, remove portions of text from the filename etc.

GPO Tool - Export/Import Group policy settings.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

String Trigger - Triggers pasting text or applications or internet links on specific strings.

Inconspicuous - Hide files in plain sight, not fully encrypted.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

IUIAutomation - Topic with framework and examples

Au3Record.exe

Share this post


Link to post
Share on other sites
jchd

Try converting the captured text with this:

Func _OemToStr($sOEM, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & StringLen($sOEM) & "]")
    DllStructSetData($tText, 1, $sOEM)
    ConsoleWrite(_vardump($tText) & @LF)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_OemToStr

You may have to explicitely pass the source codepage as parameter.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
careca

Hi, thank you for the reply, but it doesn't seem to be working somehow.

I tested this way, it's faster and easier:

file 1.txt with: Hello - 你好

#include <StringConstants.au3>
Global $iCodepage = 65001
$FO = FileOpen(@ScriptDir&'\1.txt')
$FR =FileRead($FO)
FileClose($FO)
MsgBox(64, '$FR', $FR)
;ConsoleWrite('$FR - '& $FR &' - '&@MSEC&@CRLF)
;$t1_unicode = Stringtobinary($FR,4)
;$t2_unicode = BinaryToString($t1_unicode, 4)
;ConsoleWrite('$t2_unicode - '& $t2_unicode &' - '&@MSEC&@CRLF)
;MsgBox(64, '$t2_unicode', $t2_unicode)
MsgBox(64, 'Read', _OemToStr($FR, $iCodepage))

Func _OemToStr($sOEM, $iCodepage)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & StringLen($sOEM) & "]")
    DllStructSetData($tText, 1, $sOEM)
    ;ConsoleWrite(_vardump($tText) & @LF)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", StringLen($sOEM), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_OemToStr

 


Spoiler

Paster - Main function is to paste text, but has more functions. (No longer mantained, switched to String Trigger)

Renamer - Rename files and folders, remove portions of text from the filename etc.

GPO Tool - Export/Import Group policy settings.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

String Trigger - Triggers pasting text or applications or internet links on specific strings.

Inconspicuous - Hide files in plain sight, not fully encrypted.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

IUIAutomation - Topic with framework and examples

Au3Record.exe

Share this post


Link to post
Share on other sites
jchd

Sorry I left a debugging aid in between.

But what's the difference?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
careca

No difference, "Hello - 你好" becomes "Hello - ??" same as not converting at all.

Either from console, or a txt file, when autoit handles them, they become jibberish.

 


Spoiler

Paster - Main function is to paste text, but has more functions. (No longer mantained, switched to String Trigger)

Renamer - Rename files and folders, remove portions of text from the filename etc.

GPO Tool - Export/Import Group policy settings.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

String Trigger - Triggers pasting text or applications or internet links on specific strings.

Inconspicuous - Hide files in plain sight, not fully encrypted.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

IUIAutomation - Topic with framework and examples

Au3Record.exe

Share this post


Link to post
Share on other sites
jchd

This kind of conversion requires that you correctly determine the actual encoding (and codepage when applicable) of the source.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
careca

Thank you for the help, i didn't notice this is a royal mess, thought i could just convert one or two types of encoding and it's done, but it's so much bigger than that.

As i looked for the code page of a language such as Polish...there's no way i can mess with the encoding for all the countries that use something else that's not unicode. The time and effort for this just doesn't compensate the results.

I only wish that one day we can get a uniformized standard for all countries, something that works everywhere and for everyone.

I actually learned a bit about this, but still a lot to learn and i thank you for the effort and help.

 


Spoiler

Paster - Main function is to paste text, but has more functions. (No longer mantained, switched to String Trigger)

Renamer - Rename files and folders, remove portions of text from the filename etc.

GPO Tool - Export/Import Group policy settings.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

String Trigger - Triggers pasting text or applications or internet links on specific strings.

Inconspicuous - Hide files in plain sight, not fully encrypted.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

IUIAutomation - Topic with framework and examples

Au3Record.exe

Share this post


Link to post
Share on other sites
jchd

The unique standard you're after is just that: Unicode. With it you can unambiguously and portably represent any character humanity ever used (albeit it's still a work in progress).

The mess is 8-bit character codepages and 2-byte codepages for some regions like Asia.

To focus again on your issue: in a CMD console type CHCP <Enter>. The system will return the actual codepage in force, which you should be able to use as parameter for the ANSI -> Unicode conversion function.

DOS used codepage 852 for central/eastern languages. Windows uses 1252 or 1250. Just check what's used in your own setup. If I'm not mistaken then your console "ŠŚŤŽŹšśťžźŁĄŻłąşŞĽľżŔÁÂĂÄĹĆÇ..." will convert verbatim in AutoIt native Unicode strings.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
begginner

I had similar problem as careca after executing dos and powershell commands by "Run"  and displaying results in Gui  using $STDOUT_CHILD . Instead of polish characters there was bunch of cryptic gibberish.  I used  _WinAPI_WideCharToMultiByte to do conversion  with any codepages but it didn't work.  I was so desperate to 

use StringReplace but problem was some of characters wasn't even appeared in output so was no way to replace them.  Then I tried this function that "jchd" posted

and it TOTALY WORKED !!! . THANK YOU SO MUCH for your help

Share this post


Link to post
Share on other sites
begginner
Posted (edited)

BTW  I tried exactly same command as   careca did  it with _OemToStr function and it works no problem

This is consolewrite output in SciTe before _OemToStr function

 

console2.JPG.af81f117957449b71be932575449bb3f.JPG

 

and after

 

consloe.JPG.4383f36d15b955d0ea35d3416e82dbd3.JPG

 

and this is Gui window

gui.JPG.457f09942cfcfd3fe41a0f6907c25929.JPG

CHCP command output "Active code page: 852"

Hope it helps

 

Edited by begginner
  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×