Issue with accented characters (CLI → clibpoard)

[littlebigman]

Hello,

I use the following script to 1) run a command-line application (ie. "DOS" app), 2) copy its output to the clipboard, and 3) paste it to Notepad as a simple way to search the data.

I notice that accented chars are wrong, eg. "é" becomes "é"

Is it because of the DOS to Windows character pages?

[…]
;run CLI app
Local $iPID = Run($WholeLine, $szDir, @SW_HIDE,$STDOUT_CHILD)
If @error Then Exit MsgBox($MB_ICONERROR, "Error", "Running failed.")
Local $sOutput = StdoutRead($iPID)
If @error Then Exit ConsoleWrite("Error returned by StdOutRead! @error=" & @error & ", @extended=" & @extended & @CRLF)
While True
    $sOutput &= StdoutRead($iPID)
    If @error Then ExitLoop
WEnd

;Poor man's search: copy STDOUT into clipboard, launch Notepad, paste
ClipPut($sOutput)
$iPID = Run("notepad.exe")
Opt("WinTitleMatchMode", 2) ;1=start, 2=subStr, 3=exact, 4=advanced, -1 to -4=Nocase
Local $hHdl = WinWait("[TITLE:Untitled - Notepad2; CLASS:Notepad2]","",10)
if  not $hHdl then Exit MsgBox($MB_ICONERROR,"Error","Windows not found")
Sleep(500)
Send("^v") ;paste
Sleep(1000)
Send("^{HOME}") ;jump to first line in notepad

Thank you.

[Nine]

Take a look at  _WinAPI_OemToChar()

[littlebigman]

Thanks. Looks like I'm using it wrong: This time, "é" turns into "+¬".

;run CLI app
Local $iPID = Run($WholeLine, $szDir, @SW_HIDE,$STDOUT_CHILD)
If @error Then Exit MsgBox($MB_ICONERROR, "Error", "Running failed.")
Local $sOutput = StdoutRead($iPID)
If @error Then Exit ConsoleWrite("Error returned by StdOutRead! @error=" & @error & ", @extended=" & @extended & @CRLF)
While True
    $sOutput &= StdoutRead($iPID)
    If @error Then ExitLoop
    ;ConsoleWrite($sOutput & @CRLF)
WEnd

;convert from DOS to ANSI
$sOutput = _WinAPI_OemToChar ($sOutput)

;Poor man's search: copy STDOUT into clipboard, launch Notepad, paste
ClipPut($sOutput)

[water]

I use the following function (taken from the forum):

; #INTERNAL_USE_ONLY# ===========================================================================================================
; Name ..........: _Ansi2Oem
; Description ...: Translates a string from ANSI to OEM.
; Syntax ........: _Ansi2Oem($sText)
; Parameters ....: $sText - Text to translate
; Return values .: Success - Returns the translated text
;                  Failure - None
; Author ........: Zedna
; Modified ......:
; Remarks .......:
; Related .......: Oem2Ansi (see link below)
; Link ..........: https://www.autoitscript.com/forum/topic/201398-german-umlauts-and-console-programs/?do=findComment&comment=1445253
; Example .......:
; ===============================================================================================================================
Func _Ansi2Oem($sText)
    Local $aText
    $aText = DllCall('user32.dll', 'Int', 'CharToOem', 'str', $sText, 'str', '')
    Return ConsoleWrite($aText[2])
EndFunc   ;==>_Ansi2Oem

 

[Nine]

Lets try something simple that you and I can actually run :

#include <WinAPIConv.au3>
#include <Constants.au3>

Local $iPID = Run(@ComSpec & " /c dir c:\", "", @SW_SHOW, $STDERR_MERGED)
ProcessWaitClose($iPID)
Local $sStream = StdoutRead($iPID)
MsgBox($MB_OK, "", _WinAPI_OemToChar($sStream))

Does that work ?

[ioa747]

Out of curiosity, have you checked this? or unchecked?

Settings > Time & Language >  Language >  Administrative language settings >  Change system locale

[littlebigman]

  On 4/21/2024 at 1:33 PM, water said:

I use the following function (taken from the forum):

; #INTERNAL_USE_ONLY# ===========================================================================================================
; Name ..........: _Ansi2Oem
; Description ...: Translates a string from ANSI to OEM.
; Syntax ........: _Ansi2Oem($sText)
; Parameters ....: $sText - Text to translate
; Return values .: Success - Returns the translated text
;                  Failure - None
; Author ........: Zedna
; Modified ......:
; Remarks .......:
; Related .......: Oem2Ansi (see link below)
; Link ..........: https://www.autoitscript.com/forum/topic/201398-german-umlauts-and-console-programs/?do=findComment&comment=1445253
; Example .......:
; ===============================================================================================================================
Func _Ansi2Oem($sText)
    Local $aText
    $aText = DllCall('user32.dll', 'Int', 'CharToOem', 'str', $sText, 'str', '')
    Return ConsoleWrite($aText[2])
EndFunc   ;==>_Ansi2Oem

 

Expand  

The input file is UTF8, so that didn't work.

  On 4/21/2024 at 1:36 PM, Nine said:

Does that work ?

Expand  

Nope, still wrong characters.

;Local $iPID = Run(@ComSpec & " /c dir c:\", "", @SW_SHOW, $STDERR_MERGED)
Local $iPID = Run(@ComSpec & " /c type test.utf8.txt", "", @SW_SHOW, $STDERR_MERGED)

 

  On 4/21/2024 at 2:32 PM, ioa747 said:

Out of curiosity, have you checked this? or unchecked?

Settings > Time & Language >  Language >  Administrative language settings >  Change system locale

Expand  

That solved it (reboot required.)

Thanks all.

[jchd]

FYI, on my system (Win10 x64 20h2) checking this setting has side effects: when I run a very simple piece of code from Scite, random strings are displayed in Scite console.

[ioa747]

  On 4/21/2024 at 3:54 PM, littlebigman said:

That solved it (reboot required.)

Expand  

checked ? or unchecked?

[littlebigman]

  On 4/21/2024 at 4:10 PM, ioa747 said:

checked ? or unchecked?

Expand  

From unchecked to checked.

  On 4/21/2024 at 4:04 PM, jchd said:

FYI, on my system (Win10 x64 20h2) checking this setting has side effects: when I run a very simple piece of code from Scite, random strings are displayed in Scite console.

Expand  

Thanks for the tip. If it occurs on mine, I'll revert back, and use an alternative — the application I need to use also as the option of saving the output to a file instead.

Checking the option in Windows added a second keyboard layout while I only had one before. No biggie, though.

[Nine]

As @jchd, checking this box does not work for me either.  I am unable to write french characters correctly in Scite console.  Did not test it extensively as I have no benefit of using it, or I may say, I have no issue without it...

[jchd]

@Nine, I fixed that by 1) setting this In SciTeUserProperties:

code.page=65001
output.code.page=65001

and using a good Unicode font:

font.monospace=font:DejaVu Sans Mono,size:11

Then I use CW() instead of ConsoleWrite():

Func CW($s = "")
    If @Compiled Then
        _CUI_ConsoleWrite($s)
    Else
        _ConsoleWrite($s)
    EndIf
EndFunc   ;==>CW


Func _CUI_ConsoleWrite(ByRef $s)
    Local Static $hDll = DllOpen("kernel32.dll")
    Local Static $hCon = __CUI_ConsoleInit($hDll)
    DllCall($hDll, "bool", "WriteConsoleW", "handle", $hCon, "wstr", $s & @LF, "dword", StringLen($s) + 1, "dword*", 0, "ptr", 0)
    Return
EndFunc   ;==>_CUI_ConsoleWrite


; internal use only
Func __CUI_ConsoleInit(ByRef $hDll)
    DllCall($hDll, "bool", "AllocConsole")
;~  The following 2 lines don't work for compiled scripts due to a MS bug
;~      see last post in thread https://developercommunity.visualstudio.com/t/setconsoleoutputcpcp-utf8-results-in-code-page-850/413190
;~      where MS support acknowledges it's a (still unfixed) bug in Windows itself
;~      So you can only display current locale codepage using current console font in a console when your script is compiled
;~      When running your script from SciTE, the UTF8 setting of SciTE overrides this bug and Unicode can be displayed
;~  DllCall("Kernel32.dll", "bool", "SetConsoleCP", "uint", 65001)
;~  DllCall("Kernel32.dll", "bool", "SetConsoleOutputCP", "uint", 65001)
    Return DllCall($hDll, "handle", "GetStdHandle", "int", -11)[0]
EndFunc   ;==>__CUI_ConsoleInit


; Unicode-aware ConsoleWrite
Func _ConsoleWrite(ByRef $s)
    ConsoleWrite(BinaryToString(StringToBinary($s & @LF, 4), 1))
EndFunc   ;==>_ConsoleWrite

 

[jchd]

Using the above changes, run this:

$s = "Большая проблема  大问题  बड़ी समस्या  مشكلة كبيرة Test Title 😭" & @LF
$s &= "Our familly " & ChrW(0xD83D) & ChrW(0xDC68) & ChrW(0xD83C) & ChrW(0xDFFB) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC69) & ChrW(0xD83C) & ChrW(0xDFFF) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC66) & ChrW(0xD83C) & ChrW(0xDFFD & @LF)

CW($s)

Result:

Большая проблема  大问题  बड़ी समस्या  مشكلة كبيرة Test Title 😭
Our familly 👨🏻‍👩🏿‍👦🏽

[ioa747]

more...

https://www.softwarert.com/change-language-windows-command-prompt/

[littlebigman]

  On 4/21/2024 at 4:12 PM, littlebigman said:

Checking the option in Windows added a second keyboard layout while I only had one before. No biggie, though.

Expand  

This seems to work to remove that second keyboard and go back to normal. I had a bad feeling about making changes to the Region section…and wasn't disappointed.