Jump to content
GillesMaisonneuve

How to read unicode utf8 string from sub process (Run) with StdoutRead()?

Recommended Posts

Good morning,

 

I am trying to read a Unicode utf8 string from a perl subprocess via StdoutRead.

I use an AUtoIt GUI and display result in an 'Edit' control (see my code below) using 'Courier New', a font that can handle Unicode characters.

I was expecting a result looking like (CMD console):

++$ chcp 65001>NUL: & perl -Mutf8 -CS -e "use 5.018; binmode STDOUT,q(:utf8); say qq(\x{03A9})" & chcp 850>NUL:
Ω

Instead I get someting like this (see downward the screen copy):

++$ chcp 1250>NUL: & perl -Mutf8 -CS -e "use 5.018; binmode STDOUT,q(:utf8); say qq(\x{03A9})" & chcp 850>NUL:
Ω

Obviously while I was expecting to receive an utf8 char, it seems to have been converted to Windows ANSI codepage 1250 (Windows default for Western/Central Europe, right ?)

What am I doing wrong? Is there someone who could guide me?

 

Here is my code and my output in the GUI.

Creating and configuring the Edit control:

Local $Edit1 = GUICtrlCreateEdit( "", 20, 110, 780, 500, BitOr($GUI_SS_DEFAULT_EDIT,$ES_MULTILINE,$ES_READONLY) )
GUICtrlSetData($Edit1, "This field will contain text result from external Perl command")
GUICtrlSetFont($Edit1, 10, $FW_THIN, $GUI_FONTNORMAL, "Courier New")

 

Executing Perl command (note: `-Mutf8` and `-CS` garantees that I work in utf8 and STDOUT accepts wide-characters):

local $ExePath = 'perl.exe -Mutf8 -CS '     ;~ if perl in PATH, no need for full path C:\Perl\bin\perl.exe
local $Params = '-e "use 5.018; use utf8; use charnames q(:full);  binmode STDOUT,q(:utf8);' & _
                'say scalar localtime; say qq(\N{GREEK CAPITAL LETTER OMEGA})"'
local $Cmd = $ExePath & ' ' & $Params
Local $iPID = Run($Cmd, "", @SW_HIDE, BitOR($STDERR_CHILD, $STDOUT_CHILD))

 

Reading STDOUT and displaying it into the Edit control:

While 1
    $sOutput &= StdoutRead($iPID)
    If @error Then ; Exit the loop if the process closes or StdoutRead returns an error.
        ExitLoop
    EndIf
WEnd
If $sOutput <> '' Then
    GUICtrlSetData($Edit1, $sOutput)
EndIf

 

And now, what I get on my GUI:

 

AutoIt_stdoutread_unicode_00.png

Share this post


Link to post
Share on other sites
Posted (edited)

@GillesMaisonneuve

Edited by FrancescoDiMuro
Snip.

Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites

"Courier New" definitely handles unicode characters.

Ω is 0xCE 0xA9 = 0x03A9 = 'Ω' encoded in UTF8. You need to convert the UTF8 string to the native encoding used by AutoIt (a subset of UTF16-LE named UCS2).

#include <StringConstants.au3>

Local $sOutput = Chr(0xCE) & Chr(0xA9)

If $sOutput <> '' Then
    $sOutput = BinaryToString($sOutput, $SB_UTF8)
;~     GUICtrlSetData($Edit1, $sOutput)
    MsgBox(0, "", $sOutput)
EndIf

 


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By Skysnake
      'F‌inal' ; <========= problem between F and i 'F‌inal' ; <========= problem between F and i 'Final' ; <========= Notepad ANSI, no problem
      F‌inal

      PostgreSQL complains about that red dot.  Notepad++ marks the first word.
      Notepad said it contains Unicode.  Saved as ANSI and put back. Red dot is gone.
       
      the Text is inserted manually into an AutoIt Input box.  Things get done to it, and it ends up in a SQL database.  The data seems fine, but when I start generating reports, all kinds of funny problems show up.
       
      If I can ID that character, I can remove it.
      Any ideas?
    • By Funtime60
      I have a Parent Script that starts multiple, concurrent child processes. I cannot figure out how to apply the technique for using StdoutRead in the examples in a way that listens to them all simultaneously.
      For reference, there can be any even number of, or single, child processes and there is an 1D array, $PIDS, that stores the PIDS of all the child processes and they should be outputting to another 1D array $sOutput.
      Thank you for your time. Please just point me in the right direction. I'm sorry if I missed another topic that covers this.
    • By Blueman
      Hey Guys,

      I have a (i think) simple question, but i can't seem to get the answer though the help-files.
      Hope that you can help me with this issue.
      - I am opening a SSH Connection with the following function
      Func _Connect($session,$usr,$pass) $exec = @ScriptDir & "\PLINK.EXE" If Not FileExists($exec) Then _Err("PLINK.EXE Not Found!",0) $pid = Run($exec & " -load " & $session & " -l " & $usr & " -pw " & $pass, @ScriptDir, @SW_HIDE, 0x1 + 0x8) ;Run SSH.EXE If Not $pid Then _Err("Failed to connect",0) $currentpid = $pid ;$rtn = _Read($pid) ;Check for Login Success - Prompt ;MsgBox(48,"","1") ;sleep(5000) ;Wait for connection Return $pid EndFunc - This will connect to a CMS Server with a vt100 interface where a dynamic report is generated every 20 seconds.
      - Then i will read the contents with the following Function
      Func _Read($pid) $SSHREADTIMEOUT = 0 If Not $pid Then Return -1 Local $dataA Local $dataB Do $SSHREADTIMEOUT += 1 $dataB = $dataA sleep(100) $dataA &= StdOutRead($pid) If @error Then ExitLoop Until ($dataB = $dataA And $dataA And $dataB) OR $SSHREADTIMEOUT == 50 Return $dataA EndFunc This all goes correctly, but i can only read the contents once.
      When i try to read the contents again i get nothing.
      Maybe because the CMS isn't changing, but the values in the report is changing every 20 seconds.
      I need to somehow read al of the contents every time i perform a Read action, but how?
      Yes, i can use it in a While loop, but also then i get nothing or a small line of text and not the whole report.
       
      Any Idea?
       
      Thanks Guys!
       
      --Edit--
      I have fixed the problem by changing the terminal session to a vt220 session.
      The only problem now is that i want to send the: "Data link escape" command and that is something i cannot fix,.
      I have tried;
      StdinWrite($Pid,Hex(0x10)) StdinWrite($Pid,Chr(16)) StdinWrite($Pid,{DLE}) But nothing seems to work.  
      -- Edit-2 --
      Guys, fixed that too!
      Forgot to add the number '5' to actually execute the assignment.
      So fixed it by using;
       
      StdinWrite($Pid,Chr(16)) StdinWrite($Pid,"5")
      Thanks for reading with me
       
    • By rudi
      Hello,
      to provide an easy to use starter to capture traffic on all NICs found, I can successfully get all the interfaces of TSHARK.EXE (the command line version that's automatically installed along with wireshark) with this script:
      #include <AutoItConstants.au3> #include <Array.au3> $TS_WD = "C:\Program Files\Wireshark" $TS_exe = $TS_WD & "\tshark.exe" if not FileExists($TS_exe) Then MsgBox(48,"Fatal Error","No Wireshark Commandline Tool ""TSHARK.EXE"" found:" & @CRLF & _ $TS_exe) Exit EndIf $DString = "" $PIDGetIFs = Run($TS_exe & " -D", $TS_WD, @SW_HIDE, $STDERR_MERGED) While ProcessExists($PIDGetIFs) $DString &= StdoutRead($PIDGetIFs) WEnd ; MsgBox(0,"IFs",$DString) $aNICs = StringSplit($DString, @CRLF, 1) _ArrayDisplay($aNICs) $RegExIF = "^(?:\d+\. )(\\.*?})(?: \()(.*?)(\))$" ; $1 = TSHARK Interface Name, $2 = Windows Interface Name ; ... get the names to run TSHARK with the appropriate interface string  
      When I run TSHARK.EXE using this line directly, I see a continuously growing number telling the number of packets captured so far.
      "C:\Program Files\Wireshark\tshark.exe" -i \Device\NPF_{AEB931E9-E5FA-4DA5-8328-D87BDF53805C} -b duration:300 -b files:600 -w "y:\TShark-Ringbuffer\LAN-Verbindung\TSHARK-Com-0317_17---LAN-Verbindung___.pcap" Using this script, I *DO* see the first output line "Capturing on 'LAN-Verbindung'", but I cannot get hold of the continuously growing number of packets captured so far.
      #include <AutoItConstants.au3> $WD="C:\Program Files\Wireshark" $CMD='"C:\Program Files\Wireshark\tshark.exe" -i \Device\NPF_{AEB931E9-E5FA-4DA5-8328-D87BDF53805C} -b duration:300 -b files:600 -w "y:\TShark-Ringbuffer\LAN-Verbindung\TSHARK-Com-0317_17---LAN-Verbindung___.pcap"' ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $CMD = ' & $CMD & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $PID=Run($CMD,$WD,@SW_SHOW,$stderr_merged) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $OutputAll="" While ProcessExists($PID) $output=StdoutRead($PID) $OutputAll&=$output ToolTip($OutputAll) if $output <> "" then ConsoleWrite("""" & $output & """" & @CRLF) Sleep(1000) WEnd ConsoleWrite("Process vanished" & @CRLF)  
      This is the output of SciTE, when I let TSHARK.EXE run for a short while, the "close" it's "box" ...
       
      --> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop @@ Debug(10) : $CMD = "C:\Program Files\Wireshark\tshark.exe" -i \Device\NPF_{AEB931E9-E5FA-4DA5-8328-D87BDF53805C} -b duration:300 -b files:600 -w "y:\TShark-Ringbuffer\LAN-Verbindung\TSHARK-Com-0317_17---LAN-Verbindung___.pcap" >Error code: 0 @@ Debug(13) : $PID = 10948 >Error code: 0 "Capturing on 'LAN-Verbindung' " Process vanished  
      Howto catch the "growing-packet-number" TSHARK.EXE is writing continuously to the same "window position"???
       
       
      Regards, Rudi.
       


    • By ModemJunki
      OK, the odd stuff seems to happen to me. Not sure if this belongs here or in the deployment forum.
      The Question: Why would the console output results from a command-line tool need to know the working dir on one system but not the other?
      Though in the end I solved the issue I still am not understanding why StdoutRead seems to behave differently as I describe below.
      Preamble: Windows 10 image deployment to two different HP workstation hardware platforms. After deployment set up some disk space using Intel RAID. I script the Intel RSTe client (Intel® Rapid Storage Technology enterprise). Same version of the RSTe client and same storage controller hardware (Intel C600+/C220+ series chipset SATA RAID controller). The only difference is in the driver - one system has 4.2.0.1136 and the other has 5.2.0.1194, but even after I update the older driver the anomaly persists. There are of course other differences - the older system has different CPUs (Xeon E5-2620 vs Xeon Silver 4108), different chipsets, different SMBIOS, etc..). Both have Nvidia Quadro cards with driver 385.90, but the older has 3x P2000 cards and the newer has 3x Quadro M4000 and a Quadro K1200.
      To script RAID creation and maintenance I have a function that uses StdoutRead with Intels command line client tool "rstcli64.exe".
      On the older system, the below call to StdoutRead fails. On the newer system, it works. This only started happening on the older hardware when I updated the Windows version 1607 image to build 14393.2189 (but the Windows image used is the same for both hardware targets).
      $s_DoRST = Run(@ComSpec & " /c " & $s_RSTcmd, "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) But if I change it to use @ScriptDir as the working dir, it works for both systems.
      $s_DoRST = Run(@ComSpec & " /c " & $s_RSTcmd, @ScriptDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)  The code below is the one that works on both systems (hint: the comment " ; this only deals with one device" is because I've no hardware with more than one ATAPI device to test with).
      $s_RSTExe = ".\rstcli64.exe" ; for production (compiled executable) ; #FUNCTION# ==================================================================================================================== ; Name ..........: GetDiskInfoFromRSTE() ; Description ...: Queries for hard disks attached to the controller ; Syntax ........: GetDiskInfoFromRSTE() ; Parameters ....: None ; Return value...: An array of attached hard disk datas from the RST command line tool without ATAPI devices such as optical drives ; =============================================================================================================================== Func _GetDiskInfoFromRSTE() Local $s_RSTcmd = $s_RSTExe & " -I -d" Local $s_DoRST, $a_RSTDisksTemp, $s_rawdata, $numDISK = 0 If $Debug <> 0 Then ConsoleWrite("Get disk info: " & $s_RSTcmd & @CRLF) ; useful during dev but cannot be used if #requireadmin $s_DoRST = Run(@ComSpec & " /c " & $s_RSTcmd, @ScriptDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) ProcessWaitClose($s_DoRST) $s_rawdata = StdoutRead($s_DoRST) $a_RSTDisksTemp = StringSplit($s_rawdata, @CRLF) For $i = $a_RSTDisksTemp[0] To 1 Step -1 ; since we are deleting items, we have to go backwards through the array If $a_RSTDisksTemp[$i] = "" Or $a_RSTDisksTemp[$i] = "--DISK INFORMATION--" Then _ArrayDelete($a_RSTDisksTemp, $i) EndIf Next $a_RSTDisksTemp[0] = UBound($a_RSTDisksTemp) If IsArray($a_RSTDisksTemp) Then For $j = $a_RSTDisksTemp[0] - 1 To 1 Step -1 ; we need to remove the ATAPI device(s), since we delete we have to step through in reverse If StringInStr($a_RSTDisksTemp[$j], "Disk Type: SATA") Then $numDISK = $numDISK + 1 EndIf Next For $j = $a_RSTDisksTemp[0] - 1 To 1 Step -1 ; we need to remove the ATAPI device(s), since we delete we have to step through in reverse If StringInStr($a_RSTDisksTemp[$j], "ATAPI") Then ; we have to delete the ID entry BEFORE and AFTER the ATAPI entry to remove the ATAPI device (always delete from the bottom up!). _ArrayDelete($a_RSTDisksTemp, $j + 1) ; delete entry below _ArrayDelete($a_RSTDisksTemp, $j) ; delete entry _ArrayDelete($a_RSTDisksTemp, $j - 1) ; delete entry above ExitLoop ; this only deals with one device. If there were more we would need to capture all ATAPI device ID locations into another array then delete the original array entries based on the index numbers in the new array. EndIf Next $a_RSTDisksTemp[0] = UBound($a_RSTDisksTemp) If $a_RSTDisksTemp[0] <> 1 Then If $Debug <> 0 Then ConsoleWrite("Found " & $numDISK & " disk(s) attached to controller." & @CRLF) _infoLog("_GetDiskInfoFromRSTE() found " & $numDISK & " disk(s) attached to controller.", $msgSource) Else If $Debug <> 0 Then ConsoleWrite("No disks found to be attached to controller." & @CRLF) _infoLog("No disks found to be attached to controller.", $msgSource) EndIf Else If $Debug <> 0 Then ConsoleWrite("Unable to query RAID controller. Check hardware and version of RST client software." & @CRLF) _errorLog("Unable to query RAID controller. Check hardware.", $msgSource) Exit EndIf Return $a_RSTDisksTemp EndFunc ;==>_GetDiskInfoFromRSTE  
×
×
  • Create New...