Unicode stringToBinary for unsupported language

[carl1905]

Hi, I'm looking for a way to convert string to original binary.

The reversal process was already solved. BinaryToString for unsupported language

How can I convert unicode string to original binary format(Shift-JIS)?

$text(Unicode) = "データのダウンロードに失敗しました。" = "0xFFFEC730FC30BF306E30C030A630F330ED30FC30C9306B303159576557307E3057305F300230"

<Original form>
$stringToBinary(Shift-JIS) = "0x8366815B835E82CC835F83458393838D815B836882C98EB8947382B582DC82B582BD8142"

 

Edited May 2, 2016 by carl1905

[jchd]

Use _WinAPI_WideCharToMultiByte().

[carl1905]

1 hour ago, jchd said:

Use _WinAPI_WideCharToMultiByte().

Hi. You mean something like this? But it doesn't seem to work.

#include <WinAPI.au3>
Global Const $CP_SHIFT_JIS = 932

$String = 0xFFFEC730FC30BF306E30C030A630F330ED30FC30C9306B303159576557307E3057305F300230
$test = _WinAPI_MultiByteToWideChar($String, $CP_SHIFT_JIS)
MsgBox($MB_SYSTEMMODAL, "Title", $test)

 

Edited May 2, 2016 by carl1905

[jchd]

Much simpler:

$text(Unicode) = "データのダウンロードに失敗しました。"
$test = _WinAPI_MultiByteToWideChar($text, $CP_SHIFT_JIS)
MsgBox($MB_SYSTEMMODAL, "Title", $test)

But of course the MsgBox won't display JIS correctly.

[carl1905]

1 hour ago, jchd said:

Much simpler:

$text(Unicode) = "データのダウンロードに失敗しました。"
$test = _WinAPI_MultiByteToWideChar($text, $CP_SHIFT_JIS)
MsgBox($MB_SYSTEMMODAL, "Title", $test)

But of course the MsgBox won't display JIS correctly.

I can't understand what do you mean. Exactly what I want to do is read unicode from txt file and convert it to binary(Shift-JIS).

Let's assume that I read unicode from txt file,

"0xFFFEC730FC30BF306E30C030A630F330ED30FC30C9306B303159576557307E3057305F300230"

named it as $text then

$text = "0xFFFEC730FC30BF306E30C030A630F330ED30FC30C9306B303159576557307E3057305F300230"

So, the next step is..

$test = _WinAPI_MultiByteToWideChar($text, $CP_SHIFT_JIS)

And then the $test will be..

"0x8366815B835E82CC835F83458393838D815B836882C98EB8947382B582DC82B582BD8142"

Is it right? I think that I missed something..

Edited May 2, 2016 by carl1905

[jchd]

You confuse the ASCII reading of a binary UTF16-LE file with a UTF16-LE (e.g. native AutoIt) string. Did you try to run my example code?

As you code it, the variable $text contains exactly what is inside the double quotes, i.e. not what you want.

First, don't read the text as binary, don't make the BOM part of the string. The simplest way is to FileRead the file into $text.

[carl1905]

2 hours ago, jchd said:

You confuse the ASCII reading of a binary UTF16-LE file with a UTF16-LE (e.g. native AutoIt) string. Did you try to run my example code?

As you code it, the variable $text contains exactly what is inside the double quotes, i.e. not what you want.

First, don't read the text as binary, don't make the BOM part of the string. The simplest way is to FileRead the file into $text.

Thank you. Here is sample txt file and code that I made. It generates Shift-JIS txt from text file but it have some problems.

1) The original number of strings are 3 not 2. But in the 'NEW_sample.txt' there is only 2 strings.

2) I want to add 0x00

$plus = 0x00

at the end of string but my code convert it to 0x30.

#include <File.au3>
#include <Binary.au3>
#include <winapi.au3>
Global Const $CP_SHIFT_JIS = 932
Dim $NEWdata
$TxtPath = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)",1)
If @error = 1 Then Exit
_FileReadToArray($TxtPath,$NEWdata)

$strings = 3

$Newtext = ""
$plus = 0x00
For $i = 1 To $strings
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i],"<cf>",@CRLF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i],"<lf>",@LF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i],"<cr>",@CR)
    $bNewText = _WinAPI_WideCharToMultiByte ($NEWdata[$i],$CP_SHIFT_JIS)
    $Newtext &= $bNewText
    $Newtext &= $plus
Next
$Newfile = $Newtext
$hNewfile = FileOpen ("NEW_"&CompGetFileName($TxtPath), 2+16)
FileWrite ($hNewfile, $Newfile)
FileClose ($hNewfile)
TrayTip ("Import", "Finish!", 3)
sleep (3000)

Func CompGetFileName($Path)
If StringLen($Path) < 4 Then Return -1
$ret = StringSplit($Path,"\",2)
If IsArray($ret) Then
Return $ret[UBound($ret)-1]
EndIf
If @error Then Return -1
EndFunc

sample.txt

Edited May 2, 2016 by carl1905

[jchd]

#include <File.au3>
#include <Winapi.au3>
;~ #include <Binary.au3>    ; I don't know what this is supposed to refer to, but move this outside std UDFs directory

Global Const $CP_SHIFT_JIS = 932
Local $TxtPath = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)", 1)
If @error = 1 Then Exit
Local $NEWdata = FileReadToArray($TxtPath)
Local $Newtext
For $i = 0 To UBound($NEWdata) - 1
    ; are those 3 replacements necessary?
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<cf>", @CRLF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<lf>", @LF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<cr>", @CR)
    $Newtext &= _WinAPI_WideCharToMultiByte($NEWdata[$i], $CP_SHIFT_JIS) & Chr(0)
Next
$hNewfile = FileOpen("NEW_" & CompGetFileName($TxtPath), 2 + 16)
FileWrite($hNewfile, $Newtext)
FileClose($hNewfile)
TrayTip("Import", "Finish!", 3)
Sleep(3000)

Func CompGetFileName($Path)
    If StringLen($Path) < 4 Then Return -1
    $ret = StringSplit($Path, "\", 2)
    If IsArray($ret) Then
        Return $ret[UBound($ret) - 1]
    EndIf
    If @error Then Return -1
EndFunc   ;==>CompGetFileName

This modified version should do what you want. Beware the difference between 0x00 and Chr(0).

 

Edited May 2, 2016 by jchd

[carl1905]

1 hour ago, jchd said:

#include <File.au3>
#include <Winapi.au3>
;~ #include <Binary.au3>    ; I don't know what this is supposed to refer to, but move this outside std UDFs directory

Global Const $CP_SHIFT_JIS = 932
Local $TxtPath = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)", 1)
If @error = 1 Then Exit
Local $NEWdata = FileReadToArray($TxtPath)
Local $Newtext
For $i = 0 To UBound($NEWdata) - 1
    ; are those 3 replacements necessary?
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<cf>", @CRLF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<lf>", @LF)
    $NEWdata[$i] = StringRegExpReplace($NEWdata[$i], "<cr>", @CR)
    $Newtext &= _WinAPI_WideCharToMultiByte($NEWdata[$i], $CP_SHIFT_JIS) & Chr(0)
Next
$hNewfile = FileOpen("NEW_" & CompGetFileName($TxtPath), 2 + 16)
FileWrite($hNewfile, $Newtext)
FileClose($hNewfile)
TrayTip("Import", "Finish!", 3)
Sleep(3000)

Func CompGetFileName($Path)
    If StringLen($Path) < 4 Then Return -1
    $ret = StringSplit($Path, "\", 2)
    If IsArray($ret) Then
        Return $ret[UBound($ret) - 1]
    EndIf
    If @error Then Return -1
EndFunc   ;==>CompGetFileName

This modified version should do what you want. Beware the difference between 0x00 and Chr(0).

 

Thank you. However, even though I used your modified code, in my computer the output text file is different from you.

NEW_sample.txt

Edited May 2, 2016 by carl1905

[jchd]

All I can say is that I obtain the result shown running the exact code on the posted text file verbatim, under both release and beta, both X86 and x64.

Your setup must be differing somehow.

 

[carl1905]

4 hours ago, jchd said:

All I can say is that I obtain the result shown running the exact code on the posted text file verbatim, under both release and beta, both X86 and x64.

Your setup must be differing somehow.

 

Well.. I tested it on another computer, but the result is still same that of mine. I will check what is the problem.

[carl1905]

It seems that there are nothing wrong in the code and I use 3.3.14.0 version. Even though I delete autoit and reinstall but I still have no idea why I can't get the same result.

[Trong]

OK?
 

expandcollapse popup

#include <WinAPI.au3>
Global Const $CP_SHIFT_JIS = 932

Global $sFilePathIN = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)", 3)
If @error Then Exit
Global $sFilePathOUT = FileSaveDialog("Select local to save output", _SplitPath($sFilePathIN, 6), "Text (*.txt)", 16, "NEW_" & _SplitPath($sFilePathIN, 5))
If @error Then Exit
FileChangeDir(_SplitPath($sFilePathIN, 6))

Global $sHexAdd = "00"

Local $hOpen = FileOpen($sFilePathIN, 16384)
Local $sContent = FileRead($hOpen)
FileClose($hOpen)
$sContent = StringRegExpReplace($sContent, "<cf>", @CRLF)
$sContent = StringRegExpReplace($sContent, "<lf>", @LF)
$sContent = StringRegExpReplace($sContent, "<cr>", @CR)
$sContent = _WinAPI_WideCharToMultiByte($sContent, $CP_SHIFT_JIS)
$hOpen = FileOpen($sFilePathOUT, 2 + 8 + 16)
FileWrite($hOpen, Binary(StringToBinary($sContent) & $sHexAdd))
FileClose($hOpen)
TrayTip("Import", "Finish!", 3)
MsgBox(32, "", "Import Finish!", 3)


Func _SplitPath($sFilePath, $sType = 0)
    Local $sDrive, $sDir, $sFileName, $sExtension, $sReturn
    Local $aArray = StringRegExp($sFilePath, "^\h*((?:\\\\\?\\)*(\\\\[^\?\/\\]+|[A-Za-z]:)?(.*[\/\\]\h*)?((?:[^\.\/\\]|(?(?=\.[^\/\\]*\.)\.))*)?([^\/\\]*))$", 1)
    If @error Then
        ReDim $aArray[5]
        $aArray[0] = $sFilePath
    EndIf
    $sDrive = $aArray[1]
    If StringLeft($aArray[2], 1) == "/" Then
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\/")
    Else
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\\")
    EndIf
    $aArray[2] = $sDir
    $sFileName = $aArray[3]
    $sExtension = $aArray[4]
    If $sType = 1 Then Return $sDrive
    If $sType = 2 Then Return $sDir
    If $sType = 3 Then Return $sFileName
    If $sType = 4 Then Return $sExtension
    If $sType = 5 Then Return $sFileName & $sExtension
    If $sType = 6 Then Return $sDrive & $sDir
    If $sType = 7 Then Return $sDrive & $sDir & $sFileName
    Return $aArray
EndFunc   ;==>_SplitPath

 

Function:

#include <WinAPI.au3>

;~ _ConvertEncodingFile(@DesktopDir & "\sample.txt", @DesktopDir & "\NEW_sample.txt", 932) ;iCodepage: SHIFT_JIS = 932

Func _ConvertEncodingFile($sFilePathIN, $sFilePathOUT = Default, $iCodePage = 65001) ;iCodepage: UTF-8=65001
    ; iCodepage: https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756(v=vs.85).aspx
    If Not FileExists($sFilePathIN) Then Return SetError(1, 0, 0)
    Local $hOpen = FileOpen($sFilePathIN, 16384)
    If $sFilePathOUT = Default Then $sFilePathOUT = $sFilePathIN
    Local $sContent = FileRead($hOpen)
    If @error Or StringStripWS($sContent, 8) = "" Then Return SetError(2, 0, 0)
    FileClose($hOpen)
    $sContent = _WinAPI_WideCharToMultiByte($sContent, $iCodePage, 1)
    If @error Then Return SetError(3, @error, 0)
    $hOpen = FileOpen($sFilePathOUT, 2 + 8 + 16)
    FileWrite($hOpen, $sContent)
    If @error Then Return SetError(4, @error, 0)
    FileClose($hOpen)
    Return 1
EndFunc   ;==>_ConvertEncodingFile

 

Edited May 7, 2016 by Trong
#

[carl1905]

4 hours ago, VIP said:

OK?

Thanks for your help. This is the result from you code.

Current_result

However, the one that I expected is.. like this.

Expected result what I want

[Trong]

On 5/2/2016 at 10:10 PM, carl1905 said:

2) I want to add 0x00

$plus = 0x00

 

 

You need read and understand the script!  and change it to match your purpose!

expandcollapse popup

#include <WinAPI.au3>
Global Const $CP_SHIFT_JIS = 932

Global $sFilePathIN = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)", 3)
If @error Then Exit
Global $sFilePathOUT = FileSaveDialog("Select local to save output", _SplitPath($sFilePathIN, 6), "Text (*.txt)", 16, "NEW_" & _SplitPath($sFilePathIN, 5))
If @error Then Exit
FileChangeDir(_SplitPath($sFilePathIN, 6))

Local $hOpen = FileOpen($sFilePathIN, 16384)
Local $sContent = FileRead($hOpen)
FileClose($hOpen)
$sContent = StringRegExpReplace($sContent, "<cf>", @CRLF)
$sContent = StringRegExpReplace($sContent, "<lf>", @LF)
$sContent = StringRegExpReplace($sContent, "<cr>", @CR)
$sContent = _WinAPI_WideCharToMultiByte($sContent, $CP_SHIFT_JIS, 1)
$hOpen = FileOpen($sFilePathOUT, 2 + 8 + 16)
FileWrite($hOpen, $sContent)
FileClose($hOpen)
TrayTip("Import", "Finish!", 3)
MsgBox(32, "", "Import Finish!", 3)


Func _SplitPath($sFilePath, $sType = 0)
    Local $sDrive, $sDir, $sFileName, $sExtension, $sReturn
    Local $aArray = StringRegExp($sFilePath, "^\h*((?:\\\\\?\\)*(\\\\[^\?\/\\]+|[A-Za-z]:)?(.*[\/\\]\h*)?((?:[^\.\/\\]|(?(?=\.[^\/\\]*\.)\.))*)?([^\/\\]*))$", 1)
    If @error Then
        ReDim $aArray[5]
        $aArray[0] = $sFilePath
    EndIf
    $sDrive = $aArray[1]
    If StringLeft($aArray[2], 1) == "/" Then
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\/")
    Else
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\\")
    EndIf
    $aArray[2] = $sDir
    $sFileName = $aArray[3]
    $sExtension = $aArray[4]
    If $sType = 1 Then Return $sDrive
    If $sType = 2 Then Return $sDir
    If $sType = 3 Then Return $sFileName
    If $sType = 4 Then Return $sExtension
    If $sType = 5 Then Return $sFileName & $sExtension
    If $sType = 6 Then Return $sDrive & $sDir
    If $sType = 7 Then Return $sDrive & $sDir & $sFileName
    Return $aArray
EndFunc   ;==>_SplitPath

 

Edited May 7, 2016 by Trong

[carl1905]

5 minutes ago, VIP said:

 

You need read and understand the script!  and change it to match your purpose!

expandcollapse popup

#include <WinAPI.au3>
Global Const $CP_SHIFT_JIS = 932

Global $sFilePathIN = FileOpenDialog("Select the TXT file", @ScriptDir, "text files (*.txt)", 3)
If @error Then Exit
Global $sFilePathOUT = FileSaveDialog("Select local to save output", _SplitPath($sFilePathIN, 6), "Text (*.txt)", 16, "NEW_" & _SplitPath($sFilePathIN, 5))
If @error Then Exit
FileChangeDir(_SplitPath($sFilePathIN, 6))

Local $hOpen = FileOpen($sFilePathIN, 16384)
Local $sContent = FileRead($hOpen)
FileClose($hOpen)
$sContent = StringRegExpReplace($sContent, "<cf>", @CRLF)
$sContent = StringRegExpReplace($sContent, "<lf>", @LF)
$sContent = StringRegExpReplace($sContent, "<cr>", @CR)
$sContent = _WinAPI_WideCharToMultiByte($sContent, $CP_SHIFT_JIS)
$hOpen = FileOpen($sFilePathOUT, 2 + 8 + 16)
FileWrite($hOpen, StringToBinary($sContent))
FileClose($hOpen)
TrayTip("Import", "Finish!", 3)
MsgBox(32, "", "Import Finish!", 3)


Func _SplitPath($sFilePath, $sType = 0)
    Local $sDrive, $sDir, $sFileName, $sExtension, $sReturn
    Local $aArray = StringRegExp($sFilePath, "^\h*((?:\\\\\?\\)*(\\\\[^\?\/\\]+|[A-Za-z]:)?(.*[\/\\]\h*)?((?:[^\.\/\\]|(?(?=\.[^\/\\]*\.)\.))*)?([^\/\\]*))$", 1)
    If @error Then
        ReDim $aArray[5]
        $aArray[0] = $sFilePath
    EndIf
    $sDrive = $aArray[1]
    If StringLeft($aArray[2], 1) == "/" Then
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\/")
    Else
        $sDir = StringRegExpReplace($aArray[2], "\h*[\/\\]+\h*", "\\")
    EndIf
    $aArray[2] = $sDir
    $sFileName = $aArray[3]
    $sExtension = $aArray[4]
    If $sType = 1 Then Return $sDrive
    If $sType = 2 Then Return $sDir
    If $sType = 3 Then Return $sFileName
    If $sType = 4 Then Return $sExtension
    If $sType = 5 Then Return $sFileName & $sExtension
    If $sType = 6 Then Return $sDrive & $sDir
    If $sType = 7 Then Return $sDrive & $sDir & $sFileName
    Return $aArray
EndFunc   ;==>_SplitPath

 

Ok. I'll try it.

[carl1905]

On 2016. 5. 2. at 6:59 AM, jchd said:

Your setup must be differing somehow.

 Hi, jchd. I figure out what caused the difference. When I change the system locale to English. I can get the same result that of yours. However, when I change the locale back to the Korean, I get the wrong result. You may also check it.

 Anyway, would autoit also can handle locale? I mean, is there a way to adjust the modified code working on Multibyte locale system? Do I have to run it English locale only?

Edited May 7, 2016 by carl1905

[jchd]

Jeez, I don't see right now where the locale impacts processing. To find out, can you add debug statements (BinaryLen, MsgBox and StringLen, for instance) at various places of the script and compare between En and Kr locale settings. BTW I'm using French locale but that doesn't differ much from En as far as encoding is concerned.

[carl1905]

47 minutes ago, jchd said:

Jeez, I don't see right now where the locale impacts processing. To find out, can you add debug statements (BinaryLen, MsgBox and StringLen, for instance) at various places of the script and compare between En and Kr locale settings. BTW I'm using French locale but that doesn't differ much from En as far as encoding is concerned.

Yes. For example, in this sample code, $output_len is different from locale. English locale calculates it as 16, but Korean locale calculates it as 8.

#include <WinAPI.au3>

Global Const $CP_SHIFT_JIS = 932

$string = "よ、未来の美人！"
$unicode_len = BinaryLen($string)
MsgBox(0,"title",$unicode_len) ;=> It saids 8 on both English and Korean locale.

$output = _WinAPI_WideCharToMultiByte ($string , 932 , True )
$output_len = BinaryLen($output)
MsgBox(0,"title",$output_len) ;=> It saids 16 on English locale. However, it saids 8 on Korean locale.

$hNewfile = FileOpen("NEW_test.txt", 2 + 16)
FileWrite($hNewfile, $output)
FileClose($hNewfile)
TrayTip("Import", "Finish!", 3)
Sleep(3000)

 

Edited May 7, 2016 by carl1905

[jchd]

3 hours ago, carl1905 said:

It saids 16 on English locale. However, it saids 8 on Korean locale.

That is expected: only asian (korean, japanese, ...) codepages use actual multibyte encoding. Western codepages only see a string of bytes, each of them being an individual character.

What surprises me more is that you can only obtain 2 strings in your output with korean locale while looking at the hex dump of the result we (westerners) can see 3 strings.