carl1905 Posted May 8, 2016 Author Posted May 8, 2016 12 hours ago, jchd said: That is expected: only asian (korean, japanese, ...) codepages use actual multibyte encoding. Western codepages only see a string of bytes, each of them being an individual character. What surprises me more is that you can only obtain 2 strings in your output with korean locale while looking at the hex dump of the result we (westerners) can see 3 strings. Thanks for your opinion. So.. there is no simple way to acquire corrected string on Korean locale, right?
jchd Posted May 8, 2016 Posted May 8, 2016 I ain't saying that. Can you try this (slightly simplified) version? #include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) For $i = 0 To UBound($data) - 1 $text = _WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0) FileWrite($hfile, DllStructGetData($text, 1)) FileWrite($hfile, Chr(0)) Next FileClose($hfile) I'm going to bet that the Chr(0) caused havoc in the previous code. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 8, 2016 Author Posted May 8, 2016 3 hours ago, jchd said: I ain't saying that. Can you try this (slightly simplified) version? #include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) For $i = 0 To UBound($data) - 1 $text = _WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0) FileWrite($hfile, DllStructGetData($text, 1)) FileWrite($hfile, Chr(0)) Next FileClose($hfile) I'm going to bet that the Chr(0) caused havoc in the previous code. Here is the result file. It still doens't work on Korean locale. NEW_sample.txt
jchd Posted May 8, 2016 Posted May 8, 2016 (edited) expandcollapse popup#include "..\include\dump.au3" #include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) _ConsoleWrite(_vardump($data) & @LF) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) Local $pos For $i = 0 To UBound($data) - 1 $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0) _ConsoleWrite(@LF & _vardump($text) & @LF) $pos = FileGetPos($hfile) FileWrite($hfile, DllStructGetData($text, 1)) ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF) ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF) $pos = FileGetPos($hfile) Next ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF) FileClose($hfile) Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True) Local $sUnicodeType = "wstr" If Not IsString($vUnicode) Then $sUnicodeType = "struct*" Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _ "ptr", 0, "int", 0, "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "") Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]") $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _ "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "") If $bRetString Then Return DllStructGetData($tMultiByte, 1) Return $tMultiByte EndFunc ;==>__WinAPI_WideCharToMultiByte Dump.au3 Sorry for being bold, but can you run this from SciTE and paste here the console result and the hex of the file produced. Above is the link to download my Dump include. Edited May 8, 2016 by jchd This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 9, 2016 Author Posted May 9, 2016 17 hours ago, jchd said: expandcollapse popup#include "..\include\dump.au3" #include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) _ConsoleWrite(_vardump($data) & @LF) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) Local $pos For $i = 0 To UBound($data) - 1 $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0) _ConsoleWrite(@LF & _vardump($text) & @LF) $pos = FileGetPos($hfile) FileWrite($hfile, DllStructGetData($text, 1)) ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF) ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF) $pos = FileGetPos($hfile) Next ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF) FileClose($hfile) Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True) Local $sUnicodeType = "wstr" If Not IsString($vUnicode) Then $sUnicodeType = "struct*" Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _ "ptr", 0, "int", 0, "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "") Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]") $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _ "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "") If $bRetString Then Return DllStructGetData($tMultiByte, 1) Return $tMultiByte EndFunc ;==>__WinAPI_WideCharToMultiByte Dump.au3 Sorry for being bold, but can you run this from SciTE and paste here the console result and the hex of the file produced. Above is the link to download my Dump include. Thanks for your research, I copy and paste Dump.au3 to 'C:\Program Files (x86)\AutoIt3\Include' and run the code on my computer but I get an totally empty file. (0kb) NEW_sample.txt
jchd Posted May 9, 2016 Posted May 9, 2016 Oops, I forgot to mention that you need to use the latest beta since the support of Maps is already included in the UDF. Run that from SciTE. Again, it isn't advisable to place user UDFs in the standard AutoIt UDF folder. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 10, 2016 Author Posted May 10, 2016 9 hours ago, jchd said: Oops, I forgot to mention that you need to use the latest beta since the support of Maps is already included in the UDF. Run that from SciTE. Again, it isn't advisable to place user UDFs in the standard AutoIt UDF folder. Oh, I can get the right result!! NEW_sample.txt
jchd Posted May 10, 2016 Posted May 10, 2016 The difference lies in the WideCharToMultiByte function from the std UDF, where the dllstruct was using the char type, while it need the byte type to work correctly. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
jchd Posted May 10, 2016 Posted May 10, 2016 Now, can you please check this version and tell me whether it also works, so that I give the correct advice in the ticket I created? Spoiler expandcollapse popup#include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) ;~ _ConsoleWrite(_vardump($data) & @LF) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) ;~ Local $pos For $i = 0 To UBound($data) - 1 $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 1) ;~ _ConsoleWrite(@LF & _vardump($text) & @LF) ;~ $pos = FileGetPos($hfile) ;~ FileWrite($hfile, DllStructGetData($text, 1)) FileWrite($hfile, $text) ;~ ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF) ;~ ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF) ;~ $pos = FileGetPos($hfile) Next ;~ ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF) FileClose($hfile) Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True) Local $sUnicodeType = "wstr" If Not IsString($vUnicode) Then $sUnicodeType = "struct*" Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _ "ptr", 0, "int", 0, "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "") Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]") $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _ "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "") If $bRetString Then Return DllStructGetData($tMultiByte, 1) Return $tMultiByte EndFunc ;==>__WinAPI_WideCharToMultiByte carl1905 1 This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
Moderators Melba23 Posted May 10, 2016 Moderators Posted May 10, 2016 jchd, I have just asked you the same question in Trac! M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area
jchd Posted May 10, 2016 Posted May 10, 2016 Hi Melba, That's the correct fix. Now the question remains whether requesting the result as string (not struct) also works or not in the OP's case. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
jchd Posted May 11, 2016 Posted May 11, 2016 @carl1905, Rather than just "like", I'd prefer a yes or no statement about whether the (new) code hidden in this post actually delivers the right result The difference here is that we request __WinAPIWideCharToMultiByte to return a string instead of a struct (the struct being used inside this function still being using byte itself). Sorry if that's painful but we ought to fix that correctly. carl1905 1 This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 11, 2016 Author Posted May 11, 2016 (edited) On Tuesday, May 10, 2016 at 4:07 PM, jchd said: Now, can you please check this version and tell me whether it also works, so that I give the correct advice in the ticket I created? Reveal hidden contents expandcollapse popup#include <WinAPI.au3> Global Const $CP_SHIFT_JIS = 932 Local $TxtPath = "sample.txt" Local $data = FileReadToArray($TxtPath) ;~ _ConsoleWrite(_vardump($data) & @LF) Local $text Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16) ;~ Local $pos For $i = 0 To UBound($data) - 1 $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 1) ;~ _ConsoleWrite(@LF & _vardump($text) & @LF) ;~ $pos = FileGetPos($hfile) ;~ FileWrite($hfile, DllStructGetData($text, 1)) FileWrite($hfile, $text) ;~ ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF) ;~ ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF) ;~ $pos = FileGetPos($hfile) Next ;~ ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF) FileClose($hfile) Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True) Local $sUnicodeType = "wstr" If Not IsString($vUnicode) Then $sUnicodeType = "struct*" Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _ "ptr", 0, "int", 0, "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "") Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]") $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _ "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0) If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "") If $bRetString Then Return DllStructGetData($tMultiByte, 1) Return $tMultiByte EndFunc ;==>__WinAPI_WideCharToMultiByte Hi. I'm on a mobile. However, I can't test your code right now because I have a conference of atomic physics until saturday. I'll test it on sunday. :-) Edited May 11, 2016 by carl1905
jchd Posted May 11, 2016 Posted May 11, 2016 Ah, sorry for that. Have fun launching fast neutrons around! mLipok and carl1905 2 This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 18, 2016 Author Posted May 18, 2016 On 2016. 5. 11. at 10:27 PM, jchd said: Ah, sorry for that. Have fun launching fast neutrons around! First of all, I'm sorry for my late reply. I had lots of works to do, so I didn't have a chance to look at it. Here is the result files! It seems that it works correctly. NEW_sample.txt
jchd Posted May 18, 2016 Posted May 18, 2016 Thanks for checking and glad you survived the neutron bombing. More seriously the test shows that we don't have to request the result as a DllStruct since the default String type works as well. carl1905 1 This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
carl1905 Posted May 19, 2016 Author Posted May 19, 2016 On 2016. 5. 18. at 3:30 PM, jchd said: Thanks for checking and glad you survived the neutron bombing. More seriously the test shows that we don't have to request the result as a DllStruct since the default String type works as well. Thanks for your research!
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now