Jump to content

Recommended Posts

Posted
  On 5/7/2016 at 6:18 PM, jchd said:

That is expected: only asian (korean, japanese, ...) codepages use actual multibyte encoding. Western codepages only see a string of bytes, each of them being an individual character.

What surprises me more is that you can only obtain 2 strings in your output with korean locale while looking at the hex dump of the result we (westerners) can see 3 strings.

Expand  

Thanks for your opinion. So.. there is no simple way to acquire corrected string on Korean locale, right?

Posted

I ain't saying that.

Can you try this (slightly simplified) version?

#include <WinAPI.au3>

Global Const $CP_SHIFT_JIS = 932

Local $TxtPath = "sample.txt"
Local $data = FileReadToArray($TxtPath)
Local $text
Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16)
For $i = 0 To UBound($data) - 1
    $text = _WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0)
    FileWrite($hfile, DllStructGetData($text, 1))
    FileWrite($hfile, Chr(0))
Next
FileClose($hfile)

I'm going to bet that the Chr(0) caused havoc in the previous code.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted
  On 5/8/2016 at 9:53 AM, jchd said:

I ain't saying that.

Can you try this (slightly simplified) version?

#include <WinAPI.au3>

Global Const $CP_SHIFT_JIS = 932

Local $TxtPath = "sample.txt"
Local $data = FileReadToArray($TxtPath)
Local $text
Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16)
For $i = 0 To UBound($data) - 1
    $text = _WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0)
    FileWrite($hfile, DllStructGetData($text, 1))
    FileWrite($hfile, Chr(0))
Next
FileClose($hfile)

I'm going to bet that the Chr(0) caused havoc in the previous code.

Expand  

Here is the result file. It still doens't work on Korean locale.

 

NEW_sample.txt

Posted (edited)
#include "..\include\dump.au3"

#include <WinAPI.au3>

Global Const $CP_SHIFT_JIS = 932

Local $TxtPath = "sample.txt"
Local $data = FileReadToArray($TxtPath)
_ConsoleWrite(_vardump($data) & @LF)
Local $text
Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16)
Local $pos
For $i = 0 To UBound($data) - 1
    $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0)
    _ConsoleWrite(@LF & _vardump($text) & @LF)
    $pos = FileGetPos($hfile)
    FileWrite($hfile, DllStructGetData($text, 1))
    ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF)
    ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF)
    $pos = FileGetPos($hfile)
Next
ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF)
FileClose($hfile)

Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True)
    Local $sUnicodeType = "wstr"
    If Not IsString($vUnicode) Then $sUnicodeType = "struct*"
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _
            "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "")

    Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]")

    $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _
            "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0)
    If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "")

    If $bRetString Then Return DllStructGetData($tMultiByte, 1)
    Return $tMultiByte
EndFunc   ;==>__WinAPI_WideCharToMultiByte

Dump.au3

Sorry for being bold, but can you run this from SciTE and paste here the console result and the hex of the file produced. Above is the link to download my Dump include.

 

Edited by jchd
  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted
  On 5/8/2016 at 2:32 PM, jchd said:
#include "..\include\dump.au3"

#include <WinAPI.au3>

Global Const $CP_SHIFT_JIS = 932

Local $TxtPath = "sample.txt"
Local $data = FileReadToArray($TxtPath)
_ConsoleWrite(_vardump($data) & @LF)
Local $text
Local $hfile = FileOpen("NEW_" & $TxtPath, 2 + 16)
Local $pos
For $i = 0 To UBound($data) - 1
    $text = __WinAPI_WideCharToMultiByte($data[$i], $CP_SHIFT_JIS, 0)
    _ConsoleWrite(@LF & _vardump($text) & @LF)
    $pos = FileGetPos($hfile)
    FileWrite($hfile, DllStructGetData($text, 1))
    ConsoleWrite(FileGetPos($hfile) - $pos & " bytes written to file in binary mode" & @LF)
    ConsoleWrite("Output file is now " & FileGetPos($hfile) & " bytes long" & @LF)
    $pos = FileGetPos($hfile)
Next
ConsoleWrite(@LF & "Final output file is " & FileGetPos($hfile) & " bytes long" & @LF)
FileClose($hfile)

Func __WinAPI_WideCharToMultiByte($vUnicode, $iCodePage = 0, $bRetString = True)
    Local $sUnicodeType = "wstr"
    If Not IsString($vUnicode) Then $sUnicodeType = "struct*"
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, "int", -1, _
            "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    If @error Or Not $aResult[0] Then Return SetError(@error + 20, @extended, "")

    Local $tMultiByte = DllStructCreate("byte[" & $aResult[0] & "]")

    $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", $iCodePage, "dword", 0, $sUnicodeType, $vUnicode, _
            "int", -1, "struct*", $tMultiByte, "int", $aResult[0], "ptr", 0, "ptr", 0)
    If @error Or Not $aResult[0] Then Return SetError(@error + 10, @extended, "")

    If $bRetString Then Return DllStructGetData($tMultiByte, 1)
    Return $tMultiByte
EndFunc   ;==>__WinAPI_WideCharToMultiByte

Dump.au3

Sorry for being bold, but can you run this from SciTE and paste here the console result and the hex of the file produced. Above is the link to download my Dump include.

 

Expand  

Thanks for your research, I copy and paste Dump.au3 to 'C:\Program Files (x86)\AutoIt3\Include' and run the code on my computer but I get an totally empty file. (0kb)

NEW_sample.txt

Posted

Oops, I forgot to mention that you need to use the latest beta since the support of Maps is already included in the UDF. Run that from SciTE.

Again, it isn't advisable to place user UDFs in the standard AutoIt UDF folder.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted
  On 5/9/2016 at 2:53 PM, jchd said:

Oops, I forgot to mention that you need to use the latest beta since the support of Maps is already included in the UDF. Run that from SciTE.

Again, it isn't advisable to place user UDFs in the standard AutoIt UDF folder.

Expand  

Oh, I can get the right result!! 

NEW_sample.txt

Posted

The difference lies in the WideCharToMultiByte function from the std UDF, where the dllstruct was using the char type, while it need the byte type to work correctly.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted

Now, can you please check this version and tell me whether it also works, so that I give the correct advice in the ticket I created?

  Reveal hidden contents

 

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

  • Moderators
Posted

jchd,

I have just asked you the same question in Trac!

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

  Reveal hidden contents

 

Posted

Hi Melba,

That's the correct fix. Now the question remains whether requesting the result as string (not struct) also works or not in the OP's case.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted

@carl1905,

Rather than just "like", I'd prefer a yes or no statement about whether the (new) code hidden in this post actually delivers the right result :huh2:

The difference here is that we request __WinAPIWideCharToMultiByte to return a string instead of a struct (the struct being used inside this function still being using byte itself). Sorry if that's painful but we ought to fix that correctly.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted (edited)
  On 5/10/2016 at 7:07 AM, jchd said:

Now, can you please check this version and tell me whether it also works, so that I give the correct advice in the ticket I created?

  Reveal hidden contents

 

Expand  

Hi. I'm on a mobile. However, I can't test your code right now because I have a conference of atomic physics until saturday. I'll test it on sunday. :-)

Edited by carl1905
Posted

Ah, sorry for that. Have fun launching fast neutrons around!

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted
  On 5/11/2016 at 1:27 PM, jchd said:

Ah, sorry for that. Have fun launching fast neutrons around!

Expand  

First of all, I'm sorry for my late reply. I had lots of works to do, so I didn't have a chance to look at it.

Here is the result files! It seems that it works correctly.

NEW_sample.txt

Posted

Thanks for checking and glad you survived the neutron bombing.
More seriously the test shows that we don't have to request the result as a DllStruct since the default String type works as well.

  Reveal hidden contents

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Posted
  On 5/18/2016 at 6:30 AM, jchd said:

Thanks for checking and glad you survived the neutron bombing.
More seriously the test shows that we don't have to request the result as a DllStruct since the default String type works as well.

Expand  

Thanks for your research!

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...