Testfile CJK - result is empty ?

[Myicq]

I am doing some test with iconv, to see what characters can be converted. To do this, I am creating a list of chinese characters, according to the list here.

For blocks 1 and 2 this works as expected.

But for some reason, blocks 3, 4 and 5 does not return a byte for the chinese character.

Am I missing something ?

My code :

$of = FileOpen("utf_block4.txt", 2+128) ; 32 = UTF16LE, 128 = UTF8 BOM
$myvar = ""

for $i = 0x2A700 to 0x2B734
        $myvar &= hex($i, 6) & " : >" & ChrW($i) & "<" & @CRLF
Next

FileWrite($of, $myvar)
FileWriteLine($of, "")
FileClose($of)

Edit: the forum won't let me post what I should get for the code.

Working version of block 1, first 5 lines - to see example output - is:

4E00 : >一<
4E01 : >丁<
4E02 : >丂<
4E03 : >七<
4E04 : >丄<

Edited March 30, 2015 by Myicq

[Myicq]

OK, so I found the culprit:

 

ChrW(x):

UNICODEcode A unicode code in the range 0-65535 (e.g., 65 returns the capital letter A).

 

Now, what do I do to return a string above 0xFFFF codepoints ?

[Myicq]

Thanks to trancexx  - you are a true Wizard.

Func _ChrW($iCodePoint) ; By trancexx
    If $iCodePoint <= 0xFFFF Then Return ChrW($iCodePoint)
; 
; uncomment if you need the @error value
;    If $iCodePoint > 0x10FFFF Then Return SetError(1, 0, "")
    Local $tOut = DllStructCreate("word[2]")
    DllStructSetData($tOut, 1, BitShift($iCodePoint, 10) + 0xD7C0, 1)
    DllStructSetData($tOut, 1, BitAND($iCodePoint, 0x3FF) + 0xDC00, 2)
    Return BinaryToString(DllStructGetData(DllStructCreate("byte[4]", DllStructGetPtr($tOut)), 1), 2)
EndFunc

Would be nice to have support for codepoints above 0xFFFF natively. But for now, problem solved.

[jchd]

Yeah, currently AutoIt is limited to UCS-2 charset, which is (roughly) the same as Unicode plane 0, representing every codepoint into a single 16-bit unit. Hence higher Unicode planes can't be represented natively like them would under UTF-16.

This will be a growing concern in future as higher planes become more commonly used.