AU3 Unicode support issue. [Solved] :(

[Armand]

hey there,

i have a Unicode support issue in AU3, i get some unicoded text in 3 different languages from a text file and try to display it in a listview, 2 of the languages are showing just fine [Eng/Heb] however the third one doesn't [RUS] any idea how to fix it / why does it occur ?!

P.S-> just to make sure the letters are not scrambled by my computer i've spitted the text into a msgbox() one line before the actual listview implementation...

Pic:

Edited May 2, 2008 by Armand

[MrCreatoR]

Hi Armand, i handle the same problem using a Unicode encoding of the file that stores the displayed data.

Just try example bellow with this file:

#include <GuiConstants.au3>

$GUI = GUICreate("Test Script", 300, 200)

$ListView = GUICtrlCreateListView("Test", 20, 10, 260, 180)

GUICtrlCreateListViewItem(FileReadLine("Au3_Unicode.txt", 1), $ListView)
GUICtrlCreateListViewItem(FileReadLine("Au3_Unicode.txt", 2), $ListView)
GUICtrlCreateListViewItem(FileReadLine("Au3_Unicode.txt", 3), $ListView)

GUICtrlSendMsg($ListView, $LVM_SETCOLUMNWIDTH, 0, -1)

GUISetState(@SW_SHOW, $GUI)

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit
    EndSwitch
WEnd

Edited April 27, 2008 by MsCreatoR

[Armand]

@MsCreatoR

приьет אחשלם [hey mate]

{took me an hour to try and write it in rus}

Thanks man but...

well the thing is that it shows it just fine in my case also, but only in a msgbox... see the pic i've put...

Bahh !

P.s-> i don't know russian and that's why the text in russian is not readable, it's just the same chars as you type שלומיק in heb.

Edited April 27, 2008 by Armand

[MrCreatoR]

well the thing is that it shows it just fine in my case also, but only in a msgbox... see the pic i've put

See my post again, i corrected the example to ListView, for me it's work very well in my russian system, and i tested on Hebrue system as well, it shows all three languages correctly, as long as you keep the file encoding as Unicode. And your pictures shows russian correctly (just the meaning is missing)

[Armand]

ummzz... as long as u say it should work fine i'm happy.... cause well.. in my system that listview u sent me::

Pic:

bah... i guess it's because the main language in my system is ENG, HEB is secondary and rus is just accessory...

EDIT>>

Thanks a lot for the response mate !!!!

Edited April 28, 2008 by Armand

[MrCreatoR]

Are you using the Unicode version of AutoIt3.exe? add this to the example:

MsgBox(0, "", @Unicode)

It returns 1?

i guess it's because the main language in my system is ENG, HEB is secondary and rus is just accessory

I tested on the same system. You can check my last app that uses the same method: UserJS Manager for Opera

Just swith the language to russian (Ctrl + F12 - Language - select in the combobox RU and then OK), the elements should be displayed correctly, if they not, please let me know.

[Armand]

Now u made me sad... it shows russian just fine [i guess]... but not inside the table though, i have nothing russian to put there...

P.s, loved u'r program... awesome work... looks real pro.

[Armand]

Well.. i've done some digging with it... i've noticed u'r .lng file is set to "Unicode" encoding... when i changed it to "UTF-8" it appears to fail to show the correct strings...

My text file is set to UTF-8,

>>any way to make the transform of UTF8 string to Unicode encoding within AU3?

>>any way to make AU3 show UTF8 strings properly ?! [it does show hebrew just fine !!!]

EDIT>>>>

Well, just worked it out with that text file and @MsCreatoR's example... but not in my program...

however still in my program it shows the text fine in a msgbox but not in the listview.... any idea ?! Gush i hate bugs....

2nd EDIT>>>>

i've now tried to save the from within the script the UTF8 text which refuses to show in the listview but shows fine in msgboxes and then renamed it to the filename u used in u'r example and replaced the filename form the example... it works fine the above example... but not in my program - WHY ???? WHY ???? BAHHHHHHHH!!!!!!!!!

Edited April 28, 2008 by Armand

[Siao]

Don't muck around and post a short and exact example of how you are doing it. 300+ posts later and you still don't get the basic concept of getting help - POST CODE.

How else are we supposed to know what exactly you are doing wrong? Maybe you are opening the file without UTF8 mode switch, maybe you trying it on a listview created with _GUICtrlListView_Create... who knows.

[Armand]

@Siao

with all due respect Holy-Great-One, i can't post my entire code here... it just won't fit into the post... it's way over 15K lines with all of it's includes and so... nor will i give away the source after almost a year of hard labor....

I am creating the listview with:

$SearchList = GUICtrlCreateListView("#|Title|Release|Source|Links Type|#Links|Size|Downloads|Rating|Verified at|Language|Date|Uploader|Category", _
            214, $SearchGroupY + 45, $SearchW - 5, $SearchH - 65, -1, BitOR($LVS_EX_GRIDLINES, $LVS_EX_HEADERDRAGDROP, $LVS_EX_FULLROWSELECT, 0x00010000, $LVS_EX_SUBITEMIMAGES))

The text i get is in UTF8, i have mentioned that i'm saving the text into a file one line before i'm trying to push it into the listview using:

MsgBox(0, "Title:::", $CurrentSearchResults[$currentpage - 1][$currententry][0])
                    $FileHw = FileOpen("Title3lan.txt", 137)
                    FileWrite($FileHw, $CurrentSearchResults[$currentpage - 1][$currententry][0])
                    FileClose($FileHw)
$index = _GUICtrlListView_AddItem($SearchList_Hndl, $currententry + 1)
            _GUICtrlListView_SetItemText($SearchList, $index, $CurrentSearchResults[$currentpage - 1][$currententry][0], 1)

when i open that same .txt file and display it into the example listview given above it works just fine....

also when i use the save to option of notepad, it says the file is a UTF8 file.

Any idea ?!

THANKS IN ADVANCE!!!

>> sorry for wasting u'r time.

Edited April 28, 2008 by Armand

[MrCreatoR]

@Armand

First of all, if you can post a working example of your problem (that can reproduce the problem that you get), it will be much easyer to understand your problem, you don't have to put all your source code, just make a separate example that reproduce the problem.

But as i see it, your problem is that you writing to file with UTF-8 encoding, what is 137 in second parameter of FileOpen function?

Try to write it in Unicode:

$hFile = FileOpen("test.txt", 33) ;1 = Write mode (append to end of file) + 32 = Use Unicode UTF16 Little Endian mode
FileWrite($hFile, "Это тест")
FileClose($hFile)

Or you could just encode all strings that you get from file (UTF-8 string) to ANSI ecnoding:

$StringFromFile = _UTF8ToAnsi(FileReadLine("test.txt", 1))

Func _UTF8ToAnsi($sUTF8)
    Local $iLen = StringLen($sUTF8)
    Local $stBuf = DllStructCreate("byte[" & $iLen * 2 & "];byte[2]")
    Local $aRet = DllCall("Kernel32.dll", "int", "MultiByteToWideChar", _
        "int", 65001, "int", 0, _
        "str", $sUTF8, "int", -1, _
        "ptr", DllStructGetPtr($stBuf), "int", $iLen * 2 + 2)
    Local $stOut = DllStructCreate("char[" & $iLen & "];char")
    $aRet = DllCall("kernel32.dll", "int", "WideCharToMultiByte", _
        "int", 0, "int", 0, _
        "ptr", DllStructGetPtr($stBuf), "int", -1, _
        "ptr", DllStructGetPtr($stOut), "int", $iLen + 1, _
        "int", 0, "int", 0)
    Return DllStructGetData($stOut, 1)
EndFunc

Then it should be displayed more correctly (again, the UTF-8 strings).

Edited April 28, 2008 by MsCreatoR

[Siao]

Basically, all official ListView UDFs are ANSI, they don't trully support Unicode. When you _GUICtrlListView_SetItemText or whatever, ANSI text is used, so you will always have problems displaying symbols that aren't part of current codepage.

Rewrite the necessary UDF functions to support unicode ("wchar" instead of "char" for buffer structs, "MessageW" instead of "MessageA" for SendMessage calls), or simply use GuiCtrlCreateListViewItem. With total of controls upped to 64k in recent betas, you probably won't hit the limit.

Edited April 28, 2008 by Siao

[Armand]

@ Siao & @ MsCreatoR

THANKs ALOT !!!!!

problem solved , just had to switch the function and use the built in... much easier.

[Armand]

@Siao

The built-in function looks rather chumpy and jumpy... and it seems like the UDF is already built for UNICODE as i couldn't find any "Char(" or "MessageW(" functions.

These where found to be of use in the UDF:

also checked out the '_SendMessage()' function used in the UDF, it should support Unicode.

also checked out the '_WinAPI_MultiByteToWideChar()' function and it also should support Unicode.

Any idea ?!

[ProgAndy]

also checked out the '_SendMessage()' function used in the UDF, it should support Unicode.

also checked out the '_WinAPI_MultiByteToWideChar()' function and it also should support Unicode.

Yes, there is an _SendMessageA, which uses ansi, but

_SendMessage() uses just SendMessage and adds no W. If Theres no w, AutoIT tries to use A(nsi) Function, and not Unicode.

[MrCreatoR]

SendMessage is fine, you should use the unicode messages itself, for example «$LVM_SETITEMA» should be «$LVM_SETITEMW». All «char» i think should be replaced with «wchar»..

[MrCreatoR]

Modified _GUICtrlListView_AddSubItem:

Func _GUICtrlListView_AddSubItem($hWnd, $iIndex, $sText, $iSubItem, $iImage = -1)
    If $Debug_LV Then _GUICtrlListView_ValidateClassName($hWnd)
    Local $iMask, $iBuffer, $pBuffer, $tBuffer, $iItem, $pItem, $tItem, $pMemory, $tMemMap, $pText, $iResult

    $iBuffer = StringLen($sText) + 1
    $tBuffer = DllStructCreate("wchar Text[" & $iBuffer & "]")
    $pBuffer = DllStructGetPtr($tBuffer)
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    $iMask = $LVIF_TEXT
    If $iImage <> -1 Then $iMask = BitOR($iMask, $LVIF_IMAGE)
    DllStructSetData($tBuffer, "Text", $sText)
    DllStructSetData($tItem, "Mask", $iMask)
    DllStructSetData($tItem, "Item", $iIndex)
    DllStructSetData($tItem, "SubItem", $iSubItem)
    DllStructSetData($tItem, "Image", $iImage)
    If IsHWnd($hWnd) Then
        If _WinAPI_InProcess($hWnd, $_lv_ghLastWnd) Then
            DllStructSetData($tItem, "Text", $pBuffer)
            $iResult = _SendMessage($hWnd, $LVM_SETITEMW, 0, $pItem, 0, "wparam", "ptr")
        Else
            $iItem = DllStructGetSize($tItem)
            $pMemory = _MemInit($hWnd, $iItem + $iBuffer, $tMemMap)
            $pText = $pMemory + $iItem
            DllStructSetData($tItem, "Text", $pText)
            _MemWrite($tMemMap, $pItem, $pMemory, $iItem)
            _MemWrite($tMemMap, $pBuffer, $pText, $iBuffer)
            $iResult = _SendMessage($hWnd, $LVM_SETITEMW, 0, $pMemory, 0, "wparam", "ptr")
            _MemFree($tMemMap)
        EndIf
    Else
        DllStructSetData($tItem, "Text", $pBuffer)
        $iResult = GUICtrlSendMsg($hWnd, $LVM_SETITEMW, 0, $pItem)
    EndIf
    Return $iResult <> 0
EndFunc   ;==>_GUICtrlListView_AddSubItem

[Armand]

Bah... if it's just these mild changes why doesn't they change the UDF to support unicode by default ?!

by changing what u've mentioned above [char>>wchar and $LVM_SETITEMA>>$LVM_SETITEMW] can i produce any code degradation ?!

EDIT>>

just checked the results of this transformation... not working still showing '?????' Q marks...

expandcollapse popup

Func _GUICtrlListView_AddItem($hWnd, $sText, $iImage = -1)
    Return _GUICtrlListView_InsertItem($hWnd, $sText, -1, $iImage)
EndFunc   ;==>_GUICtrlListView_AddItem

Func _GUICtrlListView_InsertItem($hWnd, $sText, $iIndex = -1, $iImage = -1)
    If $Debug_LV Then _GUICtrlListView_ValidateClassName($hWnd)
    Local $iBuffer, $pBuffer, $tBuffer, $iItem, $pItem, $tItem, $pMemory, $tMemMap, $pText, $iMask, $iResult
    Local $iParam = _GUICtrlListView_GetItemCount($hWnd)
    If $iIndex = -1 Then $iIndex = 999999999
    
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tItem, "Param", 0)
    If $sText <> -1 Then
        $iBuffer = StringLen($sText) + 1
        $tBuffer = DllStructCreate("wchar Text[" & $iBuffer & "]")
        $pBuffer = DllStructGetPtr($tBuffer)
        DllStructSetData($tBuffer, "Text", $sText)
        DllStructSetData($tItem, "Text", $pBuffer)
        DllStructSetData($tItem, "TextMax", $iBuffer)
    Else
        DllStructSetData($tItem, "Text", -1)
    EndIf
    $iMask = BitOR($LVIF_TEXT, $LVIF_PARAM)
    If $iImage >= 0 Then $iMask = BitOR($iMask, $LVIF_IMAGE)
    DllStructSetData($tItem, "Mask", $iMask)
    DllStructSetData($tItem, "Item", $iIndex)
    DllStructSetData($tItem, "Image", $iImage)
    If IsHWnd($hWnd) Then
        If _WinAPI_InProcess($hWnd, $_lv_ghLastWnd) Or ($sText = -1) Then
            $iResult = _SendMessage($hWnd, $LVM_INSERTITEMA, 0, $pItem, 0, "wparam", "ptr")
        Else
            $iItem = DllStructGetSize($tItem)
            $pMemory = _MemInit($hWnd, $iItem + $iBuffer, $tMemMap)
            $pText = $pMemory + $iItem
            DllStructSetData($tItem, "Text", $pText)
            _MemWrite($tMemMap, $pItem, $pMemory, $iItem)
            _MemWrite($tMemMap, $pBuffer, $pText, $iBuffer)
            $iResult = _SendMessage($hWnd, $LVM_INSERTITEMA, 0, $pMemory, 0, "wparam", "ptr")
            _MemFree($tMemMap)
        EndIf
    Else
        $iResult = GUICtrlSendMsg($hWnd, $LVM_INSERTITEMA, 0, $pItem)
    EndIf
    Return $iResult
EndFunc   ;==>_GUICtrlListView_InsertItemoÝ÷ Ø2¢çÇoyÈZ¶,z-ý½æÞz{hÛh[(jëh×6Func _GUICtrlListView_SetItemText($hWnd, $iIndex, $sText, $iSubItem = 0)
    If $Debug_LV Then _GUICtrlListView_ValidateClassName($hWnd)
    Local $iBuffer, $pBuffer, $tBuffer, $iItem, $pItem, $tItem, $pMemory, $tMemMap, $pText, $iResult
    Local $i_cols, $a_text, $SeparatorChar = Opt('GUIDataSeparatorChar')

    If $iSubItem = -1 Then
        $i_cols = _GUICtrlListView_GetColumnCount($hWnd)
        $a_text = StringSplit($sText, $SeparatorChar)
        If $i_cols > $a_text[0] Then $i_cols = $a_text[0]
        For $i = 1 To $i_cols
            $iResult = _GUICtrlListView_SetItemText($hWnd, $iIndex, $a_text[$i], $i - 1)
            If Not $iResult Then ExitLoop
        Next
        Return $iResult
    EndIf

    $iBuffer = StringLen($sText) + 1
    $tBuffer = DllStructCreate("wchar Text[" & $iBuffer & "]")
    $pBuffer = DllStructGetPtr($tBuffer)
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tBuffer, "Text", $sText)
    DllStructSetData($tItem, "Mask", $LVIF_TEXT)
    DllStructSetData($tItem, "item", $iIndex)
    DllStructSetData($tItem, "SubItem", $iSubItem)
    If IsHWnd($hWnd) Then
        If _WinAPI_InProcess($hWnd, $_lv_ghLastWnd) Then
            DllStructSetData($tItem, "Text", $pBuffer)
            $iResult = _SendMessage($hWnd, $LVM_SETITEMW, 0, $pItem, 0, "wparam", "ptr")
        Else
            $iItem = DllStructGetSize($tItem)
            $pMemory = _MemInit($hWnd, $iItem + $iBuffer, $tMemMap)
            $pText = $pMemory + $iItem
            DllStructSetData($tItem, "Text", $pText)
            _MemWrite($tMemMap, $pItem, $pMemory, $iItem)
            _MemWrite($tMemMap, $pBuffer, $pText, $iBuffer)
            $iResult = _SendMessage($hWnd, $LVM_SETITEMW, 0, $pMemory, 0, "wparam", "ptr")
            _MemFree($tMemMap)
        EndIf
    Else
        DllStructSetData($tItem, "Text", $pBuffer)
        $iResult = GUICtrlSendMsg($hWnd, $LVM_SETITEMW, 0, $pItem)
    EndIf
    Return $iResult <> 0
EndFunc   ;==>_GUICtrlListView_SetItemText

Edited May 1, 2008 by Armand

[GaryFrost]

_GUICtrlListView_SetUnicodeFormat($hList, True)

[MrCreatoR]

In _GUICtrlListView_InsertItem() you did not changed the messages to unicode versions ($LVM_INSERTITEMA to $LVM_INSERTITEMW)...

expandcollapse popup

Func _GUICtrlListView_InsertItem($hWnd, $sText, $iIndex = -1, $iImage = -1)
    If $Debug_LV Then _GUICtrlListView_ValidateClassName($hWnd)
    Local $iBuffer, $pBuffer, $tBuffer, $iItem, $pItem, $tItem, $pMemory, $tMemMap, $pText, $iMask, $iResult
    Local $iParam = _GUICtrlListView_GetItemCount($hWnd)
    If $iIndex = -1 Then $iIndex = 999999999
    
    $tItem = DllStructCreate($tagLVITEM)
    $pItem = DllStructGetPtr($tItem)
    DllStructSetData($tItem, "Param", 0)
    If $sText <> -1 Then
        $iBuffer = StringLen($sText) + 1
        $tBuffer = DllStructCreate("wchar Text[" & $iBuffer & "]")
        $pBuffer = DllStructGetPtr($tBuffer)
        DllStructSetData($tBuffer, "Text", $sText)
        DllStructSetData($tItem, "Text", $pBuffer)
        DllStructSetData($tItem, "TextMax", $iBuffer)
    Else
        DllStructSetData($tItem, "Text", -1)
    EndIf
    $iMask = BitOR($LVIF_TEXT, $LVIF_PARAM)
    If $iImage >= 0 Then $iMask = BitOR($iMask, $LVIF_IMAGE)
    DllStructSetData($tItem, "Mask", $iMask)
    DllStructSetData($tItem, "Item", $iIndex)
    DllStructSetData($tItem, "Image", $iImage)
    If IsHWnd($hWnd) Then
        If _WinAPI_InProcess($hWnd, $_lv_ghLastWnd) Or ($sText = -1) Then
            $iResult = _SendMessage($hWnd, $LVM_INSERTITEMW, 0, $pItem, 0, "wparam", "ptr")
        Else
            $iItem = DllStructGetSize($tItem)
            $pMemory = _MemInit($hWnd, $iItem + $iBuffer, $tMemMap)
            $pText = $pMemory + $iItem
            DllStructSetData($tItem, "Text", $pText)
            _MemWrite($tMemMap, $pItem, $pMemory, $iItem)
            _MemWrite($tMemMap, $pBuffer, $pText, $iBuffer)
            $iResult = _SendMessage($hWnd, $LVM_INSERTITEMW, 0, $pMemory, 0, "wparam", "ptr")
            _MemFree($tMemMap)
        EndIf
    Else
        $iResult = GUICtrlSendMsg($hWnd, $LVM_INSERTITEMW, 0, $pItem)
    EndIf
    Return $iResult
EndFunc   ;==>_GUICtrlListView_InsertItem