Unicode and StringRegExpReplace issues

[Ascend4nt]

Now there is another nice toy. Thank you.

Thanks, but I realized I should have swapped the hex order so it can be directly transferred to a PCRE. Here's the fixed version:

$sInputStr=InputBox("Unicode-Hex converter","Enter Unicode character(s) in box to see Hexadecimal equivalents","","",360,140)
If $sInputStr<>"" Then MsgBox(0,"Hex equivalent","Original string:"&@CRLF&$sInputStr&@CRLF&"Hexadecimal equivalents (enter as \x## for values starting in 00, otherwise \x{####}):"&@CRLF& _
    StringRegExpReplace(StringTrimLeft(StringToBinary($sInputStr,2),2),"(..)(..)","0x$2$1, "))

Edited May 20, 2010 by Ascend4nt

[jchd]

I didn't follow the thread completely and I apologize in advance if this is way off target.

Anyway, here we go. I've transposed into AutoIt some functions I use for one of my SQLite extensions. This is the part where I deal with Unicode casing and unaccenting. I transposed it from C after "some" editing... Please don't look at the code, it's ugly because I didn't have enough time to change the structure of some tables and, to be fully honest, I've little incentive to do so right now.

I believe the Unicode tries are correct as per Unicode v5.1, except for a small number of codepoints that I needed/whished to handle differently from the Unicode tables. The exceptions I introduced are few and I may miss some here. For instance, greek digamma Ϝ, ϝ are subject to casing, german Eszet ß and uppercase Eszet ẞ (U+1E9E) are handled in a special way, finally several symbols are expanded differently by the unaccenting function: fraction symbols use solidus [aka slash] instead of Unicode-recommended fraction bar, ™ gives TM, © gives (.C.) [without the dots!], ® gives (.R.) [without the dots!], all ligatures are expanded, and so on.

Living out the gory details, the OP might well find the _UnicodeUnaccent($str) function handy to do [part of] the job.

Short example (I leave it here so that the fancy characters will display nicely):

Local $s = "éÔµ€kißjàIJçè™…é$énÑÞþÆвℝÅ⁉fiflӸӡӁຳИЙ¼Ϝ½ϝ¾¿ϳʩʪʫʭǽ©ª®ij" & ChrW(0x1E9E)

_ConsoleWrite("_Lower = " & _UnicodeLower($s) & @LF)

_ConsoleWrite("Lower = " & StringLower($s) & @LF & @LF)

_ConsoleWrite("_Upper = " & _UnicodeUpper($s) & @LF)

_ConsoleWrite("Upper = " & StringUpper($s) & @LF & @LF)

_ConsoleWrite("_Title = " & _UnicodeTitle($s) & @LF & @LF)

_ConsoleWrite("_Fold = " & _UnicodeFold($s) & @LF & @LF)

_ConsoleWrite("_Unaccent = " & _UnicodeUnaccent($s) & @LF & @LF)

produces:

_Lower = éôµ€kißjàijçè™…é$énñþþæð²ℝå⁉fiflӹӡӂຳий¼ϝ½ϝ¾¿ϳʩʪʫʭǽ©ª®ijss

Lower = éôµ€kißjàijçè™…é$énñþþæð²ℝÅ⁉fiflӹӡӂຳий¼Ϝ½ϝ¾¿ϳʩʪʫʭǽ©ª®ijẞ

_Upper = ÉÔΜ€KISSJÀIJÇÈ™…É$ÉNÑÞÞÆвℝÅ⁉fiflӸӠӁຳИЙ¼Ϝ½Ϝ¾¿ϳʩʪʫʭǼ©ª®IJẞ

Upper = ÉÔµ€KIßJÀIJÇÈ™…É$ÉNÑÞÞÆвℝÅ⁉fiflӸӠӁຳИЙ¼Ϝ½ϝ¾¿ϳʩʪʫʭǼ©ª®IJẞ

_Title = ÉÔΜ€KISSJÀIJÇÈ™…É$ÉNÑÞÞÆвℝÅ⁉fiflӸӠӁຳИЙ¼Ϝ½Ϝ¾¿ϳʩʪʫʭǼ©ª®IJẞ

_Fold = éôμ€kißjàijçè™…é$énñþþæð²ℝå⁉fiflӹӡӂຳий¼ϝ½ϝ¾¿ϳʩʪʫʭǽ©ª®ijß

_Unaccent = eOμ€kissjaIJceTM...e$enNÞþAEÐ2RA!?fiflЫӡЖາИИ1/4Ϝ1/2ϝ3/4¿ϳʩʪʫʭae©a®ijSS

testtries.au3

Edit: posting this way was a bad idea: display is distorded, better try by yourself!

Edited May 21, 2010 by jchd

[GEOSoft]

I like this too. You people are just going hog wild building my repertoire of RegExp functions and I just know they are all going to come in handy.

[jchd]

I've taken some more time to read he heart of the thread. You guys are confusing ANSI and Unicode.

For instance:

$teststring = '123€ transformed by PCRE(™)'
$teststring = StringRegExpReplace($teststring, "[\x{2122}]", "TM")
$teststring = StringRegExpReplace($teststring, "[\x{20ac}]", "Euro")
_consolewrite($teststring & @LF)

works as expected if you lunch it from a UTF-8 + BOM encoded file. The code posted before was incorrect: 2122 is a hex value and \x99 is a control character in Unicode.

In the same way, the range posted from \x80 to \x9F is a range of Unicode control characters, so there is little chance they match text.

In short, there is no bug at all for me, or there is something big I miss.

Edited May 21, 2010 by jchd