Fast string manipulation

[IronFine]

Hi everyone,

I'm currently working on a project that involves heavy string manipulation, where I am working with large datasets. There I need to match specific patterns, modify the matches, and then replace the changes back into the string.

I am already mixed the use of StringReplace and StringRegExpReplace, depending where either one is fater (determined that for most of the cases once). Also I am using the case-sensitivity flag/replace count where possible. I tried to also use CLR and calling a C# hoping that that would be faster, but with the overhead it is only faster in some cases.

Are there any other tipps that I can use?

Edited April 11 by IronFine

[Nine]

There is not much you can do using standard String* functions.  Most of them (when modifying the original string) makes a copy of the string.  With small strings, it is not a big deal.  But when it comes to large strings, it can be quite lengthy to execute multiple string operations over a single string.

One method would be to perform in place replacements.  That is quite easy if you replace N characters with some other N characters.  Also it depends where the strings are coming from and where they go (text files ?).  Do they have known formats they are following (like json or xml) ?

Best way to help you, would be for you to provide a real life example, so we can try to look into a solution tailored to your situation.

[jchd]

What @Nine said, but beware of this:

13 hours ago, Nine said:

That is quite easy if you replace N characters with some other N characters.

Not that easy.

Windows uses UTF16-LE internally, which means that a "character" (here I mean actually "one Unicode codepoint") can use either one or two 16-bit words. in RAM. Also, don't forget that AutoIt uses a subset of UTF16-LE named UCS-2, allowing full use of the Unicode BMP (Basic Multilingual Plane), i.e. codepoints U+0000 to U+FFFF. Hence, counting "characters" is not trivial.

Common "characters" in non-english languages like à, Ê, ù, ç, ñ, ... can be represented using different normalization forms, see https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

Moretheless what we, humans, mean by "character" may require more than one Unicode codepoint. This is the difference between glyph (what is displayed by the font renderer) and codepoint (the actual Unicode content). For instance this string

$s = ChrW(0xD83D) & ChrW(0xDC68) & ChrW(0xD83C) & ChrW(0xDFFB) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC69) & ChrW(0xD83C) & ChrW(0xDFFF) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC66) & ChrW(0xD83C) & ChrW(0xDFFD)

displays as in a Unicode-aware environment.

And that is only speaking of valid representations: one may find out-of-spec representations in input strings, spanning representation of what we perceive as a "character" over an unbounded string of Unicode codepoints.

Edited April 12 by jchd

[IronFine]

Thank you both for your reply.

@Nine my script is processing text-files/source code with nested expressions. Each expression is replaced with a token, operating from top to bottom.

Once everything is tokenized, the string for the nested expression is manipulated. Once all that is done, all the token will be replaced with the new/manipulated string - bottom to top. 

[Nine]

Again, without a real example from your text files, not much we can recommend to help you.

[junkew]

you could quickly split it into single (unicode) chars with stringsplit and then iterate each character basically build your own tokenizer. 
Not neccesarily fast in AutoIt
 

Local $sText = "Winter"& ChrW(9731) & "Snow" & ChrW(9731) & "Ice"
; Split in single (unicode)characters
Local $aResult = StringSplit($sText, "")

local $foundStr=""

For $i = 1 To $aResult[0]
    $char=$aResult[$i]
    if ($char=chrw(9731)) Then
        _ConsoleWrite($foundStr & " rain" & @CRLF)
        $foundStr=""
    Else
        $foundStr&=$char
    endif
Next
_ConsoleWrite($foundStr & " rain" & @CRLF)

Func _ConsoleWrite($sString)
    Local $aResult = DllCall("kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sString, "int", -1, _
                                "ptr", 0, "int", 0, "ptr", 0, "ptr", 0)
    If @error Then Return SetError(1, @error, 0)
    Local $tText = DllStructCreate("char[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "WideCharToMultiByte", "uint", 65001, "dword", 0, "wstr", $sString, "int", -1, _
                            "ptr", DllStructGetPtr($tText), "int", $aResult[0], "ptr", 0, "ptr", 0)
    If @error Then Return SetError(2, @error, 0)
    ConsoleWrite(DllStructGetData($tText, 1))
EndFunc