Fast string manipulation

[IronFine]

Hi everyone,

I'm currently working on a project that involves heavy string manipulation, where I am working with large datasets. There I need to match specific patterns, modify the matches, and then replace the changes back into the string.

I am already mixed the use of StringReplace and StringRegExpReplace, depending where either one is fater (determined that for most of the cases once). Also I am using the case-sensitivity flag/replace count where possible. I tried to also use CLR and calling a C# hoping that that would be faster, but with the overhead it is only faster in some cases.

Are there any other tipps that I can use?

Edited Saturday at 06:11 PM by IronFine

[Nine]

There is not much you can do using standard String* functions.  Most of them (when modifying the original string) makes a copy of the string.  With small strings, it is not a big deal.  But when it comes to large strings, it can be quite lengthy to execute multiple string operations over a single string.

One method would be to perform in place replacements.  That is quite easy if you replace N characters with some other N characters.  Also it depends where the strings are coming from and where they go (text files ?).  Do they have known formats they are following (like json or xml) ?

Best way to help you, would be for you to provide a real life example, so we can try to look into a solution tailored to your situation.

[jchd]

What @Nine said, but beware of this:

13 hours ago, Nine said:

That is quite easy if you replace N characters with some other N characters.

Not that easy.

Windows uses UTF16-LE internally, which means that a "character" (here I mean actually "one Unicode codepoint") can use either one or two 16-bit words. in RAM. Also, don't forget that AutoIt uses a subset of UTF16-LE named UCS-2, allowing full use of the Unicode BMP (Basic Multilingual Plane), i.e. codepoints U+0000 to U+FFFF. Hence, counting "characters" is not trivial.

Common "characters" in non-english languages like à, Ê, ù, ç, ñ, ... can be represented using different normalization forms, see https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization

Moretheless what we, humans, mean by "character" may require more than one Unicode codepoint. This is the difference between glyph (what is displayed by the font renderer) and codepoint (the actual Unicode content). For instance this string

$s = ChrW(0xD83D) & ChrW(0xDC68) & ChrW(0xD83C) & ChrW(0xDFFB) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC69) & ChrW(0xD83C) & ChrW(0xDFFF) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC66) & ChrW(0xD83C) & ChrW(0xDFFD)

displays as in a Unicode-aware environment.

And that is only speaking of valid representations: one may find out-of-spec representations in input strings, spanning representation of what we perceive as a "character" over an unbounded string of Unicode codepoints.

Edited yesterday at 09:07 AM by jchd