RegExp question

[mLipok]

I have such example:

#include <Array.au3>
_Example()
Func _Example()
    Local $sTest = _
            "1a" & @CRLF & _
            "1b" & @CRLF & _
            "2" & @CRLF & _
            "1c" & @CRLF & _
            "1d" & @CRLF & _
            "2" & @CRLF & _
            "1e" & @CRLF & _
            "1f" & @CRLF & _
            "1g" & @CRLF & _
            "2" & @CRLF & _
            ""
    Local $aResult = StringRegExp($sTest, '(?is)1.*?2', $STR_REGEXPARRAYGLOBALMATCH)
    ConsoleWrite(_ArrayToString($aResult, @CRLF & @CRLF & @CRLF) & @CRLF )
EndFunc   ;==>_Example

Could somebody be so nice and explain 2 things:

Why the result is:

 

1a
1b
2


1c
1d
2


1e
1f
1g
2

I expected to get the following results.

How to get:

1b
2


1d
2


1g
2

?

Edited Monday at 10:51 PM by mLipok

[argumentum]

Spoiler

This is a classic regex trap! Let’s break down exactly why your current pattern is over-matching, and then look at how to get your expected output.

1. Why you got the first result

Your current regex is: (?is)1.*?2

Here is what is happening under the hood:

• (?is): This enables case-insensitivity (i) and "dot-all" mode (s). Dot-all means the dot . matches everything, including newlines (@CRLF).

• 1: The engine looks for the first literal 1 it can find. In your string, that's the 1 in "1a".

• .*?: This tells the engine to match any characters lazily (as few as possible) until it hits the next token.

• 2: The engine stops matching as soon as it hits the first 2.

Because it starts at "1a" and stops at the first "2", your first match group captures everything from 1a down to the first 2. Then it repeats the process starting from 1c, and then 1e.

The lazy quantifier .*? doesn't mean "skip to the closest 1 before a 2"; it just means "stop at the very first 2 you see after you've already found a 1".

2. How to get your expected result

To get only the 1 that immediately precedes the 2, you need to tell the regex engine: "Match a 1, but do not allow any other 1s to exist between it and the trailing 2."

You can achieve this by replacing the lazy dot .*? with a negated character class or a negative lookahead.

The Solution: Using a Negative Lookahead

The cleanest way to handle this with multi-character lines and newlines is to ensure that while we scan ahead to find a 2, we never cross over another 1.

Here is the updated AutoIt code:

AutoIt

#include <Array.au3>
#include <StringConstants.au3> ; Required for $STR_REGEXPARRAYGLOBALMATCH

_Example()

Func _Example()
    Local $sTest = _
            "1a" & @CRLF & _
            "1b" & @CRLF & _
            "2" & @CRLF & _
            "1c" & @CRLF & _
            "1d" & @CRLF & _
            "2" & @CRLF & _
            "1e" & @CRLF & _
            "1f" & @CRLF & _
            "1g" & @CRLF & _
            "2" & @CRLF & _
			""
			
    ; The new pattern: matches '1', ensures no other '1' follows before the '2'
    Local $aResult = StringRegExp($sTest, '(?is)1(?:(?!1).)*?2', $STR_REGEXPARRAYGLOBALMATCH)
    
    ConsoleWrite(_ArrayToString($aResult, @CRLF & @CRLF & @CRLF) & @CRLF )
EndFunc   ;==>_Example

How the new pattern works: 1(?:(?!1).)*?2

• 1: Finds a literal 1 (e.g., the one in 1a).

• (?: ... )*?: A lazy non-capturing group that loops through characters one by one.

• (?!1).: This is the magic. Before matching the next character (.), it looks ahead to make sure it is not another 1. If it is another 1 (like when moving past 1a towards 1b), the match fails for 1a, forcing the engine to discard 1a and start the whole match over again at 1b.

• 2: Closes the match once it hits the 2.

..I don't know RegEx but if the answer is good, then is a good answer 👍

[pixelsearch]

5 hours ago, mLipok said:

Why the result is...

I think it's because the smallest match you get with the ungreedy "?" in pattern concerns only the "2" found in the pattern.

* With "?" you got 3 matches, each one starting with the first "1" found, ending with the closest "2" found (ungreedy stops at the closest "2" +++)

* Without "?" the match would start at first "1" found, ending at... the last "2" found (greedy !) so it would return only 1 big match (instead of 3) corresponding to the whole text.

5 hours ago, mLipok said:

I expected to get the following results. How to get...

I tried this pattern and it seems to work :

(?i)1[^1]*?2

It matches anything between "1" and "2" starting at the last "1" found before "2" . Also we don't need the "s" option anymore as there are no "." in the pattern

Result, the one you expected :

1b
2
1d
2
1g
2

Hope it helps

[MattyD]

Yeah @pixelsearch's explanation is is correct.

But to be fair I think the answer could vary depending on the behavior we're after.

• How shall we deal with an 11a, 12, 22, or 122 etc? Do the 1's and 2's "count" here?
• Do we only need to check for digits after a newline, or do we match anywhere in the string?
• Can text legally exist after a "1f" etc, but before a newline? Can we run into rogue digits there?

Just food for thought

Edited yesterday at 08:32 AM by MattyD

[pixelsearch]

4 hours ago, MattyD said:

Just food for thought

...makes us all greedy 
One thought concerning the ungreedy character "?" I added in the pattern (didn't want to add it at first, then added it after 2nd thought)

(?i)1[^1]*?2

Please note that the result would be same using mLipok's text, with or without the ungreedy "?" , but...
I preferred to add it in case of a text getting superfluous "2" (like explained in your concerns), for example a text ending like this :

...
1g
2
yyy2zzz

Now the "?" becomes crucial :

* With "?" (ungreedy), the 3rd group matches the following, which seems correct according to OP's indications.

1g
2

* Without "?" (greedy), the 3rd group matches the following, which doesn't seem correct :

1g
2
yyy2

Edited 20 hours ago by pixelsearch
typo

[jchd]

@mLipok You overcomplicate the required pattern.

#include <Array.au3>
    Local $sTest = _
            "1a" & @CRLF & _
            "1b" & @CRLF & _
            "2" & @CRLF & _
            "1c" & @CRLF & _
            "1d" & @CRLF & _
            "2" & @CRLF & _
            "1e" & @CRLF & _
            "1f" & @CRLF & _
            "1g" & @CRLF & _
            "2" & @CRLF & _
            ""
    Local $aResult = StringRegExp($sTest, '(?i)1.\R2', $STR_REGEXPARRAYGLOBALMATCH)
    ConsoleWrite(_ArrayToString($aResult, @CRLF & @CRLF & @CRLF) & @CRLF )

Using (?is) and .*? allows the engine to match at 1 until including a 2.

[genius257]

Why do we keep the (?i), when we don't need case insensitive matching?

My suggestion: 

(?m)^1\w\R2$

Of course this is only based on the limited testing data we have now.

You can always swing by https://regex101.com/, to test different patterns with the PCRE (Legacy) flavor selected, and compare time/steps for the matching. Even try the regex debugger feature, if you need to see how it matches step for step, and where your pattern might not match.

Edited 15 hours ago by genius257