Please review doc file for RegExps

[jchd]

You all know that AutoIt now links to the current PCRE16 v8.33 library (no more UTF8 jig) and includes UCP (Unicode Character Properties) support.

This has significant impact over a number of metacharacters and features. The documentation of StringRegExp needs to reflect these changes. In my way I also found a number of errors or vague phrasing which can mislead beginners.

Please review the revised doc file and post your comments pointing out my own typos, thinkos, errors and omissions.

Old doc file from 3.3.9.15 beta

Proposed new doc file

Summary of changes

PCRE v8.33 pattern reference document

[guinness]

Wow, just wow. It's gets my vote at least. I can commit what you have there if you would like?

[guinness]

My only concern is the PCRE html file.

[Jon]

I'll add one somewhere in the HTML folder and upload one to the online docs place as well

[jchd]

I've invited for review known regexp power user Malkey and AZJIO who found the doc perfectible. I'd love to hear from a significant number of users/MVPs/Devs as well.

About the reference doc pcrepattern.html I suggest we insert the version as well so we can match it with subsequent or old releases and the link from the help file.

[guinness]

Most tend to review once it has been added to the help file I have found in my experience being around here.

[James]

Awesome! I'll have a proper look through later on - but from my quick scan through everything looked good.

[AZJIO]

I'll try to translate into English that as I formulated a description of the meta-characters

StringRegExp (it is transferred to Google)

A - the absolute beginning of the text, doesn't depend on a flag "(?m)" and match only 1 time

z - the absolute end of the text, doesn't depend on a flag "(?m)" and match only 1 time

b - the boundary between characters, one of which matches W and the other matches w

c - ... For example, cA = 001, cZ = 032, cM = r = 015

n - the link to the found group in a pattern. For example, 1, 2, 3

For the class of [... ] should be a separate table

<i>Flag = 1 or 2 :</i>

if <i>flag</i> = 1 or 2, @error accepts the following values:

1 Return array of matches.
2 Return array of matches including the full match (Perl / PHP style).

1 Search to the first match, and returns an array of groups, and in the absence of group-element array with a complete coincidence template (see notes).
 
2 Search to the first match, and returns an array containing the items in the first full match with the template, the following elements - the coincidence of these groups (style Perl / PHP).

Edited August 9, 2013 by AZJIO

[jchd]

c - ... For example, cA = 001, cZ = 032, cM = r = 015

 

n - the link to the found group in a pattern. For example, 1, 2, 3

Thanks AZJIO.

In fact I started dispatching regexp elements in more logical tables and I only now discover that's what you did in your russian help.

There was another big mistake in the Group description:

(ab)+ will match "ab" or "abab", but not "aba"

changed to

(ab)+c will match "abc" or "ababc", but not "abac"

I'd love to remove documentation of both prehistory remnants cM and octal but then it could mislead unsuspecting users.

[trancexx]

There was another big mistake in the Group description:

(ab)+ will match "ab" or "abab", but not "aba"

changed to

(ab)+c will match "abc" or "ababc", but not "abac"

What exactly was "big mistake" that required changing?

By previous description this:

$sString = "aba"
$sOut = StringRegExpReplace($sString, "(ab)+", "xx")

ConsoleWrite($sOut & @CRLF)

...should print xxa. And really if I run that script I get exactly that.

That's a bit anal change if you want my opinion considering the meaning was clear enough (and correctly described).

[Malkey]

Praise

@jchd

Well done compiling all that information together.

 

Personal comments

I hadn't come across "K" before. It will be useful.

It wasn't until I was playing around with the Unicode property shortcuts, p{...}, that I realised this help document is for the AutoIt beta version and the PCRE configure options are not set the same as on the current AutoIt release version.

I couldn't find a definitive list of regular expression Unicode property codes on the web, at least a list with Xan in it.


Correction

In order to avoid confusion with the "it":-

[:space:]
    ASCII white space (not quite the same as s: it includes VT, Chr(11)) (same as [fnrtv ] or p{Xps})

change to something like:-

[:space:]
    ASCII white space (same as [fnrtv ] or p{Xps}) (not quite the same as s because s does not match VT, Chr(11))

Comment: It looks like the current help file wrongly says s matches the Vertical Tab.


Possible Additions

g    Back-references 1, g1, and g{1} are all identical. In the pattern'(a)(©(d)g{-1}' the sequence g{-1} is a reference to the most recently started capturing subpattern before g, that is equivalent to 4, matching the letter "d". Similarly, g{-2} would be equivalent to 3, matching the letter "c".

k{name}     Back reference to a named subpattern.  Perl syntax k<name> or k'name', or  Python syntax (?P=name), and, Perl 5.10's unified back reference syntax, in which g can be used for both numeric and named references, is also supported.


Query

Is there any chance of the AutoIt user being able to optionally use the PCRE callout feature one day?
 

Playtime examples

; All commented results are from running AutoIt Beta Version 3.3.9.15

Local $l = "AutoIt 3.0 - Version 3.0 ¼ " & Chr(128) & ChrW(255) & Chr(11)
ConsoleWrite("0 " & $l & @LF) ;                            --> AutoIt 3.0 - Version 3.0 ¼ €ÿ

Local $xl = StringRegExpReplace($l, '\p{N}', '@') ;        --> AutoIt @.@ - Version @.@ @ €ÿ
ConsoleWrite("1 " & $xl & @LF)

Local $x2 = StringRegExpReplace($l, '\p{Xan}', '@') ;      --> @@@@@@ @.@ - @@@@@@@ @.@ @ €@
ConsoleWrite("2 " & $x2 & @LF)

Local $x3 = StringRegExpReplace($l, '\p{Nd}', '@') ;       --> AutoIt @.@ - Version @.@ ¼ €ÿ
ConsoleWrite("3 " & $x3 & @LF)

Local $x4 = StringRegExpReplace($l, 'AutoIt \K3.0', '@') ; --> AutoIt @ - Version 3.0 ¼ €ÿ
ConsoleWrite("4 " & $x4 & @LF)

Local $x5 = StringRegExpReplace($l, '[[:space:]]', '@') ;  --> AutoIt@3.0@-@Version@3.0@¼@€ÿ@
ConsoleWrite("5 " & $x5 & @LF)

Local $x6 = StringRegExpReplace($l, '\s', '@') ;           --> AutoIt@3.0@-@Version@3.0@¼@€ÿ
ConsoleWrite("6 " & $x6 & @LF)

Local $li = ">abcdc<"
Local $x7 = StringRegExpReplace($li, '(a)(b)(c)(d)\g{-2}', '@') ;      --> >@<
ConsoleWrite("7 " & $x7 & @LF)

Local $l2 = "rah rah RAH RAH rah RAH"
Local $x8 = StringRegExpReplace($l2, '(?<p1>(?i)rah)\s+\k<p1>', '@') ; --> @ @ rah RAH
ConsoleWrite("8 " & $x8 & @LF)

Local $x9 = StringRegExpReplace($l2, "(?'p1'(?i)rah)\s+\g{p1}", '@') ; --> @ @ rah RAH
ConsoleWrite("9 " & $x9 & @LF)

[jchd]

trancexx,

"(ab)+" does match "aba", just opposite to what was said.

Malkey,

Yeah, I'm undergoing more changes.

[Ascend4nt]

Hmm, a couple of new escape sequences I haven't seen used before. Very nice, jchd.

The only thing I might say could use some clarification is that the 'simple assertions'  (which the PCRE docs list as b, B, A, G, Z, z, ^ and $) don't actually consume characters and act sort of like look-ahead or look-behind assertions. or anchors.. or something

[jchd]

Correct, I'm trying to distiguish assertions and actual character matches. Descriptions say "Matches at this or that" vs "Matches this or that".

I've already experimented with several tables breakout and I'm looking to clarify further. Unfortunately regexps elements are not very easy to describe in isolation due to interdependance of many things. Beyond anything I'd like the page to be more beginner-friendly yet more complete and accurate than the old one.

Also introduction of UCP implies a more detailed description, but it is a huge step forward for people manipulating non-english texts.

[Ascend4nt]

Correct, I'm trying to distiguish assertions and actual character matches. Descriptions say "Matches at this or that" vs "Matches this or that".

Yeah, that makes sense to me now but when I was first learning regular expressions they both looked the same. And I really didn't have a concept of anchors or consuming characters etc. Still, at least its something.

[guinness]

These are correct...there was a mistake in the StringRegExp.

(?= ... )
    Positive look ahead.
(?<= ... )
    Positive look behind.
(?! ... )
    Negative look ahead.
(?<! ... )
    Negative look behind.

Edited August 14, 2013 by guinness

[jchd]

I've seen that. They now appear in a separate table. Everything was mixed and it was hard for beginners to distiguish matching specifiers, assertions, grouping, options, ... I'm finishing a rewrite as separate tables with a bit more explanation of what is what at the expense of more real estate.

It won't replace the official reference documentation but I hope that the new page will reveal less cryptic to beginners and easier to reference for people already wet with regexps.

[guinness]

It won't replace the official reference documentation but I hope that the new page will reveal less cryptic to beginners and easier to reference for people already wet with regexps.

Most tutorials I have seen only touch on the basics, so what you're proposing is quite a bit already.

[jchd]

I see what you mean. I don't aim at a tutorial but rather a decently complete summary for a little more than basic use, clear enough for a wide range of regexp "savviness". While exposing regexp semantics is a bit intricate due to their notation collisions, we can focus on PCRE exclusively.

I'm not going to even mention named patterns, subroutines, recursion, backtracking control, ...

Edited August 14, 2013 by jchd

[guinness]

Incorrect usage of the term tutorial, my mistake. I should have perhaps used cheatsheet!