StringRegExp to recognize spaces, numbers and special characters

[Carm01]

All,

 

I have a part of my code below, and what this does is take a specifically formatted text file, search for specific words or phrases to include or exclude( exclude code ).

 

The include and exclude are handled by separate txt files; one for each. I was able to finally understand how to use case insensitive, but I am having difficulty after several hours failing and looking up examples of how to include multiple words or phrases including spaces.

 

There seem to be a plethora of examples stripping spaces.

 

I need the spaces as this is what i am searching for 

 

For example if my include file has:

 

red

sugar

 

My exclude file has:

 

sugar coated

 

The output lines will be sentences ( or file lines ) with the words red, sugar contained in the line as included but if I am excluding the phrase 'sugar coated' it appears to only see the word 'sugar' and capture the entire content and display it ignoring the phrase 'sugar coated', thus excluding the entire line.

 

That is a perfect example, and here is a piece of code i threw together as a subroutine for the exclude portion:

Func verify1()
For $k = 1 To $d
$line66 = "(?i) " & FileReadLine($file6, $k)
If StringRegExp($line, $line66) = 1 Then
$Sno = 1
ExitLoop
EndIf
Next
$Sno = 0
EndFunc   ;==>verify1

the lines specifically are:

$line66 = "(?i) " & FileReadLine($file6, $k)
If StringRegExp($line, $line66) = 1 Then

 

This is what I am using to detect the exclude text. I have something similar for the include text. 

 

I need to understand better on how to allow for phrases, numbers, spaces and any special character regardless of how many words or spaces exist. This is not as straightforward as i thought

 

Thank You in Advance

[Jury]

So, for example, you want to exclude the phrase  'sugar coated' from all lines in the file then you want to find all lines with  red or sugar contained in any line throughout the file?

[Kyan]

Is somewhat hard to understand what you're trying to archive, you just want to capture phrases with words present in your include.txt? And if there's some word/sentence of exclude.txt you simply ignore them?

I would say with regex you cannot do it in that way, for example, your include file will match all sugar phrases, even if coated is ahead, 

You will need to put all line matches of include, and then subtract the matchs of exclude

Here's one only with one regex that looks for 'sugar' without coated being ahead, or for the word 'red'...

Local $hFile, $sLine, $l=1
$hFile = FileOpen(@DesktopDir&"\MyText.txt")
$xIncludeExp = 'sugar\s(?!coated)|red'
While 1
    $sLine = FileReadLine($hFile,$l)
    If @error Then ExitLoop
    If StringRegExp($sLine,'(?i)'&$xIncludeExp) = 1 Then ConsoleWrite($l&" OK"&@LF)
    $l+=1
WEnd
FileClose($hFile)
Exit

if you want to export matched lines, just replace If StringRegExp($sLine,'(?i)'&$xIncludeExp) = 1 Then ConsoleWrite($l&" OK"&@LF) with If StringRegExp($sLine,'(?i)'&$xIncludeExp) = 1 Then FileWriteLine(@DesktopDir&"YourCapturedPhrases.txt",$sLine)

[jguinch]

Carm01, I don't really understand what you want to do...

Could you post some small examples of include/exclude files + sentences and the result you expect ?

It will be clearer for everyone

[kylomas]

Carm01,

Something like this?

local $exclude, $text, $pattern = '(?i)'

; simulate text file
$text &= 'I have a sugar coated brown fox.' & @CRLF
$text &= 'Now I want to grill it.  No, not ask questions of it, but cook it.' & @LF
$text &= 'Does anyone have a good brown fox^ recipe?'

;simulate exclude file
$exclude &= 'sugar' & @lf
$exclude &= 'grill' & @crlf
$exclude &= 'fox^ recipe' & @crlf

; handle SRE special chars, if needed
$exclude = stringregexpreplace($exclude,'[\^]','\\^')

local $aExclude = stringregexp($exclude,'(.*)\R',3)

for $1 = 0 to ubound($aExclude) - 1
    $pattern &= $1 < ubound($aExclude) - 1 ? $aExclude[$1] & ' |' : ' ' & $aExclude[$1]
Next

ConsoleWrite('' & @CRLF)
ConsoleWrite('! --- Pattern = [' & $pattern & ']' & @CRLF)
ConsoleWrite('' & @CRLF)
ConsoleWrite(stringregexpreplace($text,$pattern,'') & @CRLF)
ConsoleWrite('' & @CRLF)

kylomas

edit: added SRE char handling for "^"

Edited October 7, 2014 by kylomas

[jchd]

kylomas,

You can put Q...E at good use to ignore special characters or subpatterns. Of course you could also guard against E occurring by itself inside the search word, but that may be unlikely enough to ignore it.

[kylomas]

@jchd - Thanks...

[Carm01]

Carm01,

Something like this?

local $exclude, $text, $pattern = '(?i)'

; simulate text file
$text &= 'I have a sugar coated brown fox.' & @CRLF
$text &= 'Now I want to grill it.  No, not ask questions of it, but cook it.' & @LF
$text &= 'Does anyone have a good brown fox^ recipe?'

;simulate exclude file
$exclude &= 'sugar' & @lf
$exclude &= 'grill' & @crlf
$exclude &= 'fox^ recipe' & @crlf

; handle SRE special chars, if needed
$exclude = stringregexpreplace($exclude,'[\^]','\\^')

local $aExclude = stringregexp($exclude,'(.*)\R',3)

for $1 = 0 to ubound($aExclude) - 1
    $pattern &= $1 < ubound($aExclude) - 1 ? $aExclude[$1] & ' |' : ' ' & $aExclude[$1]
Next

ConsoleWrite('' & @CRLF)
ConsoleWrite('! --- Pattern = [' & $pattern & ']' & @CRLF)
ConsoleWrite('' & @CRLF)
ConsoleWrite(stringregexpreplace($text,$pattern,'') & @CRLF)
ConsoleWrite('' & @CRLF)

kylomas

edit: added SRE char handling for "^"

This is closer, but....

Instead of omitting the word out of the sentence, the entire sentence is ignored, thus nothing written

IF the sentence read 'I have a sugar coated brown fox.' then that entire sentence would be passed by and no lines written

If the sentence read 'I have a tar coated brown fox.' then the entire sentence would have been captured and exported to a new file and line.

I also have an include list as to what i am searching for.

If the include list has the word/phrase  'tar coated' it would find sentences with those words or phrases , BUT if it had anything from the exclude list, nothing would be written"

IF the sentence was: 

' I have a sugar coated brown fox and a tar coated brown fox' this would result in nothing being written as a word or phrase in the sentence is on the ban list. 

Should have included that part ..

caps and lower case should be treated equally.

Sorry for not making that clear. , and thank you all for your assistance with this 

I also included some example txt files, the raw text and the results. I am really looking how to use this 

StringRegExp  to get it to recognize caps, phrases, and lower case treated equally, include numbers, symbols, etc...

exclude.txt

include.txt

rawtext.txt

results.txt

Edited October 9, 2014 by Carm01

[jchd]

Carm01,

I've hard time figuring out what you want exactly. The keyword here is "exactly". So let me ask some questions to make things clearer:

A/ Should input be treated literally? Input: "I'm digging for diamond, silver           and ----   gold" Should that match "silver and gold" from include list?

B/ Should include/exclude lists be treated literally? Input: "I'm digging for diamond, silver and gold" Should that match "silver   */and/*   gold" from include list?

C/ What is the typical size (in characters) of the include/exclude lists?

If the answers are:

A/ Yes. No.

B/ Yes. No.

C/ Small enough/small enough

then this should work for you:

#include <Array.au3>

Local $aText = FileReadToArray("include.txt")
Local $sInclude = "(?=.*(?:\Q" & _ArrayToString($aText, "\E|\Q") & "\E))"
$aText = FileReadToArray("exclude.txt")
Local $sExclude = "(?!.*(?:\Q" & _ArrayToString($aText, "\E|\Q") & "\E))"
Local $aText = 0
Local $sInput = FileRead("rawtext.txt")
Local $aResult = StringRegExp($sInput, "(?im)" & $sExclude & $sInclude & "^.*(?:$|\R)", 3)
_ArrayDisplay($aResult)

Edited October 9, 2014 by jchd

[Carm01]

 

Carm01,

I've hard time figuring out what you want exactly. The keyword here is "exactly". So let me ask some questions to make things clearer:

A/ Should input be treated literally? Input: "I'm digging for diamond, silver           and ----   gold" Should that match "silver and gold" from include list?

B/ Should include/exclude lists be treated literally? Input: "I'm digging for diamond, silver and gold" Should that match "silver   */and/*   gold" from include list?

C/ What is the typical size (in characters) of the include/exclude lists?

If the answers are:

A/ Yes. No.

B/ Yes. No.

C/ Small enough/small enough

then this should work for you:

#include <Array.au3>

Local $aText = FileReadToArray("include.txt")
Local $sInclude = "(?=.*(?:\Q" & _ArrayToString($aText, "\E|\Q") & "\E))"
$aText = FileReadToArray("exclude.txt")
Local $sExclude = "(?!.*(?:\Q" & _ArrayToString($aText, "\E|\Q") & "\E))"
Local $aText = 0
Local $sInput = FileRead("rawtext.txt")
Local $aResult = StringRegExp($sInput, "(?im)" & $sExclude & $sInclude & "^.*(?:$|\R)", 3)
_ArrayDisplay($aResult)

I like how easily this was done. 100% different from my method using file line reads, then file line writes, which was easy for me, but this little thing is stupid fast. 

technically this is what i am looking for for the searching part, but would like to output this to a file instead of an array box.

[guinness]

Which is trivial to do. Look at a for to loop in the help file and learn about arrays.

[mikell]

Or look at how to File Write From an Array in UDFs/File Management 

[jchd]

Or reverse the idea and use StringRegExpReplace to void any line which does NOT satisfy the requirements.

Exercise left to the readers.

[Carm01]

Or look at how to File Write From an Array in UDFs/File Management 

at the end

 

;_ArrayDisplay($aResult)

Local $sFilePath = @ScriptDir & "Examples.txt"

 

_FileWriteFromArray($sFilePath, $aResult, 1)

 

I guess it was too lat/early for my brain...