_WordCount - Find the total number of words in a document.

[guinness]

I came up with this snippet but feel it can be improved. It will count the number of words in a document but it's a 'clumsy' regular expression, so any input would be appreciated.

ConsoleWrite(_WordCount(@ScriptFullPath) & ' words' & @CRLF)

Func _WordCount($sFilePath)
    Local $aArray = StringRegExp(FileRead($sFilePath), '([S*]+)', 3)
    Return UBound($aArray)
EndFunc   ;==>_WordCount

[czardas]

I think you should remove FileRead and input a string instead. I don't think it gets any simpler than that.

I'm working on a project like this ATM, but I am currently trying a heuristic approach to word recognition. It's kind of a fuzzy logic function. I don't know how good it will turn out to be.

[guinness]

Wait I should use w instead of S. You gave me the inspiration and I will create a better example.

[guinness]

I had a dumb moment when I created this today, the SRE above is terrible as a word (in my humble opinion) should be made of A-Z, a-z, 0-9 and _ with the addition of - too. What about this? As I said above, I am constantly looking around the forums searching for ideas and your 'word wrap' thread spurred me on.

Local $sWordString = 'This is a simple example to count the number of words' & @CRLF & _
        'in a text document or string. For example this-is-considered-as-one-word in Word 2010.' & @CRLF & @CRLF & _
        'So why not have a go at creating a better regular expression.'
ConsoleWrite(_WordCount($sWordString) & ' words in the following document:' & @CRLF & $sWordString & @CRLF)

Func _WordCount($sString)
    Local $aArray = StringRegExp($sString, '[w-]+', 3)
    Return SetError(@error, @extended, UBound($aArray))
EndFunc   ;==>_WordCount

[Malkey]

The RE pattern '([s*]+)' is capturing all the non-whitespace characters and any asterisks characters one or many times. But an asterisk is a non-whitespace character.

This example correctly returned 88,685 words from a 925KB text file.

ConsoleWrite(_WordCount(@TempDir & 'Words.txt') & @CRLF)

Func _WordCount($sFilePath)
StringRegExpReplace(FileRead($sFilePath), 'S+', "")
Return @extended
EndFunc ;==>_WordCount

Edit:

And this example will slowly check the above result.

#include <Word.au3>

$oWordApp = _WordCreate(@TempDir & 'Words.txt', 0, 0)
$oDoc = _WordDocGetCollection($oWordApp, 0)
ConsoleWrite("Title: " & _WordDocPropertyGet($oDoc, "words") & @CRLF)
_WordQuit($oWordApp)

Edited July 31, 2012 by Malkey

[guinness]

Cheers Malkey, like I said it was bad code.

In your opinion which is the best approach mine or yours? As what I find most confusing is the fact there are many expressions to achieving the same output, how do I know which is the optimised version? Or does this come with practice? Thanks.

[czardas]

I had a dumb moment when I created this today, the SRE above is terrible as a word (in my humble opinion) should be made of A-Z, a-z, 0-9 and _ with the addition of - too. What about this? As I said above, I am constantly looking around the forums searching for ideas and your 'word wrap' thread spurred me on.

I'm not sure if this is really enough. I think the extended set of latin letters available in ascii should also be included, at least. You also need apostrophe. I'm not sure if apostrophe should be allowed to repeat or not.

Here's the core set of letters I intend to find a match for.

$sAlphaEx = ""
For $i = 65 To 255
    If $i <> 215 And $i <> 247 Then $sAlphaEx &= Chr($i)
    If $i = 90 Then $i += 6
    If $i = 122 Then $i += 69
Next

MsgBox(0, "", $sAlphaEx)

Add to this the option to include other non alphabet characters such as underscore.

Edit

The code above misses 7 letters from the extended ascii set. No's 138, 140, 142, 154, 156, 158 and159 (oops)

Edited August 1, 2012 by czardas

[Spiff59]

It's interesting that an old-fashioned (and possibly flawed?) approach seems to be nearly twice as fast as SRE (at least with small amounts of input):

Local $sWordString = 'This is a simple example to count the number of words' & @CRLF & _
        'in a text document or string. For example this-is-considered-as-one-word in Word 2010.' & @CRLF & @CRLF & _
        'So why not have a go at creating a better regular expression.'

$timer = TimerInit()
For $x = 1 to 1000
$cnt = _WordCount($sWordString)
Next
$timer = Round(TimerDiff($timer) / 1000, 2)
ConsoleWrite($cnt & ' words in the following document:' & @CRLF & $sWordString & @CRLF)
ConsoleWrite("time = " & $timer & @CRLF & @CRLF)

$timer = TimerInit()
For $x = 1 to 1000
$cnt = _WordCount2($sWordString)
Next
$timer = Round(TimerDiff($timer) / 1000, 2)
ConsoleWrite($cnt & ' words in the following document:' & @CRLF & $sWordString & @CRLF)
ConsoleWrite("time = " & $timer & @CRLF & @CRLF)

Func _WordCount($sString)
    Local $aArray = StringRegExp($sString, '[w-]+', 3)
    Return SetError(@error, @extended, UBound($aArray))
EndFunc   ;==>_WordCount

Func _WordCount2($sString)
    $sString = StringStripWS($sString, 7)
    Local $aArray = StringSplit($sString, " " & @CRLF)
    Return $aArray[0]
EndFunc

[guinness]

Thanks Spiff59. It's certainly the most obvious solution despite having some possible flaws as you said. I personally took up the challenge to learn regular expressions.

[czardas]

After a day of trying to find a comprehensive list of hyphenated words in current English, I have decided to abandon the database approach. Here's what I think is a good approach. Using a flag it should be possible to make a flexible word count function that will give the user more control. When MS word tells me that '''''''''''''' '''''''''''''''' is two words, I get suspicious.

No code - just match criteria: (other possible combinations can also be indicated by the flag)

English Alphabet + apostrophe + hyphen - default flag

Extended latin alphabet + apostrophe + hyphen

English Alphabet + apostrophe + hyphen + numbers + underscore

Extended latin alphabet + apostrophe + hyphen + numbers + underscore

Any non white space - overriding flag

I hope this provides you with further inspiration. Any thoughts or suggestions?

'

Edited August 1, 2012 by czardas