Fetch english words only

[GEOSoft]

News flash! WinRAR is based on 7z.

My main complaint is people using it when there is native zip support in Windows. For me it doesn't matter as far as extraction goes because I use Universal Extractor which covers almost all of them including setup files.

[Shaarad]

But finding each separate word in the entire 'wordlist.txt' will be a bit slower process ??

[jchd]

Using an SQLite database makes lookup a very fast process, the issue isn't there.

Yet many posters still stick on the OP's _first_ post wording. The issue is that his/her problem is rather ill-posed.

I did a little experiment and compute (using SQLite of course) the intersection of a single french words list having 336531 entries with the union of GEOsoft english wordlist recently posted in this thread and a couple of other english word lists found with a quick Google search. This union has 121942 entries.

The intersection of both wordlist has 11521 entries. With that many words it's not difficult to build meaningful french sentences out of it.

Now such frenglish sentences should they qualify as probably semantically incorrect english or semantically correct french?

The argument goes as well the other way round.

Hence a french sentence may be accepted as pure english by a simple-minded english wordlist filter.

If you rule it the other way and remove french words instead, you also have a problem because of the non-empty intersection of both wordlists.

This stands true for any couple of languages close enough to have a significant number of common words.

If you want to sort out sentence with reasonable language criterion, you have to rely on semantic analysis and that is much, much harder.

Hopefully, the OP found that the english and foreign sentences were regularly intermixed in his file, so the issue is over.

[GEOSoft]

Run a time check against that to see but for sure your method is much slower.

You are using FileRead() once for each word. The code I posted uses FileRead() once in total to load the contents to a variable.

A loop is a necessity for this code but loops are slow and you nested 2 of them to make it even slower

If you remove the 3 lines used for diplaying the array from my code you are left with 16 lines. Removing the blank lines from yours leaves you with 29 lines wich is no big deal but then add in the line count for string.au3 and file.au3 and you are up around the 1100 line mark.

Now stop and think about it. Which will be faster?

I also have a rule that I apply to UDF files I'm going to #Include. Unless it's only for testing purposes I will only #Include a file if I am going to use at least 20% of the functions in that file and even at that I'm more apt to break out the functions I need and put them in a separate file to be #Included.

File.au3 is 760 lines and String.au3 is 420 lines. All that for the sake of 2 functions?

Sorry to disappoint but this is one of those jobs that makes Regular Expressions worth learning.

[GEOSoft]

jchd

There is no doubt in my mind that a database would be a faster check here. It becomes more complex when you use a ridiculous input string like I did; which is a combination of French, German and English. That of course built a good stress test for the over all functionality of the script but would not likely ever be encountered.

By the way, I would be interested in seeing those other word lists you found. I thought I had already covered the bases for that.

I have a script already that will cheack word lists against the one I have and then after the words have been validated it will add them to the lsit and sort it. Now there is a couple of slow functions and I really should do a bit more of a re-write to speed them up more than what I have. I think the last test I did was something like 6 or 8 seconds to unique sort my word lists and build the word list which I posted. And no; I didn't sit here with an open dictionary and manually enter all the words in it. That is for someone else to do.

Edited May 22, 2011 by GEOSoft

[jchd]

Here's the source I used. I merged it with your list using "insert into wordsEN select * from geowords" with an ignore clause on primary key (= the word itself).

Frenglish words found with "select * from wordsEN intersect select * from wordsFR". You see what I mean.

[GEOSoft]

I thank you and I shall verify and add those words at some point today.

[GEOSoft]

Just an FYI

I finally got a chance to combine the word lists and generate a new file. I used my existing (88710 words) and merged it with the NA English file from Zyzzyva (10109581 words) for a total of 198291 words then ran it through my script to create a new list of 121928 unique and sorted words. Total elapsed time from beginning of merge to completion was a long 19.37 seconds but worth the time to get it right. Required time for the operation I normally use that script for is just over 4 seconds but in this case we are talking about 2 huge lists and it takes time to create the unique, sorted array and then write the file.

I'm not going to change the attached file but at some point I will post a link to a copy of the new file for those that are interested in it.

[Shaarad]

Does anybody know where can I get a list of all english words (with their meanings) as a txt / pdf / doc file ?

[GEOSoft]

I have a complete list but you will have to do a dictionary search to get the definitions.

EDIT: The list does NOT contain words that are hyphenated and it does not contain contractions.

Edited June 4, 2011 by GEOSoft

[Shaarad]

okay, no problem...please give me the link of your list file...

and if you have a good dictionary software (freeware) please tell me the name so that I'll AutoIt and then get the required file...

[GEOSoft]

I've been meaning to upload that list for a while now and finally got around to doing so.

Here is the direct link to the file

http://dundats.mvps.org/autoit/files/wordlist.zip

You can also find it by going to my web site and in the left menu just click Miscellaneous and you will see it there.

EDIT:

I also have a couple of dictionary functions I use to validate the words. They could easily be modified to download the definitions or even just the grammar part for the word. If anyone needs that I'll see if I can make them a bit more generic and add them to my Grammar UDF.

Edited June 4, 2011 by GEOSoft

[Shaarad]

thanks a lot !

Now what I am searching is a freeware dictionary software where I can find all the words in your list.

[GEOSoft]

You can use the online ones.

Primarily, I use Oxford, Merriam Webster and Google Dictionary but there are several more.

I have a small database here the someone sent me which contains words and their grammar parts but it's bt no means conplete and I would really like to expand on it to add more words as well as possibly definitions but that would be like rebuilding Merriam Websters database and overall it would be a nightmare since Grammar rules do change over a period of time and in particular when it comes to compound words. Contractions are no joy to work with either and that is one reason they are not in that word list. Of course the other reason is I originally started the list as an easy way for my wife to fix her Scrabble dictionary which breaks on a regular basis.

EDIT: I should add this.

If you are working with English grammar then there is a UDF on my site that has several valuable functions in it.

Left Menu >> Code >> My Extra UDFs >> Grammar.au3

Edited June 4, 2011 by GEOSoft

[Shaarad]

nope. I just want simple meanings of the words. I don't want synonyms, grammar etc

[BrewManNH]

You might want to look at WordWeb for a dictionary program for your definitions.

[Shaarad]

Thanks BrewManNH,

I think that was I was looking for !

[GEOSoft]

@BrewManNH

That's also a good one. I haven't been there for quite a while and forgot about it. New function coming up.