Removing Doubles

[e2e4au]

I have a plain text file that contains 1 word per line.

This text file has a huge amount of words in it, some words appear more than

once. I have tried several times to create a script the will remove all doubles without success (very frustrating ) Could someone help PLEASE.

Sample text file to work with.

arcane

has

its

experts

and

its

devotees

And

every

group

of

these

appear

to

have

one

place

where

they

congregate

to

talk

discuss

exchange

their

information

and

impart

the

newest

gossip

Shipping

movements

in

the

eastern

Mediterranean

hardly

form

a

subject

on

which

doctorates

are

earned

but

in

that

area

they

do

form

a

subject

of

great

interest

arcane

has

its

experts

and

its

devotees

And

every

group

of

these

appear

to

have

one

place

where

they

congregate

to

talk

discuss

exchange

their

information

and

impart

the

newest

gossip

Shipping

movements

in

the

eastern

Mediterranean

hardly

form

[trids]

I would sort it first (with a DOS sort) .. and then process each line sequentially, skipping it if it's the same as the previous one.

HTH

Edit: spelling etc.

Edited June 17, 2005 by trids

[GaryFrost]

#include <Array.au3>

Dim $a_text
$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
While 1
    $found = 0
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    If IsArray($a_text) Then
        For $i = 0 To UBound($a_text) - 1
            If ($a_text[$i] == $line) Then
                $found = 1
                ExitLoop
            EndIf
        Next
    EndIf
    If (Not $found) Then
        If IsArray($a_text) Then
            ReDim $a_text[UBound($a_text) + 1]
        Else
            Dim $a_text[1]
        EndIf
        $a_text[UBound($a_text) - 1] = $line
    EndIf
WEnd

FileClose($file)
_ArrayDisplay($a_text, "Unique")