removing all identical lines in a text file

[brodie28]

I am looking for a way to remove all identical lines from a text file. All I can find on the web are lame programs that need you to register for them to work...

I couldnt find any in the examples page and really dont feel like making one myself right now.

Does anyone know of one?

In case you dont understand what I mean... Say a text file was

dog

dog

cat

horse

dog

horse

cow

Then what would be returned would be

dog

cat

horse

cow

Same order, no duplicates.

[NeoFoX]

Hey...

Like this?? Where the test.txt is your example..

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @lf   
    EndIf
Wend

MsgBox(1,"Total",$total)

FileClose($file)

Greetz,

Neo

[Andreik]

Hey...

Like this?? Where the test.txt is your example..

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @lf   
    EndIf
Wend

MsgBox(1,"Total",$total)

FileClose($file)

Greetz,

Neo

If the file is:

dog

cat

dog

horse

horse cow

cow

your code not work.

#include <File.au3>
$OPEN = FileOpenDialog("OPEN",@ScriptDir,"All (*.*)",1)
If Not @error Then
    $FILE = FileOpen($OPEN,0)
    $NEW_FILE = ""
    For $INDEX = 1 To _FileCountLines($OPEN)
        $LINE = FileReadLine($FILE,$INDEX)
        $SEARCH = 0
        For $FIND = 1 To $INDEX-1
            If FileReadLine($FILE,$FIND) = $LINE Then 
                $SEARCH = 1
                ExitLoop
            EndIf
        Next
        If $SEARCH = 0 Then $NEW_FILE &= $LINE & @CRLF
    Next
    $SAVE = FileSaveDialog("SAVE",@ScriptDir,"Text (*.TXT)")
    If Not @error Then
        $FILE = FileOpen($SAVE & ".TXT",2)
        FileWrite($FILE,$NEW_FILE)
        FileClose($FILE)
    EndIf
EndIf

[brodie28]

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @CRLF 
    EndIf
Wend

;MsgBox(1,"Total",$total)

Fileclose($file)
FileOpen ( "test.txt", 2 )

FileWrite($file, $total)
FileClose($file)

That ended up doing the job... I could have thrown this together myself I was just feeling lazy I guess

[NeoFoX]

That wasn't in the file so not included >_< he can try to do something himself...

Greetz

[Andreik]

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @CRLF 
    EndIf
Wend

;MsgBox(1,"Total",$total)

Fileclose($file)
FileOpen ( "test.txt", 2 )

FileWrite($file, $total)
FileClose($file)

That ended up doing the job... I could have thrown this together myself I was just feeling lazy I guess

@brodie28

The same thing like NeoFoX.

@NeoFoX

That was just one example, I guess you do not want to compare 5 names of animals.

The script must be more generalized.