Sign in to follow this  
Followers 0
brodie28

removing all identical lines in a text file

6 posts in this topic

I am looking for a way to remove all identical lines from a text file. All I can find on the web are lame programs that need you to register for them to work...

I couldnt find any in the examples page and really dont feel like making one myself right now.

Does anyone know of one?

In case you dont understand what I mean... Say a text file was

dog

dog

cat

horse

dog

horse

cow

Then what would be returned would be

dog

cat

horse

cow

Same order, no duplicates.

Share this post


Link to post
Share on other sites



Hey...

Like this?? Where the test.txt is your example..

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @lf   
    EndIf
Wend

MsgBox(1,"Total",$total)

FileClose($file)

Greetz,

Neo


[center][font="Arial"]--- The Neo and Only --- [/font][font="Arial"]--Projects---[/font]Image to Text converterText to ASCII converter[/center]

Share this post


Link to post
Share on other sites

Hey...

Like this?? Where the test.txt is your example..

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @lf   
    EndIf
Wend

MsgBox(1,"Total",$total)

FileClose($file)

Greetz,

Neo

If the file is:

dog

cat

dog

horse

horse cow

cow

your code not work.

#include <File.au3>
$OPEN = FileOpenDialog("OPEN",@ScriptDir,"All (*.*)",1)
If Not @error Then
    $FILE = FileOpen($OPEN,0)
    $NEW_FILE = ""
    For $INDEX = 1 To _FileCountLines($OPEN)
        $LINE = FileReadLine($FILE,$INDEX)
        $SEARCH = 0
        For $FIND = 1 To $INDEX-1
            If FileReadLine($FILE,$FIND) = $LINE Then 
                $SEARCH = 1
                ExitLoop
            EndIf
        Next
        If $SEARCH = 0 Then $NEW_FILE &= $LINE & @CRLF
    Next
    $SAVE = FileSaveDialog("SAVE",@ScriptDir,"Text (*.TXT)")
    If Not @error Then
        $FILE = FileOpen($SAVE & ".TXT",2)
        FileWrite($FILE,$NEW_FILE)
        FileClose($FILE)
    EndIf
EndIf

When the words fail... music speaks

Share this post


Link to post
Share on other sites

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @CRLF 
    EndIf
Wend

;MsgBox(1,"Total",$total)

Fileclose($file)
FileOpen ( "test.txt", 2 )

FileWrite($file, $total)
FileClose($file)

That ended up doing the job... I could have thrown this together myself I was just feeling lazy I guess

Share this post


Link to post
Share on other sites

$file = FileOpen("test.txt", 0)

; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
Global $total = ""

While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    if not StringInStr($total,$line,0) >= 1 Then
        $total = $total & $line & @CRLF 
    EndIf
Wend

;MsgBox(1,"Total",$total)

Fileclose($file)
FileOpen ( "test.txt", 2 )

FileWrite($file, $total)
FileClose($file)

That ended up doing the job... I could have thrown this together myself I was just feeling lazy I guess

@brodie28

The same thing like NeoFoX.

@NeoFoX

That was just one example, I guess you do not want to compare 5 names of animals.

The script must be more generalized.


When the words fail... music speaks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0