Find Duplicate Lines in a Text File?

[logcomptechs]

Is their a way to read a text file by line and delete any duplicate lines? I was looking at _FileReadtoArray, but wasn't sure on how I would go about comparing, because I do not want to delete the original line, just duplicates of that line. Any help would be much appreciated, thanks!

[TerarinK]

How do you want it got many functions, one uses the sort ability of the arrays and then does the find duplications. However got this one that find each one and searches the whole arrays then put it in a new array. Yes they both have there appeals but what do you need it for, something fast or you have to keep the array in the same order, ie....

[bogQ]

Is their a way to read a text file by line and delete any duplicate lines? I was looking at _FileReadtoArray, but wasn't sure on how I would go about comparing, because I do not want to delete the original line, just duplicates of that line. Any help would be much appreciated, thanks!

i think that in this case you can use _ArraySearch and if find on other palce diffrent then reading position (Index of array to start searching at) to delite.

[logcomptechs]

How do you want it got many functions, one uses the sort ability of the arrays and then does the find duplications. However got this one that find each one and searches the whole arrays then put it in a new array. Yes they both have there appeals but what do you need it for, something fast or you have to keep the array in the same order, ie....

No need to keep a order or anything like that. Just need something to find duplicated lines and delete. I am going to try _ArraySearch but not 100% sure how to still search for duplicates.

[Xenobiologist]

Hi,

FileReadToArray, then ArrayUnique FileWriteFromArray done

Mega

[SpookMeister]

a brute force method:

#include <File.au3>
$OldPath = @ScriptDir & "\oldfile.txt"
$NewPath = @ScriptDir & "\newfile.txt"
Dim $a_NewFile[1]
_FileReadToArray($OldPath, $a_OldFile)
; loop through the array of the orriginal file
For $i = 1 To $a_OldFile[0]
    $line = $a_OldFile[$i]
    $count = 0
    ; check to see if the current line has existed in the old file
    For $x = 1 To $i
        If $line = $a_OldFile[$x] Then $count += 1
    Next
    ; if it only shows up once, then add it to a new array
    If $count = 1 Then
        $a_NewFile[0] = UBound($a_NewFile)
        ReDim $a_NewFile[$a_NewFile[0] + 1]
        $a_NewFile[$a_NewFile[0]] = $line
    EndIf
Next
_FileWriteFromArray($NewPath, $a_NewFile, 1)

but I like Xeno's method better

Edited March 25, 2009 by SpookMeister

[logcomptechs]

Hi,

FileReadToArray, then ArrayUnique FileWriteFromArray done

Mega

Thank You, that was extremely easy!

#Include <File.au3>
#include <Array.au3>
Dim $oFile,$nFile

_FileReadToArray("old_text.txt",$oFile)
$nFile = _ArrayUnique($oFile)
_FileWriteFromArray("new_text.txt",$nFile)

[Xenobiologist]

Thank You, that was extremely easy!

You're welcome. That's AutoIt

Mega

[TerarinK]

Perhaps you could clear this, everytime I start up the script and end it it will find the number of entries in the file and use that as a holder.

[Xenobiologist]

Perhaps you could clear this, everytime I start up the script and end it it will find the number of entries in the file and use that as a holder.

I don't know whether I got what you mean, but try this

#include <Array.au3>
#include <File.au3>
; Doppelte Zeilen löschen aus Datei

ConsoleWrite(_delelteDoubleLinesFromFile("test2.txt") & @CRLF)

Func _delelteDoubleLinesFromFile($filePath)
    Local $fileContent[1]
    If Not FileExists($filePath) Then Return -1
    If Not _FileReadToArray($filePath, $fileContent) Then Return -2
    $fileContent = _ArrayUnique($fileContent, 1, 1)
    _FileWriteFromArray($filePath, $fileContent, 1)
    Return 1
EndFunc  ;==>_delelteDoubleLinesFromFile

Mega

[TerarinK]

Thank you then

[JustDoIt]

NM.

Edited August 28, 2010 by JustDoIt

[bogQ]

Probably sort array and check if next line is identical to the curent line?

[trandunghcm]

On 26/3/2009 at 0:31 AM, logcomptechs said:

 

Thank You, that was extremely easy!

 

 

#Include <File.au3>
#include <Array.au3>
Dim $oFile,$nFile

_FileReadToArray("old_text.txt",$oFile)
$nFile = _ArrayUnique($oFile)
_FileWriteFromArray("new_text.txt",$nFile)

Good idea!

But i dont  like the first and sencond lines  created by Your Code. How to delete it! Please hepl me, thankyou!

[trandunghcm]

20 minutes ago, trandunghcm said:

Good idea!

But i dont  like the first and sencond lines  created by Your Code. How to delete it! Please hepl me, thankyou!

i have done! i was edited 

$iBase =2

_FileWriteFromArray("new_text.txt",$nFile,2)