Sign in to follow this  
Followers 0
JimC

Delete Duplicate lines in a text file

4 posts in this topic

Is there a way to delete duplicate lines in a text file ? I am having problem figuring out how to do that. Any help would be greatly appreciated.

Share this post


Link to post
Share on other sites



Is there a way to delete duplicate lines in a text file ? I am having problem figuring out how to do that. Any help would be greatly appreciated.

it will be a long Process but it can be done. I wouldn't try it on large files though.

_FileReadToArray()

Then

_ArraySearch()

FileWriteLine()

This means that you have to to test each element of the array to see if another element contains the same text.

If it's a known text string then it gets simpler

_FileReadToArray()

Then

_ArrayFindAll()

That would allow you to use _ArrayDelete() on the duplicate elements before doing the FileWriteLine()


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

Here is a quick and dirty way:

#include <file.au3>
#include <array.au3>

Dim $oldArray, $newArray[1]
Dim $inFile = "in.txt"
Dim $outFile = "out.txt"

_FileReadToArray($inFile, $oldArray)

For $X = 1 to $oldArray[0]

    $result = _ArraySearch($newArray,$oldArray[$X])

    If $result < 0 Then 
        $newArray[0] += 1
        _ArrayAdd($newArray, $oldArray[$X])
    EndIf
Next

FileDelete($outFile)

For $X = 1 to $newArray[0]
    FileWriteLine($outFile,$newArray[$X])
Next

;Currently there is a bug in this function, it will add an extra space at the top of the file
;_FileWriteFromArray("out.txt",$newArray, 1)

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Hi,

Quicker as;

$arrdupes0=_ArrayDupes( $arrItems0,  1,  0); _ArrayDupes(ByRef $arrItemsF, $iDelete = 0, $iDetails = 0)oÝ÷ Ù«­¢+ØìÉÉåÕÁÍà¹ÔÌ(ì}ÉÉåÕÁÌ¡    åIÀÌØíÉÉ%ѵͰÀÌØí¥±ÑôÀ°ÀÌØí¥Ñ¥±ÌôÀ¤(¥¹±Õµ½¹(¥¹±Õ±ÐíÉÉä¹ÔÌÐì(¥¹±ÕÅÕ½ÐíÉÉåÕÁÌà¹ÔÌÅÕ½Ðì)1½°ÀÌØí¥9Õ´ôÄÀÀÀ)1½°ÀÌØíÉÉ%ѵÍlÀÌØí¥9Õµt)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä($ÀÌØíÉÉ%ѵÍlÀÌØí¥tôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä´àÀ($ÀÌØíÉÉ%ѵÍlÀÌØí¤¬ÈÁtôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä´àÀ($ÀÌØíÉÉ%ѵÍlÀÌØí¤¬ÔÁtôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ(íø}ÉÉå¥ÍÁ±ä ÀÌØíÉÉ%ѵ̰ÅÕ½ÐìÀÌØíÉÉ%ѵÌÅÕ½Ðì¤(ÀÌØíÉÉ%ѵÌÀôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉ%ѵÌÄôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉ%ѵÌÈôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉÕÁÌÀõ}ÉÉåÕÁÌ ÀÌØíÉÉ%ѵÌÀ°Ä°À¤ì}ÉÉåÕÁÌ¡    åIÀÌØíÉÉ%ѵͰÀÌØí¥±ÑôÀ°ÀÌØí¥Ñ¥±ÌôÀ¤)}ÉÉå¥ÍÁ±ä ÀÌØíÉÉÕÁÌÀ°ÅÕ½ÐìÀÌØíÉÉÕÁÌÀìÀÌØíÉÉ%ѵÌÀìÅÕ½ÐìµÀíU   ½Õ¹ ÀÌØíÉÉ%ѵÌÀ¤´Ä¤(ÀÌØíÉÉÕÁÌÄõ}ÉÉåÕÁÌ ÀÌØíÉÉ%ѵÌİİĤ)}ÉÉå¥ÍÁ±ä ÀÌØíÉÉÕÁÌÄ°ÅÕ½ÐìÀÌØíÉÉÕÁÌÄìÀÌØíÉÉ%ѵÌÄìÅÕ½ÐìµÀíU   ½Õ¹ ÀÌØíÉÉ%ѵÌĤ´Ä¤
Best, Randall

ArrayDupes8.zip

Edited by randallc

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0