Detect duplicate lines

[AcidCorps]

I have a txt file that contains about 200 lines of data, each line is a different set of numbers and letters. I want to have a script scan each line and copy them to a new txt document except I want it to be able to detect duplicate lines and only copy them once if it already existed in a previous line.

[Xenobiologist]

Hi,

#include <file.au3>
#include <Array.au3>
Dim $aLines

If Not _FileReadToArray("duplicate.txt", $aLines) Then
    MsgBox(4096, "Error", " Error reading log to Array     error:" & @error)
    Exit
EndIf
_ArrayDisplay($aLines, "Files")
Global $array = RemoveDuplicates($aLines)
_ArrayDisplay($array, "Removed Duplicates")

Func RemoveDuplicates($avData)
    Local $avData2 = $avData
    Local $iCount = 0
    For $i = 0 To UBound($avData) - 1
        $iCount = 0
        For $ii = 0 To UBound($avData) - 1
            If $ii > UBound($avData2) - 1 Then ExitLoop
            If $avData2[$ii] = $avData[$i] Then
                If $iCount > 0 Then
                    _ArrayDelete($avData2, $ii)
                    _ArrayDelete($avData2, $i)
                EndIf
                $iCount += 1
            EndIf
        Next
    Next
    $avData2[0] = UBound($avData2)
    Return $avData2
EndFunc   ;==>RemoveDuplicates

So long,

Mega

[AcidCorps]

works perfect except I'd like the output to be another txt instead of a msg diolog.

[/dev/null]

Or even better would be an ini file with [section] and then DataLine1 for the keys or something similar to that. I've been playing around with it but can't figure out how to do this

do you also want a cold beer delivered with the script?

[Toady]

_FileWriteFromArray()

wow, help file is amazing!

[SmOke_N]

Hi,

#include <file.au3>
#include <Array.au3>
Dim $aLines

If Not _FileReadToArray("duplicate.txt", $aLines) Then
    MsgBox(4096, "Error", " Error reading log to Array     error:" & @error)
    Exit
EndIf
_ArrayDisplay($aLines, "Files")
Global $array = RemoveDuplicates($aLines)
_ArrayDisplay($array, "Removed Duplicates")

Func RemoveDuplicates($avData)
    Local $avData2 = $avData
    Local $iCount = 0
    For $i = 0 To UBound($avData) - 1
        $iCount = 0
        For $ii = 0 To UBound($avData) - 1
            If $ii > UBound($avData2) - 1 Then ExitLoop
            If $avData2[$ii] = $avData[$i] Then
                If $iCount > 0 Then
                    _ArrayDelete($avData2, $ii)
                    _ArrayDelete($avData2, $i)
                EndIf
                $iCount += 1
            EndIf
        Next
    Next
    $avData2[0] = UBound($avData2)
    Return $avData2
EndFunc   ;==>RemoveDuplicates

So long,

Mega

That's a big function, should search for "_ArrayUnique", much faster.

[AcidCorps]

do you also want a cold beer delivered with the script?

Cold beer would be nice

_FileWriteFromArray()

Thanks got it from there.

[herewasplato]

That's a big function, should search for "_ArrayUnique", much faster.

The link - I think :-)

http://www.autoitscript.com/forum/index.ph...st&p=230907

Edit: just a test edit to see if the link stays intact

Edit2: it did stay intact

Edited June 19, 2007 by herewasplato

[SmOke_N]

The link - I think :-)

http://www.autoitscript.com/forum/index.ph...st&p=230907

Edit: just a test edit to see if the link stays intact

Edit2: it did stay intact

This is probably the better link:

http://www.autoitscript.com/forum/index.ph...st&p=245675

[AliceDemetri]

I've come a lot sense this post. Wanted to thank SmOke_N I have used the _RemoveDuplicates functions is hundreds of my scripts with minor tweaks for various outcomes.