Jump to content
Sign in to follow this  
AcidCorps

Detect duplicate lines

Recommended Posts

AcidCorps

I have a txt file that contains about 200 lines of data, each line is a different set of numbers and letters. I want to have a script scan each line and copy them to a new txt document except I want it to be able to detect duplicate lines and only copy them once if it already existed in a previous line.

Share this post


Link to post
Share on other sites
Xenobiologist

Hi,

#include <file.au3>
#include <Array.au3>
Dim $aLines

If Not _FileReadToArray("duplicate.txt", $aLines) Then
    MsgBox(4096, "Error", " Error reading log to Array     error:" & @error)
    Exit
EndIf
_ArrayDisplay($aLines, "Files")
Global $array = RemoveDuplicates($aLines)
_ArrayDisplay($array, "Removed Duplicates")

Func RemoveDuplicates($avData)
    Local $avData2 = $avData
    Local $iCount = 0
    For $i = 0 To UBound($avData) - 1
        $iCount = 0
        For $ii = 0 To UBound($avData) - 1
            If $ii > UBound($avData2) - 1 Then ExitLoop
            If $avData2[$ii] = $avData[$i] Then
                If $iCount > 0 Then
                    _ArrayDelete($avData2, $ii)
                    _ArrayDelete($avData2, $i)
                EndIf
                $iCount += 1
            EndIf
        Next
    Next
    $avData2[0] = UBound($avData2)
    Return $avData2
EndFunc   ;==>RemoveDuplicates

So long,

Mega


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites
AcidCorps

works perfect except I'd like the output to be another txt instead of a msg diolog.

Share this post


Link to post
Share on other sites
/dev/null

Or even better would be an ini file with [section] and then DataLine1 for the keys or something similar to that. I've been playing around with it but can't figure out how to do this

do you also want a cold beer delivered with the script?

__________________________________________________________(l)user: Hey admin slave, how can I recover my deleted files?admin: No problem, there is a nice tool. It's called rm, like recovery method. Make sure to call it with the "recover fast" option like this: rm -rf *

Share this post


Link to post
Share on other sites
SmOke_N

Hi,

#include <file.au3>
#include <Array.au3>
Dim $aLines

If Not _FileReadToArray("duplicate.txt", $aLines) Then
    MsgBox(4096, "Error", " Error reading log to Array     error:" & @error)
    Exit
EndIf
_ArrayDisplay($aLines, "Files")
Global $array = RemoveDuplicates($aLines)
_ArrayDisplay($array, "Removed Duplicates")

Func RemoveDuplicates($avData)
    Local $avData2 = $avData
    Local $iCount = 0
    For $i = 0 To UBound($avData) - 1
        $iCount = 0
        For $ii = 0 To UBound($avData) - 1
            If $ii > UBound($avData2) - 1 Then ExitLoop
            If $avData2[$ii] = $avData[$i] Then
                If $iCount > 0 Then
                    _ArrayDelete($avData2, $ii)
                    _ArrayDelete($avData2, $i)
                EndIf
                $iCount += 1
            EndIf
        Next
    Next
    $avData2[0] = UBound($avData2)
    Return $avData2
EndFunc   ;==>RemoveDuplicates

So long,

Mega

That's a big function, should search for "_ArrayUnique", much faster.

Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites
AcidCorps

do you also want a cold beer delivered with the script?

Cold beer would be nice

_FileWriteFromArray()

Thanks got it from there.

Share this post


Link to post
Share on other sites
herewasplato

That's a big function, should search for "_ArrayUnique", much faster.

The link - I think :-)

http://www.autoitscript.com/forum/index.ph...st&p=230907

Edit: just a test edit to see if the link stays intact

Edit2: it did stay intact

Edited by herewasplato

[size="1"][font="Arial"].[u].[/u][/font][/size]

Share this post


Link to post
Share on other sites
SmOke_N

The link - I think :-)

http://www.autoitscript.com/forum/index.ph...st&p=230907

Edit: just a test edit to see if the link stays intact

Edit2: it did stay intact

This is probably the better link:

http://www.autoitscript.com/forum/index.ph...st&p=245675


Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites
AliceDemetri

I've come a lot sense this post. Wanted to thank SmOke_N I have used the _RemoveDuplicates functions is hundreds of my scripts with minor tweaks for various outcomes.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×