Remove duplicated lines in a text file

[Sercankd]

i am newbie in autoit so i cant create good programs

i need remove lines they are duplicated

i have a list that contains some important accounts and there is lines with same username but different password i want remove only other lines after first search result in text file

example

first search result is real one and i dont want delete it

asdasd31 - 505052 - asd@asd.com

there is line with same username but password different and this one should removed

asdasd31 - 346363 - sercan@sercan.com

thanks in advance

[Zedna]

Use forum Search engine:

http://www.autoitscript.com/forum/index.ph...ray++duplicates

http://www.autoitscript.com/forum/index.ph...ray++duplicates

Edited December 3, 2008 by Zedna

[Sercankd]

i tried one but it deletes only lines with same email :S

expandcollapse popup

#Region;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=icon.ico
#AutoIt3Wrapper_Res_Comment=Sercankd ECSRO
#AutoIt3Wrapper_Res_Description=Kayıt Programı
#AutoIt3Wrapper_Res_Fileversion=1
#EndRegion;**** Directives created by AutoIt3Wrapper_GUI ****
#include <IE.au3>
#include <file.au3>
Global $file
Global $toplam
Opt("TrayMenuMode", 1)
TraySetIcon("Shell32.dll", 13)
HotKeySet("{END}", "Kapat")
Func Kapat()
    Exit 0;
EndFunc  ;==>Kapat
If FileExists("kayitlar.txt") Then
While 1 = 1
    $file = FileOpen("kayitlar.txt", 1);kayit dosyası
    $email = "asd@asd.com";email
    $password = Random(100000, 999999, 1);password 100000 ile 999999 arası rastgele
    $username = "Sercan" & Random(10, 9999, 1);default sercan + rastgele sayı 10 ile 9999 arası
    FileWrite($file, $username & " - " & $password & " - " & $email & @CRLF);dosyaya yaz kayıt tut
    $sayi = Random(100, 999);rastgele fonksiyonu
    $oIE = _IECreate("http://221.195.40.141:1026/fembria/",0,0,0,1)
    _IELoadWait($oIE)
    $oForm = _IEFormGetObjByName($oIE, "register")
    $Usr = _IEGetObjByName($oIE, "account")
    _IEFormElementSetValue($Usr, $username)
    $Usr1 = _IEGetObjByName($oIE, "myemail")
    _IEFormElementSetValue($Usr1, $email)
    $Usr2 = _IEGetObjByName($oIE, "username")
    _IEFormElementSetValue($Usr2, $username)
    $Usr3 = _IEGetObjByName($oIE, "pasword")
    _IEFormElementSetValue($Usr3, $password)
    $Usr4 = _IEGetObjByName($oIE, "pssword")
    _IEFormElementSetValue($Usr4, $password)
    $Btn = _IEGetObjByName($oIE, "submit")
    _IEAction($Btn, "click")
    $toplam = _FileCountLines("kayitlar.txt")
    TrayTip("Toplam", "Toplam Üyelik Sayısı: " & $toplam , 2, 1)
    Sleep(8000)
    _IEQuit($oIE)
    FileClose($file)
WEnd
Else
        MsgBox(48,"Hata!", "Kayıt dosyası bulunamadı.Program kapatılıyor.")
EndIf

this is the code i wrote for output registered accounts and

and this is the results

Sercan674 - 822856 - asd@asd.com

Sercan281 - 530128 - asd@asd.com

Sercan494 - 626108 - asd@asd.com

Sercan674 - 505645 - asd@asd.com

you can see there is 2x Sercan674 and i want remove all sercan674s except first one

Edited December 3, 2008 by Sercankd

[Sercankd]

any help?

[KaFu]

I was already thinking about to help... but looking at the provided source... I doubt you have a respectable reason for any help...

Edit: Typo

Edited December 4, 2008 by KaFu

[Zedna]

This script looks like malicious code.

Moderator should delete above script and lock this thread.

[Sercankd]

thats not a malicious code.its just registering some important accounts for me before registration closes for game server forever

[KaFu]

thats not a malicious code.its just registering some important accounts for me before registration closes for game server forever

Hmmm, okay, cheating @ games is not that malicious... not nice but not too bad either ...

Threw something together from help-file, look at it and adopt something like it in your script...

$file = FileOpen("input.txt", 0)
$output_text = ""
; Check if file opened for reading OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
; Read in lines of text until the EOF is reached
While 1
    $line = FileReadLine($file)
    If @error = -1 Then ExitLoop
    
    if Not StringInStr($output_text,stringleft($line,StringInStr($line," - "))) then
        $output_text &= $line & @CRLF
    EndIf
Wend
FileClose($file)

$file = FileOpen("output.txt", 1)
; Check if file opened for writing OK
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
FileWrite($file, $output_text)
FileClose($file)

Edited December 4, 2008 by KaFu

[myspacee]

a bit confused, but works.

expandcollapse popup

#include <array.au3>
#include <file.au3>
#include <guiconstants.au3>
#include <string.au3>
;~ #include <array.au3>

;---------------------------------------- ask user for txt file
$message = "Choose text file to make your Wordlist"
dim $chars
$var = FileOpenDialog($message, @DesktopCommonDir & "\", " Text file (*.txt)", 1)

If @error Then
    MsgBox(4096,"","No File chosen")
Else
;-------------------------------------------------------- open File
    $filename_user = StringReplace($var, "|", @CRLF)
    $file = FileOpen($filename_user, 0)

;-------------------------------------------------------- Check if file opened for reading OK
    If $file = -1 Then
        MsgBox(0, "Error", "Unable to open file.")
        Exit
    elseif $file <> -1 Then
        $chars = FileRead($file)
;-------------------------------------------------------- create array
        ToolTip("Create array")
        $array = StringSplit($chars, @CRLF)

;-------------------------------------------------------- order in array - delete dupes
        ToolTip("delete dupes - sorting...")
        _ArraySort($Array, 0, 0, 0, 0)
        
        ToolTip("delete dupes")
        $array = _ArrayUnique($array)
        
        ToolTip("delete dupes - sorting...")
        _ArraySort($Array, 0, 0, 0, 0)

;-------------------------------------------------------- write wordlist
        ToolTip("write wordlist")
        FileDelete(@ScriptDir & "\test_sorted.txt")
        $file_sorted = FileOpen(@ScriptDir & "\test_sorted.txt", 2)
        
        for $i = 1 to UBound($array) - 1
            FileWrite($file_sorted, $array[$i] & @CRLF)
        next
        
        FileClose($file)
        FileClose($file_sorted)
;-------------------------------------------------------- clean blank lines
    ToolTip("clean blank lines")
        $file = FileOpen(@ScriptDir & "\test_sorted.txt", 0)

        FileDelete(@ScriptDir & "\Wordlist_from_text.txt")
        $file_sorted = FileOpen(@ScriptDir & "\Wordlist_from_text.txt", 2)

; Read in lines of text until the EOF is reached
        While 1
            $line = FileReadLine($file)
            If @error = -1 Then ExitLoop
                
            $line1 = StringStripWS($line,1)
            $line1 = StringStripCR($line1)

            if $line1 <> "" Then
                FileWrite($file_sorted, $line1 & @CRLF)
            EndIf
        Wend


    EndIf
endif

ToolTip("")
FileClose($file)
FileClose($file_sorted)
FileDelete(@ScriptDir & "\test_sorted.txt")


Func _ArrayUnique($aArray, $iDimension = 1, $iBase = 0, $iCase = 0, $vDelim = "|")
Local $iUboundDim
;$aArray used to be ByRef, but litlmike altered it to allow for the choosing of 1 Array Dimension, without altering the original array
If $vDelim = "|" Then $vDelim = Chr(01); by SmOke_N, modified by litlmike
If Not IsArray($aArray) Then Return SetError(1, 0, 0);Check to see if it is valid array

;Checks that the given Dimension is Valid
    If Not $iDimension > 0 Then
        Return SetError(3, 0, 0);Check to see if it is valid array dimension, Should be greater than 0
    Else
;If Dimension Exists, then get the number of "Rows"
    $iUboundDim = UBound($aArray, 1);Get Number of "Rows"
    If @error Then Return SetError(3, 0, 0);2 = Array dimension is invalid.

;If $iDimension Exists, And the number of "Rows" is Valid:
        If $iDimension > 1 Then;Makes sure the Array dimension desired is more than 1-dimensional
            Local $aArrayTmp[1];Declare blank array, which will hold the dimension declared by user
            For $i = 0 To $iUboundDim - 1;Loop through "Rows"
            _ArrayAdd($aArrayTmp, $aArray[$i][$iDimension - 1]);$iDimension-1 to match Dimension
            Next
            _ArrayDelete($aArrayTmp, 0);Get rid of 1st-element which is blank
        Else;Makes sure the Array dimension desired is 1-dimensional
    ;If Dimension Exists, And the number of "Rows" is Valid, and the Dimension desired is not > 1, then:
    ;For the Case that the array is 1-Dimensional
            If UBound($aArray, 0) = 1 Then;Makes sure the Array is only 1-Dimensional
            Dim $aArrayTmp[1];Declare blank array, which will hold the dimension declared by user
            For $i = 0 To $iUboundDim - 1
            _ArrayAdd($aArrayTmp, $aArray[$i])
            Next
            _ArrayDelete($aArrayTmp, 0);Get rid of 1st-element which is blank
        Else;For the Case that the array is 2-Dimensional
            Dim $aArrayTmp[1];Declare blank array, which will hold the dimension declared by user
            For $i = 0 To $iUboundDim - 1
            _ArrayAdd($aArrayTmp, $aArray[$i][$iDimension - 1]);$iDimension-1 to match Dimension
            Next
            _ArrayDelete($aArrayTmp, 0);Get rid of 1st-element which is blank
        EndIf
    EndIf
EndIf

Local $sHold;String that holds the Unique array info
For $iCC = $iBase To UBound($aArrayTmp) - 1;Loop Through array
;If Not the case that the element is already in $sHold, then add it
    If Not StringInStr($vDelim & $sHold, $vDelim & $aArrayTmp[$iCC] & $vDelim, $iCase) Then _
    $sHold &= $aArrayTmp[$iCC] & $vDelim
Next

If $sHold Then
    $aArrayTmp = StringSplit(StringTrimRight($sHold, StringLen($vDelim)), $vDelim, 1);Split the string into an array
    Return $aArrayTmp;SmOke_N's version used to Return SetError(0, 0, 0)
EndIf

Return SetError(2, 0, 0);If the script gets this far, it has failed
EndFunc;==>_ArrayUnique

Pay attention !

script makes alphabetic order to result file.

m.

Edited December 4, 2008 by myspacee