How to delete strings within a string that are repeats

[billthecreator]

i have a file with over 800 strings

how would i write a script that would delete the repeats

example:

This is String 1

This is String 2

This is String 3

This is String 4

This is String 1

This is String 3

This is String 8

and i want to remove the ones that are repeating.

also, it being over 800 lines, i started writing a for loop, but had no luck.

i also searched for something with deleting string repeats, and had no luck.

is there anyone who has done this before and would like to help me?

thanks.

Edited September 14, 2010 by billthecreator

[UnDeFiNeD]

You should try out this guy's UDF, it looks like just what you need http://www.autoitscript.com/forum/index.php?showtopic=119596

[funkey]

expandcollapse popup

#Include <File.au3>

Local $aFile
_FileReadToArray(@ScriptDir & "\File.txt", $aFile)

_Array2DDblDel($aFile)

_FileWriteFromArray(@ScriptDir & "\FileNew.txt", $aFile, 1)


;----------------------------------------------------------------------------------------------------------------------
; Function  _Array2DDblDel(ByRef $ARRAY [, $CASESENS=0])
;
; Description - From an 1D/2D Array will delete double entries (2D -> combination by '[n][0]' to '[n][x]').
;    - Autodetection 1D/2D Array
;    - By using string, you can choose case sensitivity.
;
; Parameter  $ARRAY:   Array to sort
; optional $CASESENS:  Case sensitivity off[0] or on[1] (default 0)
;
; Return  Succes   ByRef Array without doubles
;        Count of doubles
;    Failure   0 and set @error = 1; no array
;
; Author  BugFix ([email="bugfix@autoit.de"]bugfix@autoit.de[/email])
;----------------------------------------------------------------------------------------------------------------------
Func _Array2DDblDel(ByRef $ARRAY, $CASESENS=0)
 Local $arTmp[1] = [''], $dbl = 0, $count = 0, $x, $l, $val, $valTmp, $i, $k
 If ( Not IsArray($ARRAY) ) Then
  SetError(1)
  Return 0
 EndIf
 Local $Ubound2nd = UBound($ARRAY,2)
 If @error = 2 Then
  For $i = 0 To UBound($ARRAY)-1
   $dbl = 0
   For $k = 0 To UBound($arTmp)-1
    Switch $CASESENS
     Case 0
      If $arTmp[$k] = $ARRAY[$i] Then 
       $dbl = 1
       $count += 1
      EndIf
     Case 1
      If $arTmp[$k] == $ARRAY[$i] Then
       $dbl = 1
       $count += 1
      EndIf
    EndSwitch 
   Next
   If $dbl = 0 Then
    If $arTmp[0] = "" Then
     $arTmp[0] = $ARRAY[$i]
    Else
     ReDim $arTmp[UBound($arTmp)+1]
     $arTmp[UBound($arTmp)-1] = $ARRAY[$i]
    EndIf
   Else
    $dbl = 0
   EndIf
  Next
 Else
  ReDim $arTmp[1][$Ubound2nd]
  $arTmp[0][0] = ''
  $x = 0
  For $i = 0 To UBound($ARRAY)-1
   $dbl = 0
   $val = ''
   $valTmp = ''
   For $l = 0 To $Ubound2nd-1
    $val &= $ARRAY[$i][$l]
   Next
   For $k = 0 To UBound($arTmp)-1
    For $l = 0 To $Ubound2nd-1
     $valTmp &= $arTmp[$k][$l]
    Next
    Switch $CASESENS
     Case 0
      If  $valTmp = $val Then
       $dbl = 1
       $count += 1
      EndIf
     Case 1  
      If  $valTmp == $val Then
       $dbl = 1
       $count += 1
      EndIf
    EndSwitch
    $valTmp = ''
   Next
   If $dbl = 0 Then
    If $x = 1 Then ReDim $arTmp[UBound($arTmp)+1][$Ubound2nd]
    For $l = 0 To $Ubound2nd-1
     If $arTmp[0][0] = '' Or $x = 0 Then 
      $arTmp[0][$l] = $ARRAY[0][$l]
      If $l = $Ubound2nd-1 Then $x = 1
     Else
      $arTmp[UBound($arTmp)-1][$l] = $ARRAY[$i][$l]
      $x = 2
      If $l = $Ubound2nd-1 Then $x = 1
     EndIf
    Next
   Else
    $dbl = 0
   EndIf
  Next
 EndIf
 $ARRAY = $arTmp
 Return $count
EndFunc ; ==>_ArrayDblDel

[billthecreator]

perfect!

thank you so much

[trancexx]

You can try some cool stuff too.

$sString = FileRead("YourFile.txt")

For $sLine In StringRegExp($sString, "(.*?)\v+|", 3)
    StringRegExpReplace($sString, "\Q" & $sLine & "\E", $sLine)
    If @extended > 1 Then $sString = StringRegExpReplace($sString, "\Q" & $sLine & "\E\v+", @CRLF, @extended - 1)
Next

FileWrite("YourFile_New.txt", $sString)