Remove duplicated strings in a file

[GordonFreeman]

Hi, I have a text file that is as follows:

Note: one mark per line

ACM

ACM

ACM

AJE

AJE

AJE

AJE

AJE

ALMA

ALMA

I need to turn in:

ACM

AJE

ALMA

It's just an example because the file has several marks (27,000 lines).

[jdelaney]

look into _ArrayUnique, after reading each line in the file to an array.

[Mechaflash]

after reading each line in the file to an array.

aka _FileReadToArray()

EDIT: Note that arrays can handle some 16 million elements. so you're good =D

Edited June 15, 2012 by mechaflash213

[Malkey]

Comparing the removal of duplicate strings in a file by the array method and the Regular Expression method.

Using a 1,011 line file, we have :-

Array method Time taken approx.2.9 secs:

Reg.Exp. method Time taken approx.0.4 secs.

If the RunWait Sort function is not needed, that is, if the file is already sorted, the Reg.Exp. method's completion time is approx.0.01 secs.

The array method:

expandcollapse popup

#include <File.au3>
#include <Array.au3>

Local $sDupString = _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ALMA" & @CRLF & _
        "ALMA" & @CRLF
Local $sDup
For $i = 1 To 500
    $sDup = Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & @CRLF
    $sDupString &= $sDup & $sDup
Next

Local $sFileName = "DupFile.txt"
If FileExists($sFileName) Then FileDelete($sFileName)
FileWrite($sFileName, $sDupString)

Local $begin = TimerInit()
Local $aDupArray
_FileReadToArray($sFileName, $aDupArray)
;_ArrayDisplay($aDupArray,0)
;_ArrayDelete($aModArray,0)

Local $aModArray = _ArrayUnique($aDupArray)
;_ArrayDelete($aModArray,0)

Local $sModString = _ArrayToString($aModArray, @CRLF, 2)

ConsoleWrite("Time: " & TimerDiff($begin) & @LF) ; Time: 2917.63390849968
MsgBox(0, "Results", StringStripWS($sModString, 2))
;_ArrayDisplay($aModArray)

FileDelete($sFileName)

The Regular Expression method:

Local $sDupString = _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ALMA" & @CRLF & _
        "ALMA" & @CRLF
Local $sDup
For $i = 1 To 500
    $sDup = Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & Chr(Random(66, 90, 1)) & @CRLF
    $sDupString &= $sDup & $sDup
Next
Local $sFileName = "DupFile.txt"

If FileExists($sFileName) Then FileDelete($sFileName)
FileWrite($sFileName, $sDupString)
;ShellExecute($sFileName)
Local $begin = TimerInit()

RunWait(@ComSpec & " /c Sort " & $sFileName & " /O " & $sFileName, "", @SW_HIDE) ; Sort file

Local $sModString = StringRegExpReplace(FileRead($sFileName), "([^\v]*)(\v+)(\1\2*)*", "$1$2")
; Or
;Local $sModString = StringRegExpReplace(FileRead($sFileName) & @CRLF, "([^\v]*\v+)(\1*)", "\1")

ConsoleWrite("Time: " & TimerDiff($begin) & @LF) ; Time: 370.465418593286
MsgBox(0, "Results", StringStripWS($sModString, 2))

FileDelete($sFileName)

[AZJIO]

expandcollapse popup

Func _CreateFile($Path_In)

Local $hFile, $i, $s = ''

For $i = 0 To 1000000

  $s &= Random(1, 1000, 1) & @CRLF

Next

$hFile = FileOpen($Path_In, 2)

FileWrite($hFile, $s)

FileClose($hFile)

EndFunc



$Path_In = @ScriptDir & 'test_in.txt'

_CreateFile($Path_In)

$Path_Out = @ScriptDir & 'test_Out.txt'

$sText = FileRead($Path_In)

$err = 0

$timer = TimerInit()

$aText_Out = _StringUnique($sText)

If @error Then $err = @error

$timer = Round(TimerDiff($timer) / 1000, 2)

If $err Then

MsgBox(0, 'error', 'not found' & @CRLF & 'Time = ' & $timer & 'sec')

Exit

Else

$hFile = FileOpen($Path_Out, 2)

FileWrite($hFile, $aText_Out)

FileClose($hFile)

EndIf



MsgBox(0, "Time", 'Time = ' & $timer & 'sec')



; не учитывает регистр String = StRiNg = STRING

; not case sensitive, String = StRiNg = STRING

Func _StringUnique($sText, $sep = @CRLF)

Local $i, $k, $aText, $s, $Trg = 0, $LenSep

If StringInStr($sText, '[') And $sep <> '[' Then

  For $i = 0 To 255

   $s = Chr($i)

   If Not StringInStr($sText, $s) Then

    If StringInStr($sep, $s) Then ContinueLoop

    $sText = StringReplace($sText, '[', $s)

    $Trg = 1

    ExitLoop

   EndIf

  Next

  If Not $Trg Then Return SetError(1, 0, '')

EndIf



$LenSep = StringLen($sep)

$aText = StringSplit($sText, $sep, 1)

Assign('/', 2, 1)

$k = 0

$sText = ''

For $i = 1 To $aText[0]

  If Not IsDeclared($aText[$i] & '/') Then

   Assign($aText[$i] & '/', 0, 1)

   $sText &= $aText[$i] & $sep

   $k += 1

  EndIf

Next

If $k = 0 Then Return SetError(2, 0, '')

If $Trg Then $sText = StringReplace($sText, $s, '[')

Return StringTrimRight($sText, $LenSep)

EndFunc

Edited June 16, 2012 by AZJIO

[guinness]

Malkey,

Nice regular expression.

[AZJIO]

Malkey,

Local $sModString = StringRegExpReplace(FileRead($sFileName), '([^R]+?)(R+)(1R+)+', '1' & @CRLF)

[UEZ]

Built-in array unique function is slow. Try alternatives:

Br,

UEZ

[guinness]

Without writing to a file... (thanks to Malkey for the SRE.)

expandcollapse popup

#include <Constants.au3>

Local $sString = _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ACM" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "AJE" & @CRLF & _
        "ALMA" & @CRLF & _
        "ALMA" & @CRLF

$sString = _Sort($sString)
$sString = _RemoveDuplicates($sString)
ConsoleWrite($sString & @CRLF)

Func _RemoveDuplicates($sData)
;~  Return StringRegExpReplace($sData, '([^v]*)(v+)(12*)*', '$1$2') ; By Malkey
;~  Return StringRegExpReplace($sData, '([^v]*v+)(1*)', '1') ; By Malkey
    Return StringRegExpReplace($sData, '([^R]+?)(R+)(1R+)+', '1' & @CRLF) ; By AZJIO
EndFunc   ;==>_RemoveDuplicates

Func _Sort($sSortList)
    Local $iPID = Run('sort.exe', @SystemDir, @SW_HIDE, $STDIN_CHILD + $STDOUT_CHILD), $sOutput = ''

    StdinWrite($iPID, $sSortList)
    StdinWrite($iPID)

    While 1
        $sOutput &= StdoutRead($iPID)
        If @error Then
            ExitLoop
        EndIf
    WEnd
    Return $sOutput
EndFunc   ;==>_Sort

Edited June 17, 2012 by guinness

[AZJIO]

guinness

$s = _
'1' &@CRLF& _
'1' &@CRLF& _
'10' &@CRLF& _
'10'
MsgBox(0, 'True?', StringRegExpReplace($s, '([^v]*)(v+)(12*)*', '$1$2'))
MsgBox(0, 'True?', StringTrimRight(StringRegExpReplace($s & @CRLF, '([^R]+?)(R+)(1R+)+', '1' & @CRLF), 2))

1

1

10

10

=

1

0

10

[guinness]

AZJIO,

OK, point taken.

[AZJIO]

guinness

#include <Constants.au3>
Local $sString = _
     "ACM" & @CRLF & _
     "ACM" & @CRLF & _
     "AcM" & @CRLF & _
     "ACM" & @CRLF & _
     "AJE" & @CRLF & _
     "AJE" & @CRLF & _
     "AjE" & @CRLF & _
     "AJE" & @CRLF & _
     "AJE" & @CRLF & _
     "AJE" & @CRLF & _
     "ALMA" & @CRLF & _
     "ALMA" & @CRLF & _
     "aLMA" & @CRLF & _
     "ALMA" & @CRLF
$sString = _Sort($sString)
$sString = _RemoveDuplicates($sString)
MsgBox(0, 'True? Yes?', $sString)
Func _RemoveDuplicates($sData)
Return StringRegExpReplace($sData, '([^v]*)(v+)(12*)*', '$1$2') ; By Malkey
EndFunc ;==>_RemoveDuplicates
Func _Sort($sSortList)
Local $iPID = Run('sort.exe', @SystemDir, @SW_HIDE, $STDIN_CHILD + $STDOUT_CHILD), $sOutput = ''
StdinWrite($iPID, $sSortList)
StdinWrite($iPID)
While 1
     $sOutput &= StdoutRead($iPID)
     If @error Then
         ExitLoop
     EndIf
WEnd
Return $sOutput
EndFunc ;==>_Sort

the result of an incorrect

ACM

AcM

ACM

AJE

AjE

AJE

ALMA

aLMA

ALMA

Edited June 17, 2012 by AZJIO

[guinness]

I understood. I changed my post with your SRE.

[AZJIO]

The problem is not in SPE

Sorting is case-insensitive. Link to the group in the regular expression is an exact match. Even if the SPE changed the result will be the same

(?i)([^R]+?)(R+)(1R+)+

(?i) - does not work for Cyrillic

If there is no LF for the last line you need to change the regular expression

(?i)([^R]+?)(R+)(1(R+|z))+

Edited June 17, 2012 by AZJIO