Jump to content
Sign in to follow this  

Delete duplicate lines from a file, but with some exceptions

Recommended Posts

A simple script like the next one can delete duplicate lines from files:


$InputFile = "C:\Test.txt"

Local $aFile, $aArray
_FileReadToArray($InputFile, $aFile)
$aArray = _ArrayUnique($aFile, 1, 1)
_FileWriteFromArray($InputFile, $aArray, 1)

How can I preserve, for example the duplicate empty lines and the duplicate lines that contain a string like %16422% ?

Share this post

Link to post
Share on other sites

This _ArrayUniqueEX() function appears to work.


Local $aFile, $aArray
Local $InputFile = @ScriptDir & "\Test-2.txt"
Local $OutputFile = @ScriptDir & "\Test-2b.txt"

$aFile = FileReadToArray($InputFile)

$aArray = _ArrayUniqueEX($aFile)

_FileWriteFromArray($OutputFile, $aArray, 1)

Func _ArrayUniqueEX(Const ByRef $aArray)
    Local $sData = ChrW(160), $sSep = ChrW(160)
    For $i = 0 To UBound($aArray) - 1
        If $aArray[$i] = "" Or _                                                  ; Add blank lines.
                StringInStr($aArray[$i], "%16422%") <> 0 Or _                     ; Add lines with "%16422%" present.
                StringInStr($sData, ChrW(160) & $aArray[$i] & ChrW(160)) = 0 Then ; Add lines that do not already exist in $sData.
            $sData &= $aArray[$i] & $sSep
    Return StringSplit(StringTrimRight(StringTrimLeft($sData, 1), 1), $sSep, 0)
EndFunc   ;==>_ArrayUniqueEX

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Create New...