Jump to content
Sign in to follow this  

Delete duplicate lines from a file, but with some exceptions

Recommended Posts


A simple script like the next one can delete duplicate lines from files:


$InputFile = "C:\Test.txt"

Local $aFile, $aArray
_FileReadToArray($InputFile, $aFile)
$aArray = _ArrayUnique($aFile, 1, 1)
_FileWriteFromArray($InputFile, $aArray, 1)

How can I preserve, for example the duplicate empty lines and the duplicate lines that contain a string like %16422% ?

Share this post

Link to post
Share on other sites

This _ArrayUniqueEX() function appears to work.


Local $aFile, $aArray
Local $InputFile = @ScriptDir & "\Test-2.txt"
Local $OutputFile = @ScriptDir & "\Test-2b.txt"

$aFile = FileReadToArray($InputFile)

$aArray = _ArrayUniqueEX($aFile)

_FileWriteFromArray($OutputFile, $aArray, 1)

Func _ArrayUniqueEX(Const ByRef $aArray)
    Local $sData = ChrW(160), $sSep = ChrW(160)
    For $i = 0 To UBound($aArray) - 1
        If $aArray[$i] = "" Or _                                                  ; Add blank lines.
                StringInStr($aArray[$i], "%16422%") <> 0 Or _                     ; Add lines with "%16422%" present.
                StringInStr($sData, ChrW(160) & $aArray[$i] & ChrW(160)) = 0 Then ; Add lines that do not already exist in $sData.
            $sData &= $aArray[$i] & $sSep
    Return StringSplit(StringTrimRight(StringTrimLeft($sData, 1), 1), $sSep, 0)
EndFunc   ;==>_ArrayUniqueEX
  • Like 1

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  


Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.