Sign in to follow this  
Followers 0
mrbig1479

removing duplicate from a 2D array

13 posts in this topic

Hey , i have this array

$MyArray [100][5]

inside that array , there are duplicate lines

(a single element can be a duplicate , but i want to remove only cases where the all the elements are the same

for example

$MyArray [1][0] = A

$MyArray [1][1] = B

$MyArray [1][2] = C

$MyArray [1][3] = D

$MyArray [1][4] = E

$MyArray [1][5] = F

I Don't want to remove array [2][all the elements] in a case like this

$MyArray [2][0] = A

$MyArray [2][1] = B

$MyArray [2][2] = C

$MyArray [2][3] = D

$MyArray [2][4] = E

$MyArray [2][5] = XXX

but i do want to remove in a case like this (where is eqaul to $MyArray [1][all the elements] )

$MyArray [3][0] = A

$MyArray [3][1] = B

$MyArray [3][2] = C

$MyArray [3][3] = D

$MyArray [3][4] = E

$MyArray [3][5] = F

I hope i explained well

Share this post


Link to post
Share on other sites



(a single element can be a duplicate , but i want to remove only cases where the all the elements are the same)

One way is to create a 1D array by concatenating all elements in 2nd dimension with suitable delimiter (say "|"), pass thru _ArrayUnique and then use StringSplit($aArray[$i], "|", 2) to recreate your 2D array.

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

#Include <Array.au3>

Dim $MyArray[100][6] = [ _
["A", "B", "C", "D", "E", "F"], _
["A", "B", "C", "D", "E", "XXX"], _
["A", "B", "C", "D", "E", "F"], _
["A", "B", "C", "QQ", "E", "F"], _
["1", "2", "3", "4", "5", "6"], _
["A", "B", "C", "D", "E", "XXX"]]
;etc..

_ArrayDisplay($MyArray) ;before output
$outputArray = _CompareSecondDimension($MyArray)
_ArrayDisplay($outputArray) ;after output

Func _CompareSecondDimension(Const ByRef $csdArray)
    Local $aString[UBound($csdArray)], $rtnArray = $csdArray, $iCount = 0
    For $a = 0 to UBound($csdArray) - 1
        For $b = 0 to UBound($csdArray, 2) - 1
            $aString[$a] &= $csdArray[$a][$b]
        Next
        If _ArraySearch($aString, $aString[$a]) <> $a Then
            _ArrayDelete($rtnArray, $a - $iCount)
            $iCount += 1
        EndIf
    Next
    Return $rtnArray
EndFunc

;before output

[0]|A|B|C|D|E|F

[1]|A|B|C|D|E|XXX

[2]|A|B|C|D|E|F

[3]|A|B|C|QQ|E|F

[4]|1|2|3|4|5|6

[5]|A|B|C|D|E|XXX

[6]||||||

[7]||||||

[8]||||||

...etc...

;after output

[0]|A|B|C|D|E|F

[1]|A|B|C|D|E|XXX

[2]|A|B|C|QQ|E|F

[3]|1|2|3|4|5|6

[4]||||||

If I understood what you're trying to do correctly, then the above script SHOULD work.

It works by making a 1D array of all the concatenated second dimensions, it then searches it against what it's already done, and if it finds a match, it removes it from the return array.

I threw this together in about 20 minutes, so odds are, it can likely be optimized and enhanced, but the principle is there.

Edited by JRouleau

Share this post


Link to post
Share on other sites

Is that enough?

#include <Array.au3>
; Array

Global $a[4][5] = [[1, 2, 3, 4, 5],[1, 2, 3, 4, 5],[1, 2, 3, 4, 9],[1, 2, 3, 4, 8]]

_ArrayDisplay($a)

For $i = 1 To UBound($a, 1) - 2
    If $a[$i][0] = $a[0][0] And _
            $a[$i][1] = $a[0][1] And _
            $a[$i][2] = $a[0][2] And _
            $a[$i][3] = $a[0][3] And _
            $a[$i][4] = $a[0][4] Then
        _ArrayDelete($a, $i)
    EndIf
Next
_ArrayDisplay($a)

Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

Another example.

#include <Array.au3>

local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "D", "E", "XXX"], _
                     ["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "QQ", "E", "F"], _
                     ["1", "2", "3", "4", "5", "6"], _
                     ["A", "B", "C", "D", "E", "XXX"]]
                     ;etc..

Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray)

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = $MyArray[$i][0] & $MyArray[$i][1] & $MyArray[$i][2] & _
                $MyArray[$i][3] & $MyArray[$i][4] & $MyArray[$i][5] & @CRLF
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            $aNoDupsArray[$iCount][0] = $MyArray[$i][0]
            $aNoDupsArray[$iCount][1] = $MyArray[$i][1]
            $aNoDupsArray[$iCount][2] = $MyArray[$i][2]
            $aNoDupsArray[$iCount][3] = $MyArray[$i][3]
            $aNoDupsArray[$iCount][4] = $MyArray[$i][4]
            $aNoDupsArray[$iCount][5] = $MyArray[$i][5]
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc ;==>_ArrayRemoveDuplicateRows
1 person likes this

Share this post


Link to post
Share on other sites

Another example.

#include <Array.au3>

local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "D", "E", "XXX"], _
                     ["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "QQ", "E", "F"], _
                     ["1", "2", "3", "4", "5", "6"], _
                     ["A", "B", "C", "D", "E", "XXX"]]
                     ;etc..

Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray)

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = $MyArray[$i][0] & $MyArray[$i][1] & $MyArray[$i][2] & _
                $MyArray[$i][3] & $MyArray[$i][4] & $MyArray[$i][5] & @CRLF
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            $aNoDupsArray[$iCount][0] = $MyArray[$i][0]
            $aNoDupsArray[$iCount][1] = $MyArray[$i][1]
            $aNoDupsArray[$iCount][2] = $MyArray[$i][2]
            $aNoDupsArray[$iCount][3] = $MyArray[$i][3]
            $aNoDupsArray[$iCount][4] = $MyArray[$i][4]
            $aNoDupsArray[$iCount][5] = $MyArray[$i][5]
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc ;==>_ArrayRemoveDuplicateRows

works great !!

Thanks

Share this post


Link to post
Share on other sites

#7 ·  Posted

I created this account just to say thank you to JonathanR - your solution works flawless let and is exactly what I was looking for.  Had a hard time find a thread on how to delete duplicate rows in a 2d array.  Saved me hours of work writing my own function.  Thank you!

Share this post


Link to post
Share on other sites

#8 ·  Posted

I noticed my old example of post #5 did not allow for blank elements in a 2D array, and neither does JonathanR's example of post #3.   That is, both examples take these two rows as duplicates:-
    ["", "1", "", "", "3", ""], _
    ["1", "3", "", "", "", ""], _
because of the blank elements.

My modified example to allow for blank elements.

#include <Array.au3>

Local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..
_ArrayDisplay($MyArray, "Before")
Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray, "Unique")

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = "|" & $MyArray[$i][0] & "|" & $MyArray[$i][1] & "|" & $MyArray[$i][2] & "|" & _
                $MyArray[$i][3] & "|" & $MyArray[$i][4] & "|" & $MyArray[$i][5] & "|" & @CRLF ; Adding "|" allows for blank elements in array.
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            For $j = 0 To UBound($MyArray, 2) - 1
                $aNoDupsArray[$iCount][$j] = $MyArray[$i][$j]
            Next
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc   ;==>_ArrayRemoveDuplicateRows

And, a modified JonathanR's example allowing for blank elements.

#include <Array.au3>

Dim $MyArray[100][6] = [ _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..

_ArrayDisplay($MyArray) ;before output
$outputArray = _CompareSecondDimension($MyArray)
_ArrayDisplay($outputArray) ;after output

Func _CompareSecondDimension(Const ByRef $csdArray)
    Local $aString[UBound($csdArray)], $rtnArray = $csdArray, $iCount = 0
    For $a = 0 To UBound($csdArray) - 1
        For $b = 0 To UBound($csdArray, 2) - 1
            $aString[$a] &= "|" & $csdArray[$a][$b] & "|"
        Next
        ;ConsoleWrite($aString[$a] & @CRLF)
        If _ArraySearch($aString, $aString[$a]) <> $a Then
            _ArrayDelete($rtnArray, $a - $iCount)
            $iCount += 1
        EndIf
    Next
    Return $rtnArray
EndFunc   ;==>_CompareSecondDimension

 

Share this post


Link to post
Share on other sites

#9 ·  Posted

Thank you Malkey for pointing out the issue with blank elements.  I'm using JonathanR's code because it short and works.  But I'm hoping someone here can show me how to make it delete in reverse.  I notice _ArraySearch has an option to set to 0 to search in reverse and I tried stepping ( STEP - 1 ) from high to 0 for the $a loop.  Can't get it to work for me.

I discovered that I need it to make the highest numbered rows the authoritative.  Currently, it's making the first entry authoritative and deleting everything else that follows that is a duplicate.  I want it to keep the last entry and delete all duplicates above it.  Any takers?  I'm at a loss.  Thank you.

Share this post


Link to post
Share on other sites

#10 ·  Posted

Instead of creating a throw away array and using _ArraySearch as JonathanR does, I create a throw away string and use StringInStr.

This example deletes the first occurring duplicate rows , or, keeps the row with the highest array index of the duplicate rows.

#include <Array.au3>

Local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..
_ArrayDisplay($MyArray, "Before")
Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)
_ArrayDisplay($aDuplicatelessArray, "Unique")

; Remove first or lowest occurring duplicate row
Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = UBound($MyArray) - 1 To 0 Step -1
        $sString = ""
        For $k = 0 To UBound($MyArray, 2) - 1
            $sString &= "|" & $MyArray[$i][$k] & "|" & @CRLF ; Adding "|" allows for blank elements in array.
        Next
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= "|" & $sString & "|" & @CRLF ; Update $sNoDupStr string
            For $j = 0 To UBound($MyArray, 2) - 1
                $aNoDupsArray[$iCount][$j] = $MyArray[$i][$j]
            Next
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    ;_ArrayDisplay($aNoDupsArray)

    ; Invert array
    Local $iLen1 = UBound($aNoDupsArray), $iLen2 = UBound($aNoDupsArray, 2), $aRet[$iLen1][$iLen2]
    For $i = 0 To $iLen1 - 1
        For $j = 0 To $iLen2 - 1
            $aRet[$iLen1 - 1 - $i][$j] = $aNoDupsArray[$i][$j]
        Next
    Next
    Return $aRet
EndFunc   ;==>_ArrayRemoveDuplicateRows

#cs 
 Array before :-
    A|B|C|D|E|F
    A|B|C|D|E|XXX
    A|B|C|D|E|F
    1|2|3|4|5|6
    A|B|C|D|E|F
    A|B|C|QQ|E|F
    A|B|C|D|E|XXX
    |1|||3|
    1|3||||
    |||||
    |||||
    |||||
    ...Etc...

 Array returned:-
    1|2|3|4|5|6
    A|B|C|D|E|F
    A|B|C|QQ|E|F
    A|B|C|D|E|XXX
    |1|||3|
    1|3||||
    |||||
#ce

 

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

Remove duplicate rows (for ArrayWorkshop.au3 - see my signature).

#include 'ArrayWorkshop.au3'

Local $aArray = [[1,2,3],[1,2,3],[4,5,6],[1,2,3]]
_ArrayUniqueXD($aArray)

 

Edited by czardas

Share this post


Link to post
Share on other sites

#12 ·  Posted

To erase all duplicate rows except keep the last duplicate row, try:-

#include <Array.au3>
#include 'ArrayWorkshop.au3'

Local $aArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]

_ReverseArray($aArray)
_ArrayUniqueXD($aArray)
_ReverseArray($aArray)
_ArrayDisplay($aArray)

 

1 person likes this

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

@Malkey It took me a minute or two to figure out what you meant. Then I read post #9. :)

Edited by czardas

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0