Remove Duplicate Listview items...

Rahul Rohela · June 24, 2006

Hi,

Below is the script to remove duplicate listviewitems from list view. But this code has problem that its taking more than 5 min to compile 615 items & during process CPU usage is going up to 100%..

Please help to make it faster with leas CPU utilization...

It would be great that if someone create UDF for this.

#include <GUIConstants.au3>
#Include <GuiListView.au3>
#Include <Date.au3>
$time1 = _NowTime(3)

GUICreate("listview items",220,250, 100,200,-1,$WS_EX_ACCEPTFILES)
GUISetBkColor (0x00E0FFFF); will change background color
$lable = GUICtrlCreateLabel ( "",-1,200, 200)
$lable1 = GUICtrlCreateLabel ( "",-1,220, 200)

$listview = GuiCtrlCreateListView ("col1  |col2|col3  ",10,10,200,150);,$LVS_SORTDESCENDING)
$button = GuiCtrlCreateButton ("Value?",75,170,70,20)
$item1=GuiCtrlCreateListViewItem("item2|col22|col23",$listview)
$item2=GuiCtrlCreateListViewItem("...item1|col12|col13",$listview)
$item3=GuiCtrlCreateListViewItem("item3|col32|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item400|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item4|col32|col33",$listview)
GuiCtrlCreateListViewItem("item200|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item300|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item50|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item70|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item90|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item1000|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item501|col32*|col33",$listview)
;$input1=GuiCtrlCreateInput("",20,200, 150)
GuiCtrlSetState(-1,$GUI_DROPACCEPTED); to allow drag and dropping
GuiSetState()
GUICtrlSetData($item2,"|ITEM1")
GUICtrlSetData($item3,"||COL33")

For $x = 16 To 600
    GUICtrlCreateListViewItem("item" & $x, $listview)
Next

Do
  $msg = GuiGetMsg ()
     
   Select
      Case $msg = $button
         _deleteduplicate()
      Case $msg = $listview
         MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
   EndSelect
Until $msg = $GUI_EVENT_CLOSE

Func _deleteduplicate()
$count = _GUICtrlListViewGetItemCount($listview)
$countY = _GUICtrlListViewGetItemCount($listview)
GUICtrlsetdata ($lable1 ,$count)    
     
;msgbox(0,"First cehck ","$count is " & $count & " $countY is " & $county)


    $x = 0
    Do  
        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0,"listview items")
        $y = $x
        Do
;msgbox(0,"Count of y", $countY)
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $Y, 0,"listview items")
            GUICtrlsetdata ($lable ,"X is " & $x & " Y is " & $y)
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                If $x <> $Y Then
;msgbox(0,"Second check1 ","X is " & $x & " Y is " & $y)
                _GUICtrlListViewDeleteItem($listview, $Y)
                
        ;$y = $x
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                Else
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                $y = $y +1
                Endif
            Else
            $countY = _GUICtrlListViewGetItemCount($listview)
            $count = _GUICtrlListViewGetItemCount($listview)
            $y = $y +1
            Endif
        
        Until $y = $countY
;msgbox(0," ","$y is " & $y & " $countY is " & $countY)
    $x = $x +1
    Until $x = $count
    msgbox(0,"time","First time "  & $time1 & " Second time " & _NowTime(3))

EndFunc

Above code is for beta v3.1.1.126

As same code after making following cahanges for beta v3.1.1.127

http://www.autoitscript.com/forum/index.php?showtopic=27914

http://www.autoitscript.com/forum/index.php?showtopic=27921

in GuiListView.au3 is not working as expected

same code processing time increased by 6 more min that mean total 12 min to compile same 615 items.

Secomd whole Listvew is flikering with new beta not with old.

NEW beta release is not good for LISTVEW funcations.

you can test both scripts on new and old beta & decide.

Please post comments on it after testing both

Script for new beta

#include <GUIConstants.au3>
#Include <GuiListView.au3>
#Include <Date.au3>
$time1 = _NowTime(3)

GUICreate("listview items",220,250, 100,200,-1,$WS_EX_ACCEPTFILES)
GUISetBkColor (0x00E0FFFF); will change background color
$lable = GUICtrlCreateLabel ( "",-1,200, 200)
$lable1 = GUICtrlCreateLabel ( "",-1,220, 200)

$listview = GuiCtrlCreateListView ("col1  |col2|col3  ",10,10,200,150);,$LVS_SORTDESCENDING)
$button = GuiCtrlCreateButton ("Value?",75,170,70,20)
$item1=GuiCtrlCreateListViewItem("item2|col22|col23",$listview)
$item2=GuiCtrlCreateListViewItem("...item1|col12|col13",$listview)
$item3=GuiCtrlCreateListViewItem("item3|col32|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item400|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item4|col32|col33",$listview)
GuiCtrlCreateListViewItem("item200|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item300|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item50|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item70|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item90|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item1000|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item501|col32*|col33",$listview)
;$input1=GuiCtrlCreateInput("",20,200, 150)
GuiCtrlSetState(-1,$GUI_DROPACCEPTED); to allow drag and dropping
GuiSetState()
GUICtrlSetData($item2,"|ITEM1")
GUICtrlSetData($item3,"||COL33")

For $x = 16 To 600
    GUICtrlCreateListViewItem("item" & $x, $listview)
Next

Do
  $msg = GuiGetMsg ()
     
   Select
      Case $msg = $button
         _deleteduplicate()
      Case $msg = $listview
         MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
   EndSelect
Until $msg = $GUI_EVENT_CLOSE

Func _deleteduplicate()
$count = _GUICtrlListViewGetItemCount($listview)
$countY = _GUICtrlListViewGetItemCount($listview)
GUICtrlsetdata ($lable1 ,$count)    
     
;msgbox(0,"First cehck ","$count is " & $count & " $countY is " & $county)


    $x = 0
    Do  
        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        $y = $x
        Do
;msgbox(0,"Count of y", $countY)
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $Y, 0);,"listview items")
            GUICtrlsetdata ($lable ,"X is " & $x & " Y is " & $y)
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                If $x <> $Y Then
;msgbox(0,"Second check1 ","X is " & $x & " Y is " & $y)
                _GUICtrlListViewDeleteItem($listview, $Y)
                
        ;$y = $x
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                Else
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                $y = $y +1
                Endif
            Else
            $countY = _GUICtrlListViewGetItemCount($listview)
            $count = _GUICtrlListViewGetItemCount($listview)
            
            $y = $y +1
            Endif
        
        Until $y = $countY
;msgbox(0," ","$y is " & $y & " $countY is " & $countY)
    $x = $x +1
    Until $x = $count
    msgbox(0,"time","First time "  & $time1 & " Second time " & _NowTime(3))

EndFunc

Screen shot for time diffrence between both beta 126 & 127(With 127 LISTVIEW fliker problem is also there)

Edited June 24, 2006 by Rahul Rohela

Rahul Rohela · June 24, 2006

Now test the Beta 127 & 126 with fillowing script...

#include <GUIConstants.au3>
#Include <GuiListView.au3>
#Include <Date.au3>


GUICreate("listview items",220,250, 100,200,-1,$WS_EX_ACCEPTFILES)
GUISetBkColor (0x00E0FFFF) ; will change background color
$lable = GUICtrlCreateLabel ( "",-1,200, 200)
$lable1 = GUICtrlCreateLabel ( "",-1,220, 200)

$listview = GuiCtrlCreateListView ("col1  |col2|col3  ",10,10,200,150);,$LVS_SORTDESCENDING)
$button = GuiCtrlCreateButton ("Value?",75,170,70,20)
;$item1=GuiCtrlCreateListViewItem("item2|col22|col23",$listview)
;$item2=GuiCtrlCreateListViewItem("...item1|col12|col13",$listview)
;$item3=GuiCtrlCreateListViewItem("item3|col32|col33",$listview)
;GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item400|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item4|col32|col33",$listview)
;GuiCtrlCreateListViewItem("item200|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item300|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item50|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item70|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item500|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item200|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item300|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item400|col32*|col33",$listview)
;GuiCtrlCreateListViewItem("item101|col32*|col33",$listview)
;$input1=GuiCtrlCreateInput("",20,200, 150)
GuiCtrlSetState(-1,$GUI_DROPACCEPTED)  ; to allow drag and dropping
GuiSetState()
;GUICtrlSetData($item2,"|ITEM1")
;GUICtrlSetData($item3,"||COL33")

For $a = 1 To 100
    GUICtrlCreateListViewItem("item" & $a, $listview)
Next
$aa = 100
For $b = 101 To 200
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 200
For $b = 201 To 300
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 300
For $b = 301 To 400
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 400
For $b = 401 To 500
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 500
For $b = 501 To 600
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 600
For $b = 601 To 700
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 700
For $b = 701 To 800
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 800
For $b = 801 To 900
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 900
For $b = 901 To 1000
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1000
For $b = 1001 To 1100
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1100
For $b = 1101 To 1200
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1200
For $b = 1201 To 1300
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1300
For $b = 1301 To 1400
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1400
For $b = 1401 To 1500
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1500
For $b = 1501 To 1600
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1600
For $b = 1601 To 1700
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1700
For $b = 1701 To 1800
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1800
For $b = 1801 To 1900
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 1900
For $b = 1901 To 2000
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2000
For $b = 2001 To 2100
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2100
For $b = 2101 To 2200
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2200
For $b = 2201 To 2300
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2300
For $b = 2301 To 2400
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2400
For $b = 2401 To 2500
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
$aa = 2500
For $b = 2501 To 2600
    
    GUICtrlCreateListViewItem("item" & ($b-$aa), $listview)
Next
Do
  $msg = GuiGetMsg ()
     
   Select
      Case $msg = $button
         _deleteduplicate()
      Case $msg = $listview
         MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
   EndSelect
Until $msg = $GUI_EVENT_CLOSE

Func _deleteduplicate()
$time1 = _NowTime(3)
$count = _GUICtrlListViewGetItemCount($listview)
$countY = _GUICtrlListViewGetItemCount($listview)
GUICtrlsetdata ($lable1 ,$count)    
     
;msgbox(0,"First cehck ","$count is " & $count & " $countY is " & $county)


    $x = 0
    Do  
        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        $y = $x
        Do
        ;msgbox(0,"Count of y", $countY)
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $Y, 0);,"listview items")
            GUICtrlsetdata ($lable ,"X is " & $x & " Y is " & $y)
            ;If $x > 20 then
            ;Msgbox(0,"countgdsdh","X is " & $x & " Y is " & $y)
            ;Endif
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                If $x <> $Y Then
            ;msgbox(0,"Second check1 ","X is " & $x & " Y is " & $y)
                _GUICtrlListViewDeleteItem($listview, $Y)
                
            ;$y = $x
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                Else
                $countY = _GUICtrlListViewGetItemCount($listview)
                $count = _GUICtrlListViewGetItemCount($listview)
                $y = $y +1
                Endif
            Else
            $countY = _GUICtrlListViewGetItemCount($listview)
            $count = _GUICtrlListViewGetItemCount($listview)
            
            $y = $y +1
            Endif
        
        Until $y = $countY
        ;msgbox(0," ","$y is " & $y & " $countY is " & $countY)
    $x = $x +1
    Until $x = $count
    msgbox(0,"time","First time "  & $time1 & " Second time " & _NowTime(3))

EndFunc

Time diff result attached...

GaryFrost · June 24, 2006

in GuiListView.au3 is not working as expected
same code processing time increased by 6 more min that mean total 12 min to compile same 615 items.
Secomd whole Listvew is flikering with new beta not with old.
NEW beta release is not good for LISTVEW funcations.

1st item should start at 0 index of listview

2nd item for compare should start at ending index of listview and stop and 1st item + 1

this way deletes are starting at the end

your listview count will change everytime you delete an item

I believe this change in the dup function cuts your time in half.

Before you say something is bad, make sure you know how to use it.

Func _deleteduplicate()
    $time1 = _NowTime(3)
    $count = _GUICtrlListViewGetItemCount($listview)
    GUICtrlSetData($lable1, $count)
    
    For $x = 0 To $count - 1
        GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        For $y = _GUICtrlListViewGetItemCount($listview) - 1 To $x + 1 Step - 1
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $y, 0);,"listview items")
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                _GUICtrlListViewDeleteItem($listview, $y)
                $count -= 1
                GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
            EndIf
        Next
    Next
    MsgBox(0, "time", "First time " & $time1 & " Second time " & _NowTime(3))
    
EndFunc   ;==>_deleteduplicate

Skruge · June 25, 2006

btw- I hope you didn't type out all those lines...

Since you're not re-using $a, $aa, or $b you can populate your listview with these three lines:

For $aa = 0 To 2599 
        GUICtrlCreateListViewItem("item" & Mod($aa, 100) + 1, $listview)
Next

Rahul Rohela · June 26, 2006

1st item should start at 0 index of listview

2nd item for compare should start at ending index of listview and stop and 1st item + 1

this way deletes are starting at the end

your listview count will change everytime you delete an item

I believe this change in the dup function cuts your time in half.

Before you say something is bad, make sure you know how to use it.

Func _deleteduplicate()
    $time1 = _NowTime(3)
    $count = _GUICtrlListViewGetItemCount($listview)
    GUICtrlSetData($lable1, $count)
    
    For $x = 0 To $count - 1
        GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        For $y = _GUICtrlListViewGetItemCount($listview) - 1 To $x + 1 Step - 1
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $y, 0);,"listview items")
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                _GUICtrlListViewDeleteItem($listview, $y)
                $count -= 1
                GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
            EndIf
        Next
    Next
    MsgBox(0, "time", "First time " & $time1 & " Second time " & _NowTime(3))
    
EndFunc   ;==>_deleteduplicate

I am not good in programming... i am just using this to solve my day to day activities... i faced the problem with my script( Not good enough) so i post it here for help....

Second my little script performance difference in both beta was noticeable... so i put that here too..

gafrost every thing you mentioned was in my mind but you are Great.

Thx for all your help..

gafrost Check the same data with you script, with both 126 & 127 beta & time diffrence between both is 6 min... with 127 process is slow...

i am in great confusion that which one to use..

Edited June 26, 2006 by Rahul Rohela

randallc · June 26, 2006

Hi,

I think you could do this all a lot faster; are you prepared to sort the list first?

If so, you know when you have exhausted all the matches and can go quickly to the next item; I am not sure if this misses one or deletes an extra?; (using 126)

;ListViewExample126.au3
#include <GUIConstants.au3>
#Include <GuiListView.au3>
#Include <Date.au3>
$time1 = _NowTime(3)

GUICreate("listview items",220,250, 100,200,-1,$WS_EX_ACCEPTFILES)
GUISetBkColor (0x00E0FFFF); will change background color
$lable = GUICtrlCreateLabel ( "",-1,200, 200)
$lable1 = GUICtrlCreateLabel ( "",-1,220, 200)

$listview = GuiCtrlCreateListView ("col1  |col2|col3  ",10,10,200,150);,$LVS_SORTDESCENDING)
$button = GuiCtrlCreateButton ("Value?",75,170,70,20)
$item1=GuiCtrlCreateListViewItem("item2|col22|col23",$listview)
$item2=GuiCtrlCreateListViewItem("...item1|col12|col13",$listview)
$item3=GuiCtrlCreateListViewItem("item3|col32|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item400|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item4|col32|col33",$listview)
GuiCtrlCreateListViewItem("item200|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item300|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item3|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item50|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item70|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item90|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item1000|col32*|col33",$listview)
GuiCtrlCreateListViewItem("item501|col32*|col33",$listview)
;$input1=GuiCtrlCreateInput("",20,200, 150)
GuiCtrlSetState(-1,$GUI_DROPACCEPTED); to allow drag and dropping
GuiSetState()
GUICtrlSetData($item2,"|ITEM1")
GUICtrlSetData($item3,"||COL33")
Dim $B_DESCENDING[_GUICtrlListViewGetSubItemsCount ($listview) ]
For $x = 16 To 600
    GUICtrlCreateListViewItem("item" & $x, $listview)
Next
Do
  $msg = GuiGetMsg ()
     
   Select
      Case $msg = $button
        _GUICtrlListViewSort($listview,$B_Descending, 0)
        _deleteduplicate()
      Case $msg = $listview
         MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
   EndSelect
Until $msg = $GUI_EVENT_CLOSE
Func _deleteduplicate()
    $time1 = _NowTime(3)
    $count = _GUICtrlListViewGetItemCount($listview)
    GUICtrlSetData($lable1, $count)
    For $x =$count - 1 To 0 step -1
        GUICtrlSetData($lable, "X is " & $x & " New count " & $count)        
        $Itemtxt1 = _GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        do
if $x>0 then
            $Itemtxt2 = _GUICtrlListViewGetItemText($listview, $x- 1 , 0);,"listview items")
            If StringUpper($Itemtxt1) = StringUpper($Itemtxt2) Then
                _GUICtrlListViewDeleteItem($listview, $x- 1)
                $count -= 1
                $x -= 1
                GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
            EndIf
           EndIf
            until  StringUpper($Itemtxt1)<>StringUpper($Itemtxt2) or $x = 0
    Next
    MsgBox(0, "time", "First time " & $time1 & " Second time " & _NowTime(3))  
EndFunc

Does that get close? - just a few seconds to get all the dupes.

You could keep an array of the items, index them, and sort them again after if you really need the original order; let me know.

[i the sort is too slow, you can use my _GUICtrlListView.au3, from the link in my sig.. but probably not necessary for what you are doing]

Best, Randallc

[EDIT1 -missed " $x -= 1"]

[EDIT -missed "if $x>0 then"

"..."

"EndIf"]

[EDIT1 -missed " until StringUpper($Itemtxt1)<>StringUpper($Itemtxt2) or $x = 0"]

Attached is super-fast dupes removal; needs to include the _GUICtrlListView.au3 from link in my sig (another8x as fast; 2 secs total?)

Edited June 27, 2006 by randallc

Rahul Rohela · June 27, 2006

Hi,
I think you could do this all a lot faster; are you prepared to sort the list first?
If so, you know when you have exhausted all the matches and can go quickly to the next item; I am not sure if this misses one or deletes an extra?; (using 126)

[/autoit]Does that get close? - just a few seconds to get all the dupes.
You could keep an array of the items, index them, and sort them again after if you really need the original order; let me know.
[i the sort is too slow, you can use my _GUICtrlListView.au3, from the link in my sig.. but probably not necessary for what you are doing]
Best, Randallc
[EDIT1 -missed " $x -= 1"]
[EDIT -missed "if $x>0 then"
"..."
"EndIf"]
[EDIT1 -missed " until StringUpper($Itemtxt1)<>StringUpper($Itemtxt2) or $x = 0"]
Attached is super-fast dupes removal; needs to include the _GUICtrlListView.au3 from link in my sig (another8x as fast; 2 secs total?)

You have done great job..... With your first script Total time to (total 2600 items) remove duplicate item is 7Sec...

Previously it was 4 min

But your second script is slow... Please check if i am doing something wrong....

Test both attached scripts..

You could keep an array of the items, index them, and sort them again after if you really need the original order; let me know.

Please guide how i can implement this??

ListViewDupes.au3

Remove_Duplicate_items_list.au3

randallc · June 27, 2006

Hi,

You have done great job..... With your first script Total time to (total 2600 items) remove duplicate item is 7Sec...

I think you are being generous, as , if you include the sort, you are talking 49-63secs depending on the machine;

I agree the routine I gave was slow, as I only checked it when there were very few dupes to dlete; for the currnet script, I have it in about 4-5 secs [including sort!] depending on your machine (faster if NOT compiled on mine.. go figure!)

;ListViewDupes2.au3 0_8
;~ #include <GUIConstants.au3>
;~ #Include <GuiListView.au3>
;~ #Include <Date.au3>
#include <_GUICtrlListView.au3>
$time1 = _NowTime(3)
GUICreate("listview items",220,250, 100,200,-1,$WS_EX_ACCEPTFILES)
GUISetBkColor (0x00E0FFFF); will change background color
$lable = GUICtrlCreateLabel ( "",-1,200, 200)
$lable1 = GUICtrlCreateLabel ( "",-1,220, 200)
$listview = _GuiCtrlCreateListView ("col1  |col2|col3  ",10,10,200,150);,$LVS_SORTDESCENDING)
$button = GuiCtrlCreateButton ("Value?",75,170,70,20)
GuiCtrlSetState(-1,$GUI_DROPACCEPTED); to allow drag and dropping
GuiSetState()
$timerstamp0=TimerInit()
_LockAndWait3()
For $aa = 0 To 2599
    _GUICtrlCreateListViewItem("item" & Mod($aa, 100) + 1, $listview)
Next
_ResetLockWait3()
Do
    $msg = GuiGetMsg ()
    Select
        Case $msg = $button
            _deleteduplicate()
        Case $msg = $listview
            MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
            ;__GUICtrlListViewSort($listview,$B_DESCENDING, GUICtrlGetState($listview))
            ;MsgBox(0,"listview", "clicked="& GuiCtrlGetState($listview),2)
    EndSelect
Until $msg = $GUI_EVENT_CLOSE
Func _deleteduplicate()
    $time1 = _NowTime(3)
    local $ar_New[1]
    $timerstamp1=TimerInit()
    __GUICtrlListViewSort($listview,-1, 0)
    $count = _GUICtrlListViewGetItemCount($listview)
    GUICtrlSetData($lable1, $count)
    For $x =$count  To 1 step -1
        ;GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
        $Itemtxt1 = __GUICtrlListViewGetItemText($listview, $x, 0);,"listview items")
        do
            if $x>0 then
                $Itemtxt2 = __GUICtrlListViewGetItemText($listview, $x- 1 , 0);,"listview items")
                If StringUpper($Itemtxt1) == StringUpper($Itemtxt2) Then
                    $count -= 1
                    $x -= 1
                    ;GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
                EndIf
            EndIf
        until  StringUpper($Itemtxt1)<>StringUpper($Itemtxt2) or $x=0
        _ArrayAdd($ar_New,$Itemtxt1)
    Next
        GUICtrlSetData($lable, "X is " & $x & " New count " & $count)
    if  $ar_New[0]=="" then _ArrayDelete($ar_New,0)
    _ArrayReverse($ar_New)
    ConsoleWrite("b4refresh :"&round(TimerDiff($timerstamp1)) & " mseconds "&@lf)
    _refreshArray($ar_New)
    $time2 = _NowTime(3)
    ConsoleWrite("Finished :"&round(TimerDiff($timerstamp1)) & " mseconds "&@lf)
    MsgBox(0, "time", "Finished :"&round(TimerDiff($timerstamp1)) & " mseconds ")
    MsgBox(0, "time", "First time " & $time1 & " Second time " & $time2)
EndFunc   ;==>_deleteduplicate
Func _refreshArray(byref $ar_NewArray)
    _LockAndWait3()
    __GUICtrlListViewDeleteAllItems($listview)
    for $b=0 to UBound($ar_NewArray)-1
        _GUICtrlCreateListViewItem($ar_NewArray[$b],$listview)
    next
    _ResetLockWait3()
    _GUIListViewReDim()
EndFunc   ;==>_refreshArray

Best, Randall Edited June 27, 2006 by randallc

randallc · June 28, 2006

Hi,

Quicker again, and same speed (almost) whether there are few dupes or many;

attached.

Best, Randall

Rahul Rohela · July 7, 2006

Hi,
Quicker again, and same speed (almost) whether there are few dupes or many;
attached.
Best, Randall

Hi

This script is very fast but not removing all DUPlicates. Screen shot attached.

randallc · July 7, 2006

Hi,

Strange! - Works for me!, 100 result.

?Check you have latest version of "_GUICtrlListView.au3"? 0_80 at least?

Else we have a difference between machines; perhaps change your #include "_GUICtrlListView.au3" instead of using <>, to ensure using correct version?

Best, Randallc

Edited July 7, 2006 by randallc

Rahul Rohela · July 7, 2006

Using Beta 128 & download the latest _GUICtrlListView.au3 & Finally its going all well.. Thx for for your lightening fast script..

You could keep an array of the items, index them, and sort them again after if you really need the original order; let me know.

How to put it in original order after Sort & removal

randallc · July 10, 2006

Hi,

Now even faster; can you please test run the dupes8 for me?; needs _GUIctrlListview.au3 0_97 from sig link.

[sorry, I replied on wrong thread before... Randallc]

How to put it in original order after Sort & removal

-See "compare" thread

resortafter dupes

Edited July 10, 2006 by randallc

Rahul Rohela · July 12, 2006

Hi,
Now even faster; can you please test run the dupes8 for me?; needs _GUIctrlListview.au3 0_97 from sig link.

[sorry, I replied on wrong thread before... Randallc]
-See "compare" thread
resortafter dupes

Unbelievable.....

Click and result is there( for different combinations its between 200ms to 300ms)

You have done great job...

One more thing This code is very big but very useful.. If its possible please create UDF to Delete Listviewitems Duplicates & to Compare two ListViewitems ...

randallc · July 16, 2006

OK,

This is working for dupes from the main "_GUICtrlListView.au3" [link in signature.]"_108"

Example

"ListViewDupes10.au3"

==============================================================

"Compare will take longer; for practicality, the user may need to select which GUI to leave as fast-sorting after a "compare", at least initially; rather than all Listviews. [otherwise complictaed and item number in each list can only be 1/3 of 4000 approx; maybe 1/10th for safety, without heaps of slow programming]

Is this an issue?

Best, randall

Sign In

Remove Duplicate Listview items...

Recommended Posts

Rahul Rohela

Rahul Rohela

GaryFrost

Skruge

Rahul Rohela

randallc

Rahul Rohela

randallc

randallc

Rahul Rohela

randallc

Rahul Rohela

randallc

Rahul Rohela

randallc

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta