Sign in to follow this  
Followers 0
Jewtus

Diff two arrays with a progress bar

7 posts in this topic

I'm working with two csv files that I'm parsing into two arrays. I'm then comparing them to find the duplication and remove them from the first array. This works great on 100 or so records, but I'm trying to compare arrays with more than 70,000 records so I wanted to add in a loading bar so I can tell how far/how much longer it will take.

This is my code:

ProgressSet(0,0&"%","Checking already searched")
$aProcess = _ParseCSV($oOutfile,"|","",0)
$aAlreadyChecked = _ParseCSV($AlreadyProcessed,"|","",0)
For $a = UBound($aProcess) -1 to 0 Step -1
                for $b = 0 to UBound($aAlreadyChecked) -1
                                if $aProcess[$a][0] = $aAlreadyChecked[$b][0] Then
                                                _ArrayDelete($aProcess, $a)
                                                MsgBox(0,"",($a-UBound($aProcess)) & @TAB & $b)
                                                ProgressSet(($b/$a),Round($b/$a)&"%","Cleaning up")
                                                ExitLoop
                                EndIf
                Next
Next

I cannot get the percentage logic to show anything that seems rational or accurate. Does anyone know of a more efficient way of doing this or how to fix the progressset to actually show how far in the process it already is?

Share this post


Link to post
Share on other sites

#2 ·  Posted (edited)

Is that working for you?

#include <Array.au3>

Global $array1[100000], $array2[111111], $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()
For $i = 0 To UBound($array1) - 1
    $array1[$i] = Random(0, 100000, 1)
    $array2[$i] = Random(0, 111111, 1)
Next
ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)

Global $aResult = ArrayCompare($array1, $array2)
ConsoleWrite(UBound($aResult) & @CRLF)
;~ _ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

Br,

UEZ

Edited by UEZ
1 person likes this

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

I tried replacing the arrays but they are 2D arrays so I get an error "array variable has incorrect number of subscripts or subscript dimensions range exceeded"

What would I need to do it fix that? I tried this:

Global $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()

Global $aResult = ArrayCompare($aProcess, $aAlreadyChecked)
ConsoleWrite(UBound($aResult) & @CRLF)
_ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i][0])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i][0]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

which seems to function, but it doesn't seem to be able to see the difference in the two files. The result array ends up being a 1D version of the first array.

Share this post


Link to post
Share on other sites

Jewtus,

I'm then comparing them to find the duplication

 

Please define what you mean by "duplication"...

kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Something that exists in both arrays

EX: 

Array1

[1,2,3,4]

Array2

[3,4,5,6]

I want to remove 3 and 4 from array1 because they exist in both lists.

Edited by Jewtus

Share this post


Link to post
Share on other sites

Jewtus,

These are 1D arrays.  In post #3 you allude to a 2D aray.  Do you want to eliminate dups anywhere they exist, or, only by column?

kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

I want to eliminate the entire row if there is a match in the first column

 

I'm looking at search results and I'm comparing them to a new set of search results, but I'm trying to avoid doing more work on the results that I've already processed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • Ascer
      By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1+ System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
    • FMS
      By FMS
      Hello,
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
      TEST_labels.idx1-ubyte
    • ur
      By ur
      I am reading a CSV file which is tab seperated as below.
      Local $aArray = FileReadToArray($file) And now, I am splitting this main array record wise so that Array contains internally another arrow to represent each row.
      For $i = 0 to (UBound($aArray) - 1) ;MsgBox(0,"",$aArray[$i]) $aArray[$i] = StringSplit(StringStripCR($aArray[$i]), Chr(9),2);Chr(9) for tab ;_ArrayDisplay($aArray[$i]) Next Afther that, _ArrayDIsplay is able to see the individual internal arrays.
      _ArrayDisplay($aArray[1]) But If I try to access the individual element of it as below.It is not showing any result.
      MsgBox(0,"",$aArray[1][1]) Any suggestion, below is the sample csv file.
      New Text Document.csv
    • Jibsbrown
      By Jibsbrown
      Need some help understanding why the ConsoleWrite works inside 2nd For loop but not out side. Between Audit Wiki, Help file , Forum searching (lots of code reading), and YouTube ( shout out to TutsTeach), I have not been able to find the reason why. 
      $sIniPath = "installLog.ini" ; - Get section name $iniSctionNames = IniReadSectionNames($sIniPath) ; - Get Keys and Vaules For $a = 1 to UBound($iniSctionNames) - 1 $keys = IniReadSection($sIniPath , $iniSctionNames[$a]) For $b = 1 to UBound($keys) - 1 $oldSysInfo = IniRead($sIniPath , $iniSctionNames[1], $keys[$b][0], "") $PntIPInfo = IniRead($sIniPath , $iniSctionNames[2], $keys[$b][0], "") $NewPCInfor = IniRead($sIniPath , $iniSctionNames[3], $keys[$b][0], "") ;ConsoleWrite($oldSysInfo & @LF) Next ;ConsoleWrite($oldSysInfo & @LF) Next ConsoleWrite($oldSysInfo) My intention is to use the variables later for Listboxes. Any explanation, forum post links or whatever would help. Sorry also very very new to Autoit.
      Also here's the ini file.
      [OldSysInfo] 4=192.168.0.4|DESKTOP-RDIU2SN|R90M05Q8 5=192.168.0.5|SD0123456789101|R9WGP9P 6=192.168.0.6|SD0123456789102|R9WGP9PT 3=192.168.0.3|DESKTOP-3RS4LKL|R9WGP9P 23=192.168.0.23|SD0123456789102|MXL1234P5I [PrinterIp] 50=192.168.0.50 48=192.168.0.48 47=192.168.0.47 [NewSysInfo] newPC = SD0123456789adfs|192.168.0.185|2UA1234FTR Thank you for your time.
    • amphoric
      By amphoric
      Hi,
      I am struggling converting my 1D array to a 2D array and then showing that in my List View. Below is my current code, where I would like to take the array from the input box, and have that displayed as a list within the list view this is a list of computer names, and then update the 2nd column of each line once I have ran a ping and a registry change on each computer name, with either 'Completed' and 'Failed'
      I have checked out the AutoIt Arrays page here: https://www.autoitscript.com/wiki/Arrays , but unfortunately I cannot get my head around it. Any help would be greatly appreciated.
      I hope this makes sense.
      Thanks
       
      #include <Date.au3> #include <ButtonConstants.au3> #include <ComboConstants.au3> #include <EditConstants.au3> #include <GUIConstantsEx.au3> #include <StaticConstants.au3> #include <WindowsConstants.au3> #include <Array.au3> #include <MsgBoxConstants.au3> #include <GUIListBox.au3> #include <GuiListView.au3> #Region ### $Form1 = GUICreate("Title", 515, 533, -1, -1) GUISetBkColor(0xFFFFFF) $Pic2 = GUICtrlCreatePic("Images\pic.jpg", 16, 16, 212, 28) $Label1 = GUICtrlCreateLabel("Label", 240, 16, 271, 33, $SS_CENTER) GUICtrlSetFont(-1, 17, 800, 0, "") $Combo1 = GUICtrlCreateCombo("", 256, 406, 241, 25, BitOR($GUI_SS_DEFAULT_COMBO,$CBS_SIMPLE)) $Label2 = GUICtrlCreateLabel("Please select the location", 45, 401, 210, 20, $SS_CENTER) GUICtrlSetFont(-1, 10, 800, 0, "") $ChangeButton = GUICtrlCreateButton("Change", 286, 487, 75, 25) $RePromptButton = GUICtrlCreateButton("Re-prompt User", 366, 487, 139, 25) $Checkbox1 = GUICtrlCreateCheckbox("I have ensured that my selection is correct.", 10, 487, 273, 25) GUICtrlSetFont(-1, 10, 400, 0, "") $ManLocation = GUICtrlCreateLabel("Please enter the location:", 59, 448, 189, 20, $SS_RIGHT) GUICtrlSetFont(-1, 10, 800, 0, "") GUICtrlSetState(-1, $GUI_HIDE) $ManLocInput = GUICtrlCreateInput("", 256, 445, 241, 24) GUICtrlSetState(-1, $GUI_HIDE) $Label5 = GUICtrlCreateLabel("for this device:", 138, 418, 109, 20, $SS_CENTER) GUICtrlSetFont(-1, 10, 800, 0, "") $Label3 = GUICtrlCreateLabel("Asset Number:", 98, 72, 107, 20, $SS_CENTER) GUICtrlSetFont(-1, 10, 800, 0, "") $Input1 = GUICtrlCreateInput("", 215, 69, 241, 24) $AddMulti = GUICtrlCreateLabel("Add Multiple Assets", 356, 55, 98, 13, $SS_RIGHT) GUICtrlSetFont(-1, 8, 400, 6, "") GUICtrlSetColor(-1, 0x808080) GUICtrlSetCursor (-1, 0) $List1 = GUICtrlCreateListView("", 16, 104, 481, 278) $Add = GUICtrlCreateButton("Add", 461, 68, 35, 25) GUISetState(@SW_SHOW) #EndRegion ### ;### StartForm ### $FormaMultiple = GUICreate("Title", 277, 418, -1, -1) GUISetBkColor(0xFFFFFF) $Label1 = GUICtrlCreateLabel("Please add 1 asset number per line.", 8, 11, 253, 20, $SS_CENTER) GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif") $OK = GUICtrlCreateButton("OK", 56, 375, 75, 25) $Edit1 = GUICtrlCreateEdit("", 20, 40, 233, 329) $Cancel = GUICtrlCreateButton("Cancel", 136, 375, 75, 25) GUISetState(@SW_HIDE) ;### EndForm ### ;### Add List View Columns START ### ;### Add List View Columns END ### ;### Read Locations START ### ;Read the LOCAL ini file SECTIONS Global $aSections = IniReadSectionNames("Locations.ini") ;If the above succeeded, convert the array to a string with each item separated by a | (pipe) and set the default selected item to $aSections[1] If(Not @Error) Then GUICtrlSetData($Combo1, _ArraytoString($aSections, "|", 1), $aSections[1]) ;### Read Locations END ### ;### Require Admin START ### ;#RequireAdmin ;### Require Admin END ### While 1 $nMsg = GUIGetMsg() ;### Data Validation START ### ;Set Checkbox = ENABLE/DISABLE if not GUICtrlRead($Combo1) = "" AND GUICtrlRead($Combo1) = "Not In List" then if not GUICtrlRead($ManLocInput) = "" Then If not ControlCommand($Form1, "", $Checkbox1, "IsEnabled") Then GUICtrlSetState($Checkbox1, $GUI_ENABLE) Else If ControlCommand($Form1, "", $Checkbox1, "IsEnabled") Then GUICtrlSetState($Checkbox1, $GUI_DISABLE) EndIf Else if GUICtrlRead($Combo1) = "" Then If ControlCommand($Form1, "", $Checkbox1, "IsEnabled") Then GUICtrlSetState($Checkbox1, $GUI_DISABLE) Else If not ControlCommand($Form1, "", $Checkbox1, "IsEnabled") Then GUICtrlSetState($Checkbox1, $GUI_ENABLE) EndIf EndIf ;Set Button = ENABLE/DISABLE If GUICtrlRead($Checkbox1) = 1 Then If Not ControlCommand($Form1, "", $ChangeButton, "IsEnabled") Then GUICtrlSetState($ChangeButton, $GUI_ENABLE) Else If ControlCommand($Form1, "", $ChangeButton, "IsEnabled") Then GUICtrlSetState($ChangeButton, $GUI_DISABLE) EndIf ;### Data Validation END ### Switch $nMsg Case $GUI_EVENT_CLOSE Exit case $Add Case $AddMulti GUISetState(@SW_SHOW, $FormaMultiple) Case $Cancel GUISetState(@SW_HIDE, $FormaMultiple) Case $OK $sText = GUICtrlRead($Edit1) $aText = StringSplit($sText, @CRLF, 1) $bText = _ArrayToString($aText, ";", 1, -1) $cText = StringStripWS($bText, $STR_STRIPALL ) $dText = StringReplace($cText,";;",";") GUICtrlSetData($Input1, $dText) GUICtrlSetData($Edit1, "") GUISetState(@SW_HIDE, $FormaMultiple) For $i = 1 To $aText[0] Next Case $Combo1 ; If the combo is actioned Switch GUICtrlRead($Combo1) Case "Not In List" ; Show the manual input controls GUICtrlSetState($ManLocation, $GUI_SHOW) GUICtrlSetState($ManLocInput, $GUI_SHOW) if ControlCommand($Form1, "", $Checkbox1, "IsChecked") then ControlCommand($Form1, "", $Checkbox1, "Uncheck") GUICtrlSetData($ManLocInput, "") Case Else ; Hide them GUICtrlSetState($ManLocation, $GUI_HIDE) GUICtrlSetState($ManLocInput, $GUI_HIDE) if ControlCommand($Form1, "", $Checkbox1, "IsChecked") then ControlCommand($Form1, "", $Checkbox1, "Uncheck") GUICtrlSetData($ManLocInput, "") EndSwitch Case $ChangeButton ChangeReg() Case $RePromptButton RemoveReg() EndSwitch WEnd Func ChangeReg() $MultiAssets = StringSplit(GUICtrlRead($Input1), ";", 1) For $i = 1 To $MultiAssets[0] ; Ping with a timeout of 250ms. Local $iPing = Ping($MultiAssets[$i], 250) If $iPing Then ; If a value greater than 0 was returned then do the following: if not GUICtrlRead($ManLocInput) = "" Then RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "Location", "REG_SZ", "# - " & GUICtrlRead($ManLocInput)) RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "Location", "REG_SZ", "# - " & GUICtrlRead($ManLocInput)) RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "LocationLog", "REG_SZ", @MDAY & "/" & @MON & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC) RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "LocationLog", "REG_SZ", @MDAY & "/" & @MON & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC) MsgBox($MB_SYSTEMMODAL, "Completed", "Completed") Else If not GUICtrlRead($Input1) = "" Then RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "Location", "REG_SZ", GUICtrlRead($Combo1)) RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "Location", "REG_SZ", GUICtrlRead($Combo1)) RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "LocationLog", "REG_SZ", @MDAY & "/" & @MON & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC) RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "LocationLog", "REG_SZ", @MDAY & "/" & @MON & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC) MsgBox($MB_SYSTEMMODAL, "Completed", $MultiAssets[$i] & @CRLF & " Changed to: " & GUICtrlRead($Combo1)) Else MsgBox($MB_SYSTEMMODAL, "Error", "Please ensure all boxes are completed") EndIf EndIf Else MsgBox($MB_SYSTEMMODAL, "Error", "This asset may be OFFLINE, unable to complete") EndIf Next EndFunc ;==>PingReg Func RemoveReg() $MultiAssets = StringSplit(GUICtrlRead($Input1), ";", 1) For $i = 1 To $MultiAssets[0] ; Ping with a timeout of 250ms. Local $iPing = Ping($MultiAssets[$i], 250) If $iPing Then ; If a value greater than 0 was returned then do the following: if not GUICtrlRead($ManLocInput) = "" Then RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "Location", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "Location", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "LocationLog", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "LocationLog", "REG_SZ", "") MsgBox($MB_SYSTEMMODAL, "Completed", "Completed") Else If not GUICtrlRead($Input1) = "" Then RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "Location", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "Location", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM\SOFTWARE\Location", "LocationLog", "REG_SZ", "") RegWrite("\\" & $MultiAssets[$i] & "\HKLM64\SOFTWARE\Location", "LocationLog", "REG_SZ", "") MsgBox($MB_SYSTEMMODAL, "Completed", $MultiAssets[$i] & @CRLF & "CLEARED") Else MsgBox($MB_SYSTEMMODAL, "Error", "Please ensure all boxes are completed") EndIf EndIf Else MsgBox($MB_SYSTEMMODAL, "Error", "This asset may be OFFLINE, unable to complete") EndIf Next EndFunc ;==>PingReg1