Jump to content
Sign in to follow this  
Jewtus

Diff two arrays with a progress bar

Recommended Posts

I'm working with two csv files that I'm parsing into two arrays. I'm then comparing them to find the duplication and remove them from the first array. This works great on 100 or so records, but I'm trying to compare arrays with more than 70,000 records so I wanted to add in a loading bar so I can tell how far/how much longer it will take.

This is my code:

ProgressSet(0,0&"%","Checking already searched")
$aProcess = _ParseCSV($oOutfile,"|","",0)
$aAlreadyChecked = _ParseCSV($AlreadyProcessed,"|","",0)
For $a = UBound($aProcess) -1 to 0 Step -1
                for $b = 0 to UBound($aAlreadyChecked) -1
                                if $aProcess[$a][0] = $aAlreadyChecked[$b][0] Then
                                                _ArrayDelete($aProcess, $a)
                                                MsgBox(0,"",($a-UBound($aProcess)) & @TAB & $b)
                                                ProgressSet(($b/$a),Round($b/$a)&"%","Cleaning up")
                                                ExitLoop
                                EndIf
                Next
Next

I cannot get the percentage logic to show anything that seems rational or accurate. Does anyone know of a more efficient way of doing this or how to fix the progressset to actually show how far in the process it already is?

Share this post


Link to post
Share on other sites

Is that working for you?

#include <Array.au3>

Global $array1[100000], $array2[111111], $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()
For $i = 0 To UBound($array1) - 1
    $array1[$i] = Random(0, 100000, 1)
    $array2[$i] = Random(0, 111111, 1)
Next
ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)

Global $aResult = ArrayCompare($array1, $array2)
ConsoleWrite(UBound($aResult) & @CRLF)
;~ _ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

I tried replacing the arrays but they are 2D arrays so I get an error "array variable has incorrect number of subscripts or subscript dimensions range exceeded"

What would I need to do it fix that? I tried this:

Global $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()

Global $aResult = ArrayCompare($aProcess, $aAlreadyChecked)
ConsoleWrite(UBound($aResult) & @CRLF)
_ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i][0])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i][0]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

which seems to function, but it doesn't seem to be able to see the difference in the two files. The result array ends up being a 1D version of the first array.

Share this post


Link to post
Share on other sites

Something that exists in both arrays

EX: 

Array1

[1,2,3,4]

Array2

[3,4,5,6]

I want to remove 3 and 4 from array1 because they exist in both lists.

Edited by Jewtus

Share this post


Link to post
Share on other sites

I want to eliminate the entire row if there is a match in the first column

 

I'm looking at search results and I'm comparing them to a new set of search results, but I'm trying to avoid doing more work on the results that I've already processed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • By 3Jake
      Hello Smart People!
      I have a multi-column .CSV that I would like to draw information from, in order to populate (send) that info to some fields in an Oracle form.  This would be for account-creation in Oracle.

      I have the basic script to navigate (tab) through the fields in the Oracle form, but I am SUPER-new to AutoIT and don’t quite see how to get the script to import and use variables from a .CSV

      For example:

      My script just waits for the page to be active, then enters data and tabs between fields like this:

      ; Wait for the window to be active WinWaitActive("Oracle is Cool - E-Business is the Best") ; User Name Send("jsmith") Send("{TAB}") ; Password Send("Password1") Send("{TAB}") Send("Password1") Send("{TAB}") ; Description Send("John Smith") Send("{TAB}") (etc. etc. etc.…)

      The winning solution would take one row at a time, feed it into fields on the Oracle form, save, and then start in on the next row

      It looks like I’d need a combination of “FileReadToArray” ( or“FileReadLine”?) and “StringSplit” to loop through the values and set them to variables, which would then replace my current "hard-coded" values to be typed them into the form?

      I found the following example in the “Help” snippets for stringsplit()… this seems like it’s in the ballpark but I’m having some trouble wrapping this around what I’ve put together, since each entry in the array would have multiple elements per line instead of just “Day”:

      Func Example() Local $aDays = StringSplit("Mon,Tues,Wed,Thur,Fri,Sat,Sun", ",") ;Split the string of days using the delimiter "," and the default flag value. #cs The array returned will contain the following values: $aDays[1] = "Mon" $aDays[2] = "Tues" $aDays[3] = "Wed" ... $aDays[7] = "Sun" #ce For $i = 1 To $aDays[0] ; Loop through the array returned by StringSplit to display the individual values. MsgBox($MB_SYSTEMMODAL, "", "$aDays[" & $i & "] - " & $aDays[$i]) Next EndFunc ;==>Example

      This post also seems like a similar example, but I don’t need it to be so selective RE: one specific column:
      https://www.autoitscript.com/forum/topic/166261-how-to-read-csv-specific-row-and-columns


      Any advice would be appreciated!  Thanks for taking a minute to look!  
    • By Blitzkid
      Hello, i want to search several directories for files with the largest numbers behind them (Like "video123") . They dont have a datatype. But there are also files with longer names and datatypes in these folders (Like "video778.mp4"). Is it possible to filter the _FileListToArray Syntax from
      to smth. like
       
      Here is my Code
      #include <ButtonConstants.au3> #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> #include <array.au3> #include <File.au3> $filedir = @ScriptDir & "\" _checkfile() Func _checkfile() ConsoleWrite("______________________" & @CRLF) Local $arr[3] = ["music", "picture", "video"] For $i = 0 To UBound($arr) - 1 Local $arrayfiles = _FileListToArray($filedir & $arr[$i], $arr[$i] & "*", 1) If @error = 1 Then ConsoleWrite($arr[$i] & "Error 1") EndIf If @error = 4 Then ConsoleWrite($arr[$i] & "Error 2") ;Exit EndIf $arrayfilter = _ArrayMax($arrayfiles, 0, 1) Global $stringfiles = StringReplace($arrayfilter, $arr[$i], "") ConsoleWrite($arrayfilter & @CRLF) Next EndFunc ;==>_checkfile  
    • By jmp
      Script running good but error in line 7.
      When i run this script :
      #include <IE.au3> #include <Array.au3> $oIE = _IEAttach ("Shop") $oTable = _IETableGetCollection ($oIE, 1) $aTableData = _IETableWriteToArray ($oTable) For $inumber = 1 To UBound($aTableData) -1 $table = $aTableData[4][$inumber] MsgBox(0, "", $table) Next I got Error: array variable has incorrect number of subscripts or subscript dimension range exceeded
    • By nacerbaaziz
      Good morning guys, i hope that you're all well.
      guys, i have a problem and i hope that you can help me
      i've created an 3d array
      the array Contain a Categories info
      as folow
      $array[n][0][0] = Categorie name
      $array[n][0][1] = Categorie file path
      $array[n][0][2] = Categorie contents number
      $array[n][m][0] = link name
      $array[n][m][1] = link url
      $array[n][m][2] = link section name
      in my tool i want to add an option to delete a Category
      as you know the Categorie mean that must delete a region from the array
      when i tried to use _arrayDelete
      with the 2d array it work well
      but here i couldn't find any way to do that, can any one help me please?
      thanks in advance.
    • By nacerbaaziz
      hello sirs, please help me
      i tried to create a function that read a folder files to 3d array
      e.g
      $array[n][0][0] = ctName
      $array[n][0][1] = ctFilePath
      $array[n][0][2] = crtsections number
      $array[n][m][0] = KeyName
      $array[n][m][1] = KeyVal
      $array[n][m][2] = keySectionName
       
      that the array
      when i put one file into the folder all things work fine
      but when i put more than one file
      the last file worked fine but the others only the first key is showing
      please can you help me to correct this problem
      here is the example with the folder
      please accept my greetings
      and thanks in advance
       
      array3d.zip
×
×
  • Create New...