Jump to content
Sign in to follow this  
Jewtus

Diff two arrays with a progress bar

Recommended Posts

Jewtus

I'm working with two csv files that I'm parsing into two arrays. I'm then comparing them to find the duplication and remove them from the first array. This works great on 100 or so records, but I'm trying to compare arrays with more than 70,000 records so I wanted to add in a loading bar so I can tell how far/how much longer it will take.

This is my code:

ProgressSet(0,0&"%","Checking already searched")
$aProcess = _ParseCSV($oOutfile,"|","",0)
$aAlreadyChecked = _ParseCSV($AlreadyProcessed,"|","",0)
For $a = UBound($aProcess) -1 to 0 Step -1
                for $b = 0 to UBound($aAlreadyChecked) -1
                                if $aProcess[$a][0] = $aAlreadyChecked[$b][0] Then
                                                _ArrayDelete($aProcess, $a)
                                                MsgBox(0,"",($a-UBound($aProcess)) & @TAB & $b)
                                                ProgressSet(($b/$a),Round($b/$a)&"%","Cleaning up")
                                                ExitLoop
                                EndIf
                Next
Next

I cannot get the percentage logic to show anything that seems rational or accurate. Does anyone know of a more efficient way of doing this or how to fix the progressset to actually show how far in the process it already is?

Share this post


Link to post
Share on other sites
UEZ

Is that working for you?

#include <Array.au3>

Global $array1[100000], $array2[111111], $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()
For $i = 0 To UBound($array1) - 1
    $array1[$i] = Random(0, 100000, 1)
    $array2[$i] = Random(0, 111111, 1)
Next
ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)

Global $aResult = ArrayCompare($array1, $array2)
ConsoleWrite(UBound($aResult) & @CRLF)
;~ _ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

Br,

UEZ

Edited by UEZ
  • Like 1

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites
Jewtus

I tried replacing the arrays but they are 2D arrays so I get an error "array variable has incorrect number of subscripts or subscript dimensions range exceeded"

What would I need to do it fix that? I tried this:

Global $i, $t, $fProgress
ConsoleWrite("Creating test array... ")
$t = TimerInit()

Global $aResult = ArrayCompare($aProcess, $aAlreadyChecked)
ConsoleWrite(UBound($aResult) & @CRLF)
_ArrayDisplay($aResult)


Func ArrayCompare(ByRef $a1, $a2)
    ConsoleWrite("Sorting 2nd array... ")
    Local $t = TimerInit()
    _ArraySort($a2)
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF)
    Local $i, $c = 0, $iUB = UBound($a1) > UBound($a2) ? UBound($a1) : UBound($a2), $aNew[$iUB], $iUB = UBound($a1) - 1
    ConsoleWrite("Searching " & $iUB & " elements in " & UBound($a2) - 1 & " elements ... ")
    AdlibRegister("Show_Progress", 500)
    $fProgress = 0
    ProgressOn("Progress Meter", "Be patient, searching for duplicates...", "0%")
    $t = TimerInit()
    For $i = 0 To $iUB
        If _ArrayBinarySearch($a2, String($a1[$i][0])) > -1 Then
            ContinueLoop
        Else
            $aNew[$c] = $a1[$i][0]
            $c += 1
        EndIf
        $fProgress = $i / $iUB * 100
    Next
    ReDim $aNew[$c]
    ConsoleWrite("done in " & Round(TimerDiff($t), 2) & " ms." & @CRLF & @CRLF)
    AdlibUnRegister("Show_Progress")
    ProgressOff()
    Return $aNew
EndFunc

Func Show_Progress()
    ProgressSet($fProgress, StringFormat("%.2f %", $fProgress))
EndFunc

which seems to function, but it doesn't seem to be able to see the difference in the two files. The result array ends up being a 1D version of the first array.

Share this post


Link to post
Share on other sites
kylomas

Jewtus,

I'm then comparing them to find the duplication

 

Please define what you mean by "duplication"...

kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites
Jewtus

Something that exists in both arrays

EX: 

Array1

[1,2,3,4]

Array2

[3,4,5,6]

I want to remove 3 and 4 from array1 because they exist in both lists.

Edited by Jewtus

Share this post


Link to post
Share on other sites
kylomas

Jewtus,

These are 1D arrays.  In post #3 you allude to a 2D aray.  Do you want to eliminate dups anywhere they exist, or, only by column?

kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites
Jewtus

I want to eliminate the entire row if there is a match in the first column

 

I'm looking at search results and I'm comparing them to a new set of search results, but I'm trying to avoid doing more work on the results that I've already processed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • rm4453
      By rm4453
      Hello,
       
      I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.
      I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.
      Any help at all would be very much appreciated!
       
      *EDIT --> The array I am left with after parsing is $array[0-50000][16]
    • TrashBoat
      By TrashBoat
      So Im trying to make a simple 2d game and make some sort of collision detection so why not to make a 2 dimensional array but i have no clue how  to write it in multiple lines
      Global $map[5,5] = [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0] something like this but it doesn't work
    • Zein
      By Zein
      #include "..\Include\Array.au3" #include "..\Include\File.au3" #include "..\Include\AutoItConstants.au3" Local $aRetArray Local $sFilePath = "n.csv" _FileReadToArray($sFilePath, $aRetArray, ",") ; _FileReadToArray($sFilePath, $aRetArray, $FRTA_COUNT, ",") _ArrayDisplay($aRetArray, "Original", Default, 8) The above code shows two versions of _FileReadToArray and both don't work as expected.
      The first one doesn't use the comma as a delimiter. (so I get a single column array)  I tried adding "Default" between $aRetArray and "," then it told me it had an incorrect number of parameters. 

      I looked again at the documentation:
       
      #include <File.au3> _FileReadToArray ( $sFilePath, ByRef $vReturn [, $iFlags = $FRTA_COUNT [, $sDelimiter = ""]] )
      And I with or without the flags params I should be getting a 2D array due to my file being a csv. 
      I then tried a regular flag, $FRTA_COUNT, and it tells me that I'm using a variable $FRTA_COUNT while it's not declared. Tried putting in 1 instead and it told me again, incorrect number of params. 

       
    • ternal
      By ternal
      Hi,
      Recently I have had the need to do a sort and then do a second sort while the item of the first sort stays the same ( double sorting , first on column x then while column x is the same sort column y).
      I did not put much efffort into error checking but so far I did not need it.
      For my applications so far it works perfectly however if someone is willing I want to test this extensivly.
      If anyone has big lists of random stuff to sort could you try this out please?
      #include <Array.au3> ; #FUNCTION# ==================================================================================================================== ; Name ..........: _ArraySort_Double ; Description ...: ; Syntax ........: _ArraySort_Double (Byref $array[, $first_index = Default[, $second_index = Default[, $ascending = Default]]]) ; Parameters ....: $array - 2d array to sort. ; $first_index - [optional] first column to sort. Default is 0. ; $second_index - [optional] second column to sort. Default is 1. ; $ascending - [optional] ascending/descending. Default is 1. ; Return values .: 1 if no errors occured , -1 if errors occured ; Author ........: Ternal ; Remarks .......: Needs excessive testing. ; Related .......: _arraysort() ; =============================================================================================================================== Func _ArraySort_Double (byref $array, $first_index = Default, $second_index = Default, $ascending = Default) Local $temp_value Local $counter = 1 If UBound($array, $UBOUND_DIMENSIONS) <> 2 Then MsgBox(0, "error", "error") return -1 EndIf If $first_index = Default Then $first_index = 0 If $second_index = Default Then $second_index = 1 If $ascending = Default Then $ascending = 1 _ArraySort($array, $ascending, 0, 0, $first_index); you can alter settings of primary sort here If @error Then MsgBox(0, "error", @error) return -1 EndIf $temp_value = $array[0][$first_index] For $x = 1 to UBound($array, 1) - 1 If Mod( $x, 10000) = 0 Then ConsoleWrite("at " & $x & " of a total : " & UBound($array, 1) & @CRLF) If $array[$x][$first_index] = $temp_value Then $counter+= 1 If $x = UBound($array, 1) - 1 Then; do last line here(if last line is not a new item) _ArraySort($array, $ascending, $x - $counter, $x, $second_index);you can alter settings of secondary sort here(don't forget to place line 34 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf EndIf Else If $counter > 0 Then ;at least 2 of the same _ArraySort($array, $ascending, $x - $counter, $x - 1, $second_index);you can alter settings of secondary sort here(don't forget to place line 29 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf $counter = 1 EndIf EndIf $temp_value = $array[$x][$first_index] Next Return 1 EndFunc Kind regards, Ternal
    • TrashBoat
      By TrashBoat
      So I've made this script that detects how long i have held down my left mouse button for and stores the information in an array and then sorts its using _ArraySort but the output is half sorted half broken.
      Here's my script:
      HotKeySet("{F1}","_exit") #include <Misc.au3> #include <Timers.au3> #include <Array.au3> Local $dll = DllOpen("user32.dll") $on = False Global $array[0] While(1) If _IsPressed(01,$dll) Then $timer = _Timer_Init() While _IsPressed(01,$dll) Sleep(1) WEnd $time = _Timer_Diff($timer) _ArrayAdd($array,"Time: " & Floor($time) & " ms") ;~ ConsoleWrite("Time: " & Floor($time) & " ms" & @CRLF) EndIf Sleep(50) WEnd Func _exit() _ArraySort($array) _ArrayDisplay($array) Exit EndFunc And the output:

      See how its not sorted?  What is the problem here?
×