czardas

_ArrayUniqueR

17 posts in this topic

#1 ·  Posted (edited)

This seems currently a hot topic. Anyway I have been thinking of trying out this idea for a while. This is in experimental stage and uses an old idea of declaring uniquely named variables followed by testing for their existence. I always thought it was an interesting approach. You can set the second parameter to case sensitive. R stands for Row. It returns either a one dimensional array or a two dimensional array of unique rows. Three datatypes are ignored: namely nested arrays, objects and DLLstructs. If someone knows a way to test these items quickly for uniqueness, or represent the data as binary then please enlighten me. Rows containing these items are simply just left in place. I'm not sure how efficient or what limitations this code has. Testing begins in the morning - oh, it already is morning. Meanwhile you might like to have a play with it.

; #FUNCTION# ===================================================================================================================
; Name...........: _ArrayUniqueR
; Description ...: Removes duplicate items (or rows) from a one (or two) dimensional array.
; Syntax.........: _ArrayUniqueR($aArray [, $bCaseSense = False])
; Parameters.....; $aArray - The array containing duplicate data to be removed.
; Return values .: Success - Returns the unique array [ByRef].
;                  $bCaseSense - [Optional] Set to true for case sensitive matches. Default value = False
;                  Failure sets @error as follows
;                  |@error = 1 1st parameter is not an array.
;                  |@error = 2 1st parameter has too many dimensions.
; Author ........: czardas
; Comments ......; Works with strings, numbers, binary, boolean values, pointers, functions and keywords (Null and Default).
;                  Integers of the same magnitude are treated as duplicates regardless of data type.
;                  All elements within a row must be duplicated (in the same positions) before removal takes place.
;                  This function does not remove rows containing objects, DLLStructs or other arrays.
;                  If you modify _ArrayUniqueR, avoid using variable names that could collide with internally generated names.
; ==============================================================================================================================

Func _ArrayUniqueR(ByRef $aArray, $bCaseSense = Default)
    If Not IsArray($aArray) Then Return SetError(1) ; not an array

    Local $iDim = UBound($aArray, 0)
    If $iDim > 2 Then Return SetError(2) ; wrong number of dimensions

    Local $iRows = UBound($aArray)
    If $iRows < 2 Then Return ; array is already unique

    If $bCaseSense = Default Then $bCaseSense = False

    Local $iCols = ($iDim = 1) ? 1 : UBound($aArray, 2), _
    $iItems = 0, $aFunction[1] = [0], $vElement, $iInt, $sName

    ; for each item, or row, generate unique names using word characters only
    For $i = 0 To $iRows -1
        $sName = '' ; clear previous name
        For $j = 0 To $iCols -1
            $vElement = ($iDim = 1) ? $aArray[$i] : $aArray[$i][$j] ; get the data contained in each element

            ; use non-hexadecimal characters (other than x) as datatype identifiers and convert the data to hexadecimal where necessary
            Switch VarGetType($vElement)
                Case 'String'
                    If Not $bCaseSense Then $vElement = StringUpper($vElement) ; generates a case insensitive name segment
                    $sName &= 's' & StringToBinary($vElement, 4) ; UTF8 [$SB_UTF8]
                Case 'Int32', 'Int64' ; use decimal without conversion
                    ; the minus sign of a negative integer is replaced with 'g': to distinguish it from the positive value
                    $sName &= ($vElement < 0) ? 'g' & StringTrimLeft($vElement, 1) : 'i' & $vElement
                Case 'Double' ; may be an integer
                    $iInt = Int($vElement)
                    $sName &= ($vElement = $iInt) ? (($iInt < 0) ? 'g' & StringTrimLeft($iInt, 1) : 'i' & $iInt) : 'h' & Hex($vElement)
                Case 'Bool' ; True or False
                    $sName &= ($vElement = True) ? 't' : 'v'
                Case 'Binary'
                    $sName &= 'y' & $vElement
                Case 'Ptr'
                    $sName &= 'p' & $vElement
                Case 'Keyword' ; Default or Null (other variable declarations are illegal)
                    $sName &= ($vElement = Default) ? 'w' : 'n'
                Case 'Function', 'UserFunction' ; string conversion will fail
                    For $k = 1 To $aFunction[0] ; unique functions are stored in a separate array
                        If $vElement = $aFunction[$k] Then ; this function has been encountered previously
                            $sName &= 'u' & $k
                            ContinueLoop 2
                        EndIf
                    Next
                    $aFunction[0] += 1
                    If $aFunction[0] > UBound($aFunction) -1 Then ReDim $aFunction[$aFunction[0] + 10]
                    $aFunction[$aFunction[0]] = $vElement
                    $sName &= 'u' & $aFunction[0]
                Case Else ; Array, Object or DLLStruct
                    $sName = False ; set to ignore
                    ExitLoop
            EndSwitch
        Next

        If $sName Then
            If IsDeclared($sName) = -1 Then ContinueLoop ; [$DECLARED_LOCAL] this row has been seen previously
            Assign($sName, "", 1) ; [else] declare each unique row as a new variable name
        EndIf

        ; overwrite the next row (assumes that the first duplicate will be found quite quickly)
        Switch $iDim
            Case 1
                $aArray[$iItems] = $aArray[$i]
            Case 2
                For $j = 0 To $iCols -1
                    $aArray[$iItems][$j] = $aArray[$i][$j]
                Next
        EndSwitch

        $iItems += 1
    Next

    Switch $iDim
        Case 1
            ReDim $aArray[$iItems]
        Case 2
            ReDim $aArray[$iItems][$iCols]
    EndSwitch
EndFunc ;==> _ArrayUniqueR

Example:

#include <Array.au3>

Local $aOddMix[14] = [Null, Null, True, True, False, False, Default, Default, Binary('0x0001'), Binary('0x0001'), Ptr(0x0001), Ptr(0x0001), MsgBox, MsgBox]
_ArrayDisplay($aOddMix)
$aUnique = _ArrayUniqueR($aOddMix)
_ArrayDisplay($aOddMix)

 

Edited by czardas

Share this post


Link to post
Share on other sites



I don't see any logical reason to be using Assign and IsDeclared in that code.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

I first thought this might compare objects, it does not, but it rules it out and shines a bit of light on my dopeyness a bit:)

$obj1 = ObjCreate("scripting.dictionary")
If Not IsObj($obj1) Then Exit MsgBox(0,0,1)

$obj2 = ObjCreate("scripting.dictionary")
If Not IsObj($obj2) Then Exit MsgBox(0,0,2)
$obj2.Add("Key1", "Value1")

If $obj1 = $obj2 Then MsgBox(0,0,3)

MsgBox(0,0, _ObjCompare($obj1, $obj2))

Func _ObjCompare(ByRef $o1, ByRef $o2)
    if ObjName($o1, 1) <> ObjName($o2, 1) Then Return False
    if ObjName($o1, 2) <> ObjName($o2, 2) Then Return False
    if ObjName($o1, 3) <> ObjName($o2, 3) Then Return False
    if ObjName($o1, 4) <> ObjName($o2, 4) Then Return False
    if ObjName($o1, 5) <> ObjName($o2, 5) Then Return False
    if ObjName($o1, 6) <> ObjName($o2, 6) Then Return False
    if ObjName($o1, 7) <> ObjName($o2, 7) Then Return False
    Return True
EndFunc

 


AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

How would you do it?

Why do you need it when the variable already exists?


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

I'm not sure what you mean, I don't declare any variables that already exist. There are other ways to do this, but I think using a delimited string would be too slow. I haven't tried using a scripting dictionary previously, so I'm unaware of the limitations.

Share this post


Link to post
Share on other sites

This part

If $sName Then
    If IsDeclared($sName) = -1 Then ContinueLoop
    Assign($sName, "", 1)
EndIf

If $sName was not declared, would the script not fail with "Variable not declared" error?

JohnOne, he's not checking if the $sName is or isn't declared. He's checking if the value held by $sName is declared.

1 person likes this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

...Yea, obviously you aren't from Manchester.

1 person likes this

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

The first version of this function completely failed the integer test I ran just now. I was too tired yesterday to try everything. The Hex conversion method I was using was completely doomed to failure. The following remark (about Hex) in the help file still has me slightly baffled.

Omitting the second parameter (length) or using the keyword Default results in automatic sizing of the output; 8-characters display for numbers that can fit in that and 16-characters display for others.

Even if it worked the way I had expected, there was still a serious flaw in my logic. Two's compliment (and the potential collision range 0x80000000 - 0xFFFFFFFF) which prevents stripping out leading zeros, makes comparison of the Hex unsuitable. So how to solve this problem using minimal processing? Update in the first post - now works with all types of integers.

Testing Integers

#include <Array.au3>

Local $aIntegers[2000000]
For $i = 0 To    1999999
    $aIntegers[$i] = Number(Random(-500, 500, 1), Random(1, 3, 1)) ; Int32, Int64 or Double [randomly cast]
Next

Local $iTimer = TimerInit()
_ArrayUniqueR($aIntegers) ; returns (~) 1001 unique integers in approx 22 seconds using 2GB RAM
ConsoleWrite(TimerDiff($iTimer) & @LF)
_ArrayDisplay($aIntegers)

As many as 3003 integers (of different types) may be generated (and duplicated) in the code above, but only 1001 unique values can ever appear in the returned array - because approximately 67% of the two million randomly produced numbers differ only in type, and not in magnitude. I'll document this function after I've had more time to play around with it.

Edited by czardas

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

I have added documentation and commented the code. I also removed some superfluous syntax which was only there in case it might be needed. Hopefully it will be easier to understand now. The method used is slower than the current version of _ArrayUnique() but it is still much faster than earlier versions of _ArrayUnique(). It also behaves differently in several ways.

There are limitations to using this approach. I estimate that it can handle about 30 MB of unique string data. This estimate may vary on more powerful computers - yet to be tested. With a mixture of data types, the true size may also vary significantly. The more duplicates there are in an array, the more elements it can handle. In plain English - this function is good for hundreds of thousands of rows of medium length, a couple of million numbers, or a couple of hundred unique short stories. It's perfect for something like small business accounts stored in tables - if you don't mind waiting a few seconds.

Some of the (collision avoidance) solutions presented in the code could potentially be used in other situations. Careful consideration is needed for all possible collision scenarios when implementing methods of this nature - perhaps interesting from a design perspective.

Edited by czardas

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

The above was intended to test a concept and act as an intermediary online backup: posted here in case it might also be of interest to you. Until I created _PreDim() - formerly called _ArrayForceDim() and found here: https://www.autoitscript.com/forum/topic/179817-_arrayforcedim/ - I was not sure how to proceed further. It seems I've been having a bit of naming crisis recently: describing what some functions do is tricky enough without having to do it in one or two camel-cased words. :think:

This new function, '_UniqueRegion', works for arrays of up to five dimensions; and uniqueness can be applied to any available dimension. The notes in the code offer further explanation. Additional functions are planned which employ the same kind of logic. Some multi-loop processes may be exported to helper functions: to be shared with other functions yet to be written.

Optimization can wait. You may want to consider where latency occurs in this script and see if you spot the same thing that I did (look inside the loops). The difference may not be so great, although I think the unoptimized script is possibly a little easier to understand, so I am posting it as it is for now.

; #FUNCTION# ===================================================================================================================
; Name...........: _UniqueRegion
; Description ...: Removes duplicate items from an array (or duplicate regions from a multidimensional array).
; Syntax.........: _UniqueRegion($aArray [, $bCaseSense = False [, $iDimension = 1]])
; Parameters.....; $aArray - The array containing duplicate regions to be removed.
;                  $bCaseSense - [Optional] Set to true for case sensitive matches. Default value = False
;                  $iDimension - [Optional] Integer value - the dimension to which uniqueness applies. Default = 1st dimension
; Return values .: Success - Returns the unique array [ByRef].
;                  Failure sets @error as follows:
;                  |@error = 1 The 1st parameter is not an array.
;                  |@error = 2 The 1st parameter has too many dimensions.
;                  |@error = 3 The 3rd parameter is not a valid dimension.
; Author ........: czardas
; Comments ......; This function is limited to (medium sized) arrays of up to five dimensions.
;                  Unique array regions are defined as follows:
;                  1. unique items within a one dimensional array
;                  2. unique item collections within a two dimensional array (rows or columns)
;                  3. unique tables within a three dimensional array
;                  4. unique cuboidal areas within a four dimension array
;                  5. unique four dimensional cuboids within a five dimensional array
;                  All elements within a region must be duplicated (in juxtaposed positions) before removal takes place.
;                  Works with strings, numbers, binary, boolean values, pointers, functions and keywords (Null and Default).
;                  Integers of the same magnitude are treated as duplicates regardless of data type.
;                  This function does not remove regions containing objects, DLLStructs or other arrays.
;                  If you modify _UniqueRegion, avoid using variable names that could collide with internally generated names.
; Example .......; _UniqueRegion($aArray, Default, 2) ; [$iDimension = 2] deletes duplicate COLUMNS from a 2D array [concept ^^]
; ==============================================================================================================================

Func _UniqueRegion(ByRef $aArray, $bCaseSense = Default, $iDimension = 1)
    If Not IsArray($aArray) Then Return SetError(1) ; not an array

    Local $iDim = UBound($aArray, 0) ; get the number of dimensions
    If $iDim > 5 Then Return SetError(2) ; wrong number of dimensions

    $iDimension = ($iDimension = Default) ? 1 : Int($iDimension)
    If $iDimension < 1 Or $iDimension > $iDim Then Return SetError(3) ; [or override ambiguous input with $iDimension = 1]

    Local $aBound[6]
    For $i = 1 To $iDim
        $aBound[$i] = UBound($aArray, $i) ; get the size of each dimension
    Next
    If $aBound[$iDimension] < 2 Then Return ; the array is already unique

    For $i = $iDim + 1 To 5
        $aBound[$i] = 1 ; set the upper bounds of dimensions to be added
    Next
    _Predim($aArray, 5) ; adding more dimensions introduces some latency - too bad because this reduces complexity!

    Local $sExpression = '$aArray' ; to access elements at their original indeces
    For $i = 1 To 5
        If $i <> $iDimension Then
            $sExpression &= '[$' & $i & ']' ; default expression = '$aArray[$iNext][$2][$3][$4][$5]'
        Else
            $sExpression &= '[$iNext]'
        EndIf
    Next

    $aBound[0] = $aBound[$iDimension] ; the first loop will run to the bounds of the specified dimension
    $aBound[$iDimension] = 1 ; whichever loop this relates to must only run once on each encounter

    Local $iItems = 0, $aFunction[1] = [0], $vElement, $iInt, $sName
    If $bCaseSense = Default Then $bCaseSense = False

    For $iNext = 0 To $aBound[0] -1
        $sName = '' ; clear previous name

        For $5 = 0 To $aBound[5] -1
            For $4 = 0 To $aBound[4] -1
                For $3 = 0 To $aBound[3] -1
                    For $2 = 0 To $aBound[2] -1
                        For $1 = 0 To $aBound[1] -1
                            $vElement = Execute($sExpression) ; get the data contained in each element

                            ; use non-hexadecimal characters (other than x) as datatype identifiers and convert the data to hexadecimal where necessary
                            Switch VarGetType($vElement)
                                Case 'String'
                                    If Not $bCaseSense Then $vElement = StringUpper($vElement) ; generates a case insensitive name segment
                                    $sName &= 's' & StringToBinary($vElement, 4) ; UTF8 [$SB_UTF8]
                                Case 'Int32', 'Int64' ; use decimal without conversion
                                    ; the minus sign of a negative integer is replaced with 'g': to distinguish it from the positive value
                                    $sName &= ($vElement < 0) ? 'g' & StringTrimLeft($vElement, 1) : 'i' & $vElement
                                Case 'Double' ; may be an integer
                                    $iInt = Int($vElement)
                                    $sName &= ($vElement = $iInt) ? (($iInt < 0) ? 'g' & StringTrimLeft($iInt, 1) : 'i' & $iInt) : 'h' & Hex($vElement)
                                Case 'Bool' ; True or False
                                    $sName &= ($vElement = True) ? 't' : 'v'
                                Case 'Binary'
                                    $sName &= 'y' & $vElement
                                Case 'Ptr'
                                    $sName &= 'p' & $vElement
                                Case 'Keyword' ; Default or Null (other variable declarations are illegal)
                                    $sName &= ($vElement = Default) ? 'w' : 'n'
                                Case 'Function', 'UserFunction' ; string conversion will fail
                                    For $k = 1 To $aFunction[0] ; unique functions are stored in a separate array
                                        If $vElement = $aFunction[$k] Then ; this function has been encountered previously
                                            $sName &= 'u' & $k
                                            ContinueLoop 2
                                        EndIf
                                    Next
                                    $aFunction[0] += 1
                                    If $aFunction[0] > UBound($aFunction) -1 Then ReDim $aFunction[$aFunction[0] + 10]
                                    $aFunction[$aFunction[0]] = $vElement
                                    $sName &= 'u' & $aFunction[0]
                                Case Else ; Array, Object or DLLStruct
                                    $sName = False ; set to ignore
                                    ExitLoop 5
                            EndSwitch
                        Next
                    Next
                Next
            Next
        Next

        If $sName Then
            If IsDeclared($sName) = -1 Then ContinueLoop ; [$DECLARED_LOCAL] this region has been seen previously
            Assign($sName, "", 1) ; [else] declare each unique region as a new variable name
        EndIf

        ; overwrite the next region (assumes that the first duplicate region will be found quite quickly)
        For $5 = 0 To $aBound[5] -1
            For $4 = 0 To $aBound[4] -1
                For $3 = 0 To $aBound[3] -1
                    For $2 = 0 To $aBound[2] -1
                        For $1 = 0 To $aBound[1] -1
                            Switch $iDimension
                                Case 1
                                    $aArray[$iItems][$2][$3][$4][$5] = $aArray[$iNext][$2][$3][$4][$5]
                                Case 2
                                    $aArray[$1][$iItems][$3][$4][$5] = $aArray[$1][$iNext][$3][$4][$5]
                                Case 3
                                    $aArray[$1][$2][$iItems][$4][$5] = $aArray[$1][$2][$iNext][$4][$5]
                                Case 4
                                    $aArray[$1][$2][$3][$iItems][$5] = $aArray[$1][$2][$3][$iNext][$5]
                                Case 5
                                    $aArray[$1][$2][$3][$4][$iItems] = $aArray[$1][$2][$3][$4][$iNext]
                            EndSwitch
                        Next
                    Next
                Next
            Next
        Next

        $iItems += 1
    Next

    _Predim($aArray, $iDim) ; remove the extra dimensions
    $aBound[$iDimension] = $iItems ; reset the bounds

    Switch $iDim ; remove the remaining duplicate array regions
        Case 1
            ReDim $aArray [$aBound[1]]
        Case 2
            ReDim $aArray [$aBound[1]] [$aBound[2]]
        Case 3
            ReDim $aArray [$aBound[1]] [$aBound[2]] [$aBound[3]]
        Case 4
            ReDim $aArray [$aBound[1]] [$aBound[2]] [$aBound[3]] [$aBound[4]]
        Case 5
            ReDim $aArray [$aBound[1]] [$aBound[2]] [$aBound[3]] [$aBound[4]] [$aBound[5]]
    EndSwitch

EndFunc ;==> _UniqueRegion


; #FUNCTION# ===================================================================================================================
; Name...........: _Predim
; Description ...: Changes the size of an array by adding, or removing, dimensions.
; Syntax.........: _Predim($aArray, $iDimensions [, $iPush = False])
; Parameters.....; $aArray - The original array.
;                  $iDimensions - The number of dimensions in the returned array.
;                  $bPush - [Optional] If set to True, new dimensions are created on, or removed from, the left [see comments].
; Return values .: Returns the modified array ByRef.
;                  Failure sets @error as follows:
;                  |@error = 1 The first parameter is not an array.
;                  |@error = 2 The requested array has more than 7 dimensions.
;                  |@error = 3 The original array has more than 7 dimensions.
; Author.........: czardas
; Comments ......; This function works for up to 7 dimensions.
;                  By default, new dimensions are added to the right in a standard sequence: $aArray[7][6] ==> $aArray[7][6][1]
;                  Dimensions are removed in reverse sequence: $aArray[7][6] ==> $aArray[7]
;                  When the $bPush parameter is set to True, the original array will be pushed to higher dimensions:
;                  $aArray[7][6] ==> $aArray[1][7][6], or the process reversed: $aArray[7][6] ==> $aArray[6]
; ==============================================================================================================================

Func _Predim(ByRef $aArray, $iDimensions, $bPush = False)
    If Not IsArray($aArray) Then Return SetError(1)

    $iDimensions = Int($iDimensions)
    If $iDimensions < 1 Or $iDimensions > 7 Then Return SetError(2)

    Local $iPreDims = UBound($aArray, 0) ; current number of dimensions
    If $iPreDims = $iDimensions Then Return ; no change
    If $iPreDims > 7 Then Return SetError(3) ; too many dimensions

    ; get the size of each original dimension
    Local $a[$iPreDims +1]
    For $i = 1 To $iPreDims
        $a[$i] = UBound($aArray, $i)
    Next

    Local $sElement = '[$1][$2][$3][$4][$5][$6][$7]' ; array syntax used to access the original array elements

    If $bPush Then ; prefix dimensions, or delete from the left
        Local $iOffset = Abs($iDimensions - $iPreDims)
        If $iPreDims > $iDimensions Then
            For $i = 1 To $iDimensions ; shift elements to lower indices
                $a[$i] = $a[$i + $iOffset]
            Next
            $sElement = '$aArray' & StringLeft('[0][0][0][0][0][0]', $iOffset * 3) & StringLeft($sElement, $iDimensions * 4)

        Else
            ReDim $a [$iDimensions +1] ; make space for more dimensions
            For $i = $iDimensions To $iOffset +1 Step -1 ; shift elements to higher indices
                $a[$i] = $a[$i - $iOffset]
            Next
            For $i = 1 To $iOffset ; assign the size of each additional dimension [1][1][1]... etc...
                $a[$i] = 1
            Next

            $sElement = '$aArray' & StringMid($sElement, 1 + $iOffset * 4, $iPredims * 4)
        EndIf

    Else ; Default behaviour = append dimensions, or delete from the right
        ReDim $a [$iDimensions +1] ; modify the number of dimensions

        For $i = $iPredims + 1 To $iDimensions ; assign the size of each new dimension ...[1][1][1] etc...
            $a[$i] = 1
        Next
        $sElement = '$aArray' & StringLeft($sElement, $iPredims * 4)

        Local $2 = 0, $3 = 0, $4 = 0, $5 = 0, $6 = 0, $7 = 0 ; required to reference new higher dimensions
    EndIf

    ; add or remove dimensions
    Switch $iDimensions
        Case 1
            Local $aNewArray[$a[1]]
            For $1 = 0 To $a[1] -1
                $aNewArray[$1] = Execute($sElement)
            Next

        Case 2
            Local $aNewArray[$a[1]][$a[2]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    $aNewArray[$1][$2] = Execute($sElement)
                Next
            Next

        Case 3
            Local $aNewArray[$a[1]][$a[2]][$a[3]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    For $3 = 0 To $a[3] -1
                        $aNewArray[$1][$2][$3] = Execute($sElement)
                    Next
                Next
            Next

        Case 4
            Local $aNewArray[$a[1]][$a[2]][$a[3]][$a[4]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    For $3 = 0 To $a[3] -1
                        For $4 = 0 To $a[4] -1
                            $aNewArray[$1][$2][$3][$4] = Execute($sElement)
                        Next
                    Next
                Next
            Next

        Case 5
            Local $aNewArray[$a[1]][$a[2]][$a[3]][$a[4]][$a[5]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    For $3 = 0 To $a[3] -1
                        For $4 = 0 To $a[4] -1
                            For $5 = 0 To $a[5] -1
                                $aNewArray[$1][$2][$3][$4][$5] = Execute($sElement)
                            Next
                        Next
                    Next
                Next
            Next

        Case 6
            Local $aNewArray[$a[1]][$a[2]][$a[3]][$a[4]][$a[5]][$a[6]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    For $3 = 0 To $a[3] -1
                        For $4 = 0 To $a[4] -1
                            For $5 = 0 To $a[5] -1
                                For $6 = 0 To $a[6] -1
                                    $aNewArray[$1][$2][$3][$4][$5][$6] = Execute($sElement)
                                Next
                            Next
                        Next
                    Next
                Next
            Next

        Case 7
            Local $aNewArray[$a[1]][$a[2]][$a[3]][$a[4]][$a[5]][$a[6]][$a[7]]
            For $1 = 0 To $a[1] -1
                For $2 = 0 To $a[2] -1
                    For $3 = 0 To $a[3] -1
                        For $4 = 0 To $a[4] -1
                            For $5 = 0 To $a[5] -1
                                For $6 = 0 To $a[6] -1
                                    For $7 = 0 To $a[7] -1
                                        $aNewArray[$1][$2][$3][$4][$5][$6][$7] = Execute($sElement)
                                    Next
                                Next
                            Next
                        Next
                    Next
                Next
            Next
    EndSwitch

    $aArray = $aNewArray
EndFunc ;==> _Predim

So far only one example (deleting duplicate columns from a table). Note the difference when you omit the 3rd parameter.

#include <Array.au3> ; For _ArrayDisplay()
Local $aTest = [[1,2,2,1],[1,2,2,1]]
_ArrayDisplay($aTest)

_UniqueRegion($aTest, Default, 2) ; remove duplicate regions encountered within the second dimension
_ArrayDisplay($aTest)

 

Edited by czardas

Share this post


Link to post
Share on other sites

Just what I need right now, thanks for sharing.

Share this post


Link to post
Share on other sites

You're welcome. This function is an earlier version. The most recent version is called _ArrayUniqueXD and can be found in ArrayWorkshop (see my signature). This version should work fine too. :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • dejhost
      By dejhost
      Hello ,
      Here are three stepts that I would like to speed up - if possible: 
      STEP 1: I am generating an array, containing binary numbers  up to a certain amount of digits. My script adds leading zeros, so that each number has equal amount of digits.
      Example: 14 digit Binary array:
      I am using the code
      For $i = 0 to 2^$bit-1 ; $bit amount of digits. for example: 14 $binary = ( Dec2Bin($i) ) ; Check length of binary string $adig = $bit - StringLen($binary) ; Determine how many leading 0 have to be added $zeros = "" For $j = 1 To $adig ; add leading "0"s $zeros = $zeros & "0" Next $BinArray[$i] = $zeros & $binary ;Write binary-number to file, leading "0" Next Func Dec2Bin($D) Return (BitShift($D, 1) ? Dec2Bin(BitShift($D, 1)) : "") & BitAnd($D, 1) EndFunc ;==> Dec2Bin() AutoIt v3.3.12.0   to generate the binary number. 
      STEP 2: I reduce the array to unique values. In my application, the binary-numbers do not have a start-bit or end-bit. This means, that
        are actually the same numbers, and only one of them is to remain in the array. All alterations aren't unique and shall be removed. Here is my code:
      For $i = 0 to Ubound($BinArray)-1 ; shift through all rows For $j = 1 to $bit ; shift through all the bits If $i = Ubound($BinArray) Then ; exit before exceeding the arrays boundries ExitLoop 2 EndIf $BinArray[$i] = StringRight ( $BinArray[$i], 1 ) & StringLeft ( $BinArray[$i], $bit-1 ) $BinArray = _ArrayUnique($BinArray, 0, 0, 0, 0, $ARRAYUNIQUE_AUTO) If @error <> 0 Then Msgbox(0, "Error in _ArrayUnique", "The Error Code is " & @error & " Abort.") Exit EndIf Next Next STEP 3: Finally, I write the remaining array into a text-file.
      For $i = 0 to Ubound($BinArray)-1 FileWrite($hFileOpen, $i & @TAB & $BinArray[$i] & @CRLF) Next  
      So my question is: any idea how to speed up this procedure? There certainly is a way to do this smarter. Btw.: Step 2 is optional.
      Thanks for helping,
      dejhost 
    • Massi
      By Massi
      Hi, I have update to the most recent Autoit version. I have a little problem with _ArrayUnique function.
      If I have one array with two rows. _ArrayUnique crashes. I haven't any problem if there are more than two rows. I know that seems useless use this function with two rows, but these rows are the result of a search (generally I have more than two rows).
      "C:\Program Files (x86)\AutoIt3\Include\Array.au3" (2290) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
      Local $vFirstElem = ( ($iDims = 1) ? ($aArray[$iBase]) : ($aArray[$iColumn][$iBase]) )
      Local $vFirstElem = ( ($iDims = 1) ? ($aArray[$iBase]) : (^ ERROR
       
      #include <Array.au3> Dim $Test1[3][6]=[["11","12","13","14","15","16"],["21","22","23","24","25","26"],["21","22","23","24","25","26"]] Dim $Test[2][6]=[["11","12","13","14","15","16"],["21","22","23","24","25","26"]] Dim $result=0 _ArrayDisplay($Test1) _ArrayDisplay(_ArrayUnique($Test1, 2)) _ArrayDisplay($Test) _ArrayDisplay(_ArrayUnique($Test, 2)) I put a workaround that I don't use _ArrayUnique function if there are only two results
    • Jfish
      By Jfish
      I think there may be a bug with _ArrayUnique introduced in 3.3.14 (or it could just be me).  I updated ticket #3078 with the attached files to demonstrate.  If you run test.au3 against the test data file in 3.3.12 it works and you see a unique array from _arrayDisplay.  If you do the same in 3.3.14 the _arrayDisplay view will be empty.
      I read the script breaking changes and the updates but I don't think it should have cause this.  Please let me know if I am doing something wrong.
      #include <Excel.au3> #include <Array.au3> func _openSourceData() $sourcefile=FileOpenDialog("Please select source file",@ScriptDir,"All (*.*)") If @error Then ; Display the error message. MsgBox($MB_SYSTEMMODAL, "", "No file(s) were selected.") else ; Create application object Local $oAppl = _Excel_Open() If @error Then Exit MsgBox(16, "Excel UDF: _Excel_BookOpen Example", "Error creating the Excel application object." & @CRLF & "@error = " & @error & ", @extended = " & @extended) ; ***************************************************************************** ; Open an existing workbook and return its object identifier. ; ***************************************************************************** Local $sWorkbook = $sourcefile $oWorkbook = _Excel_BookOpen($oAppl, $sWorkbook, Default, Default, True) If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_BookOpen Example 1", "Error opening '" & $sWorkbook & "'." & @CRLF & "@error = " & @error & ", @extended = " & @extended) $sourceArray=_Excel_RangeRead($oWorkbook) _ArrayDisplay($sourceArray) $unique=_ArrayUnique($sourceArray) _ArrayDisplay($unique) EndIf EndFunc _openSourceData()  
      test.au3
      testData2.xlsx
    • souldjer777
      By souldjer777
      Good Morning AutoIT Geniuses 

      I have version 3.3.14.1.
      I am experiencing an error with the default Array.au3 or _ArrayUnique 
      I believe it is with this single line in my au3 file...$aUniqueHostname = _ArrayUnique ($array01, 1)
      SendAndLog("GEN_CSVMinRowLimit01 - Started", $tempzipdir & '\' & $LogFileName01, True) ; CSV Minimum Row Limit (FYI - default was originally set to 5000) $array01 = $twoDarray MsgBox (0, "", "You are here 1") $aUniqueHostname = _ArrayUnique ($array01, 1) ; FYI - The program never makes it to here: MsgBox (0, "", "You are here 2") "C:\Program Files (x86)\AutoIt3\Include\array.au3" (2297) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
      If IsInt($aArray[$iBase]) Then
      If IsInt(^ ERROR
      ; Title .........: Array
      ; AutoIt Version : 3.3.14.1  
       ; Autocheck of first element
          If $iIntType = $ARRAYUNIQUE_AUTO Then
              If IsInt($aArray[$iBase]) Then
              ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
                  Switch VarGetType($aArray[$iBase])
                      Case "Int32"
                          $iIntType = $ARRAYUNIQUE_FORCE32
                      Case "Int64"
                          $iIntType = $ARRAYUNIQUE_FORCE64
                  EndSwitch
              Else
                  $iIntType = $ARRAYUNIQUE_FORCE32
              EndIf
          EndIf
      I don't believe I've ever seen an error in the default au3 files. Is this something I am doing wrong?
      I'm just trying to run my apps through the new version of AutoIT since the fix was put in for...
      AutoIt3Help.exe reworked and digitally signed I have looked at bug trackers...
      https://www.autoitscript.com/trac/autoit/ticket/3110
      https://www.autoitscript.com/trac/autoit/ticket/3078
      Is this related somehow - what version of AutoIT do you recommend I use?
      Thanks!
       
    • Gibbo
      By Gibbo
      Recently I found myself needing to "Uniqueify" large arrays of data. Not happy with the speed I decided to check out how _ArrayUnique worked and if it could be improved.
      Below is my overhauled version of _ArrayUnique.

      "Uniqueify" An array of 500 random numbers (average of 5 results)
      Old: 274.597787502462
      New: 240.237994573652

      "Uniqueify" An array of 1000 random numbers (average of 5 results)
      Old: 1070.06629280595
      New: 920.537746095923

      Still not fast but faster


      Changed ALL Dim's to Local's From the help file: "You should use Local or Global, instead of Dim, to explicitly state which scope is desired for a variable/constant/array." Changed: Dim $aArrayTmp[1] ;Declare blank array, which will hold the dimension declared by user For $i = 0 To $iUboundDim - 1 _ArrayAdd($aArrayTmp, <itemtoadd> ) ;$iDimension-1 to match Dimension Next _ArrayDelete($aArrayTmp, 0) ;Get rid of 1st-element which is blank To: Local $aArrayTmp[$iUboundDim] ;Declare array, which will hold the dimension declared by user For $i = 0 To $iUboundDim - 1 ;Loop through "Rows" $aArrayTmp[$i] = <itemtoadd> ;$iDimension-1 to match Dimension Next It is silly to declare a "blank" array (and delete the first "empty value")if you already know it's intended dimentions This also removed the need for _ArrayAdd (No more ReDim's YAY!) Changed If,Then, Else statements to Switch statements where appropriate for ease of reading (they are supposed to be faster too) Trimmed: If Not $iDimension > 0 Then Return SetError(3, 0, 0) ;Check to see if it is valid array dimension, Should be greater than 0 Else To: If $iDimension < 1 Then Return SetError(3, 0, 0) ;Check to see if it is valid array dimension, Should be greater than 0 Changed: $aArrayTmp = StringSplit(StringTrimRight($sHold, StringLen($vDelim)), $vDelim, 1) ;Split the string into an array Return $aArrayTmp ;SmOke_N's version used to Return SetError(0, 0, 0) To: $sHold = StringTrimRight($sHold, StringLen($vDelim)) $aArrayTmp = StringSplit($sHold, $vDelim, 1) ;Split the string into an array Return $aArrayTmp ;SmOke_N's version used to Return SetError(0, 0, 0) For clarity

      ; #FUNCTION# ==================================================================================================================== ; Name...........: _ArrayUnique ; Description ...: Returns the Unique Elements of a 1-dimensional array. ; Syntax.........: _ArrayUnique($aArray[, $iDimension = 1[, $iBase = 0[, $iCase = 0[, $vDelim = "|"]]]]) ; Parameters ....: $aArray - The Array to use ; $iDimension - [optional] The Dimension of the Array to use ; $iBase - [optional] Is the Array 0-base or 1-base index. 0-base by default ; $iCase - [optional] Flag to indicate if the operations should be case sensitive. ; 0 = not case sensitive, using the user's locale (default) ; 1 = case sensitive ; 2 = not case sensitive, using a basic/faster comparison ; $vDelim - [optional] One or more characters to use as delimiters. However, cannot forsee its usefullness ; Return values .: Success - Returns a 1-dimensional array containing only the unique elements of that Dimension ; Failure - Returns 0 and Sets @Error: ; 0 - No error. ; 1 - Returns 0 if parameter is not an array. ; 2 - _ArrayUnique failed for some other reason ; 3 - Array dimension is invalid, should be an integer greater than 0 ; Author ........: SmOke_N ; Modified.......: litlmike, Gibbo ; Remarks .......: Returns an array, the first element ($array[0]) contains the number of strings returned, the remaining elements ($array[1], $array[2], etc.) contain the unique strings. ; Related .......: _ArrayMax, _ArrayMin ; Link ..........: ; Example .......: Yes ; =============================================================================================================================== Func _ArrayUnique($aArray, $iDimension = 1, $iBase = 0, $iCase = 0, $vDelim = "|") Local $iUboundDim ;$aArray used to be ByRef, but litlmike altered it to allow for the choosing of 1 Array Dimension, without altering the original array If $vDelim = "|" Then $vDelim = Chr(01) ; by SmOke_N, modified by litlmike If Not IsArray($aArray) Then Return SetError(1, 0, 0) ;Check to see if it is valid array ;Checks that the given Dimension is Valid If $iDimension < 1 Then Return SetError(3, 0, 0) ;Check to see if it is valid array dimension, Should be greater than 0 ;If Dimension Exists, then get the number of "Rows" $iUboundDim = UBound($aArray, 1) ;Get Number of "Rows" If @error Then Return SetError(3, 0, 0) ;2 = Array dimension is invalid. ;If $iDimension Exists, And the number of "Rows" is Valid: Switch $iDimension = 1 Case False ;Makes sure the Array dimension desired is more than 1-dimensional Local $aArrayTmp[$iUboundDim] ;Declare array, which will hold the dimension declared by user For $i = 0 To $iUboundDim - 1 ;Loop through "Rows" $aArrayTmp[$i] = $aArray[$i][$iDimension - 1] ;$iDimension-1 to match Dimension Next Case Else ;Makes sure the Array dimension desired is 1-dimensional ;If Dimension Exists, And the number of "Rows" is Valid, and the Dimension desired is not > 1, then: ;For the Case that the array is 1-Dimensional Switch UBound($aArray, 0) Case 1 ;Makes sure the Array is only 1-Dimensional Local $aArrayTmp[$iUboundDim] ;Declare array, which will hold the dimension declared by user For $i = 0 To $iUboundDim - 1 $aArrayTmp[$i] = $aArray[$i] Next Case Else ;For the Case that the array is 2-Dimensional Local $aArrayTmp[$iUboundDim] ;Declare array, which will hold the dimension declared by user For $i = 0 To $iUboundDim - 1 $aArrayTmp[$i] = $aArray[$i][$iDimension - 1] ;$iDimension-1 to match Dimension Next EndSwitch EndSwitch Local $sHold ;String that holds the Unique array info For $iCC = $iBase To $iUboundDim - 1 ;Loop Through array ;If Not the case that the element is already in $sHold, then add it Switch StringInStr($vDelim & $sHold, $vDelim & $aArrayTmp[$iCC] & $vDelim, $iCase) Case False $sHold &= $aArrayTmp[$iCC] & $vDelim EndSwitch Next If $sHold Then $sHold = StringTrimRight($sHold, StringLen($vDelim)) $aArrayTmp = StringSplit($sHold, $vDelim, 1) ;Split the string into an array Return $aArrayTmp ;SmOke_N's version used to Return SetError(0, 0, 0) EndIf Return SetError(2, 0, 0) ;If the script gets this far, it has failed EndFunc ;==>_ArrayUnique