IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

It starts with spying on the windows to get the basic information. See first thread on a basic way of spying

I was unable to install this AV program as it gives an installation error so only can help you high level

0. It helps if you install inspect.exe (in windows SDK), not fully needed

1. Make a const reference to your main ADAware window

;~ ClassName: "GuiFramework::HtmlView" Name: "Ad-Aware Antivirus Installation"
$cADName="name:=Ad*"

2. Get the Element reference

$oAD=_UIA_getFirstObjectOfElement($oDesktop, $cADName, $treescope_children)
if not isobj($oAD) Then
findThemAll($oDesktop,$treescope_children)
EndIf

3. Debug, if it is not fully working but you get the main window you could do this to get all elements in the console window shown

findThemAll($oAD,$treescope_subtree)

[junkew]

Example 9 Windows media player

 

Updated UIAWrappers extensively to abstract more from the (complex) MS UIA framework

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
HotKeySet("{ESC}", "Terminate")

;~ Turn debugging UIA on by default, dumps most details to consolewindow for the moment, later will use a logfile
;~ _UIA_setVar("Global.Debug",true)
;~ _UIA_setVar("Global.Highlight",true) ;- Highlights object when found

;~ Set the system under test variables
_UIA_setVar("SUT1.Folder","C:\Program Files (x86)\Windows Media Player")
_UIA_setVar("SUT1.Workingdir","C:\Program Files (x86)\Windows Media Player\")
_UIA_setVar("SUT1.EXE","wmplayer.exe")
_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))
_UIA_setVar("SUT1.Parameters","")
_UIA_setVar("SUT1.Processname","wmplayer.exe")
_UIA_setVar("SUT1.Windowstate",@SW_RESTORE)

_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))
_UIA_setVar("SUT1.Parameters","")
_UIA_setVar("SUT1.Processname","wmplayer.exe")

;~ Set the system under test objects to recognize
_UIA_setVar("WMP.mainwindow","classname:=WMPlayerApp")
_UIA_setVar("WMP.Playlists","classname:=SysTreeView32")
_UIA_setVar("WMP.Firstlist","name:=(Afspeellijst.*)|(Playlist.*)")
_UIA_setVar("WMP.Playbutton","name:=playGlyph")
_UIA_setVar("WMP.Volumebutton","name:=Volume")

;~ Start system under test
_UIA_DEBUG("Starting system under test" & @CRLF)
_UIA_StartSUT("SUT1")

;~ Get the main WMP element
;~ _UIA_DEBUG("Action 1 Finding main window" & @CRLF)
;~ $oWMP=_UIA_getFirstObjectOfElement($oDesktop, _UIA_getVar("WMP.mainwindow"), $treescope_children)
;~ $oWMP.setfocus()

;~ Action on the logical object as defined
_UIA_action("WMP.MAINWINDOW","setfocus")
;~ Retrieve the actual object from the runtime type information
$oWMP=_UIA_getVar("RTI.WMP.mainwindow")

;~ Get the playlists
;~ _UIA_DEBUG("Action 2 Finding playlists" & @CRLF)
$oPL=_UIA_getFirstObjectOfElement($oWMP, _UIA_getVar("WMP.playlists"), $treescope_subtree)
$oFirstPL=_UIA_getObjectByFindAll($oPL, _UIA_getVar("WMP.Firstlist"), $treescope_subtree)

;~ See if its expanded/collapsed
_UIA_DEBUG("Action 3 Expanding if needed" & @CRLF)
dim $state
;~ Get the pattern
$oExpandP=_UIA_getpattern($oFirstPL,$UIA_ExpandCollapsePatternId)
if isobj($oExpandP) Then
;~ Get the state
;~ Global Const $ExpandCollapseState_Collapsed=0
;~ Global Const $ExpandCollapseState_Expanded=1
;~ Global Const $ExpandCollapseState_PartiallyExpanded=2
;~ Global Const $ExpandCollapseState_LeafNode=3
    $oExpandP.CurrentExpandCollapseState($state)
    if $state=$ExpandCollapseState_Collapsed Then
        _UIA_action("WMP.Firstlist","leftdoubleclick")
    EndIf
endif

;~ _UIA_DEBUG("Action 4 Starting with the playbutton" & @CRLF)
;~ $oPlayButton=_UIA_getObjectByFindAll($oWMP, _UIA_getVar("WMP.Playbutton"), $treescope_subtree)
_UIA_action("WMP.Playbutton","left")

_UIA_DEBUG("Action 5 And finding the volume" & @CRLF)
$oVolumeButton=_UIA_getObjectByFindAll($oWMP, _UIA_getVar("WMP.Volumebutton"), $treescope_subtree)
$oValueP=_UIA_getpattern($oVolumeButton,$UIA_ValuePatternId)

if isobj($oValueP) Then
dim $volume
    $oValuep.currentvalue($volume)
    _UIA_Debug("Volume is " & $volume & @CRLF)
EndIf

; The End
Func Terminate()
    consolewrite("Exiting")
    $running=false
;~  Exit 0
EndFunc   ;==>Terminate

Edited November 5, 2013 by junkew

[Shrapnel]

junkew,

You are  a wizzard!

I just started exploring this, but it is AWESOME

[Shrapnel]

I know this is not much, but I am trying to contribute what i can to this.

Modified the _UIA_action function as follows:

• Merged all the mouse clicking cases into one and added a right click option
• Removed "MouseMove" commands since "MouseClick" can move AND click mouse
• made $p1 = number of clicks, $p2 = speed of mousemove and $p3 = number of ms to sleep at end of func

Func _UIA_action($obj, $strAction, $p1 = 0, $p2 = 0, $p3 = 0)
    Local $tPattern, $t, $x, $y, $mouseButton = "left"
    $strAction = StringLower($strAction)
    Switch $strAction
        Case "leftclick", "left", "rightclick", "right"
            $t = StringSplit(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId),";")
            $x = $t[1] + ($t[3] / 2)
            $y = $t[2] + ($t[4] / 2)
            If StringInStr($strAction, "right") Then $mouseButton = "right"
            If $p1 = 0 Then $p1 = 1
            MouseClick($mouseButton, $x, $y, $p1, $p2)
            If $p3 > 0 Then Sleep($p3)
        Case "setvalue using keys"
            $obj.setfocus
            Send("^a")
            Send($p1)
            If $p2 > 0 Then Sleep($p2)
        Case "invoke"
            $tPattern=_UIA_getPattern($obj,$UIA_InvokePatternId)
            $tPattern.invoke()
        Case "focus"
            $obj.setfocus()
        Case Else
    EndSwitch
EndFunc

edit: improvements

Edited November 1, 2013 by Shrapnel

[junkew]

Based on the request in this thread I enhanced the UIAwrappers more extensively to make life much easier

'?do=embed' frameborder='0' data-embedContent>>

http://www.machsupport.com/software/mach3/

Example 10 Automating mach 3 (AFX windows and other hard to get recognized by AutoIT)

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
HotKeySet("{ESC}", "Terminate")

;~ Turn debugging UIA on by default, dumps most details to consolewindow for the moment, later will use a logfile
;~ _UIA_setVar("Global.Debug",true)
;~ _UIA_setVar("Global.Highlight",true) ;- Highlights object when found

;~ Set the system under test variables
_UIA_setVar("SUT1.Folder","C:\Mach3")
_UIA_setVar("SUT1.Workingdir","C:\Mach3\")
_UIA_setVar("SUT1.EXE","Mach3.exe")
_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))
_UIA_setVar("SUT1.Parameters","/p Mach3Mill")
_UIA_setVar("SUT1.Processname","mach3.exe")
_UIA_setVar("SUT1.Windowstate",@SW_RESTORE)

_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))

;~ Set the system under test objects to recognize and abstract logical and physical names for readability in main script

_UIA_setVar("MACH3.mainwindow","classname:=Afx:00400000:b:00010003:00000006.*")
;~ _UIA_setVar("MACH3.mainwindow","name:=Mach3 CNC  Demo index:=1")

_UIA_setVar("MACH3.editgcode","name:=Edit G-Code")
_UIA_setVar("MACH3.btnCyclestart","automationid:=8142")
_UIA_setVar("MACH3.btnStop","automationid:=8140")
_UIA_setVar("MACH3.btnRewind","automationid:=8295")

_UIA_setVar("MACH3.mnuFile","name:=((File)|(Bestand)); index:=2")
_UIA_setVar("MACH3.mnuOpen","name:=Load.*")

_UIA_setVar("MACH3.dlgOpen","name:=Open*")
_UIA_setVar("MACH3.dlgOpen.edtFilename","name:=((Filename:)|(Bestandsnaam:)); indexrelative:=2")
_UIA_setVar("MACH3.dlgOpen.btnOpen","name:=Open* index:=6")

_UIA_setVar("NOTEPAD.mainwindow","classname:=Notepad")
_UIA_setVar("NOTEPAD.edit","classname:=Edit")
_UIA_setVar("NOTEPAD.mnuFile","name:=((File)|(Bestand))")
_UIA_setVar("NOTEPAD.mnuSave","name:=((Save.*)|(Opslaan.*))")
_UIA_setVar("NOTEPAD.btnClose","name:=((Close)|(Sluiten))")

_UIA_setVar("NOTEPAD.dlgSave","name:=((Save as.*)|(Opslaan als.*))")
_UIA_setVar("NOTEPAD.dlgSave.edtFilename","name:=((Filename:)|(Bestandsnaam:)); indexrelative:=1")
_UIA_setVar("NOTEPAD.dlgSave.btnSave","name:=((Save)|(Opslaan)) index:=4")
_UIA_setVar("NOTEPAD.dlgSave.btnYes","name:=((Yes)|(Ja))")

;~ Start system under test
_UIA_DEBUG("Starting system under test" & @CRLF)
_UIA_StartSUT("SUT1")
_UIA_DEBUG("SUT is started" & @CRLF)

_UIA_DEBUG("PID of SUT1 is " & _UIA_getVar("SUT1.PID") & @CRLF)

;~ Main script
_UIA_DEBUG("*** Main script started ***" & @CRLF)
_UIA_action("MACH3.mainwindow","focus")
_UIA_action("MACH3.editgCode", "left")

_UIA_action("NOTEPAD.mainwindow","focus")
_UIA_action("NOTEPAD.edit","sendkeys","f600{ENTER}g1 Z-2{ENTER}g1 y-13.5{ENTER}g1 z-14{ENTER}g1 z-26{ENTER}g1 z-38{ENTER}g1 y-35{ENTER}")
_UIA_action("NOTEPAD.mnuFile","left")
_UIA_action("NOTEPAD.mnuSave","left")
_UIA_action("NOTEPAD.dlgSave.edtFilename","sendkeys","demofile.txt")
_UIA_action("NOTEPAD.dlgSave.btnSave","left")
_UIA_action("NOTEPAD.dlgSave.btnYes","left")
_UIA_action("NOTEPAD.btnClose","left")

_UIA_action("MACH3.mnuFile","left")
_UIA_action("MACH3.mnuOpen","left")
_UIA_action("MACH3.dlgOpen.edtFilename","sendkeys","demofile.txt")
_UIA_action("MACH3.dlgOpen.btnOpen","left")

sleep(3000) ;~ Just 3 seconds to be able to load the script

_UIA_action("MACH3.btnStop","left")
_UIA_action("MACH3.btnRewind","left")
_UIA_action("MACH3.btnCycleStart","left")

; The End
Func Terminate()
    consolewrite("Exiting")
    $running=false
;~  Exit 0
EndFunc   ;==>Terminate

Edited November 18, 2013 by junkew

[Melba23]

junkew,

Excellent work. Thanks for all the effort you have put (and are still putting) into this.

M23

[Shrapnel]

The application I am trying to automate has Data Grid View controls and your spy tool is able to identify them, but I am unsure how to loop through all of the rows/columns and put the data into a 2d array.

I will be attempting to code this myself, but I was hoping someone may have already figured this out, or at least have a good idea on how to go about accomplishing this.

Anyways, here is the spy information of the Data Grid View control itself, and the spy information of one of the cells of the Data Grid view control (items that had no value were removed)

Data Grid View:

expandcollapse popup

Mouse position is retrieved 900-402
At least we have an element [DataGridView][WindowsForms10.Window.8.app.0.1a6f050_r13_ad1]
Having the following values for all properties: 
Title is: <DataGridView>    Class   := <WindowsForms10.Window.8.app.0.1a6f050_r13_ad1>  controltype:= <UIA_TableControlTypeId>  ,<50036>    , (0000C374)    

UIA_AutomationIdPropertyId :=dgvWIP
UIA_BoundingRectanglePropertyId :=283;402;1354;372
UIA_ClassNamePropertyId :=WindowsForms10.Window.8.app.0.1a6f050_r13_ad1
UIA_ControlTypePropertyId :=50036
UIA_CulturePropertyId :=0
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FrameworkIdPropertyId :=WinForm
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=True
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=False
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=True
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=False
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_LegacyIAccessibleChildIdPropertyId :=0
UIA_LegacyIAccessibleNamePropertyId :=DataGridView
UIA_LegacyIAccessibleRolePropertyId :=24
UIA_LegacyIAccessibleStatePropertyId :=1048580
UIA_LocalizedControlTypePropertyId :=table
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_NamePropertyId :=DataGridView
UIA_NativeWindowHandlePropertyId :=1179694
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=8912
UIA_ProviderDescriptionPropertyId :=[pid:148,hwnd:0x12002E Main:Nested [pid:8912,hwnd:0x12002E Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=42;1179694
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

Data Grid View Cell:

expandcollapse popup

Mouse position is retrieved 400-461
At least we have an element [OrderNo Row 0][]
Having the following values for all properties: 
Title is: <OrderNo Row 0>   Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    

UIA_BoundingRectanglePropertyId :=361;448;70;22
UIA_ControlTypePropertyId :=50025
UIA_CulturePropertyId :=0
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=True
UIA_HelpTextPropertyId :=DataGridViewTextBoxCell(DataGridViewCell)
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=False
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=True
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=False
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_LegacyIAccessibleChildIdPropertyId :=0
UIA_LegacyIAccessibleHelpPropertyId :=DataGridViewTextBoxCell(DataGridViewCell)
UIA_LegacyIAccessibleNamePropertyId :=OrderNo Row 0
UIA_LegacyIAccessibleRolePropertyId :=29
UIA_LegacyIAccessibleStatePropertyId :=3145734
UIA_LegacyIAccessibleValuePropertyId :=580902
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_NamePropertyId :=OrderNo Row 0
UIA_NativeWindowHandlePropertyId :=0
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=8012
UIA_ProviderDescriptionPropertyId :=[pid:8012,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

[junkew]

First of all within Windows UI Automation documentation is stated that those patterns are available for windows explorer

Grid Used for controls that support grid functionality such as sizing and moving to a specified cell, without header information. The “large icon view” in Windows® Explorer is an example of a control that follows the Grid control pattern. GridItem Used for controls within grids. For example, each cell in Explorer’s “details view” could follow the GridItem pattern.

So steps are

1. Get a UI Element object

2. Check the Is.....Pattern property

Seems to be false for your objects so no detailed pattern available that abstracts rows/cells (weird anyway as it seems just to be a WPF control)

UIA_IsGridItemPatternAvailablePropertyId :=False 
UIA_IsGridPatternAvailablePropertyId :=False

So to determine the way to handle your grid I suggest do do the following

1. Get your main UI element that contains the grid

2. _UIA_DumpThemAll(<UI Reference>, $treescope_subtree)

Then you can see the logic of the whole tree and build your own logic to iterate over the row's

see example 1 on the treewalkers on how to iterate thru the three

to work with /get  the gridpattern replace the _UIA_getPattern with this function (will be put later in a new version of UIAWrapper UDF)

expandcollapse popup

func _UIA_getPattern($obj,$patternID)
local $patternArray[21][3]=[ _
    [$UIA_ValuePatternId    ,           $sIID_IUIAutomationValuePattern    ,        $dtagIUIAutomationValuePattern], _
    [$UIA_InvokePatternId   ,           $sIID_IUIAutomationInvokePattern   ,        $dtagIUIAutomationInvokePattern], _
    [$UIA_SelectionPatternId,           $sIID_IUIAutomationSelectionPattern,        $dtagIUIAutomationSelectionPattern], _
    [$UIA_LegacyIAccessiblePatternId,   $sIID_IUIAutomationLegacyIAccessiblePattern,$dtagIUIAutomationLegacyIAccessiblePattern], _
    [$UIA_SelectionItemPatternId,       $sIID_IUIAutomationSelectionItemPattern,    $dtagIUIAutomationSelectionItemPattern], _
    [$UIA_RangeValuePatternId,          $sIID_IUIAutomationRangeValuePattern,       $dtagIUIAutomationRangeValuePattern], _
    [$UIA_ScrollPatternId,              $sIID_IUIAutomationScrollPattern,           $dtagIUIAutomationScrollPattern], _
    [$UIA_GridPatternId,                $sIID_IUIAutomationGridPattern,             $dtagIUIAutomationGridPattern], _
    [$UIA_GridItemPatternId,            $sIID_IUIAutomationGridItemPattern,         $dtagIUIAutomationGridItemPattern], _
    [$UIA_MultipleViewPatternId,        $sIID_IUIAutomationMultipleViewPattern,     $dtagIUIAutomationMultipleViewPattern], _
    [$UIA_WindowPatternId,              $sIID_IUIAutomationWindowPattern,           $dtagIUIAutomationWindowPattern], _
    [$UIA_DockPatternId,                $sIID_IUIAutomationDockPattern,             $dtagIUIAutomationDockPattern], _
    [$UIA_TablePatternId,               $sIID_IUIAutomationTablePattern,            $dtagIUIAutomationTablePattern], _
    [$UIA_TextPatternId,                $sIID_IUIAutomationTextPattern,             $dtagIUIAutomationTextPattern], _
    [$UIA_TogglePatternId,              $sIID_IUIAutomationTogglePattern,           $dtagIUIAutomationTogglePattern], _
    [$UIA_TransformPatternId,           $sIID_IUIAutomationTransformPattern,        $dtagIUIAutomationTransformPattern], _
    [$UIA_ScrollItemPatternId,          $sIID_IUIAutomationScrollItemPattern,       $dtagIUIAutomationScrollItemPattern], _
    [$UIA_ItemContainerPatternId,       $sIID_IUIAutomationItemContainerPattern,    $dtagIUIAutomationItemContainerPattern], _
    [$UIA_VirtualizedItemPatternId,     $sIID_IUIAutomationVirtualizedItemPattern,  $dtagIUIAutomationVirtualizedItemPattern], _
    [$UIA_SynchronizedInputPatternId,   $sIID_IUIAutomationSynchronizedInputPattern,$dtagIUIAutomationSynchronizedInputPattern], _
    [$UIA_ExpandCollapsePatternId,      $sIID_IUIAutomationExpandCollapsePattern,   $dtagIUIAutomationExpandCollapsePattern] _
        ]

    local $pPattern
    local $sIID_Pattern
    local $sdTagPattern
    local $i

    for $i=0 to ubound($patternArray)-1
        if $patternArray[$i][0]=$patternId Then
;~          consolewrite("Pattern identified " & @crlf)
            $sIID_Pattern=$patternArray[$i][1]
            $sdTagPattern=$patternArray[$i][2]
        EndIf
    next
;~  consolewrite($patternid & $sIID_Pattern & $sdTagPattern & @CRLF)

    $obj.getCurrentPattern($PatternId, $pPattern)
    $oPattern=objCreateInterface($pPattern, $sIID_Pattern, $sdtagPattern)
    if isobj($oPattern) Then
;~      consolewrite("UIA found the pattern" & @CRLF)
        return $oPattern
    Else
        _UIA_Debug("UIA WARNING ** NOT ** found the pattern" & @CRLF)
    endif
EndFunc

Edited November 4, 2013 by junkew

[Shrapnel]

Thanks junkew. I will study the information you provided. Thanks again for the great work!

[Jfish]

Junkew -

This is outstanding! The Spy you built is incredibly powerful. I would love to see Au3Info updated to include this functionality.

Well done.

[Jfish]

I am still playing with this trying to figure out how to click on found items and on example #2 to click the start menu button I get the following error:

**** All childs of the taskbar ****
Title is:   Class=Shell_TrayWnd
Title is: Start Class=Button
Title is:   Class=TrayNotifyWnd
Title is: Clock Class=TrayClockWClass
Title is: Show desktop  Class=TrayShowDesktopButtonWClass
Title is:   Class=SysPager
Title is: User Promoted Notification Area   Class=ToolbarWindow32
Title is: Fully charged (100%)  Class=
Title is: Unidentified network No network access ryanco.com Internet access Class=
Title is: Solve PC issues: 1 important message
5 total messages    Class=
Title is: Speakers / Headphones: 32%    Class=
Title is: Dropbox 2.0.22
Up to date  Class=
Title is: Connectify: Share your Internet
Hotspot disabled.   Class=
Title is: NotificationChevron   Class=Button
Title is:   Class=ReBarWindow32
Title is: Running applications  Class=MSTaskSwWClass
Title is: Running applications  Class=MSTaskListWClass
Title is: UIAutomation  Class=
Title is: Microsoft Excel 2010  Class=
Title is: Windows Virtual PC    Class=
Title is: Internet Explorer Class=
Title is: Microsoft Word    Class=
Title is: Mozilla Firefox   Class=
Title is: AutoIt Form Designer  Class=
Title is: Microsoft Outlook 2010    Class=
Title is: C:\Users\RC01712\Desktop\UIAutomation\startbtn.au3 - SciTE [7 of 7]   Class=
**** try to click on the start button of the taskbar ****
C:\Users\RC01712\Desktop\UIAutomation\UIAWrappers.au3 (475) : ==> Variable used without being declared.:
$objUIAutomation.createPropertyCondition($PropertyId, $tVal, $pCondition)
$objUIAutomation.createPropertyCondition($PropertyId, ^ ERROR
->11:15:54 AutoIt3.exe ended.rc:1
>Exit code: 1    Time: 0.728

Any thoughts on why?

[Shrapnel]

I wanted to get everyone's feed back on my idea to improve the _UIA_StartSUT function. Please let me know what you think!

Edit: Any suggestions/improvements are very welcome!

Modified _UIA_StartSUT function:

Func _UIA_StartSUT($SUT_VAR, $Return_hWnd = True)
    local $fullName = _UIA_getVar( $SUT_VAR & ".Fullname")
    local $processName = _UIA_getVar($SUT_VAR & ".processname")
    local $app2Start = $fullName & " " & _UIA_getVar($SUT_VAR & ".Parameters")
    local $workingDir =  _UIA_getVar($SUT_VAR & ".Workingdir")
    local $windowState = _UIA_getVar($SUT_VAR & ".Windowstate")
    local $result

    if fileexists($fullName) Then
        $result = processexists($processName)
        if $result = 0 Then
            _UIA_Debug("Starting : " & $app2Start & " from " & $workingDir)
            $result = Run($app2Start,$workingDir, $windowState )
            ProcessWait($processName,60)
            Sleep(500)

            _UIA_setVar($SUT_VAR & ".PID", $result)
        Else
            _UIA_setVar($SUT_VAR & ".PID", $result)
        EndIf
    Else
        _UIA_Debug("No clue where to find the system under test (SUT) on your system, please start manually:" & @CRLF )
        _UIA_Debug($app2Start & @CRLF)
    EndIf

    If $Return_hWnd Then
        Return _UIA_getSUThWnd($SUT_VAR)
    EndIf
EndFunc

Added _UIA_getSUThWnd function

Func _UIA_getSUThWnd($SUT_VAR)
    If Not ProcessExists(_UIA_getVar($SUT_VAR & ".PID")) Then
        Return _UIA_Debug("System Under Test Process ID does not exist" & @CRLF )
    Else
        _UIA_setVar($SUT_VAR & ".hWnd", _GetHwndFromPID(_UIA_getVar($SUT_VAR & ".PID")))
        Return HWnd(_UIA_getVar($SUT_VAR & ".hWnd")) ; must be converted to HWnd to work correctly
    EndIf
EndFunc

Added _GetHwndFromPID function from

Func _GetHwndFromPID($PID)
    ;http://www.autoitscript.com/forum/topic/86680-pid-window-handle-hwnd-conversion/#entry621521
    $hWnd = 0
    $stPID = DllStructCreate("int")
    Do
        $winlist2 = WinList()
        For $i = 1 To $winlist2[0][0]
            If $winlist2[$i][0] <> "" Then
                DllCall("user32.dll", "int", "GetWindowThreadProcessId", "hwnd", $winlist2[$i][1], "ptr", DllStructGetPtr($stPID))
                If DllStructGetData($stPID, 1) = $PID Then
                    $hWnd = $winlist2[$i][1]
                    ExitLoop
                EndIf
            EndIf
        Next
        Sleep(100)
    Until $hWnd <> 0
    Return $hWnd
EndFunc ;==>_GetHwndFromPID

small very simple example of use:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Version=Beta
#AutoIt3Wrapper_Change2CUI=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

_UIA_setVar("SUT1.Folder", @WindowsDir & "\System32")
_UIA_setVar("SUT1.Workingdir",@WindowsDir & "\System32\")
_UIA_setVar("SUT1.EXE","calc.exe")
_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))
_UIA_setVar("SUT1.Parameters","")
_UIA_setVar("SUT1.Processname","calc.exe")
_UIA_setVar("SUT1.Windowstate",@SW_RESTORE)

_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))


;~ Start system under test
MsgBox(0,"", "Get winow title of SUT from _UIA_StartSUT = " & WinGetTitle(_UIA_StartSUT("SUT1", True)))

WinSetTitle(_UIA_getSUThWnd("SUT1"), "", "New Title") ; Change the title of the calculator window

MsgBox(0,"", "The window title of the calculator was changed from 'Calculator' to 'New Title'")

MsgBox(0,"", "Now you can get the updated window title by using _UIA_getSUThWnd = " & WinGetTitle(_UIA_getSUThWnd("SUT1")))

Exit

Edited November 5, 2013 by Shrapnel

[junkew]

@JFish: Examples fixed, download examples and UIWrappers from first post and retry.

 

@Shrapnel: Nice suggestion, I implemented in a little different way see updated UIWrappers

;~ Wait for the window to be there
        $oSUT=_UIA_getObjectByFindAll($oDesktop, "processid:=" & $result2, $treescope_children)
        if not isobj($oSUT) Then
            _UIA_Debug("No window found in SUT : " & $app2Start & " from " & $workingDir)
        Else
        ;~ Add it to the Runtime Type Information
            _UIA_setVar("RTI." & $SUT_VAR & ".PID", $result2)
            _UIA_setVar("RTI." & $SUT_VAR & ".HWND", hex(_UIA_getPropertyValue($oSUT, $UIA_NativeWindowHandlePropertyId)))
;~          _UIA_DumpThemAll($oSUT,$treescope_subtree)
        EndIf

Tricky part is with splashscreens in that case you could have the wrong HWND in RTI.SUTx.HWND

[Shrapnel]

Fo the RTI.SUT.HWND, you should use the hwnd() function instead of hex()

It seems to be a limitation to the data dictionary to not save the hwnd() value, or not save it in the correct way, so it must be converted each time you call it. see examples below

Edit _UIA_StartSUT function as follows:

;~ Wait for the window to be there
        $oSUT=_UIA_getObjectByFindAll($oDesktop, "processid:=" & $result2, $treescope_children)
        if not isobj($oSUT) Then
            _UIA_Debug("No window found in SUT : " & $app2Start & " from " & $workingDir)
        Else
        ;~ Add it to the Runtime Type Information
            _UIA_setVar("RTI." & $SUT_VAR & ".PID", $result2)
            _UIA_setVar("RTI." & $SUT_VAR & ".HWND", HWnd(_UIA_getPropertyValue($oSUT, $UIA_NativeWindowHandlePropertyId)))
                _UIA_Debug("just get HWND = " & _UIA_getVar("RTI.SUT1.HWND") & @CRLF)
                _UIA_Debug("convert HWND with HWND function = " & hwnd(_UIA_getVar("RTI.SUT1.HWND")) & @CRLF)

            _UIA_setVar("RTI." & $SUT_VAR & ".HWND", Hex(_UIA_getPropertyValue($oSUT, $UIA_NativeWindowHandlePropertyId)))
                _UIA_Debug("just hex = " & _UIA_getVar("RTI.SUT1.HWND") & @CRLF)
                _UIA_Debug("Hex converted to HWND with HWND function= " & HWnd(_UIA_getVar("RTI.SUT1.HWND")) & @CRLF)

;~          _UIA_DumpThemAll($oSUT,$treescope_subtree)
        EndIf

Run this script

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Version=Beta
#AutoIt3Wrapper_Change2CUI=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

_UIA_setVar("SUT1.Folder", @WindowsDir & "\System32")
_UIA_setVar("SUT1.Workingdir",@WindowsDir & "\System32\")
_UIA_setVar("SUT1.EXE","calc.exe")
_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))
_UIA_setVar("SUT1.Parameters","")
_UIA_setVar("SUT1.Processname","calc.exe")
_UIA_setVar("SUT1.Windowstate",@SW_RESTORE)

_UIA_setVar("SUT1.Fullname", _UIA_getVar("SUT1.Folder") & "\"& _UIA_getVar("SUT1.EXE"))

_UIA_StartSUT("SUT1")

Exit

And view the console output (red text is the only hwnd autoit will recognize):

just get HWND = 590240
convert HWND with HWND function = 0x000901A0
just hex = 000901A0
Hex converted to HWND with HWND function= 0x00000000

Edited November 6, 2013 by Shrapnel

[Shrapnel]

So maybe the solution to this is to setVar as hwnd() and in the getVar() function to return hwnd() if the SUT begins with RTI and ends with hwnd

 

Edit: something like this:

Modified _UIA_getVar() func:

func _UIA_getVar($varName)
    if $UIA_VARS.exists($varName) Then
        If StringRight($varname, 5) = ".HWND" Then
            If StringLeft($varname, 4) = "RTI." Then
                Return HWnd($UIA_Vars($varName))
            Else
                Return HWnd($UIA_Vars("RTI." & $varName))
            EndIf
        Else
            Return $UIA_Vars($varName)
        EndIf
    Else
        SetError(1) ;~ Not defined in repository
        return "*** ERROR *** " & $varname & " not found!"
    endif
EndFunc

Modified _UIA_StartSUT() so it sets mainwindow in RTI. (side note: $result2 is redundant variable. could just use $result)

expandcollapse popup

Func _UIA_StartSUT($SUT_VAR, $hasWindow = True)
    Local $fullName=_UIA_getVar( $SUT_VAR & ".Fullname")
    Local $processName=_UIA_getVar($SUT_VAR & ".processname")
    Local $app2Start=$fullName & " " & _UIA_getVar($SUT_VAR & ".Parameters")
    Local $workingDir= _UIA_getVar($SUT_VAR & ".Workingdir")
    Local $windowState=_UIA_getVar($SUT_VAR & ".Windowstate")
    Local $result, $result2   ; Holds the process id's

    If FileExists($fullName) Then
;~      Only start new instance when not found
        $result2 = processexists($processName)
        If $result2 = 0 Then
            _UIA_Debug("Starting : " & $app2Start & " from " & $workingDir & @CRLF)
            $result = run($app2Start,$workingDir, $windowState )
            $result2 = ProcessWait($processName, 60)
;~          Sleep(500) ;~ Just to give the system some time to show everything
        EndIf

#cs ### Old Section ###
        $oSUT = _UIA_getObjectByFindAll($oDesktop, "processid:=" & $result2, $treescope_children)

        If Not IsObj($oSUT) Then
            _UIA_Debug("No window found in SUT : " & $app2Start & " from " & $workingDir & @CRLF)
        Else
        ;~ Add it to the Runtime Type Information
            _UIA_setVar("RTI." & $SUT_VAR & ".PID", $result2)
            _UIA_setVar("RTI." & $SUT_VAR & ".HWND", HWnd(_UIA_getPropertyValue($oSUT, $UIA_NativeWindowHandlePropertyId)))
        EndIf
#ce; ### Old Section ###

#Region ### New Section ###
        If $hasWindow Then
            Do
                Sleep(100)
                $oSUT = _UIA_getObjectByFindAll($oDesktop, "processid:=" & $result2, $treescope_children)
            Until IsObj($oSUT)

            ;~ Add it to the Runtime Type Information
            _UIA_setVar("RTI." & $SUT_VAR & ".PID", $result2)
            _UIA_setVar("RTI." & $SUT_VAR & ".HWND", HWnd(_UIA_getPropertyValue($oSUT, $UIA_NativeWindowHandlePropertyId)))
            _UIA_setVar("RTI." & $SUT_VAR & ".mainwindow", $oSUT)
        EndIf
#EndRegion ### New Section  ###

    Else
        _UIA_Debug("No clue where to find the system under test (SUT) on your system, please start manually:" & @CRLF )
        _UIA_Debug($app2Start & @CRLF)
    EndIf
EndFunc

Edited November 6, 2013 by Shrapnel

[Jfish]

It is hard for me to contribute to any meaningful part of this code as I am still trying to understand it all (though I love it and have used it for some things already).  I think the original simple Spy was awesome.  My tiny contribution is shrinking it down and adding some functionality with some additional hotkeys to copy to the clipboard, open and paste into Notepad, and toggle the mouse capture on and off.  I find it easier to spy things this way.  Hope it is helpful to someone.

expandcollapse popup

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
#include <Misc.au3>

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

dim $oldUIElement ; To keep track of latest referenced element

;~ Some references for reading
;~ [url=http://support.microsoft.com/kb/138518/nl]http://support.microsoft.com/kb/138518/nl[/url]  tagpoint structures attention point
;~ [url=http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/]http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/[/url]
;~ [url=http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx]http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx[/url]

HotKeySet("{ESC}", "Close") ; Set ESC as a hotkey to exit the script.
HotKeySet("^w", "GetElementInfo") ; Set Hotkey Ctrl+M to get some basic information in the GUI
HotKeySet("^c", "_copyedit")
HotKeySet("^p", "_pasteedit")
HotKeySet("^t", "_togglemouse")

#Region ### START Koda GUI section ### Form=c:\users\rc01712\desktop\uiautomation\spy2.kxf
$Form1_1 = GUICreate("Spy 2", 630, 479, 192, 132)
$Edit1 = GUICtrlCreateEdit("", 8, 16, 385, 441)
GUICtrlSetData(-1, "Edit1")
$Label1 = GUICtrlCreateLabel("Control+w to capture information", 400, 48, 201, 20)
$Label2 = GUICtrlCreateLabel("Control+p to open Notepad and paste", 400, 100, 221, 20)
$Label3 = GUICtrlCreateLabel("Control+c to copy Info to Clipboard", 400, 74, 211, 20)
$Label4 = GUICtrlCreateLabel("Esc to exit", 400, 146, 64, 20)
$Label5 = GUICtrlCreateLabel("Control+t to toggle mouse capture", 400, 125, 209, 20)
$Clear = GUICtrlCreateButton("Clear edit", 448, 192, 121, 25)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

_UIA_Init()

dim $content=""
dim $toggle=0

; Run the GUI until the dialog is closed
While true
$msg = GUIGetMsg()
sleep(100)
if _ispressed(01) AND $toggle=1 Then
getelementinfo()
endif
if $msg=$clear Then
    GUICtrlSetData($Edit1,"")
EndIf
If $msg = $GUI_EVENT_CLOSE Then ExitLoop
WEnd

func _copyEdit()
    $content=guictrlread($Edit1)
    ClipPut($content)
EndFunc

func _pasteedit()
    run("notepad.exe")
    _UIA_setVar("Notepad.mainwindow","classname:=Notepad") ; works
    _UIA_setVar("Notepad.document","classname:=Edit")
    _UIA_action("Notepad.Mainwindow","setfocus"); works
    send("^v")
EndFunc

func _togglemouse()
    if $toggle=0 then
        $toggle=1
    Else
        $toggle=0
    EndIf
    ;MsgBox("","",$toggle)
EndFunc


Func GetElementInfo()
Local $hWnd
Local $tStruct = DllStructCreate($tagPOINT) ; Create a structure that defines the point to be checked.
;~ Local $tStruct = DllStructCreate("INT64,INT64")
;~ Local $tStruct =_AutoItObject_DllStructCreate($tagPoint)

ToolTip("")

$x=MouseGetPos(0)
$y=MouseGetPos(1)
DllStructSetData($tStruct, "x", $x)
DllStructSetData($tStruct, "y", $y)
consolewrite(DllStructGetData($tStruct,"x") & DllStructGetData($tStruct,"y"))

;~ consolewrite("Mouse position is retrieved " & @crlf)
$objUIAutomation.ElementFromPoint($tStruct,$pUIElement)
;~ $objUIAutomation.ElementFromPoint(DllStructGetPtr($tStruct),$pUIElement)
;~ consolewrite("Element from point is passed, trying to convert to object ")
$oUIElement = objcreateinterface($pUIElement,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)

if isobj($oldUIElement) Then
if $oldUIElement=$oUIElement then
return
endif
endif
_WinAPI_RedrawWindow(_WinAPI_GetDesktopWindow(), 0, 0, $RDW_INVALIDATE + $RDW_ALLCHILDREN) ; Clears Red outline graphics.
GUICtrlSetData($Edit1, "Mouse position is retrieved " & $x & "-" & $y & @CRLF)
$oldElement=$oUIElement

If IsObj($oUIElement) Then
;~  ConsoleWrite("At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF)
GUICtrlSetData($Edit1, "At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF,1)
    $text1="Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= " _
& "<" &  _UIA_getControlName(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ">" & @TAB  _
& ",<" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB  _
& ", (" &  hex(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ")" & @TAB & @CRLF
$text1= $text1 & @CRLF & getAllPropertyValues($oUIElement)
ConsoleWrite($text1)
GUICtrlSetData($Edit1, "Having the following values for all properties: " & @crlf & $text1 & @CRLF, 1)


_GUICtrlEdit_LineScroll($Edit1, 0, 0 - _GUICtrlEdit_GetLineCount($Edit1))

$t=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf

EndFunc   ;==>Example

Func Close()
Exit
EndFunc   ;==>Close

[Shrapnel]

The changes I made in post #55 caused a "bug" in _UIA_Action() because it was not checking if the main window already existed in RTI.

To account for this, I modified the _UIA_Action() function as follows:

expandcollapse popup

Func _UIA_action($obj_or_string, $strAction, $p1=0, $p2=0, $p3=0)
    Local $tPattern
    Local $x, $y
;~  Local $objElement
    Local $oElement

;~ If we are giving a description then try to make an object first by looking from repository
;~ Otherwise assume an advanced description we should search under one of the previously referenced elements at runtime
    If isobj($obj_or_string) Then
        $oElement=$obj_or_string
        $obj=$obj_or_string
    Else
        $tPhysical=_UIA_getVar($obj_or_string)
;~ If not found in repository assume its a physical description
        If @error=1 Then
            _UIA_DEBUG("Finding object (bypassing repository) with physical description " & $tPhysical & @CRLF)
            $tPhysical=$obj_or_string
        EndIf

        _UIA_DEBUG("Finding object with physical description " & $tPhysical & @CRLF)

#Region ###; Updated If Statement ###
            If stringinstr($obj_or_string,".mainwindow") Then ;~ if its a mainwindow reference find it under the desktop
                If IsObj(_UIA_getVar("RTI." & $obj_or_string)) Then ; If object already exists in RTI, then use it
                    $oElement = _UIA_getVar("RTI." & $obj_or_string)
;~                  _UIA_setVar("RTI.MAINWINDOW", $oElement) ; <-------------------------------------------------------### Not sure if this should be here or not. junkew, please advise ###
                Else ; Else, find the object
                    $startElement = "Desktop"
                    $oStart = $oDesktop
                    $oElement =_UIA_getObjectByFindAll($oStart, $tPhysical, $treescope_subtree,$obj_or_string)
                    _UIA_setVar("RTI.MAINWINDOW",$oElement)
                EndIf
            Else ;~     Find the object under the last referenced mainwindow / parent window
                $startElement = "RTI." & stringleft($obj_or_string,stringinstr($obj_or_string,".")) & "mainwindow"
                $oStart = _UIA_getVar($startElement)
                $oElement = _UIA_getObjectByFindAll($oStart, $tPhysical, $treescope_subtree)
            EndIf
#EndRegion ###; Updated If Statement ###

        If IsObj($oElement) Then ; Continue the action by setting the $obj value to an UIA element
            $obj = $oElement
        Else
            Return SetError(1)
        Endif
    EndIf

    _UIA_setVar("RTI.ACTIONCOUNT", _UIA_getVar("RTI.ACTIONCOUNT") + 1)
    _UIA_DEBUG("Action " & _UIA_getVar("RTI.ACTIONCOUNT") & " " & $strAction & " on " & $obj_or_string & @CRLF & "   ")

    $strAction = StringLower($strAction)
    Switch $strAction
        Case "leftclick", "left", "rightclick", "right"
            Local $mouseButton = "left"
            $t = StringSplit(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId),";")
            $x = $t[1] + ($t[3] / 2)
            $y = $t[2] + ($t[4] / 2)
            If StringInStr($strAction, "right") Then $mouseButton = "right"
            If $p1 = 0 Then $p1 = 1
            MouseClick($mouseButton, $x, $y, $p1, $p2)
            If $p3 > 0 Then Sleep($p3)
        Case "setvalue using keys"
            $obj.setfocus
            Send("^a")
            Send($p1)
            If $p2 > 0 Then Sleep($p2)
        Case "invoke"
            $tPattern=_UIA_getPattern($obj,$UIA_InvokePatternId)
            $tPattern.invoke()
        Case "focus"
            $obj.setfocus()
        Case Else
    EndSwitch
EndFunc

[Shrapnel]

 

It is hard for me to contribute to any meaningful part of this code as I am still trying to understand it all (though I love it and have used it for some things already).  I think the original simple Spy was awesome.  My tiny contribution is shrinking it down and adding some functionality with some additional hotkeys to copy to the clipboard, open and paste into Notepad, and toggle the mouse capture on and off.  I find it easier to spy things this way.  Hope it is helpful to someone.

 

Very nice Jfish!

Edited November 7, 2013 by Shrapnel

[Shrapnel]

Ok, i kinda feel like im spamming this thread , but I'm hoping some find it useful.

I added two functions to UIAWrappers.au3, _UIA_setVarsFromArray and _UIA_getVarsFromArray

Func _UIA_setVarsFromArray(ByRef $_array)
    If Not IsArray($_array) Then Return 0
    For $x = 1 To $_array[0][0] Step 1
        _UIA_setVar($_array[$x][0], $_array[$x][1])
    Next
EndFunc


Func _UIA_getVarsFromArray(ByRef $_array)
    If Not IsArray($_array) Then Return 0
    Local $returnArray = $_array
    For $x = 1 To $returnArray[0][0] Step 1
        $returnArray[$x][1] = _UIA_getVar($returnArray[$x][0])
    Next
    Return $returnArray
EndFunc

Here is an example of their use:

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <Array.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

Local $CalcFolder = @WindowsDir & "\system32\"

Local $SUT_Array [8][2]= [ _
    [7,2], _
    ["Calculator.Folder",       $CalcFolder], _
    ["Calculator.Workingdir",   StringTrimRight($CalcFolder, 1)], _
    ["Calculator.exe",          "calc.exe"], _
    ["Calculator.Fullname",     $CalcFolder & "calc.exe"], _
    ["Calculator.Parameters",   ""], _
    ["Calculator.Processname",  "calc.exe"], _
    ["Calculator.Windowstate",  @SW_RESTORE] _
]

_ArrayDisplay($SUT_Array, "$SUT_Array")

_UIA_setVarsFromArray($SUT_Array)

ConsoleWrite("_UIA_getVar(""Calculator.Folder"") = " & _UIA_getVar("Calculator.Folder") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.Workingdir"") = " & _UIA_getVar("Calculator.Workingdir") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.exe"") = " & _UIA_getVar("Calculator.exe") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.Fullname"") = " & _UIA_getVar("Calculator.Fullname") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.Parameters"") = " & _UIA_getVar("Calculator.Parameters") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.Processname"") = " & _UIA_getVar("Calculator.Processname") & @CRLF)
ConsoleWrite("_UIA_getVar(""Calculator.Windowstate"") = " & _UIA_getVar("Calculator.Windowstate") & @CRLF)

Local $tempArray = _UIA_getVarsFromArray($SUT_Array)

_ArrayDisplay($tempArray, "$tempArray")

_UIA_StartSUT("Calculator")

Exit

This could be useful because now you can create arrays that contain all the controls, RTI, etc of an object and easily set/get them in the data dictionary.

Edited November 8, 2013 by Shrapnel

[junkew]

It makes indeed sense to have the definitions of the elements split from the actions.

My intention is to get those from definition files (still in doubt if I should do it in INI files or XML files) but will include your array method logic too as then you can compile everything without dependencies on external files when you have a small script.

Not sure If I will switch to your convention for starting the SUT. Within the framework its a little easier to use SUT1 thru SUT9, SUTnn as reference and only once the main SUT references are there use the

calculator.button1, calculator.button2 convention for the elements but I am open for suggestions

the mainwindow should be defined upfront and if detected nonexisting in UIA action function it should not be added (but maybe I have to add logic to recover/search from another starting point in the tree)

@jfish: I will be working on a more extended spy with logical tree and screenshots and other shortcut keys so please post more suggestions/features you would like in a UIA based spy