Jump to content
matt55777

_UIA_getObjectByFindAll does not find $oP0 / simplespy does not disambiguate objects with similar properties [Bug]&[Workaround]

Recommended Posts

Posted (edited)

Hi,
I am automating a powerpoint addon. Runs fine so far except at some point I need to click an object but the simplespy code fails to find the $oP0 object. The code below

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Présentation1 - PowerPoint;controltype:=UIA_WindowControlTypeId;class:=PPTFrameClass", $treescope_children)
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=Office Timeline;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_ListControlTypeId;class:=ListView", $treescope_children)
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Otl.Views.CustomControls.WizardPage;controltype:=UIA_ListItemControlTypeId;class:=ListBoxItem", $treescope_children)
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Data;controltype:=UIA_GroupControlTypeId;class:=Expander", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("Data.mainwindow", "title:=Data;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Data;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

Consolewrite("$oUIElement obj ? " & isobj($oUIElement) & " P0 " & isobj($oP0) & " P1 " & isobj($oP1) & _
              " P2 " & isobj($oP2) & " P3 " & isobj($oP3) & " P4 " & isobj($oP4))

returns:

Console:
  >>> $oUIElement obj ? 0 P0 0 P1 1 P2 1 P3 1 P4 1

So $oP4 - $oP1 are ok but $oP0 is not found by _UIA_getObjectByFindAll.

The inspect tool from Microsoft gives me the following for object $oP0:

How found:  Selected from tree...
Name:   "Data"
ControlType:    UIA_GroupControlTypeId (0xC36A)
LocalizedControlType:   "group"
BoundingRectangle:  {l:1129 t:84 r:1165 b:684}
IsEnabled:  true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:   false
AcceleratorKey: ""
AccessKey:  ""
ProcessId:  11580
RuntimeId:  [7.2D3C.16F6631]
AutomationId:   "PART_Expander"
FrameworkId:    "WPF"
ClassName:  "Expander"
IsControlElement:   true
IsContentElement:   true
ProviderDescription:    "[pid:11580,hwnd:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]"
IsPeripheral:   [Not supported]
LiveSettingProperty:    [Not supported]
IsPassword: false
ItemStatus: ""
ItemType:   ""
IsRequiredForForm:  false
HelpText:   ""
FlowsFrom:  [Not supported]
ClickablePoint: {x:1147 y:384}
Orientation:    0
OptimizeForVisualContent:   [Not supported]
PositionInSet:  [Not supported]
SizeOfSet:  [Not supported]
Level:  [Not supported]
AnnotationTypes:    [Not supported]
AnnotationObjects:  [Not supported]
LandmarkType:   [Not supported]
LocalizedLandmarkType:  [Not supported]
FullDescription:    [Not supported]
Annotation.AnnotationTypeName:  [Not supported]
Annotation.AnnotationTypeId:    [Not supported]
Annotation.Author:  [Not supported]
Annotation.DateTime:    [Not supported]
Annotation.Target:  [Not supported]
Drag.DropEffect:    [Not supported]
Drag.DropEffects:   [Not supported]
Drag.GrabbedItems:  [Not supported]
Drag.IsGrabbed: [Not supported]
DropTarget.DropTargetEffect:    [Not supported]
DropTarget.DropTargetEffects:   [Not supported]
ExpandCollapse.ExpandCollapseState: Collapsed (0)
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    "Développer"
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "Data"
LegacyIAccessible.Role: groupement (0x14)
LegacyIAccessible.State:    réduit,pouvant être actif (0x100400)
LegacyIAccessible.Value:    ""
ObjectModel.UnderlyingObjectModel:  [Error: calling getter for this property: hr=0xFFFFFFFF80070057 - Paramètre incorrect.]
SpreadsheetItem.AnnotationObjects:  [Not supported]
SpreadsheetItem.AnnotationTypes:    [Not supported]
SpreadsheetItem.Formula:    [Not supported]
Style.ExtendedProperties:   [Not supported]
Style.FillColor:    [Not supported]
Style.FillPatternColor: [Not supported]
Style.FillPatternStyle: [Not supported]
Style.Shape:    [Not supported]
Style.StyleId:  [Not supported]
Style.StyleName:    [Not supported]
Transform2.CanZoom: [Not supported]
Transform2.ZoomLevel:   [Not supported]
Transform2.ZoomMinimum: [Not supported]
Transform2.ZoomMaximum: [Not supported]
IsAnnotationPatternAvailable:   [Not supported]
IsDragPatternAvailable: [Not supported]
IsDockPatternAvailable: false
IsDropTargetPatternAvailable:   [Not supported]
IsExpandCollapsePatternAvailable:   true
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsObjectModelPatternAvailable:  [Not supported]
IsRangeValuePatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsScrollPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:  [Not supported]
IsSpreadsheetPatternAvailable:  [Not supported]
IsStylesPatternAvailable:   [Not supported]
IsSynchronizedInputPatternAvailable:    true
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    [Not supported]
IsTextEditPatternAvailable: [Not supported]
IsTextPatternAvailable: false
IsTextPattern2Available:    [Not supported]
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:   [Not supported]
IsValuePatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsWindowPatternAvailable:   false
IsCustomNavigationPatternAvailable: [Not supported]
FirstChild: "Data" button
LastChild:  "" button
Next:   [null]
Previous:   [null]
Other Props:    Object has no additional properties
Children:   "Data" button
    "Import from Excel" text
    "" custom
    "Data" text
    "" button
    "" button
    "" button
    "" button
    "" button
Ancestors:  "Otl.Views.CustomControls.WizardPage" list item
    "" list view
    "Office Timeline" window
    "Présentation1 - PowerPoint" fenêtre

 

How can I troubleshoot this?

 

EDIT :

It looks like the problem is that simplespy does not check for parents having the same name in the tree.

In my case, i had:

fogWjlp.png

I want to press the "Data" button but as you can see there are 2 Otl.Views.CustomControls.WizardPage objects and _UIA_getObjectByFindAll selects the wrong one which explains why $oP0 is not found.

As a workaround, I added

ConsoleWrite(_UIA_getBasePropertyInfo($oUIElement))

to the findThemAll function to try to find a disambiguation variable. In my case, automationid := <DataSelectionPage> worked when I added it in _UIA_getObjectByFindAll

Local $oP1=_UIA_getObjectByFindAll($oP2, "AutomationId:=DataSelectionPage;Title:=Otl.Views.CustomControls.WizardPage;controltype:=UIA_ListItemControlTypeId;class:=ListBoxItem", $treescope_children)

 

 

 

Edited by matt55777
new information

Share this post


Link to post
Share on other sites
Posted (edited)

Use index or indexrelative properties. See examples.zip how to disambuguate by index position. Thats one of the reasons for calling it a simplespy.

extended with example

  1.  Made with simplespy on calc
  2. Added the run and winwaitactive
  3. Extended the concept of oUIElement to oUIElement1,2,3 with index:=nn
  4. Extended the highlghlighting of the buttons by addiing UIA_Action on the 3 elements
;~  *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $iPID = Run("calc.exe")
winwaitactive("[REGEXPTITLE:Calc*]")

_UIA_setVar("oP1","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow")   ;Calculator
_UIA_setVar("oP2","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow")   ;Calculator
_UIA_setVar("oP3","Title:=;controltype:=UIA_GroupControlTypeId;class:=LandmarkTarget")  ;
_UIA_setVar("oP4","Title:=Number pad;controltype:=UIA_GroupControlTypeId;class:=NamedContainerAutomationPeer")  ;Number pad

;~ $oUIElement=_UIA_getObjectByFindAll("Seven.mainwindow", "title:=Seven;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement1","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=1") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")
_UIA_setVar("oUIElement2","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=2") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")
_UIA_setVar("oUIElement3","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=3") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")

_UIA_action("oUIElement1","highlight")
;~_UIA_action("oUIElement","click")
_UIA_action("oUIElement2","highlight")

_UIA_action("oUIElement3","highlight")

 

Edited by junkew

Share this post


Link to post
Share on other sites

Thanks for your patience, I have been using your framework only for a couple of days so I am still learning! I take note that index & indexrelative can also be used to disambiguate.

I think I remember from Microsoft site that automationID is guaranteed to be unique amongst siblings. It means that if it is defined then automationid is enough to choose the right node all the time, correct? If that is the case, maybe simplespy could be enhanced to include this in the code generator when automationid exists?!

Share this post


Link to post
Share on other sites

I have never used automationid as i think its only unique for the running instance and other reason is that i prefer name or title stuff in it  so i know myself which element it is. Future version i will put on todo to have it configurable per classtype

Share this post


Link to post
Share on other sites

Another quote
 

Quote

When it is available the AutomationIdProperty of an element is expected to always be the same in any instance of the application regardless of the local language. The value should be unique among sibling elements but not necessarily unique across the entire desktop.

Share this post


Link to post
Share on other sites

in same article you can read this is true for 1 build but not neccesarily for next (nightly) build. Close collaboration with developers is needed to have accessibility identification with any property. If you treat the app as a black box then in one case title,class is unique and in others you need more.

Feel free to modify simplespy to your need.

Share this post


Link to post
Share on other sites

I looked thru my W10 system for AutomationId and see its not allways having a value but however I think there are some usefull purposes in automation to make use of this when its filled (I make then the assumption developers fill this properly)

Rules I will apply in future spy

  1.  If there iis an automation id use that and put in comments the alternative description (as is used now)

It will help a little in

  • Multilanguage application's   
    • Calc.exe:   
      • name := ((Calculator)|(Rekenmachine))  as its not having an automationid
      • AutomationId:=equalsButton                 (instead of a regex)
    • Windows itself is inconsistent directly on start and taskbar many are not having an automationid

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...