Introspection and Actions; UIA_PaneControlTypeId and A Grid Within

[Sn3akyP3t3]

I'm trying to conduct UI automation using AutoIt through use of the UI Automation library provided by @junkew.  The reason why I'm using the UI Automation library and not just AutoIt's out of box controls is because the application I'm trying to automate was created using both standard and non-standard windows controls.  Some were built in-house which is likely the reason for the difficulty I'm experiencing, but I believe there should be a way to do it because both the spy tool provided by @junkew and the Windows inspect.exe tool can both provide information about individual elements of the Grid, but I'm failing to find a coded solution.  I would appreciate help in the right direction.

NOTE: I changed the name of the element of interest as well as the Window since the application is not mine to dispense.

At this moment I'm trying to solve two problems:

1. Perform some allowed UI Automation action on the element of a Grid with the specified text.
2. Extract all the elements within the Grid for analysis.  Similar to the idea of a dump I guess.

The simplespy.exe application produces this code, which works to some degree.  It puts the Grid in focus which brings the previously clicked row into focus, but it does not select a desired element or row with the text specified.

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow", $treescope_children)   
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer", $treescope_children)    
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP0,"setfocus")
_UIA_setVar("SalesorderEntry.mainwindow","title:=RowOfInterestText;classname:=GRID")
_UIA_action("SalesorderEntry.mainwindow","setfocus") 

simplespy.exe also produced this information about the focused element:

Mouse position is retrieved 2177-480
At least we have an element [RowOfInterestText][GRID]

Having the following values for all properties: 
Title is: <RowOfInterestText>   Class   := <GRID>   controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;318;588;466
*** Parent Information top down ***
5: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1200
"Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769"   
4: Title is: <WindowTitle>  Class   := <ATL:4476F950>   controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    2020;250;1256;583
"Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950"   
3: Title is: <> Class   := <WTL_SplitterWindow> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow"  
2: Title is: <> Class   := <WTL_PaneContainer>  controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer"   
1: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"  
0: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;305;1224;479
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"

expandcollapse popup

*** Detailed properties of the highlighted element ***
UIA_title:= <RowOfInterestText>
UIA_text:= <RowOfInterestText>
UIA_regexptitle:= <RowOfInterestText>
UIA_class:= <GRID>
UIA_regexpclass:= <GRID>
UIA_iaccessiblechildId:= <0>
UIA_id:= <249>
UIA_handle:= <4526846>
UIA_RuntimeId:= <42;4526846>
UIA_BoundingRectangle:= <2037;318;588;466>
UIA_ProcessId:= <6892>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_Name:= <RowOfInterestText>
UIA_HasKeyboardFocus:= <True>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_AutomationId:= <249>
UIA_ClassName:= <GRID>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <4526846>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Win32>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <RowOfInterestText>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048580>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:9208,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False> 

I experienced success as well with the Windows inspect.exe in identifying the text of the element of interest when I put focus on it:

expandcollapse popup

This is what inspect.exe shows:
How found:  Mouse move (2161,483)
hwnd=0x004512FE 32bit class="GRID" style=0x52810371 ex=0x4
RuntimeId:  "[42.4526846]"
BoundingRectangle:  {l:2037 t:318 r:2625 b:784}
ProcessId:  6892
ControlType:    UIA_PaneControlTypeId (0xC371)
LocalizedControlType:   "pane"
Name:   "RowOfInterestText"
AccessKey:  ""
HasKeyboardFocus:   true
IsKeyboardFocusable:    true
IsEnabled:  true
AutomationId:   "249"
ClassName:  "GRID"
HelpText:   ""
IsPassword: false
NativeWindowHandle: 0x4512FE
IsOffscreen:    false
FrameworkId:    "Win32"
ProviderDescription:    "[pid:6872,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]"
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    ""
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "RowOfInterestText"
LegacyIAccessible.Role: client (0xA)
LegacyIAccessible.State:    focused,focusable (0x100004)
LegacyIAccessible.Value:    ""
IsDockPatternAvailable: false
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsRangeValuePatternAvailable:   false
IsScrollPatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsTablePatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTextPatternAvailable: false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsValuePatternAvailable:    false
IsWindowPatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsSynchronizedInputPatternAvailable:    false
FirstChild: [null]
LastChild:  [null]
Next:   "Menu" text
Previous:   [null]
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "" dialog
    "" dialog
    "" pane
    "" pane
    "WindowTitle" window
    "Desktop" pane
    [ No Parent ]

[junkew]

Hard to tell how to deal with it

* no grid pattern

Suggestion is to look at the tree of inspect.exe how the grid is organized and if it has child items

within code you could then tree to iterate over the childitems of the grid

[Sn3akyP3t3]

@junkew Thanks.  Looks like there is no tree like structure with child items to work with.  I may have to use some other techniques then such as Sikuli, if that's accurate enough with the OpenCV OCR portion, for this part.