IUIAutomation embedded Chrome_RenderWidgetHostHWND

[BigDaddyO]

For the past few years i've been doing automated regression testing of the HealthCare app called Facets.

We just installed a new version which has a new web-like module that is an empty box that I can't see inside with any tools.  I've tried the Au3Info, SimpleSpy, UIASpy, and even that Powershell spy and nothing can see inside to the individual dropdowns, edits, or anything within Chrome_RenderWidgetHostHWND.

I've tried enabling all the accessibility options in Chrome browser and leaving it open prior to launching Facets but that didn't seem to do anything.

Also, I can't right click in that control to use Inspect either.

 

Here is the code I have so far, and it never finds any items under the Chrome_RenderWidgetHostHWND

expandcollapse popup

#include ".\Include\UIAWrappers.au3"

WinActivate("Facets - Billing 360", "")

;Find the Billing 360 Chrome Render Widget Host
Local $oFacetsWin = _UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Facets;controltype:=UIA_WindowControlTypeId;class:=Afx", $treescope_children)
If IsObj($oFacetsWin) Then
    ConsoleWrite("Found Object $oFacetsWin" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oFacetsWin" & @CRLF)
    Exit
EndIf
_UIA_Action($oFacetsWin,"setfocus")


Local $oWorkspace = _UIA_getObjectByFindAll($oFacetsWin, "Title:=Workspace;controltype:=UIA_PaneControlTypeId;class:=MDIClient", $treescope_children)
If IsObj($oWorkspace) Then
    ConsoleWrite("Found Object $oWorkspace" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oWorkspace" & @CRLF)
    Exit
EndIf
_UIA_Action($oWorkspace,"setfocus")


Local $oFacetsEmbededWin = _UIA_getObjectByFindAll($oWorkspace, "Title:=Billing 360;controltype:=UIA_WindowControlTypeId;class:=Afx", $treescope_children)
If IsObj($oFacetsEmbededWin) Then
    ConsoleWrite("Found Object $oFacetsEmbededWin" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oFacetsEmbededWin" & @CRLF)
    Exit
EndIf
_UIA_Action($oFacetsEmbededWin,"setfocus")


Local $oPane = _UIA_getObjectByFindAll($oFacetsEmbededWin, "Title:=Billing 360;controltype:=UIA_PaneControlTypeId;class:=Afx", $treescope_children)
If IsObj($oPane) Then
    ConsoleWrite("Found Object $oPane" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oPane" & @CRLF)
    Exit
EndIf
_UIA_Action($oPane,"setfocus")


;There are multiples of the next item, looking for the one who's Height > 100; $UIA_BoundingRectanglePropertyId: l=289,t=72,w=1596,h=824
Local $pElements, $iElements
$oPane.FindAll($treescope_children, $UIA_oTRUECondition, $pElements)                            ;Find all objects under the $oChromePane
$oPaneElements = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)    ;Convert the objects into an array to loop through

;If there are NO children found then there is nothing to search
If _UIA_IsElement($oPaneElements) Then
    $oPaneElements.Length($iElements)       ;identify the number of elements found that are stored in the array
EndIf
If $iElements < 1 Then
    ConsoleWrite("Did not find any child items under the Chrome_RenderWidgetHostHWND" & @CRLF)
    Exit
EndIf

;Loop through each item looking for the Height
Local $oPaneAfx
For $i = 0 To $iElements - 1    ;zero based
    $oPaneElements.GetElement($i, $UIA_pUIElement)
    $oPaneAfx = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    ;$tmpStr = _UIA_getBasePropertyInfo($itUIA_oUIElement)
    ;ConsoleWrite("Child Element " & $i & @CRLF & $tmpStr & @CRLF & @CRLF)
    $iIUIpropID = _UIA_getPropertyValue($oPaneAfx, $UIA_AutomationIdPropertyId) ;just used for consolewrite below
    $sRect = _UIA_getPropertyValue($oPaneAfx, $UIA_BoundingRectanglePropertyId) ;should return something like l=289,t=72,w=1596,h=824
    If $sRect <> "" Then
        ConsoleWrite("Looking at AutomationIdPropertyId " & $iIUIpropID & ", bounding rectangle (" & $sRect & ")" & @CRLF)
        $aRect = StringSplit($sRect, ";")                                           ;Split by ; as we are only interested in the last item (Height)
        If $aRect[UBound($aRect) - 1] >= 100 Then                                   ;If Height is > 100 then it's the one we want
            ConsoleWrite("Found correct Object $oPaneAfx" & @CRLF)
            ExitLoop
        EndIf
    EndIf
Next


If IsObj($oPaneAfx) Then
    ;Find the Pane that contains the Embedded Browser Window
    Local $oPaneCefBrowser = _UIA_getObjectByFindAll($oPaneAfx, "controltype:=UIA_PaneControlTypeId;class:=CefBrowserWindow", $treescope_children)
    If IsObj($oPaneCefBrowser) Then
        ConsoleWrite("Found Object $oPaneCefBrowser" & @CRLF)
    Else
        ConsoleWrite("Failed to find Object $oPaneCefBrowser" & @CRLF)
        Exit
    EndIf
    _UIA_Action($oPaneCefBrowser,"setfocus")
Else
    ConsoleWrite("Failed to find the Afx Pane with > 100 height" & @CRLF)
    Exit
EndIf

;Find the Embedded Browser Window
Local $oChromeWin = _UIA_getObjectByFindAll($oPaneCefBrowser, "controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin", $treescope_children)
If IsObj($oChromeWin) Then
    ConsoleWrite("Found Object $oChromeWin" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oChromeWin" & @CRLF)
    Exit
EndIf
_UIA_Action($oChromeWin,"setfocus")


;Now to find the actual browser chrome screen within the window.
;  When you JUST open Billing 360;  controltype:=UIA_PaneControlTypeId  If you select a dropdown or anything;  controltype:=UIA_DocumentControlTypeId
;Local $oChromePane = _UIA_getObjectByFindAll($oChromeWin, "Title:=Chrome Legacy Window;controltype:=UIA_PaneControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)
Local $oChromePane = _UIA_getObjectByFindAll($oChromeWin, "controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)
If IsObj($oChromePane) Then
    ConsoleWrite("Found Object $oChromePane" & @CRLF)
Else
    ConsoleWrite("Failed to find Object $oChromePane" & @CRLF)
    Exit
EndIf
_UIA_Action($oChromePane,"setfocus")



;Now the hard part.  Try and find all items within the chrome widget that the spy's can't see.

Local $pElements, $iElements
$oChromePane.FindAll($treescope_children, $UIA_oTRUECondition, $pElements)                          ;Find all objects under the $oChromePane and populate the variables with them
$oChromeElements = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)  ;Convert the objects into an array to loop through

;If there are NO children found then there is nothing to search
If _UIA_IsElement($oChromeElements) Then
    $oChromeElements.Length($iElements)     ;identify the number of elements found that are stored in the array
EndIf

If $iElements < 1 Then
    ConsoleWrite("Did not find any child items under the Chrome_RenderWidgetHostHWND" & @CRLF)
    Exit
EndIf

ConsoleWrite("Total Child Elements found (" & $iElements & ")" & @CRLF)

;Loop through each item and write out the info of it.
For $i = 0 To $iElements - 1    ;zero based
    $oChromeElements.GetElement($i, $UIA_pUIElement)
    $itUIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    $tmpStr = _UIA_getBasePropertyInfo($itUIA_oUIElement)
    ConsoleWrite("Child Element " & $i & @CRLF & $tmpStr & @CRLF & @CRLF)
Next

Anybody have some other ideas on what I could try to allow interaction with this new module?

Thanks

[Nine]

Seems to me that you got an embedded Chrome in there.  Like for IE, right-click Inspect does not work.  But there was the possibility of getting the whole source.  Can you check if you got something similar when you right-click it ?

[BigDaddyO]

Sadly Right Click does nothing

[junkew]

I think in the simplespy is something in comment how to do below. So in an au3 program You start before the program with embedded chromium.

How Chrome detects the presence of Assistive Technology

For performance reasons Chromium waits until it detects the presence of assistive technology before enabling full support for accessibility APIs. 

 

Windows: Chrome calls NotifyWinEvent with EVENT_SYSTEM_ALERT and the custom object id of 1. If it subsequently receives a WM_GETOBJECT call for that custom object id, it assumes that assistive technology is running.

[junkew]

Found this back

 

[BigDaddyO]

Sweet!  Thanks Junkew!

I can now see the individual controls in that module.