Control Application WindowsForms10 Internet Explorer_Server .HTM

[souldjer777]

Good Morning Guru's,

I have an exe that calls a .dll which contains several .htm files.  It's bascially an htm inside of an exe frame.  If I inspect the application with the "AutoIt v3 Window Info" utility I see...

Outside EXE...

>>>> Window <<<<
Title:    User Client
Class:    WindowsForms10.Window.8.app.0.378734a

(Everything Else is N/A)

Inside the frame of the EXE...

>>>> Window <<<<
Title:    User Client
Class:    WindowsForms10.Window.8.app.0.378734a

>>>> Control <<<<
Class:    Internet Explorer_Server
Instance:    1
ClassnameNN:    Internet Explorer_Server1
Name:    
Advanced (Class):    [CLASS:Internet Explorer_Server; INSTANCE:1]

I HAD to use another application called "AccViewer.exe" or  Accessible Info Viewer to see more detail...

Class(NN): Internet Explorer_Server1

Example of some nested htm files within the "User Client.exe"...

res://CliHTMLRes.dll/Console.htm

res://CliHTMLRes.dll/Home.htm

res://CliHTMLRes.dll/Navigation.htm

Win Title:(Blank - not available)
Text:(Blank - not available)

Class(NN): Internet Explorer_Server1

Name: res://CliHTMLRes.dll/Console.htm

Description: MSAAHTML Registered Handler

I don't know how to control the application with the "embedded" htm files.  I don't want to resort to throwing key strokes to the application - that's a shot in the dark.

I've had no luck with trying to access the embedded _IEAttach - Getting 0 and 7 errors

#include <IE.au3>
#include <Array.au3>

ShellExecute ( "C:\Program Files\User Client.exe" )
MsgBox (0, "Executed", "")
WinWait ("User Client")
MsgBox (0, "Waiting", "")
WinActivate ("User Client")
MsgBox (0, "Activated", "")
$hWnd = WinGetHandle("User Client") ; got this from window info
MsgBox (0, "Info", $hWnd)
$oIE = _IEAttach ($hWnd, "embedded",1)
MsgBox (0, "IEAttach", $oIE & " " & @ERROR )
$oLinks = _IEBodyReadText($oIE)
MsgBox (0, "Read", $oLinks)

I'm still looking into it... but losing patience so I thought I'd reach out   Thanks Everyone!

PS... have already tried

$oIE = _IEAttach("User Client", "embedded")

$oIE = _IEAttach ($hWnd, "embedded",1)

$oIE = _IEAttach('[CLASS:Internet Explorer_Server; INSTANCE:1]','embedded')

Edited May 28, 2014 by souldjer777

[junkew]

See examples section on iuiautomation and check what simple spy sees

[souldjer777]

Alright, I downloaded your Example.zip and UIA_V0_4.zip from...

 

'?do=embed' frameborder='0' data-embedContent>>

 

Put the UIA files in the "Includes" directory and ran the simplespy.au3

 

Got a very long list and it's detecting the controls

 

Not sure if I should post my results here... results are insane

expandcollapse popup

Mouse position is retrieved 1276-488
At least we have an element [About][]
Having the following values for all properties: 
Title is: <About>   Class   := <>   controltype:= <UIA_EditControlTypeId>   ,<50004>    , (0000C354)    
*** Parent Information ***
Title is: <About>   Class   := <>   controltype:= <UIA_HyperlinkControlTypeId>  ,<50005>    , (0000C355)    
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=Alt+B
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=1260;479;32;15
UIA_ClassNamePropertyId :=
UIA_ClickablePointPropertyId :=
UIA_ControllerForPropertyId :=
UIA_ControlTypePropertyId :=50004
UIA_CulturePropertyId :=0
UIA_DescribedByPropertyId :=
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FlowsToPropertyId :=
UIA_FrameworkIdPropertyId :=
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemContainingGridPropertyId :=
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=False
UIA_HelpTextPropertyId :=
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=True
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=False
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=True
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_ItemStatusPropertyId :=
UIA_ItemTypePropertyId :=
UIA_LabeledByPropertyId :=
UIA_LegacyIAccessibleChildIdPropertyId :=1
UIA_LegacyIAccessibleDefaultActionPropertyId :=Jump
UIA_LegacyIAccessibleDescriptionPropertyId :=
UIA_LegacyIAccessibleHelpPropertyId :=
UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=Alt+B
UIA_LegacyIAccessibleNamePropertyId :=About
UIA_LegacyIAccessibleRolePropertyId :=42
UIA_LegacyIAccessibleSelectionPropertyId :=
UIA_LegacyIAccessibleStatePropertyId :=6291520
UIA_LegacyIAccessibleValuePropertyId :=res://CliHTMLRes.dll/
UIA_LocalizedControlTypePropertyId :=edit
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_MultipleViewSupportedViewsPropertyId :=
UIA_NamePropertyId :=About
UIA_NativeWindowHandlePropertyId :=0
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=2176
UIA_ProviderDescriptionPropertyId :=[pid:2176,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionselectionPropertyId :=
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_SelectionItemSelectionContainerPropertyId :=
UIA_TableColumnHeadersPropertyId :=
UIA_TableItemColumnHeaderItemsPropertyId :=
UIA_TableRowHeadersPropertyId :=
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_TableItemRowHeaderItemsPropertyId :=
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_ValueValuePropertyId :=res://CliHTMLRes.dll/
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

I'll look over your examples now and see what I can do! Edited May 29, 2014 by souldjer777

[souldjer777]

Okay... so I have a slight problem... I have an element in the gui that changes... in particular - it's the percent of the disk that's encrypted.  Is there a wild card I can use to just test against the "Encrypting " portion ?  As you can see it's read as "Encrypting ( 2%)"

Mouse position is retrieved 735-285
At least we have an element [Encrypting ( 2%) ][]
Having the following values for all properties:
Title is: <Encrypting ( 2%) >    Class   := <>    controltype:= <UIA_EditControlTypeId>    ,<50004>    , (0000C354)  

; Condition to find text "Encrypting " UIA_EditControlTypeId
  Local $pCondition, $pCondition1, $pCondition2
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_EditControlTypeId, $pCondition1 )
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Encrypting ", $pCondition2 )
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "Encrypting condition ERR" & @CRLF )
  ConsoleWrite( "Encrypting condition OK" & @CRLF )

Thanks in advance!

Edited June 4, 2014 by souldjer777

[LarsJ]

I don't think you can use wildcards. But you can find all edit elements, and search for the proper one in a loop:

 

; Condition to find all edit elements
Local $pCondition
$oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_EditControlTypeId, $pCondition )
If Not $pCondition Then Return ConsoleWrite( "Property condition ERR" & @CRLF )
ConsoleWrite( "Property condition OK" & @CRLF )

; Find all edit elements
Local $pUIElementArray, $oUIElementArray, $iElements
$oWindow.FindAll( $TreeScope_Descendants, $pCondition, $pUIElementArray ) ; <<<< Use your own $oWindow element <<<<<<<<<<<
$oUIElementArray = ObjCreateInterface( $pUIElementArray, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
$oUIElementArray.Length( $iElements )
If Not $iElements Then Return ConsoleWrite( "Find edit elements ERR" & @CRLF )
ConsoleWrite( "Find all edit elements OK. Elements: " & $iElements & @CRLF )

; Find proper edit element
Local $pEdit, $oEdit, $sName, $oProperEdit
For $i = 0 To $iElements - 1
  $oUIElementArray.GetElement( $i, $pEdit )
  $oEdit = ObjCreateInterface( $pEdit, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oEdit.GetCurrentPropertyValue( $UIA_NamePropertyId, $sName )
  ConsoleWrite( "Name of edit element " & $i & ": " & $sName & @CRLF )
  If StringInStr( $sName, "Encrypting" ) Then
    ConsoleWrite( "This is the proper edit element" & @CRLF )
    $oProperEdit = $oEdit
    ExitLoop
  EndIf
Next

[souldjer777]

Thank you!  I tried it out and it's finding the element even though it's not anywhere on the screen - I'm sure it's something I did... I'm looking into it...  Thanks again! 

Update: I know why now - it's detecting hidden elements not even on the screen - elements that are not visible.  How do I exclude those please! 

Edited June 5, 2014 by souldjer777

[junkew]

You have to use UIA_IsOffscreenPropertyId :=False

with the wrappers you get something like this

#include "UIAWrappers.au3"

const cmyWindow:="name:=User Client"
const cmyEdit:="name:=encrypting; isoffscreen:=true"

$oWindow=_UIA_getFirstObjectOfElement($UIA_oDesktop, $cMyWindow, $treescope_children)
$oMyEdit=_UIA_getFirstObjectOfElement($oWindow, $cMyEdit, $treescope_children)

if you need more properties to be abstracted extend this array in UIAWrappers

;~ Propertynames to match to numeric values
local $UIA_propertiesSupportedArray[11][3]=[ _
    ["name",$UIA_NamePropertyId], _
    ["title",$UIA_NamePropertyId], _
    ["automationid",$UIA_AutomationIdPropertyId], _
    ["classname", $UIA_ClassNamePropertyId], _
    ["class", $UIA_ClassNamePropertyId], _
    ["iaccessiblevalue",$UIA_LegacyIAccessibleValuePropertyId], _
    ["iaccessiblechildId", $UIA_LegacyIAccessibleChildIdPropertyId], _
    ["controltype", $UIA_ControlTypePropertyId,1], _
    ["processid", $UIA_ProcessIdPropertyId], _
    ["acceleratorkey", $UIA_AcceleratorKeyPropertyId], _
    ["isoffscreen",$UIA_IsOffscreenPropertyId]  _
]

and in cuitautomation2.au3 you can find all property constants 

just below line

;module UIA_PropertyIds