Hooking other applications custom controls?

[chibill]

In an application that I am automating (And writing an API for) I have came across the problem of being unable to hook into elements that are custom to the application and not a generic type of element. Any idea how I would hook into them?

 

For example on is an image display that shows an image.

[FrancescoDiMuro]

Hi @chibill
Have you already tried Control* functions?

Francesco

[chibill]

Yes I have. Because they are custom control classes those functions don't seem to return anything useful. Which is why I have asked this question

[junkew]

Check within FAQ 31 the different spying tools. If it are images still it helps to make the area to search smaller before using imagesearch-findbmp solutions )also desc ribed in frequently asked questions'

[chibill]

I have used most of the tools there. My problem is the object is a custom Class one that autoit doesn’t know how to deal with. I will give the class name when I get home

[Earthshine]

Use Inspect.exe

[chibill]

Where can i find that? 

[junkew]

 

 

[chibill]

In the FAQ Inspect.exe just says in the windows SDK. I don't know what SDK its talking about. And the problem is not that I can't select the image/other control.

 

The problem is once I have it I can't do anything useful with it. Because it is a custom control.

[chibill]

According to Au3Info the Control has a class of WindowsForms10.Window.8.app.0.3553390 which doesn't help much

[chibill]

Output from SimpleSky

 

Quote

 

    https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/?do=findComment&comment=1156373
At least we have an element title: [] class: [WindowsForms10.Window.8.app.0.3553390]

Having the following values for all properties: 
Title is: <>    Class   := <WindowsForms10.Window.8.app.0.3553390>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1057;306;85;32
*** Parent Information top down ***
2: Title is: < EOS DIGITAL REBEL XS>    Class   := <WindowsForms10.Window.8.app.0.3553390>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    898;116;260;641
"Title:= EOS DIGITAL REBEL XS;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390""    
1: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.3553390>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    901;255;254;161
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390""    
0: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.3553390>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    911;267;234;143
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390""    

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:= EOS DIGITAL REBEL XS;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390")    ; EOS DIGITAL REBEL XS
_UIA_setVar("oP2","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390")    ;
_UIA_setVar("oP3","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390")    ;

_UIA_setVar(".mainwindow","title:=;classname:=WindowsForms10.Window.8.app.0.3553390")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action(".mainwindow","setfocus")

;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP2=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:= EOS DIGITAL REBEL XS;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390", $treescope_children)    
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3553390", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar(".mainwindow","title:=;classname:=WindowsForms10.Window.8.app.0.3553390")
_UIA_action(".mainwindow","setfocus")

*** Detailed properties of the highlighted element ***
UIA_class:= <WindowsForms10.Window.8.app.0.3553390>
UIA_regexpclass:= <WindowsForms10.Window.8.app.0.3553390>
UIA_iaccessiblechildId:= <0>
UIA_id:= <olcIso>
UIA_handle:= <591566>
UIA_RuntimeId:= <42;591566>
UIA_BoundingRectangle:= <1057;306;85;32>
UIA_ProcessId:= <36828>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_AutomationId:= <olcIso>
UIA_ClassName:= <WindowsForms10.Window.8.app.0.3553390>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <591566>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <WinForm>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048576>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:35524,providerId:0x906CE Main:Nested [pid:36828,providerId:0x906CE Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

 

 

[junkew]

So you have bounding rectangle with x y so you can move mouse and click mouse in the rectangle.

[chibill]

clicking on the control will not let me see what it is currently set to.

[junkew]

So you let me guess what you want? Ocr, snapshot, gettext, ......

If its a textbox and you can activate it most likely you can get text with sending copy paste to clipboard

If its an image with decent readable text tesseract can help out if you just want the image you can capture it as you have the dimensions