UI Automation Question - Looking for info on Button - (Moved)

[JohnTWI]

I am trying to automate clicking a button. The Auto It window Info tool did not give me any button info. So downloaded the "Simple UIA Spy".

The results are below, but I am still not seeing button information. Thank you in advance.

.

 

Mouse position is retrieved 115-57
At least we have an element title: [panel] class: [wxWindowClassNR]

Having the following values for all properties: 
Title is: <panel>    Class   := <wxWindowClassNR>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    0;47;1920;34
*** Parent Information top down ***
1: Title is: <Thrive>    Class   := <wxWindowClass>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    -8;-8;1936;1176
"Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass""    
0: Title is: <panel>    Class   := <wxWindowClass>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    0;47;1920;1113
"Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass""    

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass")    ;Thrive
_UIA_setVar("oP2","Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass")    ;panel

_UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

_UIA_action("panel.mainwindow","setfocus")

;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR")
_UIA_action("panel.mainwindow","setfocus")

*** Detailed properties of the highlighted element ***
UIA_title:= <panel>
UIA_text:= <panel>
UIA_regexptitle:= <panel>
UIA_class:= <wxWindowClassNR>
UIA_regexpclass:= <wxWindowClassNR>
UIA_iaccessiblechildId:= <0>
UIA_id:= <-224>
UIA_handle:= <1311508>
UIA_RuntimeId:= <42;1311508>
UIA_BoundingRectangle:= <0;47;1920;34>
UIA_ProcessId:= <1728>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_Name:= <panel>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_AutomationId:= <-224>
UIA_ClassName:= <wxWindowClassNR>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <1311508>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Win32>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <panel>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048576>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:6460,hwnd:0x140314 Main:Nested [pid:1728,hwnd:0x140314 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

[Jos]

Moved to the appropriate forum.

Moderation Team

@JohnTWI,

You have managed to post ALL you threads in the wrong post so lets consider this the last one we have to move, as it can't be this hard to post it in the appropriate forum after been show this many times!!!  OK?

Jos

Edited June 22, 2021 by Jos

[junkew]

Most likely this depends on how the developers build the application 

see https://www.wxwidgets.org/docs/tutorials/accessibility/

So I think you can only fallback to 

1. OCR like tesseract

2. Hook really deep into the system to drawtext api's (but thats not easy, maintainable, .... and probably easier to find a commercial tool that can recognize wx... controls)

 

[JohnTWI]

Can anyone direct me to a good post that uses ocr to click a button?

[JohnTWI]

I installed OCR via the youtube link below:

And OCR and associated button click work in Remote Desktop mode.

 

I was able to get the OCR to work and able to click the problem button.  While running in Remote Desktop mode.

But now the mouse click below associated with the OCR does not work when running in console mode.

MouseClick($MOUSE_CLICK_LEFT, $x1, $y1, 2)

Does anyone have a better idea for mouse click in console mode?

Thank you in advance.

[junkew]

What do you mean with does not work in console mode? Did you make a basic reproducer without OCR that mouseclick is not working?

Besides the ocr an alternative could be to click percentagewise based on the height/width of the screen you could calculate by making a wrapper around mouseclick

pseudocode

func mouseclickByPercentage($btn, $xp1, $yp1, $yp, $n) 
    $arr=wingetpos("Remote desktop title")
    
    $recalculatedX= round($xp1 * $arr[2])
    $recalculatedY= round($xy1 * $arr[3])
    ;mousemove($recalculatedx, $recalculatedy)
    mouseclick($btn, $recalculatedx, $recalculatedy, $n)

endfunc

mouseclickbypercentage($mouse_click_left, 0.5, 0.5,2) should then click somewhere in the middle of your screen

[JohnTWI]

Responding to junkew question regarding "console mode". Maintaining an active desktop after RDP disconnect, is what I meant by console mode (info below on how I am running console mode). I was able to make OCR work with an active RDP session. I will review your pseudocode example, Thank you for the feedback.