Jump to content

UI Automation Question - Looking for info on Button - (Moved)


Recommended Posts

I am trying to automate clicking a button. The Auto It window Info tool did not give me any button info. So downloaded the "Simple UIA Spy".

The results are below, but I am still not seeing button information. Thank you in advance.

.aButtonInfo.thumb.PNG.529e831e59ffef4a86168fd78ba07b9e.PNG

 

Mouse position is retrieved 115-57
At least we have an element title: [panel] class: [wxWindowClassNR]

Having the following values for all properties: 
Title is: <panel>    Class   := <wxWindowClassNR>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    0;47;1920;34
*** Parent Information top down ***
1: Title is: <Thrive>    Class   := <wxWindowClass>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    -8;-8;1936;1176
"Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass""    
0: Title is: <panel>    Class   := <wxWindowClass>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    0;47;1920;1113
"Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass""    


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass")    ;Thrive
_UIA_setVar("oP2","Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass")    ;panel

_UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

_UIA_action("panel.mainwindow","setfocus")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR")
_UIA_action("panel.mainwindow","setfocus")


*** Detailed properties of the highlighted element ***
UIA_title:= <panel>
UIA_text:= <panel>
UIA_regexptitle:= <panel>
UIA_class:= <wxWindowClassNR>
UIA_regexpclass:= <wxWindowClassNR>
UIA_iaccessiblechildId:= <0>
UIA_id:= <-224>
UIA_handle:= <1311508>
UIA_RuntimeId:= <42;1311508>
UIA_BoundingRectangle:= <0;47;1920;34>
UIA_ProcessId:= <1728>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_Name:= <panel>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_AutomationId:= <-224>
UIA_ClassName:= <wxWindowClassNR>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <1311508>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Win32>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <panel>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048576>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:6460,hwnd:0x140314 Main:Nested [pid:1728,hwnd:0x140314 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

Link to post
Share on other sites
  • Developers
Posted (edited)

Moved to the appropriate forum.

Moderation Team

@JohnTWI,

You have managed to post ALL you threads in the wrong post so lets consider this the last one we have to move, as it can't be this hard to post it in the appropriate forum after been show this many times!!!  OK?

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to post
Share on other sites

Most likely this depends on how the developers build the application 

see https://www.wxwidgets.org/docs/tutorials/accessibility/

So I think you can only fallback to 

1. OCR like tesseract

2. Hook really deep into the system to drawtext api's (but thats not easy, maintainable, .... and probably easier to find a commercial tool that can recognize wx... controls)

 

Link to post
Share on other sites

I installed OCR via the youtube link below:

And OCR and associated button click work in Remote Desktop mode.

 

I was able to get the OCR to work and able to click the problem button.  While running in Remote Desktop mode.

But now the mouse click below associated with the OCR does not work when running in console mode.

MouseClick($MOUSE_CLICK_LEFT, $x1, $y1, 2)

Does anyone have a better idea for mouse click in console mode?

Thank you in advance.

Link to post
Share on other sites

What do you mean with does not work in console mode? Did you make a basic reproducer without OCR that mouseclick is not working?

Besides the ocr an alternative could be to click percentagewise based on the height/width of the screen you could calculate by making a wrapper around mouseclick

pseudocode

func mouseclickByPercentage($btn, $xp1, $yp1, $yp, $n) 
    $arr=wingetpos("Remote desktop title")
    
    $recalculatedX= round($xp1 * $arr[2])
    $recalculatedY= round($xy1 * $arr[3])
    ;mousemove($recalculatedx, $recalculatedy)
    mouseclick($btn, $recalculatedx, $recalculatedy, $n)

endfunc

mouseclickbypercentage($mouse_click_left, 0.5, 0.5,2) should then click somewhere in the middle of your screen

Link to post
Share on other sites

Responding to junkew question regarding "console mode". Maintaining an active desktop after RDP disconnect, is what I meant by console mode (info below on how I am running console mode). I was able to make OCR work with an active RDP session. I will review your pseudocode example, Thank you for the feedback.

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...