JohnTWI Posted June 22, 2021 Share Posted June 22, 2021 I am trying to automate clicking a button. The Auto It window Info tool did not give me any button info. So downloaded the "Simple UIA Spy". The results are below, but I am still not seeing button information. Thank you in advance. . Mouse position is retrieved 115-57 At least we have an element title: [panel] class: [wxWindowClassNR] Having the following values for all properties: Title is: <panel> Class := <wxWindowClassNR> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 0;47;1920;34 *** Parent Information top down *** 1: Title is: <Thrive> Class := <wxWindowClass> controltype:= <UIA_WindowControlTypeId> ,<50032> , (0000C370) -8;-8;1936;1176 "Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass"" 0: Title is: <panel> Class := <wxWindowClass> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 0;47;1920;1113 "Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass"" ;~ *** Standard code maintainable *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) _UIA_setVar("oP1","Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass") ;Thrive _UIA_setVar("oP2","Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass") ;panel _UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR") ;~ Actions split away from logical/technical definition above can come from configfiles ;~_UIA_Action("oP1","highlight") _UIA_Action("oP1","setfocus") ;~_UIA_Action("oP2","highlight") _UIA_Action("oP2","setfocus") _UIA_action("panel.mainwindow","setfocus") ;~ *** Standard code Flexible*** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Thrive;controltype:=UIA_WindowControlTypeId;class:=wxWindowClass", $treescope_children) _UIA_Action($oP1,"setfocus") Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=panel;controltype:=UIA_PaneControlTypeId;class:=wxWindowClass", $treescope_children) _UIA_Action($oP0,"setfocus") _UIA_setVar("panel.mainwindow","title:=panel;classname:=wxWindowClassNR") _UIA_action("panel.mainwindow","setfocus") *** Detailed properties of the highlighted element *** UIA_title:= <panel> UIA_text:= <panel> UIA_regexptitle:= <panel> UIA_class:= <wxWindowClassNR> UIA_regexpclass:= <wxWindowClassNR> UIA_iaccessiblechildId:= <0> UIA_id:= <-224> UIA_handle:= <1311508> UIA_RuntimeId:= <42;1311508> UIA_BoundingRectangle:= <0;47;1920;34> UIA_ProcessId:= <1728> UIA_ControlType:= <50033> UIA_LocalizedControlType:= <pane> UIA_Name:= <panel> UIA_HasKeyboardFocus:= <False> UIA_IsKeyboardFocusable:= <True> UIA_IsEnabled:= <True> UIA_AutomationId:= <-224> UIA_ClassName:= <wxWindowClassNR> UIA_Culture:= <0> UIA_IsControlElement:= <True> UIA_IsContentElement:= <True> UIA_IsPassword:= <False> UIA_NativeWindowHandle:= <1311508> UIA_IsOffscreen:= <False> UIA_Orientation:= <0> UIA_FrameworkId:= <Win32> UIA_IsRequiredForForm:= <False> UIA_IsDockPatternAvailable:= <False> UIA_IsExpandCollapsePatternAvailable:= <False> UIA_IsGridItemPatternAvailable:= <False> UIA_IsGridPatternAvailable:= <False> UIA_IsInvokePatternAvailable:= <False> UIA_IsMultipleViewPatternAvailable:= <False> UIA_IsRangeValuePatternAvailable:= <False> UIA_IsScrollPatternAvailable:= <False> UIA_IsScrollItemPatternAvailable:= <False> UIA_IsSelectionItemPatternAvailable:= <False> UIA_IsSelectionPatternAvailable:= <False> UIA_IsTablePatternAvailable:= <False> UIA_IsTableItemPatternAvailable:= <False> UIA_IsTextPatternAvailable:= <False> UIA_IsTogglePatternAvailable:= <False> UIA_IsTransformPatternAvailable:= <False> UIA_IsValuePatternAvailable:= <False> UIA_IsWindowPatternAvailable:= <False> UIA_ValueIsReadOnly:= <True> UIA_RangeValueValue:= <0> UIA_RangeValueIsReadOnly:= <True> UIA_RangeValueMinimum:= <0> UIA_RangeValueMaximum:= <0> UIA_RangeValueLargeChange:= <0> UIA_RangeValueSmallChange:= <0> UIA_ScrollHorizontalScrollPercent:= <0> UIA_ScrollHorizontalViewSize:= <100> UIA_ScrollVerticalScrollPercent:= <0> UIA_ScrollVerticalViewSize:= <100> UIA_ScrollHorizontallyScrollable:= <False> UIA_ScrollVerticallyScrollable:= <False> UIA_SelectionCanSelectMultiple:= <False> UIA_SelectionIsSelectionRequired:= <False> UIA_GridRowCount:= <0> UIA_GridColumnCount:= <0> UIA_GridItemRow:= <0> UIA_GridItemColumn:= <0> UIA_GridItemRowSpan:= <1> UIA_GridItemColumnSpan:= <1> UIA_DockDockPosition:= <5> UIA_ExpandCollapseExpandCollapseState:= <3> UIA_MultipleViewCurrentView:= <0> UIA_WindowCanMaximize:= <False> UIA_WindowCanMinimize:= <False> UIA_WindowWindowVisualState:= <0> UIA_WindowWindowInteractionState:= <0> UIA_WindowIsModal:= <False> UIA_WindowIsTopmost:= <False> UIA_SelectionItemIsSelected:= <False> UIA_TableRowOrColumnMajor:= <2> UIA_ToggleToggleState:= <2> UIA_TransformCanMove:= <False> UIA_TransformCanResize:= <False> UIA_TransformCanRotate:= <False> UIA_IsLegacyIAccessiblePatternAvailable:= <True> UIA_LegacyIAccessibleChildId:= <0> UIA_LegacyIAccessibleName:= <panel> UIA_LegacyIAccessibleRole:= <10> UIA_LegacyIAccessibleState:= <1048576> UIA_IsDataValidForForm:= <False> UIA_ProviderDescription:= <[pid:6460,hwnd:0x140314 Main:Nested [pid:1728,hwnd:0x140314 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]> UIA_IsItemContainerPatternAvailable:= <False> UIA_IsVirtualizedItemPatternAvailable:= <False> UIA_IsSynchronizedInputPatternAvailable:= <False> Link to comment Share on other sites More sharing options...
Developers Jos Posted June 22, 2021 Developers Share Posted June 22, 2021 (edited) Moved to the appropriate forum. Moderation Team @JohnTWI, You have managed to post ALL you threads in the wrong post so lets consider this the last one we have to move, as it can't be this hard to post it in the appropriate forum after been show this many times!!! OK? Jos Edited June 22, 2021 by Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past. Link to comment Share on other sites More sharing options...
junkew Posted June 24, 2021 Share Posted June 24, 2021 Most likely this depends on how the developers build the application see https://www.wxwidgets.org/docs/tutorials/accessibility/ So I think you can only fallback to 1. OCR like tesseract 2. Hook really deep into the system to drawtext api's (but thats not easy, maintainable, .... and probably easier to find a commercial tool that can recognize wx... controls) FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets Link to comment Share on other sites More sharing options...
JohnTWI Posted June 29, 2021 Author Share Posted June 29, 2021 Can anyone direct me to a good post that uses ocr to click a button? Link to comment Share on other sites More sharing options...
JohnTWI Posted June 30, 2021 Author Share Posted June 30, 2021 I installed OCR via the youtube link below: And OCR and associated button click work in Remote Desktop mode. I was able to get the OCR to work and able to click the problem button. While running in Remote Desktop mode. But now the mouse click below associated with the OCR does not work when running in console mode. MouseClick($MOUSE_CLICK_LEFT, $x1, $y1, 2) Does anyone have a better idea for mouse click in console mode? Thank you in advance. Link to comment Share on other sites More sharing options...
junkew Posted July 7, 2021 Share Posted July 7, 2021 What do you mean with does not work in console mode? Did you make a basic reproducer without OCR that mouseclick is not working? Besides the ocr an alternative could be to click percentagewise based on the height/width of the screen you could calculate by making a wrapper around mouseclick pseudocode func mouseclickByPercentage($btn, $xp1, $yp1, $yp, $n) $arr=wingetpos("Remote desktop title") $recalculatedX= round($xp1 * $arr[2]) $recalculatedY= round($xy1 * $arr[3]) ;mousemove($recalculatedx, $recalculatedy) mouseclick($btn, $recalculatedx, $recalculatedy, $n) endfunc mouseclickbypercentage($mouse_click_left, 0.5, 0.5,2) should then click somewhere in the middle of your screen FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets Link to comment Share on other sites More sharing options...
JohnTWI Posted July 7, 2021 Author Share Posted July 7, 2021 Responding to junkew question regarding "console mode". Maintaining an active desktop after RDP disconnect, is what I meant by console mode (info below on how I am running console mode). I was able to make OCR work with an active RDP session. I will review your pseudocode example, Thank you for the feedback. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now