Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


Recommended Posts

  • Replies 870
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Automate all windows and browser applications with one UDF function library. Based on the microsoft automation API this library high level supports Recognition of conttrols from EDGE, Chrome, FF

Quickstart  Unzip the 2 zips (UIA*.zip and examples*.zip) of post #1 Start simplespy.au3 Hover your mouse over the element you want to handle Press ctrl+w Copy / past

@TheSaint no problem, bear in mind I am working (slowly) on a new spy to integrate all different spying tools so far seen. Will have different tabs with more/les information, more different ways

Posted Images

I used simplespy to get the code for an element I want to click on Power BI Online webpage. If I have the web page open in Microsoft Edge (I'm required to use this browser) and run the code below it works just fine.

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
_UIA_setVar("oUIElement","Title:=Power BI Workload2;ControlType:=UIA_HyperlinkControlTypeId;classname:=name trimmedTextWithEllipsis ng-star-inserted")
_UIA_action("oUIElement","click")

If I add a step to open the browser and web page first then it doesn't work. It doesn't give an error. It just doesn't click the web page element.

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

ShellExecute("https://app.powerbi.com/groups/me/list")
Sleep("4000")
_UIA_setVar("oUIElement","Title:=Power BI Workload2;ControlType:=UIA_HyperlinkControlTypeId;classname:=name trimmedTextWithEllipsis ng-star-inserted")
_UIA_action("oUIElement","click")

I thought maybe it page didn't have focus after it opened so I added a WinActivate for the browser window but that didn't help. I noticed that when the browser is opened using ShellExecute the search box o the page has focus and I can see the cursor flashing there. I tried sending a click to the page to get the cursor out of the search box in case that was an issue but that didn't help either. 

Any ideas on what the issue may be?

Link to post
Share on other sites

First thing to do is to use some highlight action's to see if its finding anything

I frequently have this as part of debugging and simplespy is giving you those basics

  1. Find and highlight the browser
  2. Find and highlight the document area
  3. Start finding and highlighting some elements
  4. DumpThemAll function will dump whatever is on screen to a lenghty file that you can analyze

Only if above 4 work I start with clicking, screenshotting etc.

The UIAWrappers contains "smart" finding where I know from there are issues in it so to really make sure you are not hitting an issue of the wrappers you can try to use the lower level MS UIA functions. 

Take also a look at @LarsJspy application and examples in the forum to get more feeling with the underlying patters for invoke which the wrappers have hidden away.

 

 

 

Link to post
Share on other sites

@junkew I have my code working but I have an issue. I have my code loop to perform the same actions 5 times. On the second pass, when it executes the first click, I get this very vague error message:

"C:\Program Files (x86)\AutoIt3\Include\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR
>Exit code: 1    Time: 559.6

I don't know exactly which line in my code is causing the error. Do you know what this error means, or is there something I can do to turn on more detailed error info?

Link to post
Share on other sites
Posted (edited)

I tried to capture the error using If @error but that didn't work. I tried to using a unique name for the _UIA_setVar and _UIA_Action for each pass (i.e. element1 for pass one, element2 for pass two) and that worked. Is this the best way to avoid the error or is there something else I can do?

Edited by PeterlFF
Link to post
Share on other sites

Most likely you hit a bug in the wrappers. It uses an internal caching logic in an array with "RTI." prefix for Runtime Type Information 
You can clean this "cache" with below function

func _UIAResetRTIVars()
    for $i=(ubound($uia_vars, 1)-1) to 0 step -1
        if stringleft($uia_vars[$i][0],stringlen($cRTI_Prefix))=$cRTI_Prefix Then
            _ArrayDelete($UIA_Vars,$i)
        EndIf
    next
EndFunc

;~ example
_ArrayDisplay($UIA_Vars)
_UIAResetRTIVars()
_ArrayDisplay($UIA_Vars)
Link to post
Share on other sites

@junkew I have tried this on 2 other systems and I have had issues. The first system I was able to adjust the code a little and it now is working. On the third system it errors out on the first click it tries to make. I have confirmed it has the same version of windows and is using Microsoft Edge like my system. 

The error is Line 26600, Error: Subscript used on non-accessible variable. This is the first click I send so no need to clean the cache. 

The error just lists my EXE so not sure where line 26600 is. Is there anyway to get more detailed error information?

Link to post
Share on other sites

1. Get example 1 to work and understand it

2. Use simplespy on your application and get that generated code working

3. Give your details of the error: Most likely your description of the object is just incorrect or just differently nested. All this information is in the log after you run _uia_dumpthemall

4. Most likely it errors in Func _UIA_action

and you can see in log :Not an object failing action

            _UIA_LOG("Not an object failing action " & $strAction & " on " & $obj_or_string & @CRLF, $UIA_Log_Wrapper)
or it errors on 

            $t = StringSplit(_UIA_getPropertyValue($obj2ActOn, $UIA_BoundingRectanglePropertyId), ";")

Link to post
Share on other sites
  • 1 month later...
On 4/23/2022 at 3:57 AM, junkew said:

You can set textvalues directly but then you have to study how this uia model works. Its a huge model and there are many ways to get values into a textbox completely depending on its class. Why do you not want to activate a window?

I can type into text box with type and typetext: _UIA_action("comment","type","this is a test")

comment is title of the window which has a textbox inside it. But it activates the window first which I don't want.

This one shows a red border around the window but doesn't activate or type anything. _UIA_action("comment","send","this is a test")

This is the inspection of textbox: image.png.982949018168301398c91b40a0dd6ac1.png

and sometimes it is like this: image.png.7cc39796a9ca46445137c88141035451.png

I need to send a text into textbox without activating the window.

I appreciate any help.

 

Link to post
Share on other sites

For that you have to study the code within the wrapper functions.

the type action is like sending keys. 

you are looking more for settextvalue action

It really depends on the text box if you can set or retrieve values without activating it. Look for uia textpattern and valuepattern.

Many controls do not fully support these patterns and as such the wrappers fallback to just some kind of sendkeys after setting focus to the control.

 

 

Link to post
Share on other sites

flaui and this uiawrappers all make use of the microsoft technology UIA automation and boils down to uiautomationcore.dll which is the unmanaged com version dll. Not sure where flaui defines uia2/uia3 but here is where MS has documented the foundation.
https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiautocore-overview a usefull library where i based the uiawrappers udf on.

This wrapper is around what flaui calls uia2 but as said I don't know that description from MS itself. UIA3 you refer to is the .net managed dll version but both give also access to iaccessible.iaccessible2 interface.

Link to post
Share on other sites
  • 2 weeks later...
Posted (edited)

Hello,

I have been using this tool for a while and I like it and now I'm running into a roadblock. I'm able to script where I can open this program, open the UIA_MenuItemControlTypeId based on my criteria. Then a popup menu opens. After that, I am not able to get any elements because it's reading Chrome_RenderWiedgetHostWHND. The pop up menu titled "ANALOG_3_1" has multiple buttons and I'd like to click on the one that says "Detail" (when my mouse hovers to this button it says Detail".  But there's no way of determining what the name actually is. So I just hit CTRL+W on this button below is what I have. I also tried to do something with the UIA_BoundingRectangle properties but no luck. Any suggestions?

UIA_BoundingRectangle:= <1012;541;18;18>

   

 

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=DeltaV Live;controltype:=UIA_WindowControlTypeId;class:=Window")  ;Inhouse software
_UIA_setVar("oP2","Title:=ANALOG_3_1;controltype:=UIA_WindowControlTypeId;class:=Window")   ;ANALOG_3_1
_UIA_setVar("oP3","Title:=Display Viewer Shell;controltype:=UIA_CustomControlTypeId;class:=ShellView")  ;Display Viewer Shell
_UIA_setVar("oP4","Title:=Graphic Host;controltype:=UIA_CustomControlTypeId;class:=GraphicHostView")    ;Graphic Host
_UIA_setVar("oP5","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsFormsHost") ;
_UIA_setVar("oP6","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.34cbfc8_r9_ad1") ;
_UIA_setVar("oP7","Title:=;controltype:=UIA_PaneControlTypeId;class:=CefBrowserWindow") ;
_UIA_setVar("oP8","Title:=;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_0")   ;
_UIA_setVar("oP9","Title:=Inhouse Software;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")  ;Inhouse software
_UIA_setVar("oP10","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP11","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP12","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP13","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP14","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP15","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP16","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;
_UIA_setVar("oP17","Title:=;controltype:=UIA_GroupControlTypeId;class:=")   ;

;~ $oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_GroupControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=;controltype:=UIA_GroupControlTypeId;class:=") ;ControlType:=UIA_GroupControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
_UIA_Action("oP6","setfocus")
;~_UIA_Action("oP7","highlight")
_UIA_Action("oP7","setfocus")
;~_UIA_Action("oP8","highlight")
_UIA_Action("oP8","setfocus")
;~_UIA_Action("oP9","highlight")
_UIA_Action("oP9","setfocus")
;~_UIA_Action("oP10","highlight")
_UIA_Action("oP10","setfocus")
;~_UIA_Action("oP11","highlight")
_UIA_Action("oP11","setfocus")
;~_UIA_Action("oP12","highlight")
_UIA_Action("oP12","setfocus")
;~_UIA_Action("oP13","highlight")
_UIA_Action("oP13","setfocus")
;~_UIA_Action("oP14","highlight")
_UIA_Action("oP14","setfocus")
;~_UIA_Action("oP15","highlight")
_UIA_Action("oP15","setfocus")
;~_UIA_Action("oP16","highlight")
_UIA_Action("oP16","setfocus")
;~_UIA_Action("oP17","highlight")
_UIA_Action("oP17","setfocus")

_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP16=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=DeltaV Live;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)    
_UIA_Action($oP16,"setfocus")
Local $oP15=_UIA_getObjectByFindAll($oP16, "Title:=ANALOG_3_1;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) 
_UIA_Action($oP15,"setfocus")
Local $oP14=_UIA_getObjectByFindAll($oP15, "Title:=Display Viewer Shell;controltype:=UIA_CustomControlTypeId;class:=ShellView", $treescope_children)    
Local $oP13=_UIA_getObjectByFindAll($oP14, "Title:=Graphic Host;controltype:=UIA_CustomControlTypeId;class:=GraphicHostView", $treescope_children)  
Local $oP12=_UIA_getObjectByFindAll($oP13, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsFormsHost", $treescope_children)   
_UIA_Action($oP12,"setfocus")
Local $oP11=_UIA_getObjectByFindAll($oP12, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.34cbfc8_r9_ad1", $treescope_children)   
_UIA_Action($oP11,"setfocus")
Local $oP10=_UIA_getObjectByFindAll($oP11, "Title:=;controltype:=UIA_PaneControlTypeId;class:=CefBrowserWindow", $treescope_children)   
_UIA_Action($oP10,"setfocus")
Local $oP9=_UIA_getObjectByFindAll($oP10, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_0", $treescope_children)  
_UIA_Action($oP9,"setfocus")
Local $oP8=_UIA_getObjectByFindAll($oP9, "Title:=DeltaV Operations;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children) 
_UIA_Action($oP8,"setfocus")
Local $oP7=_UIA_getObjectByFindAll($oP8, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_GroupControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_GroupControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <1012;541;18;18>
UIA_ProcessId:= <12272>
UIA_ControlType:= <50026>
UIA_LocalizedControlType:= <group>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <False>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleRole:= <20>
UIA_LegacyIAccessibleState:= <0>
UIA_LegacyIAccessibleDefaultAction:= <click ancestor>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:12272,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:UIAutomationCore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>
UIA_OptimizeForVisualContent:= <False>
UIA_IsObjectModelPatternAvailable:= <False>
UIA_AnnotationAnnotationTypeId:= <0>
UIA_IsAnnotationPatternAvailable:= <False>
UIA_IsTextPattern2Available:= <False>
UIA_StylesStyleId:= <0>
UIA_StylesFillColor:= <0>
UIA_StylesFillPatternColor:= <0>
UIA_IsStylesPatternAvailable:= <False>
UIA_IsSpreadsheetPatternAvailable:= <False>
UIA_IsSpreadsheetItemPatternAvailable:= <False>
UIA_Transform2CanZoom:= <False>
UIA_IsTransformPattern2Available:= <False>
UIA_LiveSetting:= <0>
UIA_IsTextChildPatternAvailable:= <False>
UIA_IsDragPatternAvailable:= <False>
UIA_DragIsGrabbed:= <False>
UIA_DragDropEffects:= <True>
UIA_IsDropTargetPatternAvailable:= <False>
UIA_DropTargetDropTargetEffects:= <True>
UIA_Transform2ZoomLevel:= <1>
UIA_Transform2ZoomMinimum:= <1>
UIA_Transform2ZoomMaximum:= <1>
UIA_IsTextEditPatternAvailable:= <False>
UIA_IsPeripheral:= <False>
UIA_IsCustomNavigationPatternAvailable:= <False>
UIA_PositionInSet:= <0>
UIA_SizeOfSet:= <0>
UIA_Level:= <0>
UIA_LandmarkType:= <0>

 

Edited by phat_2008
Link to post
Share on other sites
  1. For this you can use index/indexrelative:=nn
    so you first find an object with a name and then just +/- nn objects
    It can take a while to learn how to count but with the inspect.exe you should be able to identify the value for nn
    Check in the examples for index....  should become something like below (or another object thats closeby with an easy property)
    Local $oP9=_UIA_getObjectByFindAll($oP9, "Title:=DeltaV Operations;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND;indexrelative:=5", $treescope_children)
  2. Do something like above with a treewalker. See examples how to use that

Check also FAQ31 with references to tools from @LarsJ that makes life easier with spying and getting barebone uia code
 

 

Link to post
Share on other sites

Thank junkew for the reply. When Using Spy the Parent groups highlighted are two different icons/buttons but both uses the same property values. When using inspect.exe there's no object property that I can use to invoke some clicking.

 

 

image.png.6c2fed3380c266775a8a8769747b797d.png
  

 

Inspect.exe

image.png.902d461eab2397684fa0b2bcbb9a72ca.png

Link to post
Share on other sites

There are some calculated proporties that you doet see in inspect like index and indexrelative which you  can see in the wrapper source. So you can use 

Index or indexrelative and Just experiment with the value. Advice is to taken first a look in the examples to see how that works. Uia dump them all function can help in this.

Link to post
Share on other sites

Hello,

I am hopping here from AHK forums to hopefully get some advice about automating Skype, since Autoit has a much longer history with UIA :).

The thing is, Skype (and other Electron/Chromium apps) is very fussy with UIAutomation. ElementFromPoint and GetFocusedElement seems to work fine, but using FindAll with TrueCondition on the Skype window element doesn't display the whole UIA tree, and neither does recursively iterating over the Skype window elements with a TreeWalker with TrueCondition nor RawViewWalker: the content (Chromium) part is always missing. I've tried sending the WM_GETOBJECT message to Chrome_RenderWidgetHostHWND1 control, but it seems Chromium is activated, just not accessible. Setting up an event handler to send WM_GETOBJECT after a EVENT_SYSTEM_ALERT doesn't work. Also if it accessibility was turned off, then ElementFromPoint wouldn't work, so I think the problem is somewhere else. Activating the window or bringing to focus beforehand doesn't matter, the content still isn't found. 

When I run the following snippet with Skype open:

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Skype;controltype:=UIA_PaneControlTypeId", $treescope_children)
_UIA_Action($oP1,"setfocus")
_UIA_DumpThemAll($oP1, $treescope_subtree)

I get this:

<log space="preserve">
<logline level="5" timestamp="20220712-134100996">
<information> Information _UIA_VersionInfo version: T0.7-0; Release date: 20190501; OS Version: WIN_10</information>
</logline>
<logline level="5" timestamp="20220712-134100996"> _UIA_normalizeExpression Title:=Skype;controltype:=UIA_PaneControlTypeId;elements 1-2 in properties array</logline>
<logline level="5" timestamp="20220712-134100997"> _UIA_normalizeExpression property 1 Title:=Skype</logline>
<logline level="5" timestamp="20220712-134100997"> name:[Title] value:[Skype] having index 3</logline>
<logline level="5" timestamp="20220712-134100999"> _UIA_normalizeExpression property 2 controltype:=UIA_PaneControlTypeId</logline>
<logline level="5" timestamp="20220712-134101999"> name:[controltype] value:[50033] having index 15</logline>
<logline level="5" timestamp="20220712-134101000"> *** Try to get a list of elements *** treescopetype:=2</logline>
<logline level="0" timestamp="20220712-134101362"> _UIA_getObjectByFindAll walk thru the tree with n elements where n equals 11</logline>
<logline level="5" timestamp="20220712-134101398"> Found match with element: 4 the Name is: <Title is: <Skype> Class := <Chrome_WidgetWin_1> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) rect := < 10;0;620;680> hwnd := < 2427216> acceleratorkey := < > automationid := <> </logline>
<logline level="5" timestamp="20220712-134101398">
<propertymatching/>
</logline>
<logline level="5" timestamp="20220712-134101405"> Storing parent for found object in RTI as RTI.PARENT Desktop 1</logline>
<logline level="0" timestamp="20220712-134101539"> </logline>
<logline level="0" timestamp="20220712-134101541"> Quickly referenced object : <action>Action 1 setfocus on _UIA_IsElement:=1 Parameters 1:=0 2:=0 3:=0 4:=0</action></logline>
<logline level="0" timestamp="20220712-134102161">
<treedump>
<treeheader>***** Dumping tree *****</treeheader>
<elementinfo>Title is: <Skype> Class := <Chrome_WidgetWin_1> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <Intermediate D3D Window> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_TitleBarControlTypeId> ,<50037> , (0000C375) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <System> Class := <> controltype:= <UIA_MenuBarControlTypeId> ,<50010> , (0000C35A) , acceleratorkey:= <> , automationid:= <MenuBar> </elementinfo>
<elementinfo>Title is: <System> Class := <> controltype:= <UIA_MenuItemControlTypeId> ,<50011> , (0000C35B) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <Minimize> Class := <> controltype:= <UIA_ButtonControlTypeId> ,<50000> , (0000C350) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <Maximize> Class := <> controltype:= <UIA_ButtonControlTypeId> ,<50000> , (0000C350) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <Close> Class := <> controltype:= <UIA_ButtonControlTypeId> ,<50000> , (0000C350) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
</treedump>
</logline>
</log>

When I inspect the Skype window with Accessibility Insights, the tree also contains Window element with CurrentName="Skype" and Document element with CurrentName="Skype", both of which are missing from _DumpThemAll.

I've tested this on other Windows builds and the result seems to be the same. Sometimes when I kill all skype.exe processes and start Skype freshly, it works until Skype has been hidden behind another window, then it stops working and nothing gets it working again. In my main build, the FindAll/TreeWalker methods haven't worked even after killing the Skype processes. 

Perhaps user @LarsJ has ideas what might be going on? A bug in UIA?

Edited by poisonkiller
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By bobmcrae
      I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?
      #include "UIAWrappers.au3" _UIA_setVar("Global.Debug", False) _UIA_setVar("Global.Debug.File", False) _UIA_setVar("Global.Highlight", False) _UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window") _UIA_action('DPN','setFocus') _UIA_setVar("DPN.firstName","AutomationId:=txtFirstName") _UIA_setVar("DPN.lastName", "AutomationId:=txtLastName") _UIA_Action('DPN.lastName','setvalue','last name') _UIA_setVar("DPN.ID", "AutomationId:=txtSubjectId") _UIA_setVar("DPN.DOB", "AutomationId:=PART_TextBox") _UIA_setVar("DPN.Ft", "AutomationId:=txtSubjectHeight") _UIA_setVar("DPN.In", "AutomationId:=txtSubjectHeight2") _UIA_Action('DPN.firstName','setvalue','first name') _UIA_Action('DPN.ID','setvalue','ID012345') _UIA_Action('DPN.DOB','setvalue','1/31/1932') _UIA_Action('DPN.Ft','setvalue','6') _UIA_Action('DPN.In','setvalue','1') SimpleSpy output:
      ;~ *** Standard code *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children) Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children) Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) ;~ First find the object in the parent before you can do something ;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) _UIA_action($oUIElement,"click")  
    • By jackchen
      OK,I've made my "Chrome_KeepLastTab.au3" work.This script adds some features to Chrome browser:
      1. Double click on a tab to close the tab.
      2. Keep last tab:This script monitors your mouse clicks and hotkeys,if you are about to close the last tab within Chrome(click close button on the last tab, middle click/double click on the last tab or press Ctrl + w or Ctrl + {F4}), a new tab will be open and then the old tab be closed.
      #include <WindowsConstants.au3> #include <WinAPI.au3> #include "CUIAutomation2.au3" #AutoIt3Wrapper_UseX64=Y ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os ConsoleWrite("@OSArch: " & @OSArch & ", @AutoItX64: " & @AutoItX64 & @CRLF) Global $DoubleClickTime = 500 Global $UIA_oUIAutomation ; The main library core CUI automation reference Global $hMouseEvent, $hMouseHook Global $aMouseEvent[2] Global $KeepLastTab = True ; settings from ini file If $KeepLastTab Then ;The main object with acces to the windows automation api 3.0 $UIA_oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation) If IsObj($UIA_oUIAutomation) Then HotKeySet("^w", "Hotkey_Event") ; Ctrl + w to close tab HotKeySet("^{F4}", "Hotkey_Event") ; Ctrl + {F4} to close tab $DoubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")[0] OnAutoItExitRegister("UnhookMouse") ; Register mouse events callback $hMouseEvent = DllCallbackRegister("Mouse_Event", "int", "int;ptr;ptr") $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) EndIf EndIf While 1 Sleep(100) WEnd ToolTip("") ; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/ Func Mouse_Event($nCode, $wParam, $lParam) Local $info, $mouseData, $time, $timeDiff If $nCode < 0 Then Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf $tPoint = _WinAPI_GetMousePos() $hWnd = _WinAPI_WindowFromPoint($tPoint) ; if mouse is on the widget window(class: Chrome_RenderWidgetHostHWND), ; use $hWnd = _WinAPI_GetParent($hWnd) to get the parent Chrome window If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_") Then ; Chrome_WidgetWin_1: Chrome window ; Chrome_WidgetWin_2: Chrome menu ; ignore non Chrome window Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) EndIf ToolTip($hWnd & " - " & _WinAPI_GetClassName($hWnd)) ;$tagPOINT = "struct;long X;long Y;endstruct" Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo" $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam) $mouseData = DllStructGetData($info, 3) $time = DllStructGetData($info, 5) $timeDiff = $time - $aMouseEvent[1] Local $block Switch $wParam Case $WM_LBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "LClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "LDClick" Else $aMouseEvent[0] = "LClick" EndIf $block = KeepLastTab($hWnd, $aMouseEvent[0]) Case $WM_MBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "MClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "MDClick" Else $aMouseEvent[0] = "MClick" $block = KeepLastTab($hWnd, $aMouseEvent[0]) EndIf EndSwitch If Not $block Then _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf EndFunc ;==>Mouse_Event Func UnhookMouse() _WinAPI_UnhookWindowsHookEx($hMouseHook) $hMouseHook = 0 DllCallbackFree($hMouseEvent) $hMouseEvent = 0 EndFunc ;==>UnhookMouse Func Hotkey_Event() Local $block Local $hWnd = WinGetHandle(WinGetTitle("[ACTIVE]")) If StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_1") Then ; Chrome_WidgetWin_1, Chrome window $block = KeepLastTab($hWnd, "Hotkey") EndIf If Not $block Then HotKeySet("^w") Send(@HotKeyPressed) HotKeySet("^w", "Hotkey_Event") EndIf EndFunc Func KeepLastTab($hWnd, $action = "LClick") ; Possible $action value: LClick, LDClick, MClick, Hotkey ConsoleWrite(@CRLF & $action & " on a Chrome window: " & $hWnd & @CRLF) Local $aMousePos = MouseGetPos() Local $pChrome, $oChrome $UIA_oUIAutomation.ElementFromHandle($hWnd, $pChrome) ; Window $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oChrome) Then ConsoleWrite("Failed to get Chrome object from hWnd." & @CRLF) Return EndIf ;$UIA_ControlTypePropertyId = 30003 $oChromeTabs = UIA_getFirstElement($oChrome, $UIA_ControlTypePropertyId, $UIA_TabControlTypeId, $treescope_subtree) If Not IsObj($oChromeTabs) Then ConsoleWrite("Failed to get Chrome tab bar object." & @CRLF) Return EndIf Local $t $oChromeTabs.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1]) Then ; Mouse not on tabs bar ConsoleWrite("Mouse is not on the tab bar. Ignore and return..." & @CRLF) Return EndIf Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition) $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition) If Not IsObj($oCondition) Then Return $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements) $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray) If Not IsObj($oAutomationElementArray) Then ConsoleWrite("Failed to find all Chrome tabs. " & @CRLF) Return EndIf $oAutomationElementArray.Length($iLength) Local $UIA_pUIElement, $oTab2 Local $iTabs = $iLength - 1 If $iTabs > 1 Then ; more than one tab If $action = "LDClick" Then ConsoleWrite("There are " & $iTabs & " tabs within Chrome window. " & @CRLF) For $i = 1 To $iTabs $oAutomationElementArray.GetElement($i, $UIA_pUIElement) $oTab2 = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab2) Then ContinueLoop $oTab2.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then ContinueLoop If $aMousePos[0] >= $t[0] And $aMousePos[0] <= $t[2]+$t[0] And $aMousePos[1] >= $t[1] And $aMousePos[1] <= $t[3]+$t[1] Then ConsoleWrite("You double clicked on one of " & $iTabs & " tabs. Close the tab and return..." & @CRLF) HotKeySet("^w") Send("^w") HotKeySet("^w", "Hotkey_Event") Return True EndIf Next EndIf Return EndIf ConsoleWrite("There is ONLY one tab within Chrome window. " & @CRLF) $oAutomationElementArray.GetElement(1, $UIA_pUIElement) $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab) Then ConsoleWrite("Failed to get the last tab object." & @CRLF) Return EndIf Local $rtTab $oTab.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $rtTab) If UBound($rtTab) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $rtTab[0] Or $aMousePos[0] > $rtTab[2]+$rtTab[0] Or $aMousePos[1] < $rtTab[1] Or $aMousePos[1] > $rtTab[3]+$rtTab[1]) Then ; Mouse not on the last tab ConsoleWrite("Mouse is not on the last tab. Ignore and return..." & @CRLF) Return EndIf If $action = "LClick" Then $oTabClose = UIA_getFirstElement($oTab, $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $treescope_subtree) If Not IsObj($oTabClose) Then ConsoleWrite("Failed to get the last tab close object." & @CRLF) Return EndIf $oTabClose.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1] Then ; Mouse not on the tab close button Return EndIf EndIf ; open a new tab within chrome ConsoleWrite("The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing." & @CRLF) Send("^t") ConsoleWrite("A new tab created!" & @CRLF) Sleep(100) ConsoleWrite("Close the old tab and return..." & @CRLF) ;~ Local $pPattern ;~ $oTabClose.GetCurrentPattern($UIA_InvokePatternId, $pPattern) ;~ $oPattern = ObjCreateInterface($pPattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern) ;~ If IsObj($oPattern) Then ;~ ConsoleWrite("Invoke to close the tab..." & @CRLF) ;~ $oTabClose.SetFocus() ;~ $oPattern.Invoke() ;~ EndIf _WinAPI_UnhookWindowsHookEx($hMouseHook) $aMousePos = MouseGetPos() If $aMousePos[0] >= $rtTab[0] And $aMousePos[0] <= $rtTab[2]+$rtTab[0] And $aMousePos[1] >= $rtTab[1] And $aMousePos[1] <= $rtTab[3]+$rtTab[1] Then MouseClick("middle", $aMousePos[0], $aMousePos[1], 1, 0) Else MouseClick("middle", $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2, 1, 0) ; close the tab MouseMove($aMousePos[0], $aMousePos[1], 0) ; move mouse back to previous position EndIf ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEDOWN, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) ;~ Sleep(100) ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEUP, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) Return True ; to block mouse click/hot key EndFunc Func UIA_getFirstElement($obj, $propertyID, $tval, $treeScope) Local $pCondition, $oCondition $UIA_oUIAutomation.CreatePropertyCondition($propertyID, $tval, $pCondition) $oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition) Local $UIA_oUIElement, $UIA_pUIElement $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement) $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If IsObj($UIA_oUIElement) Then Return $UIA_oUIElement Return SetError(1, 0, "") EndFunc ;==>UIA_getFirstElement My question is related to #AutoIt3Wrapper_UseX64 when run this script on 64-bit Win 7.
      No mater I set #AutoIt3Wrapper_UseX64=Y or #AutoIt3Wrapper_UseX64=N, this script works very well on hotkey event, while mouse clicks sometimes works if #AutoIt3Wrapper_UseX64=N and sometimes works on Y. Can some one test this and finger out what's wrong?
      Info from SciTe if #AutoIt3Wrapper_UseX64=Y :
      @OSArch: X64, @AutoItX64: 1 Hotkey on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... Info if #AutoIt3Wrapper_UseX64=N or comment out this line:
      @OSArch: X64, @AutoItX64: 0 Hotkey on a Chrome window: 0x00140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x00140330 Failed to get Chrome tab bar object.Hotkey events and mouse click events share the same function KeepLastTab($hWnd, $action = "LClick"),Why this function triggered by hotkey works on both 32-bit and 64-bit while that triggered by mouse events failed on 32-bit autoit?
×
×
  • Create New...