Jump to content

Using UI Automation Code in AutoIt


Recommended Posts

Hi LarsJ,

Thanks for the info I have tried the change to pCondition0 and this works to get the array length back 👍

But as you mentioned it is not the most efficient way to do a "FindAll" from the main window. 

But changing to the $oTree1 doesn't work.

 

Local $pElements
  $oTree1.FindAll( $TreeScope_Descendants, $pCondition0, $pElements )
  ConsoleWrite( "$$oTree1.FindAll()" & @CRLF )

 

Scite output is 

Quote

$oUIAutomation OK
$oDesktop OK
--- Mixxx window ---
$pCondition Mixxx OK
$oMixxx OK
$pCondition0 OK
$oTree1 WLibrarySidebar OK
--- Find window/control ---
$pAndCondition1 OK
$oTreeItem1 OK
"C:\_\Apps\AutoIT3\COM OO\_IUIAutomation Native\Mixxx TEST.au3" (74) : ==> The requested action with this object has failed.:
$oTree1.FindAll( $TreeScope_Descendants, $pCondition0, $pElements )
$oTree1^ ERROR

 

This is how the IUspy looks like of the TV Control

image.thumb.png.cd2dd952965ccb517a2ce29a3f0865dc.png

Link to post
Share on other sites
  • Replies 150
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Automating Notepad This example is a response to a request by mLipok for a simple Notepad automation. Task: Create a simple script to fill up "Edit1" with HelloWorld, click "Save As...", enter

This example is about using Microsoft UI Automation APIs in AutoIt. Ie. it's about using interfaces, objects, methods, properties and constants as they are defined directly by Microsoft. In AutoI

How to topics, 1 - 7 How to topics is a guide to use UIASpy to identify windows and controls and to create sample code to obtain information and perform actions. Topics 1 - 7: Introductory top

Posted Images

It works for me with mixxx-2.3.3-win64 and Windows 7. Code executed as 64 bit code:

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n     ; If target application is running as 32 bit code
#AutoIt3Wrapper_UseX64=y      ; If target application is running as 64 bit code

Opt( "MustDeclareVars", 1 )

#include "UIA_Constants.au3"  ; Can be copied from UI Automation UDFs: https://www.autoitscript.com/forum/index.php?showtopic=201683
;#include "UIA_Functions.au3" ; Can be copied from UI Automation UDFs
;#include "UIA_SafeArray.au3" ; Can be copied from UI Automation UDFs
;#include "UIA_Variant.au3"   ; Can be copied from UI Automation UDFs

Example()

Func Example()
  ; Create UI Automation object
  Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
  ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement( $pDesktop )
  $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
  If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
  ConsoleWrite( "$oDesktop OK" & @CRLF )

  ; --- Find window/control ---

  ConsoleWrite( "--- Find window/control ---" & @CRLF )

  Local $pCondition0
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "MixxxMainWindow", $pCondition0 )
  If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
  ConsoleWrite( "$pCondition0 OK" & @CRLF )

  Local $pWindow1, $oWindow1
  $oDesktop.FindFirst( $TreeScope_Children, $pCondition0, $pWindow1 )
  $oWindow1 = ObjCreateInterface( $pWindow1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
  If Not IsObj( $oWindow1 ) Then Return ConsoleWrite( "$oWindow1 ERR" & @CRLF )
  ConsoleWrite( "$oWindow1 OK" & @CRLF )

  ; --- Find window/control ---

  ConsoleWrite( "--- Find window/control ---" & @CRLF )

  Local $pCondition1
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "WLibrarySidebar", $pCondition1 )
  If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
  ConsoleWrite( "$pCondition1 OK" & @CRLF )

  Local $pTree1, $oTree1
  $oWindow1.FindFirst( $TreeScope_Descendants, $pCondition1, $pTree1 )
  $oTree1 = ObjCreateInterface( $pTree1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
  If Not IsObj( $oTree1 ) Then Return ConsoleWrite( "$oTree1 ERR" & @CRLF )
  ConsoleWrite( "$oTree1 OK" & @CRLF )

  ; --- Find window/control ---

  ConsoleWrite( "--- Find window/control ---" & @CRLF )

  Local $pCondition2, $pCondition3, $pAndCondition3
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_TreeItemControlTypeId, $pCondition2 )
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Tracks", $pCondition3 )
  $oUIAutomation.CreateAndCondition( $pCondition2, $pCondition3, $pAndCondition3 )
  If Not $pAndCondition3 Then Return ConsoleWrite( "$pAndCondition3 ERR" & @CRLF )
  ConsoleWrite( "$pAndCondition3 OK" & @CRLF )

  Local $pTreeItem1, $oTreeItem1
  $oTree1.FindFirst( $TreeScope_Descendants, $pAndCondition3, $pTreeItem1 )
  $oTreeItem1 = ObjCreateInterface( $pTreeItem1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
  If Not IsObj( $oTreeItem1 ) Then Return ConsoleWrite( "$oTreeItem1 ERR" & @CRLF )
  ConsoleWrite( "$oTreeItem1 OK" & @CRLF )

  ; --- Find all windows/controls ---

  ConsoleWrite( "--- Find all windows/controls ---" & @CRLF )

  Local $pCondition4
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_TreeItemControlTypeId, $pCondition4 )
  If Not $pCondition4 Then Return ConsoleWrite( "$pCondition4 ERR" & @CRLF )
  ConsoleWrite( "$pCondition4 OK" & @CRLF )

  Local $pElements1
  $oTree1.FindAll( $TreeScope_Descendants, $pCondition4, $pElements1 )
  If Not $pElements1 Then Return ConsoleWrite( "$pElements1 ERR" & @CRLF )
  ConsoleWrite( "$pElements1 OK" & @CRLF )

  ; --- Code Snippets ---

  ConsoleWrite( "--- Code Snippets ---" & @CRLF )

  ; --- Create an UI Automation element array from pointer ---

  ConsoleWrite( "--- Create an UI Automation element array from pointer ---" & @CRLF )

  Local $oUIElementArray1, $iLength1 ; $pElements1 is a pointer to an UI Automation element array
  $oUIElementArray1 = ObjCreateInterFace( $pElements1, $sIID_IUIAutomationElementArray, $dtag_IUIAutomationElementArray )
  $oUIElementArray1.Length( $iLength1 )
  If Not $iLength1 Then Return ConsoleWrite( "$iLength1 = 0 ERR" & @CRLF )
  ConsoleWrite( "$iLength1 = " & $iLength1 & @CRLF )

  ; --- Traverse an UI Automation element array to get access to individual elements ---

  ConsoleWrite( "--- Traverse an UI Automation element array to get access to individual elements ---" & @CRLF )

  Local $pElement1, $oElement1, $sValue1
  For $i = 0 To $iLength1 - 1
    $oUIElementArray1.GetElement( $i, $pElement1 )
    $oElement1 = ObjCreateInterface( $pElement1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    $oElement1.GetCurrentPropertyValue( $UIA_NamePropertyId, $sValue1 )
    ConsoleWrite( "$sValue1 = " & $sValue1 & @CRLF )
  Next
EndFunc

UI Automation generally only finds visible elements. So the example only finds visible treeview items. If "Tracks" isn't a visible item, it'll cause an error in part of the code. Comment that part out.

SciTE output:

$oUIAutomation OK
$oDesktop OK
--- Find window/control ---
$pCondition0 OK
$oWindow1 OK
--- Find window/control ---
$pCondition1 OK
$oTree1 OK
--- Find window/control ---
$pAndCondition3 OK
$oTreeItem1 OK
--- Find all windows/controls ---
$pCondition4 OK
$pElements1 OK
--- Code Snippets ---
--- Create an UI Automation element array from pointer ---
$iLength1 = 17
--- Traverse an UI Automation element array to get access to individual elements ---
$sValue1 = Tracks
$sValue1 = Missing Tracks
$sValue1 = Hidden Tracks
$sValue1 = Auto DJ
$sValue1 = Crates
$sValue1 = Playlists
$sValue1 = Crates
$sValue1 = Computer
$sValue1 = Quick Links
$sValue1 = Devices
$sValue1 = Recordings
$sValue1 = History
$sValue1 = 2022-10-03
$sValue1 = Analyze
$sValue1 = iTunes
$sValue1 = Rekordbox
$sValue1 = Serato

 

Link to post
Share on other sites

Hi LarsJ,

Thanks again for the help and wonderfull tool !

I did a quick test on my side but the code example did not run as ? 

Later today I will try to debug a bit more to see where it goes wrong, and get back if I find something where it goes wrong.

 

PS :

This application seems to be created with the QT framework, definitely not native Windows controls...

But I was amazed to see that your UISpy is able to get access to these king of controls  :king:

Tried many others, but none of them seem to work, yours did 👍

Link to post
Share on other sites

Hi junkew,

Thanks for the tips, I already tried the Ranorex Spy but no success ?

The UISpy from LarsJ is just doing fine, only I can't get his code to run on my side ?

I will post later on my working version, but don't see why his version is not working on my side ??

As soon as I see some time available I will come back on this.

 

Link to post
Share on other sites
  • 2 weeks later...

Trying to figure out how to switch tabs in Chrome. For some reason calling Select() on a Chrome tab doesn't make Chrome show that tab, it still shows the previously selected tab.

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "UIA_Constants.au3"
Opt( "MustDeclareVars", 1 )
Opt("WinTitleMatchMode", 2);Match any substring in the title (case-sensitive)

Example()

Func Example()
    Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation )
    If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
    ConsoleWrite( "$oUIAutomation OK" & @CRLF )

    Local $hWindow = WinGetHandle("[TITLE:Google Chrome; CLASS:Chrome_WidgetWin_1]")
    ConsoleWrite( "$hWindow: " & $hWindow & ", " & WinGetTitle($hWindow) & @CRLF )

    Local $pElement, $oChrome
    $oUIAutomation.ElementFromHandle($hWindow, $pElement)
    $oChrome = ObjCreateInterface($pElement, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement9)
    If Not IsObj( $oChrome ) Then Return ConsoleWrite( "$oChrome ERR" & @CRLF )
    ConsoleWrite( "$oChrome OK" & @CRLF )

    Local $pCondition1, $pCondition2, $pAndCondition2
    $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_TabItemControlTypeId, $pCondition1 )
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "New Tab", $pCondition2 )
    $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pAndCondition2 )
    If Not $pAndCondition2 Then Return ConsoleWrite( "$pAndCondition2 ERR" & @CRLF )
    ConsoleWrite( "$pAndCondition2 OK" & @CRLF )

    Local $pTabItem1, $oTabItem1
    $oChrome.FindFirst( $TreeScope_Descendants, $pAndCondition2, $pTabItem1 )
    $oTabItem1 = ObjCreateInterface( $pTabItem1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oTabItem1 ) Then Return ConsoleWrite( "$oTabItem1 ERR" & @CRLF )
    ConsoleWrite( "$oTabItem1 OK" & @CRLF )


    Local $pSelectionItemPattern1, $oSelectionItemPattern1
    $oTabItem1.GetCurrentPattern( $UIA_SelectionItemPatternId, $pSelectionItemPattern1 )
    $oSelectionItemPattern1 = ObjCreateInterface( $pSelectionItemPattern1, $sIID_IUIAutomationSelectionItemPattern, $dtag_IUIAutomationSelectionItemPattern )
    If Not IsObj( $oSelectionItemPattern1 ) Then Return ConsoleWrite( "$oSelectionItemPattern1 ERR" & @CRLF )
    ConsoleWrite( "$oSelectionItemPattern1 OK" & @CRLF )

    $oSelectionItemPattern1.Select()
    ConsoleWrite( "$oSelectionItemPattern.Select()" & @CRLF )
EndFunc

 

Link to post
Share on other sites

Another question: is it possible to get Control ID of elements that don't have anything identifiable, like $UIA_ClassNamePropertyId or $UIA_NativeWindowHandlePropertyId or $UIA_AutomationIdPropertyId? AutoIt docs for Controls say:

Quote
  • ID - The internal control ID. The Control ID is the internal numeric identifier that windows gives to each control. It is generally the best method of identifying controls. In addition to the AutoIt Window Info Tool, other applications such as screen readers for the blind and Microsoft tools/APIs may allow you to get this Control ID

 

So if all controls have control ids, I should be able to somehow acquire them?

What I want to do is call AutoIt's ControlSend() on a control, for example, this pane:

736458971_Screenshot(4).thumb.png.d393d5050fc1bcb0515cd00e82a940ae.png

Link to post
Share on other sites

Do not mix uia and autoit control ids unless you understand the details on what can and what cannot be mixed.

For those hard to unique identify I made uiwrappers to use proporties like index or indexrelative.

For chrome tab Just use click. Automating things thru gui in general needs different technologies and all have their own quirks.

and frequently controlid will be automationid

Edited by junkew
Link to post
Share on other sites
6 hours ago, junkew said:

Do not mix uia and autoit control ids unless you understand the details on what can and what cannot be mixed.

For those hard to unique identify I made uiwrappers to use proporties like index or indexrelative.

and frequently controlid will be automationid

So... Are you saying that it's not possible to get Control ID of elements that don't have anything identifiable, like $UIA_ClassNamePropertyId or $UIA_NativeWindowHandlePropertyId or $UIA_AutomationIdPropertyId, like that Pane in my screenshot?

My goal here is to send a hotkey to a specific control, e.g. Ctrl+Shift+T. Can this be done without mixing UIA and AutoIt Control Ids? The only way of doing this that I know of is to use AutoIt's ControlSend(), which requires such mixing. Is there a purely UI Automation way of sending a hotkey combination to an element?

Or perhaps there is a purely Control Id way of doing this? The only reason why I'm trying to figure out the Control Id of a control using UIA is because I'm unable to get it via classical means -- I'm unable to get the Class, ClassnameNN, etc. of the control using AutoIt Window Info, as it's picking up another control that is overlayered above the one I want to select, even when toggling the "Use Spy++ control detection logic" option.

Btw, this is a separate question from the one about switching tabs in Chrome, just so that we are on the same page.

6 hours ago, junkew said:

For chrome tab Just use click.

What do you mean by a click? Do you mean a AutoIt's MoseClick() or some way of clicking via the UI Automation framework?

Doing a click via AutoIt's MouseClick() or UIA_MoseClick() from UIA UDF's UIA_Functions.au3 (it too uses AutoIt's MouseClick() inside) would not work, as the tab might be off the screen. If you have too many tabs, they overflow off-screen (off-window?) and can't be clicked on as they are not visible.

Doing a click via UI Automation's Invoke() method would not work either, as the TabItem does not implement the InvokePattern interface. It does, however, implement SelectionItemPattern interface and the currently active tab has $UIA_SelectionItemIsSelectedPropertyId set to True, while the rest of the TabItems have it set to False, but calling Select() or even AddToSelection() on a currently unselected tab doesn't seem to do anything -- Chrome still displays the tab that it has been displaying. So I wonder if I'm doing something wrong (see the code in the previous post) or perhaps Chrome simply doesn't implement TabItem's Select() to do anything meaningful like switching tabs?

6 hours ago, junkew said:

Automating things thru gui in general needs different technologies and all have their own quirks.

I specifically want to automate Chrome tab switching via the UI Automation framework. In an earlier post, Decibel has shown that it's possible to open a new tab by calling Invoke() on the New Tab button, so I thought it would be possible to interact with other UI elements too, for example tabs.

Edited by BakedCakes
Link to post
Share on other sites
  • Are you saying that it's not possible to get Control ID of elements that don't have anything identifiable
    • frequently controlid will be automationid
    • not every element has an controlid
    • In UIA you have automationid (which can be equal to controlid but not necessarily true)
  • send a hotkey to a specific control
    • This is not possible with uia as thats based on settext/setvalue by using the pattern methods
    • yes, thats based on controlsend which is based on technologies like sendkeys, keybd_event, sendinput, sendmessage (you can google these)
  • Alternative to try to select chrome tab (not tested)
    • use IAccessible interface
  • Switching tabs in Chrome
    • check example 5 and some others about chrome in the UIA wrappers UDF examples
    • If invoke() is not working then the only options are
      • keyboard send functions
      • click (as part of AutoIt)
      • rely on sendmessage(....) but then you have to find out how it works in detail thru digging in github chromium source code
    • send "ctrl+tab " key and check content of addressbar to select the right tab

sorry I do not have a clear working example as thats frequently the problem when GUI automation is needed and the logical actions are not working

Edited by junkew
Link to post
Share on other sites
On 10/16/2022 at 2:15 PM, BakedCakes said:

Or perhaps there is a purely Control Id way of doing this? The only reason why I'm trying to figure out the Control Id of a control using UIA is because I'm unable to get it via classical means -- I'm unable to get the Class, ClassnameNN, etc. of the control using AutoIt Window Info, as it's picking up another control that is overlayered above the one I want to select, even when toggling the "Use Spy++ control detection logic" option.

On 10/17/2022 at 9:08 AM, junkew said:

not every element has an controlid

I was able to find some extra controls by calling _WinAPI_EnumChildWindows() on the main window and then calling it on each of the returned controls/windows (controls like a button, a label, etc. are just windows in win32). Turns out that the UIA element tree doesn't necessity map 1:1 to controls, there might be just one control that is reporting to be an entire subtree of UIA because it draws the UI on its own. I was incorrect to assume that it's a 1:1 mapping. Oh, and calling ControlSend() on one of those extra controls did what I wanted.

On 10/17/2022 at 9:08 AM, junkew said:

use IAccessible interface

That was it, getting LegacyIAccessiblePattern of a tab and calling DoDefaultAction() on it makes Chrome switch to that tab. I thought that Legacy* methods were deprecated and likely not working, seeing how it's all grayed out in UIA Spy (while non-Legacy are in bright yellow), but no, apparently they do work.

Thank you for all of the suggestions, @junkew, my questions have been resolved :)

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...