Jump to content

Using UI Automation Code in AutoIt


Recommended Posts

This is fantastic, great work! Any chance I could get some help w/how to locate a parent element?  Maybe just a small example? I've tried to work out the syntax for a couple of days and still cannot figure it out. I'm able to locate the element by name but that shows as the $UIA_NamePropertyId (Text) and there are 3 siblings for: the Button (to expand/collapse), Image (the folder), and "Edit: TextBox". What I need to be able to do is select the parent element so that its properties show up on the screen.

Any help would be GREATLY appreciated! This is the last piece of the puzzle before I can automate the entire project. Thanks in advance!!!

I forgot to mention I working w/a tree, if that matters.

Edited by mikester
forgot to mention it was a tree element
Link to post
Share on other sites
  • Replies 137
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Automating Notepad This example is a response to a request by mLipok for a simple Notepad automation. Task: Create a simple script to fill up "Edit1" with HelloWorld, click "Save As...", enter

This example is about using Microsoft UI Automation APIs in AutoIt. Ie. it's about using interfaces, objects, methods, properties and constants as they are defined directly by Microsoft. In AutoI

How to topics, 1 - 7 How to topics is a guide to use UIASpy to identify windows and controls and to create sample code to obtain information and perform actions. Topics 1 - 7: Introductory top

Posted Images

If it helps, here's the latest section of code I tried, although I think I was closer prior to trying this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )
$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

 

Edited by mikester
Link to post
Share on other sites

I worked it out. I looked at ex1_Treewalker.au3 from junkew a little closer and that got me to this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )

$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

Hope I didn't waste anyone's time.

Link to post
Share on other sites
  • 1 month later...
  • 2 weeks later...
51 minutes ago, cyxstudio said:

UIA_ValuePatternId is not recognized. did i miss something or forgot to include something?

Global Const $UIA_ValuePatternId             = 10002

is declared in : UIA_Constants.au3

Musashi-C64.png

"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move."

Link to post
Share on other sites

Big big big thanks to @LarsJ for the very details examples and nice UISpy program.

I learned so much in your coding convention. 

and a big thanks to  @junkew also, you're the first one who helped to open the gate of UIAutomation world to me.

Still, have much much more to learn 😀

By the way, I want to mention a case that I faced when trying to go through the Notepad examples: The  Click Save As... menu item  failed.

I ran the example in the path

Quote

Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3

However, below is the log of SciTE

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3" /UserParams    
+>14:44:19 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.4.1.2.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3
+>14:44:20 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Notepad window ---
$pCondition OK
$oNotepad OK
--- Fill Edit element ---
$pCondition OK
$oEdit OK
$oValue OK
--- Open File menu ---
$pCondition1 OK
$pCondition2 OK
$pCondition OK
$oFile OK
$oInvoke OK
--- Click "Save As..." menu ---
$pCondition2 OK
$pCondition OK
$oSaveAs ERR
+>14:44:21 AutoIt3.exe ended.rc:0
+>14:44:21 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.319

The reason is $oInvokePattern.Invoke() seems to open the File menu, then close the menu right away. So there is no Save as... menu item found.

After a while, I noticed that File menu has ExpandCollapse Pattern Methods also

image.png.c94c4e4b9f9e291e84b0b9b3d0b880b7.png

Then, instead of using Invoke() to open File menu, I used the Expand() method, and it worked. The example worked very well!

Below is the code I used replacing Invoke() by Expand()

Local $pInvokePattern, $oInvokePattern
    $oMenuItem.GetCurrentPattern($UIA_InvokePatternId, $pInvokePattern)
    $oInvokePattern = ObjCreateInterface($pInvokePattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
    If Not IsObj($oInvokePattern) Then Return ConsoleWrite("$oInvokePattern ERR" & @CRLF)
    ConsoleWrite("$oInvokePattern OK" & @CRLF)
    ;~ $oInvokePattern.Invoke()
    Sleep(1000)
    ; --- ExpandCollapse Pattern (action) Object ---

    ConsoleWrite("--- ExpandCollapse Pattern (action) Object ---" & @CRLF)

    Local $pExpandCollapsePattern1, $oExpandCollapsePattern1
    $oMenuItem.GetCurrentPattern($UIA_ExpandCollapsePatternId, $pExpandCollapsePattern1)
    $oExpandCollapsePattern1 = ObjCreateInterface($pExpandCollapsePattern1, $sIID_IUIAutomationExpandCollapsePattern, $dtagIUIAutomationExpandCollapsePattern)
    If Not IsObj($oExpandCollapsePattern1) Then Return ConsoleWrite("$oExpandCollapsePattern1 ERR" & @CRLF)
    ConsoleWrite("$oExpandCollapsePattern1 OK" & @CRLF)
    $oExpandCollapsePattern1.Expand()

    Sleep(1000)
    ; --- Click Save As... menu item ---

 

Edited by LukeLe
update a case that Notepad automation fails
Link to post
Share on other sites

LukeLe, The reason for the error is that the Sleep(100) command after the $oInvoke.Invoke() statement that opens the File menu is too short. Use Sleep(200) or Sleep(300) and the code will work. The amount of delay needed depends on how much is going on on the PC at the moment the Invoke() method is executed. But the delay should preferably be as short as possible.

A better way is to use UI Automation code to verify that the File menu is open before the code continues following the Invoke(). But it might be too much to make of a simple example.

At least with regard to Microsoft programs, it's my experience that the Invoke() command works, if available.

But good that you found an alternative solution.


Later this fall I'll update the UI Automation examples with new code in the Windows 10 1809 version.

Link to post
Share on other sites
Link to post
Share on other sites

I need Help in it.Why does this error occur?

 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Run("Calc")
Sleep(1000)

Example()

Func Example()
    ; Cria UI Automation object
    Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
    ConsoleWrite("$oUIAutomation OK" & @CRLF)

    ; Get Desktop Element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement($pDesktop)
    $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
    ConsoleWrite("$oDesktop OK" & @CRLF)

    ;Espera Jameça Calculator
    WinWaitActive("Calculator ‎- Calculator", "",5)

    ConsoleWrite("--- Procura Calculator Window ---" & @CRLF)

    ; --- Copy element info ---

;~  ; $UIA_ClassNamePropertyId                            Button

    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Button", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pButton1, $oButton1,$sIID_IUIAutomationElement8,$dtag_IUIAutomationElement8,$oParent
    $oParent.FindFirst( 4, $pCondition0, $pButton1 )
    $oButton1 = ObjCreateInterface( $pButton1, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8 )
    If Not IsObj( $oButton1 ) Then Return ConsoleWrite( "$oButton1 ERR" & @CRLF )
    ConsoleWrite( "$oButton1 OK" & @CRLF )

EndFunc   ;==>Example

 

>"C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" /UserParams    
+>09:31:18 Starting AutoIt3Wrapper v.17.224.935.0 SciTE v.3.7.3.0   Keyboard:00010416  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE   UserDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper
>Running AU3Check (3.3.14.5)  params:-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7  from:C:\Users\x209080\Desktop\Notepad++\AutoIt3  input:C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3
+>09:31:19 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Users\x209080\Desktop\Notepad++\AutoIt3\autoit3_x64.exe "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Procura Calculator Window ---
--- Find window/control ---
$pCondition0 OK
"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR
->09:31:26 AutoIt3.exe ended.rc:1
+>09:31:26 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 9.801

 

Link to post
Share on other sites
Quote

"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR

(only at first glance)

Like the error message suggests, $oParent is not an object. Something like  $oParent = ObjCreateInterface(... is probably missing here.

Musashi-C64.png

"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move."

Link to post
Share on other sites

Starting a FindFirst-search for a UI element directly from the desktop is not a good idea if the only criterion is that it should be a button. Starting a search from the desktop will search through all application windows (usually 5 - 10 windows). And when it's a FindFirst-search for a button, the search will stop at the first button found in one of those windows. The probability that this is the right button is close to zero.

In UI Automation code you usually only use the desktop to find the application top window. And then you start all other searches from the application window or possibly a child window within the application.

Because all windows usually contain multiple buttons, you must at least combine the control type with the name of the button when performing a FindFirst-search to identify a button.

$oParent in rlvitorino's post should be the application top window. In my examples, I usually use $oWindow to designate this window.

Link to post
Share on other sites
  • 3 weeks later...

First, I want to say thank you so much for the tutorials, they are incredibly helpful!

Unfortunately, I think I threw myself in the deep end a bit. I'm trying to click a button on a window that looks like this (Import from disk - Single Mode)

image.png.91fe2d8632c2d35a436a6a9544685e0a.png

The problem is that using .Invoke and .Expand both take the default action, which opens a File Open dialog and freezes the script until the file is selected. Interestingly, UIA spy doesn't see the secondary menu until the dropdown arrow is clicked

All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.

Spoiler

My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts
UI-SimpleWrappers UDF - Use UI Automation more Simply-er
KeePass UDF - Automate KeePass, a password manager

Link to post
Share on other sites

Window and control automation software cannot detect a menu until it's open.

In a toolbar menu item, the dropdown arrow is usually a standalone control. Isn't it possible to identify the dropdown arrow within the toolbar menu item and then directly click the arrow with Invoke? And then you can click "Import from disk - Single Mode" in the menu with a new Invoke.

Edited by LarsJ
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...