Jump to content
LarsJ

Using UI Automation Code in AutoIt

Recommended Posts

This is fantastic, great work! Any chance I could get some help w/how to locate a parent element?  Maybe just a small example? I've tried to work out the syntax for a couple of days and still cannot figure it out. I'm able to locate the element by name but that shows as the $UIA_NamePropertyId (Text) and there are 3 siblings for: the Button (to expand/collapse), Image (the folder), and "Edit: TextBox". What I need to be able to do is select the parent element so that its properties show up on the screen.

Any help would be GREATLY appreciated! This is the last piece of the puzzle before I can automate the entire project. Thanks in advance!!!

I forgot to mention I working w/a tree, if that matters.

Edited by mikester
forgot to mention it was a tree element

Share this post


Link to post
Share on other sites

If it helps, here's the latest section of code I tried, although I think I was closer prior to trying this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )
$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

 

Edited by mikester

Share this post


Link to post
Share on other sites

I worked it out. I looked at ex1_Treewalker.au3 from junkew a little closer and that got me to this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )

$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

Hope I didn't waste anyone's time.

Share this post


Link to post
Share on other sites

Thanks junkew. My problem was that there is nothing unique about the parent, grandparent, or great grandparent. They all have the same properties so I needed a way to just get the parent directly.

Share this post


Link to post
Share on other sites

"C:\Users\hssoyr9\Documents\Outlook Files\Outlook\SeachHistory.au3" (64) : ==> Variable used without being declared.:
$oEdit1.GetCurrentPattern( $UIA_ValuePatternId, $pValue )

 

 

UIA_ValuePatternId is not recognized. did i miss something or forgot to include something?

Share this post


Link to post
Share on other sites

you forgot to include your script and all the details required for proper help, such as, what application you are automating, etc..


My resources are limited. You must ask the right questions

 

Share this post


Link to post
Share on other sites
51 minutes ago, cyxstudio said:

UIA_ValuePatternId is not recognized. did i miss something or forgot to include something?

Global Const $UIA_ValuePatternId             = 10002

is declared in : UIA_Constants.au3


Musashi-C64.png

"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move."

Share this post


Link to post
Share on other sites

Big big big thanks to @LarsJ for the very details examples and nice UISpy program.

I learned so much in your coding convention. 

and a big thanks to  @junkew also, you're the first one who helped to open the gate of UIAutomation world to me.

Still, have much much more to learn 😀

By the way, I want to mention a case that I faced when trying to go through the Notepad examples: The  Click Save As... menu item  failed.

I ran the example in the path

Quote

Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3

However, below is the log of SciTE

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3" /UserParams    
+>14:44:19 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.4.1.2.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3
+>14:44:20 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Notepad window ---
$pCondition OK
$oNotepad OK
--- Fill Edit element ---
$pCondition OK
$oEdit OK
$oValue OK
--- Open File menu ---
$pCondition1 OK
$pCondition2 OK
$pCondition OK
$oFile OK
$oInvoke OK
--- Click "Save As..." menu ---
$pCondition2 OK
$pCondition OK
$oSaveAs ERR
+>14:44:21 AutoIt3.exe ended.rc:0
+>14:44:21 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.319

The reason is $oInvokePattern.Invoke() seems to open the File menu, then close the menu right away. So there is no Save as... menu item found.

After a while, I noticed that File menu has ExpandCollapse Pattern Methods also

image.png.c94c4e4b9f9e291e84b0b9b3d0b880b7.png

Then, instead of using Invoke() to open File menu, I used the Expand() method, and it worked. The example worked very well!

Below is the code I used replacing Invoke() by Expand()

Local $pInvokePattern, $oInvokePattern
    $oMenuItem.GetCurrentPattern($UIA_InvokePatternId, $pInvokePattern)
    $oInvokePattern = ObjCreateInterface($pInvokePattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
    If Not IsObj($oInvokePattern) Then Return ConsoleWrite("$oInvokePattern ERR" & @CRLF)
    ConsoleWrite("$oInvokePattern OK" & @CRLF)
    ;~ $oInvokePattern.Invoke()
    Sleep(1000)
    ; --- ExpandCollapse Pattern (action) Object ---

    ConsoleWrite("--- ExpandCollapse Pattern (action) Object ---" & @CRLF)

    Local $pExpandCollapsePattern1, $oExpandCollapsePattern1
    $oMenuItem.GetCurrentPattern($UIA_ExpandCollapsePatternId, $pExpandCollapsePattern1)
    $oExpandCollapsePattern1 = ObjCreateInterface($pExpandCollapsePattern1, $sIID_IUIAutomationExpandCollapsePattern, $dtagIUIAutomationExpandCollapsePattern)
    If Not IsObj($oExpandCollapsePattern1) Then Return ConsoleWrite("$oExpandCollapsePattern1 ERR" & @CRLF)
    ConsoleWrite("$oExpandCollapsePattern1 OK" & @CRLF)
    $oExpandCollapsePattern1.Expand()

    Sleep(1000)
    ; --- Click Save As... menu item ---

 

Edited by LukeLe
update a case that Notepad automation fails

Share this post


Link to post
Share on other sites

Yes it can really be frustrating to find out if you have to use invoke click expand patterns. I frequently just identify the element and based on xyhw just send a mousemove and mouseclick instead of using the pattern methods.

Share this post


Link to post
Share on other sites

LukeLe, The reason for the error is that the Sleep(100) command after the $oInvoke.Invoke() statement that opens the File menu is too short. Use Sleep(200) or Sleep(300) and the code will work. The amount of delay needed depends on how much is going on on the PC at the moment the Invoke() method is executed. But the delay should preferably be as short as possible.

A better way is to use UI Automation code to verify that the File menu is open before the code continues following the Invoke(). But it might be too much to make of a simple example.

At least with regard to Microsoft programs, it's my experience that the Invoke() command works, if available.

But good that you found an alternative solution.


Later this fall I'll update the UI Automation examples with new code in the Windows 10 1809 version.

Share this post


Link to post
Share on other sites

Share this post


Link to post
Share on other sites

I need Help in it.Why does this error occur?

 

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Run("Calc")
Sleep(1000)

Example()

Func Example()
    ; Cria UI Automation object
    Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
    ConsoleWrite("$oUIAutomation OK" & @CRLF)

    ; Get Desktop Element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement($pDesktop)
    $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
    ConsoleWrite("$oDesktop OK" & @CRLF)

    ;Espera Jameça Calculator
    WinWaitActive("Calculator ‎- Calculator", "",5)

    ConsoleWrite("--- Procura Calculator Window ---" & @CRLF)

    ; --- Copy element info ---

;~  ; $UIA_ClassNamePropertyId                            Button

    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Button", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pButton1, $oButton1,$sIID_IUIAutomationElement8,$dtag_IUIAutomationElement8,$oParent
    $oParent.FindFirst( 4, $pCondition0, $pButton1 )
    $oButton1 = ObjCreateInterface( $pButton1, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8 )
    If Not IsObj( $oButton1 ) Then Return ConsoleWrite( "$oButton1 ERR" & @CRLF )
    ConsoleWrite( "$oButton1 OK" & @CRLF )

EndFunc   ;==>Example

 

>"C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" /UserParams    
+>09:31:18 Starting AutoIt3Wrapper v.17.224.935.0 SciTE v.3.7.3.0   Keyboard:00010416  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE   UserDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper
>Running AU3Check (3.3.14.5)  params:-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7  from:C:\Users\x209080\Desktop\Notepad++\AutoIt3  input:C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3
+>09:31:19 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Users\x209080\Desktop\Notepad++\AutoIt3\autoit3_x64.exe "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Procura Calculator Window ---
--- Find window/control ---
$pCondition0 OK
"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR
->09:31:26 AutoIt3.exe ended.rc:1
+>09:31:26 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 9.801

 

Share this post


Link to post
Share on other sites
Quote

"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR

(only at first glance)

Like the error message suggests, $oParent is not an object. Something like  $oParent = ObjCreateInterface(... is probably missing here.


Musashi-C64.png

"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move."

Share this post


Link to post
Share on other sites

Starting a FindFirst-search for a UI element directly from the desktop is not a good idea if the only criterion is that it should be a button. Starting a search from the desktop will search through all application windows (usually 5 - 10 windows). And when it's a FindFirst-search for a button, the search will stop at the first button found in one of those windows. The probability that this is the right button is close to zero.

In UI Automation code you usually only use the desktop to find the application top window. And then you start all other searches from the application window or possibly a child window within the application.

Because all windows usually contain multiple buttons, you must at least combine the control type with the name of the button when performing a FindFirst-search to identify a button.

$oParent in rlvitorino's post should be the application top window. In my examples, I usually use $oWindow to designate this window.

Share this post


Link to post
Share on other sites

First, I want to say thank you so much for the tutorials, they are incredibly helpful!

Unfortunately, I think I threw myself in the deep end a bit. I'm trying to click a button on a window that looks like this (Import from disk - Single Mode)

image.png.91fe2d8632c2d35a436a6a9544685e0a.png

The problem is that using .Invoke and .Expand both take the default action, which opens a File Open dialog and freezes the script until the file is selected. Interestingly, UIA spy doesn't see the secondary menu until the dropdown arrow is clicked


All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.

Spoiler

My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts

Share this post


Link to post
Share on other sites

Window and control automation software cannot detect a menu until it's open.

In a toolbar menu item, the dropdown arrow is usually a standalone control. Isn't it possible to identify the dropdown arrow within the toolbar menu item and then directly click the arrow with Invoke? And then you can click "Import from disk - Single Mode" in the menu with a new Invoke.

Edited by LarsJ

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...