IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

Automate all windows and browser applications with one UDF function library. Based on the microsoft automation API this library high level supports

• Recognition of conttrols from EDGE, Chrome, FF, Opera, Safari and Windows native apps
• Small testing framework to split object repository from coding away

Introduction

Quickstart - Getting started quickly

Simple scripts

With this module

• you can automate all applications/programs that support ui automation and/or accesibility api from microsoft
• you can recognize more controls than AutoIT can recognize "out of the box" 
• you can use concepts from other testing frameworks like

  http://download.freedesktop.org/ldtp/doc/ldtp-tutorial.pdf

  http://safsdev.sourceforge.net/Default.htm

  coded ui testing from microsoft 

Some of those controls / applications are

• chrome browser (partly mainwindow has to be done with MSAA for navigating)

  chrome://accessibility in the adress bar of chrome or start with "--force-renderer-accessibility"

• silverlight controls
• Ribbon control
• controlbars of Excel/Word
• IE and FF browsers
• Windows Media Player
• Windows clock
• AFX .. controls (partly)
• ....

•  

Based on the initial AIO Object I now have made the interface file to work with objCreateInterface function which is in the latest beta's

automate clicking and querying basic information

It gives you a lot of basic information to be able to automate clicking, querying basic information where it goes further in certain situations than AutoIt is identifying

Starting threads for background on the ui automation api of microsoft (not for starters)

• http://en.wikipedia.org/wiki/Microsoft_UI_Automation
• http://msdn.microsoft.com/en-us/library/ms747327.aspx
• Previous threads in general help/support
• Interface AutoItObject IUIAutomation
• ObjCreateInterface and struct tagPoint in method ElementFromPoint

 

• Be aware that API is not allways installed under XP/Vista see http://support.microsoft.com/kb/971513

  Within Windows 7 and Windows 8 it should be preinstalled by default. Be aware on 32 and 64 bits way of running your script

  #AutoIt3Wrapper_UseX64=Y or N

   

  Basic example of usage / showing and retrieving the default information, will post multiple examples later

  Hover your mouse to an area of interest and press ctrl+w and information will be shown in the edit box of the form

   

   

Simple spy demo (see simplespy.au3 or use latest ZIP  attachment for latest version)

 

Main features

• Recognize windows and html controls for the major browsers
• Logical and physical description for controls (UI mapping, Application map)
• Simple repository logic to abstract logical and physical descriptions
• Store Runtime Type Information in RTI. variables
• Rubberbanding/highlighting of objects
• Simple spy to help in making / identifying the physical description
• Support of regular expression(s) in identifying objects
• recognize objects on multiple properties

  supported properties: name ,title, automationid, classname, class, iaccessiblevalue, iaccessiblechildId, controltype, processid, acceleratorkey

•  The actions provided so far
   "leftclick", "left", "click", "leftdoubleclick", "leftdouble", "doubleclick", _
   "rightclick", "right", "rightdoubleclick", "rightdouble", _
   "middleclick", "middle", "middledoubleclick", "middledouble", "mousemove", "movemouse"
   "setvalue","settextvalue"
   "setvalue using keys"
   "setValue using clipboard"
   "getvalue"
   "sendkeys", "enterstring", "type", "typetext"
   "invoke"
   "focus", "setfocus", "activate"
   "close"
   "move","setposition"
   "resize"
   "minimize", "maximize", "normal", "close", "exist", "exists"
   "searchcontext", "context"
   "highlight"
   "getobject","object"
   "attach"
   "capture","screenshot", "takescreenshot"
   "dump", "dumpthemall"
   "propertyvalue", "property"

•  match on multiple properties like: 

  name:=((Zoeken.*)|(Find.*)); ControlType:=Button; acceleratorkey:=Ctrl+F

• Support for 117 different properties see $UIA_propertiesSupportedArray in uiawrappers like for example

  title, regexptitle, class, regexpclass, iaccessiblevalue, iaccessiblechildid, name, accesskey, automationid, classname 

• IAccessible, IAccessible2, ISimpleDom interfaces 
• debuglogging to a file log.txt (no output in scitewindow)

Examples

• Example 1 Iterating thru the different ways of representing the objects in the tree (#comment-1105548)
• Example 2 Finding the taskbar and clicking on the start menu button (#comment-1105680)
• Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info) (#comment-1108849)
• Example 4 that demonstrates the calculator
• Example 5 Automating chrome
• Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox
• Example 7 The chrome example modified to a firefox example
• Example 8 The other major browser Internet Explorer automated (made on Example 6 and 7)
• Example 9 Windows media player
• Example 10 Automating mach 3 (AFX windows and other hard to get recognized by AutoIT)

Lot of links are broken due to forum upgrade just search for the text like "Example 11 Demonstrate Word, Notepad and Calculator actions"

• Example 11 Demonstrate Word, Notepad and Calculator actions
• ...
• Example 13 Details 1 about the right pane of the windows explorer
• Example 14 Details 2 about the right pane of the windows explorer
• Example 15 Details 3 about the right pane of the windows explorer
• Example 16 Details 4 about the right pane of the windows explorer
• Example 17 Details 5 about the right pane of the windows explorer WITH CACHING
• Example 18 Details 6 about the right pane of the windows explorer WITH VIRTUAL ITEMS
• Example 19 Eventhandling examples
• Example 20 Eventhandling examples
• Example 21a Eventhandling examples Internet Explorer
• Example 21b Eventhandling examples Internet Explorer
• Example 22 Eventhandling examples Follow focus
• Example 23 Eventhandling examples structure changed
• Example 24 Eventhandling examples IUIAutomationEventHandler
• Example 25 SAFEARRAYS
• Example 26 IACCESSIBLE / MSAA
• Example 27 IACCESSIBLE2 / MSAA
• Example 28 IACCESSIBLE / MSAA events
• Example 29 IACCESSIBLE2 events
• Example 30 ISimpleDOM
• Example 31 Notepad window move, maximize, minimize
• Example 32 Three browsers doing the same stuff with small differences in scripting only

..

• TODO

• Build recorder
• Enhance the spy with a nicer UI
• UI for the repository (now in the script with dot notation)
• Enhance mapping / identifying on multiple properties instead of 1 combined with index
• If speed becomes an issue use the caching logic of the MS UIA framework
• Add the other patterns later
• Generalize the concept of System Under Test of starting the SUT (for testing framework purposes)
• Remote running of scripts
• Fix issue on finding within dynamic context 
• ...

edit august 18th 2013  

• initial post

Only zip files are needed to download , just unzip in 1 directory

edit july 2016

• Made V0_63 and examples 
• works with AutoIt v3.3.14
• Windows 10 tested
• Simple spy gives some basic code as a present
• Chrome latest versions seems to be having issues with IUIAutomation on tabs/buttons of mainwindow use MSAA for accessing tabsheets / buttons
• more cleanup to be in UDF style
• More comments in the source
• see changelog.txt for previous changes

edit september 2017

• All examples fixed for the IE, Firefox and Chrome browser
• Some small but essential fixes in UIAWrappers

edit april 2018 

•     Enhanced logic on fallback / dynamic search, still not perfect, to slow
•     Retested with latest Chrome, FF, Edge and IE11 and some extensions to show how to get text from the webpage (examples 5,6,7)
•     Some small bugfixes
•     Some comments as given in forum incorporated

edit may 2019

•     Speed enhancements on especially fallback searching
•     UIA.CFG works now in a better way to turn on/off debug, highlighting, debug2file
•     More stable and consistent behavior
•     Internal cleanup and refactoring of bigger functions
•     Checked with W10 (not tested on W7)
•     Added some W10 properties
•     Run with 3.3.14.5 on W10

 

UIA_V0_51.zip                 EXAMPLES_V0_5.zip     

UIA_V0_63.zip                 EXAMPLES_V0_63.zip

UIA_V0_64.zip                 EXAMPLES_V0_64.zip

 

EXAMPLES_V0_66.zip

UIA_V0_66.zip

EXAMPLES_V0_70.zip UIA_V0_70.zip

Edited May 5, 2019 by junkew
v0.70 working for IE, FireFox, Chrome, Edge

[Edano]

ok got it going on my old xp machine ..... now for more examples

[junkew]

Quickstart 

• Unzip the 2 zips (UIA*.zip and examples*.zip) of post #1
• Start simplespy.au3
• Hover your mouse over the element you want to handle
• Press ctrl+w
• Copy / paste the given source to a new AutoIT script
• If you need help
  • ask for help in new thread in general support section and post
    • screenshot that shows your element is highlighted with simplespy
    • post output of simplespy with all properties
    • attached log.txt if available in your script directory
    • use allways contents of latest zip files which could be work in progress (WIP) zip

As you will not come far if you do not understand the basics first do this

Understand AutoIT

• http://www.autoitscript.com/autoit3/docs/tutorials/helloworld/helloworld.htm
• http://www.autoitscript.com/autoit3/docs/intro/windowsadvanced.htm
• http://www.autoitscript.com/autoit3/docs/tutorials/notepad/notepad.htm

Understand testing frameworks and related concepts

• http://en.wikipedia.org/wiki/Test_automation
• http://safsdev.sourceforge.net/FRAMESDataDrivenTestAutomationFrameworks.htm
• http://en.wikipedia.org/wiki/Keyword-driven_testing
• http://ewildsch.home.xs4all.nl/testautomation.htm

 

If you understand above learn the basic way of setting focus and clicking thru UIA and the wrappers
 
Understand IUIAutomation wrappers and framework
Lets do the QuickStart steps as described above on the Start Button of Windows

• You will get this

*** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP0=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=;controltype:=Pane;class:=Shell_TrayWnd", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("Starten.mainwindow","title:=Starten;classname:=Button")
_UIA_action("Starten.mainwindow","setfocus")

This code is not yet perfect (and simplespy will improve over time) and you will have to change still manually

• _UIA_setVar is only used to get a "logical" name for a technical description
• There are many ways for UIA which varies from simple thru the wrappers to advanced based on the CUIAutomation2.au3

This is better coded

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
Local $oP0

$oP0=_UIA_action("Title:=;controltype:=Pane;class:=Shell_TrayWnd", "getobject")
_UIA_Action($oP0,"move")

_UIA_action("title:=Starten;classname:=Button;controltype:=Button","move")
_UIA_action("title:=Starten;classname:=Button;controltype:=Button","invoke")

;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","move")
;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","invoke")

Exercise

• Learn first to automate some simple stuff on notepad.exe and calc.exe by starting those applications and with ctrl+w within simplespy try some code snippets

See for latest examplecode the zip in first post

Example 1 Iterating thru the different ways of representing the objects in the tree
a. Autoit WinList function
b. UIAutomation RawViewWalker
c. UIAutomation ControlViewWalker
d. UIAutomation ContentViewWalker
e. Finding elements based on conditions (based on property id, frequently search on name or classname)
 
 

 

expandcollapse popup

;~ Example 1 Iterating thru the different ways of representing the objects in the tree
;~ a. Autoit WinList function
;~ b. UIAutomation RawViewWalker
;~ c. UIAutomation ControlViewWalker
;~ d. UIAutomation ContentViewWalker
;~ e. Finding elements based on conditions (based on property id, frequently search on name or classname)

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <MsgBoxConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

samplewinlist()

sampleTW(1)
sampleTW(2)
sampleTW(3)

ConsoleWrite("**** Desktop windows ****" & @CRLF)
findThemAll($UIA_oDesktop, $TreeScope_Children)

ConsoleWrite(@CRLF & "**** All childs of the taskbar ****")
$oTaskBar = _UIA_gettaskbar()
findThemAll($oTaskBar, $treescope_subtree)

ConsoleWrite(@CRLF & "**** Deal with browser windows ****" & @CRLF)
Local $myBrowser
;- Just get the first browser we can find
$myBrowser=getBrowser()

$tReturn=MsgBox($MB_SYSTEMMODAL + $MB_OKCANCEL , "Title", "Make sure you have 2 chrome browsers open otherwise press X button to abort.")
if $tReturn<>$IDOK  Then
    ConsoleWrite(@CRLF & "**** ABORTING chrome browser demo ****" & @CRLF)
    exit
EndIf

consolewrite(@CRLF & "**** Continue with finding 2 browser windows ****" & @CRLF)

;- Just get the first browser we can find of this type
$browserType= "Google Chrome"
;~ $browserType= "Internet Explorer"
;~ $browserType= "Opera"

$myBrowser=getBrowser($browserType,1)
if isobj($myBrowser) Then
    consolewrite("Yes found first browser " & @CRLF)
endif

ConsoleWrite(@CRLF & "**** Deal with 2nd browser window****" & @CRLF)
;- Just get the 2nd browser we can find of this type
$myBrowser=getBrowser($browserType,2)
if isobj($myBrowser) Then
    consolewrite("Yes found second browser " & @CRLF)
Else
    consolewrite("No 2nd browser found so unable to focus and highlight " & @CRLF)
    exit
endif

_UIA_Action($myBrowser,"setfocus")
_UIA_Action($myBrowser,"highlight")

Func sampleWinList()
    ConsoleWrite("sample winlist (title not empty and visible windows only)" & @CRLF)
    Local $hTimer = TimerInit()
    Local $var = WinList()

    For $i = 1 To $var[0][0]
        ; Only display visble windows
        If $var[$i][0] <> "" And IsVisible($var[$i][1]) Then
            ConsoleWrite($i & ": Title=" & $var[$i][0] & @TAB & "Handle=" & $var[$i][1] & " Class=" &  _WinAPI_GetClassName($var[$i][1]) & @CRLF)
        EndIf
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("SampleWinList took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleWinList

Func IsVisible($handle)
    If BitAND(WinGetState($handle), 2) Then
        Return 1
    Else
        Return 0
    EndIf

EndFunc   ;==>IsVisible

Func getBrowser($browserName="",$browserIndex=1)
    local $iBrowserIdx=0
;- Create treewalker
    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

;~ Get first element
    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

;~ Iterate all desktopwindows
    While IsObj($oUIElement) = True
        local $tName= _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)
        Local $tHandle= Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId))
        Local $tClassName= _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid)
;~      Browser normally put their browsername after the - (so this fails when multiple - are in title so thats why we search from right)
        local $tStr=stringmid($tname,stringinstr($tName,"-",  $STR_NOCASESENSE, -1)+2)

;- See if its a browser (based on title it could be done on class but preferred logical name instead of a classname)
;~ Private modes diffferent title to deal with only IE misbehaves
;~ InPrivate - Internet Explorer - [InPrivate]
;~ InPrivate - [InPrivate] ?- Microsoft Edge
;~ Google Chrome (Incognito)
;~ Mozilla Firefox (Private Browsing)
;~ Privé-browsen - Opera<

;~      Fix misbehave
        if $tStr="[InPrivate]" then $tStr="Internet Explorer"
        if stringinstr($tStr," (") then
            $tStr=stringleft($tstr,stringinstr($tStr," (")-1)
        EndIf
;~      consolewrite("location: " & stringinstr($tName,"-",  $STR_NOCASESENSE, -1) & "<" & $tstr & "><" & $tname & "><" &  $tclassname & "><" &   $tHandle & ">"& @CRLF)

        $itsABrowserTitle=stringinstr("Google Chrome,Internet Explorer,Mozilla Firefox,Microsoft Edge,Opera",$tStr,$STR_NOCASESENSE )
        if $itsABrowserTitle > 0 then
;~          consolewrite("** Potential match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
            if ($browserName=stringleft($tStr,stringlen($browsername))) or ($browserName="") Then
;~              consolewrite("** match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
                $iBrowserIdx=$iBrowserIdx+1
;~              consolewrite($browserIndex & $iBrowserIdx)
                if $iBrowserIDX=$browserIndex Then
                    return $oUIElement
                endif
            EndIf
        EndIf

        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    WEnd

EndFunc   ;==>sampleTW


Func sampleTW($t)
    ConsoleWrite("initializing tw " & $t & @CRLF)
    Local $hTimer = TimerInit()
    Local $i=0
;~ ' Lets show all the items of the desktop with a treewalker
    If $t = 1 Then $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    If $t = 2 Then $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)
    If $t = 3 Then $UIA_oUIAutomation.ContentViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

    While IsObj($oUIElement) = True
        ConsoleWrite($i & "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Handle=" & Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        $i=$i+1
    WEnd

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Sample tw " & $t & " took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleTW

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

edit 21 aug 2014

• Added quick start stuff

Edited May 5, 2019 by junkew

[Ascend4nt]

junkew, nice job.  I remember discussing this stuff a long while back, as I had worked on the IAccessible interface (which is also included as a 'legacy' element in IUIAutomation)

I think I got discouraged with that project because there was just too much information returned. I should really have coded it to have a filter of some sort plus a 'search' method, but laziness or disinterest set in.

I remember you had a treeview type of interface showing the IUI elements under the mouse, right?  That would have been a great start for a 'next-gen' Window Info tool.

[storme]

G'day Junkew

There is so much that the Windows API can't access and it's long overdue that UIautomation was brought into AutoIt.

I second Ascend4nt's comment about a "next-gen" Window info tool.  It'd be a great tool to explore this new way of Automating Windows.

Like

What I've always liked to see in the existing info tool is code samples to cut and paste.  In other words you select an item click a button and get a bit of code to cut and paste into your program.

I've got many ideas about it that I won't expand on here unless you're interested.

AND

I'm not just asking for something as always I'm willing to help out in any way I can if you need any help.

As you're the current expert how hard would it be to convert help file examples using standard syntax to UIautomation examples?

Thanks for the effort!

John Morrison

[junkew]

Simple scripts
The whole thread is having a history and more or less started with the complex scripts and over time more abstraction was put in UIAWrappers.au3 to make life easier. So the higher numbered examples are for starters easier to start with. This post just to give some quick reference to those examples and an advicable way to learn your way around.
 
For starters its adviceable to start just with
1. calculator and notepad example over >here
2. desktop and taskbar examples
3. combine above
 
Simple+
4. continue your steps with simple forms in IE browsers
5. continue with other browsers like chrome and firefox
6. Opera browser(s) is less open for automation
 
Complex
7. Deal with tables / grids
8. Deal with tables / grids in three different browsers 
9. testing framework with logging and configuration
10. The whole iuiautomation framework of microsoft
 
 
Further comments

• Feel free to contribute or add items on a TODO list
• My first priority is to share the basic examples and make a minimal library to make life easier for startes on this UI Automation object
• Its a powerfull model but you have to start coding to learn the basics

@Ascend4nt: First post is already having a "spy" use your mouse and click or use ctrl+m all detailed properties will show up in the edit box
In general I use inspect.exe out of the windows SDK which gives me all the details so not much priority from my side to make it in a treeview format
@storne:        I don't think it will make sense to make functions like _UIA_ControlSetFocus(.....) as UIA goes in some areas far from the logic of AutoIT so hard to map.
I will add some basic examples feel free to have certain requests or try your own
Many examples are on google for example to move an element/window: http://msdn.microsoft.com/en-us/library/ms745106.aspx

• Created UIAWrappers.AU3 and attached to first post

Example 2 Finding the taskbar and clicking on the start menu button

 

 

 

expandcollapse popup

;~ Example 2 Finding the taskbar and clicking on the start menu button
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

Consolewrite("**** All childs of the taskbar will be written to logfile see xml or txt file created in folder of example .au3 file ****" & @CRLF)
Consolewrite("** Logfile default location: " & _uia_getVar("logfilename") & @CRLF)

;~ Maintainable way for doing stuff with taskbar and startbutton by giving logical name to technical description
_UIA_setVar("oTaskbar","Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd")    ;
_UIA_setVar("oStartbutton","title:=Starten;classname:=Start")

$oTaskBar=_UIA_gettaskbar()
_UIA_DumpThemAll($otaskbar,$treescope_subtree)

local $pInvoke, $oStart
local $CaptionStartButton

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
Consolewrite("Your languagecode influences name of start button and your languagecode is: " & @oslang & @CRLF)
switch @OSLang
    case 0409
        $CaptionStartButton="start"
    case 0413
        $CaptionStartButton="Starten"
    case 1033
        $CaptionStartButton="Start"
    case Else
        MsgBox($MB_SYSTEMMODAL, "Title", "Language unknown please extend script with right caption of start button and post it in thread", 3)
EndSwitch

;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,$CaptionStartButton, $treescope_subtree)
if isobj($oStart) Then
    consolewrite("Start button found" & @CRLF)

Else
    consolewrite("I bet you did not read the messagebox. Script will stop")
    exit
EndIf

;~ Get the invoke pattern to click on the item
$oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP.invoke()

sleep(500)

;~ And doing it in a maintainable way the logical / technical definition independent of each other
_UIA_Action($UIA_oDesktop,"click",1,1)
sleep(500)
_UIA_Action("oTaskbar","setfocus")
sleep(500)
_UIA_Action("oStartButton","click")

Edited July 14, 2016 by junkew

[JohnOne]

Example works great for me.

Pretty cool stuff this junkew, and you were correct with this "I bet the text has to change to Start instead of Starten"

[junkew]

This shows a little more on the concept of

1. Find your element

2. Think what you want and find the right pattern

3. Retrieve the pattern and execute the action

 

Within the patterns you will find not likely direct support for clicking right mouse and default actions are not allways what you want so then you have to fallback to mousemove, mouseclick functions of autoit or the sendinput function from microsoft (not declared in the standard autoit library)

 

updated UIAWrappers.AU3 with some additional helper functions to get below working

 

Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info)

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

Consolewrite("**** Get the taskbar ****" & @CRLF)
$oTaskBar=_UIA_gettaskbar()
;~ Equal To
;~ _UIA_getFirstObjectOfElement($oDesktop,"classname:=Shell_TrayWnd",$TreeScope_Children)

Consolewrite("**** All childs of the taskbar ****" & @CRLF)
;~  _UIA_dumpThemAll($otaskbar,$treescope_subtree)

Dim $pInvoke ;~ Pattern invoke
Dim $oStart  ;~ Object reference to IUIElement for button start

Consolewrite("**** Different ways of assigning the startbutton to an object ****" & @CRLF)
;~ $oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"name:=Starten", $treescope_subtree)

Dim $oClock  ;~ Object reference to IUIElement for button of clock, be aware this is a panel not a button
$oClock=_UIA_getFirstObjectOfElement($oTaskbar,"classname:=TrayClockWClass", $treescope_subtree)

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)

if isobj($oStart) Then
    consolewrite("Start button found")
Else
    consolewrite("I bet the text has to change to Start instead of Starten")
EndIf

Consolewrite("Get the invoke pattern to click on the start button item. Invoke possible: " & _UIA_getPropertyValue($oUIElement, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
;~ $oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
;~ $oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP=_UIA_getpattern($oStart,$UIA_InvokePatternID)
$oInvokeP.invoke()
sleep(100)

Consolewrite("Get the menu that is after the start button" & @crlf)

$oMenuStart=_UIA_getFirstObjectOfElement($oDesktop,"Menu Start", $treescope_children)
if isobj($oMenuStart) Then
    consolewrite("Menu start found" & @crlf)
Else
    consolewrite("I bet the text has to change to Start instead of Starten" & @crlf)
EndIf

dim $oStartMenuItem
$oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"name:=SciTE Script Editor", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"Microsoft Excel 2010", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"SpotGrit", $treescope_subtree)
if isobj($oStartMenuItem) Then
    consolewrite("Scite found" & @crlf)
Else
    consolewrite("scite not found" & @crlf)
EndIf

Consolewrite("Get the pattern to click on the menu after the start button. Invoke possible: " & _UIA_getPropertyValue($oStartMenuItem, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
$oInvokeP=_UIA_getpattern($oStartMenuItem,$UIA_InvokePatternID)
;~ This would definitely fail as there is no invoke pattern
if isobj($oInvokeP) Then
    consolewrite("invoke found lets see what happens" & @crlf)
Else
    consolewrite("invoke not supported" & @crlf)
EndIf
$oInvokeP.invoke()

sleep(2000)

consolewrite("So you saw it selected but did not click" & @crlf)
;~ still you can click as you now know the dimensions where to click
dim $t
$t=stringsplit(_UIA_getPropertyValue($oStartMenuItem, $UIA_BoundingRectanglePropertyId),";")
consolewrite($t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)
;~ _winapi_mouse_event($MOUSEEVENTF_ABSOLUTE + $MOUSEEVENTF_MOVE,$t[1],$t[2])
mousemove($t[1]+($t[3]/2),$t[2]+$t[4]/2)
mouseclick("left")
sleep(2000)

Consolewrite("**** try to click on the clock (TrayClockWClass) button of the taskbar ****" & @CRLF)
if isobj($oClock) Then
    consolewrite("Clock found" & @crlf)
Else
    consolewrite("Clock not found" & @crlf)
EndIf

dim $pLegacy
;~ $oClock.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oLegacyP=_UIA_getPattern($oClock,$UIA_LegacyIAccessiblePatternId)
$oLegacyP.dodefaultaction()

dim $oClock2
$oClock2=_UIA_getFirstObjectOfElement($odesktop,"classname:=ClockFlyoutWindow",$TreeScope_Children)
_UIA_dumpthemall($oClock2,$treescope_subtree)

Exit

Edited November 5, 2013 by junkew

[junkew]

New example 4 that demonstrates on the calculator

• How to click
• How to find buttons (be aware that you have to change the captions to language of your windows)
• How to click in the menu (copy result to the clipboard)

then it uses notepad to demonstrate

• How to set a value on a textbox with the value pattern

Updated first post with latest uiawrappers.au3

 

Attention points

• Examples are build on exact match of text (so this includes tab and Ctrl+C values), later I will make some function that can find with regexp or non exact match (need treewalker for that)
• Timing / sleep is sometimes needed to give the system time to popup the menus / execute the action
• Focus of application is sometimes to be set (and sometimes not as you look on the clicking of the buttons it will even happen when there is a screen in front of the calculator)

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

run("calc.exe")
run("notepad.exe")

$oCalc=_UIA_getFirstObjectOfElement($oDesktop,"class:=CalcFrame", $treescope_children)
$oNotepad=_UIA_getFirstObjectOfElement($oDesktop,"class:=Notepad", $treescope_children)

if isobj($oCalc) Then

;~ You can comment this out just there to get the names of whats available under the calc window
_UIA_DumpThemAll($oCalc,$treescope_subtree)

    $sText="1"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Optellen"   ;Add
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="3"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Is gelijk aan"    ;Is equal to
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Bewerken"    ;Edit
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    sleep(1500)
;~  findThemAll($oCalc,$treescope_subtree)
;~  sleep(1000)

;~ Use a regular expression to identify the copy choice as there are special characters/tabs etc in the name
    $sText="Kopiëren.*"    ;Copy
    $oButton=_UIA_getObjectByFindAll($oCalc,"name:=" & $sText, $treescope_subtree)
    if isobj($oButton) Then
        consolewrite("Menuitem is there")
    Else
        consolewrite("Menuitem is NOT there")
    EndIf
    sleep(1000)

    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke
    sleep(500)

EndIf

if isobj($oNotepad) Then

$myText=clipget()

;~ You can comment this out
_UIA_dumpThemAll($oNotepad,$treescope_subtree)
    sleep(1000)

    $oNotepad.setfocus()

    $sText="Edit"
    $oEdit=_UIA_getFirstObjectOfElement($oNotepad,"classname:=" & $sText, $treescope_subtree)
    $oValueP=_UIA_getpattern($oEdit,$UIA_ValuePatternId)
    $oValueP.SetValue($myText)

EndIf

Exit

Edited November 5, 2013 by junkew

[junkew]

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ #au3check -q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ Example 5 Automating chrome
;~ This example shows how to use UI Automation with chrome (later an example will come to put stuff in the html page)
;~ 1. Have chrome started with "--force-renderer-accessibility"
;~ 2. Check with chrome://accessibility in the adress bar if accessibility is on/off, turn it on
;~ a. or close all browsers and change in script the run command to the right path of the chrome.exe
;~
;~ Apparently google changed some stuff in chrome version 29 and seems not allways to return classname properly so sometimes
;~ harder to get the right element
;- Even worse on version 42 not returning UIA controltype or classnames for certain elements
;~ Version chrome 65

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ #au3check -q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
#include <debug.au3>

#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=N  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

local $strChromeStartup="--force-renderer-accessibility"
local $strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
local $strChromeExe=$strChromeExeFolder & "chrome.exe "

if not fileexists($strChromeExe) Then
    $strChromeExeFolder=@ProgramFilesDir & "\Google\Chrome\Application\"
    $strChromeExe=$strChromeExeFolder & "chrome.exe "
EndIf

;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    if not processexists("chrome.exe") Then
        consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
        consolewrite($strChromeExeFolder & @CRLF)
        consolewrite(@ProgramFilesDir & @CRLF)
;~      C:\Program Files (x86)\Google\Chrome\Application

        consolewrite($strChromeExe & $strChromeStartup & @CRLF)
        Exit
    endif
EndIf

local $oChrome=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
$oChrome.setfocus()

;~ _UIA_DumpThemAll($oDesktop,$treescope_children)

if isobj($oChrome) Then
;~ All stuff on the page to be dumped in an xml file in log folder
_UIA_DumpThemAll($oChrome,$treescope_subtree)

;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
;~ local $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~ $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"helptext:=OmniboxViewViews", $treescope_subtree) ;works in chrome 42
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29

local $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"title:=Adres- en zoekbalk"  , $treescope_subtree) ;works in chrome 65

;~  _UIA_DumpThemAll($oChromeToolbar,$treescope_children)
local $oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)

;~  get the value of the addressbar
local $myText=""
$oValueP.CurrentValue($myText)
consolewrite("address: " & $myText & @CRLF)
sleep(500)

;~ Click all tabs, chrome hides them a little in the hierarchy but first find the tablist
local $oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
 _UIA_DumpThemall($oChromeTabs,$treescope_children)

;~  Get result with findall function alternative could be the treewalker
    local  $pCondition, $pTrueCondition
    local  $pElements, $iLength

;~  and now just find all childs which are 4 tabs and 1 new button looking like a tab
    $UIA_oUIAutomation.CreateTrueCondition($pTRUECondition)
    local $oCondition=ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition,$dtagIUIAutomationCondition)

    $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements)
    local $oAutomationElementArray = ObjCreateInterFace($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    consolewrite("We have " & $ilength & " tabs" & @CRLF)

    For $i = 0 To $iLength - 1; it's zero based
    $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        local $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        consolewrite( "Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= <" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB & @CRLF)
;~  Invoke or select them all
;~ Tricky as chrome seems to say it supports certain patterns but then they seem not to be implemented


;~ only tabs with content
if _UIA_getPropertyValue($oUIElement, $UIA_IsSelectionItemPatternAvailablePropertyId) = "True" Then
_UIA_action($oUIElement,"leftclick")
;~  $oSelectP=_UIA_getpattern($oUIElement,$UIA_SelectionItemPatternId)
;~  $oSelectP.Select()
EndIf


Next

EndIf

;~ Lets open a new tab within chrome by finding the button in the tabstrip
local $oChromeNewTab=_UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
_UIA_action($oChromeNewTab,"leftclick")

;~ Lets get the valuepattern of the addressbar
;~  $oLegacyP=_UIA_getpattern($oChromeAddressBar,$UIA_LegacyIAccessiblePatternId)
;~

_UIA_action($oChromeAddressBar,"leftclick")
_UIA_action($oChromeAddressBar,"setvalue using keys","www.autoitscript.com/forum{ENTER}")

;~ some time to load the browser page
sleep(2000)

exit

 

Edited April 11, 2018 by junkew

[trancexx]

I can't run that last example. Apparently some function definition is missing.

Is there anything special I need to do except having that script together with two includes in the same folder?

Edited August 29, 2013 by trancexx

[Jon]

Interesting.

[junkew]

@TranceXX: Just uploaded latest UIAWrappers.AU3 maybe I missed to do that. If you still get an error let me know
 

Special things that sometimes is needed (hopefully I can manage this in the UIAWrappers.AU3 later)

• Open once chrome is started the url chrome://accessibility/ to see if accessibility is on
• Chrome version 29 as Google is still changing classnames, names of properties etc. its not allways the same to find an object
  A terrible workaround is needed for html input text boxes as their name is determined by the value (seems to be specified that way by W3C) and not on the name or id attribute (only picks attribut aria-label when its available)
• Start chrome manually or remove commented run statement for starting chrome (not handling multiple instances of chrome at the moment)
• OS Language can influence stuff (I am on a dutch OS so sometimes names have to be set in english so then I sometimes switch to find by one of the other properties)
• If stuff is not working use the findThemAll(<handle to object>, <treescope>) it will write the childs with the most common properties to the output window
  Be aware on treescope_subtree as if it walks to the leafs of the hierarchy it can take minutes especially if multiple HTML windows are open

$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
findThemAll($oDesktop,$treescope_children)

• I have frequently inspect.exe open on my desktop to validate what properties are available and what values are there
• For the IAccessible2 interface (not yet as I have not converted the IDL to the autoit syntax) I use AccProbe
  http://accessibility.linuxfoundation.org/a11yweb/util/accprobe/

 

Below some background for TODO later

Chrome supports following interfaces under Windows:

• MSAA/IAccessible (complete)
• IAccessible2 (mostly complete)
• ISimpleDOM (mostly complete)
• IAccessibleEx and UI Automation (very limited)

 

@TranceXX: I should be able to get to the IAccessible2 interface by making a call to QueryInterface as Chrome supports below interfaces but I am a little less experienced in this interface/com stuff so maybe you can help out

accProbe will show what IAccessible2 can reveal out of an html page (first register the needed proxy dll)

a. IAccessible2.IDL needs to be converted to Autoit syntax

b. This should be written in AutoIT syntax
 

HWND hw = GetForegroundWindow();
    IAccessible *pIA;
    HRESULT hr = AccessibleObjectFromWindow(hw, OBJID_CLIENT, IID_IAccessible, (void**)&pIA);
    if (!SUCCEEDED(hr))
        return -1;

    const IID IID_IAccessible2 = {0xE89F726E, 0xC4F4, 0x4c19, 0xbb, 0x19, 0xb6, 0x47, 0xd7, 0xfa, 0x84, 0x78};
    ::IServiceProvider *pService = NULL; 
    hr = pIA->QueryInterface(IID_IServiceProvider, (void **)&pService); 
    if(SUCCEEDED(hr))
    {
        IAccessible2 *pIA2 = NULL; 
        hr = pService->QueryService(IID_IAccessible2, IID_IAccessible2, (void**)&pIA2); 
        if (SUCCEEDED(hr) && pIA2)
        { 
            pIA2->Release();
        } 
        pService->Release();
    }  

[trancexx]

That code written in AutoIt would be something like this (interfaces definitions missing):

Global Const $hOLEACC_DLL = DllOpen("oleacc.dll")
Global Const $OBJID_CLIENT = 0xFFFFFFFC

Func YourCodeWrittenInAutoIt($hWindow)
    Local $tIID_IAccessible = _WinAPI_GUIDFromString($sIID_IAccessible)
    Local $pIAccessible = AccessibleObjectFromWindow($hWindow, $OBJID_CLIENT, $tIID_IAccessible)
    ; Object from pointer
    Local $oIAccessible = ObjCreateInterface($pIAccessible, $sIID_IAccessible, $tagIAccessible)
    If @error Then Return -1

    Local $pIServiceProvider
    $oIAccessible.QueryInterface($sIID_IServiceProvider, $pIServiceProvider)
    ; Again object from pointer
    Local $oIServiceProvider = ObjCreateInterface($pIServiceProvider, $sIID_IServiceProvider, $tagIServiceProvider)
    If @error Then Return -2
    
    Local $pIA2
    $oIServiceProvider.QueryService($IID_IAccessible2, $IID_IAccessible2, $pIA2)
    ; And again creating object from pointer
    Local $oIA2 = ObjCreateInterface($pIA2, $sIID_IAccessible2, $tagIAccessible2)
    If @error Then Return -3
    
    ;...
EndFunc

Func AccessibleObjectFromWindow($hWindow, $iObjId, $tRefIID)
    Local $aCall = DllCall($hOLEACC_DLL, "long", "AccessibleObjectFromWindow", "hwnd", $hWindow, "dword", $iObjId, "struct*", $tRefIID, "ptr*", Null)
    If @error Or $aCall[0] < 0 Then Return SetError(1, 0, 0)
    Return $aCall[4]
EndFunc

If you look closely at that (and your) code you will see call to oIAccessible.QueryInterface(...). That probably means that you can skip one step and specify $tIID_IServiceProvider directly in earlier call to AccessibleObjectFromWindow to get $pIServiceProvider pointer directly. Edited August 29, 2013 by trancexx

[junkew]

From the uiautomation framework I can get to $oIaccessible by doing below code

The AccessibleObjectFromWindow is the MSAA way of getting the interface but thx again for the example as this will help later on getting to IAccessible2 or ISimpleDom

$oLegacyP=_UIA_getpattern($oUIElement,$UIA_LegacyIAccessiblePatternId)
    $oIAccessible = $oLegacyP.GetIAccessible()

[junkew]

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

Uodate UIAWrappers.AU3

Made a lot more comments and failure to show when an object is not retrieved/found the hierarchy of the tree is written to the console

Lots of optimizations could be done by using the cachefunctions (less interprocess communication) but on my machine it runs at a very acceptable speed

In top of script put the text in your local language of the operating system

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
 
#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
 
;~ Make this language specific
const $cToolbarByName = "name:=Google Chrome Toolbar"
const $cAddressBarByName = "name:=Adres- en zoekbalk"
const $cChromeNewTabByName="name:=Nieuw tabblad"
 
$strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
$strChromeStartup="--force-renderer-accessibility"
$strChromeExe=$strChromeExeFolder & "chrome.exe "
 
;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
    consolewrite($strChromeExe & $strChromeStartup & @CRLF)
EndIf
 
;~ Find the chrome window
$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
if not isobj($oChrome) Then
    _UIA_DumpThemAll($oDesktop,$treescope_children)
EndIf
 
;~ Make sure chrome is front window
$oChrome.setfocus()
 
if isobj($oChrome) Then
    consolewrite("Action 1" & @CRLF)
 
;~  get the chrome toolbar
;~  $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_ToolBarControlTypeId, $treescope_subtree)
    $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)
    if not isobj($oChromeToolbar) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
 
consolewrite("Action 2" & @CRLF)
;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29
    $oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
    if not isobj($oChromeAddressbar) Then
        _UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
    EndIf
 
;~  $oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)
;~  sleep(2000)
 
;~  get the value of the addressbar
;~  $myText=""
;~  $oValueP.CurrentValue($myText)
;~  consolewrite("address: " & $myText & @CRLF)
 
consolewrite("Action 3" & @CRLF)
;~ Get reference to the tabs
    $oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
    if not isobj($oChromeTabs) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
;~ Lets open a new tab within chrome
 
consolewrite("Action 4" & @CRLF)
;~  $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
    $oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
    if not isobj($oChromeNewTab) Then
        _UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
    EndIf
    _UIA_action($oChromeNewtab,"leftclick")
    sleep(500)
 
consolewrite("Action 4a" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
 
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
 
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
 
 
consolewrite("Action 4b" & @CRLF)
;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
 
sleep(500)
 
consolewrite("Action 4c retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=On", $treescope_subtree)
if not isobj($oForumLink) Then
consolewrite("*** Scripting will fail as accessibility is off ****")
MsgBox(4096, "Accessibility warning", "Accessibility is turned off, put it on by clicking on Off after Global accessibility mode", 10)
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
 
consolewrite("Action 4d" & @CRLF)
;~ $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
$oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
if not isobj($oChromeNewTab) Then
_UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
EndIf
_UIA_action($oChromeNewtab,"leftclick")
sleep(500)
consolewrite("Action 5" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName , $treescope_subtree) ;works in chrome 29
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
_UIA_action($oChromeAddressBar,"setvalue using keys","www.autoitscript.com/{ENTER}")
consolewrite("Action 6" & @CRLF) ;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
sleep(500)
consolewrite("Action 7 retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)
if not isobj($oForumLink) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oForumLink,"invoke")
sleep(3000)
consolewrite("Action 8 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
 
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf ;~ Now we get the searchfield
$oEdtSearchForum=_UIA_getObjectByFindAll($oDocument,"name:=Search...", $treescope_subtree)
if not isobj($oEdtSearchForum) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oEdtSearchForum,"focus")
_UIA_action($oEdtSearchForum,"setvalue using keys","Chrome can easy be automated with ui automation") ; {ENTER}")
sleep(500)
;~ Exit
;~ Now we press the button, see relative syntax used as the button seems not to have a name its just 2 objects further then search field
$oBtnSearch=_UIA_getObjectByFindAll($oDocument,"name:=Search...; indexrelative:=2", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
sleep(1000)
_UIA_action($oBtnSearch,"invoke")
sleep(2000)
;~ consolewrite("Action 9 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$oHyperlink=_UIA_getObjectByFindAll($oDocument,"name:=controlsend doesn't work", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
sleep(1000)
_UIA_action($oHyperlink,"invoke")
sleep(2000)
EndIf
exit
 
 

 



 

edit 1 sept 2013: added steps 4a,b,c,d to warn for accessibility is off

Edited November 17, 2013 by junkew

[ValeryVal]

Capitally!

I can't run that last example. Apparently some function definition is missing.
Is there anything special I need to do except having that script together with two includes in the same folder?

Yes! This two includes in one line is typo.

I have Example5.working nice after replacement line with $strChromeExeFolder:

$strChromeExeFolder= "Your chrome EXE folder with \"

 

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

 

Example6 has problem, though. It can't create $oChromeToolbar as object in line #48, Here:

    $oChromeToolbar =_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)

Is it concerning with my chrome language ("ru") and invalid names (Russian) of const $cToolbarByName, $cAddressBarByName, and $cChromeNewTabByName?

[junkew]

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

[ValeryVal]

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

Thank you.

It works until

$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)

So script is true before Action 7 and reports the following:

Action 7 retrieve document after clicking a hyperlink
Title is: <>    Class   := <Chrome_RenderWidgetHostHWND>    controltype:= <50030>   
"D:\Program Files\AutoIt3\Examples\IUIAutomation\UIAWrappers.au3" (424) : ==> Variable must be of type "Object".:
$obj.getCurrentPattern($PatternId, $pPattern)
$obj^ ERROR

[Lyr]

Hi

I see here some possibility for Chrome, something richer than run, winwaitactive, send("login tab password enter"). Thanks a lot. I'll experiment with it somewhen my brain's not melting from a week's work, next monday.

I have a question though: does this bring possibility for Java application?

I'm thinking about Oracle Sql Developer here. In both case (chrome and sqldev), my scenario is KeePass url/login/password/etc entry, double clic => [autoit connector] => Application running and connected. Regarding Sql Developer, AU3Info just tell me it's a SunAwtFrame, and except sending tab or clic on coordinate we can't create a new connection.

Regards,