Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


junkew
 Share

Recommended Posts

Quickstart 

  • Unzip the 2 zips (UIA*.zip and examples*.zip) of post #1
  • Start simplespy.au3
  • Hover your mouse over the element you want to handle
  • Press ctrl+w
  • Copy / paste the given source to a new AutoIT script
  • If you need help
    • ask for help in new thread in general support section and post
      • screenshot that shows your element is highlighted with simplespy
      • post output of simplespy with all properties
      • attached log.txt if available in your script directory
      • use allways contents of latest zip files which could be work in progress (WIP) zip

As you will not come far if you do not understand the basics first do this

Understand AutoIT

Understand testing frameworks and related concepts

 

If you understand above learn the basic way of setting focus and clicking thru UIA and the wrappers
 
Understand IUIAutomation wrappers and framework
Lets do the QuickStart steps as described above on the Start Button of Windows

  • You will get this
*** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP0=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=;controltype:=Pane;class:=Shell_TrayWnd", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("Starten.mainwindow","title:=Starten;classname:=Button")
_UIA_action("Starten.mainwindow","setfocus")

This code is not yet perfect (and simplespy will improve over time) and you will have to change still manually

  • _UIA_setVar is only used to get a "logical" name for a technical description
  • There are many ways for UIA which varies from simple thru the wrappers to advanced based on the CUIAutomation2.au3

This is better coded

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
Local $oP0

$oP0=_UIA_action("Title:=;controltype:=Pane;class:=Shell_TrayWnd", "getobject")
_UIA_Action($oP0,"move")

_UIA_action("title:=Starten;classname:=Button;controltype:=Button","move")
_UIA_action("title:=Starten;classname:=Button;controltype:=Button","invoke")

;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","move")
;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","invoke")

Exercise

  • Learn first to automate some simple stuff on notepad.exe and calc.exe by starting those applications and with ctrl+w within simplespy try some code snippets

See for latest examplecode the zip in first post

Example 1 Iterating thru the different ways of representing the objects in the tree
a. Autoit WinList function
b. UIAutomation RawViewWalker
c. UIAutomation ControlViewWalker
d. UIAutomation ContentViewWalker
e. Finding elements based on conditions (based on property id, frequently search on name or classname)
 
 

 

;~ Example 1 Iterating thru the different ways of representing the objects in the tree
;~ a. Autoit WinList function
;~ b. UIAutomation RawViewWalker
;~ c. UIAutomation ControlViewWalker
;~ d. UIAutomation ContentViewWalker
;~ e. Finding elements based on conditions (based on property id, frequently search on name or classname)

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <MsgBoxConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

samplewinlist()

sampleTW(1)
sampleTW(2)
sampleTW(3)

ConsoleWrite("**** Desktop windows ****" & @CRLF)
findThemAll($UIA_oDesktop, $TreeScope_Children)

ConsoleWrite(@CRLF & "**** All childs of the taskbar ****")
$oTaskBar = _UIA_gettaskbar()
findThemAll($oTaskBar, $treescope_subtree)

ConsoleWrite(@CRLF & "**** Deal with browser windows ****" & @CRLF)
Local $myBrowser
;- Just get the first browser we can find
$myBrowser=getBrowser()

$tReturn=MsgBox($MB_SYSTEMMODAL + $MB_OKCANCEL , "Title", "Make sure you have 2 chrome browsers open otherwise press X button to abort.")
if $tReturn<>$IDOK  Then
    ConsoleWrite(@CRLF & "**** ABORTING chrome browser demo ****" & @CRLF)
    exit
EndIf

consolewrite(@CRLF & "**** Continue with finding 2 browser windows ****" & @CRLF)

;- Just get the first browser we can find of this type
$browserType= "Google Chrome"
;~ $browserType= "Internet Explorer"
;~ $browserType= "Opera"

$myBrowser=getBrowser($browserType,1)
if isobj($myBrowser) Then
    consolewrite("Yes found first browser " & @CRLF)
endif

ConsoleWrite(@CRLF & "**** Deal with 2nd browser window****" & @CRLF)
;- Just get the 2nd browser we can find of this type
$myBrowser=getBrowser($browserType,2)
if isobj($myBrowser) Then
    consolewrite("Yes found second browser " & @CRLF)
Else
    consolewrite("No 2nd browser found so unable to focus and highlight " & @CRLF)
    exit
endif

_UIA_Action($myBrowser,"setfocus")
_UIA_Action($myBrowser,"highlight")

Func sampleWinList()
    ConsoleWrite("sample winlist (title not empty and visible windows only)" & @CRLF)
    Local $hTimer = TimerInit()
    Local $var = WinList()

    For $i = 1 To $var[0][0]
        ; Only display visble windows
        If $var[$i][0] <> "" And IsVisible($var[$i][1]) Then
            ConsoleWrite($i & ": Title=" & $var[$i][0] & @TAB & "Handle=" & $var[$i][1] & " Class=" &  _WinAPI_GetClassName($var[$i][1]) & @CRLF)
        EndIf
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("SampleWinList took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleWinList

Func IsVisible($handle)
    If BitAND(WinGetState($handle), 2) Then
        Return 1
    Else
        Return 0
    EndIf

EndFunc   ;==>IsVisible

Func getBrowser($browserName="",$browserIndex=1)
    local $iBrowserIdx=0
;- Create treewalker
    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

;~ Get first element
    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

;~ Iterate all desktopwindows
    While IsObj($oUIElement) = True
        local $tName= _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)
        Local $tHandle= Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId))
        Local $tClassName= _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid)
;~      Browser normally put their browsername after the - (so this fails when multiple - are in title so thats why we search from right)
        local $tStr=stringmid($tname,stringinstr($tName,"-",  $STR_NOCASESENSE, -1)+2)

;- See if its a browser (based on title it could be done on class but preferred logical name instead of a classname)
;~ Private modes diffferent title to deal with only IE misbehaves
;~ InPrivate - Internet Explorer - [InPrivate]
;~ InPrivate - [InPrivate] ?- Microsoft Edge
;~ Google Chrome (Incognito)
;~ Mozilla Firefox (Private Browsing)
;~ Privé-browsen - Opera<

;~      Fix misbehave
        if $tStr="[InPrivate]" then $tStr="Internet Explorer"
        if stringinstr($tStr," (") then
            $tStr=stringleft($tstr,stringinstr($tStr," (")-1)
        EndIf
;~      consolewrite("location: " & stringinstr($tName,"-",  $STR_NOCASESENSE, -1) & "<" & $tstr & "><" & $tname & "><" &  $tclassname & "><" &   $tHandle & ">"& @CRLF)

        $itsABrowserTitle=stringinstr("Google Chrome,Internet Explorer,Mozilla Firefox,Microsoft Edge,Opera",$tStr,$STR_NOCASESENSE )
        if $itsABrowserTitle > 0 then
;~          consolewrite("** Potential match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
            if ($browserName=stringleft($tStr,stringlen($browsername))) or ($browserName="") Then
;~              consolewrite("** match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
                $iBrowserIdx=$iBrowserIdx+1
;~              consolewrite($browserIndex & $iBrowserIdx)
                if $iBrowserIDX=$browserIndex Then
                    return $oUIElement
                endif
            EndIf
        EndIf

        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    WEnd

EndFunc   ;==>sampleTW


Func sampleTW($t)
    ConsoleWrite("initializing tw " & $t & @CRLF)
    Local $hTimer = TimerInit()
    Local $i=0
;~ ' Lets show all the items of the desktop with a treewalker
    If $t = 1 Then $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    If $t = 2 Then $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)
    If $t = 3 Then $UIA_oUIAutomation.ContentViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

    While IsObj($oUIElement) = True
        ConsoleWrite($i & "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Handle=" & Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        $i=$i+1
    WEnd

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Sample tw " & $t & " took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleTW

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

edit 21 aug 2014

  • Added quick start stuff
Edited by junkew
Link to comment
Share on other sites

junkew, nice job.  I remember discussing this stuff a long while back, as I had worked on the IAccessible interface (which is also included as a 'legacy' element in IUIAutomation)

I think I got discouraged with that project because there was just too much information returned. I should really have coded it to have a filter of some sort plus a 'search' method, but laziness or disinterest set in.

I remember you had a treeview type of interface showing the IUI elements under the mouse, right?  That would have been a great start for a 'next-gen' Window Info tool.

Link to comment
Share on other sites

G'day Junkew

There is so much that the Windows API can't access and it's long overdue that UIautomation was brought into AutoIt.

I second Ascend4nt's comment about a "next-gen" Window info tool.  It'd be a great tool to explore this new way of Automating Windows.

Like

What I've always liked to see in the existing info tool is code samples to cut and paste.  In other words you select an item click a button and get a bit of code to cut and paste into your program.

I've got many ideas about it that I won't expand on here unless you're interested.

AND

I'm not just asking for something as always I'm willing to help out in any way I can if you need any help. :)

As you're the current expert :) how hard would it be to convert help file examples using standard syntax to UIautomation examples?

Thanks for the effort!

John Morrison

Link to comment
Share on other sites

Simple scripts
The whole thread is having a history and more or less started with the complex scripts and over time more abstraction was put in UIAWrappers.au3 to make life easier. So the higher numbered examples are for starters easier to start with. This post just to give some quick reference to those examples and an advicable way to learn your way around.
 
For starters its adviceable to start just with
1. calculator and notepad example over >here
2. desktop and taskbar examples
3. combine above
 
Simple+
4. continue your steps with simple forms in IE browsers
5. continue with other browsers like chrome and firefox
6. Opera browser(s) is less open for automation
 
Complex
7. Deal with tables / grids
8. Deal with tables / grids in three different browsers 
9. testing framework with logging and configuration
10. The whole iuiautomation framework of microsoft
 
 
Further comments

  • Feel free to contribute or add items on a TODO list
  • My first priority is to share the basic examples and make a minimal library to make life easier for startes on this UI Automation object
  • Its a powerfull model but you have to start coding to learn the basics

@Ascend4nt: First post is already having a "spy" use your mouse and click or use ctrl+m all detailed properties will show up in the edit box
In general I use inspect.exe out of the windows SDK which gives me all the details so not much priority from my side to make it in a treeview format
@storne:        I don't think it will make sense to make functions like _UIA_ControlSetFocus(.....) as UIA goes in some areas far from the logic of AutoIT so hard to map.
I will add some basic examples feel free to have certain requests or try your own
Many examples are on google for example to move an element/window: http://msdn.microsoft.com/en-us/library/ms745106.aspx

  • Created UIAWrappers.AU3 and attached to first post

Example 2 Finding the taskbar and clicking on the start menu button

 

 

 

;~ Example 2 Finding the taskbar and clicking on the start menu button
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

Consolewrite("**** All childs of the taskbar will be written to logfile see xml or txt file created in folder of example .au3 file ****" & @CRLF)
Consolewrite("** Logfile default location: " & _uia_getVar("logfilename") & @CRLF)

;~ Maintainable way for doing stuff with taskbar and startbutton by giving logical name to technical description
_UIA_setVar("oTaskbar","Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd")    ;
_UIA_setVar("oStartbutton","title:=Starten;classname:=Start")

$oTaskBar=_UIA_gettaskbar()
_UIA_DumpThemAll($otaskbar,$treescope_subtree)

local $pInvoke, $oStart
local $CaptionStartButton

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
Consolewrite("Your languagecode influences name of start button and your languagecode is: " & @oslang & @CRLF)
switch @OSLang
    case 0409
        $CaptionStartButton="start"
    case 0413
        $CaptionStartButton="Starten"
    case 1033
        $CaptionStartButton="Start"
    case Else
        MsgBox($MB_SYSTEMMODAL, "Title", "Language unknown please extend script with right caption of start button and post it in thread", 3)
EndSwitch

;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,$CaptionStartButton, $treescope_subtree)
if isobj($oStart) Then
    consolewrite("Start button found" & @CRLF)

Else
    consolewrite("I bet you did not read the messagebox. Script will stop")
    exit
EndIf

;~ Get the invoke pattern to click on the item
$oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP.invoke()

sleep(500)

;~ And doing it in a maintainable way the logical / technical definition independent of each other
_UIA_Action($UIA_oDesktop,"click",1,1)
sleep(500)
_UIA_Action("oTaskbar","setfocus")
sleep(500)
_UIA_Action("oStartButton","click")
Edited by junkew
Link to comment
Share on other sites

This shows a little more on the concept of

1. Find your element

2. Think what you want and find the right pattern

3. Retrieve the pattern and execute the action

 

Within the patterns you will find not likely direct support for clicking right mouse and default actions are not allways what you want so then you have to fallback to mousemove, mouseclick functions of autoit or the sendinput function from microsoft (not declared in the standard autoit library)

 

updated UIAWrappers.AU3 with some additional helper functions to get below working

 

Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info)

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

Consolewrite("**** Get the taskbar ****" & @CRLF)
$oTaskBar=_UIA_gettaskbar()
;~ Equal To
;~ _UIA_getFirstObjectOfElement($oDesktop,"classname:=Shell_TrayWnd",$TreeScope_Children)

Consolewrite("**** All childs of the taskbar ****" & @CRLF)
;~  _UIA_dumpThemAll($otaskbar,$treescope_subtree)

Dim $pInvoke ;~ Pattern invoke
Dim $oStart  ;~ Object reference to IUIElement for button start

Consolewrite("**** Different ways of assigning the startbutton to an object ****" & @CRLF)
;~ $oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"name:=Starten", $treescope_subtree)

Dim $oClock  ;~ Object reference to IUIElement for button of clock, be aware this is a panel not a button
$oClock=_UIA_getFirstObjectOfElement($oTaskbar,"classname:=TrayClockWClass", $treescope_subtree)

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)

if isobj($oStart) Then
    consolewrite("Start button found")
Else
    consolewrite("I bet the text has to change to Start instead of Starten")
EndIf

Consolewrite("Get the invoke pattern to click on the start button item. Invoke possible: " & _UIA_getPropertyValue($oUIElement, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
;~ $oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
;~ $oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP=_UIA_getpattern($oStart,$UIA_InvokePatternID)
$oInvokeP.invoke()
sleep(100)

Consolewrite("Get the menu that is after the start button" & @crlf)

$oMenuStart=_UIA_getFirstObjectOfElement($oDesktop,"Menu Start", $treescope_children)
if isobj($oMenuStart) Then
    consolewrite("Menu start found" & @crlf)
Else
    consolewrite("I bet the text has to change to Start instead of Starten" & @crlf)
EndIf

dim $oStartMenuItem
$oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"name:=SciTE Script Editor", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"Microsoft Excel 2010", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"SpotGrit", $treescope_subtree)
if isobj($oStartMenuItem) Then
    consolewrite("Scite found" & @crlf)
Else
    consolewrite("scite not found" & @crlf)
EndIf

Consolewrite("Get the pattern to click on the menu after the start button. Invoke possible: " & _UIA_getPropertyValue($oStartMenuItem, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
$oInvokeP=_UIA_getpattern($oStartMenuItem,$UIA_InvokePatternID)
;~ This would definitely fail as there is no invoke pattern
if isobj($oInvokeP) Then
    consolewrite("invoke found lets see what happens" & @crlf)
Else
    consolewrite("invoke not supported" & @crlf)
EndIf
$oInvokeP.invoke()

sleep(2000)

consolewrite("So you saw it selected but did not click" & @crlf)
;~ still you can click as you now know the dimensions where to click
dim $t
$t=stringsplit(_UIA_getPropertyValue($oStartMenuItem, $UIA_BoundingRectanglePropertyId),";")
consolewrite($t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)
;~ _winapi_mouse_event($MOUSEEVENTF_ABSOLUTE + $MOUSEEVENTF_MOVE,$t[1],$t[2])
mousemove($t[1]+($t[3]/2),$t[2]+$t[4]/2)
mouseclick("left")
sleep(2000)

Consolewrite("**** try to click on the clock (TrayClockWClass) button of the taskbar ****" & @CRLF)
if isobj($oClock) Then
    consolewrite("Clock found" & @crlf)
Else
    consolewrite("Clock not found" & @crlf)
EndIf

dim $pLegacy
;~ $oClock.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oLegacyP=_UIA_getPattern($oClock,$UIA_LegacyIAccessiblePatternId)
$oLegacyP.dodefaultaction()

dim $oClock2
$oClock2=_UIA_getFirstObjectOfElement($odesktop,"classname:=ClockFlyoutWindow",$TreeScope_Children)
_UIA_dumpthemall($oClock2,$treescope_subtree)

Exit
Edited by junkew
Link to comment
Share on other sites

New example 4 that demonstrates on the calculator

  • How to click
  • How to find buttons (be aware that you have to change the captions to language of your windows)
  • How to click in the menu (copy result to the clipboard)
then it uses notepad to demonstrate
  • How to set a value on a textbox with the value pattern
Updated first post with latest uiawrappers.au3

 

Attention points

  • Examples are build on exact match of text (so this includes tab and Ctrl+C values), later I will make some function that can find with regexp or non exact match (need treewalker for that)
  • Timing / sleep is sometimes needed to give the system time to popup the menus / execute the action
  • Focus of application is sometimes to be set (and sometimes not as you look on the clicking of the buttons it will even happen when there is a screen in front of the calculator)
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

run("calc.exe")
run("notepad.exe")

$oCalc=_UIA_getFirstObjectOfElement($oDesktop,"class:=CalcFrame", $treescope_children)
$oNotepad=_UIA_getFirstObjectOfElement($oDesktop,"class:=Notepad", $treescope_children)

if isobj($oCalc) Then

;~ You can comment this out just there to get the names of whats available under the calc window
_UIA_DumpThemAll($oCalc,$treescope_subtree)

    $sText="1"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Optellen"   ;Add
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="3"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Is gelijk aan"    ;Is equal to
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Bewerken"    ;Edit
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    sleep(1500)
;~  findThemAll($oCalc,$treescope_subtree)
;~  sleep(1000)

;~ Use a regular expression to identify the copy choice as there are special characters/tabs etc in the name
    $sText="Kopiëren.*"    ;Copy
    $oButton=_UIA_getObjectByFindAll($oCalc,"name:=" & $sText, $treescope_subtree)
    if isobj($oButton) Then
        consolewrite("Menuitem is there")
    Else
        consolewrite("Menuitem is NOT there")
    EndIf
    sleep(1000)

    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke
    sleep(500)

EndIf

if isobj($oNotepad) Then

$myText=clipget()

;~ You can comment this out
_UIA_dumpThemAll($oNotepad,$treescope_subtree)
    sleep(1000)

    $oNotepad.setfocus()

    $sText="Edit"
    $oEdit=_UIA_getFirstObjectOfElement($oNotepad,"classname:=" & $sText, $treescope_subtree)
    $oValueP=_UIA_getpattern($oEdit,$UIA_ValuePatternId)
    $oValueP.SetValue($myText)

EndIf

Exit
Edited by junkew
Link to comment
Share on other sites

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ #au3check -q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ Example 5 Automating chrome
;~ This example shows how to use UI Automation with chrome (later an example will come to put stuff in the html page)
;~ 1. Have chrome started with "--force-renderer-accessibility"
;~ 2. Check with chrome://accessibility in the adress bar if accessibility is on/off, turn it on
;~ a. or close all browsers and change in script the run command to the right path of the chrome.exe
;~
;~ Apparently google changed some stuff in chrome version 29 and seems not allways to return classname properly so sometimes
;~ harder to get the right element
;- Even worse on version 42 not returning UIA controltype or classnames for certain elements
;~ Version chrome 65

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
;~ #au3check -q -d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w- 7
#include <debug.au3>

#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=N  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

local $strChromeStartup="--force-renderer-accessibility"
local $strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
local $strChromeExe=$strChromeExeFolder & "chrome.exe "

if not fileexists($strChromeExe) Then
    $strChromeExeFolder=@ProgramFilesDir & "\Google\Chrome\Application\"
    $strChromeExe=$strChromeExeFolder & "chrome.exe "
EndIf

;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    if not processexists("chrome.exe") Then
        consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
        consolewrite($strChromeExeFolder & @CRLF)
        consolewrite(@ProgramFilesDir & @CRLF)
;~      C:\Program Files (x86)\Google\Chrome\Application

        consolewrite($strChromeExe & $strChromeStartup & @CRLF)
        Exit
    endif
EndIf

local $oChrome=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
$oChrome.setfocus()

;~ _UIA_DumpThemAll($oDesktop,$treescope_children)

if isobj($oChrome) Then
;~ All stuff on the page to be dumped in an xml file in log folder
_UIA_DumpThemAll($oChrome,$treescope_subtree)

;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
;~ local $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~ $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"helptext:=OmniboxViewViews", $treescope_subtree) ;works in chrome 42
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29

local $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"title:=Adres- en zoekbalk"  , $treescope_subtree) ;works in chrome 65

;~  _UIA_DumpThemAll($oChromeToolbar,$treescope_children)
local $oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)

;~  get the value of the addressbar
local $myText=""
$oValueP.CurrentValue($myText)
consolewrite("address: " & $myText & @CRLF)
sleep(500)

;~ Click all tabs, chrome hides them a little in the hierarchy but first find the tablist
local $oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
 _UIA_DumpThemall($oChromeTabs,$treescope_children)

;~  Get result with findall function alternative could be the treewalker
    local  $pCondition, $pTrueCondition
    local  $pElements, $iLength

;~  and now just find all childs which are 4 tabs and 1 new button looking like a tab
    $UIA_oUIAutomation.CreateTrueCondition($pTRUECondition)
    local $oCondition=ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition,$dtagIUIAutomationCondition)

    $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements)
    local $oAutomationElementArray = ObjCreateInterFace($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    consolewrite("We have " & $ilength & " tabs" & @CRLF)

    For $i = 0 To $iLength - 1; it's zero based
    $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        local $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        consolewrite( "Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= <" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB & @CRLF)
;~  Invoke or select them all
;~ Tricky as chrome seems to say it supports certain patterns but then they seem not to be implemented


;~ only tabs with content
if _UIA_getPropertyValue($oUIElement, $UIA_IsSelectionItemPatternAvailablePropertyId) = "True" Then
_UIA_action($oUIElement,"leftclick")
;~  $oSelectP=_UIA_getpattern($oUIElement,$UIA_SelectionItemPatternId)
;~  $oSelectP.Select()
EndIf


Next

EndIf

;~ Lets open a new tab within chrome by finding the button in the tabstrip
local $oChromeNewTab=_UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
_UIA_action($oChromeNewTab,"leftclick")

;~ Lets get the valuepattern of the addressbar
;~  $oLegacyP=_UIA_getpattern($oChromeAddressBar,$UIA_LegacyIAccessiblePatternId)
;~

_UIA_action($oChromeAddressBar,"leftclick")
_UIA_action($oChromeAddressBar,"setvalue using keys","www.autoitscript.com/forum{ENTER}")

;~ some time to load the browser page
sleep(2000)

exit

 

Edited by junkew
Link to comment
Share on other sites

@TranceXX: Just uploaded latest UIAWrappers.AU3 maybe I missed to do that. If you still get an error let me know
 

Special things that sometimes is needed (hopefully I can manage this in the UIAWrappers.AU3 later)

  • Open once chrome is started the url chrome://accessibility/ to see if accessibility is on
  • Chrome version 29 as Google is still changing classnames, names of properties etc. its not allways the same to find an object
    A terrible workaround is needed for html input text boxes as their name is determined by the value (seems to be specified that way by W3C) and not on the name or id attribute (only picks attribut aria-label when its available)
  • Start chrome manually or remove commented run statement for starting chrome (not handling multiple instances of chrome at the moment)
  • OS Language can influence stuff (I am on a dutch OS so sometimes names have to be set in english so then I sometimes switch to find by one of the other properties)
  • If stuff is not working use the findThemAll(<handle to object>, <treescope>) it will write the childs with the most common properties to the output window
    Be aware on treescope_subtree as if it walks to the leafs of the hierarchy it can take minutes especially if multiple HTML windows are open
$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
findThemAll($oDesktop,$treescope_children)

 

Below some background for TODO later

Chrome supports following interfaces under Windows:
  • MSAA/IAccessible (complete)
  • IAccessible2 (mostly complete)
  • ISimpleDOM (mostly complete)
  • IAccessibleEx and UI Automation (very limited)

 

@TranceXX: I should be able to get to the IAccessible2 interface by making a call to QueryInterface as Chrome supports below interfaces but I am a little less experienced in this interface/com stuff so maybe you can help out

accProbe will show what IAccessible2 can reveal out of an html page (first register the needed proxy dll)

a. IAccessible2.IDL needs to be converted to Autoit syntax

b. This should be written in AutoIT syntax
 

HWND hw = GetForegroundWindow();
    IAccessible *pIA;
    HRESULT hr = AccessibleObjectFromWindow(hw, OBJID_CLIENT, IID_IAccessible, (void**)&pIA);
    if (!SUCCEEDED(hr))
        return -1;

    const IID IID_IAccessible2 = {0xE89F726E, 0xC4F4, 0x4c19, 0xbb, 0x19, 0xb6, 0x47, 0xd7, 0xfa, 0x84, 0x78};
    ::IServiceProvider *pService = NULL; 
    hr = pIA->QueryInterface(IID_IServiceProvider, (void **)&pService); 
    if(SUCCEEDED(hr))
    {
        IAccessible2 *pIA2 = NULL; 
        hr = pService->QueryService(IID_IAccessible2, IID_IAccessible2, (void**)&pIA2); 
        if (SUCCEEDED(hr) && pIA2)
        { 
            pIA2->Release();
        } 
        pService->Release();
    }  
Link to comment
Share on other sites

That code written in AutoIt would be something like this (interfaces definitions missing):

Global Const $hOLEACC_DLL = DllOpen("oleacc.dll")
Global Const $OBJID_CLIENT = 0xFFFFFFFC

Func YourCodeWrittenInAutoIt($hWindow)
    Local $tIID_IAccessible = _WinAPI_GUIDFromString($sIID_IAccessible)
    Local $pIAccessible = AccessibleObjectFromWindow($hWindow, $OBJID_CLIENT, $tIID_IAccessible)
    ; Object from pointer
    Local $oIAccessible = ObjCreateInterface($pIAccessible, $sIID_IAccessible, $tagIAccessible)
    If @error Then Return -1

    Local $pIServiceProvider
    $oIAccessible.QueryInterface($sIID_IServiceProvider, $pIServiceProvider)
    ; Again object from pointer
    Local $oIServiceProvider = ObjCreateInterface($pIServiceProvider, $sIID_IServiceProvider, $tagIServiceProvider)
    If @error Then Return -2
    
    Local $pIA2
    $oIServiceProvider.QueryService($IID_IAccessible2, $IID_IAccessible2, $pIA2)
    ; And again creating object from pointer
    Local $oIA2 = ObjCreateInterface($pIA2, $sIID_IAccessible2, $tagIAccessible2)
    If @error Then Return -3
    
    ;...
EndFunc

Func AccessibleObjectFromWindow($hWindow, $iObjId, $tRefIID)
    Local $aCall = DllCall($hOLEACC_DLL, "long", "AccessibleObjectFromWindow", "hwnd", $hWindow, "dword", $iObjId, "struct*", $tRefIID, "ptr*", Null)
    If @error Or $aCall[0] < 0 Then Return SetError(1, 0, 0)
    Return $aCall[4]
EndFunc
If you look closely at that (and your) code you will see call to oIAccessible.QueryInterface(...). That probably means that you can skip one step and specify $tIID_IServiceProvider directly in earlier call to AccessibleObjectFromWindow to get $pIServiceProvider pointer directly. Edited by trancexx

♡♡♡

.

eMyvnE

Link to comment
Share on other sites

From the uiautomation framework I can get to $oIaccessible by doing below code

The AccessibleObjectFromWindow is the MSAA way of getting the interface but thx again for the example as this will help later on getting to IAccessible2 or ISimpleDom

$oLegacyP=_UIA_getpattern($oUIElement,$UIA_LegacyIAccessiblePatternId)
    $oIAccessible = $oLegacyP.GetIAccessible()
Link to comment
Share on other sites

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

Uodate UIAWrappers.AU3

Made a lot more comments and failure to show when an object is not retrieved/found the hierarchy of the tree is written to the console

Lots of optimizations could be done by using the cachefunctions (less interprocess communication) but on my machine it runs at a very acceptable speed

In top of script put the text in your local language of the operating system

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
 
#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
 
;~ Make this language specific
const $cToolbarByName = "name:=Google Chrome Toolbar"
const $cAddressBarByName = "name:=Adres- en zoekbalk"
const $cChromeNewTabByName="name:=Nieuw tabblad"
 
$strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
$strChromeStartup="--force-renderer-accessibility"
$strChromeExe=$strChromeExeFolder & "chrome.exe "
 
;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
    consolewrite($strChromeExe & $strChromeStartup & @CRLF)
EndIf
 
;~ Find the chrome window
$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
if not isobj($oChrome) Then
    _UIA_DumpThemAll($oDesktop,$treescope_children)
EndIf
 
;~ Make sure chrome is front window
$oChrome.setfocus()
 
if isobj($oChrome) Then
    consolewrite("Action 1" & @CRLF)
 
;~  get the chrome toolbar
;~  $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_ToolBarControlTypeId, $treescope_subtree)
    $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)
    if not isobj($oChromeToolbar) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
 
consolewrite("Action 2" & @CRLF)
;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29
    $oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
    if not isobj($oChromeAddressbar) Then
        _UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
    EndIf
 
;~  $oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)
;~  sleep(2000)
 
;~  get the value of the addressbar
;~  $myText=""
;~  $oValueP.CurrentValue($myText)
;~  consolewrite("address: " & $myText & @CRLF)
 
consolewrite("Action 3" & @CRLF)
;~ Get reference to the tabs
    $oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
    if not isobj($oChromeTabs) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
;~ Lets open a new tab within chrome
 
consolewrite("Action 4" & @CRLF)
;~  $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
    $oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
    if not isobj($oChromeNewTab) Then
        _UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
    EndIf
    _UIA_action($oChromeNewtab,"leftclick")
    sleep(500)
 
consolewrite("Action 4a" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
 
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
 
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
 
 
consolewrite("Action 4b" & @CRLF)
;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
 
sleep(500)
 
consolewrite("Action 4c retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=On", $treescope_subtree)
if not isobj($oForumLink) Then
consolewrite("*** Scripting will fail as accessibility is off ****")
MsgBox(4096, "Accessibility warning", "Accessibility is turned off, put it on by clicking on Off after Global accessibility mode", 10)
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
 
consolewrite("Action 4d" & @CRLF)
;~ $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
$oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
if not isobj($oChromeNewTab) Then
_UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
EndIf
_UIA_action($oChromeNewtab,"leftclick")
sleep(500)
consolewrite("Action 5" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName , $treescope_subtree) ;works in chrome 29
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
_UIA_action($oChromeAddressBar,"setvalue using keys","www.autoitscript.com/{ENTER}")
consolewrite("Action 6" & @CRLF) ;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
sleep(500)
consolewrite("Action 7 retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)
if not isobj($oForumLink) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oForumLink,"invoke")
sleep(3000)
consolewrite("Action 8 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
 
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf ;~ Now we get the searchfield
$oEdtSearchForum=_UIA_getObjectByFindAll($oDocument,"name:=Search...", $treescope_subtree)
if not isobj($oEdtSearchForum) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oEdtSearchForum,"focus")
_UIA_action($oEdtSearchForum,"setvalue using keys","Chrome can easy be automated with ui automation") ; {ENTER}")
sleep(500)
;~ Exit
;~ Now we press the button, see relative syntax used as the button seems not to have a name its just 2 objects further then search field
$oBtnSearch=_UIA_getObjectByFindAll($oDocument,"name:=Search...; indexrelative:=2", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
sleep(1000)
_UIA_action($oBtnSearch,"invoke")
sleep(2000)
;~ consolewrite("Action 9 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$oHyperlink=_UIA_getObjectByFindAll($oDocument,"name:=controlsend doesn't work", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
sleep(1000)
_UIA_action($oHyperlink,"invoke")
sleep(2000)
EndIf
exit
 
 
 



 

edit 1 sept 2013: added steps 4a,b,c,d to warn for accessibility is off

Edited by junkew
Link to comment
Share on other sites

Capitally!

I can't run that last example. Apparently some function definition is missing.
Is there anything special I need to do except having that script together with two includes in the same folder?

Yes! This two includes in one line is typo.

I have Example5.working nice after replacement line with $strChromeExeFolder:

$strChromeExeFolder= "Your chrome EXE folder with \"

 

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

 

Example6 has problem, though. It can't create $oChromeToolbar as object in line #48, Here:

    $oChromeToolbar =_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)

Is it concerning with my chrome language ("ru") and invalid names (Russian) of const $cToolbarByName, $cAddressBarByName, and $cChromeNewTabByName?

The point of world view

Link to comment
Share on other sites

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

Link to comment
Share on other sites

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

Thank you.

It works until

$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)

So script is true before Action 7 and reports the following:

Action 7 retrieve document after clicking a hyperlink
Title is: <>    Class   := <Chrome_RenderWidgetHostHWND>    controltype:= <50030>   
"D:\Program Files\AutoIt3\Examples\IUIAutomation\UIAWrappers.au3" (424) : ==> Variable must be of type "Object".:
$obj.getCurrentPattern($PatternId, $pPattern)
$obj^ ERROR

The point of world view

Link to comment
Share on other sites

Hi

I see here some possibility for Chrome, something richer than run, winwaitactive, send("login tab password enter"). Thanks a lot. I'll experiment with it somewhen my brain's not melting from a week's work, next monday.

I have a question though: does this bring possibility for Java application?

I'm thinking about Oracle Sql Developer here. In both case (chrome and sqldev), my scenario is KeePass url/login/password/etc entry, double clic => [autoit connector] => Application running and connected. Regarding Sql Developer, AU3Info just tell me it's a SunAwtFrame, and except sending tab or clic on coordinate we can't create a new connection.

Regards,

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...