junkew

IUIAutomation MS framework automate chrome, FF, IE, ....

587 posts in this topic




ok got it going on my old xp machine ..... now for more examples :)

1 person likes this

[color=rgb(255,0,0);][font="'comic sans ms', cursive;"]FukuLeaks[/color][/font]

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Quickstart 

  • Unzip the 2 zips (UIA*.zip and examples*.zip) of post #1
  • Start simplespy.au3
  • Hover your mouse over the element you want to handle
  • Press ctrl+w
  • Copy / paste the given source to a new AutoIT script
  • If you need help
    • ask for help in new thread in general support section and post
      • screenshot that shows your element is highlighted with simplespy
      • post output of simplespy with all properties
      • attached log.txt if available in your script directory
      • use allways contents of latest zip files which could be work in progress (WIP) zip

As you will not come far if you do not understand the basics first do this

Understand AutoIT

Understand testing frameworks and related concepts

 

If you understand above learn the basic way of setting focus and clicking thru UIA and the wrappers
 
Understand IUIAutomation wrappers and framework
Lets do the QuickStart steps as described above on the Start Button of Windows

  • You will get this
*** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP0=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd", $treescope_children)   
_UIA_Action($oP0,"setfocus")
_UIA_setVar("Starten.mainwindow","title:=Starten;classname:=Button")
_UIA_action("Starten.mainwindow","setfocus")

This code is not yet perfect (and simplespy will improve over time) and you will have to change still manually

  • _UIA_setVar is only used to get a "logical" name for a technical description
  • There are many ways for UIA which varies from simple thru the wrappers to advanced based on the CUIAutomation2.au3

This is better coded

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)
Local $oP0

_UIA_setVar("RTI.SEARCHCONTEXT", $UIA_oDesktop)


$oP0=_UIA_action("Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd", "getobject")
_UIA_Action($oP0,"move")

_UIA_action("title:=Starten;classname:=Button;controltype:=UIA_ButtonControlTypeId","move")
_UIA_action("title:=Starten;classname:=Button;controltype:=UIA_ButtonControlTypeId","invoke")

;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","move")
;~ _UIA_action("title:=Start.*;classname:=Button;instance:=2","invoke")

Exercise

  • Learn first to automate some simple stuff on notepad.exe and calc.exe by starting those applications and with ctrl+w within simplespy try some code snippets

See for latest examplecode the zip in first post

Example 1 Iterating thru the different ways of representing the objects in the tree
a. Autoit WinList function
b. UIAutomation RawViewWalker
c. UIAutomation ControlViewWalker
d. UIAutomation ContentViewWalker
e. Finding elements based on conditions (based on property id, frequently search on name or classname)
 
 

 

;~ Example 1 Iterating thru the different ways of representing the objects in the tree
;~ a. Autoit WinList function
;~ b. UIAutomation RawViewWalker
;~ c. UIAutomation ControlViewWalker
;~ d. UIAutomation ContentViewWalker
;~ e. Finding elements based on conditions (based on property id, frequently search on name or classname)

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <MsgBoxConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

samplewinlist()

sampleTW(1)
sampleTW(2)
sampleTW(3)

ConsoleWrite("**** Desktop windows ****" & @CRLF)
findThemAll($UIA_oDesktop, $TreeScope_Children)

ConsoleWrite(@CRLF & "**** All childs of the taskbar ****")
$oTaskBar = _UIA_gettaskbar()
findThemAll($oTaskBar, $treescope_subtree)

ConsoleWrite(@CRLF & "**** Deal with browser windows ****" & @CRLF)
Local $myBrowser
;- Just get the first browser we can find
$myBrowser=getBrowser()

$tReturn=MsgBox($MB_SYSTEMMODAL + $MB_OKCANCEL , "Title", "Make sure you have 2 chrome browsers open otherwise press X button to abort.")
if $tReturn<>$IDOK  Then
    ConsoleWrite(@CRLF & "**** ABORTING chrome browser demo ****" & @CRLF)
    exit
EndIf

consolewrite(@CRLF & "**** Continue with finding 2 browser windows ****" & @CRLF)

;- Just get the first browser we can find of this type
$browserType= "Google Chrome"
;~ $browserType= "Internet Explorer"
;~ $browserType= "Opera"

$myBrowser=getBrowser($browserType,1)
if isobj($myBrowser) Then
    consolewrite("Yes found first browser " & @CRLF)
endif

ConsoleWrite(@CRLF & "**** Deal with 2nd browser window****" & @CRLF)
;- Just get the 2nd browser we can find of this type
$myBrowser=getBrowser($browserType,2)
if isobj($myBrowser) Then
    consolewrite("Yes found second browser " & @CRLF)
Else
    consolewrite("No 2nd browser found so unable to focus and highlight " & @CRLF)
    exit
endif

_UIA_Action($myBrowser,"setfocus")
_UIA_Action($myBrowser,"highlight")

Func sampleWinList()
    ConsoleWrite("sample winlist (title not empty and visible windows only)" & @CRLF)
    Local $hTimer = TimerInit()
    Local $var = WinList()

    For $i = 1 To $var[0][0]
        ; Only display visble windows
        If $var[$i][0] <> "" And IsVisible($var[$i][1]) Then
            ConsoleWrite($i & ": Title=" & $var[$i][0] & @TAB & "Handle=" & $var[$i][1] & " Class=" &  _WinAPI_GetClassName($var[$i][1]) & @CRLF)
        EndIf
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("SampleWinList took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleWinList

Func IsVisible($handle)
    If BitAND(WinGetState($handle), 2) Then
        Return 1
    Else
        Return 0
    EndIf

EndFunc   ;==>IsVisible

Func getBrowser($browserName="",$browserIndex=1)
    local $iBrowserIdx=0
;- Create treewalker
    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

;~ Get first element
    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

;~ Iterate all desktopwindows
    While IsObj($oUIElement) = True
        local $tName= _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)
        Local $tHandle= Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId))
        Local $tClassName= _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid)
;~      Browser normally put their browsername after the - (so this fails when multiple - are in title so thats why we search from right)
        local $tStr=stringmid($tname,stringinstr($tName,"-",  $STR_NOCASESENSE, -1)+2)

;- See if its a browser (based on title it could be done on class but preferred logical name instead of a classname)
;~ Private modes diffferent title to deal with only IE misbehaves
;~ InPrivate - Internet Explorer - [InPrivate]
;~ InPrivate - [InPrivate] ?- Microsoft Edge
;~ Google Chrome (Incognito)
;~ Mozilla Firefox (Private Browsing)
;~ Privé-browsen - Opera<

;~      Fix misbehave
        if $tStr="[InPrivate]" then $tStr="Internet Explorer"
        if stringinstr($tStr," (") then
            $tStr=stringleft($tstr,stringinstr($tStr," (")-1)
        EndIf
;~      consolewrite("location: " & stringinstr($tName,"-",  $STR_NOCASESENSE, -1) & "<" & $tstr & "><" & $tname & "><" &  $tclassname & "><" &   $tHandle & ">"& @CRLF)

        $itsABrowserTitle=stringinstr("Google Chrome,Internet Explorer,Mozilla Firefox,Microsoft Edge,Opera",$tStr,$STR_NOCASESENSE )
        if $itsABrowserTitle > 0 then
;~          consolewrite("** Potential match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
            if ($browserName=stringleft($tStr,stringlen($browsername))) or ($browserName="") Then
;~              consolewrite("** match ** " & $tStr & ":" & $browserName& @tab & $tname & @tab & $tclassname & @CRLF)
                $iBrowserIdx=$iBrowserIdx+1
;~              consolewrite($browserIndex & $iBrowserIdx)
                if $iBrowserIDX=$browserIndex Then
                    return $oUIElement
                endif
            EndIf
        EndIf

        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    WEnd

EndFunc   ;==>sampleTW


Func sampleTW($t)
    ConsoleWrite("initializing tw " & $t & @CRLF)
    Local $hTimer = TimerInit()
    Local $i=0
;~ ' Lets show all the items of the desktop with a treewalker
    If $t = 1 Then $UIA_oUIAutomation.RawViewWalker($UIA_pTW)
    If $t = 2 Then $UIA_oUIAutomation.ControlViewWalker($UIA_pTW)
    If $t = 3 Then $UIA_oUIAutomation.ContentViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        MsgBox(1, "UI automation treewalker failed", "UI Automation failed failed", 10)
    EndIf

    $oTW.GetFirstChildElement($UIA_oDesktop, $UIA_pUIElement)
    $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

    While IsObj($oUIElement) = True
        ConsoleWrite($i & "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Handle=" & Hex(_UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        $oTW.GetNextSiblingElement($oUIElement, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        $i=$i+1
    WEnd

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Sample tw " & $t & " took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)
EndFunc   ;==>sampleTW

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

edit 21 aug 2014

  • Added quick start stuff
Edited by junkew
1 person likes this

Share this post


Link to post
Share on other sites

junkew, nice job.  I remember discussing this stuff a long while back, as I had worked on the IAccessible interface (which is also included as a 'legacy' element in IUIAutomation)

I think I got discouraged with that project because there was just too much information returned. I should really have coded it to have a filter of some sort plus a 'search' method, but laziness or disinterest set in.

I remember you had a treeview type of interface showing the IUI elements under the mouse, right?  That would have been a great start for a 'next-gen' Window Info tool.

Share this post


Link to post
Share on other sites

G'day Junkew

There is so much that the Windows API can't access and it's long overdue that UIautomation was brought into AutoIt.

I second Ascend4nt's comment about a "next-gen" Window info tool.  It'd be a great tool to explore this new way of Automating Windows.

Like

What I've always liked to see in the existing info tool is code samples to cut and paste.  In other words you select an item click a button and get a bit of code to cut and paste into your program.

I've got many ideas about it that I won't expand on here unless you're interested.

AND

I'm not just asking for something as always I'm willing to help out in any way I can if you need any help. :)

As you're the current expert :) how hard would it be to convert help file examples using standard syntax to UIautomation examples?

Thanks for the effort!

John Morrison

1 person likes this

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Simple scripts
The whole thread is having a history and more or less started with the complex scripts and over time more abstraction was put in UIAWrappers.au3 to make life easier. So the higher numbered examples are for starters easier to start with. This post just to give some quick reference to those examples and an advicable way to learn your way around.
 
For starters its adviceable to start just with
1. calculator and notepad example over >here
2. desktop and taskbar examples
3. combine above
 
Simple+
4. continue your steps with simple forms in IE browsers
5. continue with other browsers like chrome and firefox
6. Opera browser(s) is less open for automation
 
Complex
7. Deal with tables / grids
8. Deal with tables / grids in three different browsers 
9. testing framework with logging and configuration
10. The whole iuiautomation framework of microsoft
 
 
Further comments

  • Feel free to contribute or add items on a TODO list
  • My first priority is to share the basic examples and make a minimal library to make life easier for startes on this UI Automation object
  • Its a powerfull model but you have to start coding to learn the basics

@Ascend4nt: First post is already having a "spy" use your mouse and click or use ctrl+m all detailed properties will show up in the edit box
In general I use inspect.exe out of the windows SDK which gives me all the details so not much priority from my side to make it in a treeview format
@storne:        I don't think it will make sense to make functions like _UIA_ControlSetFocus(.....) as UIA goes in some areas far from the logic of AutoIT so hard to map.
I will add some basic examples feel free to have certain requests or try your own
Many examples are on google for example to move an element/window: http://msdn.microsoft.com/en-us/library/ms745106.aspx

  • Created UIAWrappers.AU3 and attached to first post

Example 2 Finding the taskbar and clicking on the start menu button

 

 

 

;~ Example 2 Finding the taskbar and clicking on the start menu button
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

;~ To prevent some FAQ questions
consolewrite("*** Some important settings you can find in " & @scriptdir & "\UIA.CFG ***" & @CRLF)
consolewrite("*** " & _UIA_getVersionInfoString() )
consolewrite("*** If logging is turned on you can find it here :" & _UIA_getVar("logFileName") & @CRLF)

Consolewrite("**** All childs of the taskbar will be written to logfile see xml or txt file created in folder of example .au3 file ****" & @CRLF)
Consolewrite("** Logfile default location: " & _uia_getVar("logfilename") & @CRLF)

;~ Maintainable way for doing stuff with taskbar and startbutton by giving logical name to technical description
_UIA_setVar("oTaskbar","Title:=;controltype:=UIA_PaneControlTypeId;class:=Shell_TrayWnd")    ;
_UIA_setVar("oStartbutton","title:=Starten;classname:=Start")

$oTaskBar=_UIA_gettaskbar()
_UIA_DumpThemAll($otaskbar,$treescope_subtree)

local $pInvoke, $oStart
local $CaptionStartButton

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
Consolewrite("Your languagecode influences name of start button and your languagecode is: " & @oslang & @CRLF)
switch @OSLang
    case 0409
        $CaptionStartButton="start"
    case 0413
        $CaptionStartButton="Starten"
    case 1033
        $CaptionStartButton="Start"
    case Else
        MsgBox($MB_SYSTEMMODAL, "Title", "Language unknown please extend script with right caption of start button and post it in thread", 3)
EndSwitch

;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,$CaptionStartButton, $treescope_subtree)
if isobj($oStart) Then
    consolewrite("Start button found" & @CRLF)

Else
    consolewrite("I bet you did not read the messagebox. Script will stop")
    exit
EndIf

;~ Get the invoke pattern to click on the item
$oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP.invoke()

sleep(500)

;~ And doing it in a maintainable way the logical / technical definition independent of each other
_UIA_Action($UIA_oDesktop,"click",1,1)
sleep(500)
_UIA_Action("oTaskbar","setfocus")
sleep(500)
_UIA_Action("oStartButton","click")
Edited by junkew

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

This shows a little more on the concept of

1. Find your element

2. Think what you want and find the right pattern

3. Retrieve the pattern and execute the action

 

Within the patterns you will find not likely direct support for clicking right mouse and default actions are not allways what you want so then you have to fallback to mousemove, mouseclick functions of autoit or the sendinput function from microsoft (not declared in the standard autoit library)

 

updated UIAWrappers.AU3 with some additional helper functions to get below working

 

Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info)

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

Consolewrite("**** Get the taskbar ****" & @CRLF)
$oTaskBar=_UIA_gettaskbar()
;~ Equal To
;~ _UIA_getFirstObjectOfElement($oDesktop,"classname:=Shell_TrayWnd",$TreeScope_Children)

Consolewrite("**** All childs of the taskbar ****" & @CRLF)
;~  _UIA_dumpThemAll($otaskbar,$treescope_subtree)

Dim $pInvoke ;~ Pattern invoke
Dim $oStart  ;~ Object reference to IUIElement for button start

Consolewrite("**** Different ways of assigning the startbutton to an object ****" & @CRLF)
;~ $oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"name:=Starten", $treescope_subtree)

Dim $oClock  ;~ Object reference to IUIElement for button of clock, be aware this is a panel not a button
$oClock=_UIA_getFirstObjectOfElement($oTaskbar,"classname:=TrayClockWClass", $treescope_subtree)

Consolewrite("**** try to click on the start button of the taskbar ****" & @CRLF)
;~ Get the first item that has as the name: Starten change to Start for english or other text to match start button in local language
$oStart=_UIA_getFirstObjectOfElement($oTaskbar,"Starten", $treescope_subtree)

if isobj($oStart) Then
    consolewrite("Start button found")
Else
    consolewrite("I bet the text has to change to Start instead of Starten")
EndIf

Consolewrite("Get the invoke pattern to click on the start button item. Invoke possible: " & _UIA_getPropertyValue($oUIElement, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
;~ $oStart.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
;~ $oInvokeP=objCreateInterface($pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
$oInvokeP=_UIA_getpattern($oStart,$UIA_InvokePatternID)
$oInvokeP.invoke()
sleep(100)

Consolewrite("Get the menu that is after the start button" & @crlf)

$oMenuStart=_UIA_getFirstObjectOfElement($oDesktop,"Menu Start", $treescope_children)
if isobj($oMenuStart) Then
    consolewrite("Menu start found" & @crlf)
Else
    consolewrite("I bet the text has to change to Start instead of Starten" & @crlf)
EndIf

dim $oStartMenuItem
$oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"name:=SciTE Script Editor", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"Microsoft Excel 2010", $treescope_subtree)
;~ $oStartMenuItem=_UIA_getFirstObjectOfElement($oMenuStart,"SpotGrit", $treescope_subtree)
if isobj($oStartMenuItem) Then
    consolewrite("Scite found" & @crlf)
Else
    consolewrite("scite not found" & @crlf)
EndIf

Consolewrite("Get the pattern to click on the menu after the start button. Invoke possible: " & _UIA_getPropertyValue($oStartMenuItem, $UIA_IsInvokePatternAvailablePropertyId) & @CRLF)
$oInvokeP=_UIA_getpattern($oStartMenuItem,$UIA_InvokePatternID)
;~ This would definitely fail as there is no invoke pattern
if isobj($oInvokeP) Then
    consolewrite("invoke found lets see what happens" & @crlf)
Else
    consolewrite("invoke not supported" & @crlf)
EndIf
$oInvokeP.invoke()

sleep(2000)

consolewrite("So you saw it selected but did not click" & @crlf)
;~ still you can click as you now know the dimensions where to click
dim $t
$t=stringsplit(_UIA_getPropertyValue($oStartMenuItem, $UIA_BoundingRectanglePropertyId),";")
consolewrite($t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)
;~ _winapi_mouse_event($MOUSEEVENTF_ABSOLUTE + $MOUSEEVENTF_MOVE,$t[1],$t[2])
mousemove($t[1]+($t[3]/2),$t[2]+$t[4]/2)
mouseclick("left")
sleep(2000)

Consolewrite("**** try to click on the clock (TrayClockWClass) button of the taskbar ****" & @CRLF)
if isobj($oClock) Then
    consolewrite("Clock found" & @crlf)
Else
    consolewrite("Clock not found" & @crlf)
EndIf

dim $pLegacy
;~ $oClock.getCurrentPattern($UIA_InvokePatternId, $pInvoke)
$oLegacyP=_UIA_getPattern($oClock,$UIA_LegacyIAccessiblePatternId)
$oLegacyP.dodefaultaction()

dim $oClock2
$oClock2=_UIA_getFirstObjectOfElement($odesktop,"classname:=ClockFlyoutWindow",$TreeScope_Children)
_UIA_dumpthemall($oClock2,$treescope_subtree)

Exit
Edited by junkew

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

New example 4 that demonstrates on the calculator

  • How to click
  • How to find buttons (be aware that you have to change the captions to language of your windows)
  • How to click in the menu (copy result to the clipboard)
then it uses notepad to demonstrate
  • How to set a value on a textbox with the value pattern
Updated first post with latest uiawrappers.au3

 

Attention points

  • Examples are build on exact match of text (so this includes tab and Ctrl+C values), later I will make some function that can find with regexp or non exact match (need treewalker for that)
  • Timing / sleep is sometimes needed to give the system time to popup the menus / execute the action
  • Focus of application is sometimes to be set (and sometimes not as you look on the clicking of the buttons it will even happen when there is a screen in front of the calculator)
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

run("calc.exe")
run("notepad.exe")

$oCalc=_UIA_getFirstObjectOfElement($oDesktop,"class:=CalcFrame", $treescope_children)
$oNotepad=_UIA_getFirstObjectOfElement($oDesktop,"class:=Notepad", $treescope_children)

if isobj($oCalc) Then

;~ You can comment this out just there to get the names of whats available under the calc window
_UIA_DumpThemAll($oCalc,$treescope_subtree)

    $sText="1"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Optellen"   ;Add
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="3"
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Is gelijk aan"    ;Is equal to
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    $sText="Bewerken"    ;Edit
    $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke

    sleep(1500)
;~  findThemAll($oCalc,$treescope_subtree)
;~  sleep(1000)

;~ Use a regular expression to identify the copy choice as there are special characters/tabs etc in the name
    $sText="Kopiëren.*"    ;Copy
    $oButton=_UIA_getObjectByFindAll($oCalc,"name:=" & $sText, $treescope_subtree)
    if isobj($oButton) Then
        consolewrite("Menuitem is there")
    Else
        consolewrite("Menuitem is NOT there")
    EndIf
    sleep(1000)

    $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
    $oInvokeP.Invoke
    sleep(500)

EndIf

if isobj($oNotepad) Then

$myText=clipget()

;~ You can comment this out
_UIA_dumpThemAll($oNotepad,$treescope_subtree)
    sleep(1000)

    $oNotepad.setfocus()

    $sText="Edit"
    $oEdit=_UIA_getFirstObjectOfElement($oNotepad,"classname:=" & $sText, $treescope_subtree)
    $oValueP=_UIA_getpattern($oEdit,$UIA_ValuePatternId)
    $oValueP.SetValue($myText)

EndIf

Exit
Edited by junkew

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

This example shows how to use UI Automation with chrome (later an example will come to put stuff in the html page)

1. Have chrome started with "--force-renderer-accessibility"

2. Check with chrome://accessibility in the adress bar if accessibility is on/off, turn it on

a. or close all browsers and change in script the run command to the right path of the chrome.exe

 

Apparently google changed some stuff in chrome version 29 and seems not allways to return classname properly so sometimes harder to get the right element

 

 

Example 5 Automating chrome

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

$strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
$strChromeStartup="--force-renderer-accessibility"
$strChromeExe=$strChromeExeFolder & "chrome.exe "

;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
    consolewrite($strChromeExe & $strChromeStartup & @CRLF)
EndIf

$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
$oChrome.setfocus()
;~ _UIA_DumpThemAll($oDesktop,$treescope_children)

if isobj($oChrome) Then
;~  _UIA_DumpThemAll($oChrome,$treescope_children)


;~  get the toolbar
;~  $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_ToolBarControlTypeId, $treescope_subtree)
$oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,"name:=Google Chrome Toolbar" , $treescope_subtree)
;~  _UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)

;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
$oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29

;~  _UIA_DumpThemAll($oChromeToolbar,$treescope_children)
$oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)

;~  get the value of the addressbar
$myText=""
$oValueP.CurrentValue($myText)
consolewrite("address: " & $myText & @CRLF)

;~ Click all tabs
$oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
;~  _UIA_DumpThemall($oChromeTabs,$treescope_children)


;~  Get result with findall function alternative could be the treewalker
    dim $pCondition, $pTrueCondition
dim $pElements, $iLength

    $objUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition=ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition,$dtagIUIAutomationCondition)

    $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterFace($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
    $oAutomationElementArray.GetElement($i, $pUIElement)
        $oUIElement = ObjCreateInterface($pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        consolewrite( "Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= <" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB & @CRLF)
;~  Invoke or select them all
;~ Tricky as chrome seems to say it supports certain patterns but then they seem not to be implemented


;~ only tabs with content
if _UIA_getPropertyValue($oUIElement, $UIA_IsSelectionItemPatternAvailablePropertyId) = "True" Then
_UIA_action($oUIElement,"leftclick")
;~  $oSelectP=_UIA_getpattern($oUIElement,$UIA_SelectionItemPatternId)
;~  $oSelectP.Select()
EndIf
Next


;~ Lets open a new tab within chrome
$oChromeNewTab=_UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
_UIA_action($oUIElement,"leftclick")

;~ Lets get the valuepattern of the addressbar
;~  $oLegacyP=_UIA_getpattern($oChromeAddressBar,$UIA_LegacyIAccessiblePatternId)
;~

_UIA_action($oUIElement,"leftclick")
_UIA_action($oUIElement,"setvalue using keys","www.autoitscript.com/{ENTER}")

EndIf


exit
Edited by junkew

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

I can't run that last example. Apparently some function definition is missing.

Is there anything special I need to do except having that script together with two includes in the same folder?

Edited by trancexx

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

@TranceXX: Just uploaded latest UIAWrappers.AU3 maybe I missed to do that. If you still get an error let me know
 

Special things that sometimes is needed (hopefully I can manage this in the UIAWrappers.AU3 later)

  • Open once chrome is started the url chrome://accessibility/ to see if accessibility is on
  • Chrome version 29 as Google is still changing classnames, names of properties etc. its not allways the same to find an object
    A terrible workaround is needed for html input text boxes as their name is determined by the value (seems to be specified that way by W3C) and not on the name or id attribute (only picks attribut aria-label when its available)
  • Start chrome manually or remove commented run statement for starting chrome (not handling multiple instances of chrome at the moment)
  • OS Language can influence stuff (I am on a dutch OS so sometimes names have to be set in english so then I sometimes switch to find by one of the other properties)
  • If stuff is not working use the findThemAll(<handle to object>, <treescope>) it will write the childs with the most common properties to the output window
    Be aware on treescope_subtree as if it walks to the leafs of the hierarchy it can take minutes especially if multiple HTML windows are open
$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
findThemAll($oDesktop,$treescope_children)

 

Below some background for TODO later

Chrome supports following interfaces under Windows:
  • MSAA/IAccessible (complete)
  • IAccessible2 (mostly complete)
  • ISimpleDOM (mostly complete)
  • IAccessibleEx and UI Automation (very limited)

 

@TranceXX: I should be able to get to the IAccessible2 interface by making a call to QueryInterface as Chrome supports below interfaces but I am a little less experienced in this interface/com stuff so maybe you can help out

accProbe will show what IAccessible2 can reveal out of an html page (first register the needed proxy dll)

a. IAccessible2.IDL needs to be converted to Autoit syntax

b. This should be written in AutoIT syntax
 

HWND hw = GetForegroundWindow();
    IAccessible *pIA;
    HRESULT hr = AccessibleObjectFromWindow(hw, OBJID_CLIENT, IID_IAccessible, (void**)&pIA);
    if (!SUCCEEDED(hr))
        return -1;

    const IID IID_IAccessible2 = {0xE89F726E, 0xC4F4, 0x4c19, 0xbb, 0x19, 0xb6, 0x47, 0xd7, 0xfa, 0x84, 0x78};
    ::IServiceProvider *pService = NULL; 
    hr = pIA->QueryInterface(IID_IServiceProvider, (void **)&pService); 
    if(SUCCEEDED(hr))
    {
        IAccessible2 *pIA2 = NULL; 
        hr = pService->QueryService(IID_IAccessible2, IID_IAccessible2, (void**)&pIA2); 
        if (SUCCEEDED(hr) && pIA2)
        { 
            pIA2->Release();
        } 
        pService->Release();
    }  

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

That code written in AutoIt would be something like this (interfaces definitions missing):

Global Const $hOLEACC_DLL = DllOpen("oleacc.dll")
Global Const $OBJID_CLIENT = 0xFFFFFFFC

Func YourCodeWrittenInAutoIt($hWindow)
    Local $tIID_IAccessible = _WinAPI_GUIDFromString($sIID_IAccessible)
    Local $pIAccessible = AccessibleObjectFromWindow($hWindow, $OBJID_CLIENT, $tIID_IAccessible)
    ; Object from pointer
    Local $oIAccessible = ObjCreateInterface($pIAccessible, $sIID_IAccessible, $tagIAccessible)
    If @error Then Return -1

    Local $pIServiceProvider
    $oIAccessible.QueryInterface($sIID_IServiceProvider, $pIServiceProvider)
    ; Again object from pointer
    Local $oIServiceProvider = ObjCreateInterface($pIServiceProvider, $sIID_IServiceProvider, $tagIServiceProvider)
    If @error Then Return -2
    
    Local $pIA2
    $oIServiceProvider.QueryService($IID_IAccessible2, $IID_IAccessible2, $pIA2)
    ; And again creating object from pointer
    Local $oIA2 = ObjCreateInterface($pIA2, $sIID_IAccessible2, $tagIAccessible2)
    If @error Then Return -3
    
    ;...
EndFunc

Func AccessibleObjectFromWindow($hWindow, $iObjId, $tRefIID)
    Local $aCall = DllCall($hOLEACC_DLL, "long", "AccessibleObjectFromWindow", "hwnd", $hWindow, "dword", $iObjId, "struct*", $tRefIID, "ptr*", Null)
    If @error Or $aCall[0] < 0 Then Return SetError(1, 0, 0)
    Return $aCall[4]
EndFunc
If you look closely at that (and your) code you will see call to oIAccessible.QueryInterface(...). That probably means that you can skip one step and specify $tIID_IServiceProvider directly in earlier call to AccessibleObjectFromWindow to get $pIServiceProvider pointer directly. Edited by trancexx

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

From the uiautomation framework I can get to $oIaccessible by doing below code

The AccessibleObjectFromWindow is the MSAA way of getting the interface but thx again for the example as this will help later on getting to IAccessible2 or ISimpleDom

$oLegacyP=_UIA_getpattern($oUIElement,$UIA_LegacyIAccessiblePatternId)
    $oIAccessible = $oLegacyP.GetIAccessible()

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

Uodate UIAWrappers.AU3

Made a lot more comments and failure to show when an object is not retrieved/found the hierarchy of the tree is written to the console

Lots of optimizations could be done by using the cachefunctions (less interprocess communication) but on my machine it runs at a very acceptable speed

In top of script put the text in your local language of the operating system

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"
 
#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
 
;~ Make this language specific
const $cToolbarByName = "name:=Google Chrome Toolbar"
const $cAddressBarByName = "name:=Adres- en zoekbalk"
const $cChromeNewTabByName="name:=Nieuw tabblad"
 
$strChromeExeFolder=@UserProfileDir & "\AppData\Local\Google\Chrome\Application\"
$strChromeStartup="--force-renderer-accessibility"
$strChromeExe=$strChromeExeFolder & "chrome.exe "
 
;~ Start chrome
if fileexists($strChromeExe) Then
    if not processexists("chrome.exe") Then
        run($strChromeExe & $strChromeStartup,"", @SW_MAXIMIZE )
        ProcessWait("chrome.exe")
        ;~ Just to give some time to start
        sleep(10000)
    endif
Else
    consolewrite("No clue where to find chrome on your system, please start manually:" & @CRLF )
    consolewrite($strChromeExe & $strChromeStartup & @CRLF)
EndIf
 
;~ Find the chrome window
$oChrome=_UIA_getFirstObjectOfElement($oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)
if not isobj($oChrome) Then
    _UIA_DumpThemAll($oDesktop,$treescope_children)
EndIf
 
;~ Make sure chrome is front window
$oChrome.setfocus()
 
if isobj($oChrome) Then
    consolewrite("Action 1" & @CRLF)
 
;~  get the chrome toolbar
;~  $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_ToolBarControlTypeId, $treescope_subtree)
    $oChromeToolbar=_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)
    if not isobj($oChromeToolbar) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
 
consolewrite("Action 2" & @CRLF)
;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29
    $oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
    if not isobj($oChromeAddressbar) Then
        _UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
    EndIf
 
;~  $oValueP=_UIA_getpattern($oChromeAddressBar,$UIA_ValuePatternId)
;~  sleep(2000)
 
;~  get the value of the addressbar
;~  $myText=""
;~  $oValueP.CurrentValue($myText)
;~  consolewrite("address: " & $myText & @CRLF)
 
consolewrite("Action 3" & @CRLF)
;~ Get reference to the tabs
    $oChromeTabs=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_TabControlTypeId, $treescope_subtree)
    if not isobj($oChromeTabs) Then
        _UIA_DumpThemAll($oChrome,$treescope_subtree)
    EndIf
 
;~ Lets open a new tab within chrome
 
consolewrite("Action 4" & @CRLF)
;~  $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
    $oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
    if not isobj($oChromeNewTab) Then
        _UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
    EndIf
    _UIA_action($oChromeNewtab,"leftclick")
    sleep(500)
 
consolewrite("Action 4a" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName  , $treescope_subtree) ;works in chrome 29
 
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
 
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
 
 
consolewrite("Action 4b" & @CRLF)
;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
 
sleep(500)
 
consolewrite("Action 4c retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=On", $treescope_subtree)
if not isobj($oForumLink) Then
consolewrite("*** Scripting will fail as accessibility is off ****")
MsgBox(4096, "Accessibility warning", "Accessibility is turned off, put it on by clicking on Off after Global accessibility mode", 10)
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
 
consolewrite("Action 4d" & @CRLF)
;~ $oChromeNewTab= _UIA_getFirstObjectOfElement($oChromeTabs,"controltype:=" & $UIA_ButtonControlTypeId, $treescope_subtree)
$oChromeNewTab= _UIA_getObjectByFindAll($oChromeTabs, $cChromeNewTabByName,$treescope_subtree)
if not isobj($oChromeNewTab) Then
_UIA_DumpThemAll($oChromeTabs,$treescope_subtree)
EndIf
_UIA_action($oChromeNewtab,"leftclick")
sleep(500)
consolewrite("Action 5" & @CRLF)
$oChromeAddressBar=_UIA_getObjectByFindAll($oChromeToolbar, $cAddressBarByName , $treescope_subtree) ;works in chrome 29
if not isobj($oChromeAddressbar) Then
_UIA_DumpThemAll($oChromeToolbar,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oChromeAddressBar, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
_UIA_action($oChromeAddressBar,"leftclick")
_UIA_action($oChromeAddressBar,"setvalue using keys","www.autoitscript.com/{ENTER}")
consolewrite("Action 6" & @CRLF) ;~ give some time to open website
sleep(2000)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf
sleep(500)
consolewrite("Action 7 retrieve document after clicking a hyperlink" & @CRLF)
$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)
if not isobj($oForumLink) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oForumLink,"invoke")
sleep(3000)
consolewrite("Action 8 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
 
if not isobj($oDocument) Then
_UIA_DumpThemAll($oChrome,$treescope_subtree)
Else
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf ;~ Now we get the searchfield
$oEdtSearchForum=_UIA_getObjectByFindAll($oDocument,"name:=Search...", $treescope_subtree)
if not isobj($oEdtSearchForum) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
_UIA_action($oEdtSearchForum,"focus")
_UIA_action($oEdtSearchForum,"setvalue using keys","Chrome can easy be automated with ui automation") ; {ENTER}")
sleep(500)
;~ Exit
;~ Now we press the button, see relative syntax used as the button seems not to have a name its just 2 objects further then search field
$oBtnSearch=_UIA_getObjectByFindAll($oDocument,"name:=Search...; indexrelative:=2", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$t=stringsplit(_UIA_getPropertyValue($oDocument, $UIA_BoundingRectanglePropertyId),";")
_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
sleep(1000)
_UIA_action($oBtnSearch,"invoke")
sleep(2000)
;~ consolewrite("Action 9 first refresh the document control" & @CRLF)
$oDocument=_UIA_getFirstObjectOfElement($oChrome,"controltype:=" & $UIA_DocumentControlTypeId , $treescope_subtree)
if not isobj($oDocument) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
$oHyperlink=_UIA_getObjectByFindAll($oDocument,"name:=controlsend doesn't work", $treescope_subtree)
if not isobj($oBtnSearch) Then
_UIA_DumpThemAll($oDocument,$treescope_subtree)
EndIf
sleep(1000)
_UIA_action($oHyperlink,"invoke")
sleep(2000)
EndIf
exit
 
 
 



 

edit 1 sept 2013: added steps 4a,b,c,d to warn for accessibility is off

Edited by junkew
1 person likes this

Share this post


Link to post
Share on other sites

Capitally!

I can't run that last example. Apparently some function definition is missing.
Is there anything special I need to do except having that script together with two includes in the same folder?

Yes! This two includes in one line is typo.

I have Example5.working nice after replacement line with $strChromeExeFolder:

$strChromeExeFolder= "Your chrome EXE folder with \"

 

Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox

 

Example6 has problem, though. It can't create $oChromeToolbar as object in line #48, Here:

    $oChromeToolbar =_UIA_getFirstObjectOfElement($oChrome,$cToolbarByName, $treescope_subtree)

Is it concerning with my chrome language ("ru") and invalid names (Russian) of const $cToolbarByName, $cAddressBarByName, and $cChromeNewTabByName?


The point of world view

Share this post


Link to post
Share on other sites

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

Share this post


Link to post
Share on other sites

yes if you change the constants it should start to work

most likely its the language of chrome (not sure if you can put it on Dutch language as thats the Chrome I have)

You could dump all objects

* Close everything except SCITE and Chrome

Then with

findThemAll($oDesktop,$treescope_subtree)

you can dump the whole hierarchy to the output window (but it will show a lot of lines)

Thank you.

It works until

$oForumLink=_UIA_getObjectByFindAll($oDocument,"name:=Forum", $treescope_subtree)

So script is true before Action 7 and reports the following:

Action 7 retrieve document after clicking a hyperlink
Title is: <>    Class   := <Chrome_RenderWidgetHostHWND>    controltype:= <50030>   
"D:\Program Files\AutoIt3\Examples\IUIAutomation\UIAWrappers.au3" (424) : ==> Variable must be of type "Object".:
$obj.getCurrentPattern($PatternId, $pPattern)
$obj^ ERROR

The point of world view

Share this post


Link to post
Share on other sites

Hi

I see here some possibility for Chrome, something richer than run, winwaitactive, send("login tab password enter"). Thanks a lot. I'll experiment with it somewhen my brain's not melting from a week's work, next monday.

I have a question though: does this bring possibility for Java application?

I'm thinking about Oracle Sql Developer here. In both case (chrome and sqldev), my scenario is KeePass url/login/password/etc entry, double clic => [autoit connector] => Application running and connected. Regarding Sql Developer, AU3Info just tell me it's a SunAwtFrame, and except sending tab or clic on coordinate we can't create a new connection.

Regards,

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • bobmcrae
      By bobmcrae
      I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?
      #include "UIAWrappers.au3" _UIA_setVar("Global.Debug", False) _UIA_setVar("Global.Debug.File", False) _UIA_setVar("Global.Highlight", False) _UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window") _UIA_action('DPN','setFocus') _UIA_setVar("DPN.firstName","AutomationId:=txtFirstName") _UIA_setVar("DPN.lastName", "AutomationId:=txtLastName") _UIA_Action('DPN.lastName','setvalue','last name') _UIA_setVar("DPN.ID", "AutomationId:=txtSubjectId") _UIA_setVar("DPN.DOB", "AutomationId:=PART_TextBox") _UIA_setVar("DPN.Ft", "AutomationId:=txtSubjectHeight") _UIA_setVar("DPN.In", "AutomationId:=txtSubjectHeight2") _UIA_Action('DPN.firstName','setvalue','first name') _UIA_Action('DPN.ID','setvalue','ID012345') _UIA_Action('DPN.DOB','setvalue','1/31/1932') _UIA_Action('DPN.Ft','setvalue','6') _UIA_Action('DPN.In','setvalue','1') SimpleSpy output:
      ;~ *** Standard code *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children) Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children) Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) ;~ First find the object in the parent before you can do something ;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) _UIA_action($oUIElement,"click")  
    • jackchen
      By jackchen
      OK,I've made my "Chrome_KeepLastTab.au3" work.This script adds some features to Chrome browser:
      1. Double click on a tab to close the tab.
      2. Keep last tab:This script monitors your mouse clicks and hotkeys,if you are about to close the last tab within Chrome(click close button on the last tab, middle click/double click on the last tab or press Ctrl + w or Ctrl + {F4}), a new tab will be open and then the old tab be closed.
      #include <WindowsConstants.au3> #include <WinAPI.au3> #include "CUIAutomation2.au3" #AutoIt3Wrapper_UseX64=Y ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os ConsoleWrite("@OSArch: " & @OSArch & ", @AutoItX64: " & @AutoItX64 & @CRLF) Global $DoubleClickTime = 500 Global $UIA_oUIAutomation ; The main library core CUI automation reference Global $hMouseEvent, $hMouseHook Global $aMouseEvent[2] Global $KeepLastTab = True ; settings from ini file If $KeepLastTab Then ;The main object with acces to the windows automation api 3.0 $UIA_oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation) If IsObj($UIA_oUIAutomation) Then HotKeySet("^w", "Hotkey_Event") ; Ctrl + w to close tab HotKeySet("^{F4}", "Hotkey_Event") ; Ctrl + {F4} to close tab $DoubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")[0] OnAutoItExitRegister("UnhookMouse") ; Register mouse events callback $hMouseEvent = DllCallbackRegister("Mouse_Event", "int", "int;ptr;ptr") $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) EndIf EndIf While 1 Sleep(100) WEnd ToolTip("") ; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/ Func Mouse_Event($nCode, $wParam, $lParam) Local $info, $mouseData, $time, $timeDiff If $nCode < 0 Then Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf $tPoint = _WinAPI_GetMousePos() $hWnd = _WinAPI_WindowFromPoint($tPoint) ; if mouse is on the widget window(class: Chrome_RenderWidgetHostHWND), ; use $hWnd = _WinAPI_GetParent($hWnd) to get the parent Chrome window If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_") Then ; Chrome_WidgetWin_1: Chrome window ; Chrome_WidgetWin_2: Chrome menu ; ignore non Chrome window Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) EndIf ToolTip($hWnd & " - " & _WinAPI_GetClassName($hWnd)) ;$tagPOINT = "struct;long X;long Y;endstruct" Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo" $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam) $mouseData = DllStructGetData($info, 3) $time = DllStructGetData($info, 5) $timeDiff = $time - $aMouseEvent[1] Local $block Switch $wParam Case $WM_LBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "LClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "LDClick" Else $aMouseEvent[0] = "LClick" EndIf $block = KeepLastTab($hWnd, $aMouseEvent[0]) Case $WM_MBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "MClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "MDClick" Else $aMouseEvent[0] = "MClick" $block = KeepLastTab($hWnd, $aMouseEvent[0]) EndIf EndSwitch If Not $block Then _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf EndFunc ;==>Mouse_Event Func UnhookMouse() _WinAPI_UnhookWindowsHookEx($hMouseHook) $hMouseHook = 0 DllCallbackFree($hMouseEvent) $hMouseEvent = 0 EndFunc ;==>UnhookMouse Func Hotkey_Event() Local $block Local $hWnd = WinGetHandle(WinGetTitle("[ACTIVE]")) If StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_1") Then ; Chrome_WidgetWin_1, Chrome window $block = KeepLastTab($hWnd, "Hotkey") EndIf If Not $block Then HotKeySet("^w") Send(@HotKeyPressed) HotKeySet("^w", "Hotkey_Event") EndIf EndFunc Func KeepLastTab($hWnd, $action = "LClick") ; Possible $action value: LClick, LDClick, MClick, Hotkey ConsoleWrite(@CRLF & $action & " on a Chrome window: " & $hWnd & @CRLF) Local $aMousePos = MouseGetPos() Local $pChrome, $oChrome $UIA_oUIAutomation.ElementFromHandle($hWnd, $pChrome) ; Window $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oChrome) Then ConsoleWrite("Failed to get Chrome object from hWnd." & @CRLF) Return EndIf ;$UIA_ControlTypePropertyId = 30003 $oChromeTabs = UIA_getFirstElement($oChrome, $UIA_ControlTypePropertyId, $UIA_TabControlTypeId, $treescope_subtree) If Not IsObj($oChromeTabs) Then ConsoleWrite("Failed to get Chrome tab bar object." & @CRLF) Return EndIf Local $t $oChromeTabs.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1]) Then ; Mouse not on tabs bar ConsoleWrite("Mouse is not on the tab bar. Ignore and return..." & @CRLF) Return EndIf Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition) $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition) If Not IsObj($oCondition) Then Return $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements) $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray) If Not IsObj($oAutomationElementArray) Then ConsoleWrite("Failed to find all Chrome tabs. " & @CRLF) Return EndIf $oAutomationElementArray.Length($iLength) Local $UIA_pUIElement, $oTab2 Local $iTabs = $iLength - 1 If $iTabs > 1 Then ; more than one tab If $action = "LDClick" Then ConsoleWrite("There are " & $iTabs & " tabs within Chrome window. " & @CRLF) For $i = 1 To $iTabs $oAutomationElementArray.GetElement($i, $UIA_pUIElement) $oTab2 = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab2) Then ContinueLoop $oTab2.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then ContinueLoop If $aMousePos[0] >= $t[0] And $aMousePos[0] <= $t[2]+$t[0] And $aMousePos[1] >= $t[1] And $aMousePos[1] <= $t[3]+$t[1] Then ConsoleWrite("You double clicked on one of " & $iTabs & " tabs. Close the tab and return..." & @CRLF) HotKeySet("^w") Send("^w") HotKeySet("^w", "Hotkey_Event") Return True EndIf Next EndIf Return EndIf ConsoleWrite("There is ONLY one tab within Chrome window. " & @CRLF) $oAutomationElementArray.GetElement(1, $UIA_pUIElement) $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab) Then ConsoleWrite("Failed to get the last tab object." & @CRLF) Return EndIf Local $rtTab $oTab.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $rtTab) If UBound($rtTab) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $rtTab[0] Or $aMousePos[0] > $rtTab[2]+$rtTab[0] Or $aMousePos[1] < $rtTab[1] Or $aMousePos[1] > $rtTab[3]+$rtTab[1]) Then ; Mouse not on the last tab ConsoleWrite("Mouse is not on the last tab. Ignore and return..." & @CRLF) Return EndIf If $action = "LClick" Then $oTabClose = UIA_getFirstElement($oTab, $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $treescope_subtree) If Not IsObj($oTabClose) Then ConsoleWrite("Failed to get the last tab close object." & @CRLF) Return EndIf $oTabClose.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1] Then ; Mouse not on the tab close button Return EndIf EndIf ; open a new tab within chrome ConsoleWrite("The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing." & @CRLF) Send("^t") ConsoleWrite("A new tab created!" & @CRLF) Sleep(100) ConsoleWrite("Close the old tab and return..." & @CRLF) ;~ Local $pPattern ;~ $oTabClose.GetCurrentPattern($UIA_InvokePatternId, $pPattern) ;~ $oPattern = ObjCreateInterface($pPattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern) ;~ If IsObj($oPattern) Then ;~ ConsoleWrite("Invoke to close the tab..." & @CRLF) ;~ $oTabClose.SetFocus() ;~ $oPattern.Invoke() ;~ EndIf _WinAPI_UnhookWindowsHookEx($hMouseHook) $aMousePos = MouseGetPos() If $aMousePos[0] >= $rtTab[0] And $aMousePos[0] <= $rtTab[2]+$rtTab[0] And $aMousePos[1] >= $rtTab[1] And $aMousePos[1] <= $rtTab[3]+$rtTab[1] Then MouseClick("middle", $aMousePos[0], $aMousePos[1], 1, 0) Else MouseClick("middle", $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2, 1, 0) ; close the tab MouseMove($aMousePos[0], $aMousePos[1], 0) ; move mouse back to previous position EndIf ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEDOWN, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) ;~ Sleep(100) ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEUP, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) Return True ; to block mouse click/hot key EndFunc Func UIA_getFirstElement($obj, $propertyID, $tval, $treeScope) Local $pCondition, $oCondition $UIA_oUIAutomation.CreatePropertyCondition($propertyID, $tval, $pCondition) $oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition) Local $UIA_oUIElement, $UIA_pUIElement $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement) $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If IsObj($UIA_oUIElement) Then Return $UIA_oUIElement Return SetError(1, 0, "") EndFunc ;==>UIA_getFirstElement My question is related to #AutoIt3Wrapper_UseX64 when run this script on 64-bit Win 7.
      No mater I set #AutoIt3Wrapper_UseX64=Y or #AutoIt3Wrapper_UseX64=N, this script works very well on hotkey event, while mouse clicks sometimes works if #AutoIt3Wrapper_UseX64=N and sometimes works on Y. Can some one test this and finger out what's wrong?
      Info from SciTe if #AutoIt3Wrapper_UseX64=Y :
      @OSArch: X64, @AutoItX64: 1 Hotkey on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... Info if #AutoIt3Wrapper_UseX64=N or comment out this line:
      @OSArch: X64, @AutoItX64: 0 Hotkey on a Chrome window: 0x00140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x00140330 Failed to get Chrome tab bar object. Hotkey events and mouse click events share the same function KeepLastTab($hWnd, $action = "LClick"),Why this function triggered by hotkey works on both 32-bit and 64-bit while that triggered by mouse events failed on 32-bit autoit?