bobmcrae

Is there a way to specify a UIAutomation control in a way that yields faster actions?

8 posts in this topic

I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?

#include "UIAWrappers.au3"

_UIA_setVar("Global.Debug", False)
_UIA_setVar("Global.Debug.File", False)
_UIA_setVar("Global.Highlight", False)


_UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window")
_UIA_action('DPN','setFocus')

_UIA_setVar("DPN.firstName","AutomationId:=txtFirstName")
_UIA_setVar("DPN.lastName", "AutomationId:=txtLastName")
_UIA_Action('DPN.lastName','setvalue','last name')
_UIA_setVar("DPN.ID",       "AutomationId:=txtSubjectId")
_UIA_setVar("DPN.DOB",      "AutomationId:=PART_TextBox")
_UIA_setVar("DPN.Ft",       "AutomationId:=txtSubjectHeight")
_UIA_setVar("DPN.In",       "AutomationId:=txtSubjectHeight2")

_UIA_Action('DPN.firstName','setvalue','first name')
_UIA_Action('DPN.ID','setvalue','ID012345')
_UIA_Action('DPN.DOB','setvalue','1/31/1932')
_UIA_Action('DPN.Ft','setvalue','6')
_UIA_Action('DPN.In','setvalue','1')

SimpleSpy output:

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children)    
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children)    
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"click")

 

Share this post


Link to post
Share on other sites



The advantage of using the wrapper functions is that they are easy to use. The disadvantage is that they execute a large number of code lines. Especially _UIA_Action. The only thing you can do to increase performance is to avoid the wrapper functions and call the interface methods in CUIAutomation2.au3 directly.

You can look at some of my examples in the UI Automation thread to see how to do it.

Share this post


Link to post
Share on other sites

Next version I hope to have better speed of the uiaWrappers but I am not sure If I cause 2 seconds delay.

I know the logic in the wrappers is also if no treehierarchy is specified fallback to desktop and start searching the whole tree which can be huge and timeconsuming (when you turn logging on you can see what happens in detail and how many objects are traversed before finding the element)

Func _UIA_getFirstObjectOfElement($obj, $str, $treeScope) is probably quicker as it does not do a findall

I will put on my TODO some logic like

if in UIA_Action 1 property is used for finding an element directly call directly something like below

;- Change Func _UIA_getFirstObjectOfElement

$propertyID = $UIA_NamePropertyId ;- Or automation id property or .....
$tVal= ..... ;- property value

$UIA_oUIAutomation.createPropertyCondition($propertyID, $tval, $pCondition)
$oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition)
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

 

Share this post


Link to post
Share on other sites

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

Share this post


Link to post
Share on other sites

Findfirst is for sure match quicker and in your scenario your developers behave nicely on automationid.

The uiawrappers give 

A. Regular expression matching

B. Matching on index or ordinal value

C. Match with multiple properties

As said I will improve on single property and exact matching using findfirst

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Nice topic!

First i will learn, then i will ask forsome help later on :P

Really amazing job @junkew

Thanks adv.

@UPDATE:

On 19/02/2016 at 3:06 AM, bobmcrae said:

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

I'm getting:

Unknown function name.:
$aeParent = _getAeFromHandle($hWindow)
$aeParent = ^ ERROR

Some idea why?

Thanks adv.

Edited by Chiitus

Share this post


Link to post
Share on other sites

@Chiitus, it looks like perhaps you did not copy the entire  UIAutomation.au3 UDF, as the function _getAeFromHandle is at the bottom of the code above...take another look.;)

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

how did you find this

'txtUserName'

(which way did you read it?)

and why did you use it together with

$UIA_AutomationIdPropertyId

Can you also give me direct link to this application

Sunlight MiniOmni

then I can download it, use "simple spy" and try to understand your code to learn UIA

Edited by maniootek

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • careca
      By careca
      Hey there, i have the code from simplespy but have no idea about what to use, got the UIA_V0_64 files.
      I look at the examples but i get lost, i ask for guideance in this matter.
      Best regards.

       
      To get to this window simply run:
      control.exe /name Microsoft.DefaultPrograms /page pageFileAssoc
      select a filetype and click "change program"
    • junkew
      By junkew
      Introduction
      Quickstart - Getting started quickly
      Simple scripts
      With this module
      you can automate all applications/programs that support ui automation and/or accesibility api from microsoft you can recognize more controls than AutoIT can recognize "out of the box"  you can use concepts from other testing frameworks like http://download.freedesktop.org/ldtp/doc/ldtp-tutorial.pdf
      http://safsdev.sourceforge.net/Default.htm
      coded ui testing from microsoft 
      Some of those controls / applications are
      chrome browser (partly mainwindow has to be done with MSAA for navigating) chrome://accessibility in the adress bar of chrome or start with "--force-renderer-accessibility"
      silverlight controls Ribbon control controlbars of Excel/Word IE and FF browsers Windows Media Player Windows clock AFX .. controls (partly) ....  
      Based on the initial AIO Object I now have made the interface file to work with objCreateInterface function which is in the latest beta's
      automate clicking and querying basic information
      It gives you a lot of basic information to be able to automate clicking, querying basic information where it goes further in certain situations than AutoIt is identifying
      Starting threads for background on the ui automation api of microsoft (not for starters)
      http://en.wikipedia.org/wiki/Microsoft_UI_Automation http://msdn.microsoft.com/en-us/library/ms747327.aspx Previous threads in general help/support Interface AutoItObject IUIAutomation ObjCreateInterface and struct tagPoint in method ElementFromPoint  
      Be aware that API is not allways installed under XP/Vista see http://support.microsoft.com/kb/971513 Within Windows 7 and Windows 8 it should be preinstalled by default. Be aware on 32 and 64 bits way of running your script
      #AutoIt3Wrapper_UseX64=Y or N
       
      Basic example of usage / showing and retrieving the default information, will post multiple examples later
      Hover your mouse to an area of interest and press ctrl+w and information will be shown in the edit box of the form
       
       
      Simple spy demo (see simplespy.au3 or use latest ZIP  attachment for latest version)
       
      Main features
      Recognize windows and html controls for the major browsers Logical and physical description for controls (UI mapping, Application map) Simple repository logic to abstract logical and physical descriptions Store Runtime Type Information in RTI. variables Rubberbanding/highlighting of objects Simple spy to help in making / identifying the physical description Support of regular expression(s) in identifying objects recognize objects on multiple properties supported properties: name ,title, automationid, classname, class, iaccessiblevalue, iaccessiblechildId, controltype, processid, acceleratorkey
       The actions provided so far
       "leftclick", "left", "click", "leftdoubleclick", "leftdouble", "doubleclick", _
       "rightclick", "right", "rightdoubleclick", "rightdouble", _
       "middleclick", "middle", "middledoubleclick", "middledouble", "mousemove", "movemouse"
       "setvalue","settextvalue"
       "setvalue using keys"
       "setValue using clipboard"
       "getvalue"
       "sendkeys", "enterstring", "type", "typetext"
       "invoke"
       "focus", "setfocus", "activate"
       "close"
       "move","setposition"
       "resize"
       "minimize", "maximize", "normal", "close", "exist", "exists"
       "searchcontext", "context"
       "highlight"
       "getobject","object"
       "attach"
       "capture","screenshot", "takescreenshot"
       "dump", "dumpthemall"
       "propertyvalue", "property"
       match on multiple properties like:  name:=((Zoeken.*)|(Find.*)); ControlType:=Button; acceleratorkey:=Ctrl+F
      Support for 117 different properties see $UIA_propertiesSupportedArray in uiawrappers like for example title, regexptitle, class, regexpclass, iaccessiblevalue, iaccessiblechildid, name, accesskey, automationid, classname 
      IAccessible, IAccessible2, ISimpleDom interfaces  debuglogging to a file log.txt (no output in scitewindow) Examples
      Example 1 Iterating thru the different ways of representing the objects in the tree (#comment-1105548) Example 2 Finding the taskbar and clicking on the start menu button (#comment-1105680) Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info) (#comment-1108849) Example 4 that demonstrates the calculator Example 5 Automating chrome Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox Example 7 The chrome example modified to a firefox example Example 8 The other major browser Internet Explorer automated (made on Example 6 and 7) Example 9 Windows media player Example 10 Automating mach 3 (AFX windows and other hard to get recognized by AutoIT) Lot of links are broken due to forum upgrade just search for the text like "Example 11 Demonstrate Word, Notepad and Calculator actions"
      Example 11 Demonstrate Word, Notepad and Calculator actions ... Example 13 Details 1 about the right pane of the windows explorer Example 14 Details 2 about the right pane of the windows explorer Example 15 Details 3 about the right pane of the windows explorer Example 16 Details 4 about the right pane of the windows explorer Example 17 Details 5 about the right pane of the windows explorer WITH CACHING Example 18 Details 6 about the right pane of the windows explorer WITH VIRTUAL ITEMS Example 19 Eventhandling examples Example 20 Eventhandling examples Example 21a Eventhandling examples Internet Explorer Example 21b Eventhandling examples Internet Explorer Example 22 Eventhandling examples Follow focus Example 23 Eventhandling examples structure changed Example 24 Eventhandling examples IUIAutomationEventHandler Example 25 SAFEARRAYS Example 26 IACCESSIBLE / MSAA Example 27 IACCESSIBLE2 / MSAA Example 28 IACCESSIBLE / MSAA events Example 29 IACCESSIBLE2 events Example 30 ISimpleDOM Example 31 Notepad window move, maximize, minimize Example 32 Three browsers doing the same stuff with small differences in scripting only ..
      TODO Build recorder Enhance the spy with a nicer UI UI for the repository (now in the script with dot notation) Enhance mapping / identifying on multiple properties instead of 1 combined with index If speed becomes an issue use the caching logic of the MS UIA framework Add the other patterns later Generalize the concept of System Under Test of starting the SUT (for testing framework purposes) Remote running of scripts Fix issue on finding within dynamic context  ... edit august 18th 2013  
      initial post Only zip files are needed to download , just unzip in 1 directory
      edit july 2016
      Made V0_63 and examples  works with AutoIt v3.3.14 Windows 10 tested Simple spy gives some basic code as a present Chrome latest versions seems to be having issues with IUIAutomation on tabs/buttons of mainwindow use MSAA for accessing tabsheets / buttons more cleanup to be in UDF style More comments in the source see changelog.txt for previous changes edit september 2017
      All examples fixed for the IE, Firefox and Chrome browser Some small but essential fixes in UIAWrappers UIA_V0_4.zip
      EXAMPLES.zip
      EXAMPLES_V0_5.zip
      UIA_V0_51.zip
      UIA_V0_63.zip
      EXAMPLES_V0_63.zip
      UIA_V0_64.zip
      EXAMPLES_V0_64.zip
    • SorryButImaNewbie
      By SorryButImaNewbie
      Hello,
      I'm developing again (everybody ruuun! )
      I would like to develope a script that goes through basicly every user control on a window, and log things that happens, and maybe do some screenshots. I did something like this before.
      My problem, which I would like to avoid this time (to improve my understanding and skill) , was that when I was unable to get a ControlID or handler or anything, I simply did some math and clicked on the coordinates it should have been (for example, maxing the window, and knowing the initial set up I was ablo to calculate given control position). I know that this is a bad solution for a number of reasons. 
      Now I got authorization to install autoIT here, and i started to the work, AU3Info was unable to find anything on the window (this could be a problem, since autoIT doesn't see anything on it then, if I understood the help file) So I got the SimpleSpy script (source:
      )
      I added a bit of code to the original to display ID as well, what I received is this:
      Mouse position is retrieved 115-207
      At least we have an element title: [ADD] class: [Button] ID: [50000] (<-- coded this to display ID here as well)
      Having the following values for all properties: 
      Title is: <ADD>    Class   := <Button>    controltype:= <UIA_ButtonControlTypeId>    ,<50000>    , (0000C350)    10;187;120;35
      *** Parent Information top down ***
      3: Title is: <Compass>    Class   := <Window>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    -8;-8;1936;1056
      "Title:=Compass;controltype:=UIA_WindowControlTypeId;class:=Window""    
      2: Title is: <>    Class   := <MainView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;23;1920;1017
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=MainView""    
      1: Title is: <>    Class   := <TileNavigationView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;23;1920;967
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=TileNavigationView""    
      0: Title is: <>    Class   := <AreasView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;132;1920;858
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=AreasView""
       
      so far I wrote this script:
      WinActivate('Test') ;It works!! :D first official interaction Sleep(1000) ;1 sec sleep to be sure ControlClick('Test', '', '50000') If @error Then MsgBox($MB_SYSTEMMODAL, 'Error', 'ControlClick error') EndIf Sleep(1000) MsgBox(1,"Tracer message", 'ControlClick has happened') ;MouseClick() ;ControlCommand() AutoIt activates the window, but the click on the given button doesnt happen (I tried to write 50000 without ' ' on ID).
      M'I doing the @error part correctly ? (no error Msg has been displayed), sorry I rarely use AutoIT and seems to forget less and less after each neglect, but still I'm far from a proffessional
       
      Any help or suggestion is welcome, thank you for your time and insight!
    • whanenbu
      By whanenbu
      Have to start learning UIAutomation since I need to use MS EDGE or Chrome.  I downloaded Examples_V0_5.zip and UIA_V0_51.zip.
      Compiled simplespy.au3 to get started.  Appears to work.  Did Ctrl+R and get error Line 15226 Error: Subscript used on non-accessible variable.  What am I doing wrong?
      Example 1 tried to run.  It was missing CUIAutomation2.au3.  Moved it into folder from UIA.zip to Examples and now works.
      Example 2 failed.  Added switch to case 0409 and "Start" and it worked.
      Example 3 failed. Can't fix.
      Code starting at Line 50. 
      Consolewrite("Get the menu that is after the start button" & @crlf)
      $oMenuStart=_UIA_getFirstObjectOfElement($UIA_oDesktop,"Start Menu", $treescope_children)  "I tried Start, Menu Start, and Start Menu, none seem to work." 
      if isobj($oMenuStart) Then
          consolewrite("Menu start found" & @crlf)
      Else
          consolewrite("I bet the text has to change to Start instead of Starten" & @crlf)
      EndIf
       Console:
      Get the menu that is after the start button
      I bet the text has to change to Start instead of Starten
      "C:\Users\xxxx\Autoit Scripts\UI Automation\EXAMPLES_V0_5\UIAWrappers.au3" (1726) : ==> Variable must be of type "Object".:
      $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
      $t = $obj^ ERROR
       
      Example 4 fails the same as in see below.
      Code starting at Line 30
      ;~ Start the calculator and notepad
      run("calc.exe")
      run("notepad.exe")
      $oCalc=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=" & $cCalcClassname, $treescope_children)
      $oNotepad=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:="& $cNotepadClassName, $treescope_children)
      if isobj($oCalc) Then
      ;~ You can comment this out just there to get the names of whats available under the calc window
      _UIA_DumpThemAll($oCalc,$treescope_subtree)
          $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $cButton1, $treescope_subtree)
          $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
          $oInvokeP.Invoke()
      ConsoleWrite("I am hee" )
          $oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $cButtonAdd, $treescope_subtree)
          $oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
          $oInvokeP.Invoke()  This is where it fails
      Console:
      Example text is for dutch calculator so please change text to english or other language to identify controls
      "C:\Users\xxxxxxxxxxx\Autoit Scripts\UI Automation\EXAMPLES_V0_5\ex4_calculator.au3" (44) : ==> Variable must be of type "Object".:
      $oInvokeP.Invoke()
      $oInvokeP^ ERROR
      There must me something I am missing on why these codes are not working. XML file below of example 4.
       
       
      20160404-232725402.XML
    • jackchen
      By jackchen
      OK,I've made my "Chrome_KeepLastTab.au3" work.This script adds some features to Chrome browser:
      1. Double click on a tab to close the tab.
      2. Keep last tab:This script monitors your mouse clicks and hotkeys,if you are about to close the last tab within Chrome(click close button on the last tab, middle click/double click on the last tab or press Ctrl + w or Ctrl + {F4}), a new tab will be open and then the old tab be closed.
      #include <WindowsConstants.au3> #include <WinAPI.au3> #include "CUIAutomation2.au3" #AutoIt3Wrapper_UseX64=Y ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os ConsoleWrite("@OSArch: " & @OSArch & ", @AutoItX64: " & @AutoItX64 & @CRLF) Global $DoubleClickTime = 500 Global $UIA_oUIAutomation ; The main library core CUI automation reference Global $hMouseEvent, $hMouseHook Global $aMouseEvent[2] Global $KeepLastTab = True ; settings from ini file If $KeepLastTab Then ;The main object with acces to the windows automation api 3.0 $UIA_oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation) If IsObj($UIA_oUIAutomation) Then HotKeySet("^w", "Hotkey_Event") ; Ctrl + w to close tab HotKeySet("^{F4}", "Hotkey_Event") ; Ctrl + {F4} to close tab $DoubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")[0] OnAutoItExitRegister("UnhookMouse") ; Register mouse events callback $hMouseEvent = DllCallbackRegister("Mouse_Event", "int", "int;ptr;ptr") $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) EndIf EndIf While 1 Sleep(100) WEnd ToolTip("") ; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/ Func Mouse_Event($nCode, $wParam, $lParam) Local $info, $mouseData, $time, $timeDiff If $nCode < 0 Then Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf $tPoint = _WinAPI_GetMousePos() $hWnd = _WinAPI_WindowFromPoint($tPoint) ; if mouse is on the widget window(class: Chrome_RenderWidgetHostHWND), ; use $hWnd = _WinAPI_GetParent($hWnd) to get the parent Chrome window If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_") Then ; Chrome_WidgetWin_1: Chrome window ; Chrome_WidgetWin_2: Chrome menu ; ignore non Chrome window Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) EndIf ToolTip($hWnd & " - " & _WinAPI_GetClassName($hWnd)) ;$tagPOINT = "struct;long X;long Y;endstruct" Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo" $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam) $mouseData = DllStructGetData($info, 3) $time = DllStructGetData($info, 5) $timeDiff = $time - $aMouseEvent[1] Local $block Switch $wParam Case $WM_LBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "LClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "LDClick" Else $aMouseEvent[0] = "LClick" EndIf $block = KeepLastTab($hWnd, $aMouseEvent[0]) Case $WM_MBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "MClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "MDClick" Else $aMouseEvent[0] = "MClick" $block = KeepLastTab($hWnd, $aMouseEvent[0]) EndIf EndSwitch If Not $block Then _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf EndFunc ;==>Mouse_Event Func UnhookMouse() _WinAPI_UnhookWindowsHookEx($hMouseHook) $hMouseHook = 0 DllCallbackFree($hMouseEvent) $hMouseEvent = 0 EndFunc ;==>UnhookMouse Func Hotkey_Event() Local $block Local $hWnd = WinGetHandle(WinGetTitle("[ACTIVE]")) If StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_1") Then ; Chrome_WidgetWin_1, Chrome window $block = KeepLastTab($hWnd, "Hotkey") EndIf If Not $block Then HotKeySet("^w") Send(@HotKeyPressed) HotKeySet("^w", "Hotkey_Event") EndIf EndFunc Func KeepLastTab($hWnd, $action = "LClick") ; Possible $action value: LClick, LDClick, MClick, Hotkey ConsoleWrite(@CRLF & $action & " on a Chrome window: " & $hWnd & @CRLF) Local $aMousePos = MouseGetPos() Local $pChrome, $oChrome $UIA_oUIAutomation.ElementFromHandle($hWnd, $pChrome) ; Window $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oChrome) Then ConsoleWrite("Failed to get Chrome object from hWnd." & @CRLF) Return EndIf ;$UIA_ControlTypePropertyId = 30003 $oChromeTabs = UIA_getFirstElement($oChrome, $UIA_ControlTypePropertyId, $UIA_TabControlTypeId, $treescope_subtree) If Not IsObj($oChromeTabs) Then ConsoleWrite("Failed to get Chrome tab bar object." & @CRLF) Return EndIf Local $t $oChromeTabs.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1]) Then ; Mouse not on tabs bar ConsoleWrite("Mouse is not on the tab bar. Ignore and return..." & @CRLF) Return EndIf Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition) $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition) If Not IsObj($oCondition) Then Return $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements) $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray) If Not IsObj($oAutomationElementArray) Then ConsoleWrite("Failed to find all Chrome tabs. " & @CRLF) Return EndIf $oAutomationElementArray.Length($iLength) Local $UIA_pUIElement, $oTab2 Local $iTabs = $iLength - 1 If $iTabs > 1 Then ; more than one tab If $action = "LDClick" Then ConsoleWrite("There are " & $iTabs & " tabs within Chrome window. " & @CRLF) For $i = 1 To $iTabs $oAutomationElementArray.GetElement($i, $UIA_pUIElement) $oTab2 = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab2) Then ContinueLoop $oTab2.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then ContinueLoop If $aMousePos[0] >= $t[0] And $aMousePos[0] <= $t[2]+$t[0] And $aMousePos[1] >= $t[1] And $aMousePos[1] <= $t[3]+$t[1] Then ConsoleWrite("You double clicked on one of " & $iTabs & " tabs. Close the tab and return..." & @CRLF) HotKeySet("^w") Send("^w") HotKeySet("^w", "Hotkey_Event") Return True EndIf Next EndIf Return EndIf ConsoleWrite("There is ONLY one tab within Chrome window. " & @CRLF) $oAutomationElementArray.GetElement(1, $UIA_pUIElement) $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab) Then ConsoleWrite("Failed to get the last tab object." & @CRLF) Return EndIf Local $rtTab $oTab.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $rtTab) If UBound($rtTab) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $rtTab[0] Or $aMousePos[0] > $rtTab[2]+$rtTab[0] Or $aMousePos[1] < $rtTab[1] Or $aMousePos[1] > $rtTab[3]+$rtTab[1]) Then ; Mouse not on the last tab ConsoleWrite("Mouse is not on the last tab. Ignore and return..." & @CRLF) Return EndIf If $action = "LClick" Then $oTabClose = UIA_getFirstElement($oTab, $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $treescope_subtree) If Not IsObj($oTabClose) Then ConsoleWrite("Failed to get the last tab close object." & @CRLF) Return EndIf $oTabClose.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1] Then ; Mouse not on the tab close button Return EndIf EndIf ; open a new tab within chrome ConsoleWrite("The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing." & @CRLF) Send("^t") ConsoleWrite("A new tab created!" & @CRLF) Sleep(100) ConsoleWrite("Close the old tab and return..." & @CRLF) ;~ Local $pPattern ;~ $oTabClose.GetCurrentPattern($UIA_InvokePatternId, $pPattern) ;~ $oPattern = ObjCreateInterface($pPattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern) ;~ If IsObj($oPattern) Then ;~ ConsoleWrite("Invoke to close the tab..." & @CRLF) ;~ $oTabClose.SetFocus() ;~ $oPattern.Invoke() ;~ EndIf _WinAPI_UnhookWindowsHookEx($hMouseHook) $aMousePos = MouseGetPos() If $aMousePos[0] >= $rtTab[0] And $aMousePos[0] <= $rtTab[2]+$rtTab[0] And $aMousePos[1] >= $rtTab[1] And $aMousePos[1] <= $rtTab[3]+$rtTab[1] Then MouseClick("middle", $aMousePos[0], $aMousePos[1], 1, 0) Else MouseClick("middle", $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2, 1, 0) ; close the tab MouseMove($aMousePos[0], $aMousePos[1], 0) ; move mouse back to previous position EndIf ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEDOWN, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) ;~ Sleep(100) ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEUP, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) Return True ; to block mouse click/hot key EndFunc Func UIA_getFirstElement($obj, $propertyID, $tval, $treeScope) Local $pCondition, $oCondition $UIA_oUIAutomation.CreatePropertyCondition($propertyID, $tval, $pCondition) $oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition) Local $UIA_oUIElement, $UIA_pUIElement $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement) $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If IsObj($UIA_oUIElement) Then Return $UIA_oUIElement Return SetError(1, 0, "") EndFunc ;==>UIA_getFirstElement My question is related to #AutoIt3Wrapper_UseX64 when run this script on 64-bit Win 7.
      No mater I set #AutoIt3Wrapper_UseX64=Y or #AutoIt3Wrapper_UseX64=N, this script works very well on hotkey event, while mouse clicks sometimes works if #AutoIt3Wrapper_UseX64=N and sometimes works on Y. Can some one test this and finger out what's wrong?
      Info from SciTe if #AutoIt3Wrapper_UseX64=Y :
      @OSArch: X64, @AutoItX64: 1 Hotkey on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... Info if #AutoIt3Wrapper_UseX64=N or comment out this line:
      @OSArch: X64, @AutoItX64: 0 Hotkey on a Chrome window: 0x00140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x00140330 Failed to get Chrome tab bar object. Hotkey events and mouse click events share the same function KeepLastTab($hWnd, $action = "LClick"),Why this function triggered by hotkey works on both 32-bit and 64-bit while that triggered by mouse events failed on 32-bit autoit?