Jump to content
bobmcrae

Is there a way to specify a UIAutomation control in a way that yields faster actions?

Recommended Posts

I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?

#include "UIAWrappers.au3"

_UIA_setVar("Global.Debug", False)
_UIA_setVar("Global.Debug.File", False)
_UIA_setVar("Global.Highlight", False)


_UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window")
_UIA_action('DPN','setFocus')

_UIA_setVar("DPN.firstName","AutomationId:=txtFirstName")
_UIA_setVar("DPN.lastName", "AutomationId:=txtLastName")
_UIA_Action('DPN.lastName','setvalue','last name')
_UIA_setVar("DPN.ID",       "AutomationId:=txtSubjectId")
_UIA_setVar("DPN.DOB",      "AutomationId:=PART_TextBox")
_UIA_setVar("DPN.Ft",       "AutomationId:=txtSubjectHeight")
_UIA_setVar("DPN.In",       "AutomationId:=txtSubjectHeight2")

_UIA_Action('DPN.firstName','setvalue','first name')
_UIA_Action('DPN.ID','setvalue','ID012345')
_UIA_Action('DPN.DOB','setvalue','1/31/1932')
_UIA_Action('DPN.Ft','setvalue','6')
_UIA_Action('DPN.In','setvalue','1')

SimpleSpy output:

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children)    
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children)    
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"click")

 

Share this post


Link to post
Share on other sites

The advantage of using the wrapper functions is that they are easy to use. The disadvantage is that they execute a large number of code lines. Especially _UIA_Action. The only thing you can do to increase performance is to avoid the wrapper functions and call the interface methods in CUIAutomation2.au3 directly.

You can look at some of my examples in the UI Automation thread to see how to do it.

Share this post


Link to post
Share on other sites

Next version I hope to have better speed of the uiaWrappers but I am not sure If I cause 2 seconds delay.

I know the logic in the wrappers is also if no treehierarchy is specified fallback to desktop and start searching the whole tree which can be huge and timeconsuming (when you turn logging on you can see what happens in detail and how many objects are traversed before finding the element)

Func _UIA_getFirstObjectOfElement($obj, $str, $treeScope) is probably quicker as it does not do a findall

I will put on my TODO some logic like

if in UIA_Action 1 property is used for finding an element directly call directly something like below

;- Change Func _UIA_getFirstObjectOfElement

$propertyID = $UIA_NamePropertyId ;- Or automation id property or .....
$tVal= ..... ;- property value

$UIA_oUIAutomation.createPropertyCondition($propertyID, $tval, $pCondition)
$oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition)
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

 

Share this post


Link to post
Share on other sites

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

Share this post


Link to post
Share on other sites

Findfirst is for sure match quicker and in your scenario your developers behave nicely on automationid.

The uiawrappers give 

A. Regular expression matching

B. Matching on index or ordinal value

C. Match with multiple properties

As said I will improve on single property and exact matching using findfirst

Share this post


Link to post
Share on other sites

Nice topic!

First i will learn, then i will ask forsome help later on :P

Really amazing job @junkew

Thanks adv.

@UPDATE:

On 19/02/2016 at 3:06 AM, bobmcrae said:

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

I'm getting:

Unknown function name.:
$aeParent = _getAeFromHandle($hWindow)
$aeParent = ^ ERROR

Some idea why?

Thanks adv.

Edited by Chiitus

Share this post


Link to post
Share on other sites

@Chiitus, it looks like perhaps you did not copy the entire  UIAutomation.au3 UDF, as the function _getAeFromHandle is at the bottom of the code above...take another look.;)

Share this post


Link to post
Share on other sites

how did you find this

'txtUserName'

(which way did you read it?)

and why did you use it together with

$UIA_AutomationIdPropertyId

Can you also give me direct link to this application

Sunlight MiniOmni

then I can download it, use "simple spy" and try to understand your code to learn UIA

Edited by maniootek

Share this post


Link to post
Share on other sites

@junkew Thanks so much for your UDF, it makes things much faster but can you include an "action" function in your UDF?


I'm using your script to automate mstsc.exe in win2016, I can inject username + password thanks to your code, but how do i click on the ok button?

Thanks.

Share this post


Link to post
Share on other sites

That user hasn't been here since 2016, I'm not sure he'll be responding.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

sn1kzZ, Here is a list of control types and supported patterns (actions).

You should create a new thread with your own question. Add information about application, controls and actions. Then it should not be impossible to make some fast code.

 

You can also google this way:

LarsJ CUIAutomation2.au3 site:autoitscript.com

Then you'll get a list of examples with very fast code.

Share this post


Link to post
Share on other sites

As you can put text in it with setvalue it should be easy to use action leftclick on the button. Just sinplespy on your ok button and see if it highlights. Post the output and code if its not working.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By adityaparakh
      Hello ,
      I am trying to fetch data and control a Windows Program.
      The data isn't reflected in the Au3Info tool , hence used SimpleSpy UIA , for trying.
      The code is able to fetch only one row at a time.
      Most Strangely , the code doesn't retreive the same specified row consistently.
      If I run the same code , under same circumstance , it is still fetching different rows.
      This was when op2 and op3 were zero. I would not like to use this at all.

      Also , it takes 3 seconds for it search down the hiearchy and reach the row.
      I will need to fetch ALL the rows continuously for 6 hours every second , in a minimized state.
      Can you please help.
      Attached :
      1. AuInfo summary
      2. Code
      3. Simple Spy Summary
      The Actions I am trying to do (in a Minimzed State - not affecting usage of computer for other purposes)
      1. Fetch all Rows (number of rows will not always be constant
      2. Press the Check box for a specific row(s)
      3. Press the squareOff button on top
      Thanks a lot
       
       
      #include <MsgBoxConstants.au3> #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP8=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=MO Trader;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP8,"setfocus") Local $oP7=_UIA_getObjectByFindAll($oP8, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP7,"setfocus") Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP6,"setfocus") Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP5,"setfocus") Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=Day Net Position;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP3,"setfocus") Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=0.0000;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP2,"setfocus") Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=0.0000;controltype:=UIA_TableControlTypeId;class:=WindowsForms10.Window.8.app.0.3e799b_r8_ad1", $treescope_children) _UIA_Action($oP1,"setfocus") Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Data Panel;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children) ;~ First find the object in the parent before you can do something Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Row 12;ControlType:=UIA_CustomControlTypeId", $treescope_subtree) _UIA_action($oUIElement,"click") Local $string = _UIA_action($oUIElement,"getValue") MsgBox($MB_OK,"", $string) Local $oUIElement2=_UIA_getObjectByFindAll($oP0, "title:=Row 6;ControlType:=UIA_CustomControlTypeId", $treescope_subtree) _UIA_action($oUIElement2,"click") Local $string2 = _UIA_action($oUIElement2,"getValue") MsgBox($MB_OK,"", $string2)  


      SimpleSpy Row.txt
    • By rmckay
      I'm trying to set the slider in a window using RangeValuePattern.SetValue().  The window is used to set the start point for replaying market data in the app NinjaTrader 8. When using the Action window in Inspect.exe I can set a value and the slider responds.  When I run a script that selects the slider and uses .SetValue() with the same value the slider does not move.  I've tried giving the focus to the slider with no change in results.  
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 ;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code #AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code #include "C:\Users\Rod\Documents\AutoIt\Include\UIA_Constants.au3" ; Can be copied from UIASpy Includes folder ;#include "UIA_Functions.au3" ; Can be copied from UIASpy Includes folder ;#include "UIA_SafeArray.au3" ; Can be copied from UIASpy Includes folder ;#include "UIA_Variant.au3" ; Can be copied from UIASpy Includes folder Opt("MustDeclareVars", 1) Example() Func Example() ; Create UI Automation object Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation8, $sIID_IUIAutomation5, $dtag_IUIAutomation5) If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF) ConsoleWrite("$oUIAutomation OK" & @CRLF) ; Get Desktop element Local $pDesktop, $oDesktop $oUIAutomation.GetRootElement($pDesktop) $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8) If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF) ConsoleWrite("$oDesktop OK" & @CRLF) ; --- Find window/control --- ConsoleWrite("--- Find window/control ---" & @CRLF) Local $pCondition0 $oUIAutomation.CreatePropertyCondition($UIA_NamePropertyId, "Playback", $pCondition0) If Not $pCondition0 Then Return ConsoleWrite("$pCondition0 ERR" & @CRLF) ConsoleWrite("$pCondition0 OK" & @CRLF) Local $pWindow1, $oWindow1 $oDesktop.FindFirst($TreeScope_Descendants, $pCondition0, $pWindow1) $oWindow1 = ObjCreateInterface($pWindow1, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8) If Not IsObj($oWindow1) Then Return ConsoleWrite("$oWindow1 ERR" & @CRLF) ConsoleWrite("$oWindow1 OK" & @CRLF) ; --- Find window/control --- ConsoleWrite("--- Find window/control ---" & @CRLF) Local $pCondition1 $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "slider", $pCondition1) If Not $pCondition1 Then Return ConsoleWrite("$pCondition1 ERR" & @CRLF) ConsoleWrite("$pCondition1 OK" & @CRLF) Local $pSlider1, $oSlider1 $oWindow1.FindFirst($TreeScope_Descendants, $pCondition1, $pSlider1) $oSlider1 = ObjCreateInterface($pSlider1, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8) If Not IsObj($oSlider1) Then Return ConsoleWrite("$oSlider1 ERR" & @CRLF) ConsoleWrite("$oSlider1 OK" & @CRLF) ; --- RangeValue Pattern (action) Object --- ConsoleWrite("--- RangeValue Pattern (action) Object ---" & @CRLF) Local $pRangeValuePattern1, $oRangeValuePattern1 $oSlider1.GetCurrentPattern($UIA_RangeValuePatternId, $pRangeValuePattern1) $oRangeValuePattern1 = ObjCreateInterface($pRangeValuePattern1, $sIID_IUIAutomationRangeValuePattern, $dtag_IUIAutomationRangeValuePattern) If Not IsObj($oRangeValuePattern1) Then Return ConsoleWrite("$oRangeValuePattern1 ERR" & @CRLF) ConsoleWrite("$oRangeValuePattern1 OK" & @CRLF) ; --- RangeValue Pattern (action) Methods --- ConsoleWrite("--- RangeValue Pattern (action) Methods ---" & @CRLF) Local $iSetValue = 50000 $oRangeValuePattern1.SetValue($iSetValue) ConsoleWrite("$oRangeValuePattern1.SetValue()" & @CRLF) EndFunc ;==>Example Console Output
      >"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\Rod\Documents\AutoIt\NT8 PlayBack\NT8 PlayBack01.au3" /UserParams    
      +>10:39:16 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.4.1.2.0   Keyboard:00000409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:2
      +>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\Rod\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\Rod\AppData\Local\AutoIt v3\SciTE 
      >Running AU3Check (3.3.14.5)  params:-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\Rod\Documents\AutoIt\NT8 PlayBack\NT8 PlayBack01.au3
      +>10:39:16 AU3Check ended.rc:0
      >Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Users\Rod\Documents\AutoIt\NT8 PlayBack\NT8 PlayBack01.au3"    
      +>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
      $oUIAutomation OK
      $oDesktop OK
      --- Find window/control ---
      $pCondition0 OK
      $oWindow1 OK
      --- Find window/control ---
      $pCondition1 OK
      $oSlider1 OK
      --- RangeValue Pattern (action) Object ---
      $oRangeValuePattern1 OK
      --- RangeValue Pattern (action) Methods ---
      $oRangeValuePattern1.SetValue()
      +>10:39:16 AutoIt3.exe ended.rc:0
      +>10:39:16 AutoIt3Wrapper Finished.
      >Exit code: 0    Time: 1.087
      I'm a noob with autoit and tried to attach screenshots to this post.  I have shots from UIASpy and Inspect.exe.  When I try to attach them I get an "Upload Failed" message.  They show results from the respective programs.  As an alternative I included the apps results with the slider selected.
      Inspect.exe:
      How found:    Selected from tree...
      Name:    ""
      ControlType:    UIA_SliderControlTypeId (0xC35F)
      LocalizedControlType:    "slider"
      BoundingRectangle:    {l:1579 t:531 r:1801 b:549}
      IsEnabled:    true
      IsOffscreen:    false
      IsKeyboardFocusable:    true
      HasKeyboardFocus:    false
      AcceleratorKey:    ""
      AccessKey:    ""
      ProcessId:    9928
      RuntimeId:    [7.26C8.279F814]
      AutomationId:    "slider"
      FrameworkId:    "WPF"
      ClassName:    "Slider"
      IsControlElement:    true
      IsContentElement:    true
      ProviderDescription:    "[pid:9928,providerId:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]"
      IsPassword:    false
      ItemStatus:    ""
      ItemType:    ""
      IsRequiredForForm:    false
      HelpText:    ""
      ClickablePoint:    {x:-2147483648 y:-2147483648}
      Orientation:    0
      LegacyIAccessible.ChildId:    0
      LegacyIAccessible.DefaultAction:    ""
      LegacyIAccessible.Description:    ""
      LegacyIAccessible.Help:    ""
      LegacyIAccessible.KeyboardShortcut:    ""
      LegacyIAccessible.Name:    ""
      LegacyIAccessible.Role:    slider (0x33)
      LegacyIAccessible.State:    focusable (0x100000)
      LegacyIAccessible.Value:    "0"
      RangeValue.IsReadOnly:    false
      RangeValue.LargeChange:    1.000000
      RangeValue.Maximum:    691200.000000
      RangeValue.Minimum:    0.000000
      RangeValue.SmallChange:    0.100000
      RangeValue.Value:    0.000000
      IsAnnotationPatternAvailable:    false
      IsDragPatternAvailable:    false
      IsDockPatternAvailable:    false
      IsDropTargetPatternAvailable:    false
      IsExpandCollapsePatternAvailable:    false
      IsGridItemPatternAvailable:    false
      IsGridPatternAvailable:    false
      IsInvokePatternAvailable:    false
      IsItemContainerPatternAvailable:    false
      IsLegacyIAccessiblePatternAvailable:    true
      IsMultipleViewPatternAvailable:    false
      IsObjectModelPatternAvailable:    false
      IsRangeValuePatternAvailable:    true
      IsScrollItemPatternAvailable:    false
      IsScrollPatternAvailable:    false
      IsSelectionItemPatternAvailable:    false
      IsSelectionPatternAvailable:    false
      IsSpreadsheetItemPatternAvailable:    false
      IsSpreadsheetPatternAvailable:    false
      IsStylesPatternAvailable:    false
      IsSynchronizedInputPatternAvailable:    true
      IsTableItemPatternAvailable:    false
      IsTablePatternAvailable:    false
      IsTextChildPatternAvailable:    false
      IsTextEditPatternAvailable:    false
      IsTextPatternAvailable:    false
      IsTextPattern2Available:    false
      IsTogglePatternAvailable:    false
      IsTransformPatternAvailable:    false
      IsTransform2PatternAvailable:    false
      IsValuePatternAvailable:    false
      IsVirtualizedItemPatternAvailable:    false
      IsWindowPatternAvailable:    false
      IsCustomNavigationPatternAvailable:    false
      IsSelectionPattern2Available:    false
      FirstChild:    "" thumb
      LastChild:    "" thumb
      Next:    "M0,3C0,3 0,4 0,4 0,4 3,4 3,4 3,4 3,3 3,3 3,3 4,3 4,3 4,3 4,4 4,4 4,4 7,4 7,4 7,4 7,3 7,3 7,3 6,3 6,3 6,3 6,2 6,2 6,2 5,2 5,2 5,
      Previous:    "" button
      Other Props:    Object has no additional properties
      Children:    "" thumb
      Ancestors:    "Playback" window
          "Desktop 1" pane
          [ No Parent ]
       
      UIASpy:
      Treeview Element                                    Slider: Slider
                                                          
      Element Properties (identification)                 
      $UIA_AutomationIdPropertyId                         slider
      $UIA_ClassNamePropertyId                            Slider
      $UIA_ControlTypePropertyId                          $UIA_SliderControlTypeId
                                                          
      Element Properties (session unique)                 
      $UIA_ProcessIdPropertyId                            9928
      $UIA_RuntimeIdPropertyId                            7,9928,41547796
                                                          
      Element Properties (information)                    
      $UIA_BoundingRectanglePropertyId                    l=1579,t=531,w=222,h=18
      $UIA_ClickablePointPropertyId                       -2147483648,-2147483648
      $UIA_FrameworkIdPropertyId                          WPF
      $UIA_HeadingLevelPropertyId                         80050
      $UIA_LocalizedControlTypePropertyId                 slider
      $UIA_OptimizeForVisualContentPropertyId             False
      $UIA_ProviderDescriptionPropertyId                  [pid:9928,providerId:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]
                                                          
      Element Properties (has/is info)                    
      $UIA_HasKeyboardFocusPropertyId                     False
      $UIA_IsContentElementPropertyId                     True
      $UIA_IsControlElementPropertyId                     True
      $UIA_IsDataValidForFormPropertyId                   False
      $UIA_IsDialogPropertyId                             False
      $UIA_IsEnabledPropertyId                            True
      $UIA_IsKeyboardFocusablePropertyId                  True
      $UIA_IsOffscreenPropertyId                          False
      $UIA_IsPasswordPropertyId                           False
      $UIA_IsPeripheralPropertyId                         False
      $UIA_IsRequiredForFormPropertyId                    False
                                                          
      Control Patterns (element actions)                  
      $UIA_IsLegacyIAccessiblePatternAvailablePropertyId  True (LegacyIAccessiblePattern)
      $UIA_IsRangeValuePatternAvailablePropertyId         True (RangeValuePattern)
      $UIA_IsSynchronizedInputPatternAvailablePropertyId  True (SynchronizedInputPattern)
                                                          
      Control Pattern Properties                          
      $UIA_LegacyIAccessibleChildIdPropertyId             0
      $UIA_LegacyIAccessibleDefaultActionPropertyId       
      $UIA_LegacyIAccessibleDescriptionPropertyId         
      $UIA_LegacyIAccessibleHelpPropertyId                
      $UIA_LegacyIAccessibleKeyboardShortcutPropertyId    
      $UIA_LegacyIAccessibleNamePropertyId                
      $UIA_LegacyIAccessibleRolePropertyId                51 = $ROLE_SYSTEM_SLIDER
      $UIA_LegacyIAccessibleStatePropertyId               1048576 = $STATE_SYSTEM_FOCUSABLE
      $UIA_LegacyIAccessibleValuePropertyId               0
      $UIA_RangeValueIsReadOnlyPropertyId                 False
      $UIA_RangeValueLargeChangePropertyId                1
      $UIA_RangeValueMaximumPropertyId                    691199.9999999
      $UIA_RangeValueMinimumPropertyId                    0
      $UIA_RangeValueSmallChangePropertyId                0.1
      $UIA_RangeValueValuePropertyId                      2.47032822920623e-319
                                                          
      Control Pattern Methods                             
      LegacyIAccessible Pattern Methods                   
                                                          DoDefaultAction()
                                                          Select(long)
                                                          SetValue(wstr)
                                                          GetIAccessible(idispatch*)
                                                          CurrentChildId(int*)
                                                          CurrentDefaultAction(bstr*)
                                                          CurrentDescription(bstr*)
                                                          CurrentHelp(bstr*)
                                                          CurrentKeyboardShortcut(bstr*)
                                                          CurrentName(bstr*)
                                                          CurrentRole(uint*)
                                                          CurrentState(uint*)
                                                          CurrentValue(bstr*)
                                                          GetCurrentSelection(ptr*)
      RangeValue Pattern Methods                          
                                                          SetValue(ushort)
                                                          CurrentIsReadOnly(long*)
                                                          CurrentMaximum(ushort*)
                                                          CurrentMinimum(ushort*)
                                                          CurrentLargeChange(ushort*)
                                                          CurrentSmallChange(ushort*)
                                                          CurrentValue(ushort*)
      SynchronizedInput Pattern Methods                   
                                                          StartListening(long)
                                                          Cancel()
                                                          
      Parents from Desktop                                Pane: Desktop 1
                                                          Window: Playback
                                                          
      Parent to child index                               
       
      In addition to this problem, I am unable to press the start button with an autoit script or by using inspect.exe=>Action=>Toggle() or inspect.exe=>Action=>DoDefaultAction().  I'll post that after solving the current problem.

       
    • By junkew
      Automate all windows and browser applications with one UDF function library. Based on the microsoft automation API this library high level supports
      Recognition of conttrols from EDGE, Chrome, FF, Opera, Safari and Windows native apps Small testing framework to split object repository from coding away Introduction
      Quickstart - Getting started quickly
      Simple scripts
      With this module
      you can automate all applications/programs that support ui automation and/or accesibility api from microsoft you can recognize more controls than AutoIT can recognize "out of the box"  you can use concepts from other testing frameworks like http://download.freedesktop.org/ldtp/doc/ldtp-tutorial.pdf
      http://safsdev.sourceforge.net/Default.htm
      coded ui testing from microsoft 
      Some of those controls / applications are
      chrome browser (partly mainwindow has to be done with MSAA for navigating) chrome://accessibility in the adress bar of chrome or start with "--force-renderer-accessibility"
      silverlight controls Ribbon control controlbars of Excel/Word IE and FF browsers Windows Media Player Windows clock AFX .. controls (partly) ....  
      Based on the initial AIO Object I now have made the interface file to work with objCreateInterface function which is in the latest beta's
      automate clicking and querying basic information
      It gives you a lot of basic information to be able to automate clicking, querying basic information where it goes further in certain situations than AutoIt is identifying
      Starting threads for background on the ui automation api of microsoft (not for starters)
      http://en.wikipedia.org/wiki/Microsoft_UI_Automation http://msdn.microsoft.com/en-us/library/ms747327.aspx Previous threads in general help/support Interface AutoItObject IUIAutomation ObjCreateInterface and struct tagPoint in method ElementFromPoint  
      Be aware that API is not allways installed under XP/Vista see http://support.microsoft.com/kb/971513 Within Windows 7 and Windows 8 it should be preinstalled by default. Be aware on 32 and 64 bits way of running your script
      #AutoIt3Wrapper_UseX64=Y or N
       
      Basic example of usage / showing and retrieving the default information, will post multiple examples later
      Hover your mouse to an area of interest and press ctrl+w and information will be shown in the edit box of the form
       
       
      Simple spy demo (see simplespy.au3 or use latest ZIP  attachment for latest version)
       
      Main features
      Recognize windows and html controls for the major browsers Logical and physical description for controls (UI mapping, Application map) Simple repository logic to abstract logical and physical descriptions Store Runtime Type Information in RTI. variables Rubberbanding/highlighting of objects Simple spy to help in making / identifying the physical description Support of regular expression(s) in identifying objects recognize objects on multiple properties supported properties: name ,title, automationid, classname, class, iaccessiblevalue, iaccessiblechildId, controltype, processid, acceleratorkey
       The actions provided so far
       "leftclick", "left", "click", "leftdoubleclick", "leftdouble", "doubleclick", _
       "rightclick", "right", "rightdoubleclick", "rightdouble", _
       "middleclick", "middle", "middledoubleclick", "middledouble", "mousemove", "movemouse"
       "setvalue","settextvalue"
       "setvalue using keys"
       "setValue using clipboard"
       "getvalue"
       "sendkeys", "enterstring", "type", "typetext"
       "invoke"
       "focus", "setfocus", "activate"
       "close"
       "move","setposition"
       "resize"
       "minimize", "maximize", "normal", "close", "exist", "exists"
       "searchcontext", "context"
       "highlight"
       "getobject","object"
       "attach"
       "capture","screenshot", "takescreenshot"
       "dump", "dumpthemall"
       "propertyvalue", "property"
       match on multiple properties like:  name:=((Zoeken.*)|(Find.*)); ControlType:=Button; acceleratorkey:=Ctrl+F
      Support for 117 different properties see $UIA_propertiesSupportedArray in uiawrappers like for example title, regexptitle, class, regexpclass, iaccessiblevalue, iaccessiblechildid, name, accesskey, automationid, classname 
      IAccessible, IAccessible2, ISimpleDom interfaces  debuglogging to a file log.txt (no output in scitewindow) Examples
      Example 1 Iterating thru the different ways of representing the objects in the tree (#comment-1105548) Example 2 Finding the taskbar and clicking on the start menu button (#comment-1105680) Example 3 Clicking a litlle more and in the end displaying all items from the clock (thats not directly possible with AU3Info) (#comment-1108849) Example 4 that demonstrates the calculator Example 5 Automating chrome Example 6 Demonstrates all stuff within chrome to navigate html pages, find hyperlink, click hyperlink, find picture, click picture, enter data in inputbox Example 7 The chrome example modified to a firefox example Example 8 The other major browser Internet Explorer automated (made on Example 6 and 7) Example 9 Windows media player Example 10 Automating mach 3 (AFX windows and other hard to get recognized by AutoIT) Lot of links are broken due to forum upgrade just search for the text like "Example 11 Demonstrate Word, Notepad and Calculator actions"
      Example 11 Demonstrate Word, Notepad and Calculator actions ... Example 13 Details 1 about the right pane of the windows explorer Example 14 Details 2 about the right pane of the windows explorer Example 15 Details 3 about the right pane of the windows explorer Example 16 Details 4 about the right pane of the windows explorer Example 17 Details 5 about the right pane of the windows explorer WITH CACHING Example 18 Details 6 about the right pane of the windows explorer WITH VIRTUAL ITEMS Example 19 Eventhandling examples Example 20 Eventhandling examples Example 21a Eventhandling examples Internet Explorer Example 21b Eventhandling examples Internet Explorer Example 22 Eventhandling examples Follow focus Example 23 Eventhandling examples structure changed Example 24 Eventhandling examples IUIAutomationEventHandler Example 25 SAFEARRAYS Example 26 IACCESSIBLE / MSAA Example 27 IACCESSIBLE2 / MSAA Example 28 IACCESSIBLE / MSAA events Example 29 IACCESSIBLE2 events Example 30 ISimpleDOM Example 31 Notepad window move, maximize, minimize Example 32 Three browsers doing the same stuff with small differences in scripting only ..
      TODO Build recorder Enhance the spy with a nicer UI UI for the repository (now in the script with dot notation) Enhance mapping / identifying on multiple properties instead of 1 combined with index If speed becomes an issue use the caching logic of the MS UIA framework Add the other patterns later Generalize the concept of System Under Test of starting the SUT (for testing framework purposes) Remote running of scripts Fix issue on finding within dynamic context  ... edit august 18th 2013  
      initial post Only zip files are needed to download , just unzip in 1 directory
      edit july 2016
      Made V0_63 and examples  works with AutoIt v3.3.14 Windows 10 tested Simple spy gives some basic code as a present Chrome latest versions seems to be having issues with IUIAutomation on tabs/buttons of mainwindow use MSAA for accessing tabsheets / buttons more cleanup to be in UDF style More comments in the source see changelog.txt for previous changes edit september 2017
      All examples fixed for the IE, Firefox and Chrome browser Some small but essential fixes in UIAWrappers edit april 2018 
          Enhanced logic on fallback / dynamic search, still not perfect, to slow     Retested with latest Chrome, FF, Edge and IE11 and some extensions to show how to get text from the webpage (examples 5,6,7)     Some small bugfixes     Some comments as given in forum incorporated edit may 2019
          Speed enhancements on especially fallback searching     UIA.CFG works now in a better way to turn on/off debug, highlighting, debug2file     More stable and consistent behavior     Internal cleanup and refactoring of bigger functions     Checked with W10 (not tested on W7)     Added some W10 properties     Run with 3.3.14.5 on W10  
      UIA_V0_51.zip                 EXAMPLES_V0_5.zip     
      UIA_V0_63.zip                 EXAMPLES_V0_63.zip
      UIA_V0_64.zip                 EXAMPLES_V0_64.zip
       
      EXAMPLES_V0_66.zip
      UIA_V0_66.zip
      EXAMPLES_V0_70.zip UIA_V0_70.zip
    • By caramen
      Is that correct?
      With SimpleSpy i can click on controls without any include/udf ON CHROME?
    • By drapdv
      So, I have this odd issue happening with UIAWrappers.  When I run simplespy.au3 through SciTE, it finds the object I'm trying to work with, no problem. But, when I compile simplespy.exe, it can't find it.  Another script that utilizes UIAWrappers is experiencing the same phenomenon. EDIT: Compiling as 32-bit or 64-bit does not affect this behavior.
      The trouble starts at the very first object I try to get via _UIA_getObjectByFindAll, so I included the $properties2Match array and the object properties that were returned by _UIA_getPropertyValue for that first object both when running from SciTE and when compiled.
      It always finds the matches for the title and control type properties.  But when compiled, it fails to find the Automation ID property.  Any ideas as to why this would be happening?  Hoping junkew will see this and have an idea.
       
      These are the $properties2Match
      30005|Screen/PVN - ###.###.##.## - CDK Drive|0|False
      30003|50032|0|False
      30011|Portal|0|False
       
      Object properties when running from SciTE:
      <propertymatching>       
              j:1 propID:[30005] expValue:[Screen/PVN - ###.###.##.## - CDK Drive]actualValue:[Screen/PVN - ###.###.##.## - CDK Drive]1
              j:2 propID:[30003] expValue:[50032]actualValue:[50032]1
              j:3 propID:[30011] expValue:[Portal]actualValue:[Portal]1  
      </propertymatching>
       
      Object properties when script is compiled:
      <propertymatching>
              j:1 propID:[30005] expValue:[Screen/PVN - ###.###.##.## - CDK Drive]actualValue:[Screen/PVN - ###.###.##.## - CDK Drive]1
              j:2 propID:[30003] expValue:[50032]actualValue:[50032]1
              j:3 propID:[30011] expValue:[Portal]actualValue:[]0  
      </propertymatching>
       
×
×
  • Create New...