IUIAutomation MS framework automate chrome, FF, IE, ....

[Garthzilla]

I GOT IT  That was the right line of thinking.

To reiterate, a specific popup window from an uninstall that could not be suppressed was also inaccessible with AutoIT and Inspect.exe. AutoIT would stop processing more commands, IUIAutomation would crash. Inspect.exe shows:
"Cannot get object from point:
[Error: FromPoint: hr=0xFFFFFFFF80070005 - Access is denied.]" when clicking or dragging the window in question.

MSIExec runs the particular window as as "SYSTEM" and not "Administrator" and thus the access denied ssue.. My solution to get Inspect.exe working:

Run CMD.exe as Administrator (so I can do the next step)
Run CMD.exe using "psexec -i -s cmd.exe" which will run cmd.exe as the "SYSTEM" user
Ran inspect.exe from here.

Now Inspect.exe can see the window and the button. I am sure if I run these tools as SYSTEM things will start to work correctly  Thanks for your help.

[junkew]

Check what narrator from windows is speaking to you when you hover over it. Check and search @larsj threads for catching uia events to see if it tells you more. He wrote some threads and deep information on usage of uia in examples section.

Edited March 4, 2020 by junkew

[DM666]

Do I need to have a special build of autoit to use it? I downloaded UIA_V0_70.zip, unpacked to a new  folder and try to run SimpleSpy.au3

I got error messages about missing includes (WinAPISys.au3, StringConstants.au3, APIShPathConstants.au3 and more). Found them all on GitHub (like this) and put them all in my \Autoit\Includes. Clearly it doesn't went very well since I'm now get a bunch of "previously declared as a 'Const'" errors

Quote

C:\Portable\AutoIt3\Include\constants.au3(18,76) : ERROR: $OPT_COORDSRELATIVE previously declared as a 'Const'.
Global Const $OPT_COORDSRELATIVE = 0 ; Relative coords to the active window
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Portable\AutoIt3\Include\constants.au3(19,77) : ERROR: $OPT_COORDSABSOLUTE previously declared as a 'Const'.
Global Const $OPT_COORDSABSOLUTE = 1 ; Absolute screen coordinates (default)

Tried on my 3.3.8.0 and fresh install of 3.3.8.1, Win7 x64.

May be someone who get it working can share his Includes dir?

[junkew]

Try 3.3.15 it was buld for latest versions of AutoIt

[DM666]

3.3.14.5 is the most recent version I could find.

But it does the job, thank you very much!

Edited April 20, 2020 by DM666

[nooneclose]

@junkew

When I try to run the example script number 5 for chrome I get this error:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\a\Desktop\General help\AutoIt_Stuff\UIA\examples\ex5_chrome.au3" /UserParams    
+>08:30:28 Starting AutoIt3Wrapper v.18.708.1148.0 SciTE v.4.1.0.0   Keyboard:00000409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\a\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\acook\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\a\Desktop\General help\AutoIt_Stuff\UIA\examples\ex5_chrome.au3
+>08:30:28 AU3Check ended.rc:0

>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Users\a\Desktop\General help\AutoIt_Stuff\UIA\examples\ex5_chrome.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop

"C:\Users\a\Desktop\General help\AutoIt_Stuff\AU3 include files\UIAWrappers.au3" (1726) : ==> Variable must be of type "Object".:
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$t = $obj^ ERROR

->08:30:30 AutoIt3.exe ended.rc:1
+>08:30:30 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 2.613

It happens when it gets to this line;

;~  get the addressbar
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"class:=Chrome_OmniboxView", $treescope_children) ;worked in chrome 28
$oChromeAddressBar =_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29
;~  $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChrome,"name:=Adres- en zoekbalk"  , $treescope_children) ;works in chrome 29

I am using chrome version 83.0.4103

Edited July 9, 2020 by nooneclose

[junkew]

Most likely chrome version changed object ordering so the definition should be rechecked for the addressbar.

[frank10]

Great UDF junkew!

I'm trying to click on an icon on the TrayNotifyWnd. The problem is this item does not have a Name nor a class etc

It is the icon of ExpressVPN program.

So I did this way:

Local $oP3=_UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Shell_TrayWnd", $treescope_children) 
Local $oP2=_UIA_getFirstObjectOfElement($oP3, "class:=TrayNotifyWnd", $treescope_children)  
Local $oP1=_UIA_getFirstObjectOfElement($oP2, "class:=SysPager", $treescope_children)   
Local $oP0=_UIA_getFirstObjectOfElement($oP1, "class:=ToolbarWindow32", $treescope_children)

; --- FindALL     ****************   looking for NamePropertyId = ''     *****************

Local $pCondition,  $pElements
$UIA_oUIAutomation.CreateTrueCondition( $pCondition )
$oP0.FindAll( $TreeScope_Descendants, $pCondition, $pElements )
Local $oUIElementArray1, $iLength
$oUIElementArray1 = ObjCreateInterFace( $pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
$oUIElementArray1.Length( $iLength )
Local $pElement, $oElement, $sValue
For $i = 0 To $iLength - 1
  $oUIElementArray1.GetElement( $i, $pElement )
  $oElement = ObjCreateInterface( $pElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oElement.GetCurrentPropertyValue( $UIA_NamePropertyId, $sValue )
  ConsoleWrite($i &  " $sValue = " & $sValue & @CRLF )
  if ($sValue = "") Then 
    _UIA_action($oElement,"left")
  endif
next

but this does not work as the $sValue duplicates an old Name of previous element found!

These are the first entries of Tray in Scite output:

0 $sValue = Telegram Desktop
1 $sValue = AutoIt - gui_test2.au3
2 $sValue = Telegram Desktop
3 $sValue = PDF24 Creator
4 $sValue = Rimozione sicura dell'hardware ed espulsione supporti

The 2° index should be instead "" and it's the one I'm looking for, instead it duplicates the name of the 0° index "Telegram Desktop".

So, after many tries,  I found this one is working:

Local $oP3=_UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Shell_TrayWnd", $treescope_children) 
Local $oP2=_UIA_getFirstObjectOfElement($oP3, "class:=TrayNotifyWnd", $treescope_children)  
Local $oP1=_UIA_getFirstObjectOfElement($oP2, "class:=SysPager", $treescope_children)   

Local $pCondition3, $pP0, $oP0
$UIA_oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "ToolbarWindow32", $pCondition3 )
$oP1.FindFirst( $TreeScope_Descendants, $pCondition3, $pP0 )
$oP0 = ObjCreateInterface( $pP0, $sIID_IUIAutomationElement, $dtagIUIAutomationElement ) 
    
; --- FindALL     ****************   looking for NamePropertyId = ''     *****************

Local $pCondition,  $pElements
$UIA_oUIAutomation.CreateTrueCondition( $pCondition )
$oP0.FindAll( $TreeScope_Descendants, $pCondition, $pElements )
Local $oUIElementArray1, $iLength
$oUIElementArray1 = ObjCreateInterFace( $pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
$oUIElementArray1.Length( $iLength )
Local $pElement, $oElement, $sValue
For $i = 0 To $iLength - 1
  $oUIElementArray1.GetElement( $i, $pElement )
  $oElement = ObjCreateInterface( $pElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oElement.GetCurrentPropertyValue( $UIA_NamePropertyId, $sValue )
  ConsoleWrite($i &  " $sValue = " & $sValue & @CRLF )
  if ($sValue = "") Then 
    _UIA_action($oElement,"left")
  endif
next

With the correct output:

0 $sValue = Telegram Desktop
1 $sValue = AutoIt - gui_test2.au3
2 $sValue = 
3 $sValue = PDF24 Creator
4 $sValue = Rimozione sicura dell'hardware ed espulsione supporti

 

So, why the first code does not work? It seems the same equal logic to me... something wrong in _UIA_getFirstObjectOfElement or FindAll funcs?

EDIT

Now I also check the processID of the icon on tray to see if it corresponds to the correct PID .exe I'm searching, using  _WinAPI_ReadProcessMemory.

Edited August 27, 2020 by frank10

[junkew]

Draw the hierarchy of the elements on paper or check if inspect tool gives you the hierarchy. Its allways troublesome to get all constructs and finds working.

[frank10]

I attach the hierarchy:

 

Button without name is my item.

But I don't understand what's the difference writing:

Local $oP0=_UIA_getFirstObjectOfElement($oP1, "class:=ToolbarWindow32", $treescope_children)

vs

Local $pCondition3, $pP0, $oP0
$UIA_oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "ToolbarWindow32", $pCondition3 )
$oP1.FindFirst( $TreeScope_Descendants, $pCondition3, $pP0 )
$oP0 = ObjCreateInterface( $pP0, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

All the rest of code is the same... but it works only using the 2° form of $oP0.

Edited August 29, 2020 by frank10

[frank10]

@junkew,

is it possible to intercept notifications that goes on the button TrayNotify inside the  trayNotifyWnd on the bottom right of the taskbar (without opening the relative Window clicking on it),  with IUIA?

Edited August 31, 2020 by frank10

[Nine]

Here what I'm using :

expandcollapse popup

#include <Constants.au3>
#include <Array.au3>
#include "Includes\CUIAutomation2.au3"

Global Const $sCLSID_CUIAutomation8 = "{e22ad333-b25f-460c-83d0-0581107395c9}"
Global Const $sIID_IUIAutomation2 = "{34723AFF-0C9D-49D0-9896-7AB52DF8CD8A}" ; Windows 8
Global Const $dtag_IUIAutomation2 = $dtagIUIAutomation & _
    "get_AutoSetFocus hresult(bool*);" & _
    "put_AutoSetFocus hresult(bool);" & _
    "get_ConnectionTimeout hresult(dword*);" & _
    "put_ConnectionTimeout hresult(dword);" & _
    "get_TransactionTimeout hresult(dword*);" & _
    "put_TransactionTimeout hresult(dword);"


Opt("MustDeclareVars", 1)

While True
  If _WinExits() Then ExitLoop
WEnd

Func _WinExits()
  ;Local $hWnd = WinWait("[CLASS:ApplicationFrameWindow]")
  ;ConsoleWrite($hWnd & @CRLF)

  Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
  If Not IsObj($oUIAutomation) Then Exit ConsoleWrite("$oUIAutomation ERR" & @CRLF)
  ConsoleWrite("$oUIAutomation OK" & @CRLF)

;  Local $oUIAutomation2 = ObjCreateInterface($sCLSID_CUIAutomation8, $sIID_IUIAutomation2, $dtag_IUIAutomation2)
;  If Not IsObj($oUIAutomation2) Then Exit ConsoleWrite("$oUIAutomation2 ERR" & @CRLF)
;  ConsoleWrite("$oUIAutomation2 OK" & @CRLF)

; $oUIAutomation2.put_ConnectionTimeout (0)

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement($pDesktop)
  $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
  If Not IsObj($oDesktop) Then Exit ConsoleWrite("$oDesktop ERR" & @CRLF)
  ConsoleWrite("$oDesktop OK" & @CRLF)

  ;Get application window
  Local $pCondition, $pElement, $oElement
  $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "Windows.UI.Core.CoreWindow", $pCondition)
  If Not $pCondition Then Exit ConsoleWrite("$pCondition ERR" & @CRLF)
  ConsoleWrite("$pCondition OK" & @CRLF)

  $oDesktop.FindFirst($TreeScope_Descendants, $pCondition, $pElement)
  $oElement = ObjCreateInterface($pElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
  If Not IsObj($oElement) Then Return False
  ConsoleWrite("Frame Window OK" & @CRLF)

  Local $vValue
  $oElement.GetCurrentPropertyValue($UIA_NamePropertyId, $vValue)
  Return $vValue == "Centre de contrôle"
EndFunc

There is a bit of a delay, that I haven't got the time to resolve so far. 

[junkew]

You should be able to catch the events. There are some very nice examples described in the forum by @LarsJ

[frank10]

Thank you Nine, it works, but only if you have notifications popup that appears as windows. If they don't appear, the notifications go directly on the tray icon (also without a descriptive text on the tooltip, you can read them only clicking to make window appear)... so maybe we must use an event catcher to get those hidden notifications.

I looked at Larsj events, but I don't see an event to detect such notifications. Maybe I'll look deeper.

 

With this code all is fine if it finds the element oMenu:

_UIA_setVar("oMenu","Title:=Connesso a.*;controltype:=UIA_MenuItemControlTypeId;class:=MenuItem") 
local $oMenu=_UIA_action("oMenu", "getobject")

But if it cannot find the element "oMenu", the program blocks at the line: action getObject... for a lot of time...

Shouldn't it be possible to get a faster response without blocking the script? (i.e in my pc it lasts for 67 seconds!)

BTW even if the obj exists it's very slow: 34 seconds...

Edited September 2, 2020 by frank10

[frank10]

I found out the problem with ScrollViewer element...

This is the hierarchy:

 

This works:

_UIA_setVar("oPopup","Title:=;controltype:=UIA_WindowControlTypeId;class:=Popup")   
    _UIA_setVar("oContextMenu","Title:=;controltype:=UIA_MenuControlTypeId;class:=ContextMenu") 
    _UIA_setVar("oScrollViewer","Title:=;controltype:=UIA_PaneControlTypeId;class:=ScrollViewer")   
    _UIA_setVar("oMenuItem","Title:=Connesso a.*;controltype:=UIA_MenuItemControlTypeId;class:=MenuItem")
    local $oMenuItem=_UIA_action("oMenuItem", "getobject")

If there is the menuItem "Connesso a ...", it works in about 3 seconds.

But if it doesn't find the menuItem because the menu is changed, it lasts more than 1 minute to find out the object does not exist.

So, I thought getting the object oScrollViewer and using findAll to list all menuItems to check which is there, with:

_UIA_setVar("oPopup","Title:=;controltype:=UIA_WindowControlTypeId;class:=Popup")   ;
    _UIA_setVar("oContextMenu","Title:=;controltype:=UIA_MenuControlTypeId;class:=ContextMenu") ;
    _UIA_setVar("oScrollViewer","Title:=;controltype:=UIA_PaneControlTypeId;class:=ScrollViewer")   ;
    local $oScrollViewer=_UIA_action("oScrollViewer", "getobject")

    ; --- FindALL
    Local $pCondition,  $pElements
    $UIA_oUIAutomation.CreateTrueCondition( $pCondition )
    $oScrollViewer.FindAll( $TreeScope_Descendants, $pCondition, $pElements )
    Local $oUIElementArray1, $iLength1 ; $pElements is a pointer to an UI Automation element array
    $oUIElementArray1 = ObjCreateInterFace( $pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray )
    $oUIElementArray1.Length( $iLength1 )
    Local $pElement1, $oElement1, $sValue

    For $i = 0 To $iLength1 - 1
      $oUIElementArray1.GetElement( $i, $pElement1 )
      $oElement1 = ObjCreateInterface( $pElement1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
      if not isobj($oElement1) Then ConsoleWrite( "no oElement1")
        ;$oElement1.GetCurrentPropertyValue( $UIA_ClassNamePropertyId, $sValue ) ; $UIA_ClassNamePropertyId is used as example
      ;ConsoleWrite( "$sValue1 = " & $sValue & @CRLF )
      $oElement1.GetCurrentPropertyValue( $UIA_NamePropertyId, $sValue )
      ConsoleWrite($i &  " $sValue = " & $sValue & @CRLF )

    Next

But it is very slow in finding ScrollViewer, and more, it does not find the correct one!

In fact in the list there are other object names different from the menu I'm analyzing.

 

But I found out if I use the FindAll on the oContextMenu, all is good, I get the menuItems...

So, even if the simplespy and UIAspy say there is the ScrollViewer element, it must be ignored!!

 

EDIT:

there is some problem with the forum layout... I cannot align correctly the text...

Edited September 2, 2020 by frank10

[junkew]

Slow is relative. The wrappers search in the hierarchy but when it fails it tries to find in more areas which can be slow indeed as it scans many elements. Maybe the log gives you some info on what happens. 

[duzers]

Hello, Can I acces to Java applications with this project?

[junkew]

No for that you should look at jab simplespy.

[NassauSky]

Hi @junkew,

I've used this UDF a few years ago and it was nice.  I'm trying to run the Example #5 "Chrome example" as is and I'm seeing a unique error in the UDF:

Here is the resulting output and error:

address: 
We have 2 tabs
Title is: <New Tab>    Class   := <>    controltype:= <50019>    
Title is: <New Tab>    Class   := <>    controltype:= <50000>    
"T:\Cool\Autoit-T\Scripts\STATScraper\ScrapingOptionsWebDriverOrIUI\UIA_V0_70\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR

I see the UDF highlight the address bar in red then the program crashes with the above error.  

I'm running the following Chrome:

Google Chrome 

Version 85.0.4183.102 (Official Build) (64-bit)

 

Any tips, tricks, ideas or suggestions would be great.

 

Thanks for all ! :-)

[junkew]

Per chrome version google changes small things on addressbar identification properties. So it cannot find the defined element and basically tries to get a rectangle on a not found object which throws the error.