UIASpy - UI Automation Spy Tool

[jimg]

Help for the mentally weak.

The install instructions for UAISpy are totally confusing to this occasional user.  "Copy all include files to this folder" depends on where you are.  Do the two root au3 files need to be in the Autoit Include folder? Do the "Includes" need to be within the AutoIt Include folder as a sub-folder or as peers?  Sorry for the lack of insight on how this stuff all interacts.

I keep my Au3 files on a different disk.

[junkew]

I am not sure if you should try uia things then if you struckle with installing.

Do you get errors?

In the help file is described how you can add additional paths for include files. For installing this you have to put the automation include files in your includes folder either the generic one or a configured folder. 

Maybe this will help

https://www.autoitscript.com/wiki/Adding_UDFs_to_AutoIt_and_SciTE

 

[Nine]

13 hours ago, jimg said:

I keep my Au3 files on a different disk.

That's fine, this is what I have done too.  Personally, I have created subfolders to hold each UDF separately.  And I create my scripts into the corresponding subfolder when I use a specific UDF.  Just another way to classify my UDF.

[jimg]

When I attempt to run UAISpy, I get a "file not found UIA_Costansts.au3".  It has the path right, file just isn't there.

This is a slow process because I have to work on a machine at my wife's office, and I can only do that when the employee is off and my wife isn't using the VPN.

[junkew]

"file not found UIA_Costansts.au3"  if that is literally what you get you should name it UIA_Constants.au3

And I hope you have read the first post correctly

Note that UI Automation UDFs must be installed in the Includes folder.

 

[jimg]

I'm sorry if my spelling suffers when I have to jump back and forth from RDP access to the target computer.  If you meant the first post of this thread, yes I read it, and it mentions UIA_constants.au3 is enhanced for Win10, but there's no link to the file.  I would think the non-Win10 enhanced file would have been in the zip archive.

You'll be 75 years old some day and misspell an occasional word.

[jimg]

Found it.  Lost my work window till tomorrow.

[Moonscarlet]

Hello,

I am having trouble using UIASpy, it works great for me for the things I've tried. However, on this system that I am on which is built on java and accessed through internet explorer only, UIASpy can't identify each element, when I press F1 it just highlights the whole window as one block and using Au3Info shows "SunAwtCanvas" as class:

 

What can I do to be able to identify and manipulate each of these elements? Is there anything additional that needs installing or something needs to be enabled for it to work?

I would appreciate your help.

Thanks.

[junkew]

FAQ 31 will put you in the right direction it will give you

Not sure if there are many people integrating JAB so there is not a complete UDF around

[BogdanNicolescu]

How do i install it? You forgot to mention this bit of info :))

I get this error:



L.E.: Solved the problem. I would tell if i had any sort of tracking the steps, but i definetly found out how to make it to work. Man, i wish i had it recorded ...

Edited January 21, 2023 by BogdanNicolescu

[Xiangui]

Is there any detailed documentation and some tutorials? The current data is a bit underused and responsible

[junkew]

If you read the first post in this thread it will give references on how to start with uia. If not clear then tell what you did so far.

1. Install uia udf

2. Install tool based on uia udf like the spy

 

[OJBakker]

A small leftish issue.

 

Func UIASpy_TreeView_ClickItem($hWnd, $hItem, $sButton = "left", $bMove = False, $iClicks = 1, $iSpeed = 0)

Please change "left" to "primary" or "secondary".

The current behavior is quite annoying for users like me whose right hand refuses to touch a mouse. 😀

Edited June 15, 2023 by OJBakker

[NassauSky]

In the NotepadAll.au3 sample.  Near the last line there is a value set (what is typed into the edit area).  How instead would I get the existing value so I can append to what exists already? I don't see a GetValue function.

 

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
;#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "..\..\..\..\..\Includes\CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  ; Open Notepad
  ;Run( "Notepad" )
  ;Sleep( 1000 )

  ; Create UI Automation object
          Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
          If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
          ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element
          Local $pDesktop, $oDesktop
          $oUIAutomation.GetRootElement( $pDesktop )
          $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
          If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
          ConsoleWrite( "$oDesktop OK" & @CRLF )

  ; --- Find Notepad window ---
          ConsoleWrite( "--- Find Notepad window ---" & @CRLF )
          Local $pCondition0
          $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition0 ) ; Create Condition
          If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
          ConsoleWrite( "$pCondition0 OK" & @CRLF )

          Local $pNotepad, $oNotepad
          $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition0, $pNotepad ) ; From Condition Create Property
          $oNotepad = ObjCreateInterface( $pNotepad, $sIID_IUIAutomationElement, $dtagIUIAutomationElement ) ; From Property Create Object
          If Not IsObj( $oNotepad ) Then Return ConsoleWrite( "$oNotepad ERR" & @CRLF )
          ConsoleWrite( "$oNotepad OK" & @CRLF )

  ; --- Fill Edit control ---
          ConsoleWrite( "--- Fill Edit control ---" & @CRLF )

          $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "15", $pCondition0 ) ; Create Condition
          If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
          ConsoleWrite( "$pCondition0 OK" & @CRLF )

          Local $pEdit, $oEdit
          $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition0, $pEdit ) ; From Condition Create Property
          $oEdit = ObjCreateInterface( $pEdit, $sIID_IUIAutomationElement, $dtagIUIAutomationElement ) ; From Property Create Object
          If Not IsObj( $oEdit ) Then Return ConsoleWrite( "$oEdit ERR" & @CRLF )
          ConsoleWrite( "$oEdit OK" & @CRLF )

          Local $pValuePattern, $oValuePattern
          $oEdit.GetCurrentPattern( $UIA_ValuePatternId, $pValuePattern )
          $oValuePattern = ObjCreateInterface( $pValuePattern, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
          If Not IsObj( $oValuePattern ) Then Return ConsoleWrite( "$oValuePattern ERR" & @CRLF )
          ConsoleWrite( "$oValuePattern OK" & @CRLF )

          MsgBox(0,"Existing Value", $oValuePattern.GetValue())
          $oValuePattern.SetValue( "HelloWorld" )
EndFunc

Thanks.

Edited August 8, 2023 by NassauSky

[jugador]

@NassauSky

Local $sValue
$oValuePattern.CurrentValue($sValue)
ConsoleWrite( "CurrentValue:  " & $sValue & @CRLF )

 

[LarsJ]

It is easiest to read the value directly in the Edit control with UIASpy. Patterns are only used for actions. eg. to press a button

[NassauSky]

Thanks @LarsJ!!

Would you know of any UIAutomation examples that presses a button and/or clicks a mouse on a control?  or if you don't know of any examples would you possibly be able to append to the example above to click on the File menu?

🧐

Edited August 10, 2023 by NassauSky

[NassauSky]

I might have it. You can comment on the code if I'm doing it wrong but it does function.

 

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
;#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "..\..\..\..\..\Includes\CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  ; --- Open Notepad (This example requires you to open notepad manually and enter some text in it before this app runs)
  ;Run( "Notepad" ) ;If you don't want to test extracting existing text uncomment this line and next line
  ;Sleep( 1000 )

  ; Create UI Automation object (Done Once)
          Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
          If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
          ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element (Done Once)
          Local $pDesktop, $oDesktop
          $oUIAutomation.GetRootElement( $pDesktop )
          $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
          If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
          ConsoleWrite( "$oDesktop OK" & @CRLF )

  ; --- Find Notepad window --- (Done Once)
          ConsoleWrite( "--- Find Notepad window ---" & @CRLF )
          Local $pCondition0
          $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition0 ) ; Create Condition
          If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
          ConsoleWrite( "$pCondition0 OK" & @CRLF )

          Local $pNotepad, $oNotepad
          $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition0, $pNotepad ) ; From Condition Create Property
          $oNotepad = ObjCreateInterface( $pNotepad, $sIID_IUIAutomationElement, $dtagIUIAutomationElement ) ; From Property Create Object
          If Not IsObj( $oNotepad ) Then Return ConsoleWrite( "$oNotepad ERR" & @CRLF )
          ConsoleWrite( "$oNotepad OK" & @CRLF )

  ; --- Fill Notepad Edit control ---
          ConsoleWrite( "--- Fill Notepad Edit control ---" & @CRLF )

          $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "15", $pCondition0 ) ; Create Condition
                  If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
                  ConsoleWrite( "$pCondition0 OK" & @CRLF )

          Local $pNotepadEditControl, $oNotepadEditControl
          $oNotepad.FindFirst( $TreeScope_Descendants, $pCondition0, $pNotepadEditControl ) ; From Condition Create Property
          $oNotepadEditControl = ObjCreateInterface( $pNotepadEditControl, $sIID_IUIAutomationElement, $dtagIUIAutomationElement ) ; From Property Create Object
                  If Not IsObj( $oNotepadEditControl ) Then Return ConsoleWrite( "$oNotepadEditControl ERR" & @CRLF )
                  ConsoleWrite( "$oNotepadEditControl OK" & @CRLF )

          Local $pValuePattern, $oValuePattern
          $oNotepadEditControl.GetCurrentPattern( $UIA_ValuePatternId, $pValuePattern )
          $oValuePattern = ObjCreateInterface( $pValuePattern, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern )
                  If Not IsObj( $oValuePattern ) Then Return ConsoleWrite( "$oValuePattern ERR" & @CRLF )
                  ConsoleWrite( "$oValuePattern OK" & @CRLF )

          Local $sValue
          $oValuePattern.CurrentValue($sValue)  ;  MsgBox(0,"Existing Value", $sValue)
          $oValuePattern.SetValue( $sValue & @CRLF & "HelloWorld" )

  ; --- Click File Menu

    ; --- Find File Menu element ---
    ConsoleWrite("--- Find File Menu element ---" & @CRLF)

    Local $pConditionFileMenu
    $oUIAutomation.CreatePropertyCondition($UIA_NamePropertyId, "File", $pConditionFileMenu)
            If Not $pConditionFileMenu Then Return ConsoleWrite("$pConditionFileMenu ERR" & @CRLF)
            ConsoleWrite("$pConditionFileMenu OK" & @CRLF)

    Local $pFileMenu, $oFileMenu
    $oNotepad.FindFirst($TreeScope_Descendants, $pConditionFileMenu, $pFileMenu)
    $oFileMenu = ObjCreateInterface($pFileMenu, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
            If Not IsObj($oFileMenu) Then Return ConsoleWrite("$oFileMenu ERR" & @CRLF)
            ConsoleWrite("$oFileMenu OK" & @CRLF)

    ; --- Invoke Click action on File Menu ---
    Local $pInvokePattern, $oInvokePattern
    $oFileMenu.GetCurrentPattern($UIA_InvokePatternId, $pInvokePattern)
    $oInvokePattern = ObjCreateInterface($pInvokePattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
            If Not IsObj($oInvokePattern) Then Return ConsoleWrite("$oInvokePattern ERR" & @CRLF)
            ConsoleWrite("$oInvokePattern OK" & @CRLF)

    $oInvokePattern.Invoke()

EndFunc

I am just trying to make this as clean as I can so I can understand what's exactly happening (and make it easy to append different automated actions to it)

 

Thanks!

Edited August 10, 2023 by NassauSky
Added a comment to code about having notepad open ahead of time

[NassauSky]

Still playing. Interesting and maybe you can clue me in to this quirk.

If you take my working code above which clicks on the File menu and you remove the 3 lines:
 

Local $sValue
          $oValuePattern.CurrentValue($sValue)  ;  MsgBox(0,"Existing Value", $sValue)
          $oValuePattern.SetValue( $sValue & @CRLF & "HelloWorld" )

 

Then the code doesn't click the File menu. Seems strange since those 3 lines don't seem to be relevant for any following variables/functions.

[junkew]

Hard to tell but most likely notepad is not having focus at the moment you try this. Automating gui is hard as you will face many quirks in how all ui automation framework behave.