AutoIT and MSAA

[automaterk]

Good afternoon all!

I am in the process of revisiting some of my previous issues now that I have a better understanding of AutoIt and have made some decent strides in other areas of my automation. Originally I had(and still do) an issue with reading ListItems from a list box within my application. This is the only control I could not get a good read on. Even hidden elements, etc. were visible when given the right script.

Fast forward to today and I just happened to download UI Accessibility checker. Previously, nothing I've tried has been able to see more than just the list box. However, this application identifies each name in the list in the tree view. This has given me hope that there may be a solution to automate this.

You can see the control and the list tree below:

I've done a bit of research in the forums, but haven't seen anything with definitive evidence that this can be done with AutoIt. I am currently attempting to do everything in AutoItx4Java and using base AutoIt commands, which has worked well so far. Does anyone have any insight into how I might be able to interact with these elements in AutoIt or a java based solution that I might try. This would really perfect the project I've been working on and any help would be greatly appreciated.

I have looked into the UIA thread, but it doesn't seem to be as useful here as UIA inspection does not return any results.

Thank you for any and all replies!

[junkew]

did you try simplespy and inspect as described in the uia thread?
whats the output of those tools, as you use the UI Accessibility checker I am 99.99% sure that inspect and simplespy give you nformation and that way you can automate.

This thread can also help but I did not find time to enhance / investigate further. (what you post seems to be a Java application)
this makes use of the JavaAccessibilityBridge and as such called it JABSimpleSpy.
https://www.autoitscript.com/forum/topic/166830-java-object-automation-and-simple-spy/

[LarsJ]

As far as I know the UI Accessibility checker (AccCheckUI.exe) is the predecessor of Inspect.exe. Both programs are based on two methods to identify windows and controls: MSAA-techniques and UIA-techniques. Because AccCheckUI.exe is the old program I think it's primarily based on MSAA-techniques. And the new program Inspect.exe is primarily based on UIA-techniques.

If AccCheckUI.exe can identify the list items, Inspect.exe should also be able to identify the list items. I remember you didn't have much success with Inspect.exe in the old thread. But did you try to run Inspect.exe in pure MSAA mode (combo box in upper left corner)?

If you can verify that the list items can be identified with MSAA-techniques, there can be a chance to be able to automate the list box with MSAA-code (instead of UIA-code).

[automaterk]

Sorry for the late reply - I got started on something else this week and am just now coming back to all of this. Thanks for getting back with me on this - I was hoping both of you would respond!

@junkew Using Simple UIA Spy I get the following result:

expandcollapse popup

Mouse position is retrieved 1006-544
At least we have an element [][]

Having the following values for all properties: 
Title is: <>    Class   := <>   controltype:= <UIA_ListItemControlTypeId>   ,<50007>    , (0000C357)    0;1;82524432;536870911
*** Parent Information top down ***
8: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1080
"Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769"   
7: Title is: <MEDHOST>  Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    449;167;1022;746
"Title:=MEDHOST;controltype:=UIA_WindowControlTypeId;class:=LSPANEL"    
6: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;362;591;356
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
5: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;389;591;302
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
4: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    677;366;591;352
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
3: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    685;396;4096;4096
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"  
2: Title is: <Select Library List to Copy>  Class   := <LSLISTBOX>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    718;456;451;179
"Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX"    
1: Title is: <> Class   := <LSLISTSUB>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;457;436;177
"Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTSUB"   
0: Title is: <> Class   := <LSLISTLIST> controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;474;4096;160
"Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTLIST"  


;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP7=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=MEDHOST;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)    
_UIA_Action($oP7,"setfocus")
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP6,"setfocus")
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX", $treescope_children) 
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTSUB", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTLIST", $treescope_children)   
_UIA_Action($oP0,"setfocus")
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <1>
UIA_handle:= <0>
UIA_RuntimeId:= <42;3672536;2;-2147483647;3672536;-4;1>
UIA_BoundingRectangle:= <0;1;82524432;536870911>
UIA_ProcessId:= <20572>
UIA_ControlType:= <50007>
UIA_LocalizedControlType:= <list item>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <True>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <True>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <1>
UIA_LegacyIAccessibleRole:= <34>
UIA_LegacyIAccessibleState:= <3145728>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:20572,hwnd:0x0 Annotation:Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

I've attempted to use the Java Spy, but it doesn't see anything on that application. I confirmed it worked with an actual java application.

 

@larsj

Thanks for the tip on inspect.exe - I didn't realize that was an option. Here are the results, which seem to be promising.

From this information and your understanding - is there any way to use AutoIt to accomplish this? Or will I most likely need to look into a Java solution for MSAA(Which so far has not been very fruitful).

I'm not really sure where to go from here if it is possible. None of my searching threads here has really pulled anything on how to use MSAA if you can. Anything to get me started would be greatly appreciated. As always, thank you so much for your help and if you need any more info please let me know!

[junkew]

As inspect.exe gives you the result / value you are looking for you do not have to go to a java solution.

But I do not have an out of the box solution as I expected that the simplespy should give the same result as inspect.exe

• option 1 fix simple spy or based on the standard code as given by simplespy get your information with the treewalker (see examples1-3)
• option 2 more difficult but I assume 1 can work although if you see the data in inspect.exe with MSAA option turned on than this is the way to go
  If you read the thread from start to end you have all the pieces of IUIAutomation, IAccessible, IAccessible2, ISimpledomand in that there will be a name value with the right content.
  This is the area where LarsJ provided the IAccessible ... stuff (they are also in the zip of the first post)
  https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/?do=findComment&comment=1156373

 

[automaterk]

Thanks for getting back to me. I thought I would go with option 2 first as it seems like the most promising, even if it is more difficult. I have all of the files in place and have searched the thread, but I am getting issues with the code you linked me to.

"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(53,40) : error: $OBJID_WINDOW previously declared as a 'Const'.
Global Const $OBJID_WINDOW = 0x00000000
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(54,42) : error: $OBJID_TITLEBAR previously declared as a 'Const'.
Global Const $OBJID_TITLEBAR = 0xFFFFFFFE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(55,42) : error: $OBJID_SIZEGRIP previously declared as a 'Const'.
Global Const $OBJID_SIZEGRIP = 0xFFFFFFF9
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(56,39) : error: $OBJID_CARET previously declared as a 'Const'.
Global Const $OBJID_CARET = 0xFFFFFFF8
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(57,40) : error: $OBJID_CURSOR previously declared as a 'Const'.
Global Const $OBJID_CURSOR = 0xFFFFFFF7
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(58,39) : error: $OBJID_ALERT previously declared as a 'Const'.
Global Const $OBJID_ALERT = 0xFFFFFFF6
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(59,39) : error: $OBJID_SOUND previously declared as a 'Const'.
Global Const $OBJID_SOUND = 0xFFFFFFF5
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Users\Keven.Reed\Desktop\iAccessible1.au3 - 7 error(s), 0 warning(s)

I don't see where it is referencing these aside from in the Constants file. I was trying to search around AutoIt and Google and couldn't find anything about these specific variables. Is this something you have faced or that you might know how to get around?

Thanks!

 

EDIT: Also - to go ahead and grab you while you're here - I am guessing from what I've read that this will return a way of identifying the elements in the list with MSAA. I am assuming that next I will be able to use the UIA commands you've written to interact with this list in AutoIt? Sorry if I am confused on something, but wanted to go ahead and check.

Edited July 31, 2015 by automaterk

[junkew]

you should get a better text editor for searching text in au3 files in directories

• msaccessibility.au3
• msactiveaccessibility.au3

you can uncomment it there as it now seems to be in AutoItConstants nowadays.

I did not integrate (yet) anything from MSAA into the UIA framework commands.  so no UIA commands as in the UIAWrappers will not work but the plain CUIAutomation stuff will work also with MSAA, study the example from LarsJ and hopefully you grasp the concept.

 

[LarsJ]

automaterk, This script based on MSAA-code prints information in SciTE console about all elements it can find in a window. You have to add a line in top of script to get window handle. Let's see whether the code can find the listbox elements.

expandcollapse popup

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )

Example()


Func Example()

  ; Get window handle
  ;Local $hWindow = WinGetHandle( "[CLASS:Chrome_WidgetWin_1]", "" ) ; Chrome            ; http://tweepi.com/auth/login
  ;Local $hWindow = WinGetHandle( "[CLASS:MozillaWindowClass]", "" ) ; Mozilla Firefox
  ;Local $hWindow = WinGetHandle( "[CLASS:IEFrame]" )                ; Internet Explorer
  ;Local $hWindow = WinGetHandle( "[CLASS:CabinetWClass]" )          ; Windows Explorer, Vista, 7, 8
  ;Local $hWindow = WinGetHandle( "[CLASS:ExploreWClass]" )          ; Windows Explorer, Windows XP
  Local $hWindow = WinGetHandle( "Calculator" )                     ; Calculator
  ;Local $hWindow = ControlGetHandle("[CLASS:Notepad]", "", "Edit1") ; Notepad
  ;Local $hWindow = 0x000403DA                                       ; Chrome            ; http://tweepi.com/auth/login
  If Not $hWindow Then Return ConsoleWrite( "Window handle ERR" & @CRLF )
  ConsoleWrite( "Window handle OK" & @CRLF )

  ; Get Window object
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return ConsoleWrite( "Window object ERR" & @CRLF )
  ConsoleWrite( "Window object OK" & @CRLF )

  ; Window name
  Local $sName
  ConsoleWrite( @CRLF & "Window:" )
  ConsoleWrite( @CRLF & "-------" & @CRLF )
  PrintElementInfo( $oWindow, $CHILDID_SELF, "" )

  ; Get children
  ConsoleWrite( "Children:" & @CRLF )
  ConsoleWrite( "---------" & @CRLF )
  WalkTreeWithAccessibleChildren( $pWindow, 0 )

EndFunc


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return
  $oAcc.AddRef()

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "
  Next

  Local $hWnd
  WindowFromAccessibleObject( $pAcc, $hWnd )
  ConsoleWrite( $sIndent & "Window  = " & $hWnd & @CRLF & @CRLF )

  Local $vt, $pChildObj, $oChildObj, $iChildElem

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object
      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )
      If IsObj( $oChildObj ) Then
        PrintElementInfo( $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )
      EndIf

    Else ; $vt = $VT_I4

      ; Child element
      $iChildElem = DllStructGetData( $tVarChildren, $i, 3 )
      PrintElementInfo( $oAcc, $iChildElem, $sIndent )

    EndIf

  Next

EndFunc

You need CUIAutomation2.au3 and MSAccessibility.au3.

If you get errors on double-defined constants, comment out the constants in MSAccessibility.au3.

Edited August 2, 2015 by LarsJ