AutoIT and MSAA

8 posts in this topic

Good afternoon all!

I am in the process of revisiting some of my previous issues now that I have a better understanding of AutoIt and have made some decent strides in other areas of my automation. Originally I had(and still do) an issue with reading ListItems from a list box within my application. This is the only control I could not get a good read on. Even hidden elements, etc. were visible when given the right script.

Fast forward to today and I just happened to download UI Accessibility checker. Previously, nothing I've tried has been able to see more than just the list box. However, this application identifies each name in the list in the tree view. This has given me hope that there may be a solution to automate this.

You can see the control and the list tree below:


I've done a bit of research in the forums, but haven't seen anything with definitive evidence that this can be done with AutoIt. I am currently attempting to do everything in AutoItx4Java and using base AutoIt commands, which has worked well so far. Does anyone have any insight into how I might be able to interact with these elements in AutoIt or a java based solution that I might try. This would really perfect the project I've been working on and any help would be greatly appreciated.

I have looked into the UIA thread, but it doesn't seem to be as useful here as UIA inspection does not return any results.

Thank you for any and all replies!

Share this post

Link to post
Share on other sites

did you try simplespy and inspect as described in the uia thread?
whats the output of those tools, as you use the UI Accessibility checker I am 99.99% sure that inspect and simplespy give you nformation and that way you can automate.

This thread can also help but I did not find time to enhance / investigate further. (what you post seems to be a Java application)
this makes use of the JavaAccessibilityBridge and as such called it JABSimpleSpy.

1 person likes this

Share this post

Link to post
Share on other sites

As far as I know the UI Accessibility checker (AccCheckUI.exe) is the predecessor of Inspect.exe. Both programs are based on two methods to identify windows and controls: MSAA-techniques and UIA-techniques. Because AccCheckUI.exe is the old program I think it's primarily based on MSAA-techniques. And the new program Inspect.exe is primarily based on UIA-techniques.

If AccCheckUI.exe can identify the list items, Inspect.exe should also be able to identify the list items. I remember you didn't have much success with Inspect.exe in the old thread. But did you try to run Inspect.exe in pure MSAA mode (combo box in upper left corner)?

If you can verify that the list items can be identified with MSAA-techniques, there can be a chance to be able to automate the list box with MSAA-code (instead of UIA-code).

1 person likes this

Share this post

Link to post
Share on other sites

Sorry for the late reply - I got started on something else this week and am just now coming back to all of this. Thanks for getting back with me on this - I was hoping both of you would respond!

@junkew Using Simple UIA Spy I get the following result:

Mouse position is retrieved 1006-544
At least we have an element [][]

Having the following values for all properties: 
Title is: <>    Class   := <>   controltype:= <UIA_ListItemControlTypeId>   ,<50007>    , (0000C357)    0;1;82524432;536870911
*** Parent Information top down ***
8: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1080
7: Title is: <MEDHOST>  Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    449;167;1022;746
6: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;362;591;356
5: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;389;591;302
4: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    677;366;591;352
3: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    685;396;4096;4096
2: Title is: <Select Library List to Copy>  Class   := <LSLISTBOX>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    718;456;451;179
"Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX"    
1: Title is: <> Class   := <LSLISTSUB>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;457;436;177
0: Title is: <> Class   := <LSLISTLIST> controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;474;4096;160

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP7=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=MEDHOST;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)    
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX", $treescope_children) 
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTSUB", $treescope_children)    
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTLIST", $treescope_children)   
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)

*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <1>
UIA_handle:= <0>
UIA_RuntimeId:= <42;3672536;2;-2147483647;3672536;-4;1>
UIA_BoundingRectangle:= <0;1;82524432;536870911>
UIA_ProcessId:= <20572>
UIA_ControlType:= <50007>
UIA_LocalizedControlType:= <list item>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <True>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <True>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <1>
UIA_LegacyIAccessibleRole:= <34>
UIA_LegacyIAccessibleState:= <3145728>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:20572,hwnd:0x0 Annotation:Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

I've attempted to use the Java Spy, but it doesn't see anything on that application. I confirmed it worked with an actual java application.



Thanks for the tip on inspect.exe - I didn't realize that was an option. Here are the results, which seem to be promising.


From this information and your understanding - is there any way to use AutoIt to accomplish this? Or will I most likely need to look into a Java solution for MSAA(Which so far has not been very fruitful).

I'm not really sure where to go from here if it is possible. None of my searching threads here has really pulled anything on how to use MSAA if you can. Anything to get me started would be greatly appreciated. As always, thank you so much for your help and if you need any more info please let me know!

Share this post

Link to post
Share on other sites

As inspect.exe gives you the result / value you are looking for you do not have to go to a java solution.

But I do not have an out of the box solution as I expected that the simplespy should give the same result as inspect.exe

  • option 1 fix simple spy or based on the standard code as given by simplespy get your information with the treewalker (see examples1-3)
  • option 2 more difficult but I assume 1 can work although if you see the data in inspect.exe with MSAA option turned on than this is the way to go
    If you read the thread from start to end you have all the pieces of IUIAutomation, IAccessible, IAccessible2, ISimpledomand in that there will be a name value with the right content.
    This is the area where LarsJ provided the IAccessible ... stuff (they are also in the zip of the first post)


1 person likes this

Share this post

Link to post
Share on other sites

#6 ·  Posted (edited)

Thanks for getting back to me. I thought I would go with option 2 first as it seems like the most promising, even if it is more difficult. I have all of the files in place and have searched the thread, but I am getting issues with the code you linked me to.

"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(53,40) : error: $OBJID_WINDOW previously declared as a 'Const'.
Global Const $OBJID_WINDOW = 0x00000000
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(54,42) : error: $OBJID_TITLEBAR previously declared as a 'Const'.
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(55,42) : error: $OBJID_SIZEGRIP previously declared as a 'Const'.
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(56,39) : error: $OBJID_CARET previously declared as a 'Const'.
Global Const $OBJID_CARET = 0xFFFFFFF8
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(57,40) : error: $OBJID_CURSOR previously declared as a 'Const'.
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(58,39) : error: $OBJID_ALERT previously declared as a 'Const'.
Global Const $OBJID_ALERT = 0xFFFFFFF6
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(59,39) : error: $OBJID_SOUND previously declared as a 'Const'.
Global Const $OBJID_SOUND = 0xFFFFFFF5
C:\Users\Keven.Reed\Desktop\iAccessible1.au3 - 7 error(s), 0 warning(s)

I don't see where it is referencing these aside from in the Constants file. I was trying to search around AutoIt and Google and couldn't find anything about these specific variables. Is this something you have faced or that you might know how to get around?



EDIT: Also - to go ahead and grab you while you're here - I am guessing from what I've read that this will return a way of identifying the elements in the list with MSAA. I am assuming that next I will be able to use the UIA commands you've written to interact with this list in AutoIt? Sorry if I am confused on something, but wanted to go ahead and check.

Edited by automaterk

Share this post

Link to post
Share on other sites

you should get a better text editor for searching text in au3 files in directories

  • msaccessibility.au3
  • msactiveaccessibility.au3

you can uncomment it there as it now seems to be in AutoItConstants nowadays.

I did not integrate (yet) anything from MSAA into the UIA framework commands.  so no UIA commands as in the UIAWrappers will not work but the plain CUIAutomation stuff will work also with MSAA, study the example from LarsJ and hopefully you grasp the concept.


Share this post

Link to post
Share on other sites

#8 ·  Posted (edited)

automaterk, This script based on MSAA-code prints information in SciTE console about all elements it can find in a window. You have to add a line in top of script to get window handle. Let's see whether the code can find the listbox elements.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )


Func Example()

  ; Get window handle
  ;Local $hWindow = WinGetHandle( "[CLASS:Chrome_WidgetWin_1]", "" ) ; Chrome            ;
  ;Local $hWindow = WinGetHandle( "[CLASS:MozillaWindowClass]", "" ) ; Mozilla Firefox
  ;Local $hWindow = WinGetHandle( "[CLASS:IEFrame]" )                ; Internet Explorer
  ;Local $hWindow = WinGetHandle( "[CLASS:CabinetWClass]" )          ; Windows Explorer, Vista, 7, 8
  ;Local $hWindow = WinGetHandle( "[CLASS:ExploreWClass]" )          ; Windows Explorer, Windows XP
  Local $hWindow = WinGetHandle( "Calculator" )                     ; Calculator
  ;Local $hWindow = ControlGetHandle("[CLASS:Notepad]", "", "Edit1") ; Notepad
  ;Local $hWindow = 0x000403DA                                       ; Chrome            ;
  If Not $hWindow Then Return ConsoleWrite( "Window handle ERR" & @CRLF )
  ConsoleWrite( "Window handle OK" & @CRLF )

  ; Get Window object
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return ConsoleWrite( "Window object ERR" & @CRLF )
  ConsoleWrite( "Window object OK" & @CRLF )

  ; Window name
  Local $sName
  ConsoleWrite( @CRLF & "Window:" )
  ConsoleWrite( @CRLF & "-------" & @CRLF )
  PrintElementInfo( $oWindow, $CHILDID_SELF, "" )

  ; Get children
  ConsoleWrite( "Children:" & @CRLF )
  ConsoleWrite( "---------" & @CRLF )
  WalkTreeWithAccessibleChildren( $pWindow, 0 )


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "

  Local $hWnd
  WindowFromAccessibleObject( $pAcc, $hWnd )
  ConsoleWrite( $sIndent & "Window  = " & $hWnd & @CRLF & @CRLF )

  Local $vt, $pChildObj, $oChildObj, $iChildElem

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object
      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )
      If IsObj( $oChildObj ) Then
        PrintElementInfo( $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )

    Else ; $vt = $VT_I4

      ; Child element
      $iChildElem = DllStructGetData( $tVarChildren, $i, 3 )
      PrintElementInfo( $oAcc, $iChildElem, $sIndent )




You need CUIAutomation2.au3 and MSAccessibility.au3.

If you get errors on double-defined constants, comment out the constants in MSAccessibility.au3.

Edited by LarsJ
1 person likes this

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • P3rsp3ctiv3
      By P3rsp3ctiv3
      Hello how can I send a {Enter} into the edit window ($cmd)? I tried it first with:
      GUICtrlSetData($cmd, "Line 1" + "{ENTER}")
      but it doesn´t work. Can someone help me please?
      #include <ButtonConstants.au3> #include <EditConstants.au3> #include <GUIConstantsEx.au3> #include <StaticConstants.au3> #include <WindowsConstants.au3> #include <GUIConstants.au3> $Admin = GUICreate("Admin", 234, 168, 192, 229, $GUI_SS_DEFAULT_GUI, BitOR($WS_EX_OVERLAPPEDWINDOW,$WS_EX_TRANSPARENT,$WS_EX_WINDOWEDGE)) GUISetBkColor(0xB4B4B4) $exit = GUICtrlCreateButton("Close", 160, 136, 65, 25) $cmd = GUICtrlCreateEdit("",0, 0, 225, 137, $ES_AUTOVSCROLL + $WS_VSCROLL) GUICtrlSetData(-1, "") GUICtrlSetFont(-1, 10, 400, 0, "Candara") GUICtrlSetBkColor(-1, 0xE3E3E3) $cmd_clear = GUICtrlCreateButton("Clear", 9, 136, 65, 25) GUISetState(@SW_SHOW) HotKeySet("{f1}", CMD_1) HotKeySet("{f2}", CMD_2) While 1 $msg = GUIGetMsg(1) Switch $msg[0] Case $GUI_EVENT_CLOSE Exit Case $exit GUIDelete($Admin) Exit EndSwitch WEnd Func CMD_1(); GUICtrlSetData($cmd, "Line 1" & "{ENTER}") GUICtrlSetData($cmd, "Line 2" & "{ENTER}") EndFunc Func CMD_2(); GUICtrlSetData($cmd, "Line 3" & "{ENTER}") GUICtrlSetData($cmd, "Line 4" & "{ENTER}") EndFunc
    • nacerbaaziz
      By nacerbaaziz
      please i need to link a progress bar with a time can you help me?
      i want to set a progress bar for 10 sec
      am waiting for your answers
      thank you.
    • AdminC
      By AdminC
      #include <SendMessage.au3>
      #include <WinAPI.au3>
      #include <WindowsConstants.au3>
      #include <GDIPlus.au3>
      #include <GUIConstantsEx.au3>
      #include <WinAPIGdi.au3>
      #include <Array.au3>
      #include <WinAPIDiag.au3>
      $Imglocation = @ScriptDir&"\Image\testimg.bmp"
      Local $Bitmap = _GDIPlus_BitmapCreateFromFile($Imglocation)
      Local $hBitmap = _GDIPlus_BitmapCreateHBITMAPFromBitmap($Bitmap)
      Local $imgSize = _WinAPI_GetBitmapDimension($hBitmap)
      Local $imgBits = DllStructCreate('dword[' & ($imgSize.X * $imgSize.Y) & ']')
      _WinAPI_GetBitmapBits($hBitmap, DllStructGetSize($imgBits), DllStructGetPtr($imgBits))
      MsgBox(0, "", DllStructGetData($imgBits, 1, 1)) 
      DllstructGetData can't get data color B, G, R value. Who's know that? teach me plz
    • Miliardsto
      By Miliardsto
      Hello. Im trying to make my scripts safe - unnable to decompile. I search for obfuscators and other security methods but the search has come to nothing.
      Then one guy gave that idea below. If I rightly understood this idea lets we talk about example program with this secutiy method.
      Program have two parts, first is only login gui and the second part is the main program Second part (main program) is uploaded on ftp server lets say that on http://xxx/autoit/main_program.au3 So we have the first gui with login, we put correctly login and pass and this is the moment when code from http://xxx/autoit/main_program.au3 will be downloaded and executed Finally main program will be appear This is the similiar way like new games are protected by cracking.
      I have few questions in this moment about this:
      Is something like that even possible to do with the autoit? First part of program (login gui) must have somewhere given that link to download the rest of code - http://xxx/autoit/main_program.au3 to make it execute. As we know this first part of program is easy able to hack and retrieve this web url http://xxx/autoit/main_program.au3 where located is main part of program. Is the way to encrypt or secure it? If only code will be stored in .php we know it cannot be previewed. So it could for example get code from .php file instead of .au3 I know that methods works in other languages (I dont know exactly how) thats becouse I only speculates, maybe something may looks different in these solution? Other way would be compiling second part of code on web server (there are available web autoit servers) maybe this way is possible? Tell me anything U know about this ideas and if its even possible to achieve.
      Thanks for ur any response, advice or thoughts

    • dubd83
      By dubd83
      I am trying to click on a tab within an SWT_Window0. The problem is autoit can not see the tab's text and there is no controlid. I have tried using commands to obtain a handle and text with no luck. Any ideas on how to get control of this tab?