Jump to content
automaterk

AutoIT and MSAA

Recommended Posts

automaterk

Good afternoon all!

I am in the process of revisiting some of my previous issues now that I have a better understanding of AutoIt and have made some decent strides in other areas of my automation. Originally I had(and still do) an issue with reading ListItems from a list box within my application. This is the only control I could not get a good read on. Even hidden elements, etc. were visible when given the right script.

Fast forward to today and I just happened to download UI Accessibility checker. Previously, nothing I've tried has been able to see more than just the list box. However, this application identifies each name in the list in the tree view. This has given me hope that there may be a solution to automate this.

You can see the control and the list tree below:

Capture.thumb.JPG.a0deaca9da5adfd45aa863

I've done a bit of research in the forums, but haven't seen anything with definitive evidence that this can be done with AutoIt. I am currently attempting to do everything in AutoItx4Java and using base AutoIt commands, which has worked well so far. Does anyone have any insight into how I might be able to interact with these elements in AutoIt or a java based solution that I might try. This would really perfect the project I've been working on and any help would be greatly appreciated.

I have looked into the UIA thread, but it doesn't seem to be as useful here as UIA inspection does not return any results.

Thank you for any and all replies!

Share this post


Link to post
Share on other sites
junkew

did you try simplespy and inspect as described in the uia thread?
whats the output of those tools, as you use the UI Accessibility checker I am 99.99% sure that inspect and simplespy give you nformation and that way you can automate.

This thread can also help but I did not find time to enhance / investigate further. (what you post seems to be a Java application)
this makes use of the JavaAccessibilityBridge and as such called it JABSimpleSpy.
https://www.autoitscript.com/forum/topic/166830-java-object-automation-and-simple-spy/

  • Like 1

Share this post


Link to post
Share on other sites
LarsJ

As far as I know the UI Accessibility checker (AccCheckUI.exe) is the predecessor of Inspect.exe. Both programs are based on two methods to identify windows and controls: MSAA-techniques and UIA-techniques. Because AccCheckUI.exe is the old program I think it's primarily based on MSAA-techniques. And the new program Inspect.exe is primarily based on UIA-techniques.

If AccCheckUI.exe can identify the list items, Inspect.exe should also be able to identify the list items. I remember you didn't have much success with Inspect.exe in the old thread. But did you try to run Inspect.exe in pure MSAA mode (combo box in upper left corner)?

If you can verify that the list items can be identified with MSAA-techniques, there can be a chance to be able to automate the list box with MSAA-code (instead of UIA-code).

  • Like 1

Share this post


Link to post
Share on other sites
automaterk

Sorry for the late reply - I got started on something else this week and am just now coming back to all of this. Thanks for getting back with me on this - I was hoping both of you would respond!

@junkew Using Simple UIA Spy I get the following result:

Mouse position is retrieved 1006-544
At least we have an element [][]

Having the following values for all properties: 
Title is: <>    Class   := <>   controltype:= <UIA_ListItemControlTypeId>   ,<50007>    , (0000C357)    0;1;82524432;536870911
*** Parent Information top down ***
8: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1080
"Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769"   
7: Title is: <MEDHOST>  Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    449;167;1022;746
"Title:=MEDHOST;controltype:=UIA_WindowControlTypeId;class:=LSPANEL"    
6: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;362;591;356
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
5: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    664;389;591;302
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
4: Title is: <Setlib>   Class   := <LSPANEL>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    677;366;591;352
"Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL" 
3: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    685;396;4096;4096
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"  
2: Title is: <Select Library List to Copy>  Class   := <LSLISTBOX>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    718;456;451;179
"Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX"    
1: Title is: <> Class   := <LSLISTSUB>  controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;457;436;177
"Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTSUB"   
0: Title is: <> Class   := <LSLISTLIST> controltype:= <UIA_ListControlTypeId>   ,<50008>    , (0000C358)    719;474;4096;160
"Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTLIST"  


;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP7=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=MEDHOST;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)    
_UIA_Action($oP7,"setfocus")
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP6,"setfocus")
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=Setlib;controltype:=UIA_WindowControlTypeId;class:=LSPANEL", $treescope_children)  
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=Select Library List to Copy;controltype:=UIA_ListControlTypeId;class:=LSLISTBOX", $treescope_children) 
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTSUB", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_ListControlTypeId;class:=LSLISTLIST", $treescope_children)   
_UIA_Action($oP0,"setfocus")
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_ListItemControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <1>
UIA_handle:= <0>
UIA_RuntimeId:= <42;3672536;2;-2147483647;3672536;-4;1>
UIA_BoundingRectangle:= <0;1;82524432;536870911>
; Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

I've attempted to use the Java Spy, but it doesn't see anything on that application. I confirmed it worked with an actual java application.

 

@larsj

Thanks for the tip on inspect.exe - I didn't realize that was an option. Here are the results, which seem to be promising.

Capture.thumb.JPG.4d6cd9f5e5a67ce2437c05Capture.thumb.JPG.a78569d5a51b5f46d52c0c

From this information and your understanding - is there any way to use AutoIt to accomplish this? Or will I most likely need to look into a Java solution for MSAA(Which so far has not been very fruitful).

I'm not really sure where to go from here if it is possible. None of my searching threads here has really pulled anything on how to use MSAA if you can. Anything to get me started would be greatly appreciated. As always, thank you so much for your help and if you need any more info please let me know!

Share this post


Link to post
Share on other sites
junkew

As inspect.exe gives you the result / value you are looking for you do not have to go to a java solution.

But I do not have an out of the box solution as I expected that the simplespy should give the same result as inspect.exe

  • option 1 fix simple spy or based on the standard code as given by simplespy get your information with the treewalker (see examples1-3)
  • option 2 more difficult but I assume 1 can work although if you see the data in inspect.exe with MSAA option turned on than this is the way to go
    If you read the thread from start to end you have all the pieces of IUIAutomation, IAccessible, IAccessible2, ISimpledomand in that there will be a name value with the right content.
    This is the area where LarsJ provided the IAccessible ... stuff (they are also in the zip of the first post)
    https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/?do=findComment&comment=1156373

 

  • Like 1

Share this post


Link to post
Share on other sites
automaterk

Thanks for getting back to me. I thought I would go with option 2 first as it seems like the most promising, even if it is more difficult. I have all of the files in place and have searched the thread, but I am getting issues with the code you linked me to.

"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(53,40) : error: $OBJID_WINDOW previously declared as a 'Const'.
Global Const $OBJID_WINDOW = 0x00000000
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(54,42) : error: $OBJID_TITLEBAR previously declared as a 'Const'.
Global Const $OBJID_TITLEBAR = 0xFFFFFFFE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(55,42) : error: $OBJID_SIZEGRIP previously declared as a 'Const'.
Global Const $OBJID_SIZEGRIP = 0xFFFFFFF9
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(56,39) : error: $OBJID_CARET previously declared as a 'Const'.
Global Const $OBJID_CARET = 0xFFFFFFF8
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(57,40) : error: $OBJID_CURSOR previously declared as a 'Const'.
Global Const $OBJID_CURSOR = 0xFFFFFFF7
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(58,39) : error: $OBJID_ALERT previously declared as a 'Const'.
Global Const $OBJID_ALERT = 0xFFFFFFF6
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3"(59,39) : error: $OBJID_SOUND previously declared as a 'Const'.
Global Const $OBJID_SOUND = 0xFFFFFFF5
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Users\Keven.Reed\Desktop\iAccessible1.au3 - 7 error(s), 0 warning(s)

I don't see where it is referencing these aside from in the Constants file. I was trying to search around AutoIt and Google and couldn't find anything about these specific variables. Is this something you have faced or that you might know how to get around?

Thanks!

 

EDIT: Also - to go ahead and grab you while you're here - I am guessing from what I've read that this will return a way of identifying the elements in the list with MSAA. I am assuming that next I will be able to use the UIA commands you've written to interact with this list in AutoIt? Sorry if I am confused on something, but wanted to go ahead and check.

Edited by automaterk

Share this post


Link to post
Share on other sites
junkew

you should get a better text editor for searching text in au3 files in directories

  • msaccessibility.au3
  • msactiveaccessibility.au3

you can uncomment it there as it now seems to be in AutoItConstants nowadays.

I did not integrate (yet) anything from MSAA into the UIA framework commands.  so no UIA commands as in the UIAWrappers will not work but the plain CUIAutomation stuff will work also with MSAA, study the example from LarsJ and hopefully you grasp the concept.

 

Share this post


Link to post
Share on other sites
LarsJ

automaterk, This script based on MSAA-code prints information in SciTE console about all elements it can find in a window. You have to add a line in top of script to get window handle. Let's see whether the code can find the listbox elements.

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )

Example()


Func Example()

  ; Get window handle
  ;Local $hWindow = WinGetHandle( "[CLASS:Chrome_WidgetWin_1]", "" ) ; Chrome            ; http://tweepi.com/auth/login
  ;Local $hWindow = WinGetHandle( "[CLASS:MozillaWindowClass]", "" ) ; Mozilla Firefox
  ;Local $hWindow = WinGetHandle( "[CLASS:IEFrame]" )                ; Internet Explorer
  ;Local $hWindow = WinGetHandle( "[CLASS:CabinetWClass]" )          ; Windows Explorer, Vista, 7, 8
  ;Local $hWindow = WinGetHandle( "[CLASS:ExploreWClass]" )          ; Windows Explorer, Windows XP
  Local $hWindow = WinGetHandle( "Calculator" )                     ; Calculator
  ;Local $hWindow = ControlGetHandle("[CLASS:Notepad]", "", "Edit1") ; Notepad
  ;Local $hWindow = 0x000403DA                                       ; Chrome            ; http://tweepi.com/auth/login
  If Not $hWindow Then Return ConsoleWrite( "Window handle ERR" & @CRLF )
  ConsoleWrite( "Window handle OK" & @CRLF )

  ; Get Window object
  Local $pWindow, $oWindow
  AccessibleObjectFromWindow( $hWindow, $OBJID_CLIENT, $tIID_IAccessible, $pWindow )
  $oWindow = ObjCreateInterface( $pWindow, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oWindow ) Then Return ConsoleWrite( "Window object ERR" & @CRLF )
  ConsoleWrite( "Window object OK" & @CRLF )

  ; Window name
  Local $sName
  ConsoleWrite( @CRLF & "Window:" )
  ConsoleWrite( @CRLF & "-------" & @CRLF )
  PrintElementInfo( $oWindow, $CHILDID_SELF, "" )

  ; Get children
  ConsoleWrite( "Children:" & @CRLF )
  ConsoleWrite( "---------" & @CRLF )
  WalkTreeWithAccessibleChildren( $pWindow, 0 )

EndFunc


Func WalkTreeWithAccessibleChildren( $pAcc, $iLevel, $iLevels = 0 )

  If $iLevels And $iLevel = $iLevels Then Return

  ; Create object
  Local $oAcc = ObjCreateInterface( $pAcc, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAcc ) Then Return
  $oAcc.AddRef()

  Local $iChildCount, $iReturnCount, $tVarChildren

  ; Get children
  If $oAcc.get_accChildCount( $iChildCount ) Or Not $iChildCount Then Return
  If AccessibleChildren( $pAcc, 0, $iChildCount, $tVarChildren, $iReturnCount ) Then Return

  ; Indentation
  Local $sIndent = ""
  For $i = 0 To $iLevel - 1
    $sIndent &= "    "
  Next

  Local $hWnd
  WindowFromAccessibleObject( $pAcc, $hWnd )
  ConsoleWrite( $sIndent & "Window  = " & $hWnd & @CRLF & @CRLF )

  Local $vt, $pChildObj, $oChildObj, $iChildElem

  ; For each child
  For $i = 1 To $iReturnCount

    ; $tVarChildren is an array of VARIANTs with information about the children
    $vt = BitAND( DllStructGetData( $tVarChildren, $i, 1 ), 0xFFFF )

    If $vt = $VT_DISPATCH Then

      ; Child object
      $pChildObj = DllStructGetData( $tVarChildren, $i, 3 )
      $oChildObj = ObjCreateInterface( $pChildObj, $sIID_IAccessible, $dtagIAccessible )
      If IsObj( $oChildObj ) Then
        PrintElementInfo( $oChildObj, $CHILDID_SELF, $sIndent )
        WalkTreeWithAccessibleChildren( $pChildObj, $iLevel + 1, $iLevels )
      EndIf

    Else ; $vt = $VT_I4

      ; Child element
      $iChildElem = DllStructGetData( $tVarChildren, $i, 3 )
      PrintElementInfo( $oAcc, $iChildElem, $sIndent )

    EndIf

  Next

EndFunc

You need CUIAutomation2.au3 and MSAccessibility.au3.

If you get errors on double-defined constants, comment out the constants in MSAccessibility.au3.

Edited by LarsJ
  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • JuanFelipe
      By JuanFelipe
      Hello friends, I have a simple question, can I have a variable that is not reset, even if my program is closed?
      That is, I want to make a program that can only be opened once, have a variable $ ini = 0, when it is opened for the first and only time it becomes $ ini = 1, so that it can not use the program a second time, as if it was a test software.
      I do not want the variable to depend on a reading of an external file that is easily manipulated. Thank you.
    • nooneclose
      By nooneclose
      My program has to first search for names in Column D that do not match up with column C. I got that search to work using arrays. It was slow and I could not figure out how to delete them so I just manually put coded the names that do not belong. I found their cell location but I do not know how to store that location and delete it.
      This is what I have so far.
      Local $NameToDelete1[6]  = _Excel_RangeFind($OpenWorkbook, "Smith, Bill") _ArrayDisplay($NameToDelete1, "Excel UDF: _Excel_RangeFind Example 1", "", 0, "|", "Sheet|Name|Cell|Value|Formula|Comment") _Excel_RangeDelete($OpenWorkbook.ActiveSheet, $NameToDelete1[2], $xlShiftUp)  
      Please help, I wanted to have this program done yesterday but I did not see this problem until yesterday. 
    • TrashBoat
      By TrashBoat
      So Im trying to make a simple 2d game and make some sort of collision detection so why not to make a 2 dimensional array but i have no clue how  to write it in multiple lines
      Global $map[5,5] = [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0] something like this but it doesn't work
    • Zein
      By Zein
      #include "..\Include\Array.au3" #include "..\Include\File.au3" #include "..\Include\AutoItConstants.au3" Local $aRetArray Local $sFilePath = "n.csv" _FileReadToArray($sFilePath, $aRetArray, ",") ; _FileReadToArray($sFilePath, $aRetArray, $FRTA_COUNT, ",") _ArrayDisplay($aRetArray, "Original", Default, 8) The above code shows two versions of _FileReadToArray and both don't work as expected.
      The first one doesn't use the comma as a delimiter. (so I get a single column array)  I tried adding "Default" between $aRetArray and "," then it told me it had an incorrect number of parameters. 

      I looked again at the documentation:
       
      #include <File.au3> _FileReadToArray ( $sFilePath, ByRef $vReturn [, $iFlags = $FRTA_COUNT [, $sDelimiter = ""]] )
      And I with or without the flags params I should be getting a 2D array due to my file being a csv. 
      I then tried a regular flag, $FRTA_COUNT, and it tells me that I'm using a variable $FRTA_COUNT while it's not declared. Tried putting in 1 instead and it told me again, incorrect number of params. 

       
    • nooneclose
      By nooneclose
      I need to perform a subtotal in excel and I would like to automate this process using Autoit if possible like always any and all help will be greatly appreciated. 
      I can not find a good example but the two from Microsoft. Here is one of the two from msdn.microsoft.com/en-us/vba/excel-vba/articles/range-subtotal-method-excel
      I do not really understand how to translate this into AutoIt, but I gave it a try and here is what I have.
      $OpenRange      = "A1:E200" $xlSum          = -4157 $Added_Array[2] = [2, 3] $OpenRange.Subtotal("B1", $xlSum, $Added_Array, True, False, True) I just need to perform a subtotal on a range based on a header called department, and then perform a sum on the results.
×