Jump to content

JAVA object automation and simple spy


Recommended Posts

Ok thanks, will try that.

One more thing, this table is not static. It shows like live status of the program, i.e. column hedaer "input status" can change to i.e.  "active", "completed".  Is there a way to read the actual status of the table?


Link to comment
Share on other sites

  • 1 year later...

@junkew thanks for directing me here, I followed the steps here but I am unable to get it to work correctly.

After editing the correct path for my installed java (C:\Program Files (x86)\Java\jre1.8.0_121) and "WindowsAccessBridge-32.dll", I also added "#AutoIt3Wrapper_UseX64=N", the script crashes after pressing CTRL+W even though there are no errors until I press it, could you please help me figure it out?

Java Home: C:\Program Files (x86)\Java\jre1.8.0_121
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
JAVAHOME=C:\Program Files (x86)\Java\jre1.8.0_121
The current working directory: 
D:\Autoit\UIA JavaWe are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=0
Warning: The script is advicable to be run with #requireadmin
Opening C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll
  PASS : Windows accessbridge C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll opened returns: 2
 0 initializeAccessBridge is finishedWindows_run passed :
mouse position retrieved
0!>21:17:41 AutoIt3.exe ended.rc:-1073741819
+>21:17:41 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 17.79


Link to comment
Share on other sites

Maybe try other tools of internet that use Java Accessibility Bridge to see if it works on your machine
Try also the tools from Oracle. Its just not easy to get it directly working and debugging is even harder




Edited by junkew
Link to comment
Share on other sites

Thanks, I tried access bridge explorer and javamonkey, they are working but another problem is that they are only showing java control panel in the list and not detecting the system I am trying to work on.

I don't understand why the script is crashing like that.

Link to comment
Share on other sites

  • 2 months later...

I'm automating the UI of a 64bit Java app with this UDF, I'm able to button click on pushbutton element. So I proceed to select the combobox using _JAB_comboBoxSelectItem(). But I always get this return from the UDF. 

$__g_hwndJavaWindow = WinActivate("REW V5.20.9", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
$PreferencesButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Preferences", "push button")
_JAB_singleAction($__g_vmId, $PreferencesButton)
$__g_hwndJavaWindow = WinActivate("Preferences", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
_JAB_comboBoxSelectItem($__g_vmId, "Select sample rate", "192 kHz")

Console print:

Java Home: C:\Program Files (x86)\Java\jre1.8.0_331
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_331\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
_JAB_comboBoxSelectItem: this element isn't combo box

So I use JAB explorer to check on the element, I noticed all combobox name property in this app only use "-" value. So I can only identify them with Description property.


Under JAVAUI.au3, there is "WCHAR description[1024];" defined in $tagAccessibleContextInfo. But _JAB_comboBoxSelectItem() is searching by name property.

How can I modify the __getAccessibleContextInfo() to search by Description or any suggesstion? Thanks.

Func __getAccessibleContextInfo($vmId, $ac, byref $info)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "getAccessibleContextInfo", "long", $vmId, $c_JOBJECT64, $ac, "struct*", $info)
    If @error Then Return SetError(1, 0, 0)
    $info = $result[3]
    Return $result[0]



Link to comment
Share on other sites

Still not doing much with this today but I would look in detail in 


to see how they build up a list of controls

Based on that list you can iterate /traverse over it and for each control request the description and then filter out the ones you need.

probably functions like

The functions GetAccessibleChildFromContext and GetAccessibleParentFromContext enable you to walk the GUI component hierarchy, retrieving the nth child, or the parent, of a particular GUI object

Basically similar to what the explorer is doing it wraps the accessible context in an objectlist and adds "calculated" properties to each object and thats then used in building the tree and displaying the properties

private void AddChildrenProperties(PropertyList list, PropertyOptions options) {
      if ((options & PropertyOptions.Children) != 0) {
        var count = GetInfo().childrenCount;
        if (count > 0) {
          var group = list.AddGroup("Children");
          group.LoadChildren = () => {
            group.AddProperty("Count", count);
            Enumerable.Range(0, LimitSize(count)).ForEach(index => {
              var childHandle = AccessBridge.Functions.GetAccessibleChildFromContext(JvmId, _ac, index);
              var childNode = new AccessibleContextNode(AccessBridge, childHandle);
              var childGroup = group.AddGroup(string.Format("{0} [{1} of {2}]", childNode.GetTitle(), index + 1, count));
              childGroup.LoadChildren = () => {
                AddSubContextProperties(childGroup.Children, options, childNode);

To help you have to make a simple reproducer but steps would be

for each control in controls

     if __getAccessibleContextInfo() description is match then return control


Link to comment
Share on other sites

Thanks @junkew, I'm able to workout a funtion to traverse the JAB structure based on your recommendation:

;Search for page tab: Soundcard
    Local $Tab_Preferences_Soundcard = _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Soundcard", "page tab")
    ;child: panel
    Local $Panel_Preferences_Soundcard = _JAB_getFirstChildByRole($__g_vmId, $Tab_Preferences_Soundcard, "panel")

    ;Search all element in same level for description
    Local $iCount = _JAB_getChildrenCount($__g_vmId, $Panel_Preferences_Soundcard)
    For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($__g_vmId, $Panel_Preferences_Soundcard, $i)
        Local $child_ac_s1 = _JAB_getDescription($__g_vmId, $child_ac)
        If StringInStr($child_ac_s1 ,$Value) Then
            $re2 = $i
    return $child_ac

I also added this function in JAVAUI.au3 for getting description property:

Func _JAB_getDescription($vmId, $ac)
   Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
   __getAccessibleContextInfo($vmId, $ac, $acInfo)
   Return DllStructGetData($acInfo, "description")

Currently, I hit another issue with the Radio Button element. I noticed there is no singleAction function for this in JAVAUI.au3. May I know how do I send to this AccessibleInterface. Thanks


Link to comment
Share on other sites

not sure what you ask but I assume you want to click or enable a radiobutton.

If you can get the bounds property you can use mousemove/mouseclick to the right location and do your regular mouseclick action.

Getting the state checked/unchecked is probably harder but also with the bounds you can easily do a pixelgetcolor or pixelgetchecksum or a little harder bitblt the area to get the details of all pixels.


Link to comment
Share on other sites

  • 3 months later...


I'm looking at the below code and not understanding the logic.

Func _JAB_getAccessibleContextByRole($vmId, $ac, $sRole, $sIndex)
   Local $find_ac = 0
   Local $iCount =_JAB_getChildrenCount($vmId, $ac)
   If $iCount = 0 Then
   For $i = 0 To $iCount - 1
      Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
      Local $s3 = _JAB_getRole($vmId, $child_ac)
;     consolewrite($child_ac & "|" & $s1 & "," & $s3 & @CRLF)
      If $s3 = $sRole Then
         $find_ac = $child_ac
         If $find_ac = 0 Then
            $find_ac = _JAB_getAccessibleContextByRole($vmId, $child_ac, $sRole, $sIndex)
   Return $find_ac



The above for loop iterated through each single node in the above tree. It exited the for loop from panel > page tab list > page tab > panel > panel > combo box > popup menu > scroll pane > push button (first instance) to panel > page tab list > page tab > panel > panel > text. The second exit happened from panel > page tab list > page tab > panel > panel > push button (first instance - Clear) to panel > page tab list > page tab (second instance). But I would like to return find_ac when I hit panel > page tab list > page tab > panel > panel > push button (Second instance - Search).

Please pour in suggestions.


Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...