JAVA object automation and simple spy

[drago1]

Hi,

thank you for this script. I would like to automate right click on table row.

I there possibility to perform "right click" action?

[junkew]

Not sure you could try getaccessibleactions but easiest is to get xywh area and use mousemove mouseclick autoi functions

[drago1]

Ok thanks, will try that.

One more thing, this table is not static. It shows like live status of the program, i.e. column hedaer "input status" can change to i.e.  "active", "completed".  Is there a way to read the actual status of the table?

 

[junkew]

Not doing anything with this anymore but in general study source code of java ferret and nvda accessibility sources.

[Moonscarlet]

@junkew thanks for directing me here, I followed the steps here but I am unable to get it to work correctly.

After editing the correct path for my installed java (C:\Program Files (x86)\Java\jre1.8.0_121) and "WindowsAccessBridge-32.dll", I also added "#AutoIt3Wrapper_UseX64=N", the script crashes after pressing CTRL+W even though there are no errors until I press it, could you please help me figure it out?

Java Home: C:\Program Files (x86)\Java\jre1.8.0_121
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
JAVAHOME=C:\Program Files (x86)\Java\jre1.8.0_121
The current working directory: 
D:\Autoit\UIA JavaWe are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=0
Warning: The script is advicable to be run with #requireadmin
Opening C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll
  PASS : Windows accessbridge C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll opened returns: 2
 0 initializeAccessBridge is finishedWindows_run passed :
mouse position retrieved
0!>21:17:41 AutoIt3.exe ended.rc:-1073741819
+>21:17:41 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 17.79

 

[junkew]

Maybe try other tools of internet that use Java Accessibility Bridge to see if it works on your machine
Try also the tools from Oracle. Its just not easy to get it directly working and debugging is even harder

https://github.com/google/access-bridge-explorer

https://docs.oracle.com/javase/7/docs/technotes/guides/access/enable_and_test.html

 

Edited April 22, 2022 by junkew

[Moonscarlet]

Thanks, I tried access bridge explorer and javamonkey, they are working but another problem is that they are only showing java control panel in the list and not detecting the system I am trying to work on.

I don't understand why the script is crashing like that.

[VAG]

I'm automating the UI of a 64bit Java app with this UDF, I'm able to button click on pushbutton element. So I proceed to select the combobox using _JAB_comboBoxSelectItem(). But I always get this return from the UDF. 

$__g_hwndJavaWindow = WinActivate("REW V5.20.9", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
$PreferencesButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Preferences", "push button")
_JAB_singleAction($__g_vmId, $PreferencesButton)
Sleep(1000)
$__g_hwndJavaWindow = WinActivate("Preferences", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
_JAB_comboBoxSelectItem($__g_vmId, "Select sample rate", "192 kHz")

Console print:

Java Home: C:\Program Files (x86)\Java\jre1.8.0_331
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_331\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
_JAB_comboBoxSelectItem: this element isn't combo box

So I use JAB explorer to check on the element, I noticed all combobox name property in this app only use "-" value. So I can only identify them with Description property.

Under JAVAUI.au3, there is "WCHAR description[1024];" defined in $tagAccessibleContextInfo. But _JAB_comboBoxSelectItem() is searching by name property.

How can I modify the __getAccessibleContextInfo() to search by Description or any suggesstion? Thanks.

Func __getAccessibleContextInfo($vmId, $ac, byref $info)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "getAccessibleContextInfo", "long", $vmId, $c_JOBJECT64, $ac, "struct*", $info)
    If @error Then Return SetError(1, 0, 0)
    $info = $result[3]
    Return $result[0]
EndFunc

 

 

[junkew]

Still not doing much with this today but I would look in detail in 

https://github.com/google/access-bridge-explorer

to see how they build up a list of controls

Based on that list you can iterate /traverse over it and for each control request the description and then filter out the ones you need.

probably functions like

The functions GetAccessibleChildFromContext and GetAccessibleParentFromContext enable you to walk the GUI component hierarchy, retrieving the nth child, or the parent, of a particular GUI object

Basically similar to what the explorer is doing it wraps the accessible context in an objectlist and adds "calculated" properties to each object and thats then used in building the tree and displaying the properties

private void AddChildrenProperties(PropertyList list, PropertyOptions options) {
      if ((options & PropertyOptions.Children) != 0) {
        var count = GetInfo().childrenCount;
        if (count > 0) {
          var group = list.AddGroup("Children");
          group.LoadChildren = () => {
            group.AddProperty("Count", count);
            Enumerable.Range(0, LimitSize(count)).ForEach(index => {
              var childHandle = AccessBridge.Functions.GetAccessibleChildFromContext(JvmId, _ac, index);
              var childNode = new AccessibleContextNode(AccessBridge, childHandle);
              var childGroup = group.AddGroup(string.Format("{0} [{1} of {2}]", childNode.GetTitle(), index + 1, count));
              childGroup.LoadChildren = () => {
                AddSubContextProperties(childGroup.Children, options, childNode);
              };
            });
          };
        }
      }

To help you have to make a simple reproducer but steps would be

for each control in controls

     if __getAccessibleContextInfo() description is match then return control

 

[VAG]

Thanks @junkew, I'm able to workout a funtion to traverse the JAB structure based on your recommendation:

;Search for page tab: Soundcard
    Local $Tab_Preferences_Soundcard = _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Soundcard", "page tab")
    ;child: panel
    Local $Panel_Preferences_Soundcard = _JAB_getFirstChildByRole($__g_vmId, $Tab_Preferences_Soundcard, "panel")

    ;Search all element in same level for description
    Local $iCount = _JAB_getChildrenCount($__g_vmId, $Panel_Preferences_Soundcard)
    For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($__g_vmId, $Panel_Preferences_Soundcard, $i)
        Local $child_ac_s1 = _JAB_getDescription($__g_vmId, $child_ac)
        If StringInStr($child_ac_s1 ,$Value) Then
            $re2 = $i
            ExitLoop
        EndIf
    Next
    return $child_ac

I also added this function in JAVAUI.au3 for getting description property:

Func _JAB_getDescription($vmId, $ac)
   Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
   __getAccessibleContextInfo($vmId, $ac, $acInfo)
   Return DllStructGetData($acInfo, "description")
EndFunc

Currently, I hit another issue with the Radio Button element. I noticed there is no singleAction function for this in JAVAUI.au3. May I know how do I send to this AccessibleInterface. Thanks

[junkew]

not sure what you ask but I assume you want to click or enable a radiobutton.

If you can get the bounds property you can use mousemove/mouseclick to the right location and do your regular mouseclick action.

Getting the state checked/unchecked is probably harder but also with the bounds you can easily do a pixelgetcolor or pixelgetchecksum or a little harder bitblt the area to get the details of all pixels.

 

[autotest410]

@junkew

I'm looking at the below code and not understanding the logic.

Func _JAB_getAccessibleContextByRole($vmId, $ac, $sRole, $sIndex)
   Local $find_ac = 0
   Local $iCount =_JAB_getChildrenCount($vmId, $ac)
   If $iCount = 0 Then
      Return
   EndIf
   For $i = 0 To $iCount - 1
      Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
      Local $s3 = _JAB_getRole($vmId, $child_ac)
;     consolewrite($child_ac & "|" & $s1 & "," & $s3 & @CRLF)
      If $s3 = $sRole Then
         $find_ac = $child_ac
         ExitLoop
      Else
         If $find_ac = 0 Then
            $find_ac = _JAB_getAccessibleContextByRole($vmId, $child_ac, $sRole, $sIndex)
         EndIf
      EndIf
   Next
   Return $find_ac
EndFunc

 

 

The above for loop iterated through each single node in the above tree. It exited the for loop from panel > page tab list > page tab > panel > panel > combo box > popup menu > scroll pane > push button (first instance) to panel > page tab list > page tab > panel > panel > text. The second exit happened from panel > page tab list > page tab > panel > panel > push button (first instance - Clear) to panel > page tab list > page tab (second instance). But I would like to return find_ac when I hit panel > page tab list > page tab > panel > panel > push button (Second instance - Search).

Please pour in suggestions.

TIA

[junkew]

You woud have to add a static variabele that counts the instancematch as long as its not a match you continue instead of exitloop.

Just add a bunch of consolewrites and remove exitloop to learn how the code works.

 

[mLipok]

@junkew please provide current UDF and example files. Maybe GitHub site ?

Edited August 20 by mLipok

[mLipok]

20 hours ago, mLipok said:

@junkew please provide current UDF and example files. Maybe GitHub site ?

@junkew

So far, I've done some minor research – initial tests and minor code modifications.

I intend to share my proposals, but I need to be sure I'm working on the latest version of what you have.

Therefore, please share the current files.

[junkew]

Its all in this thread and I can see I updated all in post 1.