Jump to content

JAVA object automation and simple spy


Recommended Posts

  • Replies 90
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

As a starter and far from perfect and real messy code a simple spy for Java Accessibility Bridge Based on the information in general help and support '?do=embed' frameborder='0' data-embedC

;~ ******* ;~ Every AccessibleContext IntPtr must be replaced by long, including but not limited to getAccessibleContextFromHWND, getAccessibleParentFromContext, getAccessibleChildFromConte

revalidated and still works with 1.8.0.66 and autoit 3.3.15.0 Mouse position is retrieved 1519-331 JAVA window found <> Java Window Title= Handle=0x0000000000040A88 res: 1

Posted Images

Ok thanks, will try that.

One more thing, this table is not static. It shows like live status of the program, i.e. column hedaer "input status" can change to i.e.  "active", "completed".  Is there a way to read the actual status of the table?

 

Link to post
Share on other sites
  • 1 year later...

@junkew thanks for directing me here, I followed the steps here but I am unable to get it to work correctly.

After editing the correct path for my installed java (C:\Program Files (x86)\Java\jre1.8.0_121) and "WindowsAccessBridge-32.dll", I also added "#AutoIt3Wrapper_UseX64=N", the script crashes after pressing CTRL+W even though there are no errors until I press it, could you please help me figure it out?

Java Home: C:\Program Files (x86)\Java\jre1.8.0_121
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
JAVAHOME=C:\Program Files (x86)\Java\jre1.8.0_121
The current working directory: 
D:\Autoit\UIA JavaWe are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=0
Warning: The script is advicable to be run with #requireadmin
Opening C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll
  PASS : Windows accessbridge C:\Program Files (x86)\Java\jre1.8.0_121\bin\WindowsAccessBridge-32.dll opened returns: 2
 0 initializeAccessBridge is finishedWindows_run passed :
mouse position retrieved
0!>21:17:41 AutoIt3.exe ended.rc:-1073741819
+>21:17:41 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 17.79

 

Link to post
Share on other sites
Posted (edited)

Maybe try other tools of internet that use Java Accessibility Bridge to see if it works on your machine
Try also the tools from Oracle. Its just not easy to get it directly working and debugging is even harder

https://github.com/google/access-bridge-explorer

https://docs.oracle.com/javase/7/docs/technotes/guides/access/enable_and_test.html

 

Edited by junkew
Link to post
Share on other sites

Thanks, I tried access bridge explorer and javamonkey, they are working but another problem is that they are only showing java control panel in the list and not detecting the system I am trying to work on.

I don't understand why the script is crashing like that.

Link to post
Share on other sites
  • 2 months later...

I'm automating the UI of a 64bit Java app with this UDF, I'm able to button click on pushbutton element. So I proceed to select the combobox using _JAB_comboBoxSelectItem(). But I always get this return from the UDF. 

$__g_hwndJavaWindow = WinActivate("REW V5.20.9", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
$PreferencesButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Preferences", "push button")
_JAB_singleAction($__g_vmId, $PreferencesButton)
Sleep(1000)
$__g_hwndJavaWindow = WinActivate("Preferences", "")
__getAccessibleContextFromHWND($__g_hwndJavaWindow, $__g_vmId, $__g_acJavaMainWindow) ;~    Get our vmid into the global variable
_JAB_comboBoxSelectItem($__g_vmId, "Select sample rate", "192 kHz")

Console print:

Java Home: C:\Program Files (x86)\Java\jre1.8.0_331
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files (x86)\Java\jre1.8.0_331\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
_JAB_comboBoxSelectItem: this element isn't combo box

So I use JAB explorer to check on the element, I noticed all combobox name property in this app only use "-" value. So I can only identify them with Description property.

826231621_JABExplorer.jpg.54df81b61092141f24d2ac5345ab80ea.jpg

Under JAVAUI.au3, there is "WCHAR description[1024];" defined in $tagAccessibleContextInfo. But _JAB_comboBoxSelectItem() is searching by name property.

How can I modify the __getAccessibleContextInfo() to search by Description or any suggesstion? Thanks.

Func __getAccessibleContextInfo($vmId, $ac, byref $info)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "getAccessibleContextInfo", "long", $vmId, $c_JOBJECT64, $ac, "struct*", $info)
    If @error Then Return SetError(1, 0, 0)
    $info = $result[3]
    Return $result[0]
EndFunc

 

 

Link to post
Share on other sites

Still not doing much with this today but I would look in detail in 

https://github.com/google/access-bridge-explorer

to see how they build up a list of controls

Based on that list you can iterate /traverse over it and for each control request the description and then filter out the ones you need.

probably functions like

The functions GetAccessibleChildFromContext and GetAccessibleParentFromContext enable you to walk the GUI component hierarchy, retrieving the nth child, or the parent, of a particular GUI object

Basically similar to what the explorer is doing it wraps the accessible context in an objectlist and adds "calculated" properties to each object and thats then used in building the tree and displaying the properties

private void AddChildrenProperties(PropertyList list, PropertyOptions options) {
      if ((options & PropertyOptions.Children) != 0) {
        var count = GetInfo().childrenCount;
        if (count > 0) {
          var group = list.AddGroup("Children");
          group.LoadChildren = () => {
            group.AddProperty("Count", count);
            Enumerable.Range(0, LimitSize(count)).ForEach(index => {
              var childHandle = AccessBridge.Functions.GetAccessibleChildFromContext(JvmId, _ac, index);
              var childNode = new AccessibleContextNode(AccessBridge, childHandle);
              var childGroup = group.AddGroup(string.Format("{0} [{1} of {2}]", childNode.GetTitle(), index + 1, count));
              childGroup.LoadChildren = () => {
                AddSubContextProperties(childGroup.Children, options, childNode);
              };
            });
          };
        }
      }

To help you have to make a simple reproducer but steps would be

for each control in controls

     if __getAccessibleContextInfo() description is match then return control

 

Link to post
Share on other sites

Thanks @junkew, I'm able to workout a funtion to traverse the JAB structure based on your recommendation:

;Search for page tab: Soundcard
    Local $Tab_Preferences_Soundcard = _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Soundcard", "page tab")
    ;child: panel
    Local $Panel_Preferences_Soundcard = _JAB_getFirstChildByRole($__g_vmId, $Tab_Preferences_Soundcard, "panel")

    ;Search all element in same level for description
    Local $iCount = _JAB_getChildrenCount($__g_vmId, $Panel_Preferences_Soundcard)
    For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($__g_vmId, $Panel_Preferences_Soundcard, $i)
        Local $child_ac_s1 = _JAB_getDescription($__g_vmId, $child_ac)
        If StringInStr($child_ac_s1 ,$Value) Then
            $re2 = $i
            ExitLoop
        EndIf
    Next
    return $child_ac

I also added this function in JAVAUI.au3 for getting description property:

Func _JAB_getDescription($vmId, $ac)
   Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
   __getAccessibleContextInfo($vmId, $ac, $acInfo)
   Return DllStructGetData($acInfo, "description")
EndFunc

Currently, I hit another issue with the Radio Button element. I noticed there is no singleAction function for this in JAVAUI.au3. May I know how do I send to this AccessibleInterface. Thanks

385974749_radiobuttonaccessibletree.thumb.jpg.40672515f8165f40e3608dcf34542ed8.jpg

Link to post
Share on other sites

not sure what you ask but I assume you want to click or enable a radiobutton.

If you can get the bounds property you can use mousemove/mouseclick to the right location and do your regular mouseclick action.

Getting the state checked/unchecked is probably harder but also with the bounds you can easily do a pixelgetcolor or pixelgetchecksum or a little harder bitblt the area to get the details of all pixels.

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...