Jump to content

JAVA object automation and simple spy


Recommended Posts

  • Replies 90
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

As a starter and far from perfect and real messy code a simple spy for Java Accessibility Bridge Based on the information in general help and support '?do=embed' frameborder='0' data-embedC

;~ ******* ;~ Every AccessibleContext IntPtr must be replaced by long, including but not limited to getAccessibleContextFromHWND, getAccessibleParentFromContext, getAccessibleChildFromConte

revalidated and still works with 1.8.0.66 and autoit 3.3.15.0 Mouse position is retrieved 1519-331 JAVA window found <> Java Window Title= Handle=0x0000000000040A88 res: 1

Posted Images

updated first post with @fablecao javaui.au3 code and updated jabsimplespy to use that

trouble on x64 to find correct path so hardcoded in javaui.au3

If $sJavaHome = "" Then

      consolewrite("  Unable to find Java Registry using hardcoded value" & @CRLF)
      $sJavaHome = "C:\Program Files\Java\jdk1.8.0_172\jre"

   EndIf
  • in jabsimplespy source there is also an example function that iterates all java reconizable objects
  • added highlighting when spying on object
    • shows also in more detail how to get information about an identified element

 

This is just an example on how to click on the About button similar as was given by @fablecao

  • #include "JavaUI.au3"
    shellexecute("C:\Program Files\Java\jdk1.8.0_172\jre\bin\javacpl.exe")
    sleep(3000)
    
    global $wintitle = "Java Control Panel"
    
    $winHandle = WinActivate($wintitle)
    
    $result = __isJavaWindow($winHandle)
    
    Global $vmId
    Global $ac
    
    global $sName = "About..."
    global $sRole = "push button"
    
    __getAccessibleContextFromHWND($winHandle, $vmID, $ac)
    
    $re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
    
    $sName = _JAB_getName($vmId, $re_ac)
    
    Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
    __getAccessibleContextInfo($vmId, $re_ac, $acInfo)
    $acX = DllStructGetData($acInfo, "x")
    $acY = DllStructGetData($acInfo, "y")
    $acW = DllStructGetData($acInfo, "width")
    $acH = DllStructGetData($acInfo, "height")
    
    ConsoleWrite("Name:" & $sName & @CRLF)
    ConsoleWrite("rect:" & $acX & $acX+$acW & $acy & $acY+$acH & @CRLF)
    
    
    $re_ac = _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)
    $sName = _JAB_getName($vmId, $re_ac)
    
    Local $acInfo2 = DllStructCreate($tagAccessibleContextInfo)
    __getAccessibleContextInfo($vmId, $re_ac, $acInfo2)
    $acX = DllStructGetData($acInfo2, "x")
    $acY = DllStructGetData($acInfo2, "y")
    $acW = DllStructGetData($acInfo2, "width")
    $acH = DllStructGetData($acInfo2, "height")
    ConsoleWrite("Name:" & $sName & @CRLF)
    ConsoleWrite("rect:" & $acX & $acX+$acW & $acy & $acY+$acH & @CRLF)
    
    _JAB_singleAction($vmId, $re_ac)
    
    
    ;~ Must use the AccessibleContext got from the win handle which include the target element
    Func _JAB_getAccessibleContextByRole($vmId, $ac, $sRole, $sIndex)
       Local $find_ac = 0
       Local $iCount =_JAB_getChildrenCount($vmId, $ac)
       If $iCount = 0 Then
          Return
       EndIf
       For $i = 0 To $iCount - 1
          Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
          Local $s3 = _JAB_getRole($vmId, $child_ac)
    ;     consolewrite($child_ac & "|" & $s1 & "," & $s3 & @CRLF)
          If $s3 = $sRole Then
             $find_ac = $child_ac
             ExitLoop
          Else
             If $find_ac = 0 Then
                $find_ac = _JAB_getAccessibleContextByRole($vmId, $child_ac, $sRole, $sIndex)
             EndIf
          EndIf
       Next
       Return $find_ac
    EndFunc

     

 

@rlvitorino in above example another recursive function to find by role, by index is not there but I made a start in that function

 

Link to post
Share on other sites

setvalue example:

Func __setTextContents($vmId, $ac, $text)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "setTextContents", "long", $vmId, $c_JOBJECT64, $ac, "wstr", $text)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc

Java access bridge native interface function __setTextContents can do this.

Due to some bug of early version of JRE, this native function takes no effect.  So I wrote an alternative function.

Func _JAB_setValue($vmId, $ac, $sValue)
   Local $actions = DllStructCreate($tagAccessibleActions)
   __getAccessibleActions($vmId, $ac, $actions)
   Local $s1 = DllStructGetData($actions, "actionsCount")
   If $s1 = 0 Then
      consolewrite("_JAB_setValue: this element has no action" & @CRLF)
      Return
   EndIf
   Local $re1 = 0
   Local $re2 = 0
   For $i = 2 To $s1 + 1
      Local $s = DllStructGetData($actions, $i)
      If $s = "select-all" Then
         $re1 = 1
         ExitLoop
      EndIf
   Next
   If $re1 = 0 Then
      consolewrite("_JAB_setValue: this element doesn't support select-all action" & @CRLF)
      Return
   EndIf
   For $i = 2 To $s1 + 1
      Local $s = DllStructGetData($actions, $i)
      If $s = "paste-from-clipboard" Then
         $re2 = 1
         ExitLoop
      EndIf
   Next
   If $re2 = 0 Then
      consolewrite("_JAB_setValue: this element doesn't support paste-from-clipboard action" & @CRLF)
      Return
   EndIf
   Local $actionsToDo = DllStructCreate($tagAccessibleActionsToDo)
   Local $failure
   ClipPut($sValue)
   DllStructSetData($actionsToDo, "actionsCount", 2)
   DllStructSetData($actionsToDo, 2, "select-all")
   DllStructSetData($actionsToDo, 3, "paste-from-clipboard")
   __doAccessibleActions($vmId, $ac, $actionsToDo, $failure)
EndFunc

Structure $actionsToDo, first member is actions count "2", second member is first action "select-all", third member is second action "paste-from-clipboard".

Link to post
Share on other sites
;=====================================AccessibleAction================================
Global Const $MAX_ACTION_INFO = 256
Global Const $MAX_ACTIONS_TO_DO = 32

Global Const $tagAccessibleActionInfo = _
"WCHAR name[256]"

Local $tag_ActionInfo = ""
For $i = 1 To 256
   If $tag_ActionInfo = "" Then
      $tag_ActionInfo = $tagAccessibleActionInfo
   Else
      $tag_ActionInfo = $tag_ActionInfo & ";" & $tagAccessibleActionInfo
   EndIf
Next
Global Const $tagAccessibleActions = _
"INT actionsCount;" & _
$tag_ActionInfo                     ; $tAccessibleActionInfo actionInfo[256]

Local $tag_Actions = ""
For $i = 1 To 32
   If $tag_Actions = "" Then
      $tag_Actions = $tagAccessibleActionInfo
   Else
      $tag_Actions = $tag_Actions & ";" & $tagAccessibleActionInfo
   EndIf
Next
Global Const $tagAccessibleActionsToDo = _
"INT actionsCount;" & _
$tag_Actions                            ; $tAccessibleActions actions[32]

This is the usage of action in Java access bridge.

$tagAccessibleActionInfo is a Java element's action's name, 256 byte max.

$tagAccessibleActions is a structure of Java element's can-do actions, 256 members max, first member is total actions count.

$tagAccessibleActionsToDo is a structure of  Java element's to-do actions, 32 queue actions max, first member is total actionsToDo count.

You find actions from can-do list, and put them into to-do list, and execute native interface function.  $failure is used to put the execution result.

__doAccessibleActions($vmId, $ac, $actionsToDo, $failure)
Edited by fablecao
improve
Link to post
Share on other sites

Regarding JABSimpleSpy.au3 and gettiing AutoIT to interact with a JAVA application...

*Exactly how* is "JABSWITCH /ENABLE" set?

I am a java neophite and also not a java fan at large...

Thank you,
Chuck

EDIT: Answering my own question... found JABSWITCH.EXE and ran it with /ENABLE ... response was "bridge enabled"

Edited by ChuckV
answered my own question... and posted for others benefit
Link to post
Share on other sites
  • 2 weeks later...
On 7/4/2019 at 11:19 AM, fablecao said:

global $wintitle = "Java Control Panel"

$winHandle = WinActivate($wintitle)

$result = __isJavaWindow($winHandle)

Global $vmId
Global $ac

global $sName = "Edit Site List..."
global $sRole = "push button"

__getAccessibleContextFromHWND($winHandle, $vmID, $ac)

$re_ac = _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)

_JAB_singleAction($vmId, $re_ac)

Perfect example!
Now if you can demonstrate a more complete example, I'm sure we can all learn from it and @junkew can pin it aswell!

Here's what i think will cover usages:
1. click on "Security" tab

image.png.1764d915828025b6f84c4934614bf28c.png

2. Click on "Edit Site List"

image.png.64936f3091dada9208d97d8cac3d837a.png

3. Click on "Add" button

image.png.0eefb34737977146a5ad295e3f8cb59a.png

4. setValue "https://google.com"

image.png.2ee64c08c622fb2d63d4bc1a83313e1c.png

5. Click "OK" button to close window

6. Click "OK" button to close main Java window aswell

Edited by sn1kzZ
Link to post
Share on other sites

You can do it directly by using this:

 

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 for other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2022-03-07

Link to post
Share on other sites
32 minutes ago, mLipok said:

You can do it directly by using this:

 

Are you replying to me? if yes then this isn't what were discussing here, javacpl.exe is just an example, but the intention of this thread is to INTERACT with Java applets.
Not Manage Java settings like your .au3 does.

Link to post
Share on other sites

I thought you want to do only this specific action.
but in relation to automating JAVA - my experience says me that not all JAVA APP can be automated by UIA in so perfect way as in case standard WIN32 APP.
I was trying to use UIA but in some JAVA solution I need to calculate COORDINATES and use MOUSE CLICK :(

 

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 for other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2022-03-07

Link to post
Share on other sites

JAB and UIA are different technologies with different scope of object recognition but follow similar principles

  • JAB
    Focused on JAVA technology: JAVA SWING, JAVA applets
  • UIA
    Focused on Windows technology: Win32, WinForms, .... and (partly) HTML
    UIA can deal with some Java technology if that application has build in IAccessible interface
  • AutoIt / Win32 sendmessage under the surface
    • AutoIt uses similar principles as UIA but UIA is more embedded in the Windows system and as such recognizes a little more
  • Webdriver for HTML and more inner HTML visibility (if its just black box clicking, editing in browser UIA can deal with most important browser GUI)
  • Hooking injecting dll's into application of interest
    • Mostly used in commercial testautomation tools to recognize harder to get controls (like Delphi apps, )
      Interested people could look into Deviare Hooking Engine.
       

The patterns they share

  • Objects are put in a tree hierarchy
    • JAB starting from Java Window which is a child from Windows desktop
      • So this means you first need AutoIt Win* function or UIA function to find your Java Window in general then AutoIt function is easiest.
    • UIA starting at windows desktop
  • Logic your program shouild follow
    • Find mainwindow
      • Scan childs with their properties whereas objecttype and title are most important but all properties can be of interest for identifying
      • Scan childs of childs (as its a hierarchy)
    • Use a spy tool to get initial insight in hierarchy and type of object
      https://github.com/google/access-bridge-explorer/releases/tag/v1.1.0 is a decent one (JABSimpleSpy is not showing hierarchy of tree)

So you should be applying above logic. Example will follow later 😉

 

 

 

Link to post
Share on other sites

Demo fully working

  • some workarounds
    • click is done based on a x,y, h, w calculation
    • settext JAB functions I did not get to work so bypassed that with a normal send  keys function
    • some slowdown is done just with sleep function
    • seems some debugging is needed when we want to use specific JAB functions clicking on a button

 

  1.  Issue 1 is tackled clicking on a security tab is done with __requestFocus (in code I still do a mousemove and mouseclick as it seems setfocus is not updating the gui in itself)
    __requestFocus($__g_vmId, $SecurityTab)

     

  2. Popupwindow is a new windows window so based on HWND the ac should be retrieved.
    This works not in below code but when commenting out all lines with 'works ok '
     and just check the window and convert to ac it works so somehow clearing of variables should happen or speed should be slower
    No clue if tree can be refreshed (without (re)loading the whole JAB)

  3.  
;~ see first post

 

Edited by junkew
9-Sep-2019: fixed all steps
Link to post
Share on other sites

@junkew

I got it working. I just debug it. Checked my dll reference. 

Now i m tried on my jar file to click on button. I am getting following response in console.

shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(3000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
global $sName = "Send"
global $sRole = "push button"
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
$re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
_JAB_singleAction($vmId, $re_ac)
Java Home: C:\Program Files\Java\jre1.8.0_221
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
Name:
rect:0000
Name:
rect:0000
_JAB_singleAction: Only applicable for single action element like button or check box, menu, menu item

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...