Jump to content

JAVA object automation and simple spy


Recommended Posts

  • Replies 90
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

As a starter and far from perfect and real messy code a simple spy for Java Accessibility Bridge Based on the information in general help and support '?do=embed' frameborder='0' data-embedC

;~ ******* ;~ Every AccessibleContext IntPtr must be replaced by long, including but not limited to getAccessibleContextFromHWND, getAccessibleParentFromContext, getAccessibleChildFromConte

revalidated and still works with 1.8.0.66 and autoit 3.3.15.0 Mouse position is retrieved 1519-331 JAVA window found <> Java Window Title= Handle=0x0000000000040A88 res: 1

Posted Images

https://github.com/openjdk/jdk/tree/master/src/jdk.accessibility/windows/native/jaccessinspector gives an indication that at least __releaseJavaObject should be called to release memory allocated.

More CPP source on how JAB is made https://github.com/openjdk/jdk/blob/master/src/jdk.accessibility/windows/native/libwindowsaccessbridge/WinAccessBridge.cpp

and apparently I tried also in 2013 some stuff where most important part can be that certain messages have to be handled when sent from JAB to AutoIt client.

  • jaccesswalker from jdk10 and jdk11 seems to be most stable tool to spy not crashing
  • whereas jaccessinspector just quits when i try to spy on jdk8 javacpl.exe.
  • accessibility explorer is also crashing as soon as I spy on the modal dialog.

As jaccesswalker is working fine I only can say this can work but maybe issues due to

* Logic coding error

* separate thread needed

* ....

 

Edited by junkew
Link to post
Share on other sites

Demo script updated and fully working (see post 1)

  • _JAB_SingleAction seems to hang on sitelistbutton, bypassed by calculating mouse location based on acinfo x,y,h,w
;~  TODO: _JAB_singleAction($__g_vmId, $SiteListButton)
    clickac($__g_vmId, $SiteListButton)
  • button add in popup works
  • Done: Edit part
Edited by junkew
Link to post
Share on other sites
16 hours ago, KhalidAnsari said:

@junkew

I got it working. I just debug it. Checked my dll reference. 

Now i m tried on my jar file to click on button. I am getting following response in console.

shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(3000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
global $sName = "Send"
global $sRole = "push button"
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
$re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
_JAB_singleAction($vmId, $re_ac)
Java Home: C:\Program Files\Java\jre1.8.0_221
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
Name:
rect:0000
Name:
rect:0000
_JAB_singleAction: Only applicable for single action element like button or check box, menu, menu item

 

_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)

This function is not my original, you find wrong $re_ac which cause _JAB_singleAction error.

You can use  _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)

Link to post
Share on other sites

Ok will check with the original version.

added function _JAB_getAccessibleChildNByRole

usage: get an element by its index and class/role 

;~  local $re_ac=_JAB_getAccessibleContextByRole($__g_vmId, $__g_acJavaMainWindow, "push button", 1)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",1)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",2)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",3)

function

Func _JAB_getAccessibleChildNByRole($vmId, $ac, $sRole,$index=1, $depth=0)
   Static $matchCount=0

   if ($depth=0) then
       $matchCount=0
   EndIf

   Local $iCount = _JAB_getChildrenCount($vmId, $ac)
;~    consolewrite("Investigating" & $iCount & ";" & $index & ";" & $matchCount & ";" & _JAB_getRole($vmId, $ac) & ";")

    If $iCount = 0 Then
        Return -1
    EndIf

    local $retVal=-1

    For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
        Local $child_ac_s3 = _JAB_getRole($vmId, $child_ac)

;~      consolewrite($child_ac_s3 & @CRLF)

        If $child_ac_s3 = $sRole Then
            $matchCount+=1
;~          consolewrite("Partial match" & $matchCount & @CRLF)
            if $index=$matchCount Then
;~              consolewrite("Full match" & $matchCount & @CRLF)
                Return $child_ac
            endif
        EndIf

        $depth+=1
        local $retVal=_JAB_getAccessibleChildNByRole($vmId, $child_ac, $sRole, $index, $depth)

        if ($retVal <>-1) Then
            exitloop
        EndIf
   Next

   Return $retVal
EndFunc

 

Edited by junkew
function changed/added
Link to post
Share on other sites

@fablecao

I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function

See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
;~ consolewrite("Just before it causes a blocking ..." & @CRLF)
;~ _JAB_singleAction($__g_vmId, $SiteListButton)
;~ consolewrite("Just after it causes a blocking ..." & @CRLF)

    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Output from printDescription($siteListButton)

Name:Edit Site List...
rect:10681209562585
  name: <Edit Site List...>
  description: <Add, Remove, or Edit entries on the Exception Site List>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1068>
  y: <562>
  width: <141>
  height: <23>
  accessibleComponent: <1>
  accessibleAction: <1>
  name: <0>
  description: <0>
  role: <0>
  role_en_US: <0>
  states: <0>
  states_en_US: <0>
  indexInParent: <0>
  childrenCount: <0>
  x: <0>
  y: <0>
  width: <0>
  height: <0>
  accessibleComponent: <0>
  accessibleAction: <0>

 

Link to post
Share on other sites
3 hours ago, junkew said:

@fablecao

I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function

See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
;~ consolewrite("Just before it causes a blocking ..." & @CRLF)
;~ _JAB_singleAction($__g_vmId, $SiteListButton)
;~ consolewrite("Just after it causes a blocking ..." & @CRLF)

    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Output from printDescription($siteListButton)

Name:Edit Site List...
rect:10681209562585
  name: <Edit Site List...>
  description: <Add, Remove, or Edit entries on the Exception Site List>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1068>
  y: <562>
  width: <141>
  height: <23>
  accessibleComponent: <1>
  accessibleAction: <1>
  name: <0>
  description: <0>
  role: <0>
  role_en_US: <0>
  states: <0>
  states_en_US: <0>
  indexInParent: <0>
  childrenCount: <0>
  x: <0>
  y: <0>
  width: <0>
  height: <0>
  accessibleComponent: <0>
  accessibleAction: <0>

 

I don't know what behaves weird when you use _JAB_SingleAction on the sitelist button.  I test it successfully in my computer.

And from your output of printDescription($siteListButton), I find that the output of printdescription($siteListButton) inside highlightAC is different from the next output of printdescription($siteListButton).

Does the func clickSiteButton function normal?

Does the func __releaseJavaObject inside highlightAC release the AC?

Link to post
Share on other sites

By the way, I prefer to use the combination of JAB native function to write UDF.

The JAB native function __requestFocus may implement your function highlightAC.

;~ BOOL requestFocus(const long vmID, const AccessibleContext accessibleContext)
; Request focus for a component. Returns whether successful.
Func __requestFocus($vmId, $ac)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "requestFocus", "long", $vmId, $c_JOBJECT64, $ac)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc

Link to post
Share on other sites

I agree on preferring native functions over alternatives like mousemove and click .

the problem I observe

Function clicks on the button

Popup dialogwindow appears

Script hangs

When I use this below function so action is executed but then script is hanging somewhere 

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
 consolewrite("Just before it causes a blocking ..." & @CRLF)
_JAB_singleAction($__g_vmId, $SiteListButton)
consolewrite("Just after it causes a blocking ..." & @CRLF)

;~    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc
Edited by junkew
Link to post
Share on other sites
53 minutes ago, junkew said:

When I use this below function so action is executed but then script is hanging somewhere 

Maybe the same problem like in IE.au3 UDF , I mean when you click in "OpenFile Button".
IE object is waiting for end of selecting..... and for this reason, AutoIt is waiting for IE......

 

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 for other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2022-03-07

Link to post
Share on other sites

@junkew

Thanks for reply. 

Following code is closing my Java screen unexpectedly. I am trying to click on "Send button" but $re_ac returning -1 from method _JAB_getAccessibleChildNByRole

#include "Javaui.au3"
shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(5000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
ConsoleWrite("$vmID : " & $vmID & @CRLF)
ConsoleWrite("$ac : " & $ac & @CRLF)
Sleep(3000)
local $re_ac =_JAB_getAccessibleChildNByRole($vmId, $ac, "push button",1)
ConsoleWrite("$re_ac : " & $re_ac &  @CRLF)

OutPut

Java Home: C:\Program Files\Java\jre1.8.0_221
 We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files\Java\jre1.8.0_221\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
$vmID : 1247780
$ac : 1627349361864
Investigating1;1;0;frame; -- Not getting out from _JAB_getAccessibleChildNByRole
Investigating0;1;0;; -- Not getting out from _JAB_getAccessibleChildNByRole
$re_ac : -1

 

Link to post
Share on other sites
5 hours ago, junkew said:

I agree on preferring native functions over alternatives like mousemove and click .

the problem I observe

Function clicks on the button

Popup dialogwindow appears

Script hangs

When I use this below function so action is executed but then script is hanging somewhere 


func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
 consolewrite("Just before it causes a blocking ..." & @CRLF)
_JAB_singleAction($__g_vmId, $SiteListButton)
consolewrite("Just after it causes a blocking ..." & @CRLF)

;~    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Oh, maybe there is something wrong in my JAB native function.  "NONE" should be changed to "NONE:cdecl".  All of which should be corrected.

Func __releaseJavaObject($vmId, $object)
    $result = DllCall($hAccessBridgeDll, "NONE", "releaseJavaObject", "long", $vmId, $c_JOBJECT64, $object)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc
Link to post
Share on other sites
On 11/24/2018 at 3:08 AM, junkew said:

Just checked and still works nicely in 64 bits windows. 

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\x\Documents\UIA\JABSimpleSpy.au3" /UserParams    
+>19:58:07 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0   Keyboard:00020409  OS:WIN_81/  CPU:X64 OS:X64    Environment(Language:0413)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\x\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\x\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\x\Documents\UIA\JABSimpleSpy.au3
+>19:58:08 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Users\x\Documents\UIA\JABSimpleSpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
C:\Program Files\Java\jre1.8.0_181
We are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=1
Opening C:\Program Files\Java\jre1.8.0_181\bin\WindowsAccessBridge-64.dll  Windows accessbridge 32 opened 1
 0 initializeAccessBridge is finishedWindows_run passed :
0hello  name: <Settings...>
  description: <<html>Modify settings for temporary files</html>>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1028>
  y: <447>
  width: <83>
  height: <0>
+>19:58:30 AutoIt3.exe ended.rc:0
+>19:58:30 AutoIt3Wrapper Finished.

Initially I learned a lot from NVDA to use this stuff and found another nice access-bridge-explorer that can be used for porting things from JAB into AU3

 

How to use autoit3_x64.exe in SciTE?

Link to post
Share on other sites
  • 8 months later...
  • 2 months later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...