Jump to content

JAVA object automation and simple spy


Recommended Posts

  • 4 months later...

Thank you for putting this together! I am having trouble getting it to work. Your other simplespy GUI worked without any issues!

I have updated the JRE to point to "C:\Program Files\Java\jre1.8.0_45\bin\javacpl.exe" instead of 25. I have also enabled the JAB switch, however I did this with instruction through the Ease of Access menu.

Is there something I am missing? The GUI launches without any error, but CTRL+W doesn't do anything when trying to select an object.

I realize this is an older post, but there doesn't seem to be any other semi-current information on running AutoIT with a java application.

Thanks in advance!

Link to comment
Share on other sites

I will validate if it still is working (no reason to be broken)

try if you get it working with NVDA or JavaMonkey

you could add this to see if you get a window

consolewrite("The window number is " & $hWND)
$tGetMousePos = _WinAPI_GetMousePos()
    $hWnd = _WinAPI_WindowFromPoint($tGetMousePos)

consolewrite("The window number is " & $hWND)

    $result =dllcall($accessBridgeDll,"BOOL:cdecl", "isJavaWindow", "hwnd", $hWnd)
    if @error=0 Then


Link to comment
Share on other sites

Thanks for the update. I actually tried it with another application and it is working. At the time of posting, I was under the impression that this application was running in Java, but it wasn't.

Thanks again for all the work you've done on this forum!

Link to comment
Share on other sites

  • 7 months later...

revalidated and still works with and autoit

Mouse position is retrieved 1519-331
  JAVA window found <> Java Window Title= Handle=0x0000000000040A88  res: 1
Result getAccessibleContextFromHWND is <> ubound: 4
  We have a VMid 133784 ac 469679048
Call version info: PASSED VMID found : 133784
  VMVersion: <1.8.0_66>
  bridgeJavaClassVersion: <1.8.0_66>
  bridgeJavaDLLVersion: <1.8.0_66>
  bridgeWinDLLVersion: <1.8.0_66>
 *** getAccessibleContextAT result is <469679016> 
 *** getAccessibleContextAT result is <469679016> 
<html>View information about this version of the JRE</html>
push button
push button
End of getAcceccibleContextAt info


Link to comment
Share on other sites

This is a great script an is ecactly what i am looking for.

I am getting  an Error at the dll Call

$result =dllcall($accessBridgeDll, "BOOL:cdecl", "getVersionInfo", "long", $vmId, "struct*", DllStructGetPtr($tAccessBridgeVersionInfo))

Mouse position is retrieved 1072-415
  JAVA window found <> Java Window Title= Handle=0x00151304  res: 1
Result getAccessibleContextFromHWND is <> ubound: 4
  We have a VMid 397076 ac 29949844
getVersionInfo error:1
 *** getAccessibleContextAT result is <29949764>

Error 1 means "unable to use the DLL file"


My Environment Settings:
I have to Use Java Version 1.7.0_76-b13
jabswitch is enabled
javaferret.exe and JavaMonkey-32.exe are working properly with the same Java Window

For testing I modified $accessBridgeDLL to the path of WindowsAccessBridge-32.dll matching Java Version 1.7.0_76 . I got the same Error.

Why do I get an "unable to use DLL file" error when calling "getVersionInfo" and some code lines up the call "getAccessibleContextFromHWND" is working correct?

Thanks for any ideas?

Link to comment
Share on other sites

When I trying to is it I mean CTRL+W then i get:


--> Press Ctrl+F11 to Restart or Ctrl+Break -or- F11 to Stop
We are using X64 at cpu X64 Autoit 64 bit version 0
  Windows accessbridge 32 opened 1
 0 initializeAccessBridge is finishedWindows_run passed :
0hello!>11:17:08 AutoIt3.exe ended.rc:-1073741819
+>11:17:08 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 27.99


Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Codefor other useful stuff click the following button:


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 *


My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2023-04-24

Link to comment
Share on other sites

In case of @AutoItX64=0

how about this:

Const $c_JOBJECT64 = "UINT64" ;~ JOBJECT64


EDIT: like this:

Const $c_JOBJECT64 = (@AutoItX64) ? ("UINT64") : ("UINT") ;~ JOBJECT64
Const $cP_JOBJECT64 = (@AutoItX64) ? ("UINT64*") : ("UINT*") ;~ POINTER(JOBJECT64)


EDIT2: other proposal:

Global $sWow64 = (@AutoItX64) ? ("\Wow6432Node") : ("")
Global $sJavaVersion = RegRead("HKLM\SOFTWARE" & $sWow64 & "\JavaSoft\Java Runtime Environment", "CurrentVersion")
Global $sJavaHome = RegRead("HKLM\SOFTWARE" & $sWow64 & "\JavaSoft\Java Runtime Environment\" & $sJavaVersion, "JavaHome")


EDIT2: some further changes:


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <WinAPIEx.au3>
#include <debug.au3>
#include <memory.au3>
#include <Misc.au3>

HotKeySet("{ESC}", "Close") ; Set ESC as a hotkey to exit the script.
HotKeySet("^w", "GetElementInfo") ; Set Hotkey Ctrl+W to get some basic information in the GUI

Global $hAccessBridgeDll ;~ reference to the accessbridge dll

Const $tagAccessBridgeVersionInfo = "WCHAR VMversion[256];WCHAR bridgeJavaClassVersion[256];WCHAR bridgeJavaDLLVersion[256];WCHAR bridgeWinDLLVersion[256]"

Const $c_JOBJECT64 = (@AutoItX64) ? ("UINT64") : ("UINT") ;~ JOBJECT64
Const $cP_JOBJECT64 = (@AutoItX64) ? ("UINT64*") : ("UINT*") ;~ POINTER(JOBJECT64)

Local $tAccessBridgeVersionInfo = DllStructCreate($tagAccessBridgeVersionInfo)

Const $tagAccessibleContextInfo = "WCHAR name[1024];WCHAR description[1024];WCHAR role[256];WCHAR role_en_US[256];WCHAR states[256];WCHAR states_en_US[256];INT indexInParent;INT childrenCount;INT x;INT y;INT width;INT height;BOOL accessibleComponent;BOOL accessibleAction;BOOL accessibleSelection;BOOL accessibleText;BOOL accessibleInterfaces"
Local $tAccessibleContextInfo = DllStructCreate($tagAccessibleContextInfo)
;~ struct AccessibleTextInfo {
;~  jint charCount;       // # of characters in this text object
;~  jint caretIndex;      // index of caret
;~  jint indexAtPoint;    // index at the passsed in point
;~ };
Const $tagAccessibleTextInfo = "INT charCount;INT caretIndex;INT indexAtPoint"

Const $AutoSpy = 0 ;2000 ; SPY about every 2000 milliseconds automatically, 0 is turn of use only ctrl+w

;~ global $oldUIElement ; To keep track of latest referenced element
Global $frmSimpleSpy, $edtCtrlInfo, $lblCapture, $lblEscape, $lblRecord, $edtCtrlRecord, $msg
Global $i ; Just a simple counter to measure time expired in main loop

;~ Determine java location

Global $sWow64 = (@AutoItX64) ? ("\Wow6432Node") : ("")
Global $sJavaVersion = RegRead("HKLM\SOFTWARE" & $sWow64 & "\JavaSoft\Java Runtime Environment", "CurrentVersion")
Global $sJavaHome = RegRead("HKLM\SOFTWARE" & $sWow64 & "\JavaSoft\Java Runtime Environment\" & $sJavaVersion, "JavaHome")
ConsoleWrite($sJavaHome & @CRLF)

ShellExecute($sJavaHome & "\bin\javacpl.exe")

;~ _Example1_IterateJavaWindows()

#Region ### START Koda GUI section ### Form=
$frmSimpleSpy = GUICreate("Simple UIA Spy", 801, 601, 181, 4)
$edtCtrlInfo = GUICtrlCreateEdit("", 18, 18, 512, 580)
GUICtrlSetData(-1, "")
$lblCapture = GUICtrlCreateLabel("Ctrl+W to capture information", 544, 10, 528, 17)
$lblEscape = GUICtrlCreateLabel("Escape to exit", 544, 53, 528, 17)
$edtCtrlRecord = GUICtrlCreateEdit("", 544, 72, 233, 520)
GUICtrlSetData(-1, "//TO DO edtCtrlRecord")
;~ $lblRecord = GUICtrlCreateLabel("Ctrl + R to record code", 544, 32, 527, 17)
#EndRegion ### END Koda GUI section ###

;~ _UIA_Init()

;~ loadCodeTemplates() ; //TODO: To use templates per class/controltype

; Run the GUI until the dialog is closed
While True
    $msg = GUIGetMsg()
;~ if _ispressed(01) Then
;~ getelementinfo()
;~ endif

    ;Just to show anyway the information about every n ms so ctrl+w is not interfering / removing window as unwanted side effects
    $i = $i + 100
    If ($AutoSpy <> 0) And ($i >= $AutoSpy) Then
        $i = 0

    If $msg = $GUI_EVENT_CLOSE Then ExitLoop

Func GetElementInfo()
    Local $acParent = 0
    Local $vmID = 0
    Local $tStruct = DllStructCreate($tagPOINT)
    Local $description

    $description = ""

    $xMouse = MouseGetPos(0)
    $yMouse = MouseGetPos(1)
    DllStructSetData($tStruct, "x", $xMouse)
    DllStructSetData($tStruct, "y", $yMouse)

    $tGetMousePos = _WinAPI_GetMousePos()
    $hwnd = _WinAPI_WindowFromPoint($tGetMousePos)

    $result = DllCall($hAccessBridgeDll, "BOOL:cdecl", "isJavaWindow", "hwnd", $hwnd)
    If @error = 0 Then
        If $result[0] = 1 Then
            $description = "Mouse position is retrieved " & $xMouse & "-" & $yMouse & @CRLF
;~          GUICtrlSetData($edtCtrlInfo, "Mouse position is retrieved " & $xMouse & "-" & $yMouse & @CRLF)

            $description = $description & ("  JAVA window found <" & "> Java Window Title=" & " Handle=" & $hwnd & @TAB & " res: " & $result[0] & @CRLF)

            $result = DllCall($hAccessBridgeDll, "BOOL:cdecl", "getAccessibleContextFromHWND", "hwnd", $hwnd, "long*", $vmID, $cP_JOBJECT64, $acParent)
            If @error = 0 Then
                $description = $description & ("Result getAccessibleContextFromHWND is <" & $result & "> ubound: " & UBound($result) & @CRLF)
                $vmID = $result[2]
                $acParent = $result[3]
                $description = $description & ("  We have a VMid " & $vmID & " ac " & $acParent & @CRLF)

                $result = DllCall($hAccessBridgeDll, "BOOL:cdecl", "getVersionInfo", "long", $vmID, "struct*", DllStructGetPtr($tAccessBridgeVersionInfo))

                If @error = 0 Then
                    $s1 = DllStructGetData($tAccessBridgeVersionInfo, "VMVersion")
                    $s2 = DllStructGetData($tAccessBridgeVersionInfo, "bridgeJavaClassVersion")
                    $s3 = DllStructGetData($tAccessBridgeVersionInfo, "bridgeJavaDLLVersion")
                    $s4 =