Scripting to a Java based GUI

[MadMike]

Heya, this is my first post,

Am very new to Auto-It and i'm very impressed with its abilities. I've searched the forums high and low for help with this problem, but don't seem to be having any luck. Hope it isn't already posted, sorry if it is.

Anyway, here's what I want to do. I'm running a program called Web NMS 4, a network configuring and monitoring tool. Its a Java application. I can access the menus and click on certain points in the window using the mouse co-ordinates or by using Send("!f") {ENTER} and all that stuff, however, I want to do it a lot smarter and make it better at knowing what its clicking as opposed to just clicking on a co-ordinate.

I want to know is it possible to be able to read text within this java application and then compare it to see if its the button I want to click.

Also, I need to be able to click an expanding tree format in one part of the screen, then into another window (which is contained within the current window, cannot be moved outside it).

Thanks for any help you can give.

[ssubirias3]

I want to know is it possible to be able to read text within this java application and then compare it to see if its the button I want to click.

Also, I need to be able to click an expanding tree format in one part of the screen, then into another window (which is contained within the current window, cannot be moved outside it).

SciTE4AutoIt3 has some really good tools to help a noob scripting in AutoIt. If you're not using this the get your copy today! If you are using SciTE4AutoIt3, then you'll want to use Tools > AU3Info to get controlIDs and other stuff you can use with your script. Then look up the following commands in Help to accomplish your "smarter" automation. ControlClick(), ControlCommand(), ControlSend(), and ControlFocus(). Be sure to read the remarks to since they are way too often overlooked. Welcome to the forum hope this helps.

[MadMike]

Yeah, I have the SciTE4AutoIt3 program and I've written a few well working scripts as a basic test of its capabilites.

but to use the ControlClick(), ControlCommand(), ControlSend(), and ControlFocus() commands, I need Control ID's and ClassnameNN's and that kinda thing, which I cannot obtain through the AutoIt Window Info utility. All it gives me is the main window information and nothing of its contents like it would for most windows.

I've attached a pic of the program. I want to be able to click on the Icon, tis a java program. I cant get control ID's for any of the icons, I cant seem to get the option menus to open with winmenuselect() either.

What I've been doing so far is using Send("!v") commands to the program and selecting things this way and by using MouseClick( "left" ,515, 220,2) to click on the icons, however the menu changes upon selecting icons and i want to know is there another way to access these??

Edited September 10, 2007 by MadMike

[MadMike]

There's actually a few topics based on this throughout the forums, but none have been seen through and it seems no one has really gotten anywhere with them either.

How to automate Java windows/controls?

[Shevilie]

Yea Java windows are pain in da ass... You have to rely on tabs and stuff.

[ssubirias3]

... I need Control ID's and ClassnameNN's and that kinda thing, which I cannot obtain through the AutoIt Window Info utility. All it gives me is the main window information and nothing of its contents like it would for most windows.

What I've been doing so far is using Send("!v") commands to the program and selecting things this way and by using MouseClick( "left" ,515, 220,2) to click on the icons, however the menu changes upon selecting icons and i want to know is there another way to access these??

If you have OpenOffice which is is a java suit of applications, you'll see that AU3Info doesn't have any information under the Control tab. But it does tell you the Class for the window is "SALFRAME". The following code successfully uses ControlFocus() and ControlClick() to drop down the Insert menu opton within Writer.

Opt("WinTitleMatchMode", 4)
ControlFocus("[Class:SALFRAME]","","")
ControlClick("","","","left", 1, 112,11)

You assignment if you choose to accept it is

• Set AU3Info coord settings to Client. Options > Coords Mode > Client (check marked).
• Verify your Position coords under the Mouse tab match your 515, 220 coords.
• Get the java window class.
• Modify the above example code with your Class: name and mouse x,y coords.
• Test script.
• Pass "Go" and collect your $200 and have a nice day .

[MadMike]

Thanks for that ssubirias3

That way does work, I know it can all be done that way, but I want to make it smarter than that

I have found another solution though, which seems like it might work....

By using Java Ferret (Java's version of Auto-It's Window Info program) I can get the handles and classnames to call

by installing the Java Access Bridge and its incorporated dll, windowsaccessbridge.dll, I should be able to use a DllCall() to the Dll and then select menus through this. This would be a much smarter approach.

The following code is something I'm trying to get working. By using an example Java Swing program, I tried to find the controls in it using the Java Ferret.

Now I know my call to the command prompt ain't the cleanest, but I don't care about it for the moment, I really wasn't bothereb being perfect about it, I can clean that later.

Run("cmd.exe")
WinWaitActive("C:\WINDOWS\system32\cmd.exe")
Send( "cd \Progra~1\Java\jdk1.6.0_02\bin{ENTER}")
Send( "java JMenuDemo{ENTER}")

; Wait for the JMenu Usage Demo to become active - it is titled "JMenu Usage Demo" on English systems
WinWaitActive("JMenu Usage Demo")

WinActivate ( "JMenu Usage Demo" )
AutoItSetOption("SendKeyDelay", 400)

$dll = DllOpen("windowsaccessbridge.dll")
$hwnd = ControlClick("[Class:SunAWTFrame]", "&Menu 1", "", "left", 1)
$menu = ControlClick("[Class:SunAWTFrame]", "Menu Item with PlainText", "","left", 1)

ControlFocus("[Class:SunAWTFrame]","JMenu Usage Demo")
DllCall($dll, "void", "Windows_run")
DllCall($dll, "void", "AccessBridge_FocusGainedFP", "hwnd", $hwnd)
DllCall($dll, "void", "setMouseClickedFP(MouseClickedDelegate fp)")

What the program should do, is open up this Java Swing application, set the focus to the menu_bar, select first menu option, this will result in the program placing text in the window area.

I figure the problem is in the DllCall()'s

Here's the dump from Java Ferret based on the above actions of my program

expandcollapse popup

State changed event: old = focused; new = (null)

Version Information:
    Java virtual machine version: 1.6.0_02
    Access Bridge Java class version: 1.6.0_02
    Access Bridge Java DLL version: AccessBridge 2.0
    Access Bridge Windows DLL version: AccessBridge 2.0

AccessibleContext information:
    Name:  
    Description:  
    Role:  text
    Role in en_US locale:  text
    States:  enabled,focusable,visible,showing,opaque,multiple line
    States in en_US locale:  enabled,focusable,visible,showing,opaque,multiple line
    Index in parent:  0
    Children count:  0
    Bounding rectangle:  [6, 52, 493, 293]
    Top-level window name:  JMenu Usage Demo
    Top-level window role:  frame
    Parent name:  
    Parent role:  viewport
    Visible descendents count:  0

AccessibleActions info:
    Number of actions:  55
    Action 0 name: insert-content
    Action 1 name: delete-previous
    Action 2 name: delete-next
    Action 3 name: set-read-only
    Action 4 name: delete-previous-word
    Action 5 name: delete-next-word
    Action 6 name: set-writable
    Action 7 name: cut-to-clipboard
    Action 8 name: copy-to-clipboard
    Action 9 name: paste-from-clipboard
    Action 10 name: page-up
    Action 11 name: page-down
    Action 12 name: selection-page-up
    Action 13 name: selection-page-down
    Action 14 name: selection-page-left
    Action 15 name: selection-page-right
    Action 16 name: insert-break
    Action 17 name: beep
    Action 18 name: caret-forward
    Action 19 name: caret-backward
    Action 20 name: selection-forward
    Action 21 name: selection-backward
    Action 22 name: caret-up
    Action 23 name: caret-down
    Action 24 name: selection-up
    Action 25 name: selection-down
    Action 26 name: caret-begin-word
    Action 27 name: caret-end-word
    Action 28 name: selection-begin-word
    Action 29 name: selection-end-word
    Action 30 name: caret-previous-word
    Action 31 name: caret-next-word
    Action 32 name: selection-previous-word
    Action 33 name: selection-next-word
    Action 34 name: caret-begin-line
    Action 35 name: caret-end-line
    Action 36 name: selection-begin-line
    Action 37 name: selection-end-line
    Action 38 name: caret-begin-paragraph
    Action 39 name: caret-end-paragraph
    Action 40 name: selection-begin-paragraph
    Action 41 name: selection-end-paragraph
    Action 42 name: caret-begin
    Action 43 name: caret-end
    Action 44 name: selection-begin
    Action 45 name: selection-end
    Action 46 name: default-typed
    Action 47 name: insert-tab
    Action 48 name: select-word
    Action 49 name: select-line
    Action 50 name: select-paragraph
    Action 51 name: select-all
    Action 52 name: unselect
    Action 53 name: toggle-componentOrientation
    Action 54 name: dump-model

I've also attached the Java Swing Application I'm testing with, you'll need to rename it to a .java file, compile it and run it yourselves

Java Access Bridge and Java Ferret are freely available on Sun Microsystem's Java Site

Am I making this too awkward?? if so, I'll just go off on my own and do it

Just thought there seemed to be quite a few people on these forums looking to do this, developers and noobs alike

JMenuDemo.txt

[ssubirias3]

Thanks for that ssubirias3

That way does work, I know it can all be done that way, but I want to make it smarter than that

Am I making this too awkward?? if so, I'll just go off on my own and do it

Just thought there seemed to be quite a few people on these forums looking to do this, developers and noobs alike

Hi Mike - Glad the example code worked for you like a champ! Are you making this too awkard? Only you can answer that. What are you willing to do to get what you want? It kinda like your signature says .

Because AU3Info cannot see the controls you want to work with, you'll be limited using ControlClick. Know that as long as you provide the Client x,y coords this will work for you even if the window is in a different position or size. I have come across a few threads where other folks have mentioned the Java Access Bridge, perhaps a forum search could shine some light on this topic. Thanks for the information on Java Ferret. After briefly skimming the page for Java Ferret, it seems that you need the SDK installed. So I haven't done this yet, but I think your right there might be others interested in using AutoIt's DllCall() to better control their Java GUIs.

Edit: Regarding your code I'd suggest looking at the DllStruct...() functions. You may need to set up the structure correctly before using DllCall(). Also I don't think you need DllOpen() and your "void" should be an "int", but I could be wrong. There are others like SmOke_N and Martain to name a few who understand DllCall() way better than I do at this time. This edit is of course talking about this section of code you posted earlier.

$dll = DllOpen("windowsaccessbridge.dll")
$hwnd = ControlClick("[Class:SunAWTFrame]", "&Menu 1", "", "left", 1)
$menu = ControlClick("[Class:SunAWTFrame]", "Menu Item with PlainText", "","left", 1)

ControlFocus("[Class:SunAWTFrame]","JMenu Usage Demo")
DllCall($dll, "void", "Windows_run")
DllCall($dll, "void", "AccessBridge_FocusGainedFP", "hwnd", $hwnd)
DllCall($dll, "void", "setMouseClickedFP(MouseClickedDelegate fp)")

Edited September 11, 2007 by ssubirias3

[MadMike]

yeah I think I'll need to look at all of the Dll control set

but I think more than that, I'm missing something in the fundamental call to the Dll, need to test to make sure its returning true

I've added in an If loop here to only send up the message boxes if the array '$result' works, so far it doesn't

I've also found out that the return values are Boolean and according to to a topic in the Developer Forums on Dll Calls, its classed as an 'int' type

also the call to the function is not a stdcall like most, since it uses cdecl calls instead (read that somewhere, will link it tomorrow)

$dll = DllOpen("windowsaccessbridge.dll")
$hwnd = WinGetHandle("[Class:SunAWTFrame]")

ControlFocus("[Class:SunAWTFrame]","JMenu Usage Demo", "")

DllCall($dll, "int:cdecl", "initializeAccessBridge()")
DllCall($dll, "int:cdecl", "IsJavaWindow([Class:SunAWTFrame]")
$result = DllCall($dll, "int:cdecl", "GetAccessibleContextFromHWND($hWnd, $vmID, $ac)", "int", $hwnd, "int", 32, "int", 32)
If Not @error Then
    msgbox(0, "HWND Target", $result[0])  ; Text returned in param 0
    msgbox(0, "Long *vmID", $result[1])  ; Text returned in param 1
    msgbox(0, "AccessibleContext *ac", $result[2])  ; Text returned in param 2
EndIf

how can I check to make sure the .dll was called right and initialized correctly??

[MarcoIppo]

Hi,

I've tried to use 'WindowsAccessBridge.dll' but my code doen't work !!

First of all I've Opened dll file and it was opened (I sow it in loaded dll in Windows system)

$dll = DllOpen("windowsaccessbridge.dll")
        MsgBox(0, "Debug Window", "Dll Handle: " & $dll )

than I try to initialize access bridge, but I receive error n° 3 (@error = 3 "function" not found in the DLL file.)

DllCall($dll, "int:cdecl", "initializeAccessBridge()")
            $error=@error
            MsgBox(0, "Debug Window", "DllCall error: " & $error )

What's the problem?

Thank you

[kingkaj]

I'm also trying to use WindowsAccessBridgel.dll.

Could be wrong here, but perhaps there's no need to call initializeAccessBridge(). Anyway, no matter what I do it always return an error.

I've been calling the function isJavaWindow like here:

$result=DllCall("WindowsAccessBridge.dll", "int:cdecl", "isJavaWindow", "HWnd", "0x0048010E")
ConsoleWrite (@error &';'&$result[0]&';'&$result[1])

and it seems to be successful: @error gives me a 0, and I get the hwnd value back in the $result. I was expecting true or false though...

Am I doing the right thing here?

[PsaltyDS]

Hi,

I've tried to use 'WindowsAccessBridge.dll' but my code doen't work !!

First of all I've Opened dll file and it was opened (I sow it in loaded dll in Windows system)

$dll = DllOpen("windowsaccessbridge.dll")
        MsgBox(0, "Debug Window", "Dll Handle: " & $dll )

than I try to initialize access bridge, but I receive error n° 3 (@error = 3 "function" not found in the DLL file.)

DllCall($dll, "int:cdecl", "initializeAccessBridge()")
            $error=@error
            MsgBox(0, "Debug Window", "DllCall error: " & $error )

What's the problem?

Thank you

I don't think the parens "()" belong on the function name:

DllCall($dll, "int:cdecl", "initializeAccessBridge")

If there were parameters to the function, they would not be passed inside the parens in AutoIt's format, they would be comma delimited (i.e. "function", type, param") as in the help file entry for DllCall().

P.S. found a ref to the functions: JavaTM Access Bridge for Microsoft Windows, v1.0.2 FCS API Specification

This leads to an example:

$hWindow = WinGetHandle("Your Window Title")
DllCall($dll, "int:cdecl", "IsJavaWindow", "hwnd", $hWindow)

@kingkaj: I don't think you can pass the window handle by literal hex string that way.

Edited December 10, 2008 by PsaltyDS

[kingkaj]

Thanks for your reply PsaltyDS.

In fact my 2 line script returns exactly the same as the example you posted. So I can get as far as DllCall not returning an error.

Let me explain some more...

What I'm trying to do is read the text off a Java screen. I'm using Java Access Bridge 2.0.1 with JDK 1.5.0_12. JDK comes with a demo app swingset. I'm trying to read the strings from the warning dialog example:

First I need the AccessibleContext and vmID. Access Bridge API doc say to use GetAccessibleContextFromHWND.

My script then becomes:

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("[Class:SunAwtDialog]")

WinWaitActive ("[CLASS:SunAwtDialog]", "")

$result=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
ConsoleWrite (@error &';'&$result[0]&';'&$result[1])

Note the spelling of the function, it fails when you put 'GetAccessibleContextFromHWND'.

My script writes following to the console 0;0;0x000F016E, which tells me the call to the dll went fine (@error = 0), but the rest just returns what I submitted as parameters. I expected to get the AccessibleContext and vmID.

[PsaltyDS]

Thanks for your reply PsaltyDS.

In fact my 2 line script returns exactly the same as the example you posted. So I can get as far as DllCall not returning an error.

Let me explain some more...

What I'm trying to do is read the text off a Java screen. I'm using Java Access Bridge 2.0.1 with JDK 1.5.0_12. JDK comes with a demo app swingset. I'm trying to read the strings from the warning dialog example:

First I need the AccessibleContext and vmID. Access Bridge API doc say to use GetAccessibleContextFromHWND.

My script then becomes:

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("[Class:SunAwtDialog]")

WinWaitActive ("[CLASS:SunAwtDialog]", "")

$result=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
ConsoleWrite (@error &';'&$result[0]&';'&$result[1])

Note the spelling of the function, it fails when you put 'GetAccessibleContextFromHWND'.

My script writes following to the console 0;0;0x000F016E, which tells me the call to the dll went fine (@error = 0), but the rest just returns what I submitted as parameters. I expected to get the AccessibleContext and vmID.

Try this to be sure you are seeing all results:

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("[Class:SunAwtDialog]")
ConsoleWrite("Window handle $hWindow = " & $hWindow & @LF)
WinWaitActive ("[CLASS:SunAwtDialog]", "")
$avResult=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
$iErrSav = @error
$sResult = "Results:  @error = " & $iErrSav
For $n = 0 To UBound($avResult) - 1
    $sResult &= @CRLF & "[" & $n & "] = " & $avResult[$n]
Next
ConsoleWrite($sResult & @LF)

If that doesn't help, try changing "HWND" to "HWND*" in the DllCall, to pass it byRef. Then you would see the handle value changed in the returned parameters.

[ofLight]

As far as I am understanding the Windowsaccessbridge.dll it will only work on .jar files that are basicly loaded in a VM with the JDK, not compiled java apps that are run with the JRE??

Even if that was the case, the following code should work on say, \Java\jdk1.6.0_11\demo\jfc\Notepad\Notepad.jar . However all I can get is a 0 from the following code.

Sleep(3000) ;Gives me time to click on a window

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("")

$result=DllCall($dll, "int:cdecl", "isJavaWindow", "HWnd", $hWindow)
ConsoleWrite("IsJavaWindow?: "&$result[0]&@LF&'WindowHandle: '&$result[1]&@LF)

$avResult=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
$iErrSav = @error
$sResult = "Results:  @error = " & $iErrSav
For $n = 0 To UBound($avResult) - 1
    $sResult &= @CRLF & "[" & $n & "] = " & $avResult[$n]
Next
ConsoleWrite($sResult & @LF)

Output:

IsJavaWindow?: 0
WindowHandle: 0x0001076C
Results:  @error = 0
[0] = 0
[1] = 0x0001076C

Edited December 10, 2008 by ofLight

[kingkaj]

Try this to be sure you are seeing all results:

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("[Class:SunAwtDialog]")
ConsoleWrite("Window handle $hWindow = " & $hWindow & @LF)
WinWaitActive ("[CLASS:SunAwtDialog]", "")
$avResult=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
$iErrSav = @error
$sResult = "Results:  @error = " & $iErrSav
For $n = 0 To UBound($avResult) - 1
    $sResult &= @CRLF & "[" & $n & "] = " & $avResult[$n]
Next
ConsoleWrite($sResult & @LF)

If that doesn't help, try changing "HWND" to "HWND*" in the DllCall, to pass it byRef. Then you would see the handle value changed in the returned parameters.

Your code gives me the same results as my original code. Tried with hwnd* as well, no change in output.

[kingkaj]

As far as I am understanding the Windowsaccessbridge.dll it will only work on .jar files that are basicly loaded in a VM with the JDK, not compiled java apps that are run with the JRE??

I tried the Java Ferret and Monkey apps that are included with Access Bridge on jar as well as compiled Java apps and it works on both.

[seangriffin]

As far as I am understanding the Windowsaccessbridge.dll it will only work on .jar files that are basicly loaded in a VM with the JDK, not compiled java apps that are run with the JRE??

Even if that was the case, the following code should work on say, \Java\jdk1.6.0_11\demo\jfc\Notepad\Notepad.jar . However all I can get is a 0 from the following code.

Sleep(3000) ;Gives me time to click on a window

$dll = DllOpen ("WindowsAccessBridge.dll")
$hWindow = WinGetHandle("")

$result=DllCall($dll, "int:cdecl", "isJavaWindow", "HWnd", $hWindow)
ConsoleWrite("IsJavaWindow?: "&$result[0]&@LF&'WindowHandle: '&$result[1]&@LF)

$avResult=DllCall($dll, "int:cdecl", "getAccessibleContextFromHWND", "HWND", $hWindow)
$iErrSav = @error
$sResult = "Results:  @error = " & $iErrSav
For $n = 0 To UBound($avResult) - 1
    $sResult &= @CRLF & "[" & $n & "] = " & $avResult[$n]
Next
ConsoleWrite($sResult & @LF)

Output:

IsJavaWindow?: 0
WindowHandle: 0x0001076C
Results:  @error = 0
[0] = 0
[1] = 0x0001076C

I've solved this puzzle of automating Java applications!

I wrote my own DLL which bridges the gap (so to speak) between AutoIT and the Java Access Bridge. My solution is working for browser-based Java applets, as well as standalone Java GUIs.

Here is the link to my solution:

Java UDF, Automating Java Applications using the Java Access Bridge

Thanks to MadMike for suggesting the use of the Java Access Bridge. Without this suggestion I would have never come up with my solution. Thank-you!

Great to finally see Java software automated.

Edited January 27, 2009 by seangriffin