Jump to content


How to automate Java windows/controls?

  • Please log in to reply
25 replies to this topic

#1 Jon



  • Administrators
  • 10,648 posts

Posted 15 March 2007 - 07:25 AM

So I had a look at a couple of Java programs. Notably the Java Control Panel that most people will have. And it's like a total black box. Using Window Spy++ I could see the main window but even though the window was full of buttons and checkboxes and stuff - there were no child windows!

I thought automating Java controls would be a case of finding out which messages it accepts to do what - but if it doesn't even present a child window to send to I'm a bit stumped. Spy++ couldn't even see any unknown/registered messages when you click on things.

Is there a Java SDK type thing for programming these sorts of apps - that might shown how they are control internally and help out. I mean, there must be an internal command for selecting a checkbox.

Ideas welcome.

#2 Manadar



  • MVPs
  • 10,935 posts

Posted 16 March 2007 - 01:06 PM

Here's what the professionals say about Java Applets.

AutoMate is able to find the Java Applet window but recognizes nothing inside it. The Java controls are basically "invisible" to AutoMate. From Windows' point of view the Java Applet looks like a window containing one large bitmap. Because of this, most of AutoMate's interactivity actions (i.e. 'Move Mouse to Object,' 'Press,' or 'Set Text' actions) will not work within a Java Applet window.

The only possibility for AutoMate to interact with controls within a Java Applet Window is to set the focus to the Java Window and use the 'Send Keystrokes' action to tab through the fields to set and get data or press buttons.


Edit: What Larry said completely contradicts this information.

Edited by Manadar, 16 March 2007 - 01:07 PM.

#3 Jon



  • Administrators
  • 10,648 posts

Posted 16 March 2007 - 01:13 PM

Not quite. Junit looks to be a general java testing thing. But you need to use some addins to test gui stuff through a awk.robot java class. This class can mess around with a few buttons but appears to have no c++/external interface - so it's of no use :whistle:

#4 Manadar



  • MVPs
  • 10,935 posts

Posted 16 March 2007 - 01:28 PM

I've seen a few programs that decompile the .class files, that could at least give you information about the Java applet. This probably should not be considered an option...

My guess is that if you should succeed, Jon, you would be the first ever.

#5 big_daddy


  • Moderators
  • 2,591 posts

Posted 16 March 2007 - 02:22 PM

By the sounds of it this may be worth taking a look at.

IBM Rational Functional Tester

Edited by big_daddy, 16 March 2007 - 02:23 PM.

#6 TimothyGirard



  • New Members
  • 1 posts

Posted 04 May 2007 - 07:42 PM

What if we were able to capture user defined images and then OCR any text. Most text within these UI's (Flash and Java) is usually in standard fonts.

this would be a very powerfull ability for AutoIt. The ability to define a "virtual" object that could be manipulated via standard functions would no doubt push a large group of app automators (Like me) to AutoIt a lot faster. Until I can do that easily in AutoIT, I'm stuck in QuickTest Pro.

#7 DaleHohm


    Think of IE as an API...

  • MVPs
  • 5,989 posts

Posted 04 May 2007 - 11:15 PM

I have always been on the lookout for such tools and methods, but had never found anything other than trying to exploit methods that might be exposed by the applet to Javascript.

I did recently find a couple of resources dedicated to discussions of this but I have not had a chance to dig into them to see if there is anything promising.

This Wiki has a bunch of tool pointers: TestingGUIApplications and this Yahoo! group: java-gui-testing · Java GUI Testing


Edit: also seeing the source may help you... Google the DJ Java Decompiler

Edited by DaleHohm, 04 May 2007 - 11:16 PM.

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model


Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds


SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

#8 b990595



  • Normal Members
  • 3 posts

Posted 16 May 2007 - 09:52 AM

I would be nice to read java windows, maybe this can help ...

Sun have developed a Java Access Bridge, which contains DLLs to read Java windows ..

API documentation is here ..

JavaFerret is a simple app, that reads java screens ...

-Rasmus, Denmark

#9 Xenobiologist


    Xx Code~Mega xX

  • MVPs
  • 4,866 posts

Posted 07 September 2007 - 06:49 AM


any progress on this?

We use this : IBM Rational Functional Tester

But it is written in Java.

So long,


Scripts & functions [color=#0000ff;]Organize Includes Let Scite organize the include files[/color] [color=#ff0000;]*new[/color]

[color=#000000;]Yahtzee The game "Yahtzee" (Kniffel, DiceLion)[/color]

[color=#000000;]LoginWrapper Secure scripts by adding a query (authentication)[/color]

[color=#000000;]_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)[/color]

[color=#000000;]Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.[/color]

[color=#000000;]MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times[/color]

#10 MadMike



  • Active Members
  • 7 posts

Posted 10 September 2007 - 02:51 PM

Sorry I'm not an experienced developer just yet but I'm very interested if anyone has made any progress on this as well.

I've lookep up a load of how to automate Java windows/controls and this Java Access Bridge seems the best way.

I've been trying to use the Java Access Bridge and the Java Ferret and I'm not having much luck.
Just have to use DllCall to the WindowsAccessBridge.dll with the Java Access Bridge then I can start making calls to the classes within the program.
If it's too easy, then its not worth doingMike P. O'Keeffe BEng, Hardware Design EngineerLanguages: Perl, C, Expect, Visual Basic, Assembly

#11 ValeryVal


    Dig the way

  • Active Members
  • PipPipPipPipPipPip
  • 482 posts

Posted 11 September 2007 - 05:17 AM

I added ALisp

Apart from other things you can create by newLisp's tools java-based GUI.
For doing it newLisp has special guiserver.jar (GUI server) and guiserver.lsp.

Creation of java-based gui is the following process:

1. Start GuiServer demon

java.exe -jar "c:/Program Files/newlisp/guiserver.jar" 47011

You will see the demon's window worked (black window)

2. Start ALisp.au3

3. Evaluate by ALisp the expression:

(load "c:/Program Files/newlisp/guiserver.lsp")

4. Now you can evaluate lines (step by step or all as one) of some example from newlisp/guiserver folder.
ie button-demo.lsp

(gs:set-trace true)
(gs:frame 'ButtonDemo 100 100 400 300 "Click on button or color panel")
(gs:set-resizable 'ButtonDemo nil)
(gs:panel 'ColorPanel 360 200)
(gs:set-background 'ColorPanel '(0 1 0) 0.2)
(gs:button 'aButton 'button-action "color")
(gs:set-flow-layout 'ButtonDemo "center" 2 15)
(gs:add-to 'ButtonDemo 'ColorPanel 'aButton)
(gs:set-visible 'ButtonDemo true)
(gs:mouse-event 'ColorPanel 'mouse-action)

;;;; define actions
(define (button-action id)
(gs:set-color 'ColorPanel (random) (random) (random)))

(define (mouse-action id type x y button cnt mods)
(gs:set-text 'ButtonDemo (format "%8s %3d:%3d %d %d %2d" type x y button cnt mods))


Thus, you will create really java-based GUI having two handlers - button-action and mouse-action.
If you uncomment last line then GUI will independent window which handles button and mouse events.
This is a two-side body - ALisp & java window.
Maybe it is simplest platform for experiment with interaction between AutoIt and java, I think.

Remark. You should to install newlisp from this site, I think

BTW. ALisp (with help newlisp) shows how you can build simplest java-based bridge from Windows into Linux/Mac OS X/Solaris/BSD's platform
The point of world view

#12 ValeryVal


    Dig the way

  • Active Members
  • PipPipPipPipPipPip
  • 482 posts

Posted 11 September 2007 - 05:29 AM

Feed back from java-based Gui can be implemented by AutoItX.
See, example of AutoItX using (not tested)


However there are other more fast method, I think

The point of world view

#13 Antiec



  • Active Members
  • Pip
  • 52 posts

Posted 14 September 2007 - 12:43 PM


I had to automate a Java program at my work. I found two kinda nice java-based automation programs (marathon & abbot), but marathon wouldn't work with our huge program (it did work with some smaller applications) and I couldn't really do what I wanted with abbot. So I found the already mentioned Java Access Bridge. I didn't/don't know how to use it with autoit, but I managed to automate our program with JAB & C++, mostly by customizing Java Ferret though. I know a bit about how JAB works and I might be able to answer some questions if you have any. And maybe I'll try to use it with autoit too (I'd really love it).

Though... Marathon seems to be pretty nice program for the job. And I couldn't find a way to use button IDs (like name etc) straight, but I had to run the parent/child tree's (there's java monkey for finding the right routes). And somehow my program stalls when there's a new dialog window, but that might not happen with a new code.


Edit: And actually I didn't use the DLLs, but the source code. Anyways, atleast I think I have somekind of image how the calls should work... gotta test it.

Edited by Antiec, 14 September 2007 - 02:48 PM.

#14 Rocc



  • New Members
  • 1 posts

Posted 30 January 2008 - 12:31 AM

Perhaps this web site might help.

#15 junkew



  • Active Members
  • PipPipPipPipPipPip
  • 866 posts

Posted 11 March 2008 - 08:17 PM

And maybe when somebody comes around to read
it can be programmed in C (and with samples it looks like it could be done in AutoIT) but definitely is some effort.

#16 cbadal



  • Active Members
  • 10 posts

Posted 13 March 2008 - 04:41 PM

For automating Java you would really need something that understands Java.

Java swing controls are all drawn in Java drawing functions (these functions eventually call native platform dependent drawing functions).
This makes it nice for Java because they only need to write the drawing code once (and only update the low level platform dependent drawing functions in the actual JVM code).
This also means that their controls don't adhere to the Windows API (except for perhaps a top level Window, or some container).

The very old Java AWT toolkit uses native controls and if you came across a Java GUI that was written in AWT, you may be able to automate it with AutoIt.

#17 daluu



  • Active Members
  • PipPip
  • 105 posts

Posted 16 March 2009 - 07:32 PM

What about:

The free FEST framework/tool for Java? Though it's more developer & unit test centric. Works for Java applets too. Not sure how one can use it with AutoIt though.


#18 Xenobiologist


    Xx Code~Mega xX

  • MVPs
  • 4,866 posts

Posted 16 March 2009 - 09:17 PM


another tool which works quiet well is Loadrunner from HP.


Scripts & functions [color=#0000ff;]Organize Includes Let Scite organize the include files[/color] [color=#ff0000;]*new[/color]

[color=#000000;]Yahtzee The game "Yahtzee" (Kniffel, DiceLion)[/color]

[color=#000000;]LoginWrapper Secure scripts by adding a query (authentication)[/color]

[color=#000000;]_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)[/color]

[color=#000000;]Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.[/color]

[color=#000000;]MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times[/color]

#19 bo8ster



  • Active Members
  • PipPipPipPipPipPip
  • 1,189 posts

Posted 16 March 2009 - 10:26 PM

Thanks guys for all the links, really helpful!
Post your code because code says more then your words can. SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y. Use Opt("MustDeclareVars", 1)Brett F's Learning To Script with AutoIt V3Valuater's AutoIt 1-2-3, Class... is now in SessionContribution: Get SVN Rev Number, Control Handle under mouse, A Presentation using AutoIt, Log ConsoleWrite output in Scite

#20 vixLT6B



  • Banned (NOT IN USE)
  • 4 posts

Posted 19 March 2009 - 10:47 PM

I am not sure, but look at this: [link removed]

Edited by SmOke_N, 19 March 2009 - 11:36 PM.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users