Sign in to follow this  
Followers 0

How to automate Java windows/controls?

26 posts in this topic

#1 ·  Posted

So I had a look at a couple of Java programs. Notably the Java Control Panel that most people will have. And it's like a total black box. Using Window Spy++ I could see the main window but even though the window was full of buttons and checkboxes and stuff - there were no child windows!

I thought automating Java controls would be a case of finding out which messages it accepts to do what - but if it doesn't even present a child window to send to I'm a bit stumped. Spy++ couldn't even see any unknown/registered messages when you click on things.

Is there a Java SDK type thing for programming these sorts of apps - that might shown how they are control internally and help out. I mean, there must be an internal command for selecting a checkbox.

Ideas welcome.


"If you can post in the forum, that there is an issue with the forum, then the forum is working spectacularly. All else is cosmetics, and can totally suck it."
- boththose, April 2015.

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Here's what the professionals say about Java Applets.

AutoMate is able to find the Java Applet window but recognizes nothing inside it. The Java controls are basically "invisible" to AutoMate. From Windows' point of view the Java Applet looks like a window containing one large bitmap. Because of this, most of AutoMate's interactivity actions (i.e. 'Move Mouse to Object,' 'Press,' or 'Set Text' actions) will not work within a Java Applet window.

Solutions

The only possibility for AutoMate to interact with controls within a Java Applet Window is to set the focus to the Java Window and use the 'Send Keystrokes' action to tab through the fields to set and get data or press buttons.

http://www.networkautomation.com/automate/...23252F9A5553FE0

Edit: What Larry said completely contradicts this information.

Edited by Manadar

Share this post


Link to post
Share on other sites

#3 ·  Posted

Not quite. Junit looks to be a general java testing thing. But you need to use some addins to test gui stuff through a awk.robot java class. This class can mess around with a few buttons but appears to have no c++/external interface - so it's of no use :whistle:


"If you can post in the forum, that there is an issue with the forum, then the forum is working spectacularly. All else is cosmetics, and can totally suck it."
- boththose, April 2015.

Share this post


Link to post
Share on other sites

#4 ·  Posted

I've seen a few programs that decompile the .class files, that could at least give you information about the Java applet. This probably should not be considered an option...

My guess is that if you should succeed, Jon, you would be the first ever.

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

By the sounds of it this may be worth taking a look at.

IBM Rational Functional Tester

Edited by big_daddy

Share this post


Link to post
Share on other sites

#6 ·  Posted

What if we were able to capture user defined images and then OCR any text. Most text within these UI's (Flash and Java) is usually in standard fonts.

this would be a very powerfull ability for AutoIt. The ability to define a "virtual" object that could be manipulated via standard functions would no doubt push a large group of app automators (Like me) to AutoIt a lot faster. Until I can do that easily in AutoIT, I'm stuck in QuickTest Pro.

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

I have always been on the lookout for such tools and methods, but had never found anything other than trying to exploit methods that might be exposed by the applet to Javascript.

I did recently find a couple of resources dedicated to discussions of this but I have not had a chance to dig into them to see if there is anything promising.

This Wiki has a bunch of tool pointers: TestingGUIApplications and this Yahoo! group: java-gui-testing · Java GUI Testing

Dale

Edit: also seeing the source may help you... Google the DJ Java Decompiler

Edited by DaleHohm

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

#8 ·  Posted

I would be nice to read java windows, maybe this can help ...

Sun have developed a Java Access Bridge, which contains DLLs to read Java windows ..

http://java.sun.com/products/accessbridge/

API documentation is here ..

http://java.sun.com/products/accessbridge/docs/api.html

JavaFerret is a simple app, that reads java screens ...

http://java.sun.com/products/accessbridge/...javaferret.html

-Rasmus, Denmark

Share this post


Link to post
Share on other sites

#9 ·  Posted

Hi,

any progress on this?

We use this : IBM Rational Functional Tester

But it is written in Java.

So long,

Mega


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

#10 ·  Posted

Sorry I'm not an experienced developer just yet but I'm very interested if anyone has made any progress on this as well.

I've lookep up a load of how to automate Java windows/controls and this Java Access Bridge seems the best way.

I've been trying to use the Java Access Bridge and the Java Ferret and I'm not having much luck.

Just have to use DllCall to the WindowsAccessBridge.dll with the Java Access Bridge then I can start making calls to the classes within the program.


If it's too easy, then its not worth doingMike P. O'Keeffe BEng, Hardware Design EngineerLanguages: Perl, C, Expect, Visual Basic, Assembly

Share this post


Link to post
Share on other sites

#11 ·  Posted

I added ALisp

here

Apart from other things you can create by newLisp's tools java-based GUI.

For doing it newLisp has special guiserver.jar (GUI server) and guiserver.lsp.

Creation of java-based gui is the following process:

1. Start GuiServer demon

java.exe -jar "c:/Program Files/newlisp/guiserver.jar" 47011

You will see the demon's window worked (black window)

2. Start ALisp.au3

3. Evaluate by ALisp the expression:

(load "c:/Program Files/newlisp/guiserver.lsp")

4. Now you can evaluate lines (step by step or all as one) of some example from newlisp/guiserver folder.

ie button-demo.lsp

(gs:init)

(gs:set-trace true)

(gs:frame 'ButtonDemo 100 100 400 300 "Click on button or color panel")

(gs:set-resizable 'ButtonDemo nil)

(gs:panel 'ColorPanel 360 200)

(gs:set-background 'ColorPanel '(0 1 0) 0.2)

(gs:button 'aButton 'button-action "color")

(gs:set-flow-layout 'ButtonDemo "center" 2 15)

(gs:add-to 'ButtonDemo 'ColorPanel 'aButton)

(gs:set-visible 'ButtonDemo true)

(gs:mouse-event 'ColorPanel 'mouse-action)

;;;; define actions

(define (button-action id)

(gs:set-color 'ColorPanel (random) (random) (random)))

(define (mouse-action id type x y button cnt mods)

(gs:set-text 'ButtonDemo (format "%8s %3d:%3d %d %d %2d" type x y button cnt mods))

)

;(gs:listen)

Thus, you will create really java-based GUI having two handlers - button-action and mouse-action.

If you uncomment last line then GUI will independent window which handles button and mouse events.

This is a two-side body - ALisp & java window.

Maybe it is simplest platform for experiment with interaction between AutoIt and java, I think.

Remark. You should to install newlisp from this site, I think

BTW. ALisp (with help newlisp) shows how you can build simplest java-based bridge from Windows into Linux/Mac OS X/Solaris/BSD's platform


The point of world view

Share this post


Link to post
Share on other sites

#12 ·  Posted

Feed back from java-based Gui can be implemented by AutoItX.

See, example of AutoItX using (not tested)

here

However there are other more fast method, I think

;-)


The point of world view

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

Hi!

I had to automate a Java program at my work. I found two kinda nice java-based automation programs (marathon & abbot), but marathon wouldn't work with our huge program (it did work with some smaller applications) and I couldn't really do what I wanted with abbot. So I found the already mentioned Java Access Bridge. I didn't/don't know how to use it with autoit, but I managed to automate our program with JAB & C++, mostly by customizing Java Ferret though. I know a bit about how JAB works and I might be able to answer some questions if you have any. And maybe I'll try to use it with autoit too (I'd really love it).

Though... Marathon seems to be pretty nice program for the job. And I couldn't find a way to use button IDs (like name etc) straight, but I had to run the parent/child tree's (there's java monkey for finding the right routes). And somehow my program stalls when there's a new dialog window, but that might not happen with a new code.

Yours,

Lassi

Edit: And actually I didn't use the DLLs, but the source code. Anyways, atleast I think I have somekind of image how the calls should work... gotta test it.

Edited by Antiec

Share this post


Link to post
Share on other sites

#15 ·  Posted

And maybe when somebody comes around to read

http://java.sun.com/docs/books/jni/

it can be programmed in C (and with samples it looks like it could be done in AutoIT) but definitely is some effort.

Share this post


Link to post
Share on other sites

#16 ·  Posted

For automating Java you would really need something that understands Java.

Java swing controls are all drawn in Java drawing functions (these functions eventually call native platform dependent drawing functions).

This makes it nice for Java because they only need to write the drawing code once (and only update the low level platform dependent drawing functions in the actual JVM code).

This also means that their controls don't adhere to the Windows API (except for perhaps a top level Window, or some container).

The very old Java AWT toolkit uses native controls and if you came across a Java GUI that was written in AWT, you may be able to automate it with AutoIt.

Share this post


Link to post
Share on other sites

#18 ·  Posted

Hi,

another tool which works quiet well is Loadrunner from HP.

Mega


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

#19 ·  Posted

Thanks guys for all the links, really helpful!


Post your code because code says more then your words can. SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y. Use Opt("MustDeclareVars", 1)[topic="84960"]Brett F's Learning To Script with AutoIt V3[/topic][topic="21048"]Valuater's AutoIt 1-2-3, Class... is now in Session[/topic]Contribution: [topic="87994"]Get SVN Rev Number[/topic], [topic="93527"]Control Handle under mouse[/topic], [topic="91966"]A Presentation using AutoIt[/topic], [topic="112756"]Log ConsoleWrite output in Scite[/topic]

Share this post


Link to post
Share on other sites

#20 ·  Posted (edited)

I am not sure, but look at this: [link removed]

Edited by SmOke_N

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0