Flashfires

Getting X,Y Coordinates via screenshot

12 posts in this topic

Hello Folks,

I'm new to AutoIT and hoping you can help me out. 

I'm interacting with a java application and don't have access to any of the controls.
But I need to select certain values within a dropdown list.  This has proven difficult.

To accomplish this, I've taken screenshots of the values I need to select in the dropdown.
I can open the dropdown list via clicking on screen coordinates.  What I need AutoIT to do,
is give me the "x, y" coordinates of the item by looking at the screenshot so I can click it.

I hope that makes sense, let me know if you need more details.

Thanks!
Moe

Share this post


Link to post
Share on other sites



Welcome to AutoIt and the forum!

Did you have a look at this UDF (library of User Defined Functions) which lets you automate Java applications?

 


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@Flashfires it sounds like you're trying to build a Rube Goldberg machine. Why take a screenshot, and then use X,Y coords to find the position on the application, rather than just manipulating the application? As it is a Java app, the typical control IDs are unavailable. But you should be able to use IUIAutomation (in the Examples forum). If you let us know the app, or show your screenshot, we can better assist. 


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

Hey Folks, thanks for the responses!

@water Wait, you're saying AutoIT can talk to Java apps?  That's a game changer ... I was convinced it couldn't.  Thank you, I'll check that out!

@JLogan3o13 Thanks for the IUIAutomation reference, I'll check it out.  I may be over complicating things out of ignorance of the AutoIT features.

I'll give this a look, and let you know how it works out either way.  Thanks again!

Share this post


Link to post
Share on other sites

I have never tried it myself. It might have its limitations but what I have read sounds promising.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Well, I've been trying to get the data access bridge working, but no dice.  Being honest, it's probably a little over my head.  As I'm running out of time to implement a solution for this, I'll check out a few of these other suggestions to see if I have any luck.

To better explain my issue, I have a dropdown list in a Java application.  I can click it with AutoIT, but can't read the values ... cause 'Java'.  The values of the dropdown list are ever changing, as it's database driven.  But a few of the values in there are constant, and those are the ones I'm interested in.  For the sake of example, let's say the constant values of the dropdown list are "cat", "dog",  and "Fish".

I need to create a loop where I:

  • Have AutoIT click the Dropdown List
  • Select one of these Constant Values
  • Do Other Stuff
  • Repeat

With no way to communicate with the dropdown list, my only thought was to take screenshots of these constant dropdown items.  (Cat, Dog, Fish)  My hope was to have AutoIT click the dropdown, and find the coordinates of one of these constants by using these saved screenshots.  (so I can select it)  If anyone knows how to do that, it might be easier for me to implement.

Or if I'm being silly and there's a much easier way to accomplish this, please let me know.  :)

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

What is jabsimplespy giving? Which windows version and which jdk is installed? In general advice to try trial version of ranorex spy. If that works you know controls are recognizable. if simplespy and inspect.exe do not recognize its only alternative is jabsimplespy for java objects and iaccessible interfaces for qt widgets.

Edited by junkew

Share this post


Link to post
Share on other sites
On 2/10/2017 at 0:43 PM, JLogan3o13 said:

 If you let us know the app, or show your screenshot, we can better assist. 

I second that.

Share this post


Link to post
Share on other sites

#10 ·  Posted

@JLogan3o13 @spudw2k Fair enough.

I work for a hotel chain, and this app works in conjunction with our reservation system.
I'm looking to see of AutoIT can automate some very repetitive and time consuming tasks.

I'll provide a screenshot to give you a visual, and I'll just scratch out the sensitive data.

Despite not having access to the controls (it's java) interacting with this screen is easy with AutoIT.
The only issue is with the dropdown list.  The problem is that rates are added and removed regularly.
So you can't count on items being in the same position.  The dropdown list does not accept keystrokes,
so you can't just type what you're looking for.  

But the items I need to select are constants, so they'll always be there ... just maybe in a different position.
The items are BAR1, BAR2, BAR3, etc.  So unless I manage to implement java access bridge, the only
way I can think to do this is to:

  • Screen capture the above values (BAR1, BAR2, BAR3, etc)
  • Open the dropdown list in AutoIT
  • Use those screenshots to find and click the values

Disclaimer: I'm not a pro developer ... I'm savvy enough to follow code samples, and tweak them as needed.
But implementing some of the solutions so far have been over my head.  I'm still trying though.

I hope that helps, let me know if you need more details.
Thanks for the assistance, it's much appreciated!

screenshot.jpg

Share this post


Link to post
Share on other sites

#12 ·  Posted

@junkew Ok, I'll take another stab at getting the Java Access Bridge working on Monday morning.  Thanks for the link!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now