Sign in to follow this  
Followers 0
mrwizardd

Simple mouse clicking in Java app using info from simplespy?

11 posts in this topic

#1 ·  Posted (edited)

My situation:

I'm trying to automate ticket management software called HP Service Manager that's made in Java. I'm on Windows 7 64 bit and the application is 32 bit. Here's a screenshot of the Application:

2hqw0b5.jpg

We often have Emails (first, second, and third notifications) that need to be send out to users about their computers coming in, or that we received a ticket in their name and for them to stop by so we can work on it.

I have a little experience with Autoit. Definitely never made anything pretty but I could get some simple keyboard key and mouse clicking down when it was needed. That's essentially what I'm trying to do here, so it doesn't need to be pretty, just functional. 

What I'm having trouble with:

I'm having a bit of trouble getting started with this. I'm using the SimpleSpy.au3 to get info from the button I need to click in HPSM. Here's the info I get:

Mouse position is retrieved 1467-590
At least we have an element [General ][]
Having the following values for all properties: 
Title is: <General > Class   := <> controltype:= <UIA_EditControlTypeId> ,<50004> , (0000C354)
*** Parent Information ***
Title is: <Selected notebook tab - General > Class   := <> controltype:= <UIA_HyperlinkControlTypeId> ,<50005> , (0000C355)
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=1458;579;44;21
UIA_ClassNamePropertyId :=
UIA_ClickablePointPropertyId :=
UIA_ControllerForPropertyId :=
UIA_ControlTypePropertyId :=50004
UIA_CulturePropertyId :=0
UIA_DescribedByPropertyId :=
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FlowsToPropertyId :=
UIA_FrameworkIdPropertyId :=
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemContainingGridPropertyId :=
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=False
UIA_HelpTextPropertyId :=
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=True
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=False
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=True
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_ItemStatusPropertyId :=
UIA_ItemTypePropertyId :=
UIA_LabeledByPropertyId :=
UIA_LegacyIAccessibleChildIdPropertyId :=2
UIA_LegacyIAccessibleDefaultActionPropertyId :=Jump
UIA_LegacyIAccessibleDescriptionPropertyId :=
UIA_LegacyIAccessibleHelpPropertyId :=
UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=
UIA_LegacyIAccessibleNamePropertyId :=General 
UIA_LegacyIAccessibleRolePropertyId :=42
UIA_LegacyIAccessibleSelectionPropertyId :=
UIA_LegacyIAccessibleStatePropertyId :=14680128
UIA_LegacyIAccessibleValuePropertyId :=javascript:setSelected(arr_X13,'X14_p');
UIA_LocalizedControlTypePropertyId :=edit
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_MultipleViewSupportedViewsPropertyId :=
UIA_NamePropertyId :=General 
UIA_NativeWindowHandlePropertyId :=0
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=92156
UIA_ProviderDescriptionPropertyId :=[pid:92156,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionselectionPropertyId :=
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_SelectionItemSelectionContainerPropertyId :=
UIA_TableColumnHeadersPropertyId :=
UIA_TableItemColumnHeaderItemsPropertyId :=
UIA_TableRowHeadersPropertyId :=
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_TableItemRowHeaderItemsPropertyId :=
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_ValueValuePropertyId :=javascript:setSelected(arr_X13,'X14_p');
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

 

I know I must be not doing something very basic to get this to work. When I run my script below, the window will activate and I won't get any errors from Autoit, but it won't actually click this button.

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "UIAWrappers.au3"
#include <Misc.au3>
#include <Constants.au3>


#cs ----------------------------------------------------------------------------


 AutoIt Version: 3.3.12.0


#ce ----------------------------------------------------------------------------


#AutoIt3Wrapper_UseX64=Y


WinActivate("HPSM Overlay v2")
ControlClick("General", "", "controltype [UIA_HyperlinkControlTypeId]")

I've tried adding the x. before it as well. With that I get an error that Autoit doesn't recognize "x." What am I doing wrong? What would be the correct way to script it to simply click the button? I think once I get something as simple as this down I can get the rest of what I'm hoping to do. If you might have links to basic examples of automating a java app that would be greatly appreciated too! Thanks folks.

 
Edited by mrwizardd

Share this post


Link to post
Share on other sites



where did you get simplepsy.au3? can you share it to me? :zorro:


ill get to that... i still need to learn and understand a lot of codes graduated.gif

Correct answer, learn to walk before you take on that marathon.

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

What does "AU3Info" show?

It looks like the Id is 50004

So you might want to try

ControlClick("HPSM Overlay v2", "", "50004")

Syntax is: ControlClick ( "title", "text", controlID [, button = "left" [, clicks = 1 [, x [, y]]]] )

From the controlID onwards is optional.

Of course im not sure the id is that, but won't harm for sure.

EDIT: got the title wrong, if you know it is "General" then put that.

Edited by careca

Spoiler

Paster - Main function is to paste text, but has more functions.

OpenW - Open With... alternative, Open any file with any application, set it's icon, set application as default.

Renamer - Rename files and folders, remove portions of text from the filename etc.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

 

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

where did you get simplepsy.au3? can you share it to me? :zorro:

 Having trouble finding that original post where a guy made it and required a .jar file and some .au3's in the "Include" folder. But here is the script:

(EDIT: Found it! '?do=embed' frameborder='0' data-embedContent>> )

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "UIAWrappers.au3"
#include <Misc.au3>


AutoItSetOption("MustDeclareVars", 1)


#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os


dim $oldUIElement ; To keep track of latest referenced element
dim $Form1, $Edit1, $label1, $label2, $msg, $x, $y, $oUIElement, $oTW, $objParent, $oldElement, $text1, $t


;~ Some references for reading
;~ [url=http://support.microsoft.com/kb/138518/nl]http://support.microsoft.com/kb/138518/nl[/url]  tagpoint structures attention point
;~ [url=http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/]http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/[/url]
;~ [url=http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx]http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx[/url]


HotKeySet("{ESC}", "Close") ; Set ESC as a hotkey to exit the script.
HotKeySet("^w", "GetElementInfo") ; Set Hotkey Ctrl+M to get some basic information in the GUI


#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("Form1", 1024, 768, 192, 124)
$Edit1 = GUICtrlCreateEdit("", 10, 10, 600, 700)
;~ $Edit1 = GUICtrlCreateEdit("", 10, 10, 10, 10)
$Label1 = GUICtrlCreateLabel("Ctrl+W to capture information", 640,10,600,25)
$Label2 = GUICtrlCreateLabel("Escape to exit", 640,45,600,25)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###


_UIA_Init()


; Run the GUI until the dialog is closed
While true
$msg = GUIGetMsg()
sleep(100)
;~ if _ispressed(01) Then
;~ getelementinfo()
;~ endif
If $msg = $GUI_EVENT_CLOSE Then ExitLoop
WEnd


Func GetElementInfo()
Local $hWnd
Local $tStruct = DllStructCreate($tagPOINT) ; Create a structure that defines the point to be checked.
;~ Local $tStruct = DllStructCreate("INT64,INT64")
;~ Local $tStruct =_AutoItObject_DllStructCreate($tagPoint)


ToolTip("")
Global $UIA_oUIAutomation ;The main library core CUI automation reference
Global $UIA_oDesktop, $UIA_pDesktop  ;Desktop will be frequently the starting point


Global $UIA_oUIElement, $UIA_pUIElement  ;Used frequently to get an element
Global $UIA_oTW, $UIA_pTW  ;Generic treewalker which is allways available


$x=MouseGetPos(0)
$y=MouseGetPos(1)
DllStructSetData($tStruct, "x", $x)
DllStructSetData($tStruct, "y", $y)
consolewrite(DllStructGetData($tStruct,"x") & DllStructGetData($tStruct,"y"))


;~ consolewrite("Mouse position is retrieved " & @crlf)
$UIA_oUIAutomation.ElementFromPoint($tStruct,$UIA_pUIElement )
;~ $objUIAutomation.ElementFromPoint(DllStructGetPtr($tStruct),$pUIElement)
;~ consolewrite("Element from point is passed, trying to convert to object ")
$oUIElement = objcreateinterface($UIA_pUIElement,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)


$UIA_oUIAutomation.RawViewWalker($UIA_pTW)
$oTW=ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        msgbox(1,"UI automation treewalker failed", "UI Automation failed failed",10)
    EndIf


local $parentHandle


$oTW.getparentelement($oUIElement,$parentHandle)
$objParent=objcreateinterface($parentHandle,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)
If IsObj($objParent) = 0 Then
msgbox(1,"No parent", "UI Automation failed failed",10)
EndIf


if isobj($oldUIElement) Then
if $oldUIElement=$oUIElement then
return
endif
endif
_WinAPI_RedrawWindow(_WinAPI_GetDesktopWindow(), 0, 0, $RDW_INVALIDATE + $RDW_ALLCHILDREN) ; Clears Red outline graphics.
GUICtrlSetData($Edit1, "Mouse position is retrieved " & $x & "-" & $y & @CRLF)
$oldElement=$oUIElement


If IsObj($oUIElement) Then
;~  ConsoleWrite("At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF)
GUICtrlSetData($Edit1, "At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF,1)
    $text1="Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= " _
& "<" &  _UIA_getControlName(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ">" & @TAB  _
& ",<" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB  _
& ", (" &  hex(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ")" & @TAB & @CRLF


    $text1=$text1 & "*** Parent Information ***" & @CRLF


    $text1=$Text1 & "Title is: <" &  _UIA_getPropertyValue($objParent,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($objParent,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= " _
& "<" &  _UIA_getControlName(_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId)) &  ">" & @TAB  _
& ",<" &  _UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId) &  ">" & @TAB  _
& ", (" &  hex(_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId)) &  ")" & @TAB & @CRLF


$text1=$text1 & "*** Detailed properties of the highlighted element ***"


$text1= $text1 & @CRLF & _UIA_getAllPropertyValues($oUIElement)


GUICtrlSetData($Edit1, "Having the following values for all properties: " & @crlf & $text1 & @CRLF, 1)




_GUICtrlEdit_LineScroll($Edit1, 0, 0 - _GUICtrlEdit_GetLineCount($Edit1))


$t=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId),";")
_UIA_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf


EndFunc   ;==>Example


Func Close()
Exit
EndFunc   ;==>Close
Edited by mrwizardd

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

What does "AU3Info" show?

It looks like the Id is 50004

So you might want to try

ControlClick("HPSM Overlay v2", "", "50004")

Syntax is: ControlClick ( "title", "text", controlID [, button = "left" [, clicks = 1 [, x [, y]]]] )

From the controlID onwards is optional.

Of course im not sure the id is that, but won't harm for sure.

 

AU3Info is not recognizing any of the Java buttons, just the window. From what I understand, native Autoit does not support Java buttons and controls. But using that script above, it can identify them. May need some of the files I originally downloaded first. See

Junkew put together something that will allow Autoit to send commands to them inside the Java App. I tried your suggestions, thank you, but no dice :(

Edited by mrwizardd

Share this post


Link to post
Share on other sites

Sadly i don't know of any alternative, only mouseclick.


Spoiler

Paster - Main function is to paste text, but has more functions.

OpenW - Open With... alternative, Open any file with any application, set it's icon, set application as default.

Renamer - Rename files and folders, remove portions of text from the filename etc.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

 

Share this post


Link to post
Share on other sites

That would actually work great if I could actually get it to click the button being seen by simplespy. I may have to try and message Junkew about how to use that stuff.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Please read the iuiautomation thread and see what can and cannot work. It heavily depends on how the java application is build. For example lotus notes is written in java and supports full accessibility. If its the same for your application you can best assess by looking with simplespy how easy elements are highligthed. Commands and examples are in the first post attached in zip files.

And you should then not us autoit commands to control elements.

Edited by junkew
1 person likes this

Share this post


Link to post
Share on other sites

Well, at the top of the post with the spy info you got: "Mouse position is retrieved 1467-590"

Therefore:

MouseClick("left", 1467, 590)

Spoiler

Paster - Main function is to paste text, but has more functions.

OpenW - Open With... alternative, Open any file with any application, set it's icon, set application as default.

Renamer - Rename files and folders, remove portions of text from the filename etc.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

 

Share this post


Link to post
Share on other sites

 

Well, at the top of the post with the spy info you got: "Mouse position is retrieved 1467-590"

Therefore:

MouseClick("left", 1467, 590)

 But the window may not always be in the same place. Certain line items may move or the constraints may change :(

Share this post


Link to post
Share on other sites

Maybe you could do a pixel search, for a specific shade of color specific to a button.


Spoiler

Paster - Main function is to paste text, but has more functions.

OpenW - Open With... alternative, Open any file with any application, set it's icon, set application as default.

Renamer - Rename files and folders, remove portions of text from the filename etc.

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Firefox Profile Backup - Backup/restore previously saved profile.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0