Jump to content
Sign in to follow this  
guestscripter

Java/SunAwtFrame data entry and automation

Recommended Posts

guestscripter

I´ve got an already usably working script for entering a reservation in to a Java App,

but to give it more than very basic functionality and stability I´m stuck.

Is there any way beyond MouseGetCursor and WinActivate, and other than lot´s of Sleep, that I can make this more stable?

sometimes there is a popup in the app, which of course the script doesn´t see,

and sometimes the script "trips" and enters the wrong data in the wrong field, and that of course causes lots of errors and popups...

yes, there´s a "Java UDF" Topic in the example scripts section, but that didn´t work for me, and the development on that seems to have stopped years ago... (unless I´m overlooking another thread somewhere?) A newer thread about, as far as I understand it, automating hard-to-automate controls might be a direction: ?do=embed' frameborder='0' data-embedContent>...

 

Here´s a cut-out of what I´ve built so far:

Global $sWinTitle = "OPERA PMS [Version"
Global $wHandle = WinGetHandle($sWinTitle)
Global $cHandle = ControlGetHandle($wHandle, "", "")

#region Parser
;that´s a separate story
#endregion

#region Entry
_BlockWindow()

OperaSend("!.")

OperaSend($aRES[$ixGuestSurName],2)
OperaSend("{TAB}")
OperaSend($aRES[$ixGuestFirstName],2)
OperaSend("{TAB}")
OperaSend("i{ENTER}")
OperaSend("!n")
;etc...

_unBlock()
#endregion


Func OperaSend($sSend, $flag = 0)
    While Not WinActive($wHandle);force the window to be active
        WinActivate($wHandle)
    WEnd
    If $flag = 2 Then ;$flag = 2 send longer strings of text with clipput+paste
        Do
            ClipPut("")
            ClipPut($sSend)
            Sleep(128) ;seems to be necessary
        Until @error = 0
        Do
            $x = ControlSend($wHandle, "", $cHandle, "^v");{CTRLDOWN}{CTRLUP}
        Until $x = 1
        Sleep(128)
    Else ;flag = 1 Sends Raw, $flag = 0 Sends "normal"
        Do
            $x = ControlSend($wHandle, "", $cHandle, $sSend, $flag)
        Until $x = 1
        Sleep(64)
    EndIf
    Do
        Sleep(8)
    Until MouseGetCursor() = 2 ;check for non-arrow mouse cursors (busy, etc.)
    Sleep(8)
EndFunc   ;==>OperaSend


#include <Misc.au3> ; for MouseTrap Function
Func _BlockWindow()
    $pos = WinGetPos($wHandle)
    If IsHWnd($wHandle) Then
        WinActivate($wHandle)
        WinSetState($wHandle, "", @SW_DISABLE)
        WinSetTrans($wHandle, "", 255 * 0.92)
        _MouseTrap($pos[0] + 10, $pos[1] + 32)
        _BlockKeys(1)
        ConsoleWrite("Blocked" & @CR)
        $blocked = 1
    EndIf
EndFunc ;==>_BlockWindow
Func _unBlock()
    If IsHWnd($wHandle) Then
        _BlockKeys()
        _MouseTrap()
        WinSetTrans($wHandle, "", 255)
        WinSetState($wHandle, "", @SW_ENABLE)
        ConsoleWrite("Unblocked" & @CR)
        $blocked = 0
    EndIf
EndFunc ;==>_unBlock



Func _BlockKeys($flag = 0);instead of _BlockInput(), because no Admin privileges
    If $flag = 1 Then
        HotKeySet("{TAB}", "_BlockKeysDummy")
        HotKeySet("^{TAB}", "_BlockKeysDummy")
        HotKeySet("!{TAB}", "_BlockKeysDummy");doesn´t really work
        HotKeySet("+{TAB}", "_BlockKeysDummy")
        HotKeySet("#{TAB}", "_BlockKeysDummy")
        HotKeySet("{ENTER}", "_BlockKeysDummy")
        ;...etc.
    Else
        HotKeySet("{TAB}")
        HotKeySet("^{TAB}")
        HotKeySet("!{TAB}");doesn´t really work
        HotKeySet("+{TAB}")
        HotKeySet("#{TAB}")
        HotKeySet("{ENTER}")
        ;...etc.
    EndIf
EndFunc   ;==>_BlockKeys
Edited by guestscripter

Share this post


Link to post
Share on other sites
guestscripter

Is there any smarter way or more elegant to do this?

Func CSTestInWin($wHandle, $aCheckSumTest, $iCheckSumStep = 1, $bCheckSumMode = 1)
    ;$aCheckSumTest is Array where [0] is a checksum,
        ;and [1]-[4] are left, top, right and bottom coordinates respectively.
    ;The coordinates shall be relative to the window specified by the $wHandle.
        ;Option "PixelCoordMode" needs to be set to 0=relative.
    If WinExists($wHandle) And UBound($aCheckSumTest) = 5 And Opt("PixelCoordMode") = 0 Then
        Local $aCheckSum = PixelChecksum($aCheckSumTest[1], $aCheckSumTest[2], _
                                         $aCheckSumTest[3], $aCheckSumTest[4], $iCheckSumStep, $wHandle, $bCheckSumMode)
        If $aCheckSum = $aCheckSumTest[0] Then
            Return True
        Else
            Return False
        EndIf
    Else
        SetError(1)
        ConsoleWrite("_CheckSumTestInWindow found invalid parameter, or ""PixelCoordMode"" option not set to 0."&@CRLF)
        Return False
    EndIf
EndFunc   ;==>CSTestInWin
;Usage example:
Opt("PixelCoordMode",0)
;...
Local Const $JavaAppWinTitle = "xyz"
Local Const $aProfileSearchNoResult[5] = [1798928452, 140, 155, 1002, 748]
Local Const $aProfileSearchWithResults[5] = [4654546456, 140, 155, 1002, 748]
;...
$wHandleJavaApp = WinWaitActive($sJavaAppWinTitle)
;...

While NOT CSTestInWin($wHandleJavaApp,$aProfileSearchNoResult) _
          OR CSTestInWin($wHandleJavaApp,$aProfileSearchWithResults) _
          OR $bExitLoop = 0
;tight loop
WEnd

If CSTestInWin($wHandleJavaApp,$aProfileSearchNoResult) Then
    CreateNewProfile()
ElseIf CSTestInWin($wHandleJavaApp,$aProfileSearchWithResults) Then
    QueryUser_UseExistingProfileOrCreateNew()
EndIf

To clarify, this is part of a script I am building that automates software that uses Java, so not identifiable Controls to "see" or work with (although I found I can use ControlSend to send to the right window at least).

Already tried Java UDF and it looks like I need a workaround because this "Java Access Bridge" that Java UDF works with only helps with identifying Controls in "accessibility friendly" Java Software. So that way works for example for jEdit, but not for the software I want to automate. (if you know another way please let me know!)

I looked around at different things like "ImageSearch" and "FastFind".. ...UDFs that use .dll´s to offer functionality like searching for a button by a .bmp file, just then I found that the native PixelCheckSum function is already well powerfull.

Edited by guestscripter

Share this post


Link to post
Share on other sites
guestscripter

First of all thank you very much for the recommendation. I downloaded the include files and ran the "simple spy demo" from the first post in that thread.

With my java interface window in focus, and the caret in an edit box selecting the text "asdasdasdasd" as a test, I got what I have pasted below.

Maybe this can help you help me help myself?

Mouse position is retrieved 649-314
At least we have an element [MYWINDOWTITLEWASHERE ][SunAwtFrame]
Having the following values for all properties: 
Title is: <MYWINDOWTITLEWASHERE >   Class   := <SunAwtFrame>    controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    
*** Parent Information ***
Title is: <Desktop> Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=Alt+ 
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=448;156;1007;694
UIA_ClassNamePropertyId :=SunAwtFrame
UIA_ClickablePointPropertyId :=
UIA_ControllerForPropertyId :=
UIA_ControlTypePropertyId :=50032
UIA_CulturePropertyId :=0
UIA_DescribedByPropertyId :=
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FlowsToPropertyId :=
UIA_FrameworkIdPropertyId :=Win32
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemContainingGridPropertyId :=
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=True
UIA_HelpTextPropertyId :=
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=False
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=True
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=True
UIA_IsValuePatternAvailablePropertyId :=False
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=True
UIA_ItemStatusPropertyId :=
UIA_ItemTypePropertyId :=
UIA_LabeledByPropertyId :=
UIA_LegacyIAccessibleChildIdPropertyId :=0
UIA_LegacyIAccessibleDefaultActionPropertyId :=
UIA_LegacyIAccessibleDescriptionPropertyId :=
UIA_LegacyIAccessibleHelpPropertyId :=
UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=Alt+ 
UIA_LegacyIAccessibleNamePropertyId :=MYWINDOWTITLEWASHERE      
UIA_LegacyIAccessibleRolePropertyId :=10
UIA_LegacyIAccessibleSelectionPropertyId :=
UIA_LegacyIAccessibleStatePropertyId :=1048580
UIA_LegacyIAccessibleValuePropertyId :=
UIA_LocalizedControlTypePropertyId :=Fenster
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_MultipleViewSupportedViewsPropertyId :=
UIA_NamePropertyId :=MYWINDOWTITLEWASHERE     
UIA_NativeWindowHandlePropertyId :=1836932
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=2696
UIA_ProviderDescriptionPropertyId :=[pid:5780,hwnd:0x1C0784 Main:Nested [pid:2696,hwnd:0x1C0784 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=42;1836932
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionselectionPropertyId :=
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_SelectionItemSelectionContainerPropertyId :=
UIA_TableColumnHeadersPropertyId :=
UIA_TableItemColumnHeaderItemsPropertyId :=
UIA_TableRowHeadersPropertyId :=
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_TableItemRowHeaderItemsPropertyId :=
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=True
UIA_TransformCanResizePropertyId :=True
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_ValueValuePropertyId :=
UIA_WindowCanMaximizePropertyId :=True
UIA_WindowCanMinimizePropertyId :=True
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=2
UIA_WindowWindowVisualStatePropertyId :=0

I can see that it found a Window Handle and a Control Handle of some sort...

...just I suspect this control handle is the "one size fits all" that I have resorted to using so far like

$wHandleOpera = WinWaitActive($sOperaWinTitle)
Global $cHandleOpera = ControlGetHandle($wHandleOpera, "", "")
;..stuff here...
ControlSend($wHandleOpera, "", $cHandleOpera, "{TAB}")
ClipPut("asfljalsfjlasjflkasj")
ControlSend($wHandleOpera, "", $cHandleOpera, "^v")

which, in a shaky (script trips when user presses a key or window loses focus, so lots of mouse/key/etc. blocking) and limited (needing pixelchecksums to give the script any kind of "eyes") way, kind of works....

 

Or can you see something more usable in the results?

Can these results indicated yes/no if this software is potentially automatable with the IUIAutomation Functions?

Edited by guestscripter

Share this post


Link to post
Share on other sites
junkew

The UI Automation will not come far enough if it only exposes AWT Frame

Attached some code which shows JAB (Java Accessibility Bridge) can be loaded from AutoIT. Not finished at all (put on my long term TODO list)

Maybe its usefull for you or others as a starting point. 

;~ By inspecting the WindowsAccessBridge-32.dll it reveals some information about the hidden dialogs
;~   So it seems the hidden dialog is shown after you call windows_run() no clue if interaction is needed
;~
;~ Somehow it sends a message unclear if this is to the JVM to respond to
;~   push   SSZ6E73E320_AccessBridge_FromJava_Hello
;~   push   SSZ6E73E300_AccessBridge_FromWindows_Hello
;~   db 'AccessBridge-FromWindows-Hello',0
;~   db 'AccessBridge-FromJava-Hello',0

;~ Copied from this NVDA reference and translated to AutoIT
;~ http://www.webbie.org.uk/nvda/api/JABHandler-pysrc.html
;~
;~ def initialize():
;~         global isRunning
;~         if not bridgeDll:
;~                 raise NotImplementedError("dll not available")
;~         bridgeDll.Windows_run()
;~         #Accept wm_copydata and any wm_user messages from other processes even if running with higher privilages
;~***         ChangeWindowMessageFilter=getattr(windll.user32,'ChangeWindowMessageFilter',None)
;~***         if ChangeWindowMessageFilter:
;~***                 if not ChangeWindowMessageFilter(winUser.WM_COPYDATA,1):
;~***                         raise WinError()
;~***                 for msg in xrange(winUser.WM_USER+1,65535):
;~***                         if not ChangeWindowMessageFilter(msg,1):
;~***                                 raise WinError()
;~         #Register java events
;~         bridgeDll.setFocusGainedFP(internal_event_focusGained)
;~         bridgeDll.setPropertyActiveDescendentChangeFP(internal_event_activeDescendantChange)
;~         bridgeDll.setPropertyNameChangeFP(event_nameChange)
;~         bridgeDll.setPropertyDescriptionChangeFP(event_descriptionChange)
;~         bridgeDll.setPropertyValueChangeFP(event_valueChange)
;~         bridgeDll.setPropertyStateChangeFP(internal_event_stateChange)
;~         bridgeDll.setPropertyCaretChangeFP(internal_event_caretChange)
;~         isRunning=True

;create the structs
;~
;~  http://docs.oracle.com/javase/accessbridge/2.0.2/api.htm
;~
;~ #define MAX_STRING_SIZE     1024
;~ #define SHORT_STRING_SIZE    256
;~                  struct AccessBridgeVersionInfo {
;~  wchar_t VMversion[SHORT_STRING_SIZE];              // version of the Java VM
;~  wchar_t bridgeJavaClassVersion[SHORT_STRING_SIZE]; // version of the AccessBridge.class
;~  wchar_t bridgeJavaDLLVersion[SHORT_STRING_SIZE];   // version of JavaAccessBridge.dll
;~  wchar_t bridgeWinDLLVersion[SHORT_STRING_SIZE];    // version of WindowsAccessBridge.dll
;~ };

;~ Local $AccessBridgeVersionInfo=DllStructCreate("WCHAR VMversion[256];WCHAR bridgeJavaClassVersion[256];WCHAR bridgeJavaDLLVersion[256];WCHAR bridgeWinDLLVersion[256]")
Local $AccessBridgeVersionInfo=DllStructCreate("WCHAR VMversion[256];WCHAR bridgeJavaClassVersion[256];WCHAR bridgeJavaDLLVersion[256];WCHAR bridgeWinDLLVersion[256]")

;~ struct AccessibleContextInfo {
;~  wchar_ name[MAX_STRING_SIZE];        // the AccessibleName of the object
;~  wchar_ description[MAX_STRING_SIZE]; // the AccessibleDescription of the object
;~  wchar_ role[SHORT_STRING_SIZE];      // localized AccesibleRole string
;~  wchar_ states[SHORT_STRING_SIZE];    // localized AccesibleStateSet string
;~                                       //   (comma separated)
;~  jint indexInParent                   // index of object in parent
;~  jint childrenCount                   // # of children, if any
;~  jint x;                              // screen x-axis co-ordinate in pixels
;~  jint y;                              // screen y-axis co-ordinate in pixels
;~  jint width;                          // pixel width of object
;~  jint height;                         // pixel height of object
;~  BOOL accessibleComponent;            // flags for various additional
;~  BOOL accessibleAction;               // Java Accessibility interfaces
;~  BOOL accessibleSelection;            // FALSE if this object doesn't
;~  BOOL accessibleText;                 // implement the additional interface
;~  BOOL accessibleInterfaces;           // new bitfield containing additional
;~                                       //   interface flags
;~ };
;~ Local $AccessibleContextInfo=DllStructCreate("WCHAR name[1024];WCHAR description[1024];WCHAR role[256];WCHAR states[256];int indexInParent;int childrenCount;int x;int y;int width;int height;BOOL accessibleComponent;BOOL accessibleAction;BOOL accessibleSelection;BOOL accessibleText;BOOL accessibleInterfaces")
Local $AccessibleContextInfo=DllStructCreate("WCHAR name[1024];WCHAR description[1024];WCHAR role[256];WCHAR states[256];int64 indexInParent;int64 childrenCount;int64 x;int64 y;int64 width;int64 height;byte accessibleComponent;byte accessibleAction;byte accessibleSelection;byte accessibleText;byte accessibleinterfaces")


;~ #AutoIt3Wrapper_UseX64=Y

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>

local $hwnd
local $i
local $result
local $vmID
local $ac

;~ Make messages elevated
_ChangeWindowMessageFilter($WM_COPYDATA,1)
for $i=$WM_USER to $WM_USER+65536
    _ChangeWindowMessageFilter($i, 1)
Next

Func _ChangeWindowMessageFilter($iMsg, $iAction)
    Local $aCall = DllCall("user32.dll", "bool", "ChangeWindowMessageFilter", "dword", $iMsg, "dword", $iAction)
    If @error Or Not $aCall[0] Then Return SetError(1, 0, 0)
    Return 1
EndFunc

local $bridgeDLL=DLLOPEN("WindowsAccessBridge-32.dll")
;~ sleep(500)
If $bridgeDLL=true Then
    consolewrite($bridgeDLL & @CRLF)
Else
    consolewrite("DLL not found try to change to WindowsAccessBridge-64.dll if you use 64 bits autoit and windows")
endIf

;~ TODO: Handle messages received from initialize

$result =dllcall($bridgeDLL,"int:cdecl", "Windows_run")
consolewrite($result & " " & @error & " initializeAccessBridge is finished")

sleep(250)

consolewrite("Windows_run passed :" & $result & @CRLF)

Local $var = WinList()
consolewrite("Before loading all Windows:" & $var[0][0] & @CRLF)

For $i = 1 To $var[0][0]
; Only display visble windows that have a title
    If IsVisible($var[$i][1]) Then
        local $handle=wingethandle($var[$i][0])
        $result =dllcall($bridgeDLL,"BOOL:cdecl", "isJavaWindow", "hwnd", $handle)
        if @error=0 Then
            if $result[0]=1 Then
                consolewrite( $i & " Java Window Title=" & $var[$i][0] &  " Handle=" & $var[$i][1] & @TAB & " res: " &  $result[0] & @CRLF)

            local $ac=0
            local $vmID=0

            $result =dllcall($bridgeDLL,"BOOL:cdecl", "getAccessibleContextFromHWND", "hwnd", $handle, "long*", $vmID, "ptr*", $ac)

                if @error=0 Then
                    consolewrite("We have a context " & @CRLF)
                    $vmID=$result[2]
                    $ac=$result[3]

                    if @error > 0 then consolewrite("Struct error")

;~                  $result =dllcall($bridgeDLL, "BOOL:cdecl", "getVersionInfo", "long", $vmId, "struct", $AccessBridgeVersionInfo)
                    $result =dllcall($bridgeDLL, "BOOL:cdecl", "getVersionInfo", "long", $vmId, "ptr", DllStructGetPtr($AccessBridgeVersionInfo))
                    consolewrite( @error & " context found of " & $vmID & @CRLF)
                    if @error=0 Then

                        $s1=dllstructgetdata($AccessBridgeVersionInfo, "VMVersion")
                        $s2=dllstructgetdata($AccessBridgeVersionInfo, "bridgeJavaClassVersion")
                        $s3=dllstructgetdata($AccessBridgeVersionInfo, "bridgeJavaDLLVersion")
                        $s4=dllstructgetdata($AccessBridgeVersionInfo, "bridgeWinDLLVersion")

                        consolewrite("VMVersion: <" & $s1 & ">" & @CRLF)
                        consolewrite("bridgeJavaClassVersion: <" & $s2 & ">" & @CRLF)
                        consolewrite("bridgeJavaDLLVersion: <" & $s3 & ">" & @CRLF)
                        consolewrite("bridgeWinDLLVersion: <" & $s4 & ">" & @CRLF)
                    EndIf

;~ Retrieves an AccessibleContextInfo object of the AccessibleContext object ac.
                    $result =dllcall($bridgeDLL, "BOOL:cdecl", "getAccessibleContextInfo","long", $vmId, "ptr", $ac, "ptr", DllStructGetPtr($AccessibleContextInfo))
;~ Local $AccessibleContextInfo=DllStructCreate("WCHAR name[1024];WCHAR description[1024];WCHAR role[256];WCHAR states[256];int indexInParent;int childrenCount;int x;int y;int width;int height;BOOL accessibleComponent;BOOL accessibleAction;BOOL accessibleSelection;BOOL accessibleText;BOOL accessibleInterfaces")
                    if @error<>0 Then
                        consolewrite("We have an error " & @CRLF)
                    EndIf
                    $s1=dllstructgetdata($AccessibleContextInfo, "name")
                    $s2=dllstructgetdata($AccessibleContextInfo, "description")
                    consolewrite("name: <" & $s1 & ">" & @CRLF)
                    consolewrite("description: <" & $s2 & ">" & @CRLF)

;~ AccessibleContext GetAccessibleChildFromContext(long vmID, AccessibleContext ac, jint index);

;~ Returns an AccessibleContext object that represents the nth child of the object ac, where n is specified by the value index.


                Else
                    consolewrite( @error & " No context found" & @CRLF)
                endif
            EndIf
        Else
        EndIf
    endif
Next
;~ http://www.autohotkey.com/board/topic/95343-how-to-send-to-unseen-controls-in-a-java-app/
local $result =dllcall($bridgeDLL,"BOOL", "shutdownAccessBridge")

Func IsVisible($handle)
    If BitAND(WinGetState($handle), 2) Then
        Return 1
    Else
        Return 0
    EndIf

EndFunc   ;==>IsVisible
  • Like 1

Share this post


Link to post
Share on other sites
guestscripter

Thanks junkew.

For my current skill level the code looks heavy, and besides I highly doubt that the App I want to automate supports accessibility... (I´ve done some research on JAB before when trying to use the Java UDF ('?do=embed' frameborder='0' data-embedContent>>) by seangriffin). Thanks a lot anyway!

I´m marking the thread as "solved", as the question, thanks to you, has been answered as well as possible for now I reckon.

Planning to instead continue cleaning up my "dirty" method with PixelChecksums and ControlSend (to the overall window) as much as possible... (I´ve made progress with that to a point that actually looks quite satisfying)

Edited by guestscripter

Share this post


Link to post
Share on other sites
guestscripter

P.S. If anyone has interest (or can improve the direction I´m taking) here are snippets of how the "dirty" method is working. (any pointers in regards to best coding practice would be great)

Global $wHandleOpera
Global $cHandleOpera

$wHandleOpera = WinActivate($sOperaWinTitle)
$cHandleOpera = ControlGetHandle($wHandleOpera, "", "")  
Func OperaSend($sSend, $flag = 0)
    WaitForTheMouse()
    If $flag = 2 Then
        ClipPut("")
        ClipPut($sSend)
        Sleep(128);Must not be removed
        Do
            Local $x = ControlSend($wHandleOpera, "", $cHandleOpera, "^v")
        Until $x = 1
        ;Sleep(128)
    Else
        Sleep(64)
        Do
            $x = ControlSend($wHandleOpera, "", $cHandleOpera, $sSend, $flag)
        Until $x = 1
    EndIf
EndFunc   ;==>OperaSend
Func WaitForTheMouse()
    While MouseGetCursor() <> 2
        ;cr("WaitForTheMouse")
        Sleep(100)
    WEnd
EndFunc   ;==>WaitForTheMouse
Func CSTestInOpera($aCheckSumTest, $iCheckSumStep = 1, $bCheckSumMode = 1)
    ;$aCheckSumTest is Array where [0] is a checksum and [1]-[4] are left, top, right and bottom coordinates respectively.
    ;Window Handle from Global Scope. Option "PixelCoordMode" needs to be set to 0=relative.
    If WinExists($wHandleOpera) And UBound($aCheckSumTest) = 5 Then
        Local $aCheckSum = PixelChecksum($aCheckSumTest[1], $aCheckSumTest[2], $aCheckSumTest[3], $aCheckSumTest[4], $iCheckSumStep, $wHandleOpera, $bCheckSumMode)
        If $aCheckSum = $aCheckSumTest[0] Then
            Return True
        Else
            Return False
        EndIf
    Else
        SetError(1)
        Return False
    EndIf
EndFunc   ;==>CSTestInOpera
Func AdlibCheckWinStillActive()
    If Not WinActive($wHandleOpera) Then
        WinActivate($wHandleOpera)
        If @error Then
            _Exit()
        EndIf
    EndIf
EndFunc   ;==>AdlibCheckWinStillActive
Func BlockInactiveWindowsAndMouse($flag = 0);if Flag = 1 then just reblock mouse
    ;Note: Check out http://www.autoitscript.com/forum/topic/87735-blockinputex-udf/
    If $flag = 0 Then
        WinSetOnTop($wHandleOpera, "", 1)
        WinSetState($wHandleOpera, "", @SW_DISABLE)
        Global $aBlockerWinList = WinList()
        For $i = 1 To $aBlockerWinList[0][0]
            If BitAND(WinGetState($aBlockerWinList[$i][1]), 2) And BitAND(WinGetState($aBlockerWinList[$i][1]), 4) And $aBlockerWinList[$i][1] <> $wHandleOpera Then;Visible+Enabled+NotActive
                WinSetState($aBlockerWinList[$i][1], "", @SW_DISABLE)
                $aBlockerWinList[$i][0] = 1
            Else
                $aBlockerWinList[$i][0] = 0
            EndIf
        Next
        WinMove($wHandleOpera, "", Default, Default, 1024, 768)
    EndIf
    Local $aTempWinGetPos = WinGetPos($wHandleOpera)
    _MouseTrap($aTempWinGetPos[0] + Int($aTempWinGetPos[2] / 2), $aTempWinGetPos[1] + Int($aTempWinGetPos[3] / 2), $aTempWinGetPos[0] + Int($aTempWinGetPos[2] / 2), $aTempWinGetPos[1] + Int($aTempWinGetPos[3] / 2))
    If Not WinActive($wHandleOpera) Then WinActivate($wHandleOpera)
EndFunc   ;==>BlockInactiveWindowsAndMouse
Func UnDoBlockInactiveWindowsAndMouse()
    _MouseTrap()
    WinSetOnTop($wHandleOpera, "", 0)
    For $i = 1 To $aBlockerWinList[0][0]
        If $aBlockerWinList[$i][0] = 1 Then
            WinSetState($aBlockerWinList[$i][1], "", @SW_ENABLE)
        EndIf
    Next
    If Not WinActive($wHandleOpera) Then WinActivate($wHandleOpera)
    WinSetState($wHandleOpera, "", @SW_ENABLE)
EndFunc   ;==>UnDoBlockInactiveWindowsAndMouse
Edited by guestscripter

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • spikespaz
      By spikespaz
      The rule against game automation is misguided and unhelpful for the following reasons.
       
      I am a developer, using AutoIt to automate testing for a game that I am developing. Singleplayer games can be mundane and sometimes cheating can be overlooked. Multiplayer games usually have anticheat against repetitive robotic actions anyways. People want to override the control schemes of games with bad customization. Eg. Axiom Verge, Fortnite (Crouching). Game automation is not always bad. If the user feels the need to automate a singleplayer game for boring or mundane tasks, that is their choice. Similarly, if I wish to use scripts to automate testing, that is my choice.
      Game automation can be a problem for online multiplayer games, giving players a competitive advantage. This could be countered by common sense; ask OP what game they are automating, and is it an online game? What is this script for, and what does it seem to do?

      Please revise the rule as it seems very unnecessary and harmful to people seeking help with innocent attempts at game automation.
    • milkmoron
      By milkmoron
      https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/?do=findComment&comment=1156373 At least we have an element title: [Entered By:] class: [WindowsForms10.EDIT.app.0.30495d1_r9_ad1] Having the following values for all properties: Title is: <Entered By:> Class := <WindowsForms10.EDIT.app.0.30495d1_r9_ad1> controltype:= <UIA_EditControlTypeId> ,<50004> , (0000C354) 322;370;736;184 *** Parent Information top down *** 3: Title is: <View Activity> Class := <WindowsForms10.Window.8.app.0.30495d1_r9_ad1> controltype:= <UIA_WindowControlTypeId> ,<50032> , (0000C370) 306;72;784;605 "Title:=View Activity;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1"" 2: Title is: <> Class := <WindowsForms10.Window.8.app.0.30495d1_r9_ad1> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 314;103;768;542 "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1"" 1: Title is: <> Class := <WindowsForms10.Window.8.app.0.30495d1_r9_ad1> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 314;103;749;451 "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1"" 0: Title is: <Delete> Class := <WindowsForms10.Window.8.app.0.30495d1_r9_ad1> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 314;122;749;432 "Title:=Delete;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1"" ;~ *** Standard code maintainable *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) _UIA_setVar("oP1","Title:=View Activity;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1") ;View Activity _UIA_setVar("oP2","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1") ; _UIA_setVar("oP3","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1") ; _UIA_setVar("oP4","Title:=Delete;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1") ;Delete _UIA_setVar("EnteredBy:.mainwindow","title:=Entered By:;classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad1") ;~ Actions split away from logical/technical definition above can come from configfiles ;~_UIA_Action("oP1","highlight") _UIA_Action("oP1","setfocus") ;~_UIA_Action("oP2","highlight") _UIA_Action("oP2","setfocus") ;~_UIA_Action("oP3","highlight") _UIA_Action("oP3","setfocus") ;~_UIA_Action("oP4","highlight") _UIA_Action("oP4","setfocus") _UIA_action("EnteredBy:.mainwindow","setfocus") ;~ *** Standard code Flexible*** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP3=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=View Activity;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1", $treescope_children) _UIA_Action($oP3,"setfocus") Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1", $treescope_children) _UIA_Action($oP2,"setfocus") Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1", $treescope_children) _UIA_Action($oP1,"setfocus") Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Delete;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.30495d1_r9_ad1", $treescope_children) _UIA_Action($oP0,"setfocus") _UIA_setVar("EnteredBy:.mainwindow","title:=Entered By:;classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad1") _UIA_action("EnteredBy:.mainwindow","setfocus") Need some help selecting this defining it as a variable to save and then insert the text within, to the same application but different account.
    • TheSaint
      By TheSaint
      TeraCopy Timer - A program of mine, front end really, that I have been working on (on & off) for some time, but not shared here before, that I recall. Third party program TeraCopy is required.
      Many of you will be familiar with TeraCopy, and it has been discussed here at AF on several occasions, in various sub forums, including Chat.
      BE ADVISED - I still use an older version of TeraCopy (v2.27), a goody but an oldy, so have only tested my program with that. It may work with the newer v3.xx, or may be easily adjusted to do so, but I have not yet gone that route. Likewise, I only use it with Win 7 and Win XP, but I don't imagine there should be any issues with newer Windows. Add to that, my program being devised and created in AutoIt v3.3.0.0.
      So what is TeraCopy Timer and why might you need it?
      --------------------------------------------------------------------------------------
      Those familiar with TeraCopy use, know you can run multiple copy or move jobs either simultaneously or concurrently (one immediately after another has finished). With the free version at least, you have no control over precise order or timing, and no way to deal (without manual interaction) with the pesky Thumbs.db files in Win XP. TeraCopy Timer allows you to do all those things, and in a batch fashion.
      --------------------------------------------------------------------------------------
      Some screenshots and brief explanation
      Main Window
      Minimalist Window
      This new WAIT feature is the simplest and best way to use the program generally (in my view) ... but check out the following, as it is not always the case.
      Advanced Delay Window
      WARNING - It should go without saying, that you take all the usual precautions. For instance, don't use move with the only copy you have of precious files. BACKUP BACKUP BACKUP! Don't hold me liable for anything, except for being a nice guy who likes to share.
      TeraCopy Timer v2.8.zip  (see detail at Post #34)
      TeraCopy Timer v2.9.zip  (see detail at Post #35)
      Enjoy!
      OLDER DOWNLOADS
      P.S. I am not affiliated in any way with those who created and provide the excellent third party program - TeraCopy.
    • Earthshine
      By Earthshine
      This is pretty useful to me at least, I hope to others. Whenever you need to poke a button or such on a GUI, you can just make calls to this UDF and instruct it how long to wait, 0 is forever, anyway, let the code to the explaining. Feel free to help expand this UDF. I plan to create new ones as I go along that support all the Win32 and .NET Controls. Ultimately, I would like to be able to fully control any type of control, this works well with most. Just an example of what you can do.
       
      The test installs and can uninstall 7-Zip 15.14 (x64) Setup.exe
      NOTE: Logger Author(s) .....: Michael Mims (zorphnog)
      ;                    the logging script can be obtained here
      ;                   https://www.autoitscript.com/forum/topic/156196-log4a-a-logging-udf/
       
       
       
       
      WaitForControls.au3
      install_7zip_test.au3
    • rcmaehl
      By rcmaehl
      A UDF to connect and control Cisco Finesse on the Local Network via it's provided HTTP API
       
      Current Features:
      Heavyweight
      * Large feature set, containing most Finesse Desktop API functions  
      PLANNED Features:
      Heavyweight
      * Control over own Status as well as Supervisor functionality
        Helpful Premade Examples Files
      * Examples for almost every function
      * Premade CUI and GUI clients showing 
        Ready for all users, beginner and up
      * Detailed Error Reporting to help pin-point problems
      * Thorough parameter checking to prevent accidental script crashes
×