junkew Posted September 9, 2019 Author Posted September 9, 2019 I have no time to help much at the moment. You are having 0 for the rect. So not finding right ac. You find by role and not by title and role. Be aware this is a work in progress and many issues can arise on testing this stuff. FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
junkew Posted September 9, 2019 Author Posted September 9, 2019 (edited) https://github.com/openjdk/jdk/tree/master/src/jdk.accessibility/windows/native/jaccessinspector gives an indication that at least __releaseJavaObject should be called to release memory allocated. More CPP source on how JAB is made https://github.com/openjdk/jdk/blob/master/src/jdk.accessibility/windows/native/libwindowsaccessbridge/WinAccessBridge.cpp and apparently I tried also in 2013 some stuff where most important part can be that certain messages have to be handled when sent from JAB to AutoIt client. jaccesswalker from jdk10 and jdk11 seems to be most stable tool to spy not crashing whereas jaccessinspector just quits when i try to spy on jdk8 javacpl.exe. accessibility explorer is also crashing as soon as I spy on the modal dialog. As jaccesswalker is working fine I only can say this can work but maybe issues due to * Logic coding error * separate thread needed * .... Edited September 9, 2019 by junkew FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
junkew Posted September 9, 2019 Author Posted September 9, 2019 (edited) Demo script updated and fully working (see post 1) _JAB_SingleAction seems to hang on sitelistbutton, bypassed by calculating mouse location based on acinfo x,y,h,w ;~ TODO: _JAB_singleAction($__g_vmId, $SiteListButton) clickac($__g_vmId, $SiteListButton) button add in popup works Done: Edit part Edited September 10, 2019 by junkew FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
fablecao Posted September 10, 2019 Posted September 10, 2019 On 9/9/2019 at 11:11 AM, KhalidAnsari said: @junkew I got it working. I just debug it. Checked my dll reference. Now i m tried on my jar file to click on button. I am getting following response in console. shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar") sleep(3000) global $wintitle = "Login for Chat" $winHandle = WinActivate($wintitle) $result = __isJavaWindow($winHandle) Global $vmId Global $ac global $sName = "Send" global $sRole = "push button" __getAccessibleContextFromHWND($winHandle, $vmID, $ac) $re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1) _JAB_singleAction($vmId, $re_ac) Java Home: C:\Program Files\Java\jre1.8.0_221 We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0 PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1 0 Windows_run returns: Name: rect:0000 Name: rect:0000 _JAB_singleAction: Only applicable for single action element like button or check box, menu, menu item Expand _JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1) This function is not my original, you find wrong $re_ac which cause _JAB_singleAction error. You can use _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)
junkew Posted September 10, 2019 Author Posted September 10, 2019 (edited) Ok will check with the original version. added function _JAB_getAccessibleChildNByRole usage: get an element by its index and class/role ;~ local $re_ac=_JAB_getAccessibleContextByRole($__g_vmId, $__g_acJavaMainWindow, "push button", 1) local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",1) local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",2) local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",3) function expandcollapse popupFunc _JAB_getAccessibleChildNByRole($vmId, $ac, $sRole,$index=1, $depth=0) Static $matchCount=0 if ($depth=0) then $matchCount=0 EndIf Local $iCount = _JAB_getChildrenCount($vmId, $ac) ;~ consolewrite("Investigating" & $iCount & ";" & $index & ";" & $matchCount & ";" & _JAB_getRole($vmId, $ac) & ";") If $iCount = 0 Then Return -1 EndIf local $retVal=-1 For $i = 0 To $iCount - 1 Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i) Local $child_ac_s3 = _JAB_getRole($vmId, $child_ac) ;~ consolewrite($child_ac_s3 & @CRLF) If $child_ac_s3 = $sRole Then $matchCount+=1 ;~ consolewrite("Partial match" & $matchCount & @CRLF) if $index=$matchCount Then ;~ consolewrite("Full match" & $matchCount & @CRLF) Return $child_ac endif EndIf $depth+=1 local $retVal=_JAB_getAccessibleChildNByRole($vmId, $child_ac, $sRole, $index, $depth) if ($retVal <>-1) Then exitloop EndIf Next Return $retVal EndFunc Edited September 10, 2019 by junkew function changed/added FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
junkew Posted September 10, 2019 Author Posted September 10, 2019 @fablecao I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function func clickSiteListButton() $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button") highLightAC($__g_vmId, $SiteListButton) printdescription($SiteListButton) sleep(2500) ;~ TODO: Somewhere here it hangs ;~ consolewrite("Just before it causes a blocking ..." & @CRLF) ;~ _JAB_singleAction($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ..." & @CRLF) clickac($__g_vmId, $SiteListButton) sleep(1500) __releaseJavaObject($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ... 2" & @CRLF) EndFunc Output from printDescription($siteListButton) Name:Edit Site List... rect:10681209562585 name: <Edit Site List...> description: <Add, Remove, or Edit entries on the Exception Site List> role: <push button> role_en_US: <push button> states: <enabled,focusable,visible,showing,opaque> states_en_US: <enabled,focusable,visible,showing,opaque> indexInParent: <0> childrenCount: <0> x: <1068> y: <562> width: <141> height: <23> accessibleComponent: <1> accessibleAction: <1> name: <0> description: <0> role: <0> role_en_US: <0> states: <0> states_en_US: <0> indexInParent: <0> childrenCount: <0> x: <0> y: <0> width: <0> height: <0> accessibleComponent: <0> accessibleAction: <0> FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
fablecao Posted September 11, 2019 Posted September 11, 2019 On 9/10/2019 at 9:23 PM, junkew said: @fablecao I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function func clickSiteListButton() $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button") highLightAC($__g_vmId, $SiteListButton) printdescription($SiteListButton) sleep(2500) ;~ TODO: Somewhere here it hangs ;~ consolewrite("Just before it causes a blocking ..." & @CRLF) ;~ _JAB_singleAction($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ..." & @CRLF) clickac($__g_vmId, $SiteListButton) sleep(1500) __releaseJavaObject($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ... 2" & @CRLF) EndFunc Output from printDescription($siteListButton) Name:Edit Site List... rect:10681209562585 name: <Edit Site List...> description: <Add, Remove, or Edit entries on the Exception Site List> role: <push button> role_en_US: <push button> states: <enabled,focusable,visible,showing,opaque> states_en_US: <enabled,focusable,visible,showing,opaque> indexInParent: <0> childrenCount: <0> x: <1068> y: <562> width: <141> height: <23> accessibleComponent: <1> accessibleAction: <1> name: <0> description: <0> role: <0> role_en_US: <0> states: <0> states_en_US: <0> indexInParent: <0> childrenCount: <0> x: <0> y: <0> width: <0> height: <0> accessibleComponent: <0> accessibleAction: <0> Expand I don't know what behaves weird when you use _JAB_SingleAction on the sitelist button. I test it successfully in my computer. And from your output of printDescription($siteListButton), I find that the output of printdescription($siteListButton) inside highlightAC is different from the next output of printdescription($siteListButton). Does the func clickSiteButton function normal? Does the func __releaseJavaObject inside highlightAC release the AC?
fablecao Posted September 11, 2019 Posted September 11, 2019 By the way, I prefer to use the combination of JAB native function to write UDF. The JAB native function __requestFocus may implement your function highlightAC. ;~ BOOL requestFocus(const long vmID, const AccessibleContext accessibleContext) ; Request focus for a component. Returns whether successful. Func __requestFocus($vmId, $ac) $result = DllCall($hAccessBridgeDll, "bool:cdecl", "requestFocus", "long", $vmId, $c_JOBJECT64, $ac) If @error Then Return SetError(1, 0, 0) Return $result[0] EndFunc
junkew Posted September 11, 2019 Author Posted September 11, 2019 (edited) I agree on preferring native functions over alternatives like mousemove and click . the problem I observe Function clicks on the button Popup dialogwindow appears Script hangs When I use this below function so action is executed but then script is hanging somewhere func clickSiteListButton() $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button") highLightAC($__g_vmId, $SiteListButton) printdescription($SiteListButton) sleep(2500) ;~ TODO: Somewhere here it hangs consolewrite("Just before it causes a blocking ..." & @CRLF) _JAB_singleAction($__g_vmId, $SiteListButton) consolewrite("Just after it causes a blocking ..." & @CRLF) ;~ clickac($__g_vmId, $SiteListButton) sleep(1500) __releaseJavaObject($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ... 2" & @CRLF) EndFunc Edited September 11, 2019 by junkew FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
mLipok Posted September 11, 2019 Posted September 11, 2019 On 9/11/2019 at 5:06 AM, junkew said: When I use this below function so action is executed but then script is hanging somewhere Expand Maybe the same problem like in IE.au3 UDF , I mean when you click in "OpenFile Button". IE object is waiting for end of selecting..... and for this reason, AutoIt is waiting for IE...... Signature beginning:* Please remember: "AutoIt"..... * Wondering who uses AutoIt and what it can be used for ? * Forum Rules ** ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Code * for other useful stuff click the following button: Reveal hidden contents Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API * ErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 * My contribution to others projects or UDF based on others projects: * _sql.au3 UDF * POP3.au3 UDF * RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF * SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane * Useful links: * Forum Rules * Forum etiquette * Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * Wiki: * Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX IE Related: * How to use IE.au3 UDF with AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler * IE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related: * How to get reference to PDF object embeded in IE * IE on Windows 11 * I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions * EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *I also encourage you to check awesome @trancexx code: * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuff * OnHungApp handler * Avoid "AutoIt Error" message box in unknown errors * HTML editor * winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/ "Homo sum; humani nil a me alienum puto" - Publius Terentius Afer"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming" , be and \\//_. Anticipating Errors : "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty." Signature last update: 2023-04-24
KhalidAnsari Posted September 11, 2019 Posted September 11, 2019 @junkew Thanks for reply. Following code is closing my Java screen unexpectedly. I am trying to click on "Send button" but $re_ac returning -1 from method _JAB_getAccessibleChildNByRole #include "Javaui.au3" shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar") sleep(5000) global $wintitle = "Login for Chat" $winHandle = WinActivate($wintitle) $result = __isJavaWindow($winHandle) Global $vmId Global $ac __getAccessibleContextFromHWND($winHandle, $vmID, $ac) ConsoleWrite("$vmID : " & $vmID & @CRLF) ConsoleWrite("$ac : " & $ac & @CRLF) Sleep(3000) local $re_ac =_JAB_getAccessibleChildNByRole($vmId, $ac, "push button",1) ConsoleWrite("$re_ac : " & $re_ac & @CRLF) OutPut Java Home: C:\Program Files\Java\jre1.8.0_221 We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0 C:\Program Files\Java\jre1.8.0_221\bin\WindowsAccessBridge-32.dll PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1 0 Windows_run returns: $vmID : 1247780 $ac : 1627349361864 Investigating1;1;0;frame; -- Not getting out from _JAB_getAccessibleChildNByRole Investigating0;1;0;; -- Not getting out from _JAB_getAccessibleChildNByRole $re_ac : -1
fablecao Posted September 11, 2019 Posted September 11, 2019 On 9/11/2019 at 5:06 AM, junkew said: I agree on preferring native functions over alternatives like mousemove and click . the problem I observe Function clicks on the button Popup dialogwindow appears Script hangs When I use this below function so action is executed but then script is hanging somewhere func clickSiteListButton() $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button") highLightAC($__g_vmId, $SiteListButton) printdescription($SiteListButton) sleep(2500) ;~ TODO: Somewhere here it hangs consolewrite("Just before it causes a blocking ..." & @CRLF) _JAB_singleAction($__g_vmId, $SiteListButton) consolewrite("Just after it causes a blocking ..." & @CRLF) ;~ clickac($__g_vmId, $SiteListButton) sleep(1500) __releaseJavaObject($__g_vmId, $SiteListButton) ;~ consolewrite("Just after it causes a blocking ... 2" & @CRLF) EndFunc Expand Oh, maybe there is something wrong in my JAB native function. "NONE" should be changed to "NONE:cdecl". All of which should be corrected. Func __releaseJavaObject($vmId, $object) $result = DllCall($hAccessBridgeDll, "NONE", "releaseJavaObject", "long", $vmId, $c_JOBJECT64, $object) If @error Then Return SetError(1, 0, 0) Return $result[0] EndFunc
fablecao Posted September 16, 2019 Posted September 16, 2019 On 11/23/2018 at 7:08 PM, junkew said: Just checked and still works nicely in 64 bits windows. >"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\x\Documents\UIA\JABSimpleSpy.au3" /UserParams +>19:58:07 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0 Keyboard:00020409 OS:WIN_81/ CPU:X64 OS:X64 Environment(Language:0413) +> SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE UserDir => C:\Users\x\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper SCITE_USERHOME => C:\Users\x\AppData\Local\AutoIt v3\SciTE >Running AU3Check (3.3.14.5) from:C:\Program Files (x86)\AutoIt3 input:C:\Users\x\Documents\UIA\JABSimpleSpy.au3 +>19:58:08 AU3Check ended.rc:0 >Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Users\x\Documents\UIA\JABSimpleSpy.au3" --> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop C:\Program Files\Java\jre1.8.0_181 We are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=1 Opening C:\Program Files\Java\jre1.8.0_181\bin\WindowsAccessBridge-64.dll Windows accessbridge 32 opened 1 0 initializeAccessBridge is finishedWindows_run passed : 0hello name: <Settings...> description: <<html>Modify settings for temporary files</html>> role: <push button> role_en_US: <push button> states: <enabled,focusable,visible,showing,opaque> states_en_US: <enabled,focusable,visible,showing,opaque> indexInParent: <0> childrenCount: <0> x: <1028> y: <447> width: <83> height: <0> +>19:58:30 AutoIt3.exe ended.rc:0 +>19:58:30 AutoIt3Wrapper Finished. Initially I learned a lot from NVDA to use this stuff and found another nice access-bridge-explorer that can be used for porting things from JAB into AU3 https://github.com/nvaccess/nvda/blob/master/source/JABHandler.py https://github.com/google/access-bridge-explorer Expand How to use autoit3_x64.exe in SciTE?
fablecao Posted September 17, 2019 Posted September 17, 2019 On 9/16/2019 at 1:56 PM, fablecao said: #AutoIt3Wrapper_UseX64 = y Expand Oh, my God. It's the bug of SciTE-Lite, this setting takes no effect in SciTE-Lite.
fablecao Posted September 17, 2019 Posted September 17, 2019 On 9/16/2019 at 1:56 PM, fablecao said: #AutoIt3Wrapper_UseX64 = y Expand It's a pity that SciTE-Lite doesn't support autoit3wrapper.keywords.properties and other properties.
Mich90 Posted June 12, 2020 Posted June 12, 2020 I finally got a simple Java automatisation working 🙂 Inow have got the problem that I need to select the first row of a table. Is this possible? Thanks for any hint!
junkew Posted June 13, 2020 Author Posted June 13, 2020 It depends. Just chech what different tools highlight for you when you spy. It basically starts with below function. getAccessibleTableInfo FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
duzers Posted September 4, 2020 Posted September 4, 2020 Hello, It is possible to click Page tab e.g. "Java" or radio button in Security tab in Java Control Panel.
junkew Posted September 6, 2020 Author Posted September 6, 2020 Yes it is you can see jabsimplespy highlights it. You have to study java accessibility code examples to understand the concept of what and how you can retrieve the accessible objects. FAQ 31 How to click some elements, FAQ 40 Test automation with AutoIt, Multithreading CLR .NET Powershell CMDLets
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now