Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


junkew
 Share

Recommended Posts

Differs due to many different factors like platform version. As inspect.exe is the main ms tool that's the best default tool to check with. And then you also have different tools using different ways of identifying. The uia is a good base from Microsoft but commercial tools like ranorex or testcomplete can give different insights and some helpfull tools look with ocr or ai imaging basically a smart way of bitmap analyzing.

Link to comment
Share on other sites

  • 2 weeks later...

Hi,

Okay thats really useful. Sooo after trying everything above i still have the issue.  Inspect.exe does show me different trees based on the different OS versions. Not sure what to try next since it seems on windows server more elements get found(I dont want to use any other libs e.g ranorex or img search). I tried swapping out OLEACC.dll and UIAutomationCore.dll (UIA docs references these as core to the functionality of the lib) in system 32 to use the windows server dll's as a last ditch attempt. But this only broke chrome in the way that no tabs load anymore and u get the aw,snap message everytime 😂.

Link to comment
Share on other sites

Actually i did find the item in the tree, the tree as mentioned is slightly different. This is the major difference 

Provider Description: MSAA Proxy (IAccessible2) (unmanaged:UIAutomationCore.DLL)]" - Windows Pro (Not working)

Provider Description: MSAA Proxy (unmanaged:UIAutomationCore.DLL)]" - Windows server (Working)

Can i get rid of IAccessible2 somehow haha

Link to comment
Share on other sites

  • 4 months later...

Hi @junkew ,

In version V0-70, has anyone questioned the output of ex1_Treewalker? Is it common for the clipping of the first letter or more of the Window in the console output?  ie  <askbar>, <eamViewer Panel>, <ite [6 of 6]> 

Would you recommend I make any code adjustments to accommodate for this?

Thanks!

 

**** Deal with browser windows ****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>

**** Continue with finding 2 browser windows ****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>
Yes found first browser 

**** Deal with 2nd browser window****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>
location: 121<Microsoft? Edge><IUIAutomation MS framework automate chrome, FF, IE, .... - Page 45 - AutoIt Example Scripts - AutoIt Forums - Profile 1 - Microsoft? Edge><Chrome_WidgetWin_1><001201FC>
Yes found second browser 
>Exit code: 0

 

Link to comment
Share on other sites

  • 1 month later...

I'm using UIAWrapper.au3 and sometimes I get this error:

"C:\AUTOMATY\UIAWrappers.au3" (1674) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR
->14:05:04 AutoIt3.exe ended.rc:1

The biggest problem is catching the error, the script runs ~10 hours every day.
The function is called dozens of times a day.
The error occurs 2 a month, but it is irritating because it interrupts the script.

Excerpt from UIAWrappers.au3

1668        Local $t
1669        $t = StringSplit(_UIA_getPropertyValue($obj2ActOn, $UIA_BoundingRectanglePropertyId), ";")
1670    ;~  If no parameter detect the middle
1671        If $p1 = 0 Then
1672    ;~  consolewrite(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId) & $t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)
1673    ;~  _winapi_mouse_event($MOUSEEVENTF_ABSOLUTE + $MOUSEEVENTF_MOVE,$t[1],$t[2])
1674            $x = Int($t[1] + ($t[3] / 2))
1675            $y = Int($t[2] + $t[4] / 2)
1676        Else
1677    ;~      calculate location relative in the element found
1678            $x = Int($t[1] + $p1)
1679            $y = Int($t[2] + $p2)
1680        EndIf

Maybe it's worth adding a verification of the $t variable on line 1670 before further operations on the $t array?

I added an error trap - now I'm waiting for it to appear and check what goes to $t.

Link to comment
Share on other sites

  • 1 month later...

Recently I was working on simplifiaction in usage with UIA


And thus I have almostly completed new UDF to simplify UIA usage.

 

Example:

Func _Example()

    Local $hWND_UIAFound = 0
    _log(@ScriptLineNumber & ' ' & WinExists($_sExampleGuiTitle, "")

    _UIASimple_WinExist($_sExampleGuiTitle, "", "Some text to find", $hWND_UIAFound)

    _UIASimple_ButtonClick($hWND_UIAFound, "", "Some text to find", "Orange")

    _UIASimple_ButtonClick($hWND_UIAFound, "", "", "&Fake button")

    _UIASimple_ButtonClick($_sExampleGuiTitle, "OK", "Some text to find", "Banana")

EndFunc   ;==>_Example

it uses my UIASimple.au3 which I will publish when it will be finished.

And part of UIASimple.au3 UDF:

....
....
....
Global $_UIASimple_TEXT_MATCH_MODE = $PropertyConditionFlags_IgnoreCase ; https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/ne-uiautomationclient-propertyconditionflags
....
....
....
#Region - _UIASimple_*** Helpers
Func _UIASimple_ButtonClick($sWinTitle, $sWinText, $sUIAText, $sButton)
    If Not WinExists($sWinTitle, $sWinText) Then Return SetError(1, 0, False)

    Local $hWND_UIAFound = 0 ; Reset
    Local $oWindow = _UIASimple_FindWindow($sWinTitle, $sWinText, $sUIAText, $hWND_UIAFound)
    If @error Then Return SetError(2, @extended, False)

    Local $oButton = _UIASimple_FindSubObject_byName($oWindow, $sButton)
    If @error Then Return SetError(3, @extended, False)

    _UIASimple_ObjectInvoke($oButton)
    If @error Then Return SetError(4, @extended, False)

    Return True
EndFunc   ;==>_UIASimple_ButtonClick
....
....
....

 

My general rule was to map the Standard AutoIt functions to give possibility to do things in a similar way but with UIA, so that it would be easy to switch to UIA (when it is needed).

But currently I have one hm.... dilemma.

As so far I used: UIA_Constants.au3
I also saw CUIAutomation2.au3
both files have defined the same values in a different convention.
For example:

$dtag_IUIAutomationInvokePattern
$dtagIUIAutomationInvokePattern

Some other const are defined with the same names so I can't use both together.

So my dilema is:
Which file should I use and why ? CUIAutomation2.au3 or UIA_Constants.au3 ?

 

Edited by mLipok

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Codefor other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 *

 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2023-04-24

Link to comment
Share on other sites

  • 1 month later...

I'm using UIA_V0_70.zip and UIASpy to read controls from an external program.

Program has 4 ComboBoxes. They all have almost identical properties: no title, same class type, same ctrl type, same ctrl name, no handle:

Title     = 
Class     = ComboBox
Ctrl type = 50003
Ctrl name = combo box
Selected  = False
Handle    = 00000000

The only difference I can find, using UIASpy, is the RuntimeId (something like 7;18760;23387474, 7;18760;15337269, 7;18760;12405375, 7;18760;44539515) and Parent to child index (7, 8, ....).

How do I use those to click different comboboxes?

I tried this and it did'nt work:

_UIA_Action('title:=;classname:=ComboBox;RuntimeId:='&$combo000[0], "click")   ; $combo000 = array of all 4 different RuntimeIds

If I use this, it clicks the first ComboBox just fine:

_UIA_Action("title:=;classname:=ComboBox;ControlType:=50003","click")

If I use UIASpy > right click each ComboBox > Show UI Automation element = it works perfectly, each different ComboBox is highlighted.

But I don't understant what $iIdx is, in this context (first ComboBox is 20, for instance).

Thank you.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...