Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


Recommended Posts

Regarding the variable definitions with _UIA_setvar like simplespy is given those can be put in UIA.CFG like this

Example for notepad in 2 languages english/dutch

closewindow.Don'tSave=title:=((Don't Save)|(Niet opslaan));classname:=CCPushButton

Above instead of 

;~ Give logical names

;~ When closing when text is changed
;~ The three buttons
_UIA_setVar("Notepad.closewindow.Don'tSave","title:=((Don't Save)|(Niet opslaan));classname:=CCPushButton")

you just put the definitions in the separate configuration uia.cfg file so you can split your page definitions away from your coding.

Example 31 for notepad will then change to something like below

#include <Array.au3>
#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

; Run Notepad

; Wait 10 seconds for the Notepad window to appear.
WinWaitActive("[CLASS:Notepad]", "", 10)

;~ see UIA.CFG section notepad

;~ Do some actions on the logical named objects

_UIA_action("notepad.mainwindow.edit","setvalue","set value: hello world")
_UIA_action("notepad.mainwindow.edit","type","type command: hello world")

if _UIA_action("Notepad.closewindow","exists") Then


Link to comment
Share on other sites

  • 1 month later...

My problem is the same objects . 

I have an intranet page where I have to click on the next links in the table.
The identification of the link objects is not different.

UIA always finds the first object in which to indicate the next link?

The table that I serve has 200 rows. I can not put screenshots (confidential data).

To show what the problem I have created, the test.html and Simple UIA Spy code



How do you click View in the Piotr line?


Global $WindowChromePID
$sLink = @ScriptDir & "\test.html"

If FileExists("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe") Then
    $WindowChromePID = Run("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe --force-renderer-accessibility " & $sLink)
ElseIf FileExists("C:\Program Files\Google\Chrome\Application\chrome.exe") Then
    $WindowChromePID = Run("C:\Program Files\Google\Chrome\Application\chrome.exe --force-renderer-accessibility " & $sLink)
    MsgBox(8192+16, "Error","Chrome.exe not found")

_UIA_setVar("oP1","Title:=test.html - Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1")  ;test.html - Google Chrome
_UIA_setVar("oP2","Title:=;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")  ;
_UIA_setVar("oP3","Title:=;controltype:=UIA_TableControlTypeId;class:=")    ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=View;controltype:=UIA_DataItemControlTypeId;class:=") ;View
_UIA_setVar("oP6","Title:=View;controltype:=UIA_HyperlinkControlTypeId;class:=")    ;View

;~ $oUIElement=_UIA_getObjectByFindAll("View.mainwindow", "title:=View;ControlType:=UIA_TextControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=View;controltype:=UIA_TextControlTypeId;class:=") ;ControlType:=UIA_TextControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

;~ _UIA_Action("oP1","setfocus")
;~ _UIA_Action("oP2","setfocus")
;~ _UIA_Action("oP3","setfocus")
;~ _UIA_Action("oP4","setfocus")
;~ _UIA_Action("oP5","setfocus")
;~ _UIA_Action("oP6","setfocus")

;~ _UIA_action("oUIElement","highlight")

Screen from MS Inspect


Link to comment
Share on other sites

  • 1 month later...

I'm having a bit of trouble getting the library to NOT create log xml files in the LOG directory.  I want the script to run daily, so I don't need log files every day.

I have added this line to the code:

; Turn off debug log file generation - Doesn't seem to work either.


As well as setting these two in UIA.CFG:



But I keep getting XML files in the LOG directory named:  20180729-130940736.XML

With this content:

<?xml version="1.0" encoding="UTF-8"?><log space="preserve"><logline level="5" timestamp="20190729-130940737"> <information> Information _UIA_VersionInfo version: T0.7-0; Release date: 20190501; OS Version: WIN_7</information></logline>



Link to comment
Share on other sites

I'm working on a simple login script where I create a GUI and call a function that opens chrome, finds an element types a username goes to the next element and types the password. when I run the function again I get the below error so I am guessing the array needs to be cleared out before I run the function again? I basically used simplespy to generate the code. It works perfectly when I just re-run the function in Scite, but when I include it into my GUI I get the below error on the second run, first run is fine.

here is the function only

Func website()

;~ Start chrome

If processexists($strChromeExeOnly) Then
    sleep (1000)
    run($strChromeExe & $strChromeStartup & " " &"--user-data-dir=C:\AutoLogontestCrome\userData" & " " & "--disable-single-click-autofill" & " " & $MedtronicURL,"", @SW_MAXIMIZE )
    ;~ Just to give some time to start

    run($strChromeExe & $strChromeStartup & " " &"--user-data-dir=C:\AutoLogontestCrome\userData" & " " & "--disable-single-click-autofill" & " " & $MedtronicURL,"", @SW_MAXIMIZE )
    ;~ Just to give some time to start

local $oChrome=_UIA_getFirstObjectOfElement($UIA_oDesktop,"class:=Chrome_WidgetWin_1", $treescope_children)

AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=website - Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1")    ;website Network - Google Chrome
_UIA_setVar("oP2","Title:=Medtronic CareLink Network;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")    ;website
_UIA_setVar("oP3","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP6","Title:=;controltype:=UIA_TableControlTypeId;class:=")    ;
_UIA_setVar("oP7","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP8","Title:=CareLink® Network • Sign In Please enter your User ID and password to enter the Medtronic CareLink Clinician Website. User ID: Password: Sign In Patient Information Site >> Forgot Password?;controltype:=UIA_DataItemControlTypeId;class:=") ;website • Sign In Please enter your User ID and password to enter the Website. User ID: Password: Sign In bla bla >> Forgot Password?
_UIA_setVar("oP9","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP10","Title:=;controltype:=UIA_TableControlTypeId;class:=")   ;
_UIA_setVar("oP11","Title:=;controltype:=UIA_CustomControlTypeId;class:=")  ;

;~ $oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_DataItemControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=;controltype:=UIA_DataItemControlTypeId;class:=") ;ControlType:=UIA_DataItemControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

;GUICtrlSetState($Button2, $GUI_ENABLE)



>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\AutoLogontestCrome\login_GUI_v2.au3" /UserParams    
+>19:35:18 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.   Keyboard:00000409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\user1\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user1\AppData\Local\AutoIt v3\SciTE
>Running AU3Check (  from:C:\Program Files (x86)\AutoIt3  input:C:\AutoLogontestCrome\login_GUI_v2.au3
+>19:35:19 AU3Check ended.rc:0
>Running:(\Program Files (x86)\AutoIt3\autoit3.exe "C:\AutoLogontestCrome\login_GUI_v2.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"C:\Program Files (x86)\AutoIt3\Include\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR

->19:36:11 AutoIt3.exe ended.rc:1
+>19:36:11 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 53.91

Link to comment
Share on other sites

You are right I basically cache things in the array by prefixing it with rti. For runtime type information whenever the object is found it reuses that info. As I am on holiday without laptop hard to say in which example I remove items from array that start with rti.

Link to comment
Share on other sites

Did anybody recently try to run the Firefox automation example? After doing some modifications, I have been able to access tree elements like tabs and click to them, so basic things work. But I can't access the document itself, I always get ""Action 6 failed". Any Ideas?

Thanks for any help.


Link to comment
Share on other sites

On 8/8/2019 at 7:30 PM, junkew said:

Try if simplespy highlights your document area and or elements. Most likely during upgrades they change element order and as such your document identification or element tree changed. Which ff version are you running.

Today, I had the chance to do some more testing with simplespy.au3:

  • I used machine #1 with 64bit Windows 7 build 7601 and almost current Firefox 68.0 64bit
  • I ran the same tests on machine #2 with 32bit Windows 7 build 7601 and current Firefox 68.0.1 32bit
  • Same Tests were made on another machine #3 with 64bit Windows 10 1803 and current Firefox 68.0.1 32bit

More details about my environment:

  • All machines have German versions of Windows and Firefox.
  • On the 64bit machines #1 and #3, I executed simplespy.au3 in both modes x86 and x64, using "Run Script (x64)" and "Run Script (x86)"
  • During the tests, Firefox is already started when I run simplespy.au3
  • I have to confirm an UAC Dialog, before simplespy.au3 starts
  • I have autoit version installed on all machines

How I did the tests:

  • I can place my mouse over almost any Element of Firefox (tried toolbar, address bar, favourite bookmarks), press Ctrl+W and get information about that element in the simplespy window on any machine.
  • If i place my mouse anywhere in the content area of the browser (where the web content is displayed) and press Ctrl+W there, simpplespy.au3 crashes on machines #1 and #2 (with the additional information as shown below). On machine #3, everything works as expected.


  • Access to content area of firefox doesn't seem to be possible with windows 7, but works fine with windows 10

Do you have any idea what's going wrong with Windows 7?


Appendix - Additional Error Information

Machine #1, 32bit mode and Machine #2, 32bit mode

  Problemereignisname:    APPCRASH
  Anwendungsname:    AutoIt3.exe
  Anwendungszeitstempel:    5aaa71bf (different each time, of course)
  Fehlermodulname:    uiautomationcore.dll
  Fehlermodulzeitstempel:    56f58458 (different each time, of course)
  Ausnahmecode:    c0000005
  Ausnahmeoffset:    00053671
  Betriebsystemversion:    6.1.7601.
  Gebietsschema-ID:    1031
  Zusatzinformation 1:    0a9e
  Zusatzinformation 2:    0a9e372d3b4ad19135b953a78882e789
  Zusatzinformation 3:    0a9e
  Zusatzinformation 4:    0a9e372d3b4ad19135b953a78882e789

Machine #1, 64bit mode

  Problemereignisname:    APPCRASH
  Anwendungsname:    AutoIt3_x64.exe
  Anwendungszeitstempel:    5aaa71d3
  Fehlermodulname:    uiautomationcore.dll
  Fehlermodulzeitstempel:    56f58b9a
  Ausnahmecode:    c0000005
  Ausnahmeoffset:    000000000005bffe
  Betriebsystemversion:    6.1.7601.
  Gebietsschema-ID:    1031
  Zusatzinformation 1:    a04c
  Zusatzinformation 2:    a04c4c7ad5cbb615489808b507a01638
  Zusatzinformation 3:    c91a
  Zusatzinformation 4:    c91af45283a621aa3e02406f3227c8f3



Link to comment
Share on other sites

1 hour ago, Heinzl said:

Access to content area of firefox doesn't seem to be possible with windows 7

Browse to about:config, confirm the risk, find all parameters browser.tabs.remote.autostart, browser.tabs.remote.autostart.1, browser.tabs.remote.autostart.2 etc, turn them all to false and restart Firefox. Now you will see only one process Firefox.exe but IUIAutomation should work.

Oops... this worked on version 67 but does not work on 68. Sorry.

Edited by InnI
Link to comment
Share on other sites

Hard to say but I feel its a firefox issue.


You can doublecheck with inspect.exe whats happening and if you can reproduce report a bug with mozilla firefox and/or feature request.

I am on W10 nowadays and not working iwth W7 machines so hard to help in anlalysing.

validate these links on FF. I have been reading this and see litlle info on UIA but they support MSAA (and UIA puts a layer if needed overMSAA)

about:config in addressbar and then type accessibility maybe you can tweak some settings



Link to comment
Share on other sites

Thank you junkew and InnI for your assistance!

I won't spend any more time on Windows 7 and will continue work with Windows 10.

Now that I am able to find the content object of type UIA_DocumentControlTypeId, I have another question:

Is there a way to "convert" this to something that is more similar to a DomTree (like ISimpleDom)? Or, in other words: How do I acces porperties like innerHTML or innerText?

Thank you for any assistance

Link to comment
Share on other sites

This is an example on how to get whole DOM thru javascript and addressbar in chrome/ff/edge/IE.

  • Its easier in general to test this information thru IE (when its correct there chances are low its different in other browser)
  • I use UIA mainly for "black box" testing and then you are less interested in the html code itself but as you can see you can get to the information you want.
  • In 99% of the cases I just alert what I need to a popup/message alertbox and read that with "standard" functions.
  • Below example makes an additional textarea where I can put anything I want with multiple javascript miniscripts I execute thru addressbar

This textarea I then read with _UIA_action( <ID of textarea>,"getvalue). So after below script is finished you should see at the bottom of your html page a huge textarea( normally I make them 1x1 pixel)

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oChromeWindow",".*Google Chrome;controltype:=UIA_PaneControlTypeId;class:=Chrome_WidgetWin_1") ;AutoIt General Help and Support - AutoIt Forums - Google Chrome
_UIA_setVar("oAdressbar","Title:=Adres") ;ControlType:=UIA_ButtonControlTypeId;classname:=")


;~ _UIA_action("oAdressbar","highlight")
;~ _UIA_action("oAdressbar","setfocus")

local $jsSource="javascript:"
$jsSource&="var UIAel=document.createElement(""textarea"");"

;~ Alternative
;~ $jsSource="javascript:alert(document.body.innerHTML;"
;~ $jsSource&="void(0);"


more advanced

never needed it and only made a demo once with websockets


with javascript approach you can start a websocket in the page and communicate with it from AutoIT


Edited by junkew
added websocket info
Link to comment
Share on other sites

Thank you for your sample code. Unfortunately, javascript is not an option in my case. So I try to slowly approach the ISimpleDomNode. According to https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Accessibility/AT-APIs/ImplementationFeatures/MSAA#ISimpleDOMNode this should be possible in Firefox. In your sample file ex30_ISimpleDom.au3, I found that you are doing quite the same thing for chrome. So I tried to create a very simple test, which looks like this:

; Test for Firefox Access

#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os

; find Firefox Window
ConsoleWrite("Find Firefox ... ")
Local $oFF = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=MozillaWindowClass", $treescope_children)
If IsObj($oFF) Then
    ConsoleWrite("found" & @CRLF)
    _UIA_DumpThemAll($UIA_oDesktop, $treescope_children)

; find Google TAB
ConsoleWrite("Find Document ... ")
Local $oDoc = _UIA_getObjectByFindAll($oFF, "Title:=.*Google Search;controltype:=UIA_DocumentControlTypeId", $treescope_subtree)
If IsObj($oDoc) Then
    ConsoleWrite("found" & @CRLF)
    _UIA_DumpThemAll($oFF, $treescope_subtree)

; convert to Accessible
Local $oAccessible = ObjCreateInterface($oDoc, $sIID_IAccessible, $dtagIAccessible)
If IsObj($oAccessible) Then
    ConsoleWrite("Converted to IAccessible" & @CRLF)
    ConsoleWrite("Conversion failed" & @CRLF)

I would expect to get an Instance of IAccessible, but I get the following output instead:

Find Firefox ... found
Find Document ... found
Conversion failed

Maybe this is because ObjCreateInterface expects a pointer not an object?

Can you give any advice? Thanks in advance


Link to comment
Share on other sites

  • 5 weeks later...

I get an error right off the bat with a new copy of UIAWrappers.au3... it says "Variable used without being declared" and it points to "$UIA_OptimizeForVisualContentPropertyId"... Am I missing a required include or something?

Edit: (I'm following the quickstart guide with UIA_V0_70 and AutoIt V3.3.14.5)

Edit 2: Even stranger is that I can run the 'simple uia spy' without issue. There must a missing include file.

Edit 3: No idea how it happened, but apparently I had an old CUIAutomation2 file in my includes which didn't have that variable. Worked fine in the download folder because it was in the zip.

Edited by seadoggie01

All my code provided is Public Domain... but it may not work. ;) Use it, change it, break it, whatever you want.


My Humble Contributions:
Personal Function Documentation - A personal HelpFile for your functions
Acro.au3 UDF - Automating Acrobat Pro
ToDo Finder - Find #ToDo: lines in your scripts
UI-SimpleWrappers UDF - Use UI Automation more Simply-er
KeePass UDF - Automate KeePass, a password manager
InputBoxes - Simple Input boxes for various variable types

Link to comment
Share on other sites

  • 3 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Create New...