Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


junkew
 Share

Recommended Posts

released in first post as attachment version 0.66 + examplesV0.66

@rvicta download in first post v0.66 and see example 6 where in last step 10 I just reference a piece of text by name only.

Example could be

;~ Find the chrome window
local $oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Chrome_WidgetWin_1", $treescope_children)
If Not IsObj($oChrome) Then
    _UIA_DumpThemAll($UIA_oDesktop, $treescope_subtree)
    Exit
EndIf
;~ Get the document reference
$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)
If Not IsObj($oDocument) Then
    _UIA_DumpThemAll($oDocument, $treescope_subtree)
EndIf
    
Local $oUIElement=_UIA_getObjectByFindAll($oDocument, "title:=Part Number Entry;ControlType:=Text", $treescope_subtree)

_UIA_Action($oUIElement,"highlight")
consolewrite(_UIA_action($oUIElement, "property", "title"))

otherwise search in examples for indexrelative:=(+/-)n where you can reference relative n position back or forward close to another object. You can determine n by looking into the result of _UIA_DumpThemAll($oDocument,$treescope_subtree)

Link to comment
Share on other sites

Thank you, junkew, for your suggestions. This worked great for what I was trying to do. I can now read the text controls properly.

But now, I have come upon an odd behavior. This is still the same Chrome session for a customer. I have 3 (actually there are more and they exhibit this same behavior but I'm using these three as an example) edit controls on the screen. The first and third controls have different values in them. When I write their values to the console they showed the correct values. The second edit control, if it is empty on the screen, will show me the same value as the first edit control when I write it to the console. I know the index is correct for this edit control because I commented out the code to read from all other edit controls on the page and the correct one gets highlighted when running the script.

Local $oPhasedOut = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_TextControlTypeId & ";Index:=15", $treescope_subtree)
   Local $sPhasedOut = _UIA_action($oPhasedOut, "property", "title")
   ConsoleWrite("Phased Out: " & $sPhasedOut & @CRLF)

   Local $oCost = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=6", $treescope_subtree)
   Local $sCost = _UIA_Action($oCost, "getValue")
   ConsoleWrite("Cost: " & $sCost & @CRLF)

   Local $oTrade = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=7", $treescope_subtree)
   Local $sTrade = _UIA_Action($oTrade, "getValue")
   ConsoleWrite("Trade: " & $sTrade & @CRLF)

   Local $oList = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=8", $treescope_subtree)
   Local $sList = _UIA_Action($oList, "getValue")
   ConsoleWrite("List: " & $sList & @CRLF)

The output from SimplySpy for this empty edit control is:

Having the following values for all properties: 
Title is: <>    Class   := <>   controltype:= <UIA_EditControlTypeId>   ,<50004>    , (0000C354)    610;551;87;20
*** Parent Information top down ***
5: Title is: <BS1-My Favorites> Class   := <Chrome_WidgetWin_1> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    14;16;1663;1056
"Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1""   
4: Title is: <BS1-My Favorites> Class   := <Chrome_RenderWidgetHostHWND>    controltype:= <UIA_DocumentControlTypeId>   ,<50030>    , (0000C36E)    22;101;1647;963
"Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND""    
3: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    22;101;1647;808
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
2: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    22;136;1647;773
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
1: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    365;259;961;688
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
0: Title is: <Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* >  Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    375;285;1023;616
"Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=""  


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1") ;BS1-My Favorites
_UIA_setVar("oP2","Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")  ;BS1-My Favorites
_UIA_setVar("oP3","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP6","Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=")    ;Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* 

;~ $oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=;controltype:=UIA_EditControlTypeId;class:=") ;ControlType:=UIA_EditControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
_UIA_Action("oP6","setfocus")

_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP5=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)    
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)  
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <610;551;87;20>
UIA_ProcessId:= <2964>
UIA_ControlType:= <50004>
UIA_LocalizedControlType:= <edit>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <True>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <True>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <False>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleRole:= <42>
UIA_LegacyIAccessibleState:= <1048576>
UIA_LegacyIAccessibleDefaultAction:= <activate>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:2964,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

I noticed that UIA_iaccessiblevalue is not present for empty edit controls. Is there a way to check for this? Then I would probably know that there is no value.

Edited by rvicta
Link to comment
Share on other sites

Within simplespy  or uiawrappers I filter out empty values.
To be 100% sure my advice use

  • inspect as part of windows SDK
  • VisualUIAVerifyNative.exe as part of windows SDK (8.1)

I see you use index (which is probably fine for your purpose) but understand clearly also the indexrelative property as that could be usefull to get the textlabel normally nn positions before your editbox if your developers program nicely its the same distance for all textlabel controls.

Link to comment
Share on other sites

Thank you for the help, junkew.

Using $UIA_LegacyIAccessibleValuePropertyId for edit controls I am able to determine if it is empty. Using Inspect.exe I was able to see that ValuePropertId would either contain the value or an empty string.

Link to comment
Share on other sites

  • 2 weeks later...
  • 1 month later...

Many ideas to rewrite with dot syntax and more wrapped away to deal consistently with different browsers in 1 syntax using autoitobject internal udf. But as time is limited not on short term.

But if you miss something let me know so maybe easy to add.

Link to comment
Share on other sites

Skype 8 (on Windows 8.1) use Chrome (Chrome_WidgetWin_1)

Some time its able to select the Sign In button and sometime it dont. The following screenshot depicts when it do not select the individual item Not Sign in nor Create Account etc.

SimpleUIASpy_Skype.thumb.png.14e0006e1e1f2d360103c7c1d83fad0d.png

I suspect Skype is somehow changing the DOM, but still, its visible, manually its possible to click the Sign in button, when it do not highlight the control (and do not display the code), the following code do not work

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP2=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Skype;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Skype Preview;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("SigninwithMicrosoft.mainwindow", "title:=Sign in with Microsoft;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Sign in;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

Is there any way to find the exact reason and solution?

Thanks.

Update

I realized that whenever I am running simplespy in parallel, it identifies the control, therefore I copied _EventProc() from simplespy.au3 and added into my script, tested couple of times, its working everytime. Trying to understand why Chrome Accessibility has its effect on finding the control. Would still appreciate expert opinion here.

Edited by r0ash
adding more information
Link to comment
Share on other sites

@junkew @r0ash

It seems like no need to wait until chromium sends EVENT_SYSTEM_ALERT message. Just to send a message to RenderWidget control and chromium turns on the accessibility tree. Tested on Google Chrome 67 portable without  --force-renderer-accessibility parameter

$hWnd = ControlGetHandle("[RegexpTitle: - Google Chrome]", "", "Chrome_RenderWidgetHostHWND1")
_SendMessage($hWnd, $WM_GETOBJECT, 0, 1)

 

Link to comment
Share on other sites

  • 4 weeks later...

Hi I'm looking to automate a third party application called ams360 for data entry. Is it possible with this UDF? I don't think the Control Functions work correctly for me. Is there an easier alternative, it looks very complicated and I don't have alot of scripting skill. What should I be doing to learn more about how to use this UDF.

Link to comment
Share on other sites

@milkmoron I am not aware of any other easier alternative, but you can starting by reading the linked posts and examples in the first post:

Good luck :)

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

So I'm using IUIAutomation and I'm not sure if I'm doing this correctly but heres my code to select the description text area. Store the value for later and send it to another text area.

#include "UIAWrappers.au3"
ControlFocus ( "View Activity", "", "")
Local $oViewActivity = ControlGetFocus ( "View Activity" )

_UIA_SETVAR("Description","classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad118")
Local $oPolicy = _UIA_Action("Description","getvalue")

MsgBox($MB_SYSTEMMODAL, "", "The text in Name1 is: " & $oPolicy)

ControlFocus ( "Activity / Suspense", "", "")
WinActivate ( "Activity / Suspense" )

_UIA_SETVAR("FIADescription","title:=Description:")
;_UIA_SETVAR("FIADescription","classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad14
_UIA_Action("FIADescription","sendkeys", $oPolicy)

It pauses during Local $oPolicy = _UIA_Action("Description","getvalue")

Link to comment
Share on other sites

Start with simplespy and post output of that. Better done in general help and support area with new thread as the answers can be lengthy.

For beginners frequently its hard to understand gui hierarchy and z order of elements. For that simplespy and inspect are tools to help you.

First step is to have action highlight working on your element of interest from then onwards it should become easier. In the background some log*.xml is made giving you all details on how udf is trying to find your element.

Link to comment
Share on other sites

On 7/18/2018 at 11:56 AM, junkew said:

Start with simplespy and post output of that. Better done in general help and support area with new thread as the answers can be lengthy.

For beginners frequently its hard to understand gui hierarchy and z order of elements. For that simplespy and inspect are tools to help you.

First step is to have action highlight working on your element of interest from then onwards it should become easier. In the background some log*.xml is made giving you all details on how udf is trying to find your element.

Which parts need to be outputted. It has some confidential information in it.

Link to comment
Share on other sites

Really no need to quote the previous post.

The parts to post the output of are the parts that show the controls in question.

Spoiler

Renamer - Rename files and folders, remove portions of text from the filename etc.

GPO Tool - Export/Import Group policy settings.

MirrorDir - Synchronize/Backup/Mirror Folders

BeatsPlayer - Music player.

Params Tool - Right click an exe to see it's parameters or execute them.

String Trigger - Triggers pasting text or applications or internet links on specific strings.

Inconspicuous - Hide files in plain sight, not fully encrypted.

Regedit Control - Registry browsing history, quickly jump into any saved key.

Time4Shutdown - Write the time for shutdown in minutes.

Power Profiles Tool - Set a profile as active, delete, duplicate, export and import.

Finished Task Shutdown - Shuts down pc when specified window/Wndl/process closes.

NetworkSpeedShutdown - Shuts down pc if download speed goes under "X" Kb/s.

IUIAutomation - Topic with framework and examples

Au3Record.exe

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...