Jump to content
zenocon

Question on Controls / Targeting with AutoIT Info Finder

Recommended Posts

zenocon

Hello,

I'm trying to write a script to automate part of what appears to be a Winforms app.  At least when I use AutoIT Info and drop the finder on it, it lists the Window's class as: WindowsForms10.Windows.8.app.0.3551b1b_r14_ad2.

The bizarre thing is some of the controls don't seem to be targetable.  See the screen shot attached with what appears to show a WinForms GroupBox with the title "Command Window".  Inside the group box it appears to have a Label, and TextBox, and a Button.  Using the AutoIT Info Finder, I'm not able to individually address the inner components.  I can only seem to target the GroupBox. 

Attached are the results of the Finder/Control window.  Notice that it shows the control has class SWHGRP, in Advanced Mode: [CLASS:SWHGRP;INSTANCE:1].  My assumption here, though is that this is the GroupBox control.  What I want to do is automate entering some text into the TextBox in there and then clicking the button.  

If this is just a standard Winforms app, it seems odd to me that these inner controls would not be targetable, but I'm pretty new to AutoIT (although certainly not new to software development).  Using a standard mouse, I can certainly click in the TextBox, enter some text and click the button, so they are legit controls.

  • Can someone help me understand why some controls like this appear to not be able to be targeted by the Finder? 
  • If I write my own script can I just enumerate all the children and see if I can find it that way?  Is it worth trying to do this, or if the AutoIT Info Finder cannot address them, then that's likely definitive and I'm out of luck?
  • Is it possible they may have some kind of 3rd party control embedded in there like a webview -- and this might be what is causing the issue?

 

 

Screen Shot 2017-11-20 at 2.12.53 PM.png

Screen Shot 2017-11-20 at 2.11.29 PM.png

Screen Shot 2017-11-20 at 2.11.23 PM.png

Share this post


Link to post
Share on other sites
zenocon

So, I've done a lot of reading.  And I've started playing with the IUIAotmation scripts.  I will try the `simplespy` script and see if I can target those controls using the UIAutomation integration.  If there are any other tools that would be useful for introspecting windows apps / components, I'd love to hear about them -- most of the links posted in here are broken or dead, so it is a long haul to try to get updated information.

Share this post


Link to post
Share on other sites
zenocon
Just now, zenocon said:

So, I've done a lot of reading.  And I've started playing with the IUIAotmation scripts.  I will try the `simplespy` script and see if I can target those controls using the UIAutomation integration.  If there are any other tools that would be useful for introspecting windows apps / components, I'd love to hear about them -- most of the links posted in here are broken or dead, so it is a long haul to try to get updated information.

Correction: I've tried the simplespy script locally and it works fine, but I don't have easy access to the target app/controls I want to get at.  I have to physically drive to another location and try it on-site, so I'm trying to do as much homework as I can to ensure the trips are worth it, if that makes sense.

Share this post


Link to post
Share on other sites
LarsJ

Hello Mr. zenocon and welcome to our forums.

The "AutoIt Window Info" tool generally detects standard Windows controls in the Control Library, but has only very limited support for Windows Forms controls in .NET Framework.

The idea of enumerating the child windows will not really help. The controls are still Windows Forms controls and "AutoIt Window Info" tool will probably not give much information.

That some kind of 3rd party code is the reason why you cannot identify the controls is unlikely. The reason is simply the "AutoIt Window Info" tool's lack of support for these controls.

The best chance to identify the controls is the UI Automation framework and the "Simple spy tool". Microsoft's answer to this tool is Inspect.exe which is included in the Windows SDK. You can also download a copy in this post.

  • Like 1

Share this post


Link to post
Share on other sites
zenocon

Hi LarsJ -- Thanks much for the response.  I have made some further progress.  I have tried the Simple Spy Tool, but it also was not seemingly able to target the text box control I am hoping to interact with.  So, I tried Microsoft's inspect.exe that I found in the forums here and that can identify it.  The information is below from that tool.

As you can see, I know it *can* be automated since it is showing that is how I found it.  I did it through the Actions menu on that tool, and executed the Value.setValue action (which worked).  So I thought I had it easy b/c it seems to have an AutomationId, and I thought well maybe I can use that to directly address it.  However, my attempts to do so below were not successful, so I'm hoping someone else might be able to show me the err of my ways (code below)

How found:  Value.SetValue...
Name:   "Tag:"
ControlType:    UIA_EditControlTypeId (0xC354)
LocalizedControlType:   "edit"
BoundingRectangle:  {l:595 t:521 r:699 b:541}
IsEnabled:  true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:   false
AccessKey:  ""
ProcessId:  6216
RuntimeId:  [2A.440654]
AutomationId:   "27"
FrameworkId:    "Win32"
ClassName:  "Edit"
NativeWindowHandle: 0x440654
IsContentElement:   true
ProviderDescription:    "[pid:7432,hwnd:0x440654 Main:Nested [pid:6216,hwnd:0x440654 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]"
IsPeripheral:   [Not supported]
LiveSettingProperty:    [Not supported]
IsPassword: false
HelpText:   ""
FlowsFrom:  [Not supported]
OptimizeForVisualContent:   [Not supported]
PositionInSet:  [Not supported]
SizeOfSet:  [Not supported]
Level:  [Not supported]
AnnotationTypes:    [Not supported]
AnnotationObjects:  [Not supported]
LandmarkType:   [Not supported]
LocalizedLandmarkType:  [Not supported]
FullDescription:    [Not supported]
Annotation.AnnotationTypeName:  [Not supported]
Annotation.AnnotationTypeId:    [Not supported]
Annotation.Author:  [Not supported]
Annotation.DateTime:    [Not supported]
Annotation.Target:  [Not supported]
Drag.DropEffect:    [Not supported]
Drag.DropEffects:   [Not supported]
Drag.GrabbedItems:  [Not supported]
Drag.IsGrabbed: [Not supported]
DropTarget.DropTargetEffect:    [Not supported]
DropTarget.DropTargetEffects:   [Not supported]
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    ""
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "Tag:"
LegacyIAccessible.Role: editable text (0x2A)
LegacyIAccessible.State:    focusable (0x100000)
LegacyIAccessible.Value:    "dv"
ObjectModel.UnderlyingObjectModel:  [Error: calling getter for this property: hr=0xFFFFFFFF80070057 - The parameter is incorrect.]
Scroll.HorizontallyScrollable:  false
Scroll.HorizontalScrollPercent: 0.000000
Scroll.HorizontalViewSize:  100.000000
Scroll.VerticallyScrollable:    false
Scroll.VerticalScrollPercent:   0.000000
Scroll.VerticalViewSize:    100.000000
SpreadsheetItem.AnnotationObjects:  [Not supported]
SpreadsheetItem.AnnotationTypes:    [Not supported]
SpreadsheetItem.Formula:    [Not supported]
Style.ExtendedProperties:   [Not supported]
Style.FillColor:    [Not supported]
Style.FillPatternColor: [Not supported]
Style.FillPatternStyle: [Not supported]
Style.Shape:    [Not supported]
Style.StyleId:  [Not supported]
Style.StyleName:    [Not supported]
Value.IsReadOnly:   false
Value.Value:    "dv"
Transform2.CanZoom: [Not supported]
Transform2.ZoomLevel:   [Not supported]
Transform2.ZoomMinimum: [Not supported]
Transform2.ZoomMaximum: [Not supported]
IsAnnotationPatternAvailable:   [Not supported]
IsDragPatternAvailable: [Not supported]
IsDockPatternAvailable: false
IsDropTargetPatternAvailable:   [Not supported]
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsObjectModelPatternAvailable:  [Not supported]
IsRangeValuePatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsScrollPatternAvailable:   true
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:  [Not supported]
IsSpreadsheetPatternAvailable:  [Not supported]
IsStylesPatternAvailable:   [Not supported]
IsSynchronizedInputPatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    [Not supported]
IsTextEditPatternAvailable: [Not supported]
IsTextPatternAvailable: false
IsTextPattern2Available:    [Not supported]
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:   [Not supported]
IsValuePatternAvailable:    true
IsVirtualizedItemPatternAvailable:  false
IsWindowPatternAvailable:   false
IsCustomNavigationPatternAvailable: [Not supported]
FirstChild: [null]
LastChild:  [null]
Next:   "COMMAND:" text
Previous:   "Tag:" text
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "Dispatch (DSP)" pane
    "" pane
    "http://redacted/
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" tab
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "Service/STG - Dispatching - redacted - redacted Drive" window
    "Desktop" pane
    [ No Parent ]

Here is some simple code I have tried:

 

#include "../uia/UIA_V0_64/UIAWrappers.au3"
#include "../uia/UIA_V0_64/CUIAutomation2.au3"
AutoItSetOption("MustDeclareVars", 1)

Example()

Func Example()
    Local $mainApp = _UIA_getObjectByFindAll($UIA_oDesktop, "AutomationId:=Portal", $treescope_children)
    If Not IsObj($mainApp) Then _WriteErrorLog( "$mainApp ERR" & @CRLF )
    _WriteErrorLog( "$mainApp OK" & @CRLF )

    Local $editbox = _UIA_getObjectByFindAll($mainApp, "AutomationId:=27", $treescope_subtree)
    If Not IsObj($editbox) Then _WriteErrorLog( "$editbox ERR" & @CRLF )
    _UIA_Action($editbox,"setfocus")
    _UIA_Action($editbox,"highlight")
    Sleep(1000)
    _UIA_Action($editbox,"sendkeys","DV")
    _UIA_Action($editbox,"sendkeys","{ENTER}")
EndFunc

Func _WriteErrorLog($ErrorMessage)
    ConsoleWrite($ErrorMessage)
    FileWriteLine(@ScriptDir & "\" & @ScriptName & ".log", @HOUR & ":" & @MIN & ":" & @SEC & ": " & $ErrorMessage)
EndFunc   ;==>WriteErrorLog

It did not work as expected -- strangely enough it showed a completely different highlighted portion of the app, and the error log also indicated that the $editbox was not an object, which is just confusing.  $mainApp prints ok, though.  I'm using the AutomationId:=Portal on that which I gleaned through inspect.ext tool.  It seems to be an OK selector for the target app.

But here, I'm wondering why I'm not able to successfully target the text box with it's listed AutomationId:=27, or somehow it is confusing it with another?

Given the dump above of the element I want to automate, can anyone recommend a more robust selector that will allow me to interact with the control.

Thank you in advance

Share this post


Link to post
Share on other sites
junkew

Use name property to be Tag: 

Or use another object together with indexrelative property

See examples.zip with different examples of name:=value pair examples including indexrelative property. Best to calculate with inspect.exe

Post simplespy output and tell what highlights. Most likely z-order of elements is incorrect and as such object from point gives incorrect result.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.