zenocon

249 ms to find it now with this little useful snippet: https://code.msdn.microsoft.com/windowsdesktop/Enumerate-top-level-9aa9d7c1/sourcecode?fileId=44683&pathId=1184961558 Bulk of that might be in test setup, since I'm testing it in a unit test. But it is definitely much speedier.

No, it's slow b/c I'm using a TreeWalker in the UI Automation API to iterate through children of desktop element to then check name property against Regex. It takes 5-7 sec which is ridiculous. I'm now trying a different approach using "user32.dll"

Yes, I solved it. You have to get a TreeWalker and just test regex on each one. Just stinks that they don't provide it through the API. Also appears to be quite slow. Oh well.

LOL, I think my first problem is finding the app handle. With AutoIT I'm using a regular expression to nab the title, which is very custom and dynamic depending on what screen they have open. Local $hWindow = WinGetHandle("[REGEXPTITLE:^\(SVC - .*\) - App-Name$]") ; then convert it to UIElement, etc. With .NET API it's something like this: var app = AutomationElement.RootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, " no regexp here "));

I don't have Enterprise. Nonetheless, I am proficient in C#/.NET so I'm thinking probably the best thing to do is to just build a lib that does the scraping. AutoIT / IUIA proved very useful for me to learn the API, and prototype proof-of-concept that showed we are able to scrape the info we want out of the legacy API. In addition, it was an easy / quick dev lifecycle to just rebuild the script to an exe and remote it to their desktop to test it, run inspect.exe to analyze problems and rinse and repeat. But now that I have the script working, I think I probably will convert it to a .NET library using the official API.

Thank you @junkew I have looked at the automation namespace in .NET, and while I think I could probably accomplish the same with that, it is easier for me to write .au3 script and test right now -- it's a faster development cycle with my situation at the moment. The UI I'm scraping is a legacy application that I have no control over and very limited access to -- making it difficult to verify the scripts. The legacy application does not provide any proper API for integration, so this is the path we're going to do an integration, after only having moderate success with an OCR approach.

I have a script I write as .au3 which uses the IUIA wrapper functions. It is working, but I need to invoke it from a C# WinForms app. I see there is AutoITX dll that exposes the scripting language to C#/.NET, but I think I can import UDF functions like IUI wrapper using that, can I? I think the best option I suppose is to just Invoke the built .au3 executable from C#. This only presents a challenge to get the return values from the script. The script scrapes info out of a legacy Windows app that I need. I can probably dump this to a file (ugly) or perhaps pass the info through stdio stream (less ugly). What are other people doing who need to integrate with .NET apps?

The documentation on the tree scope even on Microsoft's site https://msdn.microsoft.com/en-us/library/windows/desktop/ee671699(v=vs.85).aspx isn't exactly clear to me. Let's say there's this hierarchy desktop ├── app │ ├── elem1 │ ├── elem2 │ ├── elem3 │ │ ├── elem4 │ │ └── elem5 │ ├── elem6 │ ├── elem7 │ │ ├── elem8 │ │ │ └── elem9 │ │ ├── elem10 │ │ │ └── elem11 │ ├── elem12 │ │ ├── elem13 │ │ └── elem14 I'm mainly interested in only $treescope_descendants, $treescope_subtree, $treescope_children, since I think the others aren't supported. Can someone indicate what the explicit differences are? My guess is _UIA_getObjectByFindAll($app, <selector-that-matches-elem5>, $treescope_children) would fail to find elem5 because elem5 is not a direct child of $app, but _UIA_getObjectByFindAll($app, <selector-that-matches-elem5>, $treescope_subtree) would because it will descend down recursively from all children rooted under $app. Is that a correct statement? If so, can someone explain what $treescope_descendants is for? It sounds like $treescope_descendants is the same as $treescope_subtree but it does not include the top-level element you're starting the search from, whereas $treescope_subtree does include that top-level element and all descendants recursively.

In Visual Studio, there is a Document View which shows all the UI elements. By switching to that view it shows you the parent/child hierarchy of all the elements, and it allows you to drag-n-drop elements in whatever order or hierarchy you want. Once I figured this out, I was able to re-organize my fake UI to match the output of my inspect.exe results.

I think she means automating the UI of a code editor with AutoIT to write it in JavaScript and publish on the Chrome store.... j/k

That's odd, I only posted it once

Hi, After scouring the forums for many hours, I'm trying to compile the most up to date / recent information on the options available for integrating with JavaScript / DOM -- as it relates to scraping + automation of web pages. It's my understanding there is IE.au3 script for automation of IE through a COM interface. But I believe this only works with IE and won't work with Edge, correct? Is there a COM interface that works with Edge, or any other options for integrating with Edge (other than IUIAuatomation?) I know there was also a FF.au3 UDF, but Mozilla abandoned the support for their mozrepl in favor of Web Extensions, and my understanding is that the FF.au3 UDF no longer works, is that correct? There was also a Chrome.au3 UDF, but my read on the forums indicate that this also broke many Chrome releases past. Which leaves IUIAutomation which I have been using to automate / scrape Windows apps, but when I am trying it on a website, it is not as useful. For example, if I know the exact DOM id / class, I can get at it and do whatever I need to in JavaScript very simply. With IUIAutomation, the DOM properties are not available, and most tags / elements in DOM have no useful defining characteristics to be able to get at them reliably (if they are targetable at all). Some things might be able to be done with IUIAutomation, but I see it's value in targeting website automation / scraping as fairly limited. At this point, it seems like my best option is to use IE.au3, but that forces users on IE, which is probably a showstopper. Is there another way to bridge into the DOM? I have written Web Extensions for Chrome and Firefox before. They can communicate with external processes via AJAX or messaging. I'm wondering if I can build what I need in a WebExtension and then trigger it from AutoIT Script, and gather up the results somewhere. I know there was the ISimpleDOM.au3 and some Microsoft Accessability scripts, but they seem to only be partially supported in browsers, and I didn't have a lot of luck getting those examples to run correctly.

Hi, After scouring the forums for many hours, I'm trying to compile the most up to date / recent information on the options available for integrating with JavaScript / DOM -- as it relates to scraping + automation of web pages. It's my understanding there is IE.au3 script for automation of IE through a COM interface. But I believe this only works with IE and won't work with Edge, correct? Is there a COM interface that works with Edge, or any other options for integrating with Edge (other than IUIAuatomation?) I know there was also a FF.au3 UDF, but Mozilla abandoned the support for their mozrepl in favor of Web Extensions, and my understanding is that the FF.au3 UDF no longer works, is that correct? There was also a Chrome.au3 UDF, but my read on the forums indicate that this also broke many Chrome releases past. Which leaves IUIAutomation which I have been using to automate / scrape Windows apps, but when I am trying it on a website, it is not as useful. For example, if I know the exact DOM id / class, I can get at it and do whatever I need to in JavaScript very simply. With IUIAutomation, the DOM properties are not available, and most tags / elements in DOM have no useful defining characteristics to be able to get at them reliably (if they are targetable at all). Some things might be able to be done with IUIAutomation, but I see it's value in targeting website automation / scraping as fairly limited. At this point, it seems like my best option is to use IE.au3, but that forces users on IE, which is probably a showstopper. Is there another way to bridge into the DOM? I have written Web Extensions for Chrome and Firefox before. They can communicate with external processes via AJAX or messaging. I'm wondering if I can build what I need in a WebExtension and then trigger it from AutoIT Script, and gather up the results somewhere. I know there was the ISimpleDOM.au3 and some Microsoft Accessability scripts, but they seem to only be partially supported in browsers, and I didn't have a lot of luck getting those examples to run correctly.

Hi junkew -- Tab Index does not affect Automation Index. I indicated that in my first post above. The only thing that affects it, as far as I can tell, is the order you add child components to a parent.

Ugh it appears it is just LIFO (Last In First Out). It's like a stack, push a new element into a container and it is at the top of the list. I don't see any other way to modify.

I'm building a fake WinForms GUI that I'm using to test. In one instance, it has a panel with a number of text elements. I have the inspect.exe output of the real target UI that I'm trying to automate, but I'm trying to build a fake one now that emulates it so I can more easily test. The problem is the order of the text elements in the pane is different, and I'm having a helluva time trying to figure out how to set the order -- if that's even possible. The WinForms TabIndex property doesn't do it. There is no Index property on it otherwise. I'm worried it may not even be possible, so I figured I'd ask here. Thank in advance

Consider the following image. I'm interested in all of the text elements embedded in the last pane called "Power Container". Unfortunately, this pane is embedded in 8 other panes with the same name and no other identifying characteristics. I think I can get to the target pane using indexrelative. Local $powerContainer = _UIA_getObjectByFindAll($cPowerMain, "controltype:=UIA_PaneControlTypeId;Name:=Power Container;indexrelative:=8", $treescope_subtree) If Not IsObj($powerContainer) Then Return ConsoleWrite("Could not find Power Container pane") So, now I have the pane, but I want to target individual text elements. They do seemingly have AutomationIds so I think I can use that as a selector. Problem is I'm building a fake WinForms GUI to test with since I don't have access to the real UI to automate for testing purposes, and with WinForms, there is no way to set the AutomationId. There is no other identifying information on these text elements. Their Name property contains the actual data, which varies. Except for the labels that sit in from of them. The way it is built it looks like this: My question is this: I know I can select to get the one with the static label value "Tag#" Local $tagLabel = _UIA_getObjectByFindAll($powerContainer, "controltype:='UIA_TextControlTypeId;Name:=Tag#", $treescope_subtree) Now, what I want to do is navigate to get the next sibling of this element. It doesn't look like the API allows me to do this? I think what I have to do is get a RawViewWalker and just iterate through all of them and inspect each one and figure out if it is what I'm looking for. Does this seem correct? Is there a better way to achieve this?

Solved it. The loop was wrong. It was copy/paste from another example. Didn't realize it was trying to ensure there was only one element in the array when I was looking for more.

Bizarre that I cannot find any child elements. Anyone know what I'm doing wrong here? Local $vehInfo = _UIA_getObjectByFindAll($oWindow, "Title:=Display vehicle information for RO", $treescope_subtree) If Not IsObj( $vehInfo ) Then Return elog("Could not find vehicle information popup dialog") Local $pUIElementArray, $oUIElementArray, $iElements $vehInfo.FindAll( $treescope_subtree, $UIA_oTRUECondition, $pUIElementArray ) $oUIElementArray = ObjCreateInterface( $pUIElementArray, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray ) $oUIElementArray.Length( $iElements ) If Not ( $iElements = 1 ) Then Exit ConsoleWrite( "$iElements ERR" & @CRLF ) ConsoleWrite( "$iElements OK" & @CRLF ) I have a handle on the element with the title "Display vehicle information for RO", and when I do a FindAll on it, it returns empty, but showing the inspect.exe screenshot it has child elements. Why am I coming up empty?

Hi, If I'm reading the code in UIAWrappers.au3 correctly, the function _UIA_getObjectByFindAll( ) uses the UIElement.FindAll with the true condition, so it would find any elements that match the treescope passed in. This populates an array, and then it iterates through them and tries matching the properties that were passed into _UIA_getObjectByFindAll and returns the first element that is the best match. I understand one can use the index or relativeindex properties to target a specific child, let's say when you have a panel that has a bunch of edit fields. If you know the specific index of the one you want you can use the index to get it back, but what if I want to return, say the whole array of UIElements that match a specific type. Is my best option here to just use the .FindAll( ) function directly and set my own condition to match the controltypeid?

Thank you junkew -- _UIA_Action($oTab, "setfocus") also appears to do the trick.

I have a winforms app with multiple tabs on a tab control, and I'm trying to tab to one of them. I'm trying to use the Invoke pattern but it is not working. Apparently a tab doesn't implement invoke? What other pattern can I try? #include "../uia/UIA_V0_64/UIAWrappers.au3" #include "../uia/UIA_V0_64/CUIAutomation2.au3" AutoItSetOption("MustDeclareVars", 1) Example() Func Example()     ; Get Handle to the application     Local $hWindow = WinGetHandle("Service/STG")     If Not $hWindow Then Return ConsoleWrite( "Window handle ERR" & @CRLF )     ConsoleWrite( "Window handle OK" & @CRLF )     WinActivate($hWindow)     ; Create UI Automation object     Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )     If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "UI Automation object ERR" & @CRLF )     ConsoleWrite( "UI Automation object OK" & @CRLF )     ; Get UI Automation element from window handle     Local $pWindow, $oWindow     $oUIAutomation.ElementFromHandle( $hWindow, $pWindow )     $oWindow = ObjCreateInterface( $pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )     If Not IsObj( $oWindow ) Then Return ConsoleWrite( "Automation element from window ERR" & @CRLF )     ConsoleWrite( "Automation element from window OK" & @CRLF )     Local $pCondition, $pCondition1, $pCondition2     $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_TabItemControlTypeId, $pCondition1)     $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Service/STG", $pCondition2)     $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )     If Not $pCondition Then Return ConsoleWrite("Property condition ERR" & @CRLF)     ; Find the Service/STG Tab so we can click it     Local $pTab, $oTab     $oWindow.FindFirst($TreeScope_Descendants, $pCondition, $pTab)     $oTab = ObjCreateInterface($pTab, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)     If Not IsObj($oTab) Then Return ConsoleWrite("Find Service tab ERR" & @CRLF)     ; Invoke the Service/STG Tab     Local $pInvoke, $oInvoke     $oTab.GetCurrentPattern($UIA_InvokePatternId, $pInvoke)     $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )     If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "Invoke object ERR" & @CRLF)     $oInvoke.Invoke()

Hi LarsJ -- Thanks much for the response. I have made some further progress. I have tried the Simple Spy Tool, but it also was not seemingly able to target the text box control I am hoping to interact with. So, I tried Microsoft's inspect.exe that I found in the forums here and that can identify it. The information is below from that tool. As you can see, I know it *can* be automated since it is showing that is how I found it. I did it through the Actions menu on that tool, and executed the Value.setValue action (which worked). So I thought I had it easy b/c it seems to have an AutomationId, and I thought well maybe I can use that to directly address it. However, my attempts to do so below were not successful, so I'm hoping someone else might be able to show me the err of my ways (code below) How found: Value.SetValue... Name: "Tag:" ControlType: UIA_EditControlTypeId (0xC354) LocalizedControlType: "edit" BoundingRectangle: {l:595 t:521 r:699 b:541} IsEnabled: true IsOffscreen: false IsKeyboardFocusable: true HasKeyboardFocus: false AccessKey: "" ProcessId: 6216 RuntimeId: [2A.440654] AutomationId: "27" FrameworkId: "Win32" ClassName: "Edit" NativeWindowHandle: 0x440654 IsContentElement: true ProviderDescription: "[pid:7432,hwnd:0x440654 Main:Nested [pid:6216,hwnd:0x440654 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]" IsPeripheral: [Not supported] LiveSettingProperty: [Not supported] IsPassword: false HelpText: "" FlowsFrom: [Not supported] OptimizeForVisualContent: [Not supported] PositionInSet: [Not supported] SizeOfSet: [Not supported] Level: [Not supported] AnnotationTypes: [Not supported] AnnotationObjects: [Not supported] LandmarkType: [Not supported] LocalizedLandmarkType: [Not supported] FullDescription: [Not supported] Annotation.AnnotationTypeName: [Not supported] Annotation.AnnotationTypeId: [Not supported] Annotation.Author: [Not supported] Annotation.DateTime: [Not supported] Annotation.Target: [Not supported] Drag.DropEffect: [Not supported] Drag.DropEffects: [Not supported] Drag.GrabbedItems: [Not supported] Drag.IsGrabbed: [Not supported] DropTarget.DropTargetEffect: [Not supported] DropTarget.DropTargetEffects: [Not supported] LegacyIAccessible.ChildId: 0 LegacyIAccessible.DefaultAction: "" LegacyIAccessible.Description: "" LegacyIAccessible.Help: "" LegacyIAccessible.KeyboardShortcut: "" LegacyIAccessible.Name: "Tag:" LegacyIAccessible.Role: editable text (0x2A) LegacyIAccessible.State: focusable (0x100000) LegacyIAccessible.Value: "dv" ObjectModel.UnderlyingObjectModel: [Error: calling getter for this property: hr=0xFFFFFFFF80070057 - The parameter is incorrect.] Scroll.HorizontallyScrollable: false Scroll.HorizontalScrollPercent: 0.000000 Scroll.HorizontalViewSize: 100.000000 Scroll.VerticallyScrollable: false Scroll.VerticalScrollPercent: 0.000000 Scroll.VerticalViewSize: 100.000000 SpreadsheetItem.AnnotationObjects: [Not supported] SpreadsheetItem.AnnotationTypes: [Not supported] SpreadsheetItem.Formula: [Not supported] Style.ExtendedProperties: [Not supported] Style.FillColor: [Not supported] Style.FillPatternColor: [Not supported] Style.FillPatternStyle: [Not supported] Style.Shape: [Not supported] Style.StyleId: [Not supported] Style.StyleName: [Not supported] Value.IsReadOnly: false Value.Value: "dv" Transform2.CanZoom: [Not supported] Transform2.ZoomLevel: [Not supported] Transform2.ZoomMinimum: [Not supported] Transform2.ZoomMaximum: [Not supported] IsAnnotationPatternAvailable: [Not supported] IsDragPatternAvailable: [Not supported] IsDockPatternAvailable: false IsDropTargetPatternAvailable: [Not supported] IsExpandCollapsePatternAvailable: false IsGridItemPatternAvailable: false IsGridPatternAvailable: false IsInvokePatternAvailable: false IsItemContainerPatternAvailable: false IsLegacyIAccessiblePatternAvailable: true IsMultipleViewPatternAvailable: false IsObjectModelPatternAvailable: [Not supported] IsRangeValuePatternAvailable: false IsScrollItemPatternAvailable: false IsScrollPatternAvailable: true IsSelectionItemPatternAvailable: false IsSelectionPatternAvailable: false IsSpreadsheetItemPatternAvailable: [Not supported] IsSpreadsheetPatternAvailable: [Not supported] IsStylesPatternAvailable: [Not supported] IsSynchronizedInputPatternAvailable: false IsTableItemPatternAvailable: false IsTablePatternAvailable: false IsTextChildPatternAvailable: [Not supported] IsTextEditPatternAvailable: [Not supported] IsTextPatternAvailable: false IsTextPattern2Available: [Not supported] IsTogglePatternAvailable: false IsTransformPatternAvailable: false IsTransform2PatternAvailable: [Not supported] IsValuePatternAvailable: true IsVirtualizedItemPatternAvailable: false IsWindowPatternAvailable: false IsCustomNavigationPatternAvailable: [Not supported] FirstChild: [null] LastChild: [null] Next: "COMMAND:" text Previous: "Tag:" text Other Props: Object has no additional properties Children: Container has no children Ancestors: "Dispatch (DSP)" pane "" pane "http://redacted/ "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" tab "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "" pane "Service/STG - Dispatching - redacted - redacted Drive" window "Desktop" pane [ No Parent ] Here is some simple code I have tried: #include "../uia/UIA_V0_64/UIAWrappers.au3" #include "../uia/UIA_V0_64/CUIAutomation2.au3" AutoItSetOption("MustDeclareVars", 1) Example() Func Example() Local $mainApp = _UIA_getObjectByFindAll($UIA_oDesktop, "AutomationId:=Portal", $treescope_children) If Not IsObj($mainApp) Then _WriteErrorLog( "$mainApp ERR" & @CRLF ) _WriteErrorLog( "$mainApp OK" & @CRLF ) Local $editbox = _UIA_getObjectByFindAll($mainApp, "AutomationId:=27", $treescope_subtree) If Not IsObj($editbox) Then _WriteErrorLog( "$editbox ERR" & @CRLF ) _UIA_Action($editbox,"setfocus") _UIA_Action($editbox,"highlight") Sleep(1000) _UIA_Action($editbox,"sendkeys","DV") _UIA_Action($editbox,"sendkeys","{ENTER}") EndFunc Func _WriteErrorLog($ErrorMessage) ConsoleWrite($ErrorMessage) FileWriteLine(@ScriptDir & "\" & @ScriptName & ".log", @HOUR & ":" & @MIN & ":" & @SEC & ": " & $ErrorMessage) EndFunc ;==>WriteErrorLog It did not work as expected -- strangely enough it showed a completely different highlighted portion of the app, and the error log also indicated that the $editbox was not an object, which is just confusing. $mainApp prints ok, though. I'm using the AutomationId:=Portal on that which I gleaned through inspect.ext tool. It seems to be an OK selector for the target app. But here, I'm wondering why I'm not able to successfully target the text box with it's listed AutomationId:=27, or somehow it is confusing it with another? Given the dump above of the element I want to automate, can anyone recommend a more robust selector that will allow me to interact with the control. Thank you in advance

Correction: I've tried the simplespy script locally and it works fine, but I don't have easy access to the target app/controls I want to get at. I have to physically drive to another location and try it on-site, so I'm trying to do as much homework as I can to ensure the trips are worth it, if that makes sense.

So, I've done a lot of reading. And I've started playing with the IUIAotmation scripts. I will try the `simplespy` script and see if I can target those controls using the UIAutomation integration. If there are any other tools that would be useful for introspecting windows apps / components, I'd love to hear about them -- most of the links posted in here are broken or dead, so it is a long haul to try to get updated information.