Search the Community

Please help me to get text from another program using UI I need my program to extract symbol and its description from TC2000, basically a "element title". I want it to return "INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology" (or any other text if symbol changes) my wrapper is: Mouse position is retrieved 1194-405 At least we have an element title: [INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology ] class: [TextBlock] Having the following values for all properties: Title is: <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > Class := <TextBlock> controltype:= <UIA_TextControlTypeId> ,<50020> , (0000C364) 989;356;406;183 *** Parent Information top down *** 2: Title is: <TC2000 ® Version 17.0.6435.29524> Class := <Window> controltype:= <UIA_WindowControlTypeId> ,<50032> , (0000C370) 51;24;1712;1124 "Title:=TC2000 ® Version 17.0.6435.29524;controltype:=UIA_WindowControlTypeId;class:=Window"" 1: Title is: <> Class := <WorkspaceTabControl> controltype:= <UIA_CustomControlTypeId> ,<50025> , (0000C369) 59;123;1696;1017 "Title:=;controltype:=UIA_CustomControlTypeId;class:=WorkspaceTabControl"" 0: Title is: <> Class := <WebChart> controltype:= <UIA_CustomControlTypeId> ,<50025> , (0000C369) 675;125;1080;612 "Title:=;controltype:=UIA_CustomControlTypeId;class:=WebChart"" ;~ *** Standard code maintainable *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) _UIA_setVar("oP1","Title:=TC2000 ® Version 17.0.6435.29524;controltype:=UIA_WindowControlTypeId;class:=Window") ;TC2000 ® Version 17.0.6435.29524 _UIA_setVar("oP2","Title:=;controltype:=UIA_CustomControlTypeId;class:=WorkspaceTabControl") ; _UIA_setVar("oP3","Title:=;controltype:=UIA_CustomControlTypeId;class:=WebChart") ; ;~ $oUIElement=_UIA_getObjectByFindAll("INNT InnovateBiopharmaceuticals,Inc HealthcareSector(D) Biotechnology .mainwindow", "title:=INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology ;ControlType:=UIA_TextControlTypeId", $treescope_subtree) _UIA_setVar("oUIElement","Title:=INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology ;controltype:=UIA_TextControlTypeId;class:=TextBlock") ;ControlType:=UIA_TextControlTypeId;classname:=TextBlock") ;~ Actions split away from logical/technical definition above can come from configfiles ;~_UIA_Action("oP1","highlight") _UIA_Action("oP1","setfocus") ;~_UIA_Action("oP2","highlight") _UIA_Action("oP2","setfocus") ;~_UIA_Action("oP3","highlight") _UIA_Action("oP3","setfocus") ;~_UIA_action($oUIElement","highlight") ;~_UIA_action($oUIElement,"click") ;~_UIA_action("oUIElement","highlight") ;~_UIA_action("oUIElement","click") ;~ *** Standard code Flexible*** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP2=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=TC2000 ® Version 17.0.6435.29524;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) _UIA_Action($oP2,"setfocus") Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_CustomControlTypeId;class:=WorkspaceTabControl", $treescope_children) Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_CustomControlTypeId;class:=WebChart", $treescope_children) ;~ First find the object in the parent before you can do something ;~$oUIElement=_UIA_getObjectByFindAll("INNT InnovateBiopharmaceuticals,Inc HealthcareSector(D) Biotechnology .mainwindow", "title:=INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology ;ControlType:=UIA_TextControlTypeId", $treescope_subtree) Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology ;ControlType:=UIA_TextControlTypeId", $treescope_subtree) ;~_UIA_action($oUIElement,"highlight") _UIA_action($oUIElement,"click") *** Detailed properties of the highlighted element *** UIA_title:= <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > UIA_text:= <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > UIA_regexptitle:= <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > UIA_class:= <TextBlock> UIA_regexpclass:= <TextBlock> UIA_iaccessiblechildId:= <0> UIA_id:= <Watermark> UIA_handle:= <0> UIA_RuntimeId:= <7;1244;18409771> UIA_BoundingRectangle:= <989;356;406;183> UIA_ProcessId:= <1244> UIA_ControlType:= <50020> UIA_LocalizedControlType:= <text> UIA_Name:= <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > UIA_HasKeyboardFocus:= <False> UIA_IsKeyboardFocusable:= <False> UIA_IsEnabled:= <True> UIA_AutomationId:= <Watermark> UIA_ClassName:= <TextBlock> UIA_ClickablePoint:= <1192;447> UIA_Culture:= <0> UIA_IsControlElement:= <True> UIA_IsContentElement:= <True> UIA_IsPassword:= <False> UIA_NativeWindowHandle:= <0> UIA_IsOffscreen:= <False> UIA_Orientation:= <0> UIA_FrameworkId:= <WPF> UIA_IsRequiredForForm:= <False> UIA_IsDockPatternAvailable:= <False> UIA_IsExpandCollapsePatternAvailable:= <False> UIA_IsGridItemPatternAvailable:= <False> UIA_IsGridPatternAvailable:= <False> UIA_IsInvokePatternAvailable:= <False> UIA_IsMultipleViewPatternAvailable:= <False> UIA_IsRangeValuePatternAvailable:= <False> UIA_IsScrollPatternAvailable:= <False> UIA_IsScrollItemPatternAvailable:= <False> UIA_IsSelectionItemPatternAvailable:= <False> UIA_IsSelectionPatternAvailable:= <False> UIA_IsTablePatternAvailable:= <False> UIA_IsTableItemPatternAvailable:= <False> UIA_IsTextPatternAvailable:= <False> UIA_IsTogglePatternAvailable:= <False> UIA_IsTransformPatternAvailable:= <False> UIA_IsValuePatternAvailable:= <False> UIA_IsWindowPatternAvailable:= <False> UIA_ValueIsReadOnly:= <True> UIA_RangeValueValue:= <0> UIA_RangeValueIsReadOnly:= <True> UIA_RangeValueMinimum:= <0> UIA_RangeValueMaximum:= <0> UIA_RangeValueLargeChange:= <0> UIA_RangeValueSmallChange:= <0> UIA_ScrollHorizontalScrollPercent:= <0> UIA_ScrollHorizontalViewSize:= <100> UIA_ScrollVerticalScrollPercent:= <0> UIA_ScrollVerticalViewSize:= <100> UIA_ScrollHorizontallyScrollable:= <False> UIA_ScrollVerticallyScrollable:= <False> UIA_SelectionCanSelectMultiple:= <False> UIA_SelectionIsSelectionRequired:= <False> UIA_GridRowCount:= <0> UIA_GridColumnCount:= <0> UIA_GridItemRow:= <0> UIA_GridItemColumn:= <0> UIA_GridItemRowSpan:= <1> UIA_GridItemColumnSpan:= <1> UIA_DockDockPosition:= <5> UIA_ExpandCollapseExpandCollapseState:= <3> UIA_MultipleViewCurrentView:= <0> UIA_WindowCanMaximize:= <False> UIA_WindowCanMinimize:= <False> UIA_WindowWindowVisualState:= <0> UIA_WindowWindowInteractionState:= <0> UIA_WindowIsModal:= <False> UIA_WindowIsTopmost:= <False> UIA_SelectionItemIsSelected:= <False> UIA_TableRowOrColumnMajor:= <2> UIA_ToggleToggleState:= <2> UIA_TransformCanMove:= <False> UIA_TransformCanResize:= <False> UIA_TransformCanRotate:= <False> UIA_IsLegacyIAccessiblePatternAvailable:= <True> UIA_LegacyIAccessibleChildId:= <0> UIA_LegacyIAccessibleName:= <INNT Innovate Biopharmaceuticals, Inc Healthcare Sector(D) Biotechnology > UIA_LegacyIAccessibleRole:= <41> UIA_LegacyIAccessibleState:= <0> UIA_IsDataValidForForm:= <False> UIA_ProviderDescription:= <[pid:1244,providerId:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]> UIA_IsItemContainerPatternAvailable:= <False> UIA_IsVirtualizedItemPatternAvailable:= <False> UIA_IsSynchronizedInputPatternAvailable:= <True>

I'm trying to conduct UI automation using AutoIt through use of the UI Automation library provided by @junkew. The reason why I'm using the UI Automation library and not just AutoIt's out of box controls is because the application I'm trying to automate was created using both standard and non-standard windows controls. Some were built in-house which is likely the reason for the difficulty I'm experiencing, but I believe there should be a way to do it because both the spy tool provided by @junkew and the Windows inspect.exe tool can both provide information about individual elements of the Grid, but I'm failing to find a coded solution. I would appreciate help in the right direction. NOTE: I changed the name of the element of interest as well as the Window since the application is not mine to dispense. At this moment I'm trying to solve two problems: Perform some allowed UI Automation action on the element of a Grid with the specified text. Extract all the elements within the Grid for analysis. Similar to the idea of a dump I guess. The simplespy.exe application produces this code, which works to some degree. It puts the Grid in focus which brings the previously clicked row into focus, but it does not select a desired element or row with the text specified. ;~ *** Standard code *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow", $treescope_children) _UIA_Action($oP3,"setfocus") Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer", $treescope_children) _UIA_Action($oP2,"setfocus") Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children) _UIA_Action($oP1,"setfocus") Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children) _UIA_Action($oP0,"setfocus") _UIA_setVar("SalesorderEntry.mainwindow","title:=RowOfInterestText;classname:=GRID") _UIA_action("SalesorderEntry.mainwindow","setfocus") simplespy.exe also produced this information about the focused element: Mouse position is retrieved 2177-480 At least we have an element [RowOfInterestText][GRID] Having the following values for all properties: Title is: <RowOfInterestText> Class := <GRID> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 2037;318;588;466 *** Parent Information top down *** 5: Title is: <Desktop> Class := <#32769> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 0;0;1920;1200 "Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769" 4: Title is: <WindowTitle> Class := <ATL:4476F950> controltype:= <UIA_WindowControlTypeId> ,<50032> , (0000C370) 2020;250;1256;583 "Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950" 3: Title is: <> Class := <WTL_SplitterWindow> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 2028;300;1240;502 "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow" 2: Title is: <> Class := <WTL_PaneContainer> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 2028;300;1240;502 "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer" 1: Title is: <> Class := <#32770> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 2028;300;1240;502 "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770" 0: Title is: <> Class := <#32770> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) 2037;305;1224;479 "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770" *** Detailed properties of the highlighted element *** UIA_title:= <RowOfInterestText> UIA_text:= <RowOfInterestText> UIA_regexptitle:= <RowOfInterestText> UIA_class:= <GRID> UIA_regexpclass:= <GRID> UIA_iaccessiblechildId:= <0> UIA_id:= <249> UIA_handle:= <4526846> UIA_RuntimeId:= <42;4526846> UIA_BoundingRectangle:= <2037;318;588;466> UIA_ProcessId:= <6892> UIA_ControlType:= <50033> UIA_LocalizedControlType:= <pane> UIA_Name:= <RowOfInterestText> UIA_HasKeyboardFocus:= <True> UIA_IsKeyboardFocusable:= <True> UIA_IsEnabled:= <True> UIA_AutomationId:= <249> UIA_ClassName:= <GRID> UIA_Culture:= <0> UIA_IsControlElement:= <True> UIA_IsContentElement:= <True> UIA_IsPassword:= <False> UIA_NativeWindowHandle:= <4526846> UIA_IsOffscreen:= <False> UIA_Orientation:= <0> UIA_FrameworkId:= <Win32> UIA_IsRequiredForForm:= <False> UIA_IsDockPatternAvailable:= <False> UIA_IsExpandCollapsePatternAvailable:= <False> UIA_IsGridItemPatternAvailable:= <False> UIA_IsGridPatternAvailable:= <False> UIA_IsInvokePatternAvailable:= <False> UIA_IsMultipleViewPatternAvailable:= <False> UIA_IsRangeValuePatternAvailable:= <False> UIA_IsScrollPatternAvailable:= <False> UIA_IsScrollItemPatternAvailable:= <False> UIA_IsSelectionItemPatternAvailable:= <False> UIA_IsSelectionPatternAvailable:= <False> UIA_IsTablePatternAvailable:= <False> UIA_IsTableItemPatternAvailable:= <False> UIA_IsTextPatternAvailable:= <False> UIA_IsTogglePatternAvailable:= <False> UIA_IsTransformPatternAvailable:= <False> UIA_IsValuePatternAvailable:= <False> UIA_IsWindowPatternAvailable:= <False> UIA_ValueIsReadOnly:= <True> UIA_RangeValueValue:= <0> UIA_RangeValueIsReadOnly:= <True> UIA_RangeValueMinimum:= <0> UIA_RangeValueMaximum:= <0> UIA_RangeValueLargeChange:= <0> UIA_RangeValueSmallChange:= <0> UIA_ScrollHorizontalScrollPercent:= <0> UIA_ScrollHorizontalViewSize:= <100> UIA_ScrollVerticalScrollPercent:= <0> UIA_ScrollVerticalViewSize:= <100> UIA_ScrollHorizontallyScrollable:= <False> UIA_ScrollVerticallyScrollable:= <False> UIA_SelectionCanSelectMultiple:= <False> UIA_SelectionIsSelectionRequired:= <False> UIA_GridRowCount:= <0> UIA_GridColumnCount:= <0> UIA_GridItemRow:= <0> UIA_GridItemColumn:= <0> UIA_GridItemRowSpan:= <1> UIA_GridItemColumnSpan:= <1> UIA_DockDockPosition:= <5> UIA_ExpandCollapseExpandCollapseState:= <3> UIA_MultipleViewCurrentView:= <0> UIA_WindowCanMaximize:= <False> UIA_WindowCanMinimize:= <False> UIA_WindowWindowVisualState:= <0> UIA_WindowWindowInteractionState:= <0> UIA_WindowIsModal:= <False> UIA_WindowIsTopmost:= <False> UIA_SelectionItemIsSelected:= <False> UIA_TableRowOrColumnMajor:= <2> UIA_ToggleToggleState:= <2> UIA_TransformCanMove:= <False> UIA_TransformCanResize:= <False> UIA_TransformCanRotate:= <False> UIA_IsLegacyIAccessiblePatternAvailable:= <True> UIA_LegacyIAccessibleChildId:= <0> UIA_LegacyIAccessibleName:= <RowOfInterestText> UIA_LegacyIAccessibleRole:= <10> UIA_LegacyIAccessibleState:= <1048580> UIA_IsDataValidForForm:= <False> UIA_ProviderDescription:= <[pid:9208,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]> UIA_IsItemContainerPatternAvailable:= <False> UIA_IsVirtualizedItemPatternAvailable:= <False> UIA_IsSynchronizedInputPatternAvailable:= <False> I experienced success as well with the Windows inspect.exe in identifying the text of the element of interest when I put focus on it: This is what inspect.exe shows: How found: Mouse move (2161,483) hwnd=0x004512FE 32bit class="GRID" style=0x52810371 ex=0x4 RuntimeId: "[42.4526846]" BoundingRectangle: {l:2037 t:318 r:2625 b:784} ProcessId: 6892 ControlType: UIA_PaneControlTypeId (0xC371) LocalizedControlType: "pane" Name: "RowOfInterestText" AccessKey: "" HasKeyboardFocus: true IsKeyboardFocusable: true IsEnabled: true AutomationId: "249" ClassName: "GRID" HelpText: "" IsPassword: false NativeWindowHandle: 0x4512FE IsOffscreen: false FrameworkId: "Win32" ProviderDescription: "[pid:6872,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]" LegacyIAccessible.ChildId: 0 LegacyIAccessible.DefaultAction: "" LegacyIAccessible.Description: "" LegacyIAccessible.Help: "" LegacyIAccessible.KeyboardShortcut: "" LegacyIAccessible.Name: "RowOfInterestText" LegacyIAccessible.Role: client (0xA) LegacyIAccessible.State: focused,focusable (0x100004) LegacyIAccessible.Value: "" IsDockPatternAvailable: false IsExpandCollapsePatternAvailable: false IsGridItemPatternAvailable: false IsGridPatternAvailable: false IsInvokePatternAvailable: false IsLegacyIAccessiblePatternAvailable: true IsMultipleViewPatternAvailable: false IsRangeValuePatternAvailable: false IsScrollPatternAvailable: false IsScrollItemPatternAvailable: false IsSelectionItemPatternAvailable: false IsSelectionPatternAvailable: false IsTablePatternAvailable: false IsTableItemPatternAvailable: false IsTextPatternAvailable: false IsTogglePatternAvailable: false IsTransformPatternAvailable: false IsValuePatternAvailable: false IsWindowPatternAvailable: false IsItemContainerPatternAvailable: false IsVirtualizedItemPatternAvailable: false IsSynchronizedInputPatternAvailable: false FirstChild: [null] LastChild: [null] Next: "Menu" text Previous: [null] Other Props: Object has no additional properties Children: Container has no children Ancestors: "" dialog "" dialog "" pane "" pane "WindowTitle" window "Desktop" pane [ No Parent ]

I wrote this power user tool a long while back and it originally had baked in commands and wasn't very extensible. I rewrote the tool and made use of the power and openeness of AutoIt. Some of you may find the concept and/or UI approach interesting (as well as useful) which is why I'm posting here. In all likely-hood, it could be rewritten entirely in AutoIt but that's not something I plan on doing. Good chance there are similar tools out there as well especially since I've sat on this for so long. In short, the tool provides a snazzy user interface (no two interfaces will look alike) which opens right over whatever your working on so you can quickly launch any number of chained AutoIt scripts. My all-time favorite being; normalize clipboard text. https://winclickpro.codeplex.com/ I haven't shared the tool or concept with anyone outside of publishing it so I'm curious to see what people think.

UDF provide graphical control based on static class. Button Progressbar v1.01. Fix on Syntax Error Checking Download Choose server where you will download: fxControl.zip - code.google.com (renamed from zettoControl_v1.02.zip) fxControl.zip - autoitscript.com The next version of this UDF will named as fxControl (Control Framework). Screenshot Sample Application using zettoControl Z-Compressor [link fixed] Clean PC Clean PC Source available in spoiler below 'Codice Sorgente' text, in that page. but download link: dead. Its still useful because at least the source are here while you able replace image file with what you have. zettoControl_v1.02.zip