Jump to content
Sign in to follow this  
Sn3akyP3t3

Introspection and Actions; UIA_PaneControlTypeId and A Grid Within

Recommended Posts

I'm trying to conduct UI automation using AutoIt through use of the UI Automation library provided by @junkew.  The reason why I'm using the UI Automation library and not just AutoIt's out of box controls is because the application I'm trying to automate was created using both standard and non-standard windows controls.  Some were built in-house which is likely the reason for the difficulty I'm experiencing, but I believe there should be a way to do it because both the spy tool provided by @junkew and the Windows inspect.exe tool can both provide information about individual elements of the Grid, but I'm failing to find a coded solution.  I would appreciate help in the right direction.

NOTE: I changed the name of the element of interest as well as the Window since the application is not mine to dispense.

At this moment I'm trying to solve two problems:

  1. Perform some allowed UI Automation action on the element of a Grid with the specified text.
  2. Extract all the elements within the Grid for analysis.  Similar to the idea of a dump I guess.

The simplespy.exe application produces this code, which works to some degree.  It puts the Grid in focus which brings the previously clicked row into focus, but it does not select a desired element or row with the text specified.

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow", $treescope_children)   
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer", $treescope_children)    
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP0,"setfocus")
_UIA_setVar("SalesorderEntry.mainwindow","title:=RowOfInterestText;classname:=GRID")
_UIA_action("SalesorderEntry.mainwindow","setfocus") 

simplespy.exe also produced this information about the focused element:

Mouse position is retrieved 2177-480
At least we have an element [RowOfInterestText][GRID]

Having the following values for all properties: 
Title is: <RowOfInterestText>   Class   := <GRID>   controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;318;588;466
*** Parent Information top down ***
5: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1200
"Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769"   
4: Title is: <WindowTitle>  Class   := <ATL:4476F950>   controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    2020;250;1256;583
"Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950"   
3: Title is: <> Class   := <WTL_SplitterWindow> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow"  
2: Title is: <> Class   := <WTL_PaneContainer>  controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer"   
1: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"  
0: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;305;1224;479
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"
*** Detailed properties of the highlighted element ***
UIA_title:= <RowOfInterestText>
UIA_text:= <RowOfInterestText>
UIA_regexptitle:= <RowOfInterestText>
UIA_class:= <GRID>
UIA_regexpclass:= <GRID>
UIA_iaccessiblechildId:= <0>
UIA_id:= <249>
UIA_handle:= <4526846>
UIA_RuntimeId:= <42;4526846>
UIA_BoundingRectangle:= <2037;318;588;466>
; Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False> 

I experienced success as well with the Windows inspect.exe in identifying the text of the element of interest when I put focus on it:

This is what inspect.exe shows:
How found:  Mouse move (2161,483)
hwnd=0x004512FE 32bit class="GRID" style=0x52810371 ex=0x4
RuntimeId:  "[42.4526846]"
BoundingRectangle:  {l:2037 t:318 r:2625 b:784}
ProcessId:  6892
ControlType:    UIA_PaneControlTypeId (0xC371)
LocalizedControlType:   "pane"
Name:   "RowOfInterestText"
AccessKey:  ""
HasKeyboardFocus:   true
IsKeyboardFocusable:    true
IsEnabled:  true
AutomationId:   "249"
ClassName:  "GRID"
HelpText:   ""
IsPassword: false
NativeWindowHandle: 0x4512FE
IsOffscreen:    false
FrameworkId:    "Win32"
ProviderDescription:    "[pid:6872,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]"
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    ""
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "RowOfInterestText"
LegacyIAccessible.Role: client (0xA)
LegacyIAccessible.State:    focused,focusable (0x100004)
LegacyIAccessible.Value:    ""
IsDockPatternAvailable: false
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsRangeValuePatternAvailable:   false
IsScrollPatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsTablePatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTextPatternAvailable: false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsValuePatternAvailable:    false
IsWindowPatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsSynchronizedInputPatternAvailable:    false
FirstChild: [null]
LastChild:  [null]
Next:   "Menu" text
Previous:   [null]
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "" dialog
    "" dialog
    "" pane
    "" pane
    "WindowTitle" window
    "Desktop" pane
    [ No Parent ]

Share this post


Link to post
Share on other sites

@junkew Thanks.  Looks like there is no tree like structure with child items to work with.  I may have to use some other techniques then such as Sikuli, if that's accurate enough with the OpenCV OCR portion, for this part.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By WilliamasKumeliukas
      Hi,
      I started this project alone in May 2020 as project in my spare time at work, I'm working for a IT compagny that's started opening their services to residential customers few months ago and now my position in the company kinda drifted in the doom and gloom world of repetitive tasks like: Reinstallation of Windows 10.
      The procedure is very repetitive and I started feeling like being a robot which is the main reason I started this project.
       
      ==============================FAQ==================================
      1. Q: Do you want this project to be accomplished with the usage of AutoIT ONLY or 3rd party tools / Scripts (BAT / CMD / VBS) ? A: No, if I cannot find a way using AutoIT to accomplish a task I will move to my Plan B which consist of automating an 3rd party tool to accomplish the affected task until a solution is found. 2. Q: What do I get from helping/collaborating in this project? A: I will personally take the responsability to mention you in the credits of this project. 3. Q: If I have more questions, can I ask? A: Certainly! feel free to ask any questions related to this project! 4. Q: What is the main goal of this project? A: Automating Windows 10 configuration without user interaction needed (as much as possible) ______________________________________________________________________________________________________________________________
      Current progression of the project (more might be added in future)
      « Blue = Info || Yellow = Unfinished/Untested || Purple = Could be better || Green = Done ||Red = Not Yet Started »
      *** Very early Stage ***
      Connect Network Attached Storage(NAS) (Tested & Working - AutoIT only)
      Install Adobe Reader DC, 7zip, Google Chrome (Tested & Working - using 3rd party tool)
      Change OEM Information (Tested & Working - contains both AutoIT version and CMD script)
      Disable ScreenSaver (Require testing AutoIT only)
      Change Computer Name (Require testing AutoIT only) 
      Change Default Browser (Work - using 3rd party tool)
      Change .pdf / .pdfxml from Edge to Adobe Reader DC (AutoIT only)
      Change Edge to Google Chrome as Default Browser (Tested & Working - using 3rd party tool)
      Windows Updates (Currently working on it - AutoIT only)
      Install Office 
      Activate Office (Tested on 2010 & Working - CMD Script)
      Pin to taskbar Excel, Outlook, Word, Chrome (Tested & Working - using 3rd party tool)
      GUI currently featuring the following options 
      - TreeView to check case of desired tasks
      -  Redirect ConsoleWrite with timestamp messages in richedit (errors shown in red text)
                 *** To Do ***
       - Read informations needed for tasks like: OEM, ComputerName, NAS user/password etc... from a file
       - ListView for Windows Updates
       - Probably more
       
      ***  If this project interest you, Reply here This will greatly help me to see if you'd like this project to become real  ***
      ______________________________________________________________________________________________________________________________
      2020.09.21 Update: Since this topic reached over 200+ views without a single interest shown by community, pm me and i'll share with you W10 Configurator script  
       
      Best Regards,
      ~WilliamasKumeliukas
    • By WindIt
      Hi, 
      I have a set of excel docs 1& 2
      1) In Excel 1 i have a fixed column called dimensionId that contains list of dimensionid's with in between empty cells.
      Here, For each dimesionId, i need to check corresponding columns of L,M,N ,O(senior, mid, junior, student) & check If they contain letter Y (Open excel 2) and if empty ignore,check next.
      I need to loop through the entire column of dimensionId one by one, please help me proceed with the logic.
      2) If letter Y exists, open excel 2 and check if same dimesionId exists in column A.
      New to autoit excel automation, Help me with logic to execute this.
      #include <Excel.au3>
      Global $oExcel = _Excel_Open()
      Global $oWorkbook = _Excel_BookOpen($oExcel, "D:\yta\Trial concept dimensions list.xlsx")
      Local $read1 = _Excel_RangeRead ($oWorkbook,Default,"C3")
      MsgBox(0,"",$read1)
      Attached exce 1 doc.
       
       
      exl1.xlsx exl2.xlsx
    • By Earthshine
      This is pretty useful to me at least, I hope to others. Whenever you need to poke a button or such on a GUI, you can just make calls to this UDF and instruct it how long to wait, 0 is forever, anyway, let the code to the explaining. Feel free to help expand this UDF. I plan to create new ones as I go along that support all the Win32 and .NET Controls. Ultimately, I would like to be able to fully control any type of control, this works well with most. Just an example of what you can do.
      NOTE: Logger Author(s) .....: Michael Mims (zorphnog)
      ;                    the logging script can be obtained here
      ;                   https://www.autoitscript.com/forum/topic/156196-log4a-a-logging-udf/
       
       
       
       
       
       
       
      WaitForControls.au3 log4a.au3
    • By Se7enstars
      New Modern Design UI in AutoIt in 118 line
      Please watch & like in Youtube: Link
      Full Source Code free: Link
       

×
×
  • Create New...