Sn3akyP3t3

Introspection and Actions; UIA_PaneControlTypeId and A Grid Within

3 posts in this topic

I'm trying to conduct UI automation using AutoIt through use of the UI Automation library provided by @junkew.  The reason why I'm using the UI Automation library and not just AutoIt's out of box controls is because the application I'm trying to automate was created using both standard and non-standard windows controls.  Some were built in-house which is likely the reason for the difficulty I'm experiencing, but I believe there should be a way to do it because both the spy tool provided by @junkew and the Windows inspect.exe tool can both provide information about individual elements of the Grid, but I'm failing to find a coded solution.  I would appreciate help in the right direction.

NOTE: I changed the name of the element of interest as well as the Window since the application is not mine to dispense.

At this moment I'm trying to solve two problems:

  1. Perform some allowed UI Automation action on the element of a Grid with the specified text.
  2. Extract all the elements within the Grid for analysis.  Similar to the idea of a dump I guess.

The simplespy.exe application produces this code, which works to some degree.  It puts the Grid in focus which brings the previously clicked row into focus, but it does not select a desired element or row with the text specified.

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow", $treescope_children)   
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer", $treescope_children)    
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770", $treescope_children)   
_UIA_Action($oP0,"setfocus")
_UIA_setVar("SalesorderEntry.mainwindow","title:=RowOfInterestText;classname:=GRID")
_UIA_action("SalesorderEntry.mainwindow","setfocus") 

simplespy.exe also produced this information about the focused element:

Mouse position is retrieved 2177-480
At least we have an element [RowOfInterestText][GRID]

Having the following values for all properties: 
Title is: <RowOfInterestText>   Class   := <GRID>   controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;318;588;466
*** Parent Information top down ***
5: Title is: <Desktop>  Class   := <#32769> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    0;0;1920;1200
"Title:=Desktop;controltype:=UIA_PaneControlTypeId;class:=#32769"   
4: Title is: <WindowTitle>  Class   := <ATL:4476F950>   controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    2020;250;1256;583
"Title:=WindowTitle;controltype:=UIA_WindowControlTypeId;class:=ATL:4476F950"   
3: Title is: <> Class   := <WTL_SplitterWindow> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_SplitterWindow"  
2: Title is: <> Class   := <WTL_PaneContainer>  controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WTL_PaneContainer"   
1: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2028;300;1240;502
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"  
0: Title is: <> Class   := <#32770> controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    2037;305;1224;479
"Title:=;controltype:=UIA_PaneControlTypeId;class:=#32770"
*** Detailed properties of the highlighted element ***
UIA_title:= <RowOfInterestText>
UIA_text:= <RowOfInterestText>
UIA_regexptitle:= <RowOfInterestText>
UIA_class:= <GRID>
UIA_regexpclass:= <GRID>
UIA_iaccessiblechildId:= <0>
UIA_id:= <249>
UIA_handle:= <4526846>
UIA_RuntimeId:= <42;4526846>
UIA_BoundingRectangle:= <2037;318;588;466>
UIA_ProcessId:= <6892>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_Name:= <RowOfInterestText>
UIA_HasKeyboardFocus:= <True>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_AutomationId:= <249>
UIA_ClassName:= <GRID>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <4526846>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Win32>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <RowOfInterestText>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048580>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:9208,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False> 

I experienced success as well with the Windows inspect.exe in identifying the text of the element of interest when I put focus on it:

This is what inspect.exe shows:
How found:  Mouse move (2161,483)
hwnd=0x004512FE 32bit class="GRID" style=0x52810371 ex=0x4
RuntimeId:  "[42.4526846]"
BoundingRectangle:  {l:2037 t:318 r:2625 b:784}
ProcessId:  6892
ControlType:    UIA_PaneControlTypeId (0xC371)
LocalizedControlType:   "pane"
Name:   "RowOfInterestText"
AccessKey:  ""
HasKeyboardFocus:   true
IsKeyboardFocusable:    true
IsEnabled:  true
AutomationId:   "249"
ClassName:  "GRID"
HelpText:   ""
IsPassword: false
NativeWindowHandle: 0x4512FE
IsOffscreen:    false
FrameworkId:    "Win32"
ProviderDescription:    "[pid:6872,hwnd:0x4512FE Main:Nested [pid:6892,hwnd:0x4512FE Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Nonclient:Microsoft: Non-Client Proxy (unmanaged:uiautomationcore.dll); Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]"
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    ""
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "RowOfInterestText"
LegacyIAccessible.Role: client (0xA)
LegacyIAccessible.State:    focused,focusable (0x100004)
LegacyIAccessible.Value:    ""
IsDockPatternAvailable: false
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsRangeValuePatternAvailable:   false
IsScrollPatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsTablePatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTextPatternAvailable: false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsValuePatternAvailable:    false
IsWindowPatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsSynchronizedInputPatternAvailable:    false
FirstChild: [null]
LastChild:  [null]
Next:   "Menu" text
Previous:   [null]
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "" dialog
    "" dialog
    "" pane
    "" pane
    "WindowTitle" window
    "Desktop" pane
    [ No Parent ]

Share this post


Link to post
Share on other sites



@junkew Thanks.  Looks like there is no tree like structure with child items to work with.  I may have to use some other techniques then such as Sikuli, if that's accurate enough with the OpenCV OCR portion, for this part.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • TheSaint
      By TheSaint
      TeraCopy Timer - A program of mine, front end really, that I have been working on (on & off) for some time, but not shared here before, that I recall. Third party program TeraCopy is required.
      Many of you will be familiar with TeraCopy, and it has been discussed here at AF on several occasions, in various sub forums, including Chat.
      BE ADVISED - I still use an older version of TeraCopy (v2.27), a goody but an oldy, so have only tested my program with that. It may work with the newer v3.xx, or may be easily adjusted to do so, but I have not yet gone that route. Likewise, I only use it with Win 7 and Win XP, but I don't imagine there should be any issues with newer Windows. Add to that, my program being devised and created in AutoIt v3.3.0.0.
      So what is TeraCopy Timer and why might you need it?
      I devised my program for two main reasons really, based on giving me greater control, using the Free version of TeraCopy, which has some limitations (perhaps even with the paid full version). My program utilizes the features of the TeraCopy command-line.
      1. I was fed up with Thumbs.db files regularly halting the process in Win XP or preventing a folder being deleted after a move. Only happens with Win XP.
      2. I wanted to automate delays between jobs, allowing all HDD's to rest periodically when doing large and lengthy jobs. Letting HDD's heat up too much, can have tragic results or considerably shorten their lifespan. Prevention is better than cure or just relying on monitoring software etc.
      --------------------------------------------------------------------------------------
      Those familiar with TeraCopy use, know you can run multiple copy or move jobs either simultaneously or concurrently (one immediately after another has finished). With the free version at least, you have no control over precise order or timing, and no way to deal (without manual interaction) with the pesky Thumbs.db files in Win XP. TeraCopy Timer allows you to do all those things, and in a batch fashion.
      --------------------------------------------------------------------------------------
      Some screenshots and brief explanation
      Main Window

      You can browse to set the Source and Destination paths, or like me, just use Drag & Drop to those inputs.
      Once the Destination path has been set, the MIN (minimalist GUI) button becomes available. In Minimal mode, you get a further level of automation, once initial options are set, and thus less prompts ... none in fact, as jobs are created automatically based on either COPY or MOVE.
      Minimalist Window

      In the Minimalist mode, you can only use Drag & Drop to add a source file or folder, and COPY or MOVE is permanently set when the window first opens, via a choice prompt.
      Assigning a WAIT is simple and easy, and is the latest feature added to the program ... I used a much more complex variant before that (see Advanced Delay).

      The program now displays three file size reports.
      Individual Size of the current (last added) job (Blue label).
      Total Size of all jobs combined (Black label).
      Subtotal Size for each grouping of jobs, defined by a WAIT selection (Red label).
      You toggle between Black and Red, by just clicking that label. If no WAIT has been set, then the values will be the same.
      The wait of 5 minutes in the screenshot above is set for the source shown. It means wait 5 minutes before copying (or moving) that source.
      More sources added from that point, add to a new subtotal. To see the previous subtotal, you need to be at the main (MAX) window, and select the prior job.
      So every time a WAIT is clicked for a source a new subtotal count is started.

      In the screenshot above, you can see the Job name, and that it is Job number 5 order wise. MOVE has been set for that job.
      Selecting Job 4, you will see the previous (complete) Subtotal. Selecting job 3 (in this instance), you would see the subtotal up to that job.
      Click the red Total label and it will change to Black, and show you the total size over all, as shown in the second screenshot above.
      This new WAIT feature is the simplest and best way to use the program generally (in my view) ... but check out the following, as it is not always the case.
      Advanced Delay Window

      The Advanced Delay Options, are a further level of automation, added during the early stages of development, before I thought to create the Minimalist window and show Sizes. It was before I decided to put a lot more effort into the program. As with all my programs though, it is continual use that eventually dictates what I ultimately want to happen, to make my life easier ... simpler, better, smarter, quicker.
      The main difference between this older method, and the new WAIT one, where you specify delays precisely, is that the program attempts to determine the best moments to pause, based on various factors, which you setup and can vary between different types of Job sessions.
      The chief purpose for all the advanced options, is an attempt to cater for the difference between moving a small number of big files and a lot of small files. Moving or Copying lots of small files (hundreds or thousands), as many would know, can heat up a HDD far quicker and to a much higher temperature than a small number of (even very) large files. I guess that is primarily due to the sheer number of indexes that need to be created, and with small files are done at a staggering rate.
      Which method you use, is up to you, and should be governed by the type of job. Both methods can be used together, but not advised if you don't want the possibility of unnecessary extra long delays. EDIT - That said, you could use the WAIT option just for a delayed start of the first job ... perhaps your PC is busy doing something else until then, but you want to go and do something else for a bit, and have it all done by the time you return (i.e. watch a movie).
      WARNING - It should go without saying, that you take all the usual precautions. For instance, don't use move with the only copy you have of precious files. BACKUP BACKUP BACKUP! Don't hold me liable for anything, except for being a nice guy who likes to share.
      TeraCopy Timer v2.0.zip
      Enjoy!
      P.S. I am not affiliated in any way with those who created and provide the excellent third party program - TeraCopy.
    • Earthshine
      By Earthshine
      This is pretty useful to me at least, I hope to others. Whenever you need to poke a button or such on a GUI, you can just make calls to this UDF and instruct it how long to wait, 0 is forever, anyway, let the code to the explaining. Feel free to help expand this UDF. I plan to create new ones as I go along that support all the Win32 and .NET Controls. Ultimately, I would like to be able to fully control any type of control, this works well with most. Just an example of what you can do.
       
      The test installs and can uninstall 7-Zip 15.14 (x64) Setup.exe
      NOTE: Logger Author(s) .....: Michael Mims (zorphnog)
      ;                    the logging script can be obtained here
      ;                   https://www.autoitscript.com/forum/topic/156196-log4a-a-logging-udf/
       
       
      install_7zip_test.au3
      WaitForControls.au3
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good morning everyone
      I worked a bit on a little project of mine, and I managed to do what I was trying to do, or, automate a quiz in AutoIt.
      Now, everything works fine but, in order to achieve what I'm trying to achive, I'd like to make this script run faster.
      I'll attach a portion of the code in order to let you understand what my script does.
      $objIE_h1 = _IETagNameGetCollection($objIE, "h1") ; Take the text of the answer If @error Then ConsoleWrite("Errore durante la lettura degli h1 nella pagina. - Errore: " & @error & @CRLF) Else For $objElement In $objIE_h1 $strInnerText = $objElement.innerText Next Select Case StringInStr($strInnerText, "QUESTION") ; Here is the question ; PS4 PRO $objSpans = _IETagNameGetCollection($objIE, "span") ; Answers are stored here For $objElement In $objSpans If $i >= 4 Then $strAnswer = $objElement.innerText ; Take the text of the answer n° 0, 1, 2, 3 If StringInStr($strAnswer, "CORRECT ANSWER") Then ; If I match the correct answer, then, I have to click on the input with answer id = answer_i $j = $i - 4 $objInput = _IEGetObjById($objIE, "answer_" & $j) If @error Then ConsoleWrite("Errore durante la lettura della risposta #" & $i & " - Errore: " & @error & @CRLF) Else _IEAction($objInput, "click") If @error Then ConsoleWrite("Errore durante il click sull'elemento. - Errore: " & @error) Else ConsoleWrite($j & ") Risposta: " & $strAnswer & @CRLF) ExitLoop EndIf EndIf EndIf $i+=1 Else $i+=1 EndIf Next $i = 0 $j = 0 This is what my script does.
      Questions are 20, in which I have 15 multiple choice question, 2 images ( on which I use _IEImgClick() ), and 3 text inputs ( a simple send with the text I have to enter ).
      How can I make all of these, faster?
      If I reach this goal, I've probably done one of my best goals in life, and I'd like to share my happiness with you, giving a huge donation to this fantastic community!
      Thanks for your help
    • Psyllex
      By Psyllex
      Is it possible for AutoIt to execute LALT+LSHIFT+NUMLOCK?  I've attempted the following:
       
      ;To show the desktop Send("{RWIN down}") Send("{d}") Send("{RWIN up}") WinWait(2) ;just to make sure the desktop is activated. ControlClick("", "",1[,button="left"[, clicks=2 [, x [, y]]]] ) Send("{LALT down}") Send("{CLT down}") Send("{DELETE down}") ;Send("{LSHIFT down}") ;Send("{NUMLOCK}") ;Send("{NUMLOCK}") ;Send ("{LALT up}") ;Send("{LSHIFT up}") This should bring up the MouseKeys dialog...but I'm getting nothing.  Thanks for any help!
    • WalkHappy
      By WalkHappy
      Hello, I'm in the process of recreation-ally expanding my Auto-it experience for some future awesome presentation-Al capabilities. So I have a user interface which loads a text document onto a GUI-object. The variables of this function are 1) The text document is dynamic in font size and face; giving the viewer an unpredictable horizontal/vertical scroll dimension. 2) The main objective is to animate the vertical scroll-bar so that it moves like the end of a movie with accreditation. 3) I'm trying to limit the accessibility in observation by excluding any mouse functions within the animation process. Any help would be greatly appreciated! Been trying to work on this for days now - from getting element properties, to embedding external code, and even exploiting ("$oIE.navigate") to loading several awkwardly formatted text documents in sequential order just to imitate a slightly offensive animation scheme. Anyways... Suggestions and comments welcomed please!!
      Script is from [Function GUICreate] - Doc:
      GUICreate("Embedded Web control Test", 640, 580, (@DesktopWidth - 640) / 2, (@DesktopHeight - 580) / 2, BitOR($WS_OVERLAPPEDWINDOW, $WS_CLIPSIBLINGS, $WS_CLIPCHILDREN)) GUICtrlCreateObj($oIE, 10, 40, 600, 360) $idButton_Back = GUICtrlCreateButton("Back", 10, 420, 100, 30) $idButton_Forward = GUICtrlCreateButton("Forward", 120, 420, 100, 30) $idButton_Home = GUICtrlCreateButton("Home", 230, 420, 100, 30) $idButton_Stop = GUICtrlCreateButton("Stop", 330, 420, 100, 30) GUISetState(@SW_SHOW) ;Show GUI After this.. This is the window that pop's-up: