Jump to content
FrancescoDiMuro

UIAutomation - Can't get the value of a Text control in a ListItem

Recommended Posts

Good afternoon everyone :)
Third day into learning UIAutomation and I am struggling a bit to read the value of a Text control which is contained by a ListItem control.
I am trying to learn UIAutomation following @LarsJ's UIAutomation tutorial, so I'd be very happy to get some advices without mixing this approach with the @junkew's one.
Below, there are the ListItem and the Text control inspected with UIASpy.

ListItem control:

Spoiler

image.thumb.png.0b989502282782172f21c46970e52d76.png


Text control:

Spoiler

image.thumb.png.903f6320794ae8d1e48f0aaef76ae231.png


This is the code I'm using to read the various Text controls:

Spoiler
#include "CUIAutomation2.au3"
#include <GuiListView.au3>

Opt("MustDeclareVars", 1)

Test()

Func Test()

    Local $oUIAutomation, _
          $pDesktop, _
          $oDesktop, _
          $pWindow, _
          $oWindow, _
          $pCondition, _
          $pPane1, _
          $oPane1, _
          $pPane2, _
          $oPane2, _
          $pListView, _
          $oListView, _
          $pListItems, _
          $oListItems, _
          $iListItems, _
          $pListItem, _
          $oListItem, _
          $pTextControls, _
          $oTextControls, _
          $iTextControls, _
          $pText, _
          $oText, _
          $sText, _
          $oValue, _
          $pValue

    ConsoleWrite(@CRLF & "----- UIAutomation Object -----" & @CRLF)

    ; Create UI Automation object
    $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
    ConsoleWrite("$oUIAutomation OK" & @CRLF)

    ConsoleWrite("----- Desktop -----" & @CRLF)

    ; Get Desktop element
    Global $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement($pDesktop)
    $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
    ConsoleWrite("$oDesktop OK" & @CRLF)

    ConsoleWrite("----- Main Window -----" & @CRLF)

    $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "WindowName", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oDesktop.FindFirst($TreeScope_Descendants, $pCondition, $pWindow)
    $oWindow = ObjCreateInterface($pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oWindow) Then Return ConsoleWrite("$oWindow ERR" & @CRLF)
    ConsoleWrite("$oWindow OK" & @CRLF)

    ConsoleWrite("----- First Pane -----" & @CRLF)

    ; Why these two instructions are not working?
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "AfxControlBar140s", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "59422", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_PaneControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oWindow.FindFirst($TreeScope_Descendants, $pCondition, $pPane1)
    $oPane1 = ObjCreateInterface($pPane1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oPane1) Then Return ConsoleWrite("$oPane1 ERR" & @CRLF)
    ConsoleWrite("$oPane1 OK" & @CRLF)

    ConsoleWrite("----- Second Pane -----" & @CRLF)

    ; Same question of Pane1
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "Afx:00790000:b:00010003:00000000:00000000", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "81", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_PaneControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane1.FindFirst($TreeScope_Descendants, $pCondition, $pPane2)
    $oPane2 = ObjCreateInterface($pPane2, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oPane2) Then Return ConsoleWrite("$oPane2 ERR" & @CRLF)
    ConsoleWrite("$oPane2 OK" & @CRLF)

    ConsoleWrite("----- ListView -----" & @CRLF)

    #cs

    ; The ListView, even if has two parents pane, is not found is the second one, but in the first one

    ; Same question of Pane1
    ; $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "SysListView32", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_AutomationIdPropertyId, "33217", $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_ListControlTypeId, $pCondition)
    ; $oUIAutomation.CreatePropertyCondition($UIA_NativeWindowHandlePropertyId, "0x0005091A", $pCondition)

    $oUIAutomation.CreatePropertyCondition($UIA_RuntimeIdPropertyId, "330010", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane2.FindFirst($TreeScope_Children, $pCondition, $pListView)
    $oListView = ObjCreateInterface($pListView, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oListView) Then Return ConsoleWrite("$oListView ERR" & @CRLF)
    ConsoleWrite("$oListView OK" & @CRLF)

    #ce

    $oUIAutomation.CreatePropertyCondition($UIA_ClassNamePropertyId, "SysListView32", $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oPane1.FindFirst($TreeScope_Children, $pCondition, $pListView)
    $oListView = ObjCreateInterface($pListView, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oListView) Then Return ConsoleWrite("$oListView ERR" & @CRLF)
    ConsoleWrite("$oListView OK" & @CRLF)

    ConsoleWrite("----- List Items -----" & @CRLF)

    $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_ListItemControlTypeId, $pCondition)
    If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
    ConsoleWrite("$pCondition OK" & @CRLF)

    $oListView.FindAll($TreeScope_Children, $pCondition, $pListItems)
    $oListItems = ObjCreateInterface($pListItems, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
    $oListItems.Length($iListItems)
    If Not $iListItems Then Return ConsoleWrite("$oListItems ERR" & @CRLF)
    ConsoleWrite("$oListItems OK" & @CRLF)

    For $i = 0 To $iListItems - 1

        $oListItems.GetElement($i, $pListItem)
        $oListItem = ObjCreateInterface($pListItem, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

        $oUIAutomation.CreatePropertyCondition($UIA_ControlTypePropertyId, $UIA_TextControlTypeId, $pCondition)
        If Not $pCondition Then Return ConsoleWrite("$pCondition ERR" & @CRLF)
        ConsoleWrite("$pCondition OK" & @CRLF)

        ConsoleWrite("----- Text Controls -----" & @CRLF)

        $oListItem.FindAll($TreeScope_Descendants, $pCondition, $pTextControls)
        $oTextControls = ObjCreateInterface($pTextControls, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)
        $oTextControls.Length($iTextControls)
        If Not $iTextControls Then Return ConsoleWrite("$oTextControls ERR" & @CRLF)
        ConsoleWrite("$oTextControls OK" & @CRLF)

        For $j = 0 To $iTextControls - 1

            $oTextControls.GetElement($j, $pText)
            $oText = ObjCreateInterface($pText, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

            ; None of these methods are working
            ; $oText.GetCurrentPropertyValue($UIA_NamePropertyId, $sText)
            ; $oText.GetCurrentPropertyValue($UIA_ValueValuePropertyId, $sText)

            $oText.GetCurrentPattern($UIA_LegacyIAccessiblePatternId, $pValue)
            $oValue = ObjCreateInterface($pValue, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern)
            If Not IsObj($oValue) Then Return ConsoleWrite("$oValue ERR" & @CRLF)
            ConsoleWrite("$oValue OK" & @CRLF)

            ; $oValue.CurrentName($sText)
            $oValue.CurrentValue($sText)

            ConsoleWrite("Value of [" & $i & "," & $j & "] item: " & $sText & @CRLF)
        Next

    Next

ConsoleWrite(@CRLF)

EndFunc

 


This is the output from the SciTE console:

Spoiler
----- UIAutomation Object -----
$oUIAutomation OK
----- Desktop -----
$oDesktop OK
----- Main Window -----
$pCondition OK
$oWindow OK
----- First Pane -----
$pCondition OK
$oPane1 OK
----- Second Pane -----
$pCondition OK
$oPane2 OK
----- ListView -----
$pCondition OK
$oListView OK
----- List Items -----
$pCondition OK
$oListItems OK
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [0,0] item: 
$oValue OK
Value of [0,1] item: 
$oValue OK
Value of [0,2] item: 
$oValue OK
Value of [0,3] item: 
$oValue OK
Value of [0,4] item: 
$oValue OK
Value of [0,5] item: 
$oValue OK
Value of [0,6] item: 
$oValue OK
Value of [0,7] item: 
$oValue OK
Value of [0,8] item: 
$oValue OK
Value of [0,9] item: 
$oValue OK
Value of [0,10] item: 
$oValue OK
Value of [0,11] item: 
$oValue OK
Value of [0,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [1,0] item: 
$oValue OK
Value of [1,1] item: 
$oValue OK
Value of [1,2] item: 
$oValue OK
Value of [1,3] item: 
$oValue OK
Value of [1,4] item: 
$oValue OK
Value of [1,5] item: 
$oValue OK
Value of [1,6] item: 
$oValue OK
Value of [1,7] item: 
$oValue OK
Value of [1,8] item: 
$oValue OK
Value of [1,9] item: 
$oValue OK
Value of [1,10] item: 
$oValue OK
Value of [1,11] item: 
$oValue OK
Value of [1,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [2,0] item: 
$oValue OK
Value of [2,1] item: 
$oValue OK
Value of [2,2] item: 
$oValue OK
Value of [2,3] item: 
$oValue OK
Value of [2,4] item: 
$oValue OK
Value of [2,5] item: 
$oValue OK
Value of [2,6] item: 
$oValue OK
Value of [2,7] item: 
$oValue OK
Value of [2,8] item: 
$oValue OK
Value of [2,9] item: 
$oValue OK
Value of [2,10] item: 
$oValue OK
Value of [2,11] item: 
$oValue OK
Value of [2,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [3,0] item: 
$oValue OK
Value of [3,1] item: 
$oValue OK
Value of [3,2] item: 
$oValue OK
Value of [3,3] item: 
$oValue OK
Value of [3,4] item: 
$oValue OK
Value of [3,5] item: 
$oValue OK
Value of [3,6] item: 
$oValue OK
Value of [3,7] item: 
$oValue OK
Value of [3,8] item: 
$oValue OK
Value of [3,9] item: 
$oValue OK
Value of [3,10] item: 
$oValue OK
Value of [3,11] item: 
$oValue OK
Value of [3,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [4,0] item: 
$oValue OK
Value of [4,1] item: 
$oValue OK
Value of [4,2] item: 
$oValue OK
Value of [4,3] item: 
$oValue OK
Value of [4,4] item: 
$oValue OK
Value of [4,5] item: 
$oValue OK
Value of [4,6] item: 
$oValue OK
Value of [4,7] item: 
$oValue OK
Value of [4,8] item: 
$oValue OK
Value of [4,9] item: 
$oValue OK
Value of [4,10] item: 
$oValue OK
Value of [4,11] item: 
$oValue OK
Value of [4,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [5,0] item: 
$oValue OK
Value of [5,1] item: 
$oValue OK
Value of [5,2] item: 
$oValue OK
Value of [5,3] item: 
$oValue OK
Value of [5,4] item: 
$oValue OK
Value of [5,5] item: 
$oValue OK
Value of [5,6] item: 
$oValue OK
Value of [5,7] item: 
$oValue OK
Value of [5,8] item: 
$oValue OK
Value of [5,9] item: 
$oValue OK
Value of [5,10] item: 
$oValue OK
Value of [5,11] item: 
$oValue OK
Value of [5,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [6,0] item: 
$oValue OK
Value of [6,1] item: 
$oValue OK
Value of [6,2] item: 
$oValue OK
Value of [6,3] item: 
$oValue OK
Value of [6,4] item: 
$oValue OK
Value of [6,5] item: 
$oValue OK
Value of [6,6] item: 
$oValue OK
Value of [6,7] item: 
$oValue OK
Value of [6,8] item: 
$oValue OK
Value of [6,9] item: 
$oValue OK
Value of [6,10] item: 
$oValue OK
Value of [6,11] item: 
$oValue OK
Value of [6,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [7,0] item: 
$oValue OK
Value of [7,1] item: 
$oValue OK
Value of [7,2] item: 
$oValue OK
Value of [7,3] item: 
$oValue OK
Value of [7,4] item: 
$oValue OK
Value of [7,5] item: 
$oValue OK
Value of [7,6] item: 
$oValue OK
Value of [7,7] item: 
$oValue OK
Value of [7,8] item: 
$oValue OK
Value of [7,9] item: 
$oValue OK
Value of [7,10] item: 
$oValue OK
Value of [7,11] item: 
$oValue OK
Value of [7,12] item: 
$pCondition OK
----- Text Controls -----
$oTextControls OK
$oValue OK
Value of [8,0] item: 
$oValue OK
Value of [8,1] item: 
$oValue OK
Value of [8,2] item: 
$oValue OK
Value of [8,3] item: 
$oValue OK
Value of [8,4] item: 
$oValue OK
Value of [8,5] item: 
$oValue OK
Value of [8,6] item: 
$oValue OK
Value of [8,7] item: 
$oValue OK
Value of [8,8] item: 
$oValue OK
Value of [8,9] item: 
$oValue OK
Value of [8,10] item: 
$oValue OK
Value of [8,11] item: 
$oValue OK
Value of [8,12] item:

 

As you can see, I posted some questions in the code itself because I tried different methods to do such task.
I already tried with _GUICtrlListView_GetItemText* and ControlListView() functions once "arrived" to the ListView control, but none of them worked.
Could you please direct me to the right approach to help me achieve this?
Thank you in advance.

Cheers ^_^


Click here to see my signature:

Spoiler

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites
Posted (edited)

Good to see that someone is at least trying.

When you cannot see the text value in UIASpy, I don't think you can create code to read the value yourself. This is probably because the control doesn't have adequate UI Automation support.

And since your own experiments with classic code do not work either, it seems that the control completely lacks automation code.

You can try with MSAA code in Inspect.exe. Switch to MSAA mode through the dropdown control in the upper left corner.

If that does not work either, OCR techniques are probably the only solution.

Edited by LarsJ
OCR techniques

Share this post


Link to post
Share on other sites

@LarsJ
Thanks for bumping in!
I had to download and install the Windows SDK because the link provided was broken; so, I seize the opportunity to upload it here.
I tried inspecting the Control using MSAA mode, and these are the results:

ListView control:

Spoiler

image.thumb.png.43bcc5beed475a1e8a96eee197eecf6d.png


ListItem control:

Spoiler

image.thumb.png.6d77fd656621349f3cf25cfb0f676904.png

I was looking at the ListItem control (guessing that "Voce di elenco" is a ListItem and "Elenco" is a ListView), the Value is null, and so, I don't know if I can retrieve any value from this ListView (probably not).
Just for curiosity, I inspected the same controls with UIAutomation mode, and these are the results:

ListView control:

Spoiler

image.thumb.png.674d7ace588c698fd888cf2382aa6361.png


ListItem control:

Spoiler

image.thumb.png.9decc5e13d463ad6fe93459848d46e0d.png


Text control:

Spoiler

image.thumb.png.34a1507941c18b45d11b9bed4484b66b.png

I'd like to thank you once again for the amazing tool you made, which is very simple to use and it gives you a lot of information and creates code out of the box.

Cheers ^_^


Click here to see my signature:

Spoiler

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites

You should still be able to use OCR techniques (there are many examples of using Tesseract) to obtain the value of the text control.

Implement the following functionality in a script:

  • Get the bounding rectangle of the text control with UIA code
  • Make a screenshot of this rectangle and save it as an image
  • Use Tesseract to read the text in the image
  • Validate the output text from Tesseract

I myself have used this technique successfully a few times.

Share this post


Link to post
Share on other sites

@LarsJ

Thank you for you help once again.

After some days of searching, I l've seen that the software has its own scripting language to interact with the data displayed in the controls, and so, I'm gonna study it since it's C++, and I always wanted to re-use it since I have studied it in high school.

Thanks :)


Click here to see my signature:

Spoiler

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...