UIA, is it possible to get all text from a window?

[eatfishdk]

I work with "Microsoft Dynamics NAV 2018" and often need to get data out to various places.

With the normal Autoit commands I can get some of the data out, but not stuff in tables, non edits fields etc.
With the various UIA spy tools I can see that the text is there but can't figure out how to get it all out to a string or an array.

I basically want to activate a window and press a button and all the text is read out (into a string or array), kinda like the autoit info tool does it..

Is it possible at all to read through the entire tree and get all, say the:
$UIA_ValueValuePropertyId, $UIA_LegacyIAccessibleNamePropertyId, $UIA_LegacyIAccessibleValuePropertyId
values...

I don't need to interact in any way (click buttons etc.) in the window, just get the text..
Can anybody guide me or maybe have a simple example because the UIA info in the various threads, I'm sorry to say, goes right over my head...

 

Regards, Martin

[junkew]

If you have luck you can type in your addressbar

javascript:alert(document.body.innerText)

"I'm sorry to say, goes right over my head..."

Nice moment to start with example 1-10 in the examples section on IUIAutomation.

The alternative is the WebDriver examples but also that will have a steep learning curve.

 

[eatfishdk]

Hi, thanks for the reply.

The app is a windows application and not browser based..

I've tried the various examples and they work of course, but I cant really follow..
Is there an example of just reading text, not pushing buttons or any thing? I think i've looked at all of them but might have missed something..

"Normal" Autoit code I can usually figure out, but been away from it for some years and due to personal issues the more complicated stuff is hard for me to grasp these days..

[junkew]

1. Study with inspect.exe or other spy tools referenced in FAQ 31 your tree and see if its worth the effort.
   If all data is there you can walk the tree in AutoIt in different ways
2. Best file to study: ex1_Treewalker.au3
3. especiall findthemall function is then usefull to start with
4. If ordering is important you probably have to create the walking over the tree yourself which can be hard an challenging certainly when you have to filter out unwanted information.

As you are saying tables it still can be done but gets harder as then you have to navigate the cells by row and column which is most likely possible.

Its possible when its shown to the spying tools but can be hard as UIA is a big model to understand.

 

[mLipok]

1 hour ago, eatfishdk said:

The app is a windows application and not browser based.

You could try:

WinGetText()

or the examples/solutions mentioned by @junkew or ....

Try this new UDF:

 

Edited January 10, 2022 by mLipok

[eatfishdk]

15 hours ago, junkew said:

1. Study with inspect.exe or other spy tools referenced in FAQ 31 your tree and see if its worth the effort.
   If all data is there you can walk the tree in AutoIt in different ways
2. Best file to study: ex1_Treewalker.au3
3. especiall findthemall function is then usefull to start with
4. If ordering is important you probably have to create the walking over the tree yourself which can be hard an challenging certainly when you have to filter out unwanted information.

As you are saying tables it still can be done but gets harder as then you have to navigate the cells by row and column which is most likely possible.

Its possible when its shown to the spying tools but can be hard as UIA is a big model to understand.

All the data (text) I need is visible to the spy tools, so it's there. I'm not worried about getting to much data as that is relatively easy to manage if I just get a string or array out with the help of UIA...
But I'll try the different examples and see if I can figure it out.

Thanks again for the reply.

 

 

[eatfishdk]

13 hours ago, mLipok said:

You could try:

WinGetText()

or the examples/solutions mentioned by @junkew or ....

Try this new UDF:

 

Wingettext and the other methods in "plain" autoit doesn't give me all the text, just a small subset..
I'll might take a look at the udf you linked, but a lot of the text is hidden because the size of the window so need to scroll to see it all.. 

But thanks for the suggestion!

 

[Werty]

Have you tried CTRL-A then CTRL-C ?

Edited January 11, 2022 by Werty

[eatfishdk]

9 minutes ago, Werty said:

Have you tried CTRL-A then CTRL-C ?

Erhm.. Its a joke right?

The text I need are in boxes like this:

If copy/paste was possible, I've wouldn't have looked at UIA... 

And no, you can't right click on these fields and copy... But they are available through the UIA spy tools..

 

[junkew]

Here you have the findthemall function which you can tweak to get more/less properties printed.

• Below example just dumps name and class
  • you can just extend this line with your properites 

    ConsoleWrite(
    "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB 
    & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB
    & "Value=" & _UIA_getPropertyValue($oUIElement, $UIA_ValueValuePropertyId) & @TAB
    & "Accessible name=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleNamePropertyId) & @TAB
    & "Accessible value=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleValuePropertyId) & @TAB
    & @CRLF)

     

• With UIA_DumpThemAll you get all properties which is overkill so you have to tweak
• When the class or controltype is different it could be you have to look for other properties to get the text thats relevant for you

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        
;~          if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="<your object of interest name>" then
;~              $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
;~              _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
;~              consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)           
;~          endif
        
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

[eatfishdk]

Just now, junkew said:

Here you have the findthemall function which you can tweak to get more/less properties printed.

• Below example just dumps name and class
  • you can just extend this line with your properites 

    ConsoleWrite(
    "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB 
    & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB
    & "Value=" & _UIA_getPropertyValue($oUIElement, $UIA_ValueValuePropertyId) & @TAB
    & "Accessible name=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleNamePropertyId) & @TAB
    & "Accessible value=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleValuePropertyId) & @TAB
    & @CRLF)

     

• With UIA_DumpThemAll you get all properties which is overkill so you have to tweak
• When the class or controltype is different it could be you have to look for other properties to get the text thats relevant for you

Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        
;~          if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="<your object of interest name>" then
;~              $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
;~              _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
;~              consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)           
;~          endif
        
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

Thank you very much, I'll have a look at it later today!

 

[eatfishdk]

@junkew Just wanted to say thank you for getting me started!

I've finally got some code that dumps all the stuff I need from the program.
Now I just need to sort the stuff I need to get out and make it a bit faster! 

Again! Thanks alot!'

Regards, Martin

[junkew]

Speed is probably due to the true condition which basically means return all objects which can be a huge array

$UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

These conditions you can make with and, or logic.

See below a filter on ControlType and Name. Probably in your scenario I would filter on controltype and or condition (createOrCondition)

Local $pCondition1
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_MenuItemControlTypeId, $pCondition1 )
  If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
  ConsoleWrite( "$pCondition1 OK" & @CRLF )

  Local $pCondition2 ; $UIA_NamePropertyId is LOCALIZED and maybe CASE SENSITIVE
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "File", $pCondition2 ) ; File <<<<<<<<<<<<<<<<<<<<
  If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pCondition2 OK" & @CRLF )

  ; And condition
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF

That's where I frequently use the findThemAll function for to tweak the filters.

The alternative for findall are the treewalkers where you can fully walk the element tree yourself but it depends on what you feel is most easiest to handle.

References

• See for example the different examples 1-25 that try to explain all the details

 

• @LarsJmade some wonderfull example threads and some tooling like UIASpy 

See for example

 

[eatfishdk]

Hi

I've made some good progress and are getting the text I need.
I'm going to look into those conditions when I have some more time, had some issues getting them to work right but thats me not really getting it 🙂
Looking at the posts/examples right now.

But I got it from 8-9 seconds to about one (theres a ton of stuff in the ui..) so thats pretty good for right now!

Again thanks alot @junkew for the help!

 

[mLipok]

On 1/11/2022 at 12:48 PM, junkew said:

Func findThemAll($oElementStart, $TreeScope)

What should be passed as $TreeScope ?

[mLipok]

9 minutes ago, mLipok said:

What should be passed as $TreeScope ?

I think I got it please anybody confirm that one of this following const should be used

; enum TreeScope
Global Const $TreeScope_Element     =  1
Global Const $TreeScope_Children    =  2
Global Const $TreeScope_Descendants =  4
Global Const $TreeScope_Subtree     =  7
Global Const $TreeScope_Parent      =  8
Global Const $TreeScope_Ancestors   = 16

 

Edited November 19, 2023 by mLipok

[junkew]

https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/ne-uiautomationclient-treescope

Its all based on microsoft Windows sdk header and idl files.