Jump to content

UIA, is it possible to get all text from a window?


Recommended Posts

I work with "Microsoft Dynamics NAV 2018" and often need to get data out to various places.

With the normal Autoit commands I can get some of the data out, but not stuff in tables, non edits fields etc.
With the various UIA spy tools I can see that the text is there but can't figure out how to get it all out to a string or an array.

I basically want to activate a window and press a button and all the text is read out (into a string or array), kinda like the autoit info tool does it..

Is it possible at all to read through the entire tree and get all, say the:
$UIA_ValueValuePropertyId, $UIA_LegacyIAccessibleNamePropertyId, $UIA_LegacyIAccessibleValuePropertyId
values...

I don't need to interact in any way (click buttons etc.) in the window, just get the text..
Can anybody guide me or maybe have a simple example because the UIA info in the various threads, I'm sorry to say, goes right over my head...

 

Regards, Martin

Link to post
Share on other sites

If you have luck you can type in your addressbar

javascript:alert(document.body.innerText)

"I'm sorry to say, goes right over my head..."

Nice moment to start with example 1-10 in the examples section on IUIAutomation.

The alternative is the WebDriver examples but also that will have a steep learning curve.

 

Link to post
Share on other sites

Hi, thanks for the reply.

The app is a windows application and not browser based..

I've tried the various examples and they work of course, but I cant really follow..
Is there an example of just reading text, not pushing buttons or any thing? I think i've looked at all of them but might have missed something..

"Normal" Autoit code I can usually figure out, but been away from it for some years and due to personal issues the more complicated stuff is hard for me to grasp these days..

Link to post
Share on other sites
  1. Study with inspect.exe or other spy tools referenced in FAQ 31 your tree and see if its worth the effort.
    If all data is there you can walk the tree in AutoIt in different ways
  2. Best file to study: ex1_Treewalker.au3
  3. especiall findthemall function is then usefull to start with
  4. If ordering is important you probably have to create the walking over the tree yourself which can be hard an challenging certainly when you have to filter out unwanted information.

As you are saying tables it still can be done but gets harder as then you have to navigate the cells by row and column which is most likely possible.

Its possible when its shown to the spying tools but can be hard as UIA is a big model to understand.

 

Link to post
Share on other sites
1 hour ago, eatfishdk said:

The app is a windows application and not browser based.

You could try:

WinGetText()

or the examples/solutions mentioned by @junkew or ....

Try this new UDF:

 

Edited by mLipok

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * for other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE *

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2021-10-18

Link to post
Share on other sites
15 hours ago, junkew said:
  1. Study with inspect.exe or other spy tools referenced in FAQ 31 your tree and see if its worth the effort.
    If all data is there you can walk the tree in AutoIt in different ways
  2. Best file to study: ex1_Treewalker.au3
  3. especiall findthemall function is then usefull to start with
  4. If ordering is important you probably have to create the walking over the tree yourself which can be hard an challenging certainly when you have to filter out unwanted information.

As you are saying tables it still can be done but gets harder as then you have to navigate the cells by row and column which is most likely possible.

Its possible when its shown to the spying tools but can be hard as UIA is a big model to understand.

All the data (text) I need is visible to the spy tools, so it's there. I'm not worried about getting to much data as that is relatively easy to manage if I just get a string or array out with the help of UIA...
But I'll try the different examples and see if I can figure it out.

Thanks again for the reply.

 

 

Link to post
Share on other sites
13 hours ago, mLipok said:

You could try:

WinGetText()

or the examples/solutions mentioned by @junkew or ....

Try this new UDF:

 

Wingettext and the other methods in "plain" autoit doesn't give me all the text, just a small subset..
I'll might take a look at the udf you linked, but a lot of the text is hidden because the size of the window so need to scroll to see it all.. 

But thanks for the suggestion!

 

Link to post
Share on other sites
9 minutes ago, Werty said:

Have you tried CTRL-A then CTRL-C ?

Erhm.. Its a joke right?

The text I need are in boxes like this:
image.png.f72aecd9379c31381d84e25183a97d8d.png

If copy/paste was possible, I've wouldn't have looked at UIA... 

And no, you can't right click on these fields and copy... But they are available through the UIA spy tools..

 

Link to post
Share on other sites

Here you have the findthemall function which you can tweak to get more/less properties printed.

  • Below example just dumps name and class
    • you can just extend this line with your properites 
      ConsoleWrite(
      "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB 
      & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB
      & "Value=" & _UIA_getPropertyValue($oUIElement, $UIA_ValueValuePropertyId) & @TAB
      & "Accessible name=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleNamePropertyId) & @TAB
      & "Accessible value=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleValuePropertyId) & @TAB
      & @CRLF)

       

  • With UIA_DumpThemAll you get all properties which is overkill so you have to tweak
  • When the class or controltype is different it could be you have to look for other properties to get the text thats relevant for you
Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        
;~          if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="<your object of interest name>" then
;~              $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
;~              _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
;~              consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)           
;~          endif
        
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

Link to post
Share on other sites
Just now, junkew said:

Here you have the findthemall function which you can tweak to get more/less properties printed.

  • Below example just dumps name and class
    • you can just extend this line with your properites 
      ConsoleWrite(
      "Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB 
      & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @TAB
      & "Value=" & _UIA_getPropertyValue($oUIElement, $UIA_ValueValuePropertyId) & @TAB
      & "Accessible name=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleNamePropertyId) & @TAB
      & "Accessible value=" & _UIA_getPropertyValue($oUIElement, $UIA_LegacyIAccessibleValuePropertyId) & @TAB
      & @CRLF)

       

  • With UIA_DumpThemAll you get all properties which is overkill so you have to tweak
  • When the class or controltype is different it could be you have to look for other properties to get the text thats relevant for you
Func findThemAll($oElementStart, $TreeScope)
    Local $hTimer = TimerInit()
;~  Get result with findall function alternative could be the treewalker
    Dim $pCondition, $pTrueCondition
    Dim $pElements, $iLength

    $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

    $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray)

    $oAutomationElementArray.Length($iLength)
    For $i = 0 To $iLength - 1; it's zero based
        $oAutomationElementArray.GetElement($i, $UIA_pUIElement)
        $oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        ConsoleWrite("Title is: " & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & @TAB & "Class=" & _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid) & @CRLF)
        
;~          if _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)="<your object of interest name>" then
;~              $t = StringSplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId), ";")
;~              _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])
;~              consolewrite(_UIA_getAllPropertyValues($UIA_oUIElement) & @CRLF)           
;~          endif
        
    Next

    Local $fDiff = TimerDiff($hTimer)
    Consolewrite("Findthemall took: " & $fDiff & " milliseconds" & @CRLF & @CRLF)

EndFunc   ;==>findThemAll

 

Thank you very much, I'll have a look at it later today!

 

Link to post
Share on other sites

@junkew Just wanted to say thank you for getting me started!

I've finally got some code that dumps all the stuff I need from the program.
Now I just need to sort the stuff I need to get out and make it a bit faster! 

Again! Thanks alot!'

Regards, Martin

Link to post
Share on other sites

Speed is probably due to the true condition which basically means return all objects which can be a huge array

$UIA_oUIAutomation.CreateTrueCondition($pTrueCondition)
    $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition)
;~  $oCondition1 = _AutoItObject_WrapperCreate($aCall[1], $dtagIUIAutomationCondition)
;~ Tricky to search all descendants on html objects or from desktop/root element
    $oElementStart.FindAll($TreeScope, $oCondition, $pElements)

These conditions you can make with and, or logic.

See below a filter on ControlType and Name. Probably in your scenario I would filter on controltype and or condition (createOrCondition)

Local $pCondition1
  $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_MenuItemControlTypeId, $pCondition1 )
  If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
  ConsoleWrite( "$pCondition1 OK" & @CRLF )

  Local $pCondition2 ; $UIA_NamePropertyId is LOCALIZED and maybe CASE SENSITIVE
  $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "File", $pCondition2 ) ; File <<<<<<<<<<<<<<<<<<<<
  If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
  ConsoleWrite( "$pCondition2 OK" & @CRLF )

  ; And condition
  $oUIAutomation.CreateAndCondition( $pCondition1, $pCondition2, $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF

That's where I frequently use the findThemAll function for to tweak the filters.

The alternative for findall are the treewalkers where you can fully walk the element tree yourself but it depends on what you feel is most easiest to handle.

References

  • See for example the different examples 1-25 that try to explain all the details

 

  • @LarsJmade some wonderfull example threads and some tooling like UIASpy 

See for example

 

Link to post
Share on other sites

Hi

I've made some good progress and are getting the text I need.
I'm going to look into those conditions when I have some more time, had some issues getting them to work right but thats me not really getting it 🙂
Looking at the posts/examples right now.

But I got it from 8-9 seconds to about one (theres a ton of stuff in the ui..) so thats pretty good for right now!

Again thanks alot @junkew for the help!

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...