Jump to content
WSorich

Determine text in another applications window

Recommended Posts

WSorich

Hi,

I am trying to automate a series of tedious data exporting steps. I have gone ahead and created steps for sending key strokes and clicking controls etc, but I get to a particular screen with a list which has an almost random number of tables listed.

I'd like to be able to query the list of items in the window presented by the application so I can figure out how many lines I need to move through the list before selecting.

The window (screenshot attached) is some sort of scrollable list view, where the format is like this...

(Tickbox) Item information blah blah blah (Tickbox) new item name blah blah

When I use the info tool, I can get a handle, but I cant seem to use any function to read any text info from the window.

Any hints on approaches on how to read the data out of this window, (with a short sample code) would be appreciated.

Cheers.

post-42837-0-18343900-1331707356_thumb.j

Edited by WSorich

Share this post


Link to post
Share on other sites
WSorich

Just thought of something, Just wanted to add that I have used AU3Info to try and get info on the window tick boxes or list items,

but the only thing that AU3info identifies is the handle of the outside of the window frame.

Is it possible to pull apart the window handle further to get to the line items within the list? Does anyone know how to do this?

Cheers.

Share this post


Link to post
Share on other sites
JohnOne

Am I correct in assuming window info tool will give you a handle to the window, but not a handle or specific information

on its controls, such as the one you are trying to get info on?

If so, then it is most likely the case, that this window does not use standard windows controls, and basically

means you can not get the info you want in a standard clean and safe fashion using Autoit3.


AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites
WSorich

Thanks for your reply John.

I have added the "Summary" information from AU3Info below, as I'm not 100% sure if your thinking the same as I am as to what information I can get.

Between your comments and the information below, I can see that there is some "Control" information, but I dont know how to extract anything from this.

I have tried using functions like

ControlSend($ImpWizWinTitle,"","[NAME:gridTables]", "{DOWN 20}")

And this allows me to move the cursor in the list, but I have no idea on how to extract the information at the cursor location to see what I'm pointing to.

>>>> Window <<<<
Title: Import and Export Wizard
Class: WindowsForms10.Window.8.app.0.141b42a_r15_ad1
Position: 557, 58
Size: 566, 579
Style: 0x16CF0000
ExStyle: 0x00050100
Handle: 0x0000000000040824
>>>> Control <<<<
Class: WindowsForms10.Window.8.app.0.141b42a_r15_ad1
Instance: 8
ClassnameNN: WindowsForms10.Window.8.app.0.141b42a_r15_ad18
Name: gridTables
Advanced (Class): [NAME:gridTables]
ID: 329926
Text:
Position: 16, 86
Size: 518, 364
ControlClick Coords: 206, 301
Style: 0x56210000
ExStyle: 0x00000200
Handle: 0x00000000000508C6
>>>> Mouse <<<<
Position: 787, 475
Cursor ID: 0
Color: 0xFFFFFF
>>>> StatusBar <<<<
>>>> ToolsBar <<<<
>>>> Visible Text <<<<
Choose one or more tables and views to copy.
Select Source Tables and Views
&Edit Mappings...
&Preview...
&Tables and views:
&Help
&Finish >>|
&Next >
< &Back
Cancel

>>>> Hidden Text <<<<
0
Edited by WSorich

Share this post


Link to post
Share on other sites
WSorich

Agreed that the alternate thread seems to be similar.

A couple of things that I would like to know though,

1. Is there a way to determine the "type" of listbox / combo box / grid I am dealing with here?

2. Isnt there a standard way that windows requires application developers to create a window structure

3. Once the window is created, isnt there a standard way that windows requires the structure to appear so it can be interpreted and displayed?

4. Cant we grab the window handle and determine where the information in that window is?

5. Couldnt we then "read" that structure and grab the text?

Edited by WSorich

Share this post


Link to post
Share on other sites
WSorich

Well, there has been some progress towards being able to read the text being displayed in the screen I attached in my first post.

I have come across a tool that has helped me identify more information about the window where the data is presented.

The tool was written to replace UISpy from Microsoft. The new tool is called "UIAVerify2.0".

When I use this tool, I can drill down into the window and actually see the data in the row and column I need.

All I need to do now, is decode what the data is telling me to a way I can access it in AutoIT.

The sequence of images I have attached here show;

1. Visual UI - Grid control info.jpg

- Top level at which I can determine information through AU3Info, and for which I can also get from AutoIT commands.

2. Visual UI - Grid control - Column #2 info.jpg

- Next level down where I cannot determine how to get information

3. Visual UI.jpg

- Lowest level, indicating the actual data I need to read. Cannot determine how to retrieve even though UIAVerify seems to find it easily.

Any help would be appreciated.

post-42837-0-47590100-1332299112_thumb.j

post-42837-0-42798200-1332299441_thumb.j

post-42837-0-44424400-1332299457_thumb.j

Share this post


Link to post
Share on other sites
Ruud

Hello WSorich,

It seems you're more persistent than I am :oops:

I tried the UIAVerify2.0-Tool you mentioned, and get similar results - the text I need is found and shown in the left panel.

I thought I'd try something crazy: to use autoit to read the text from UIAVerify instead of from the actual target window.

Using AU3Info and pointing it to UIAVerify, however, the text I read in the left panel is also not shown in AU3Info. The text is selectable in UIAVerify, but not copyable either, so it seems that workaround is a dead end.

------

edit:

by pointing AU3info at the right panel of UIAVerify, I can get to the text. So I could use AutoIt to alternately (1) control the target application, and (2) read the needed information from UIAVerify.

It's cumbersome, but works. Help with finding a more direct way is still appreciated.

-------

Cheers,

Ruud

Edited by Ruud

Share this post


Link to post
Share on other sites
WSorich

Lol. Now that is something I didn't think of, but for my purposes, I am trying to construct a stand alone application for this task.

Yes, I am persistent, as I am curious as to how these things work. It would seem that I don't have enough knowledge to break through to a complete solution, but I'll keep putting it out there in the hopes that another small piece of the puzzle is filled in by somebody and we can sort this out.

My current question is about the screenshot from UIA verify.

1. How can I use AutoIT to go from either a window handle or control handle down to

the next level in the tree, and then the next etc as is shown by UIA Verify.

Cheers, Wal.

Edited by WSorich

Share this post


Link to post
Share on other sites
Ruud

That'd be my question too.

Still, I cannot believe that we'd be the first with such an issue - there must have been heaps of people with that question before.

So - where are they and how do we get them to answer :oops::bye:

Edited by Ruud

Share this post


Link to post
Share on other sites
JohnOne

You are correct, there have been many, I suspect they accepted that Autoit simply cannot do this with

none standard windows controls most of the time.

If it were me, I'd be trying to figure out how this "UIA verify" does it, rather than why autoit cannot.

I've always been of the thought that you would need to use the language the control was developed in

which is probably not c++.


AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites
WSorich

Looking at the UIAVerify package, it seems like this may have UI automation functions that you can call, and that the front end that I was using is a way to test/retrieve the information needed to access the window controls/information.

This seems much like the information that AU3Info provides for you to use AutoITs functions to access the windows controls etc.

So it looks like it would be possible to use the UIAverify interface to the windows information from within an AutoIT script.

Here are the list of files in the UIAverify package.

Interop.UIAutomationClient.dll
IQueryString.dll
SampleDriver.exe
UIAComWrapper.dll
uialoggerxml.dll
UIALogging.dll
uiverify.config
VisualUIAVerify.exe
WUIATestLibrary.dll

I'll start investigating how we can call functions in the UIAverify package to retrieve information using AutoIT... without having to actually run UIAverify..

This is the same idea, but a more direct approach than Ruud initially suggested.

Share this post


Link to post
Share on other sites
Ruud

Good work - keep us posted! :oops:

Share this post


Link to post
Share on other sites
Ruud

I tried viewing the functions inside these dll's (using DLL export viewer) - but no functions were shown in any of the files.

Now admittedly I'm new to dll-calls, but I have been able to successfully interface with the from python (I tried porting my project to python and access AutoIt from there). Then again, DLL export viewer DOES show the functions when pointing it to FastFind.dll.

Does anyone know what to make of this?

Share this post


Link to post
Share on other sites
Ruud

@Autolaser: Danke fuer den Hinweis!

@WSorich: In a german-language AutoIt forum, a program ("dotNet DLL-Exporter") was posted to convert VB.Net or C# DLL files to C++ DLL files that can be used with AutoIt; it is post #20 under Autolaser's link. Seems helpful - if not here then at least somewhere down the road. MS Visual Studio is needed btw for it to work.

To both:

I am not sure if this conversion is needed on the files that we have here; even though dotNet DLL-Exporter showed some functions in the original DLL files (see screenshot 1), after there are still 0 functions after the conversion in each DLL file according to DLL export viewer (screenshot2). Below is a part of the *.il-file for the particular DLL i was working on here; all other files are in the attached ZIP, for those that are interested.

I guess I'll try to play around a bit more, see if I can use the classes defined in these files, etc.

Thanks so far!

---------uiacomwrapper.il

//  Microsoft (R) .NET Framework IL Disassembler.  Version 3.5.30729.1
//  Copyright (c) Microsoft Corporation.  All rights reserved.



// Metadata version: v2.0.50727
.module extern UIAutomationCore.dll
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                       // .zV.4..
  .ver 2:0:0:0
}
.assembly extern Interop.UIAutomationClient
{
  .ver 1:0:0:0
}
.assembly extern WindowsBase
{
  .publickeytoken = (31 BF 38 56 AD 36 4E 35 )                       // 1.8V.6N5
  .ver 3:0:0:0
}
.assembly extern System
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                       // .zV.4..
  .ver 2:0:0:0
}
.assembly extern Accessibility
{
  .publickeytoken = (B0 3F 5F 7F 11 D5 0A 3A )                       // .?_....:
  .ver 2:0:0:0
}
.assembly UIAComWrapper
{
  .custom instance void [mscorlib]System.Runtime.InteropServices.GuidAttribute::.ctor(string) = ( 01 00 24 37 34 66 32 64 62 33 64 2D 31 35 36 61   // ..$74f2db3d-156a
                                                                                                  2D 34 31 36 30 2D 61 34 30 31 2D 30 31 31 66 36   // -4160-a401-011f6
                                                                                                  30 31 65 31 62 35 36 00 00 )                    // 01e1b56..
  .custom instance void [mscorlib]System.Runtime.CompilerServices.InternalsVisibleToAttribute::.ctor(string) = ( 01 00 12 55 49 41 43 6F 6D 57 72 61 70 70 65 72   // ...UIAComWrapper
                                                                                                                 54 65 73 74 73 00 00 )                         // Tests..
  .custom instance void [mscorlib]System.Runtime.InteropServices.ComVisibleAttribute::.ctor(bool) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyFileVersionAttribute::.ctor(string) = ( 01 00 07 31 2E 30 2E 30 2E 30 00 00 )            // ...1.0.0.0..

  // --- The following custom attribute is added automatically, do not uncomment -------
  //  .custom instance void [mscorlib]System.Diagnostics.DebuggableAttribute::.ctor(valuetype [mscorlib]System.Diagnostics.DebuggableAttribute/DebuggingModes) = ( 01 00 07 01 00 00 00 00 )

  .custom instance void [mscorlib]System.Runtime.CompilerServices.CompilationRelaxationsAttribute::.ctor(int32) = ( 01 00 08 00 00 00 00 00 )
  .custom instance void [mscorlib]System.Runtime.CompilerServices.RuntimeCompatibilityAttribute::.ctor() = ( 01 00 01 00 54 02 16 57 72 61 70 4E 6F 6E 45 78   // ....T..WrapNonEx
                                                                                                             63 65 70 74 69 6F 6E 54 68 72 6F 77 73 01 )       // ceptionThrows.
  .custom instance void [mscorlib]System.Reflection.AssemblyTitleAttribute::.ctor(string) = ( 01 00 0D 55 49 41 43 6F 6D 57 72 61 70 70 65 72   // ...UIAComWrapper
                                                                                              00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyDescriptionAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyConfigurationAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyCompanyAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyProductAttribute::.ctor(string) = ( 01 00 0D 55 49 41 43 6F 6D 57 72 61 70 70 65 72   // ...UIAComWrapper
                                                                                                00 00 )
  .custom instance void [mscorlib]System.Reflection.AssemblyCopyrightAttribute::.ctor(string) = ( 01 00 28 43 6F 70 79 72 69 67 68 74 20 C2 A9 20   // ..(Copyright ..
                                                                                                  20 32 30 30 39 20 4D 69 63 72 6F 73 6F 66 74 20   //  2009 Microsoft
                                                                                                  43 6F 72 70 6F 72 61 74 69 6F 6E 00 00 )        // Corporation..
  .custom instance void [mscorlib]System.Reflection.AssemblyTrademarkAttribute::.ctor(string) = ( 01 00 00 00 00 )
  .hash algorithm 0x00008004
  .ver 1:0:0:0
}
.module UIAComWrapper.dll
// MVID: {49AEADE1-0284-4B66-869E-F9F24A7E9E7D}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003      // WINDOWS_CUI
.corflags 0x00000002

.vtfixup [1] int32 fromunmanaged at VT_01
.data VT_01 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_02
.data VT_02 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_03
.data VT_03 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_04
.data VT_04 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_05
.data VT_05 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_06
.data VT_06 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_07
.data VT_07 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_08
.data VT_08 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_09
.data VT_09 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_10
.data VT_10 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_11
.data VT_11 = int32(0)

.vtfixup [1] int32 fromunmanaged at VT_12
.data VT_12 = int32(0)

    //  ILONLY
// Image base: 0x003A0000


// =============== CLASS MEMBERS DECLARATION ===================

.class public auto ansi beforefieldinit System.Windows.Automation.BasePattern
       extends [mscorlib]System.Object
{
  .field assembly class System.Windows.Automation.AutomationElement _el
  .field assembly bool _cached
  .method assembly hidebysig specialname rtspecialname
          instance void  .ctor(class System.Windows.Automation.AutomationElement el,
                               bool cached) cil managed
  {
    // Code size       37 (0x25)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call     instance void [mscorlib]System.Object::.ctor()
    IL_0006:  nop
    IL_0007:  nop
    IL_0008:  ldarg.1
    IL_0009:  ldnull
    IL_000a:  call     bool System.Windows.Automation.AutomationElement::op_Inequality(class System.Windows.Automation.AutomationElement,
                                                                                         class System.Windows.Automation.AutomationElement)
    IL_000f:  call     void [System]System.Diagnostics.Debug::Assert(bool)
    IL_0014:  nop
    IL_0015:  ldarg.0
    IL_0016:  ldarg.1
    IL_0017:  stfld   class System.Windows.Automation.AutomationElement System.Windows.Automation.BasePattern::_el
    IL_001c:  ldarg.0
    IL_001d:  ldarg.2
    IL_001e:  stfld   bool System.Windows.Automation.BasePattern::_cached
    IL_0023:  nop
    IL_0024:  ret
  } // end of method BasePattern::.ctor

} // end of class System.Windows.Automation.BasePattern

.class public auto ansi beforefieldinit System.Windows.Automation.GridPattern
       extends System.Windows.Automation.BasePattern
{
  .class sequential ansi sealed nested public beforefieldinit GridPatternInformation
         extends [mscorlib]System.ValueType
  {
    .field private class System.Windows.Automation.AutomationElement _el
    .field private bool _isCached
    .method assembly hidebysig specialname rtspecialname
            instance void  .ctor(class System.Windows.Automation.AutomationElement element,
                                 bool isCached) cil managed
    {
      // Code size     16 (0x10)
      .maxstack  8
      IL_0000:  nop
      IL_0001:  ldarg.0


(...)

post-70958-0-61707100-1333302517_thumb.j

post-70958-0-24875600-1333302518_thumb.j

UIAVerify_Dlls_after.7z

UIAVerify_Dlls_before.7z

Edited by Ruud

Share this post


Link to post
Share on other sites
WSorich

Hi, I've been on a break in the past few days, but I have the following link for UIAVerify which I will explore to see if there is additional information available.

http://uiautomationverify.codeplex.com/

I noticed that there is a link which has a document "HowToUseUIAVerify" and there seems to be some VB code also, so maybe that will help some.

Edited by WSorich

Share this post


Link to post
Share on other sites
Ruud

I've been trying several things in the last days but the DLLs are not opening up to me I'm afraid. I'd love to hear about your progress, if you're having any ;)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×