Jump to content

UIASpy - UI Automation Spy Tool

Recommended Posts

At @LarsJ:      You're great! Thank you so much.

Now I have 2 1 questions:

13 hours ago, LarsJ said:

The problem is that there are two visitor number controls that you can see when printing the entire tree structure:

How to print the entire tree structure?

Just found the answer here myself. The search button is your friend. 🙂

13 hours ago, LarsJ said:

And you need the second control to read the text

How could I detect that there are two controls and that I should use the second one without looking at the entire tree structure first? 
How did you find out?

Edited by Exit
I answered the first question myself.

App: Au3toCmd              UDF: _SingleScript()                             

Link to post
Share on other sites
  • Replies 108
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

The GUI looks like shown in the picture: It contains a main menu, a treeview in left pane that lists UI Automation elements, a listview in right pane that displays detail information for the

faustf, I'm pretty sure your life will be much easier if you find yourself a job where you can use your hands. And neither on a mouse nor a keyboard.

Sample code creation through Sample code main menu   The Sample code main menu is used for Sample code creation. Much more code can be created through the menu than through the Detail i

Posted Images

Now I've worked a bit in UIASpy and I'm really excited.
@LarsJ made a great tool. :yes:

Now I have a few comments on the handling of the function keys.
The keys should not be handled as hotkeys but as accelerator keys.
At least if you use common keys like F1-F7.
These keys are used in many applications.
For example, the key F1, which is used in SciTe to call syntax help.
In order to use them, you always have to switch off the 'detect element' function in the UIASpy menu.

Now I have found the solution for me:
Just use the function keys in shifted mode.
Simply add a plus sign (+) at the beginning of the file 'UIASpy_Menu.au3' for the hotkey definitions.
So the keys F1-F7 are again available for other applications.

Global $aFuncKeys = [ [ "+{F1}", "UIASpy_F1" ], _ ; F-key, Function
                      [ "+{F2}", "UIASpy_F2" ], _
                      [ "+{F3}", "UIASpy_F3" ], _
                      [ "+{F4}", "UIASpy_F4" ], _
                      [ "+{F7}", "UIASpy_F7" ], _
                      [ "+{F8}", "UIASpy_F8" ] ]

If someone wants to copy it, just should do it. 

Thanks again to @LarsJ :thumbsup:

App: Au3toCmd              UDF: _SingleScript()                             

Link to post
Share on other sites

got an error when trying to run the UIASpy.au3

I just downloaded it today.

line 3787 looks like this:
Local $aElemIndexes = ($aElems[$iIdx][7])[1], $aIndex, $jCnt = 0

here is from the console:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3" /UserParams    
+>14:03:15 Starting AutoIt3Wrapper v.18.708.1148.0 SciTE v.   Keyboard:00010409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (  from:C:\Program Files (x86)\AutoIt3  input:C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3
+>14:03:16 AU3Check ended.rc:0
>Running:(\Program Files (x86)\AutoIt3\autoit3.exe "C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\Includes\UIASpy_ElemInfo.au3" (3787) : ==> Subscript used on non-accessible variable.:
Local $aElemIndexes = ($aElems[$iIdx][7])[1], $aIndex, $jCnt = 0
Local $aElemIndexes = ($aElems[$iIdx][7])^ ERROR
->14:03:17 AutoIt3.exe ended.rc:1
+>14:03:17 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 2.566

Link to post
Share on other sites
  • 3 months later...

i just went through the codes and i realise that you did not use third party control functions like controlgettext(). 

May i know how do you get properties of child control?


I tried it and it was able to detect controls that autoit info is not able to such as power builder data window.

Link to post
Share on other sites
  • 1 month later...

LarsJ, another great tool of yours!

Here in our company we are using a terminal program called "Mocha TN5250 for Windows 7/8/10" from https://mochasoft.dk/ . Currently I retrieve terminal text using ControlSend() - which works 95%. It is somehow "sensible"; e. g. it won't work if - for whatever reason - window lost focus.

I'm not able grab any information with AutoIt Window Info Tool and even your tool can't see this control/part of the GUI.

Do you have ever heard about "invisible" controls? And do you have a clue how to address such controls?

Edited by supersonic
Link to post
Share on other sites

If you are in control of the installation check if the activex control of mocha is there then you could use objcreate function. If not search for ehllapi terminal dll. If you have no development background easiest will be screenscraping with controlsend and winactivate commands. Its hard to help with terminal emulators as there are many around. 



Link to post
Share on other sites

junkew, thank you.

Yes, I am in control of the installation/system but Mocha ActiveX is not (and will not be) available, unfortunately. The ActiveX control is also not able to grab text from running sessions initiated by the standard Mocha client. And as far as I understand EHLLAPI it also isn't able to grab session text but I can build my own terminal application... Well, that's somehow beyond the goal 😊

So, I will stick with Win*() and Control*() functions - that's good enough for now...

But... If anyone else have a clue why there could be a hidden/invisible control and how to grab text from it... I would be very pleased...

Edited by supersonic
Link to post
Share on other sites

I think these terminals/consoles are simply windows and not controls. The text in the windows is written through the window's device context (hdc) with GDI functions like DrawText. That's why they can't be automated. There are a few posts on the same topic here.

Link to post
Share on other sites

Building your own terminal emulator is not that hard just long strings send back and forth to a mainframe with some bytes representing colors or fields so it depends on your goals.

Hooking drawtext is an option but hard to interpret and control.

I have never seen a terminal emulator that has no option to grab to text on clipboard. Whats the macro language of mocha offering?

I remember some open source terminal emulator is around on github but do not remember details.

Link to post
Share on other sites

@LarsJ - Sorry for bothering you again: Maybe it is a GDI window (I don't know how to evaluate that) but it is funny that rows/cols can be selected by mouse (or all text by Ctrl+A). Is this also typical for GDI windows?

@junkew - Thank you for motivating me to code my own terminal emulation. 😊 Here in our company Mocha terminal _must_ be used and unfortunately there is no way around it. I can live with that. Of course, Mocha can grab text (by keyboard, mouse and menue, macro [= disabled for all users]). My aim was to automate text grabbing (and any further processing) by a litte AutoIt script (that actually fully works for a couple of month now). All I wish is to make text grabbing more reliable.

Link to post
Share on other sites

You can easily manipulate and edit an image and extract text from an image. Rows and columns in an image are just a certain number of pixels in the horizontal and vertical directions.
With classic automation code you can automate this set of controls and with UI automation code you can automate this set of controls. There are no such controls as console or terminal controls.

Link to post
Share on other sites


Shows you can copy paste as text or csv

1. WinActivate the mocha window main window

2. Click xy counted from xy of the window to be in the terminal area

3. Send home to be at terminal position 0 0

4. Send shift arrow down 24 times and shift arrow right 80 or thru menu send select all or just ctrl a

5. Send copy either thru menu or with shortcut key

6. Clipboard functions AutoIt should be able to get it from the keyboard.

Doing 1 and 2 should be 100% reliable to make sure you are in terminal area.

Just post simplespy output based on that we can help you probably a little further at least there is a mainwindow child of desktop.

Link to post
Share on other sites

Seems to be a full GDI black area with drawtext happening.

Control is not detectable so just calculate position based on position info of main info and show that highlighted area and activate it with a mouseclick

Example (long time not played with an as400 or other IBM mainframe  ;-))

  1. Download and install mocha tn5250
  2. Connect to www.pub400.com (either with or without userid, logonscreen will come allways)
  3. Run below script (after you have done file, edit session, connect,) I did not automate the menus as thats straight forward with alt and letter keys
  • Some calculation has to be done based on the font size you have installed
  • Mocha is a limited terminal emulator, no way to easy find out the blinking cursor position and no x, y in toolbar

Functions are generic

  • setPosTN5250 with x, y will calculate position and with mouseclick cursor will be positioned
  • getTN5250Text will just return the text in the black area
  • getTN5250hwnd will determine hwnd of first windows10.* with text toolstrip1, .... 
  • selectBlockTN5250 will select a rectangle of text (will not copy it thats just another send ctrl c)
  • _UIA_DrawRect draws a rectangle 😂



#include <WinAPI.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
;~ test with mocha tn5250
;~ www.pub400.com

global $g_baseLeft
global $g_baseRight
global $g_baseTop
global $g_baseBottom
global $g_TN5250Window


func example()
    Local $aPos = WinGetPos($g_TN5250Window)
;~   $aPos[0] = X position
;~   $aPos[1] = Y position
;~   $aPos[2] = Width
;~   $aPos[3] = Height
;~ Left,right,top,bottom

;~ So some calculations for the black area

;~ Lets positon mouse over the first row
    for $iCol=1 to 80

;~ Lets positon mouse over the first coloumn
    for $iRow=1 to 32


    $screenLines=stringsplit($screenAsText, @CRLF,$STR_ENTIRESPLIT)


func setPosTN5250($x,$y)
    mouseclick("left",$g_baseLeft + 4 + ($x*9) ,$g_basetop+8 + ($y*13),1,0)

func selectBlockTN5250($x1,$y1,$x2,$y2)
    mousemove($g_baseLeft + 4 + ($x1*9) ,$g_basetop+8 + ($y1*13),0)
    mousemove($g_baseLeft + 4 + ($x2*9) ,$g_basetop+8 + ($y2*13),0)

func getTN5250Text()
    return $text

func getTN5250hwnd()
    Local $aWinList = WinList("[REGEXPCLASS:WindowsForms10.Window.8.*]")
;~     _ArrayDisplay($aWinList)
    ; Loop through the array displaying only visable windows with a title.
    For $i = 1 To $aWinList[0][0]
        If $aWinList[$i][0] <> "" And BitAND(WinGetState($aWinList[$i][1]), 2) Then
            if (stringinstr(wingettext($hwnd), "toolStrip1")  > 0) Then return $hwnd
;~             MsgBox($MB_SYSTEMMODAL, "", "Title: " & $aWinList[$i][0] & @CRLF & "Handle: " & $aWinList[$i][1])
;~      _ArrayDisplay($aWinList)


; Draw rectangle on screen.
Func _UIA_DrawRect($tLeft, $tRight, $tTop, $tBottom, $color = 0xFF, $PenWidth = 4)
    Local $hDC, $hPen, $obj_orig, $x1, $x2, $y1, $y2
    $x1 = $tLeft
    $x2 = $tRight
    $y1 = $tTop
    $y2 = $tBottom
    $hDC = _WinAPI_GetWindowDC(0) ; DC of entire screen (desktop)
    $hPen = _WinAPI_CreatePen($PS_SOLID, $PenWidth, $color)
    $obj_orig = _WinAPI_SelectObject($hDC, $hPen)

    _WinAPI_DrawLine($hDC, $x1, $y1, $x2, $y1) ; horizontal to right
    _WinAPI_DrawLine($hDC, $x2, $y1, $x2, $y2) ; vertical down on right
    _WinAPI_DrawLine($hDC, $x2, $y2, $x1, $y2) ; horizontal to left right
    _WinAPI_DrawLine($hDC, $x1, $y2, $x1, $y1) ; vertical up on left

    ; clear resources
    _WinAPI_SelectObject($hDC, $obj_orig)
    _WinAPI_ReleaseDC(0, $hDC)
EndFunc   ;==>_UIA_DrawRect


Link to post
Share on other sites

UIASpy updates to Windows 10 version 1809
Documentation for Windows 10 version 1809 updates can be found in Windows 8, Windows 8.1 and Windows 10 updates. This post is pretty much completely rewritten.

In addition, there are a few minor updates that are documented below.

A small but significant update is that in Sample code to find the application top window, $TreeScope_Descendants is replaced by $TreeScope_Children. The application top window is always a direct child of the Desktop. See the treeview in UIASpy. Therefore, this update can be made. This is a major performance update when the application window is not at the top of the Z-order of windows.

UIA_MouseClick() in UIA_Functions.au3 is extended with right-click functionality. Sample code can be generated through Sample code | Code snippets ...

In UIASpy.ini, through the DisplayScale key, you can specify the display scale you want when UIASpy starts up.

Still not implemented update: Integrating $PropertyConditionFlags_MatchSubstring (Windows 10 1809) into Sample code.

New zip-file at bottom of first post.

Link to post
Share on other sites

Thank you Mr. argumentum. I'm on AutoIt and forgot to test on Zip-file in both examples is updated.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Create New...