Sign in to follow this  
Followers 0
seangriffin

Automate Qt and QWidgets

27 posts in this topic

#1 ·  Posted (edited)

This post explains how to automate applications developed in Qt (http://qt.nokia.com) using AutoIT. This includes such popular applications as VLC, Google Earth, Skype, Virtual Box and Avidemux.

Qt based applications contain widgets which are not immediately accessible through the Au3Info tool. There are many forum posts on the subject of querying Qt based applications using Au3Info, with the result that it returns only a single QWidget object for the entire application or window.

Qt is a framework which wraps, or masks, the underlying native Win32 controls within it's GUI. Like other similar frameworks and languages (such as Java), Qt includes accessibility features which allow other non-Qt applications to interact with the framework. Tools like AutoIT can use these accessibility features to automate Qt applications.

I have not yet found a Qt accessibility bridge which AutoIT can use (such as in the form of a Windows DLL).

However, I have found a very useful tool called Ranorex Spy (http://www.ranorex.com/gui-testing-guide/spy.html), which uses the accessibility features of Qt to expose information about the QWidgets within an application, in a similar manner to Au3Info. There doesn't appear to be any license associated with Ranorex Spy, so I have attached the tool to this post for the benefit of others.

Use Ranorex Spy in the same manner that you use Au3Info. With the application you want to automate already open, drag the Finder Tool (crosshair) in Ranorex Spy over the application, and you'll notice that it can identify the widgets within the application (in the same manner that Au3Info would).

You can then use this information within the standard AutoIT function calls. Here are some examples.

If the QWidget has a "Caption/text", as identified in Ranorex Spy, then use the following AutoIT call to click on that control:

ControlClick("window title", "", "[TEXT:the Caption/text from Ranorex Spy]")

If the QWidget doesn't have a "Caption/text", then you can use the "Handle" from Ranorex Spy instead. Paste the handle value into the following AutoIT script:

Const $handle_to_find = "paste the handle from Ranorex Spy here"

For $i = 1 to 1000

    $handle = ControlGetHandle("window title", "", "[CLASS:QWidget; INSTANCE:" & $i & "]")

    if @error = 1 Then ExitLoop

    if ($handle = $handle_to_find) Then

        ConsoleWrite("Found QWidget handle " & $handle_to_find & " at instance: " & $i & @CRLF)
        ExitLoop
    EndIf
Next

Run the script and it should give you the QWidget instance number of the control. Use this instance number in the following AutoIT call to click on that control:

ControlClick("window title", "", "[CLASS:QWidget; INSTANCE:insert the instance number here]")

Hope this is useful to all those having trouble automating Qt.

Apologies everyone, but I've had to remove the "RanorexSpy.zip" attachment because I've run out of upload quota for the site. Contact me if you'd like a copy.

Edited by seangriffin
1 person likes this

Cheers, Sean.

See my other UDFs:

Chrome UDF - Automate Chrome | SAP UDF - Automate SAP | Java UDF - Automate Java Applications & Applets | Tesseract (OCR) UDF - Capture text from applications, controls and the desktop | Textract (OCR) UDF - Capture text from applications and controls | FileSystemMonitor UDF - File, Folder, Drive and Shell Monitoring | VLC (Media Player) UDF - Creating and controlling a VLC control in AutoIT | Google Maps UDF - Creating and controlling Google Maps (inc. GE) in AutoIT | SAPIListBox (Speech Recognition) UDF - Speech Recognition via the Microsoft Speech (SAPI) ListBox | eBay UDF - Automate eBay using the eBay API | ChildProc (Parallel Processing) UDF - Parallel processing functions for AutoIT | HyperCam (Screen Recording) UDF - Automate the HyperCam screen recorder | Twitter UDF - Automate Twitter using OAuth and the Twitter API | cURL UDF - a UDF for transferring data with URL syntax

See my other Tools:

Rapid Menu Writer - Add menus to DVDs in seconds | TV Player - Automates the process of playing videos on an external TV / Monitor | Rapid Video Converter - A tool for resizing and reformatting videos | [topic130531]Rapid DVD Creator - Convert videos to DVD fast and for free | ZapPF - A tool for killing processes and recycling files | Sean's eBay Bargain Hunter - Find last minute bargains in eBay using AutoIT | Sean's GUI Inspector - A scripting tool for querying GUIs | TransLink Journey Planner with maps - Incorporating Google Maps into an Australian Journey Planner | Automate Qt and QWidgets | Brisbane City Council Event Viewer - See what's going on in Brisbane, Australia

Share this post


Link to post
Share on other sites



You're my personal hero! oO

Have long been searching desperately for a way to control Teamspeak3 ... and now it should finally be possible? :) Thank you!

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

You're my personal hero! oO

Have long been searching desperately for a way to control Teamspeak3 ... and now it should finally be possible? :) Thank you!

;) glad to be of service. I too was having a rough time automating Avidemux, until I stumbled over this solution.

I would have thought many frustrated users would like this, so I tried to inform people in older forum posts about Qt automation issues, though updating old posts seems to be some sort of improper behaviour these days, and my posts have been removed. Never used to be the case. Times have changed.

Well hope people like yourself still manage to find this post.

Edited by seangriffin

Cheers, Sean.

See my other UDFs:

Chrome UDF - Automate Chrome | SAP UDF - Automate SAP | Java UDF - Automate Java Applications & Applets | Tesseract (OCR) UDF - Capture text from applications, controls and the desktop | Textract (OCR) UDF - Capture text from applications and controls | FileSystemMonitor UDF - File, Folder, Drive and Shell Monitoring | VLC (Media Player) UDF - Creating and controlling a VLC control in AutoIT | Google Maps UDF - Creating and controlling Google Maps (inc. GE) in AutoIT | SAPIListBox (Speech Recognition) UDF - Speech Recognition via the Microsoft Speech (SAPI) ListBox | eBay UDF - Automate eBay using the eBay API | ChildProc (Parallel Processing) UDF - Parallel processing functions for AutoIT | HyperCam (Screen Recording) UDF - Automate the HyperCam screen recorder | Twitter UDF - Automate Twitter using OAuth and the Twitter API | cURL UDF - a UDF for transferring data with URL syntax

See my other Tools:

Rapid Menu Writer - Add menus to DVDs in seconds | TV Player - Automates the process of playing videos on an external TV / Monitor | Rapid Video Converter - A tool for resizing and reformatting videos | [topic130531]Rapid DVD Creator - Convert videos to DVD fast and for free | ZapPF - A tool for killing processes and recycling files | Sean's eBay Bargain Hunter - Find last minute bargains in eBay using AutoIT | Sean's GUI Inspector - A scripting tool for querying GUIs | TransLink Journey Planner with maps - Incorporating Google Maps into an Australian Journey Planner | Automate Qt and QWidgets | Brisbane City Council Event Viewer - See what's going on in Brisbane, Australia

Share this post


Link to post
Share on other sites

Dang, couldnt you have posted this a year ago? Could really have used it for my old Google Earth project ;)

Joking ofcourse ;) it's nice to know it's available now :)

Share this post


Link to post
Share on other sites

Dang, couldnt you have posted this a year ago? Could really have used it for my old Google Earth project :D

Joking ofcourse ;) it's nice to know it's available now ;)

:) I hear your pain. I remember also trying to automate Avidemux about a year ago, only to give up in frustration! I'm also kicking myself for not finding this earlier.

To my surprise Qt has been around for a very long time (http://en.wikipedia.org/wiki/Qt_%28framework%29). I've actually never heard of it (and I've had to automate my own fair share of technologies in the past), though it's obviously a prominant development platform when you look at the long list of well known apps developed in it.

I hope I can help a few people with this one. And if Ranorex can produce a Qt inspection tool such as Ranorex Spy, then obviously something could be done specifically for AutoIT (as I had done earlier in my Java UDF to bridge Java accessibility with AutoIT via a DLL).


Cheers, Sean.

See my other UDFs:

Chrome UDF - Automate Chrome | SAP UDF - Automate SAP | Java UDF - Automate Java Applications & Applets | Tesseract (OCR) UDF - Capture text from applications, controls and the desktop | Textract (OCR) UDF - Capture text from applications and controls | FileSystemMonitor UDF - File, Folder, Drive and Shell Monitoring | VLC (Media Player) UDF - Creating and controlling a VLC control in AutoIT | Google Maps UDF - Creating and controlling Google Maps (inc. GE) in AutoIT | SAPIListBox (Speech Recognition) UDF - Speech Recognition via the Microsoft Speech (SAPI) ListBox | eBay UDF - Automate eBay using the eBay API | ChildProc (Parallel Processing) UDF - Parallel processing functions for AutoIT | HyperCam (Screen Recording) UDF - Automate the HyperCam screen recorder | Twitter UDF - Automate Twitter using OAuth and the Twitter API | cURL UDF - a UDF for transferring data with URL syntax

See my other Tools:

Rapid Menu Writer - Add menus to DVDs in seconds | TV Player - Automates the process of playing videos on an external TV / Monitor | Rapid Video Converter - A tool for resizing and reformatting videos | [topic130531]Rapid DVD Creator - Convert videos to DVD fast and for free | ZapPF - A tool for killing processes and recycling files | Sean's eBay Bargain Hunter - Find last minute bargains in eBay using AutoIT | Sean's GUI Inspector - A scripting tool for querying GUIs | TransLink Journey Planner with maps - Incorporating Google Maps into an Australian Journey Planner | Automate Qt and QWidgets | Brisbane City Council Event Viewer - See what's going on in Brisbane, Australia

Share this post


Link to post
Share on other sites

First, I am really happy that this Thread helped a lot of people and solved main issues.

Unfortunately, I still have a problem with Internet Explorer_Server class (in Kaspersky Anti-virus 2011) and with QWidget class (in Kaspersky Anti-virus 2012).

Here is Kav 2011 screenshot (Nothing change in RanorexSpy program as i move my mouse in the window)

Posted Image

Here is Kav 2012 screenshot (Nothing change in RanorexSpy program as i move my mouse in the window)

Posted Image

I would really appriciate any help about this issue. it is really important to me to access the inner Controls in these applications.

Thanks

Share this post


Link to post
Share on other sites

hey i've tried ranorex spy on skype but it is not working.

let's say here i've attached skype screenshot and i want to automate it.

but ranorex spy is not helpful to identify search box that is highlighted.

and it is failed to identify other components also.

post-62603-0-34221700-1313660880_thumb.p

Share this post


Link to post
Share on other sites

same problem, hope autoit can support QT framework in future....

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

Thank you seangriffin for pioneering this process.

I've been working with this example the past few days with utter frustration and now I've figured a bit more out.

I am able to use Ranorex Spy to see the field caption/text in my QT application, but found that calling their controls only worked sometimes.

Then I figured it out....

If I use Ranorex finder tool on the element first, then call

ControlClick("MyAppTitle", "", "[TEXT:comboEnv]")

it works fine

If I close the app and then open it again and try to call ControlClick... it fails.

As soon as I use ranorex to scan the field again, THEN try the ControlClick.. it works!

So its like the fields have to be scanned by Ranorex first to be instantiated as their control names.

Very tricky stuff

I am knowingly necroposting since this is still a valid topic and moot to create a new one. I am also contributing new knowledge that may help others.

Edited by DssTrainer

Share this post


Link to post
Share on other sites

@DssTrainer: Last year I tried Ranorex Spy with another application and it was the same for me to. AutoIt could only access the controls if Ranorex Spy scanned the application first.


Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the Universe
trying to produce bigger and better idiots.
So far, the Universe is winning.

Share this post


Link to post
Share on other sites

Yep. That "AutoIt could only access the controls if Ranorex Spy scanned the application first." is a general know behavior.

You might like to take a look at the Interface AutoItObject IUIAutomation topic.


"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

^^

Thanks for the link. To simplify things, I instead made a quick-n-dirty RanorexifyMe($title) function that starts ranorexspy, enables HotTracking, hovers the mouse over my QT Application window, then kills ranorexspy... leaving my QT app properly accessible. Works perfectly.

Edited by DssTrainer

Share this post


Link to post
Share on other sites

What do you do if the application you are trying to automate has a new handle for each control every time you go into it? The instance numbers stay the same but can shuffle around depending on who logs in. So we can't even count on that, and only 1 control has any caption text to it.

I can't count on things being in the same place either, because the window might not be fullscreen or the screen resolutions might be different depending on who is using it. Is there anything else I can try that will let me always get the same buttons and controls all of the time, other than handles or caption text?

Share this post


Link to post
Share on other sites

^^

Thanks for the link. To simplify things, I instead made a quick-n-dirty RanorexifyMe($title) function that starts ranorexspy, enables HotTracking, hovers the mouse over my QT Application window, then kills ranorexspy... leaving my QT app properly accessible. Works perfectly.

Hi, Can you please show me how to write the "quick-n-dirty" function that you are referring to? I tried the following but did not work :(

Run("E:\Installers\RanorexSpy.exe")
Sleep(2000)
ControlEnable("RanorexSpy", "", "&Enable")
Run(<myExe>)
Sleep(2000)

ControlFocus(<myTitle>, <myText>, "[CLASS:QWidget; INSTANCE:1]")
#include <GuiListView.au3>
_GUICtrlListView_SetHoverTime(<myTitle>, 2000)

Thanks,

-Amit

Share this post


Link to post
Share on other sites

What do you do if the application you are trying to automate has a new handle for each control every time you go into it? The instance numbers stay the same but can shuffle around depending on who logs in. So we can't even count on that, and only 1 control has any caption text to it.

I can't count on things being in the same place either, because the window might not be fullscreen or the screen resolutions might be different depending on who is using it. Is there anything else I can try that will let me always get the same buttons and controls all of the time, other than handles or caption text?

Firstly, every time I open my Qt application, the control has a different handle. Secondly, I put the caption/text into the parameter but no action...

Do you know why?

Share this post


Link to post
Share on other sites

How do I click on a particular button in the Qwidget? For example with VLC media player, the play button?

Thanks!

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

Hi, Can you please show me how to write the "quick-n-dirty" function that you are referring to? I tried the following but did not work :(

 

Run("E:\Installers\RanorexSpy.exe")
Sleep(2000)
ControlEnable("RanorexSpy", "", "&Enable")
Run(<myExe>)
Sleep(2000)

ControlFocus(<myTitle>, <myText>, "[CLASS:QWidget; INSTANCE:1]")
#include <GuiListView.au3>
_GUICtrlListView_SetHoverTime(<myTitle>, 2000)
Thanks,

-Amit

 

 

Here's my code:

Func Ranorexify($title, $text = "")

    Dim $winpos, $winsize

    ; Start RanorexSpy
    Run("C:\RanorexSpy.exe", "C:\", @SW_HIDE)

    ; Enable HotTrackerdev on RanorexSpy
    ControlCommand("RanorexSpy", "", "[TEXT:&Enable]", "Check", "")

    ; Activate Application Window
    if WinActivate($title) Then
        $winpos = WinGetPos($title, $text)
        if (IsArray($winpos)) Then
            MouseMove($winpos[0]+200, $winpos[1]+200, 0)
        Else
            MsgBox(1, "Window Not Found - " & @Scriptname, $title)
        EndIf

        ; Restore focus to Application
        WinActivate($title, $text)
    else
        MsgBox(1, "Window Not Active - " & @Scriptname, $title)
    EndIf

    ; Kill RanorexSpy
    WinKill("RanorexSpy")

EndFunc
Edited by DssTrainer

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

I assume Ranorex spy is doing some kind of handshake to activate within QT the accessibility

maybe stuff like this can work

PostMessage( HWND_BROADCAST, WM_WININICHANGE, 
SPI_SETSCREENREADER, 0 );
SystemParametersInfo( SPI_SETSCREENREADER, TRUE, NULL, SPIF_UPDATEINIFILE | 
SPIF_SENDCHANGE);

http://support.microsoft.com/kb/180958

A system environment variable "SPI_GETSCREENREADER" is referenced by some IAccessible2 Clients

or just do this

Make sure you have a folder named ‘accessible’ in your application’s folder with the executable and the file ‘qtaccessiblewidgets.dll’

Edited by junkew

Share this post


Link to post
Share on other sites

Has anyone had luck using this method with Malwarebytes Version 2?  I can't seem to figure out how to ControlClick on the big green Scan Now button?  If someone could show me how to do that step I'm sure I can figure out the rest.  RanorexSpy doesn't seem to be getting any button info from Malwarebytes. :(

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0