Jump to content

Clicking button or sending shortcut to (stubbern) Windows - (Moved)


Recommended Posts


I am trying to automate a software called "LabelImg" (https://pypi.org/project/labelImg/).

My autoit-script is started once I selected a folder with images within LabelImg. Pressing the button "Next Image" or pressing the shortcut "d" (https://github.com/tzutalin/labelImg#Hotkeys) jumps to the next image in the selected folder. This shall happen once per second.

#include <Misc.au3>
#include <MsgBoxConstants.au3>
#include <AutoItConstants.au3>
Opt("WinTitleMatchMode", 1)

Local $hDLL = DllOpen("user32.dll")

While 1

    If _IsPressed("1B", $hDLL) Then
        Local $temp = WinActivate("labelImg")
        ConsoleWrite($temp & @CRLF)
        If WinActivate("labelImg") Then
            ConsoleWrite("All Set!" & @CRLF)

        Local $temp = MouseClick($MOUSE_CLICK_RIGHT, 50, 200)
        If $temp <> 1 Then
            MsgBox(1, "$temp", $temp)


So the Send ("d")-command and the MouseClick are alternative methods to jump to the next image. Both fail.

Both ConsoleWrite's deliver proper feedback (I continiously get the handle and "All set" ).


Could you tell me what I'm doing wrong? 

Thank you.

Edited by dejhost
Link to comment
Share on other sites

  • Developers

Moved to the appropriate forum, as the Developer General Discussion forum very clearly states:


General development and scripting discussions.

Do not create AutoIt-related topics here, use the AutoIt General Help and Support or AutoIt Technical Discussion forums.

Moderation Team

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
Live for the present,
Dream of the future,
Learn from the past.

Link to comment
Share on other sites

Thanks a lot for the hints!

Simple UIA Spy tells me the following:

Mouse position is retrieved 54-217
At least we have an element title: [Next Image] class: []

Having the following values for all properties: 
Title is: <Next Image>  Class   := <>   controltype:= <UIA_ButtonControlTypeId> ,<50000>    , (0000C350)    0;180;97;43
*** Parent Information top down ***
1: Title is: <labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]>   Class   := <Qt5QWindowIcon> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    -7;-7;2208;1208
"Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon"" 
0: Title is: <Tools>    Class   := <>   controltype:= <UIA_ToolBarControlTypeId>    ,<50021>    , (0000C365)    0;43;97;1131

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon")   ;labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]
_UIA_setVar("oP2","Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=") ;Tools

;~ $oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Next Image;controltype:=UIA_ButtonControlTypeId;class:=") ;ControlType:=UIA_ButtonControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 



;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon", $treescope_children)  
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=", $treescope_children) 
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

*** Detailed properties of the highlighted element ***
UIA_title:= <Next Image>
UIA_text:= <Next Image>
UIA_regexptitle:= <Next Image>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <0;180;97;43>
UIA_ProcessId:= <14960>
UIA_ControlType:= <50000>
UIA_LocalizedControlType:= <button>
UIA_Name:= <Next Image>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Next Image>
UIA_LegacyIAccessibleDescription:= <Open the next Image>
UIA_LegacyIAccessibleRole:= <43>
UIA_LegacyIAccessibleState:= <1048704>
UIA_LegacyIAccessibleDefaultAction:= <Press>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:14960,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

Is there a way to adress the button?

I also installed Ranorex Studio (Trial), but this seems overkill for just automating simple mouse-clicks.

Edited by dejhost
Link to comment
Share on other sites

I am not sure did you see simplespy highlight your area of interest?
if so yes then the code you gave above is a solution to automate your button. As you have a uiabounding rectangle you can move the mouse to that location and send a mouseclick on that location MouseMove,  MouseClick, MouseClickDrag, MouseCoordMode (Option), MouseDown, MouseGetPos, MouseUp, MouseWheel or search for sendinput in forum.

If no then you should look at

  • developers making the QT application more accessibile (then they are also more reachable for UIA)
  • commercial testautomation solutions (which sometimes hook deeper into your system but probably then need admin access to run)
    • Ranorex
    • Testcomplete
    • ....


Link to comment
Share on other sites

@junkew: simplespy highligts an area close by, displaying a red rectangle. So it doesn't highlight the button I am hovering over. That means, I need to look into alternative methonds 😞


@AlessandroAvolio: That's interesting! Thank you for trying. I built labelImg using anconda. That's the only possible reason I can think of. I now downloaded the prebuild binaries - which refuse to run: I just get a brief terminal window displayed. Did version 1.8.0 run straight out of the box in your case?



Link to comment
Share on other sites

Based on the spy output and your comment there is a red rectangle (assuming its the size of your button) I guess your DPI scaling of the screen is not 100%.

As spy is telling you this you know its recognized by using UIA library

"title:=Next Image;ControlType:=UIA_ButtonControlTypeId"

Then your next problem is how to invoke the button. Some options

1. Use the UIA invoke pattern (thats scaling independent, but not allways available)

2. Use the mousemove/mouseclick to the element boundingrectangle position to click (but then your DPI scaling could mean some addtional handling to do)

Link to comment
Share on other sites

@junkew: it seems indeed to be a scaling problem: when I move mousecursor to a button above or below "Next", and press 'control +w' to acivate simplespy, the red rectangle moves accordingly.

However, I got the precompiled version of LabelImg running by deleting a file in the user folder (https://github.com/tzutalin/labelImg/issues/221#issuecomment-368237423). In this version, the mouseclick works just fine - just as well as sending 'd', which is a builtin hotkey for "Next image" in LabelImg.


So with this, my problem is solved. Thank you very much for your efforts! I really appreciate it.


Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Create New...