Jump to content

Clicking button or sending shortcut to (stubbern) Windows - (Moved)

Recommended Posts


I am trying to automate a software called "LabelImg" (https://pypi.org/project/labelImg/).

My autoit-script is started once I selected a folder with images within LabelImg. Pressing the button "Next Image" or pressing the shortcut "d" (https://github.com/tzutalin/labelImg#Hotkeys) jumps to the next image in the selected folder. This shall happen once per second.

#include <Misc.au3>
#include <MsgBoxConstants.au3>
#include <AutoItConstants.au3>
Opt("WinTitleMatchMode", 1)

Local $hDLL = DllOpen("user32.dll")

While 1

    If _IsPressed("1B", $hDLL) Then
        Local $temp = WinActivate("labelImg")
        ConsoleWrite($temp & @CRLF)
        If WinActivate("labelImg") Then
            ConsoleWrite("All Set!" & @CRLF)

        Local $temp = MouseClick($MOUSE_CLICK_RIGHT, 50, 200)
        If $temp <> 1 Then
            MsgBox(1, "$temp", $temp)


So the Send ("d")-command and the MouseClick are alternative methods to jump to the next image. Both fail.

Both ConsoleWrite's deliver proper feedback (I continiously get the handle and "All set" ).


Could you tell me what I'm doing wrong? 

Thank you.

Edited by dejhost
Link to post
Share on other sites
  • Developers

Moved to the appropriate forum, as the Developer General Discussion forum very clearly states:


General development and scripting discussions.

Do not create AutoIt-related topics here, use the AutoIt General Help and Support or AutoIt Technical Discussion forums.

Moderation Team

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
Live for the present,
Dream of the future,
Learn from the past.

Link to post
Share on other sites

Thanks a lot for the hints!

Simple UIA Spy tells me the following:

Mouse position is retrieved 54-217
At least we have an element title: [Next Image] class: []

Having the following values for all properties: 
Title is: <Next Image>  Class   := <>   controltype:= <UIA_ButtonControlTypeId> ,<50000>    , (0000C350)    0;180;97;43
*** Parent Information top down ***
1: Title is: <labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]>   Class   := <Qt5QWindowIcon> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    -7;-7;2208;1208
"Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon"" 
0: Title is: <Tools>    Class   := <>   controltype:= <UIA_ToolBarControlTypeId>    ,<50021>    , (0000C365)    0;43;97;1131

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon")   ;labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]
_UIA_setVar("oP2","Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=") ;Tools

;~ $oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Next Image;controltype:=UIA_ButtonControlTypeId;class:=") ;ControlType:=UIA_ButtonControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 



;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon", $treescope_children)  
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=", $treescope_children) 
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

*** Detailed properties of the highlighted element ***
UIA_title:= <Next Image>
UIA_text:= <Next Image>
UIA_regexptitle:= <Next Image>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <0;180;97;43>
UIA_ProcessId:= <14960>
UIA_ControlType:= <50000>
UIA_LocalizedControlType:= <button>
UIA_Name:= <Next Image>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Next Image>
UIA_LegacyIAccessibleDescription:= <Open the next Image>
UIA_LegacyIAccessibleRole:= <43>
UIA_LegacyIAccessibleState:= <1048704>
UIA_LegacyIAccessibleDefaultAction:= <Press>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:14960,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

Is there a way to adress the button?

I also installed Ranorex Studio (Trial), but this seems overkill for just automating simple mouse-clicks.

Edited by dejhost
Link to post
Share on other sites

I am not sure did you see simplespy highlight your area of interest?
if so yes then the code you gave above is a solution to automate your button. As you have a uiabounding rectangle you can move the mouse to that location and send a mouseclick on that location MouseMove,  MouseClick, MouseClickDrag, MouseCoordMode (Option), MouseDown, MouseGetPos, MouseUp, MouseWheel or search for sendinput in forum.

If no then you should look at

  • developers making the QT application more accessibile (then they are also more reachable for UIA)
  • commercial testautomation solutions (which sometimes hook deeper into your system but probably then need admin access to run)
    • Ranorex
    • Testcomplete
    • ....


Link to post
Share on other sites

@junkew: simplespy highligts an area close by, displaying a red rectangle. So it doesn't highlight the button I am hovering over. That means, I need to look into alternative methonds 😞


@AlessandroAvolio: That's interesting! Thank you for trying. I built labelImg using anconda. That's the only possible reason I can think of. I now downloaded the prebuild binaries - which refuse to run: I just get a brief terminal window displayed. Did version 1.8.0 run straight out of the box in your case?



Link to post
Share on other sites

Based on the spy output and your comment there is a red rectangle (assuming its the size of your button) I guess your DPI scaling of the screen is not 100%.

As spy is telling you this you know its recognized by using UIA library

"title:=Next Image;ControlType:=UIA_ButtonControlTypeId"

Then your next problem is how to invoke the button. Some options

1. Use the UIA invoke pattern (thats scaling independent, but not allways available)

2. Use the mousemove/mouseclick to the element boundingrectangle position to click (but then your DPI scaling could mean some addtional handling to do)

Link to post
Share on other sites

@junkew: it seems indeed to be a scaling problem: when I move mousecursor to a button above or below "Next", and press 'control +w' to acivate simplespy, the red rectangle moves accordingly.

However, I got the precompiled version of LabelImg running by deleting a file in the user folder (https://github.com/tzutalin/labelImg/issues/221#issuecomment-368237423). In this version, the mouseclick works just fine - just as well as sending 'd', which is a builtin hotkey for "Next image" in LabelImg.


So with this, my problem is solved. Thank you very much for your efforts! I really appreciate it.


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By TheAutomator
      I'm writing a script that uses a lot of specific mouse clicks to automate tasks on my browser, server system, and email software for work.
      Sometimes the system gets updated or someone drags a handle to the left or right and I have to put in new mouse click coordinates.

      It would be a lot more easy to label them and put them in some sort of ini file, but then i have to load every X and Y (and sometimes mouse speed) into a variable
      when loading the script.

      How would you guys go about such a script, any ideas on how to store and load lot's of click coordinates from a file?
      There are some pixel color checks, sleeps and other code between the mouse clicks too..

      Ideas i had so far:
      - string replace new coordinates in the main script with another autoit script that uses a crosshair to show every mouse position.
      - manually write an ini file and load every X, Y, Speed into a huge number of variables: $Searchinput_X = iniread...., $Searchinput_Yiniread...., $Searchinput_Speed = iniread....
      - split a textfile line by line into an array and ignore labels that begin with ";" for example (a bit like a custom ini file)

      These all seem somewhat tedious i guess..
    • By AthanBit
      I am Using Windows 10 on 32 Operating System
      On typing I use Alt + Shift so  to enable toggle typing in English and Greek
      i find it impossible to achieve the same function using autoit3 using Send ("!+")
      I would appreciate any help in solving this problem
      Also I would appreciate any help in switching to Greek language using the CAPS Key
      Thank you
    • By Danyfirex
      AT Command UDF - for control AT Modems, send SMS, get SMS
      #cs 1.0.0 2020/10/03 . First version - Danyfirex + mLipok 1.0.1 2020/10/04 . Added - Function - _ATCmd_IsPINReady - Danyfirex . Added - Function - _ATCmd_IsPINRequired - Danyfirex . Added - Function - _ATCmd_IsSIMInserted - Danyfirex . Added - Function - _ATCmd_IsSenderSupported - Danyfirex . Added - Function - _ATCmd_OnPINReques - Danyfirex . Added - Function - _ATCmd_SMS_ListTextMessages - Danyfirex . Added - Function - _ATCmd_SetPIN - Danyfirex . Added - Function - __ATCmd_GetPINCounter - Danyfirex - Added - ENUM - $ATCmd_ERR_PIN - Danyfirex - Added - ENUM - $ATCmd_ERR_SIM - Danyfirex . Changed - __ATCmd_ComposePDU() - using _ATCmd_UseUCS2() internally instead parameter - Danyfirex . Suplemented - #CURRENT# - Danyfirex . . 1.0.2 2020/10/05 . Added - ENUM - $ATCmd_MSGLIST_* - mLipok . Added - ENUM - $ATCmd_STATUS__* - mLipok - Added - ENUM - $ATCmd_ERR_PARAMETER - mLipok . Added - _ATCmd_UsePDU() - parameter validation - mLipok . Added - _ATCmd_UseUCS2() - parameter validation - mLipok . Added - more error logs . Changed - MagicNumber replaced with Standard UDF constants - mLipok . Small refactoring - mLipok . . 1.0.3 2020/10/05 . CleanUp - Danyfirex . . 1.0.4 2020/10/05 . Small refactoring - Danyfirex . CleanUp - Danyfirex . . 1.0.5 2020/10/23 . _ATCmd_FullLoging - mLipok . _ATCmd_CMEESetup() ... @WIP - mLipok . $ATCMD_STATUS_11_SUBSCRIBERNUMBER - mLipok . . 1.0.6 2020/10/25 . __ATCmd_CMSErrorParser() - mLipok . . @LAST https://www.nowsms.com/gsm-modem-cms-error-code-list https://m2msupport.net/m2msupport/at-command-to-enable-error-codes/ https://www.micromedia-int.com/en/gsm-2/73-gsm/669-cme-error-gsm-equipment-related-errors https://assets.nagios.com/downloads/nagiosxi/docs/ATCommandReference.pdf https://www.maritex.com.pl/product/attachment/40451/15b4db6d1a10eada42700f7293353776 https://www.multitech.net/developer/wp-content/uploads/2010/10/S000463C.pdf https://www.telit.com/wp-content/uploads/2017/09/Telit_AT_Commands_Reference_Guide_r24_B.pdf https://docs.rs-online.com/5931/0900766b80bec52c.pdf PDU Format / Testers / Encoders / decoders https://m2msupport.net/m2msupport/sms-at-commands/#pduformat http://smstools3.kekekasvi.com/topic.php?id=288 #ce  
    • By Drac89
      Hi Team, 
      I am trying to send credentials via autoit on a web app launched via IE. I am passing the credentials into variables and trying to send them send("$username") but username field isn't getting populated. But the password field, I can send the credential, also when I try ieformsetvalue and set the username wrt username field fetched, it's getting send but my login button remains greyed out. 
      Can angularjs keydown function interfere in the autoit send process and not allow the credentials passed if it's not typed out? This is a general question to understand if javascript or angular js etc could detect if the credentials are automated, then it might not detect it as a keydown? 
    • By Rskm
      Hi, recently i created a GUI for some calculations in AutoIT.  The GUI has 2 tabs and on the first tab, it has few inputboxes where i use to click with mouse and start entering the inputs. i recently made some changes in the position of these textboxes. I made this by changing the autoit code file instead of making changes in KODA. Now i cannot click inside these textboxes with mouse. However, i can use TAB key to cycle through/enter input in these input boxes. what could have made the inputboxes not responding to mouseclicks. thanks
  • Create New...