Gal

Mouse click and sends in the background

13 posts in this topic

I've been searching these past few days for anything that would help me automate in the background, I've made a few things work but for a couple decade old programs I can't get the control IDs, which is resulting in difficulties.

 

I've tried using the _MouseClickPlus method which calls the user32 dll sendmessage with the mouse opcodes, this worked like a charm for a new adobe apollo runtime environment I was checking.

In that case, it didn't have any control IDs or anything else, and somehow all it needed to work was the title to obtain the winhandle from.

However this doesn't work for many windows apps, not sure why.

 

All of the following problems are under windows 7

1) In paint, if I use

WinGetHandle("Untitled - Paint")

it returns zeroes, anyone have any idea why? I've gotten around this by using the title "[CLASS:MSPaintApp]", however I'm still interested in this response.

2) Next I was looking to make it click somewhere, this:

ControlClick("[CLASS:MSPaintApp]", "", "[CLASS:Afx:00000000FF0C0000:8; INSTANCE:1]", "left", 1, 275, 152)

worked, although a better solution for me would be supplying only the title / handle and coords from the window start and not from the control start, and having it click on any control.

Is there any way to do this?

3) In Paint, if I click / hold for a second the alt key, I can see the keyboard shortcuts, for example, alt + H would trigger the home tab. to what control do I use ControlSend to so that "!hp" would call "Alt h" and then select the pencil with "p", I couldn't make this work when the main whiteboard is the controlID and neither when the top toolbar is the control ID, like so:

ControlSend("[CLASS:MSPaintApp]", "", "[CLASS:NetUIHWND; INSTANCE:1]", "!hp")

4) Continuing the previous topic, in one of my older programs, alt + F for example doesn't trigger the File at the top, so I would need to send a ControlClick to it, however it doesn't have any control info, you can look at the SciTE editor, going over the file/edit/search.. with the window info tool, there's nothing about the control.

How can I send a ControlClick to the Search button in this case? Simply supplying the window is not enough, and I have a feeling ControlClick never works without specifying a control ID.

A solution to my second question would be a solution to the fourth as well :/

 

Any tips will be greatly appreciated,

Thanks!

g3trans

 

 

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

For #1:  I think you may not have the window activated.  Try this to get the handle and see if it works for you ...

run("mspaint.exe")
sleep(1000)
WinActivate("Untitled - Paint")
$paintHandle=WinGetHandle("Untitled - Paint")
ConsoleWrite(@CRLF&$paintHandle&@CRLF)

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

Edited by Jfish
1 person likes this

Build your own poker game with AutoIt: pokerlogic.au3 | Learn To Program Using FREE Tools with AutoIt

Share this post


Link to post
Share on other sites

Why would the Window need to be active to get its handle ?

This is working fine for me so what do you get?:

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : WinGetHandle("Untitled - Paint") = ' & WinGetHandle("Untitled - Paint") & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

Jos :)

1 person likes this

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

You're right, not sure why that didn't work for me the other day. 

Any idea on my other questions? 

 

Basically I would ideally be able to send a ControlClick anywhere in the window, but for now, the simplified problem would be in the SciTE editor, sending a click to open the "Search" options and then clicking "Find" for example

This would be like sending Ctrl + F if the window was on top and activated, but in the background I'm trying to go about it with ControlClick.

 

Any thoughts? 

Gal

 

 

 

Share this post


Link to post
Share on other sites

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

I'm checking out the IUI now, as for sending send instead of controlsend, that's not an option for me as I need to perform the whole automation in the background.

I'll update with my impression of the IUI

Share this post


Link to post
Share on other sites

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

This is pretty cool, the simplyspy, but it doesn't work in the background, to requires me to send one of their methods _UIA_Action with setfocus before it can perform a click on the object I recorded.

Maybe there's a way to use the object I found with the simplespy and send it to the controlclick?

 

Like using the below methods to find an object by title and class and control type, and then forwarding that to control click?

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=C:\Users\King\Desktop\simplyspy\exploratoryAutoit.au3 - SciTE-Lite [4 of 4];controltype:=UIA_WindowControlTypeId;class:=SciTEWindow", $treescope_children)
 ;_UIA_Action($oP1,"setfocus")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Application;controltype:=UIA_MenuBarControlTypeId;class:=", $treescope_children)
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Search.mainwindow", "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Search;controltype:=UIA_MenuControlTypeId;class:=#32768", $treescope_children)
 ;_UIA_Action($oP0,"setfocus")
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Replace...  Ctrl+H.mainwindow", "title:=Replace...  Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Replace... Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")

 

Share this post


Link to post
Share on other sites

The programs I want to be running in the background need to talk to com ports and actual hardware, 

I can imagine a setup where the physical computer translates the com ports input / output into tcp/ip commands to talk to a virtual machine and then that virtual machine translates it back into comm port emulation but that is too complex for me to even grasp how I would go about getting that to happen ;)

 

Right now there's a machine dedicated to running the autoit, which is less than idea because if I were to want to run several of these at the same time I'd need a machine for each run, and more external hardware which is pricier as well.

 

 

I'm sure that it's possible to send these clicks / sends in the background, I just can't find the solution.

 

Thanks for the tips though 

 

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

Edited by Karthi

Share this post


Link to post
Share on other sites
6 hours ago, Karthi said:

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

 

I haven't found a way to do something like this. If you're automating, you eventually are forced to use blockinput because if you use the mouse or keyboard you're going to directly interfere with whatever you're trying to do. You need mouse in X,Y position? Move your mouse while this is happening and it won't be where you need it. Automating while doing work yourself isn't something that autoit can do.

Maybe a virtual machine would work, but i have no experience working with it. I don't think it would work.

Share this post


Link to post
Share on other sites

@Karthi traditionally you would use ControlSend or ControlClick to interact with a window, even when it is no active. I believe the suggestion for IUI was because the OP was unable to do this. This is another reason why it is better to begin your own thread rather than tagging a question onto the end of someone else's; your needs may be different from those of the OP.

I would suggest you start with a detailed explanation on what application you're trying to automate, what the window looks like, what you find when you use the AutoIt Window Info Tool (located in the same directory where you installed AutoIt) etc. The more info you give us the better we can offer assistance.


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • lganta
      By lganta
      Hello!
      I created some scripts for a simple farming bot a few years ago and they rely on ControlSend.
      A year ago I quit that game and then installed Windows 10 (was using windows 7).
      Recently I installed that game and tried running them again and they seem to work (I have some messages displayed on the screen with the state of the bot ), except for the ControlSend part (which obviously is crucial).
      I started debugging the scripts so I created a basic script that makes use of Send. I tested this with Notepad in focus and works just fine and then with my game window in focus and it didn't work (it's supposed to write that text in an input box from the game).
      #include <MsgBoxConstants.au3> Sleep(3000); Send("some text"); MsgBox($MB_OK, "Notification", "Control was sent!");  
      Is there a way for the creators of the game to create some kind of security system against this? Or something happens because I updated to Windows 10?
      Is there something I'm missing?
      Thank you!
    • ur
      By ur
      I am clicking on a button on to submit o form UI.
      Problem is, the code working on windows 10,7, server 2012 r2 but not on windows server 2008.
      I checked the IDs also on that OS, they didn't change.
      Even the ControlClick is returning the status as success only but button is not getting clicked.
      But controlsettext on other fields are working.
      ;$MartConfigWindow is window object ControlClick($MartConfigWindow, "", "[CLASS:WindowsForms10.BUTTON.app.0.378734a; INSTANCE:2]")  
    • lattey
      By lattey
      I used the window info tool to get what I've got below, but it just doesn't click the button. I run the script, nothing happens. 
      WinActivate("SELECTION") Sleep(200) ControlClick("SELECTION", "", "[CLASS:Button;INSTANCE:2]") i think maybe the autoit cannot detect the windows. So what i do:
       
      MouseClick( "", 503, 500) local $blanktitle = WinGetTitle("[ACTIVE]") If StringInStr($blanktitle, "SELECTION") Then msgbox("","", "got") WinActivate($blanktitle) ControlClick($blanktitle, "", "[CLASS:Button;INSTANCE:2]") Endif The script can detect the windows as the it is prompt "got" message. But it just doesn't click the button.
      Please help.
      Thanks for any help you can offer.
    • rawkhopper
      By rawkhopper
      Hello everyone,
      I have a script that is automating a piece of sluggish software.  I want to automate it with a bit of caution and I am not sure the best way to do it.  ControlSend works great most of the time.  If I have it enter 10 into a text box every once in a while it will enter 1 or 11 and then hit enter.
      To overcome this I use MouseClick to select the text I just entered and then see if it matches the string it was supposed to put in before it hits enter.  This seems to work but what I love about ControlSend is there is less room for human interaction messing it up.  
      Yes I could block input but I prefer not to do that (permissions).
      Is there a better way of doing this?  Any Help would be much appreciated.
      Anyway here is the snippet of the script in question:
      Func KVSend () WinActivate ( "Window", "" ) Local $WindowPos = WinGetPos("Window", "") If $kV < 30 Then WinActivate ( "Window", "" ) ControlClick ("Window", "", 1001) ;Click in Accel Voltage box Sleep (100) ControlSend ("Window", "", 1001, $kV) ; \ kV Sleep (100) MouseMove($WindowPos[0]+130,$WindowPos[1]+75,1) MouseClick($MOUSE_CLICK_LEFT) MouseClick($MOUSE_CLICK_LEFT) Send ("^c") Local $clip = ClipGet () If $clip = $kV Then ControlSend ("Window", "", 1001, "{ENTER}") ;Hit ENTER if value is correct Sleep (100) ControlClick ("Window", "", 1518) ;Lens Clear Else Send ("{BACKSPACE}") KVSend() ; If value is incorrect try again EndIf EndIf  
    • Ubermensch
      By Ubermensch
      Hello,
      I am currently trying to automatically click the "Yes" button in the ActiveX prompt/popup message after opening the IE (html).
      At 1st, I encounter the "Allow Blocked Content". I already resolve it just by changing settings in the IE Options. But after resolving the "Allow Blocked Content", there's a popup message appear.
      I have attached the ActiveX Prompt.
      Here is the 1st code that I try to use.
      #include <IE.au3> #include <MsgBoxConstants.au3> #include <WinAPI.au3> _IECreate("C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\sampleAWD10.html",0,1,0) Local $oIE = _IEAttach("", "instance", 1) _IELoadWait($oIE) AdlibRegister("_ActiveXRun",250) Local $oLastName = _IEGetObjByName($oIE, "Text4") Local $oGetItem = _IEGetObjByName($oIE, "getitem") _IEAction($oGetItem, "click") MsgBox($MB_SYSTEMMODAL, "Form Element Value", _IEFormElementGetValue($oGetItem)) Func _ActiveXRun() $retWin = WinGetHandle("[Class:Button]","") $winTitle = "[HANDLE:" & $retWin &"]" $ctrlHandle = ControlGetHandle($winTitle,"", "[CLASS:Button; INSTANCE:2]") $ctrlTitle = "HANDLE:" & $ctrlHandle &"]" WinWaitActive($ctrlTitle,"[CLASS:Button; INSTANCE:2]",10) $k = ControlGetPos($winTitle, "","[CLASS:Button; INSTANCE:2]") $x = $k[0] $y = $k[1] WinActivate ($winTitle,"An ActiveX control on this page might be unsafe to interact with other parts of the page. Do you want to allow this interaction?") ControlFocus($winTitle,"An ActiveX control on this page might be unsafe to interact with other parts of the page. Do you want to allow this interaction?","[CLASS:Button; INSTANCE:2]") ControlClick($winTitle, "","[CLASS:Button; INSTANCE:2]","primary",1,$x,$y) ControlSend($winTitle, "", "[CLASS:Button; INSTANCE:2]", "{ENTER}", 0) EndFunc Here is the console output.
      >"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\AWD10.1.au3" /UserParams +>12:44:11 Starting AutoIt3Wrapper v.17.224.935.0 SciTE v.3.7.3.0 Keyboard:00000409 OS:WIN_10/ CPU:X64 OS:X64 Environment(Language:0409) CodePage:0 utf8.auto.check:4 +> SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE UserDir => C:\Users\april\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper SCITE_USERHOME => C:\Users\april\AppData\Local\AutoIt v3\SciTE >Running AU3Check (3.3.14.2) from:C:\Program Files (x86)\AutoIt3 input:C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\AWD10.1.au3 +>12:44:11 AU3Check ended.rc:0 >Running:(3.3.14.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\AWD10.1.au3" --> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop "C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\AWD10.1.au3" (31) : ==> Subscript used on non-accessible variable.: $x = $k[0] $x = $k^ ERROR ->12:44:24 AutoIt3.exe ended.rc:1 +>12:44:24 AutoIt3Wrapper Finished. >Exit code: 1 Time: 13.76 I also tried a simpler code.
      include <IE.au3> #include <MsgBoxConstants.au3> #include <WinAPI.au3> _IECreate("C:\Users\april\Documents\Logo\JRB\AutoIt\AutoBOT\AWD10\sampleAWD10.html",0,1,0) Local $oIE = _IEAttach("", "instance", 1) _IELoadWait($oIE) $k = ControlGetPos("[CLASS:#32770]", "","[CLASS:Button; INSTANCE:2]") $x = $k[0] $y = $k[1] ControlClick("[CLASS:#32770]", "","[CLASS:Button; INSTANCE:2]","primary",1,$x,$y) Local $oLastName = _IEGetObjByName($oIE, "Text4") Local $oGetItem = _IEGetObjByName($oIE, "getitem") _IEAction($oGetItem, "click") MsgBox($MB_SYSTEMMODAL, "Form Element Value", _IEFormElementGetValue($oGetItem)) There's no error in the console output for the 2nd code I have tried.
      And here is the Window Info for the ActiveX prompt.
      >>>> Window <<<< Title: Internet Explorer Class: #32770 Position: 580, 338 Size: 376, 146 Style: 0x94C808C4 ExStyle: 0x00010101 Handle: 0x00000000001516FA >>>> Control <<<< Class: Button Instance: 2 ClassnameNN: Button2 Name: Advanced (Class): [CLASS:Button; INSTANCE:2] ID: 1 Text: &Yes Position: 184, 77 Size: 80, 22 ControlClick Coords: 37, 12 Style: 0x50010000 ExStyle: 0x00000004 Handle: 0x00000000001215DE >>>> Mouse <<<< Position: 228, 120 Cursor ID: 0 Color: 0xFFFFFF >>>> StatusBar <<<< >>>> ToolsBar <<<< >>>> Visible Text <<<< &No An ActiveX control on this page might be unsafe to interact with other parts of the page. Do you want to allow this interaction? &Yes >>>> Hidden Text <<<<