Jump to content
Gal

Mouse click and sends in the background

Recommended Posts

Gal

I've been searching these past few days for anything that would help me automate in the background, I've made a few things work but for a couple decade old programs I can't get the control IDs, which is resulting in difficulties.

 

I've tried using the _MouseClickPlus method which calls the user32 dll sendmessage with the mouse opcodes, this worked like a charm for a new adobe apollo runtime environment I was checking.

In that case, it didn't have any control IDs or anything else, and somehow all it needed to work was the title to obtain the winhandle from.

However this doesn't work for many windows apps, not sure why.

 

All of the following problems are under windows 7

1) In paint, if I use

WinGetHandle("Untitled - Paint")

it returns zeroes, anyone have any idea why? I've gotten around this by using the title "[CLASS:MSPaintApp]", however I'm still interested in this response.

2) Next I was looking to make it click somewhere, this:

ControlClick("[CLASS:MSPaintApp]", "", "[CLASS:Afx:00000000FF0C0000:8; INSTANCE:1]", "left", 1, 275, 152)

worked, although a better solution for me would be supplying only the title / handle and coords from the window start and not from the control start, and having it click on any control.

Is there any way to do this?

3) In Paint, if I click / hold for a second the alt key, I can see the keyboard shortcuts, for example, alt + H would trigger the home tab. to what control do I use ControlSend to so that "!hp" would call "Alt h" and then select the pencil with "p", I couldn't make this work when the main whiteboard is the controlID and neither when the top toolbar is the control ID, like so:

ControlSend("[CLASS:MSPaintApp]", "", "[CLASS:NetUIHWND; INSTANCE:1]", "!hp")

4) Continuing the previous topic, in one of my older programs, alt + F for example doesn't trigger the File at the top, so I would need to send a ControlClick to it, however it doesn't have any control info, you can look at the SciTE editor, going over the file/edit/search.. with the window info tool, there's nothing about the control.

How can I send a ControlClick to the Search button in this case? Simply supplying the window is not enough, and I have a feeling ControlClick never works without specifying a control ID.

A solution to my second question would be a solution to the fourth as well :/

 

Any tips will be greatly appreciated,

Thanks!

g3trans

 

 

Share this post


Link to post
Share on other sites
Jfish

For #1:  I think you may not have the window activated.  Try this to get the handle and see if it works for you ...

run("mspaint.exe")
sleep(1000)
WinActivate("Untitled - Paint")
$paintHandle=WinGetHandle("Untitled - Paint")
ConsoleWrite(@CRLF&$paintHandle&@CRLF)

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

Edited by Jfish
  • Like 1

Build your own poker game with AutoIt: pokerlogic.au3 | Learn To Program Using FREE Tools with AutoIt

Share this post


Link to post
Share on other sites
Jos

Why would the Window need to be active to get its handle ?

This is working fine for me so what do you get?:

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : WinGetHandle("Untitled - Paint") = ' & WinGetHandle("Untitled - Paint") & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

Jos :)

  • Like 1

Visit the SciTE4AutoIt3 Download page for the latest versions  - Beta files                                How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
Gal

You're right, not sure why that didn't work for me the other day. 

Any idea on my other questions? 

 

Basically I would ideally be able to send a ControlClick anywhere in the window, but for now, the simplified problem would be in the SciTE editor, sending a click to open the "Search" options and then clicking "Find" for example

This would be like sending Ctrl + F if the window was on top and activated, but in the background I'm trying to go about it with ControlClick.

 

Any thoughts? 

Gal

 

 

 

Share this post


Link to post
Share on other sites
Gal

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

I'm checking out the IUI now, as for sending send instead of controlsend, that's not an option for me as I need to perform the whole automation in the background.

I'll update with my impression of the IUI

Share this post


Link to post
Share on other sites
Gal

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

This is pretty cool, the simplyspy, but it doesn't work in the background, to requires me to send one of their methods _UIA_Action with setfocus before it can perform a click on the object I recorded.

Maybe there's a way to use the object I found with the simplespy and send it to the controlclick?

 

Like using the below methods to find an object by title and class and control type, and then forwarding that to control click?

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=C:\Users\King\Desktop\simplyspy\exploratoryAutoit.au3 - SciTE-Lite [4 of 4];controltype:=UIA_WindowControlTypeId;class:=SciTEWindow", $treescope_children)
 ;_UIA_Action($oP1,"setfocus")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Application;controltype:=UIA_MenuBarControlTypeId;class:=", $treescope_children)
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Search.mainwindow", "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Search;controltype:=UIA_MenuControlTypeId;class:=#32768", $treescope_children)
 ;_UIA_Action($oP0,"setfocus")
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Replace...  Ctrl+H.mainwindow", "title:=Replace...  Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Replace... Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")

 

Share this post


Link to post
Share on other sites
Gal

The programs I want to be running in the background need to talk to com ports and actual hardware, 

I can imagine a setup where the physical computer translates the com ports input / output into tcp/ip commands to talk to a virtual machine and then that virtual machine translates it back into comm port emulation but that is too complex for me to even grasp how I would go about getting that to happen ;)

 

Right now there's a machine dedicated to running the autoit, which is less than idea because if I were to want to run several of these at the same time I'd need a machine for each run, and more external hardware which is pricier as well.

 

 

I'm sure that it's possible to send these clicks / sends in the background, I just can't find the solution.

 

Thanks for the tips though 

 

Share this post


Link to post
Share on other sites
Karthi

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

Edited by Karthi

Share this post


Link to post
Share on other sites
takebacker
6 hours ago, Karthi said:

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

 

I haven't found a way to do something like this. If you're automating, you eventually are forced to use blockinput because if you use the mouse or keyboard you're going to directly interfere with whatever you're trying to do. You need mouse in X,Y position? Move your mouse while this is happening and it won't be where you need it. Automating while doing work yourself isn't something that autoit can do.

Maybe a virtual machine would work, but i have no experience working with it. I don't think it would work.

Share this post


Link to post
Share on other sites
JLogan3o13

@Karthi traditionally you would use ControlSend or ControlClick to interact with a window, even when it is no active. I believe the suggestion for IUI was because the OP was unable to do this. This is another reason why it is better to begin your own thread rather than tagging a question onto the end of someone else's; your needs may be different from those of the OP.

I would suggest you start with a detailed explanation on what application you're trying to automate, what the window looks like, what you find when you use the AutoIt Window Info Tool (located in the same directory where you installed AutoIt) etc. The more info you give us the better we can offer assistance.


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • MaximusCZ
      By MaximusCZ
      Hello, and apologies if this have been discussed before, but I googled and couldnt find anything relevant.
      I have a problem where I try to send a control clink into to a window. 
      Part of window I try to send clicks to looks like this . The black border is showing when I mouse over it Au3Info, I get following information:

      I am testing its interaction with code

      The result I get is that it makes control clicks trough first line, opens and closes menus, as expected.
      When It gets to second line, it does not click a single icon. I can tell that coords are correct as the dropdown menus (No view active) gets focused(altough not rolled out). Same behaviour happens if roughly 40<y<60, so I know I am hitting good height. You can look at how it behaves here: output.webm 
      Dropdown 1 focuses at around 8 sec mark, and 13 sec mark it changes to second dropdown, defocusing first one, seemingly not registering any clinks inbetween.
       
      Have you ever encountered similar behaviour? I need to click the Key button, first on second line, with controlclick (or any other way not hijacking the mouse). I cant trigger that button by keyboard and There is no other control having control over it.
      Help
       
    • SkysLastChance
      By SkysLastChance
      Run("notepad.exe") GLOBAL $One = ('Line1' & _ 'Line2' & _ 'Line3' & _ 'Line4' & _ 'Line5') ClipPut($One) $Notepad = WinWait("[CLASS:Notepad]", "", 10) ControlSend("Untitled - Notepad", "", "", ("^v")) Instead of having this paste as Line1Line2Line3Line4Line5
      How can I make it paste it into notepad as (without double spacing.)
      Line1
      Line2
      Line3
      Line4
      Line5
    • Skysnake
      By Skysnake
      I need to automate a specific GUI from a legacy system written in C.
      The spy tool gives me this

      I use this code
      Local $swintit = WinGetTitle("[active]") ConsoleWrite("active Window " & $swintit & @CRLF) If $swintit = "FILES NOTES" Then ; If $swintit = "FILES NOTES" Then ; got Window, now atomate ConsoleWrite("got Window " & @CRLF) ; now focus, then click :) ;EditPaste ;ControlCommand("FILES NOTES", "", 404, $acno) ; paste acno GUICtrlSetState(404, $GUI_Focus) ControlCommand("FILES NOTES", "", 404, $acno) ; paste acno GUICtrlSetState(401, $GUI_Focus) ControlClick("FILES NOTES", "", 401, $acno) ; click okay EndIf ; If $swintit = "FILES NOTES" Then The idea is to paste the acno value into the edit box, then click an Okay button.
      I get a console write that the active Window is as expected, but my loop completes without ever writing anything into the GUI.
      I also tried the control ID as text "404" - same result
      What am I missing?
      Skysnake
       
    • JNutt
      By JNutt
       

       
       
      I'm trying to use mouseclick in a panel above.  I want to click on a list item, such as 'Estimated'.  (I've given up on trying to use control click for this).  I'm using window info coordinates from 'Control' tab.

      ; click 'Columns' WinActivate("Innovaya Studio with Sage (Archtectural 2017_V1.invx") MouseClick("primary", 292, 88, 1, 1) WinActivate("Select Browsing Properties") Sleep(1000) MouseClick("primary", 27, 70, 0,0) Sleep(2000) Send("{TAB}{TAB}{TAB}{TAB}{TAB}") Sleep(1000) Send("{ENTER}") ControlClick( But when I used the code Autoit doesn't click where the coordinates are.  In the scipts it is the second MouseClick that won't work.  Any ideas?
    • Miliardsto
      By Miliardsto
      Hello I got this script works below and I want to do not move cursor on screen when action is performed.
      Opt("MouseCoordMode", 1) ; cause it gets whole screen coords Local $x, $y Local $search = _ImageSearch('item.bmp', 0, $x, $y, 0) If $search = 1 Then MouseMove($x, $y,0) MouseClick("right", $x, $y)) MouseMove($xMiddle, $yMiddle,0) MouseClick("left", $xMiddle, $yMiddle) EndIf I changed this above to this below and what happen is. It clicks right button and then left button but not move mouse to $x $y and then to $xMiddle $yMiddle.
      Opt("MouseCoordMode", 1) ; cause it gets whole screen coords Local $x, $y Local $search = _ImageSearch('item.bmp', 0, $x, $y, 0) If $search = 1 Then ControlClick("","",0,"secondary",1,$x,$y) ControlClick("","",0,"primary",1,$xMiddle,$yMiddle) EndIf If needed I got handle in var $hwnd
      Please tell me how parameters in ControlClick would like be
×