Jump to content

Mouse click and sends in the background


Gal
 Share

Recommended Posts

I've been searching these past few days for anything that would help me automate in the background, I've made a few things work but for a couple decade old programs I can't get the control IDs, which is resulting in difficulties.

 

I've tried using the _MouseClickPlus method which calls the user32 dll sendmessage with the mouse opcodes, this worked like a charm for a new adobe apollo runtime environment I was checking.

In that case, it didn't have any control IDs or anything else, and somehow all it needed to work was the title to obtain the winhandle from.

However this doesn't work for many windows apps, not sure why.

 

All of the following problems are under windows 7

1) In paint, if I use

WinGetHandle("Untitled - Paint")

it returns zeroes, anyone have any idea why? I've gotten around this by using the title "[CLASS:MSPaintApp]", however I'm still interested in this response.

2) Next I was looking to make it click somewhere, this:

ControlClick("[CLASS:MSPaintApp]", "", "[CLASS:Afx:00000000FF0C0000:8; INSTANCE:1]", "left", 1, 275, 152)

worked, although a better solution for me would be supplying only the title / handle and coords from the window start and not from the control start, and having it click on any control.

Is there any way to do this?

3) In Paint, if I click / hold for a second the alt key, I can see the keyboard shortcuts, for example, alt + H would trigger the home tab. to what control do I use ControlSend to so that "!hp" would call "Alt h" and then select the pencil with "p", I couldn't make this work when the main whiteboard is the controlID and neither when the top toolbar is the control ID, like so:

ControlSend("[CLASS:MSPaintApp]", "", "[CLASS:NetUIHWND; INSTANCE:1]", "!hp")

4) Continuing the previous topic, in one of my older programs, alt + F for example doesn't trigger the File at the top, so I would need to send a ControlClick to it, however it doesn't have any control info, you can look at the SciTE editor, going over the file/edit/search.. with the window info tool, there's nothing about the control.

How can I send a ControlClick to the Search button in this case? Simply supplying the window is not enough, and I have a feeling ControlClick never works without specifying a control ID.

A solution to my second question would be a solution to the fourth as well :/

 

Any tips will be greatly appreciated,

Thanks!

g3trans

 

 

Link to comment
Share on other sites

For #1:  I think you may not have the window activated.  Try this to get the handle and see if it works for you ...

run("mspaint.exe")
sleep(1000)
WinActivate("Untitled - Paint")
$paintHandle=WinGetHandle("Untitled - Paint")
ConsoleWrite(@CRLF&$paintHandle&@CRLF)

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

Edited by Jfish

Build your own poker game with AutoIt: pokerlogic.au3 | Learn To Program Using FREE Tools with AutoIt

Link to comment
Share on other sites

  • Developers

Why would the Window need to be active to get its handle ?

This is working fine for me so what do you get?:

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : WinGetHandle("Untitled - Paint") = ' & WinGetHandle("Untitled - Paint") & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

Jos :)

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

You're right, not sure why that didn't work for me the other day. 

Any idea on my other questions? 

 

Basically I would ideally be able to send a ControlClick anywhere in the window, but for now, the simplified problem would be in the SciTE editor, sending a click to open the "Search" options and then clicking "Find" for example

This would be like sending Ctrl + F if the window was on top and activated, but in the background I'm trying to go about it with ControlClick.

 

Any thoughts? 

Gal

 

 

 

Link to comment
Share on other sites

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

I'm checking out the IUI now, as for sending send instead of controlsend, that's not an option for me as I need to perform the whole automation in the background.

I'll update with my impression of the IUI

Link to comment
Share on other sites

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

This is pretty cool, the simplyspy, but it doesn't work in the background, to requires me to send one of their methods _UIA_Action with setfocus before it can perform a click on the object I recorded.

Maybe there's a way to use the object I found with the simplespy and send it to the controlclick?

 

Like using the below methods to find an object by title and class and control type, and then forwarding that to control click?

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=C:\Users\King\Desktop\simplyspy\exploratoryAutoit.au3 - SciTE-Lite [4 of 4];controltype:=UIA_WindowControlTypeId;class:=SciTEWindow", $treescope_children)
 ;_UIA_Action($oP1,"setfocus")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Application;controltype:=UIA_MenuBarControlTypeId;class:=", $treescope_children)
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Search.mainwindow", "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Search;controltype:=UIA_MenuControlTypeId;class:=#32768", $treescope_children)
 ;_UIA_Action($oP0,"setfocus")
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Replace...  Ctrl+H.mainwindow", "title:=Replace...  Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Replace... Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")

 

Link to comment
Share on other sites

The programs I want to be running in the background need to talk to com ports and actual hardware, 

I can imagine a setup where the physical computer translates the com ports input / output into tcp/ip commands to talk to a virtual machine and then that virtual machine translates it back into comm port emulation but that is too complex for me to even grasp how I would go about getting that to happen ;)

 

Right now there's a machine dedicated to running the autoit, which is less than idea because if I were to want to run several of these at the same time I'd need a machine for each run, and more external hardware which is pricier as well.

 

 

I'm sure that it's possible to send these clicks / sends in the background, I just can't find the solution.

 

Thanks for the tips though 

 

Link to comment
Share on other sites

  • 5 months later...
6 hours ago, Karthi said:

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

 

I haven't found a way to do something like this. If you're automating, you eventually are forced to use blockinput because if you use the mouse or keyboard you're going to directly interfere with whatever you're trying to do. You need mouse in X,Y position? Move your mouse while this is happening and it won't be where you need it. Automating while doing work yourself isn't something that autoit can do.

Maybe a virtual machine would work, but i have no experience working with it. I don't think it would work.

Link to comment
Share on other sites

  • Moderators

@Karthi traditionally you would use ControlSend or ControlClick to interact with a window, even when it is no active. I believe the suggestion for IUI was because the OP was unable to do this. This is another reason why it is better to begin your own thread rather than tagging a question onto the end of someone else's; your needs may be different from those of the OP.

I would suggest you start with a detailed explanation on what application you're trying to automate, what the window looks like, what you find when you use the AutoIt Window Info Tool (located in the same directory where you installed AutoIt) etc. The more info you give us the better we can offer assistance.

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...