Jump to content
Gal

Mouse click and sends in the background

Recommended Posts

Gal

I've been searching these past few days for anything that would help me automate in the background, I've made a few things work but for a couple decade old programs I can't get the control IDs, which is resulting in difficulties.

 

I've tried using the _MouseClickPlus method which calls the user32 dll sendmessage with the mouse opcodes, this worked like a charm for a new adobe apollo runtime environment I was checking.

In that case, it didn't have any control IDs or anything else, and somehow all it needed to work was the title to obtain the winhandle from.

However this doesn't work for many windows apps, not sure why.

 

All of the following problems are under windows 7

1) In paint, if I use

WinGetHandle("Untitled - Paint")

it returns zeroes, anyone have any idea why? I've gotten around this by using the title "[CLASS:MSPaintApp]", however I'm still interested in this response.

2) Next I was looking to make it click somewhere, this:

ControlClick("[CLASS:MSPaintApp]", "", "[CLASS:Afx:00000000FF0C0000:8; INSTANCE:1]", "left", 1, 275, 152)

worked, although a better solution for me would be supplying only the title / handle and coords from the window start and not from the control start, and having it click on any control.

Is there any way to do this?

3) In Paint, if I click / hold for a second the alt key, I can see the keyboard shortcuts, for example, alt + H would trigger the home tab. to what control do I use ControlSend to so that "!hp" would call "Alt h" and then select the pencil with "p", I couldn't make this work when the main whiteboard is the controlID and neither when the top toolbar is the control ID, like so:

ControlSend("[CLASS:MSPaintApp]", "", "[CLASS:NetUIHWND; INSTANCE:1]", "!hp")

4) Continuing the previous topic, in one of my older programs, alt + F for example doesn't trigger the File at the top, so I would need to send a ControlClick to it, however it doesn't have any control info, you can look at the SciTE editor, going over the file/edit/search.. with the window info tool, there's nothing about the control.

How can I send a ControlClick to the Search button in this case? Simply supplying the window is not enough, and I have a feeling ControlClick never works without specifying a control ID.

A solution to my second question would be a solution to the fourth as well :/

 

Any tips will be greatly appreciated,

Thanks!

g3trans

 

 

Share this post


Link to post
Share on other sites
Jfish

For #1:  I think you may not have the window activated.  Try this to get the handle and see if it works for you ...

run("mspaint.exe")
sleep(1000)
WinActivate("Untitled - Paint")
$paintHandle=WinGetHandle("Untitled - Paint")
ConsoleWrite(@CRLF&$paintHandle&@CRLF)

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

Edited by Jfish
  • Like 1

Build your own poker game with AutoIt: pokerlogic.au3 | Learn To Program Using FREE Tools with AutoIt

Share this post


Link to post
Share on other sites
Jos

Why would the Window need to be active to get its handle ?

This is working fine for me so what do you get?:

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : WinGetHandle("Untitled - Paint") = ' & WinGetHandle("Untitled - Paint") & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

Jos :)

  • Like 1

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource        Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites
Gal

You're right, not sure why that didn't work for me the other day. 

Any idea on my other questions? 

 

Basically I would ideally be able to send a ControlClick anywhere in the window, but for now, the simplified problem would be in the SciTE editor, sending a click to open the "Search" options and then clicking "Find" for example

This would be like sending Ctrl + F if the window was on top and activated, but in the background I'm trying to go about it with ControlClick.

 

Any thoughts? 

Gal

 

 

 

Share this post


Link to post
Share on other sites
Gal

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

On the last issue, if you can't identify the individual properly the send won't take.  Try regular send instead of controlsend.

I'm checking out the IUI now, as for sending send instead of controlsend, that's not an option for me as I need to perform the whole automation in the background.

I'll update with my impression of the IUI

Share this post


Link to post
Share on other sites
Gal

 

I think the new paint may have ribbon-like controls that are harder to individually identify.  The IUIAutomation MS framework referenced here: https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/ may help.

This is pretty cool, the simplyspy, but it doesn't work in the background, to requires me to send one of their methods _UIA_Action with setfocus before it can perform a click on the object I recorded.

Maybe there's a way to use the object I found with the simplespy and send it to the controlclick?

 

Like using the below methods to find an object by title and class and control type, and then forwarding that to control click?

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=C:\Users\King\Desktop\simplyspy\exploratoryAutoit.au3 - SciTE-Lite [4 of 4];controltype:=UIA_WindowControlTypeId;class:=SciTEWindow", $treescope_children)
 ;_UIA_Action($oP1,"setfocus")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Application;controltype:=UIA_MenuBarControlTypeId;class:=", $treescope_children)
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Search.mainwindow", "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Search;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")
 
 Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Search;controltype:=UIA_MenuControlTypeId;class:=#32768", $treescope_children)
 ;_UIA_Action($oP0,"setfocus")
 ;~ First find the object in the parent before you can do something
 ;~$oUIElement=_UIA_getObjectByFindAll("Replace...  Ctrl+H.mainwindow", "title:=Replace...  Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Replace... Ctrl+H;ControlType:=UIA_MenuItemControlTypeId", $treescope_subtree)
 _UIA_action($oUIElement,"click")

 

Share this post


Link to post
Share on other sites
Gal

The programs I want to be running in the background need to talk to com ports and actual hardware, 

I can imagine a setup where the physical computer translates the com ports input / output into tcp/ip commands to talk to a virtual machine and then that virtual machine translates it back into comm port emulation but that is too complex for me to even grasp how I would go about getting that to happen ;)

 

Right now there's a machine dedicated to running the autoit, which is less than idea because if I were to want to run several of these at the same time I'd need a machine for each run, and more external hardware which is pricier as well.

 

 

I'm sure that it's possible to send these clicks / sends in the background, I just can't find the solution.

 

Thanks for the tips though 

 

Share this post


Link to post
Share on other sites
Karthi

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

Edited by Karthi

Share this post


Link to post
Share on other sites
takebacker
6 hours ago, Karthi said:

is it possible to do ui automation in the background? without activating the windows? so, i can do something else in the foreground. if so, how to do that?

 

I haven't found a way to do something like this. If you're automating, you eventually are forced to use blockinput because if you use the mouse or keyboard you're going to directly interfere with whatever you're trying to do. You need mouse in X,Y position? Move your mouse while this is happening and it won't be where you need it. Automating while doing work yourself isn't something that autoit can do.

Maybe a virtual machine would work, but i have no experience working with it. I don't think it would work.

Share this post


Link to post
Share on other sites
JLogan3o13

@Karthi traditionally you would use ControlSend or ControlClick to interact with a window, even when it is no active. I believe the suggestion for IUI was because the OP was unable to do this. This is another reason why it is better to begin your own thread rather than tagging a question onto the end of someone else's; your needs may be different from those of the OP.

I would suggest you start with a detailed explanation on what application you're trying to automate, what the window looks like, what you find when you use the AutoIt Window Info Tool (located in the same directory where you installed AutoIt) etc. The more info you give us the better we can offer assistance.


√-1 2^3 ∑ π, and it was delicious!

How to get your question answered on this forum!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • SkysLastChance
      By SkysLastChance
      I must be missing something. I have had this problem multiple times. Why does F6 key not send 6 times?  
      Not working 
      ControlSend("[CLASS:#32770]", "", "", ("{F6 6}")) Working
      ControlSend("[CLASS:#32770]", "", "", ("{F6}")) ControlSend("[CLASS:#32770]", "", "", ("{F6}")) ControlSend("[CLASS:#32770]", "", "", ("{F6}")) ControlSend("[CLASS:#32770]", "", "", ("{F6}")) ControlSend("[CLASS:#32770]", "", "", ("{F6}")) ControlSend("[CLASS:#32770]", "", "", ("{F6}"))  
    • tentacole
      By tentacole
      Afternoon!
      This is my first post, so I apologize if this is in the wrong place. 
      I've created a while loop to click in a certain area of an application, and have the y axis change at the end of each loop. My loop continues to click at x:27, x:10, even though the $y is adding 15 at the end of each loop. I did a Send($x) and Send($y) into a Notepad to see if the $y had changed after each loop, and the 15 was being added to $y each loop. (If this makes sense)
       
      I'm unsure where I'm going wrong, and would be extremely grateful if someone can point me in the correct direction to fix this. 
      Local $rDirectory = "H:\oDemandProject\fList.txt" Local $rLine = _FileCountLines($rDirectory) Local $x = 27 Local $y = 10 While $rline > 0 WinActivate("OnDemand", "-> 1") WinWaitActive("OnDemand", "-> 1") ControlClick("OnDemand", "", "[CLASS:AfxFrameOrView120u; INSTANCE:1]", "left", 4, $x, $y) $y += 15 $rLine -= 1 WEnd  
    • faustf
      By faustf
      hi guys
      i have a script  like this,  in windows 7 work perfect , in windows 10 not  work  , some one can explain me  why ??'
      #include <Array.au3> #include <MsgBoxConstants.au3> Global $aGPathFoto= FileReadToArray (@ScriptDir & "\TEMP\PathFoto.txt") Local $ctrl = 0 While 1 Local $hWnd = WinWait("[CLASS:#32770]", "", 10) If WinExists("[CLASS:#32770]") Then If $ctrl < 2 Then ControlSend($hWnd, "Choose File to Upload", "[CLASS:Edit; INSTANCE:1]", $aGPathFoto[0]&"{Enter}") Exit $ctrl += 1 Else ExitLoop EndIf EndIf WEnd $aGpathfoto = C:\Users\Public\Pictures\Sample Pictures\Chrysanthemum.jpg
    • Yirrlaar
      By Yirrlaar
      #Include <WinAPI.au3> local $ClassName = _WinAPI_GetClassName(WinGetHandle(WinGetTitle("[ACTIVE]"))) ControlClick(WinGetTitle("[ACTIVE]"), "", "[CLASS:"&$ClassName&"; INSTANCE:1]", "primary") When I am running this script it just gives me a ConsoleWrite output of "0x001D04A8"
      When using "AutoIT v3 Window Info" all the info I need for the ControlClick is there and displayed perfectly as if I typed it in normally.
      Upon click the button it should be clicking it, I see the following inn the console window:
      If I replace the $ClassName with the actual ClassName of "WindowsForms10.BUTTON.app.0.3296db7_r14_ad1" it works just fine, any ideas???
    • Tats
      By Tats
      Hi everyone,
      I have a problem with ControlClick function. It doesn't work no matter how I tried all the possible combination of the parameters.
      Below is the scripts of possible parameter combination:
      ;ControlClick("title","text","controlID ","button","clicks","x","y") ;title= Title, Class, Handle ;controlID= Class, ClassNN, Instance ;Title here is empty when I check it with AutoIt Info ControlClick("", "", "[ScreenKeyboard::Button18]", "left", "1", "230","890") ControlClick("", "", "[CLASS:ScreenKeyboard::Button; INSTANCE:18]", "left", "1", "230","890") ControlClick("", "", "[INSTANCE:18]", "left", "1", "230","890") ;Title here is the Class of the window ControlClick("[ScreenKeyboard::SKFrame]", "", "[ScreenKeyboard::Button18]", "left", "1", "230","890") ControlClick("[ScreenKeyboard::SKFrame]", "", "[CLASS:ScreenKeyboard::Button; INSTANCE:18]", "left", "1", "230","890") ControlClick("[ScreenKeyboard::SKFrame]", "", "[INSTANCE:18]", "left", "1", "230","890") ;Title here is the Handle of the window ControlClick("[0x000907A0]", "", "[ScreenKeyboard::Button18]", "left", "1", "230","890") ControlClick("[0x000907A0]", "", "[CLASS:ScreenKeyboard::Button; INSTANCE:18]", "left", "1", "230","890") ControlClick("[0x000907A0]", "", "[INSTANCE:18]", "left", "1", "230","890") Am I coding it correctly?
      Thanks
×