brad3260

ControlClick in apps

2 posts in this topic

#1 ·  Posted

Hello everyone, I have been beating my head against the wall all day today over this and am hoping someone can help. I work for a company who makes assistive technology and I have multiple consumers who want to use the Mail app included in Windows 10. Most of these consumers are using eye tracking technology so the MouseClick is not an option since their eyes are constantly controlling the mouse cursor so it would be a battle between that function and their eyes. And hence I need to use ControlClick. I'm having problems with it though, I can't get it to click anything in the app. I've been left clicking on things with no apparent luck. However, when I changed it to right click, something very interesting happens, the window menu (the one you get when you hit ALT + SPACE) opens every time the ControlClick runs, regardless of the X,Y coordinates as if it is right clicking at the top left of the window every time.

Based on the Window Info, the entire app is one big control whose position is 0,40 and the size is 1440,838.

Does anybody have any ideas why I can apparently only click on the top left corner of the app? Any direction is greatly appreciated!

 

Opt('MouseClickDownDelay', 100)
Opt('WinTitleMatchMode', 2)
Local $hWnd = WinActivate('Gmail')
$inc = 30
For $hi = 1 To 10 Step 1
    $hadj = $hi * $inc
    For $i = 1 To 10 Step 1
        $adj = $i * $inc
        $j = ControlClick($hWnd, "", "[CLASS:ApplicationFrameInputSinkWindow; INSTANCE:1]", "Right", 1, $hadj, $adj)
        MouseMove($hadj, $adj, 3)
    Next
Next

 

Share this post


Link to post
Share on other sites



#2 ·  Posted

You'll need to use something like IUIAutomation for newer Windows 10 Apps like Mail, Calculator etc...

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • breakbadsp
      By breakbadsp
      AutoIT AU3info doeas not detect all gui objects uniquely for .NET GUIs developed in C#.
      this is not working now i am using COM windows approach for this, But its very difficult.
      Please let me know if anyone has done it before.
    • KB505
      By KB505
      Hello,
       
      I'm starting a script in AutoIT to automize a task in a software. I need to click a several Controls. 
      I'm just starting to learn AutoIT, and discovered that we can use the Window Info tool (au3info) to identify controls precisely and targetting safely a specific button.
      However, when I try it, I realize that most of the controls in my toolbar have the same Control Info (No ID, same class, same instance, ...). The only thing that changes is the ControlClicks Coord but I don't think that it's interesting, I don't see the difference with the Mouse Coordinates. Besides, I was aiming to write a stable script that would work even if the button changes a little bit of place for example. 
      Should I use another tool that Window Info tool to detect this Controls ? Is there another way to identify a Control ? 
      Thank you in advance for your help
       
    • SorryButImaNewbie
      By SorryButImaNewbie
      Hello,
      I'm developing again (everybody ruuun! )
      I would like to develope a script that goes through basicly every user control on a window, and log things that happens, and maybe do some screenshots. I did something like this before.
      My problem, which I would like to avoid this time (to improve my understanding and skill) , was that when I was unable to get a ControlID or handler or anything, I simply did some math and clicked on the coordinates it should have been (for example, maxing the window, and knowing the initial set up I was ablo to calculate given control position). I know that this is a bad solution for a number of reasons. 
      Now I got authorization to install autoIT here, and i started to the work, AU3Info was unable to find anything on the window (this could be a problem, since autoIT doesn't see anything on it then, if I understood the help file) So I got the SimpleSpy script (source:
      )
      I added a bit of code to the original to display ID as well, what I received is this:
      Mouse position is retrieved 115-207
      At least we have an element title: [ADD] class: [Button] ID: [50000] (<-- coded this to display ID here as well)
      Having the following values for all properties: 
      Title is: <ADD>    Class   := <Button>    controltype:= <UIA_ButtonControlTypeId>    ,<50000>    , (0000C350)    10;187;120;35
      *** Parent Information top down ***
      3: Title is: <Compass>    Class   := <Window>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    -8;-8;1936;1056
      "Title:=Compass;controltype:=UIA_WindowControlTypeId;class:=Window""    
      2: Title is: <>    Class   := <MainView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;23;1920;1017
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=MainView""    
      1: Title is: <>    Class   := <TileNavigationView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;23;1920;967
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=TileNavigationView""    
      0: Title is: <>    Class   := <AreasView>    controltype:= <UIA_CustomControlTypeId>    ,<50025>    , (0000C369)    0;132;1920;858
      "Title:=;controltype:=UIA_CustomControlTypeId;class:=AreasView""
       
      so far I wrote this script:
      WinActivate('Test') ;It works!! :D first official interaction Sleep(1000) ;1 sec sleep to be sure ControlClick('Test', '', '50000') If @error Then MsgBox($MB_SYSTEMMODAL, 'Error', 'ControlClick error') EndIf Sleep(1000) MsgBox(1,"Tracer message", 'ControlClick has happened') ;MouseClick() ;ControlCommand() AutoIt activates the window, but the click on the given button doesnt happen (I tried to write 50000 without ' ' on ID).
      M'I doing the @error part correctly ? (no error Msg has been displayed), sorry I rarely use AutoIT and seems to forget less and less after each neglect, but still I'm far from a proffessional
       
      Any help or suggestion is welcome, thank you for your time and insight!
    • Akashrai
      By Akashrai
      I want to automate a client application but auto it info tool is not detecting the GUI components. I don't know what to do. please help me out. Screenshot  of first login page of application is in attachment 
       

    • automaterk
      By automaterk
      Hello all! Quick question that I couldn't find in the forums. Does anyone that uses AutoItx4Java have any insight on using outside functions(non-standard AutoIT) in Java. I can't find anything in the forums that explains how this might be possible, but I feel like it probably is.
      An example would be that I am trying to use the OCR functions from an outside library in AutoIT. This works fine in the AutoIT environment, but it would be amazing if I could also use these functions in my java code to keep it all in one place.
       
      Any help or nudge in the right direction would be great! Thanks! If you need any more information, please let me know!