Sign in to follow this  
Followers 0
hunte922

Simulating Mouse Events for On-Screen Keyboard?

12 posts in this topic

#1 ·  Posted (edited)

Hi all,

How can I simulate mouse events such as clicking in the OSK in Windows 8.1? I've tried ControlClick, SendInput, and other DLLCalls regarding mouse_event. None of them work.
 

>>>> Window <<<<
Title:  
Class:  IPTip_Main_Window
Position:   489, 809
Size:   941, 271
Style:  0x94000000
ExStyle:    0x08000088
Handle: 0x0000000000570C26

>>>> Control <<<<
Class:  DirectUIHWND
Instance:   0
ClassnameNN:    DirectUIHWND0
Name:   
Advanced (Class):   [CLASS:DirectUIHWND; INSTANCE:0]
ID: 
Text:   
Position:   0, 0
Size:   941, 271
ControlClick Coords:    45, 235
Style:  0x56000000
ExStyle:    0x00000020
Handle: 0x00000000002608EC

(Attached is an image of the On-Screen Keyboard in question.)

onscreen-keyboard.bmp

Edited by hunte922

Share this post


Link to post
Share on other sites



Hi, hunte922, can you explain why would you want to manipulate the OSK rather than just sending the keys directly? Seems it would be far less code..


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Hi, hunte922, can you explain why would you want to manipulate the OSK rather than just sending the keys directly? Seems it would be far less code..

I am trying to switch to the numberpad whenever the OSK is opened:

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      ControlClick("[CLASS:Shell_TrayWnd]", "", "TIPBand")
      If $offon And $numberpad And $kbf <> 0 Then
         Local $wPos = WinGetPos($kbf, "")
         Local $ps = PixelSearch(0, Abs($wPos[3]), Abs($wPos[0]/2), Abs($wPos[3]/2), 0x1D1C21, 0, 5, $kbf);Searches for the bottom-left button, I couldn't get its position directly
         If @error = 0 Then
            ;???
         EndIf
      EndIf
   EndIf
EndFunc

 

Edited by hunte922
Sorry for the lack of usable code, the script I'm using KB() in contains business-related stuff.

Share this post


Link to post
Share on other sites

But why are you using the OSK at all? For what reason are you using it that can't be done some other way, probably easier?


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

But why are you using the OSK at all? For what reason are you using it that can't be done some other way, probably easier?

I'm using a touch device with no keyboard. I might be able to download/create a third-party OSK, but the risk (reliability) vs reward (auto switch to numberpad) isn't that great.

Share this post


Link to post
Share on other sites

What are you doing with the on screen keyboard, where is the input going to, have you tried using ControlSend instead of the OSK? You're trying to automate something that can probably be done without it, if you'd just explain what it is you're doing. So far all we know is that you're trying to put the OSK into numpad mode.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

What are you doing with the on screen keyboard, where is the input going to, have you tried using ControlSend instead of the OSK? You're trying to automate something that can probably be done without it, if you'd just explain what it is you're doing. So far all we know is that you're trying to put the OSK into numpad mode.

I'm only trying to open it and then switch it to the numberpad; nothing else automated. From thereon is user input.

Share this post


Link to post
Share on other sites

I don't have a Win8 box, but I checked with a colleague and he suggested looking into setting the registry before opening TabTip.

Try looking here...

HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7

KeyboardLayoutPreference = 1 DWORD (should set numlock mode on, 0 should turn it off)

Hope that helps!

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

I don't have a Win8 box, but I checked with a colleague and he suggested looking into setting the registry before opening TabTip.

Try looking here...

HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7

KeyboardLayoutPreference = 1 DWORD (should set numlock mode on, 0 should turn it off)

Hope that helps!

Unfortunately, that's not the view I was hoping for. That changes the entire layout of the keyboard instead of switching to the numberpad. My preferred effect happens when pressing the bottom-left button entitled "&123".

 

Edit: I tried changing it anyway during script use, and none of these combinations actually work (#RequireAdmin is enabled:)

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      If $offon Then
         Local $dw = 0x00000000
         If $numberpad Then
            $dw = 0x00000001
         EndIf
         RegWrite("HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_USERS\" &  _GetSID() & "\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         ;MsgBox(0,"",RegRead("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference"))
      EndIf
      ControlClick("[CLASS:Shell_TrayWnd]", "", "TIPBand")
   EndIf
EndFunc

Func _GetSID($sComputerName = @ComputerName, $sUsername = @UserName)
   Local $oWMIService = ObjGet("winmgmts:{impersonationLevel=impersonate}!//" & $sComputerName & "/root/cimv2")
   Local $oColItems = $oWMIService.ExecQuery("Select * From Win32_UserAccount")
   If IsObj($oColItems) Then
      For $oObjectItem In $oColItems
         If $sUsername = $oObjectItem.Name Then
            Return $oObjectItem.SID
         EndIf
      Next
   EndIf
   Return SetError(1, 0, 0)
EndFunc

 

Edit 2: The below script works. Although it's not what I originally wanted, it will suffice (albeit with a longer delay than the last script.)

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      If $offon Then
         Local $dw = 0x00000000
         If $numberpad Then
            $dw = 0x00000001
         EndIf
         RegWrite("HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         ShellExecute("C:\Program Files\Common Files\microsoft shared\ink\TabTip.exe", "", "C:\Program Files\Common Files\microsoft shared\ink")
         ProcessWait("TabTip.exe")
      Else
         ProcessClose("TabTip.exe")
         ProcessWaitClose("TabTip.exe")
      EndIf
   EndIf
EndFunc

But my question still stands, and I still heavily prefer to switch to the other numberpad due to the clear, large buttons.

Edited by hunte922

Share this post


Link to post
Share on other sites

You might want to study the code (found searching the forum ) : https://www.autoitscript.com/forum/topic/98007-my-virtual-keyboard/

Since Windows 10 removes the keypad in MuffinMan's solution, it looks like I'll have to make a keypad that is integrated into my entire script (if I still won't be able to simulate mouse events on the OSK correctly.) The script you posted won't work due to how it's constructed and can't easily be incorporated into my own.

Share this post


Link to post
Share on other sites

I am also trying to spoof mouse events to the touch keyboard in Win 10.  I have a touch screen and I am writing a program that translates the touches into mouse events.  My program works great with standard Windows programs.  For example, I can open Notepad, open menus in Notepad, double click to open programs or folders, etc.  But when I bring up the virtual keyboard and touch it, my mouse cursor is placed over the keyboard but the keys are not pressed.  I've tried spoofing the mouse events with a combination of SetCursorPos and mouse_event, and I also tried SendInput.  Neither work with the touch keyboard.

It appears as if the virtual keyboard is at some lower level, almost like a driver.  It will respond to mouse events created at a driver level with an actual mouse, but it will not respond to mouse events created at the user level. 

How can I spoof mouse events that will activate keys on the touch keyboard?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0