Simulating Mouse Events for On-Screen Keyboard?

[hunte922]

Hi all,

How can I simulate mouse events such as clicking in the OSK in Windows 8.1? I've tried ControlClick, SendInput, and other DLLCalls regarding mouse_event. None of them work.
 

>>>> Window <<<<
Title:  
Class:  IPTip_Main_Window
Position:   489, 809
Size:   941, 271
Style:  0x94000000
ExStyle:    0x08000088
Handle: 0x0000000000570C26

>>>> Control <<<<
Class:  DirectUIHWND
Instance:   0
ClassnameNN:    DirectUIHWND0
Name:   
Advanced (Class):   [CLASS:DirectUIHWND; INSTANCE:0]
ID: 
Text:   
Position:   0, 0
Size:   941, 271
ControlClick Coords:    45, 235
Style:  0x56000000
ExStyle:    0x00000020
Handle: 0x00000000002608EC

(Attached is an image of the On-Screen Keyboard in question.)

onscreen-keyboard.bmp

Edited July 21, 2015 by hunte922

[JLogan3o13]

Hi, hunte922, can you explain why would you want to manipulate the OSK rather than just sending the keys directly? Seems it would be far less code..

[hunte922]

Hi, hunte922, can you explain why would you want to manipulate the OSK rather than just sending the keys directly? Seems it would be far less code..

I am trying to switch to the numberpad whenever the OSK is opened:

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      ControlClick("[CLASS:Shell_TrayWnd]", "", "TIPBand")
      If $offon And $numberpad And $kbf <> 0 Then
         Local $wPos = WinGetPos($kbf, "")
         Local $ps = PixelSearch(0, Abs($wPos[3]), Abs($wPos[0]/2), Abs($wPos[3]/2), 0x1D1C21, 0, 5, $kbf);Searches for the bottom-left button, I couldn't get its position directly
         If @error = 0 Then
            ;???
         EndIf
      EndIf
   EndIf
EndFunc

 

Edited July 21, 2015 by hunte922
Sorry for the lack of usable code, the script I'm using KB() in contains business-related stuff.

[BrewManNH]

But why are you using the OSK at all? For what reason are you using it that can't be done some other way, probably easier?

[hunte922]

But why are you using the OSK at all? For what reason are you using it that can't be done some other way, probably easier?

I'm using a touch device with no keyboard. I might be able to download/create a third-party OSK, but the risk (reliability) vs reward (auto switch to numberpad) isn't that great.

[BrewManNH]

What are you doing with the on screen keyboard, where is the input going to, have you tried using ControlSend instead of the OSK? You're trying to automate something that can probably be done without it, if you'd just explain what it is you're doing. So far all we know is that you're trying to put the OSK into numpad mode.

[hunte922]

What are you doing with the on screen keyboard, where is the input going to, have you tried using ControlSend instead of the OSK? You're trying to automate something that can probably be done without it, if you'd just explain what it is you're doing. So far all we know is that you're trying to put the OSK into numpad mode.

I'm only trying to open it and then switch it to the numberpad; nothing else automated. From thereon is user input.

[MuffinMan]

I don't have a Win8 box, but I checked with a colleague and he suggested looking into setting the registry before opening TabTip.

Try looking here...

HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7

KeyboardLayoutPreference = 1 DWORD (should set numlock mode on, 0 should turn it off)

Hope that helps!

[hunte922]

I don't have a Win8 box, but I checked with a colleague and he suggested looking into setting the registry before opening TabTip.

Try looking here...

HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7

KeyboardLayoutPreference = 1 DWORD (should set numlock mode on, 0 should turn it off)

Hope that helps!

Unfortunately, that's not the view I was hoping for. That changes the entire layout of the keyboard instead of switching to the numberpad. My preferred effect happens when pressing the bottom-left button entitled "&123".

 

Edit: I tried changing it anyway during script use, and none of these combinations actually work (#RequireAdmin is enabled:)

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      If $offon Then
         Local $dw = 0x00000000
         If $numberpad Then
            $dw = 0x00000001
         EndIf
         RegWrite("HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference", "REG_DWORD", $dw)
         RegWrite("HKEY_USERS\" &  _GetSID() & "\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         ;MsgBox(0,"",RegRead("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\TouchKeyboard\Users\" & _GetSID(), "KeyboardLayoutPreference"))
      EndIf
      ControlClick("[CLASS:Shell_TrayWnd]", "", "TIPBand")
   EndIf
EndFunc

Func _GetSID($sComputerName = @ComputerName, $sUsername = @UserName)
   Local $oWMIService = ObjGet("winmgmts:{impersonationLevel=impersonate}!//" & $sComputerName & "/root/cimv2")
   Local $oColItems = $oWMIService.ExecQuery("Select * From Win32_UserAccount")
   If IsObj($oColItems) Then
      For $oObjectItem In $oColItems
         If $sUsername = $oObjectItem.Name Then
            Return $oObjectItem.SID
         EndIf
      Next
   EndIf
   Return SetError(1, 0, 0)
EndFunc

 

Edit 2: The below script works. Although it's not what I originally wanted, it will suffice (albeit with a longer delay than the last script.)

Func KB($offon, $numberpad)
   Sleep(1)
   Local $kbf = _WinAPI_FindWindow("IPTip_Main_Window", "")
   Local $state = False
   If $kbf <> 0 And (WinGetState($kbf) = 7 Or WinGetState($kbf) = 15) Then
      $state = True
   EndIf
   If $state <> $offon Then
      If $offon Then
         Local $dw = 0x00000000
         If $numberpad Then
            $dw = 0x00000001
         EndIf
         RegWrite("HKEY_CURRENT_USER\Software\Microsoft\TabletTip\1.7", "KeyboardLayoutPreference", "REG_DWORD", $dw)
         ShellExecute("C:\Program Files\Common Files\microsoft shared\ink\TabTip.exe", "", "C:\Program Files\Common Files\microsoft shared\ink")
         ProcessWait("TabTip.exe")
      Else
         ProcessClose("TabTip.exe")
         ProcessWaitClose("TabTip.exe")
      EndIf
   EndIf
EndFunc

But my question still stands, and I still heavily prefer to switch to the other numberpad due to the clear, large buttons.

Edited July 21, 2015 by hunte922

[PACaleala]

You might want to study the code (found searching the forum ) : https://www.autoitscript.com/forum/topic/98007-my-virtual-keyboard/

[hunte922]

You might want to study the code (found searching the forum ) : https://www.autoitscript.com/forum/topic/98007-my-virtual-keyboard/

Since Windows 10 removes the keypad in MuffinMan's solution, it looks like I'll have to make a keypad that is integrated into my entire script (if I still won't be able to simulate mouse events on the OSK correctly.) The script you posted won't work due to how it's constructed and can't easily be incorporated into my own.

[TonyG]

I am also trying to spoof mouse events to the touch keyboard in Win 10.  I have a touch screen and I am writing a program that translates the touches into mouse events.  My program works great with standard Windows programs.  For example, I can open Notepad, open menus in Notepad, double click to open programs or folders, etc.  But when I bring up the virtual keyboard and touch it, my mouse cursor is placed over the keyboard but the keys are not pressed.  I've tried spoofing the mouse events with a combination of SetCursorPos and mouse_event, and I also tried SendInput.  Neither work with the touch keyboard.

It appears as if the virtual keyboard is at some lower level, almost like a driver.  It will respond to mouse events created at a driver level with an actual mouse, but it will not respond to mouse events created at the user level. 

How can I spoof mouse events that will activate keys on the touch keyboard?