Jump to content

Android Visual Control by USB. NO SDK Needed, NO SCREEN or TOUCH Screen Needed.

Recommended Posts


This is an small script to Control Android Devices by USB using adb, you can use it with phones with a Cracked Screen or not working Touch Screen.

What it Does:
- Show your Android Screen on your Desktop.
- Allow you to Swipe or Tap on the screen.
- Allow you to send BACK button event.
- Allow you to send WakeUp event.
- Allow you to move in List Menu using Swipe gestures or Mouse Wheel Forward or Backward.
- Refresh Screen with Wheel Click(Wheel Down)(To update your device screen on your desktop view).
- Allow you to unlock your Device with Pattern only 3 points pattern for now.

What it does not:
- Complex Unlock Pattern not supported yet I'm working on that.
- Complex Swipe gesture not supported yet, only straight lines.
- PC Keyboard to write SMS or Email not Supported yet, I'm working on that.
- Volume UP or Down not supported yet I will add it soon.
- No configuration interface or file yet, keys and actions are pre set on script code. I will change that.

What it Requires:
- In order to Compile the Script you need 3 files:
adb.exe, AdbWinApi.dll, AdbWinUsbApi.dll
- In order to use the script without these files or without compile it you need to copy that files to user Temp Dir, or windows Temp dir I mean @TempDir.

Keys Used and Notes:
- Left Click will tap on the screen.
- Keep Left Click down, move the mouse and release it with perform a Swipe operation.(Mouse Drag = Swipe).
- Right Click will go back.
- Enter on the keyboard will send WakeUp event(Power Button)
- Wheel Down will Refresh the screen on your desktop, also F5.(Take last pictures of your android device screen)

IMPORTANT: There is an $scale variable on the script that scale down your Phone screenshot image to a good resolution to show on your Desktop, I use 1.5 of scale factor because my phone resolution is 480x800 and my Desktop Screen is 1366x768 and that scale factor is good enough for me.(I will calculate that in the future to make it automatic).

IMPORTANT 2: If you feel is very slow feel free to decrease Sleep() values from 400 to 200, that will be faster but you will need to Refresh with Wheel Click after you perform certain actions.

Thanks and Credits:
UDFs\MouseControl.au3 Base on Low Level Mouse Hook by @_Kurt all credits to him.
UDFs\GUICtrlPicPNG.au3 Based on PNG work around by @UEZ from the help file, All credits to him.

Kind Regards


Edited by alien4u
Adding Swipe Operation Instruction.
  • Like 1

Share this post

Link to post
Share on other sites

cool , how can i integrat it inside an application am developing to help ime in my word?

Share this post

Link to post
Share on other sites
On 10/10/2016 at 5:22 AM, WormIT said:

Using screen capture to show Android's screens and control will make your Android system lag. :D

One of the purposes of this code is to learn and also access to an Android Device with a broken screen, is not really about efficiency.

Share this post

Link to post
Share on other sites

Use the TeamViewer UDF instead of screen cap?

2015 - Still no flying cars, instead blankets with sleeves.

Share this post

Link to post
Share on other sites

Bad idea to use Team Viewer as an intermediator, not only does it not work while the phone is not connected to the internet, you have to totally trust them with your device. I think there are better ways to do this... something like a VNC Server, I only know droidVNC but it does not work on my phone... But screen cap does work for now :)

AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)


My contributions to the AutoIt Community

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) mean to do that to anybody!!!



Share this post

Link to post
Share on other sites
8 hours ago, Lakes said:

Use the TeamViewer UDF instead of screen cap?

The idea here is you will not need any software installed on your Android Device with the broken screen.
You only need USB Debugging Enable that all.

As an example, a family:
Man, Woman, Kid they all enable USB debugging and connect at least once to this computer or any computer they can run the Script.
6 months or a year from that moment, if the computer operating system was not re-installed or the phone system, if they get a not working touch-screen or display they will be able to connect to their phones and save data or do some things... regardless what they have installed on their phones.

Kind Regards

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • nacerbaaziz
      By nacerbaaziz

      Hi dear
      This is the first include file I designed
      This work is especially directed to NVDA free screen reader users
      It contains a set of functions that enable you to control the program
      Such as forcing the program to read a custom text
      Force it to stop talking
      Find out if the program is running
      And show custom text in the Braille screen
      Available functions are
      to load the dll file
      to UnLoad the dll file
      to speak a custom text
      to show custom text in the Braille screen
      to Force the NVDA  to stop talking
      to check if the NVDA is running
      important note :
      All of these functions depend on a nvdaControllerClient32.dll
      I've added it in the attachments, as well as a file for examples, and other files
      As the source and examples in other languages
      For those who wanted to download the free screen reader, this is the download link from the official website
      i hope you like this topic
      I hope you will try it and give me your opinions
      Thank you all members and administrators for their help
      now with the  Attachments
    • RHolmes
      By RHolmes
      I wrote a script a few months ago that was working at the time. When I tried it today it wouldn't retrieve any controls.  The application successfully launches, but I cannot figure out how to retrieve the control from the window that is launched. I'm on a windows 10 machine using AutoIt v3.3.14.2. The only thing I can think of that has changed is windows updates? Code is below, any help is greatly appreciated.    ; Notes: ; HandleError( handleToCheck, MsgToLogOnFailure, terminateAutoItOnFail ) : function that simply checks the handle and quits AutoIt if not present ; all of this works well FileChangeDir( $CLIENT_APPLICATION_DIR ); Run( "Client.exe" ) Local $hClient = WinWaitActive( $CLIENT_TITLE, "", 10 ) $terminateOnFail = 1 HandleError( $hClient, "LaunchClient::Error: Failed to launch client. Either timed-out or failed.", $terminateOnFail ) LogToFile( "Client launched, waiting for system to ready." ) Sleep( 5000 ) ; this part does not work ; $SYSTEM_INDICATOR is a global variable. I have tried these values: "SystemIndicatorWindow" (Text), "Qt5QWindowIcon101" (ClassNN), and ; "[CLASS:Qt5QWindowIcon; INSTANCE:101]" Local $hStatusIndicator = ControlGetHandle( $hClient, "", $SYSTEM_INDICATOR ) HandleError( $hStatusIndicator, "CheckStatus::Error: couldn't retrieve control: " & $SYSTEM_INDICATOR, $terminateOnFail ) This is what the spy reveals: 

      Edit: I just tried this code and it works for notepad++.
      FileChangeDir( "C:\Program Files\Notepad++\" ); Run( "notepad++.exe" ) Local $hNotePad = WinWaitActive( "new 1 - Notepad++", "", 10 ) If $hNotePad = 0 Or $hNotePad = -1 Then MsgBox( $MB_SYSTEMMODAL, "Error", "Error getting app handle." ) EndIf Sleep( 1000 ) Local $hNewFileBtn = ControlGetHandle( $hNotePad, "", "[CLASS:ToolbarWindow32; INSTANCE:1]" ) If $hNewFileBtn = 0 Or $hNewFileBtn = -1 Then MsgBox( $MB_SYSTEMMODAL, "Error", "Error getting button handle." ) EndIf MsgBox( $MB_SYSTEMMODAL, "Success", "Success." )  
    • rcmaehl
      By rcmaehl
      A UDF with Extended Functions for Window Management
      Fixes WinGetClassList's barbaric returning of a @LF separated string instead of an array.
      Potential Uses:
      Automating applications that change their controls' handles/classes on each launch (e.g. half of Cisco's programs)
      WindowEx.zip  (v0.4)
      10/04/2016 (v0.4): _WinGetClassNNList Fixed : Not Returning an Index when using $2D_ARRAY _WinGetClassNNList Fixed : Not Properly returning $aArray[x][1] on Classes with instances > 9 when using $2D_ARRAY 10/03/2016 (v0.3): _WinGetClassList Added : Exactly the same as WinGetClassList but returns a more civilized Array _WinGetClassNNList Added : Returns Classes and their instances in either a 1D or 2D array depending on Flags _WindowGetHandleList Renamed: _WinGetHandleList SCRIPT BREAKING! _WindowGetHandleListFromPos Renamed: _WinGetHandleListFromPos SCRIPT BREAKING! 10/01/2016 (v0.2): WindowsExConstants.au3 Added : Flags in _WindowGetHandleListFromPos _WindowGetHandleListFromPos Removed: ConsoleWrite left in during debug _WindowGetHandleListFromPos Added : Flag for if part of a Control is at $X, $Y return it as well. 10/01/2016 (v0.1): _WindowGetHandleList Added : Retrieves the handles of classes from a window. _WindowGetHandleListFromPos Added : Retrieves the handles of classes at a specific position from a window. Known and Reported Bugs:
      None reported To Do:
      To Be Decided. Opinions welcome! Upcoming Changes:
      To Be Decided.
    • BigMac121263
      By BigMac121263
      I know that this theme was discussed several times but I read all These threats with no helping result.
      I have two pieces of TEMPer1F USB Thermometer.
      It is a Standard Windows HID device with no add. Driver needed.
      With the device I got an App reading from it and showing Graphs and much more and which can handle several devices.
      With the Software came also a RDingUSB.dll file handling the communication with the device.
      I can call the dll without any error. But the Problem is how to Access the device using the DLL and Access more than one of same VID/PID
      by Serial or anything else useful.
      I have also sources in c# doing the Job, but no experience in C# to successfuly port it to AutoIT.
      So my question: Can anyone help me to realize the simple pull of the temperature value from the devices based on C# source?
      The rest to write it to MySQL DB and repeat it etc. is no Problem for me.
      The C# part with the DLL functions class:
      using System;
      using System.Runtime.InteropServices;
      namespace TEMPer
        internal class RDing
          public static extern IntPtr OpenUSBDevice(int VID, int PID);
          public static extern IntPtr CloseUSBDevice(IntPtr hDevice);
          public static extern bool WriteUSB(IntPtr hDevice, byte[] pBuffer, uint dwBytesToWrite, ref ulong lpNumberOfBytesWritten);
          public static extern bool ReadUSB(IntPtr hDevice, byte[] pBuffer, uint dwBytesToRead, ref ulong lpNumberOfBytesRead);
          public static extern ushort GetInputLength(IntPtr hDevice);
          public static extern ushort GetOutputLength(IntPtr hDevice);
          public static extern uint GetErrorMsg(ref string[] lpErrorMsg, uint dwErrorMsgSize);
      The C# code snippet working with the functions of the class above is in the attachment
      this is the interesting part where the data is sent and pulled which I
      do not understand to transcode to AutoIT
      Important my devices are the type2 (TEMPer1F) :
      So in the end I Need: Call DLL, read all devices in array
      Loop the Array until ubound(Array) and for each device in Array read the temperature value and prepare the data to
      decimal 6,1 (SQL Format) and °C
      So how to pull the value from the devices?
      Maybe somwbody finds the important part in the C#  (the code above is for different models, mine is TEMPer1F
      Or maybe someone has solved this and has maybe a uDF for this
    • JRSmile
      By JRSmile
      i searched the forum to communicate with a logitech G600 without using Shortcuts or Default Mouse Keys.
      the code below lets u use your sidekeys whenever you press down the ring key.
      before using the code you will have to unassign the keys in the logitech mouse driver software.
      The source of my code is a wacom usb script, wich was heavily modified to work.
      you might find some parts wierd, thats because this is my first attempt to communicate via raw hid. 
      #Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_Change2CUI=y #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <APISysConstants.au3> #include <GUIConstantsEx.au3> #include <StaticConstants.au3> #include <WinAPIGdi.au3> #include <WinAPIMisc.au3> #include <WinAPISys.au3> #include <WindowsConstants.au3> #cs ; #include "CommMG.au3" $mgdebug = True Global $sportSetError $tset = _ComGetPortNames() for $i = 0 to UBound($tset) -1 if $tset[$i][1] = "Serielles USB-Gerät" Then $COMPort = StringTrimLeft($tset[$i][0],3) ConsoleWrite("Teensy an Port COM" & $COMPort & " gefunden!" & @CRLF) EndIf Next $resOpen = _CommSetPort($COMPort,$sportSetError,31250, 8,0,1,0,0,0) if $resOpen = 0 then ConsoleWrite($sportSetError & @LF) ;~ Exit EndIf #ce $atagRID_DEVICE_INFO_HID = 'struct;dword VendorId;dword ProductId;dword VersionNumber;ushort UsagePage;ushort Usage;endstruct' $atagRID_INFO_HID = 'dword Size;dword Type;' & $atagRID_DEVICE_INFO_HID & ';dword Unused[2]' Opt('TrayAutoPause', 0) ; Create GUI Global $g_hForm = GUICreate('G600', 100, 25, -1, -1, BitOR($WS_CAPTION, $WS_POPUP, $WS_SYSMENU)) Local $tRID = DllStructCreate($tagRAWINPUTDEVICE) DllStructSetData($tRID, 'Flags', $RIDEV_INPUTSINK) DllStructSetData($tRID, 'hTarget', $g_hForm) DllStructSetData($tRID, 'UsagePage', 0x80) ; Logitech G600 DllStructSetData($tRID, 'Usage', 0x0A) ; _WinAPI_RegisterRawInputDevices($tRID) ; Now iterate to find other devices Local $tText, $aData = _WinAPI_EnumRawInputDevices() If IsArray($aData) Then ReDim $aData[$aData[0][0] + 1][3] $tText = DllStructCreate('wchar[256]') For $i = 1 To $aData[0][0] If _WinAPI_GetRawInputDeviceInfo($aData[$i][0], $tText, 256, $RIDI_DEVICENAME) Then $aData[$i][2] = DllStructGetData($tText, 1) Else $aData[$i][2] = '' EndIf If $aData[$i][1] = $RIM_TYPEHID Then $devInf = DllStructCreate($atagRID_INFO_HID) If _WinAPI_GetRawInputDeviceInfo($aData[$i][0], $devInf, DllStructGetSize($devInf), $RIDI_DEVICEINFO ) Then If DllStructGetData($devInf, 'VendorId') = 0x046D And DllStructGetData($devInf, 'ProductId') = 0xC24A then ; G600 VID & PID ConsoleWrite ("Device Info:-" & @CRLF) ConsoleWrite ('VendorId: ' & Hex(DllStructGetData($devInf, 'VendorId'),4) & @CRLF) ConsoleWrite ('ProductId: ' & Hex(DllStructGetData($devInf, 'ProductId'),4) & @CRLF) ConsoleWrite ('VersionNumber: ' & DllStructGetData($devInf, 'VersionNumber') & @CRLF) ConsoleWrite ('UsagePage: ' & Hex(DllStructGetData($devInf, 'UsagePage'),2) & @CRLF) ConsoleWrite ('Usage: ' & Hex(DllStructGetData($devInf, 'Usage'),2) & @CRLF) DllStructSetData($tRID, 'UsagePage', DllStructGetData($devInf, 'UsagePage')) DllStructSetData($tRID, 'Usage', DllStructGetData($devInf, 'Usage')) _WinAPI_RegisterRawInputDevices($tRID) EndIf EndIf EndIf Next EndIf ; Register WM_INPUT message GUIRegisterMsg($WM_INPUT, 'WM_INPUT') GUISetState(@SW_SHOW) Global $structHID_DATA = "struct;" & _ "dword Type;" & _ "dword Size;" & _ "handle hDevice;" & _ "wparam wParam;" & _ "dword dwSizeHid;" & _ "dword dwCount;" & _ "endstruct;" Global $structWACOM_PEN_DATA = "struct;" & _ "dword Type;" & _ "dword Size;" & _ "handle hDevice;" & _ "wparam wParam;" & _ "dword dwSizeHid;" & _ "dword dwCount;" & _ "ubyte bRawData00;" & _ "ubyte penvsEraser;" & _ "word x;" & _ "word y;" & _ "word proximity;" & _ "word pressure;" & _ "ubyte bRawData09;" & _ "ubyte bRawData10;" & _ "ubyte bRawData11;" & _ "ubyte bRawData12;" & _ "ubyte bRawData13;" & _ "ubyte bRawData14;" & _ "ubyte bRawData15;" & _ "ubyte bRawData16;" & _ "ubyte bRawData17;" & _ "ubyte bRawData18;" & _ "ubyte bRawData19;" & _ "ubyte bRawData20;" & _ "ubyte bRawData21;" & _ "ubyte bRawData22;" & _ "ubyte bRawData23;" & _ "ubyte bRawData24;" & _ "ubyte bRawData25;" & _ "ubyte bRawData26;" & _ "ubyte bRawData27;" & _ "ubyte bRawData28;" & _ "ubyte bRawData29;" & _ "ubyte bRawData30;" & _ "ubyte bRawData31;" & _ "ubyte bRawData32;" & _ "ubyte bRawData33;" & _ "ubyte tilt_ba;" & _ "ubyte bRawData35;" & _ "ubyte tilt_na;" & _ "ubyte bRawData37;" & _ "endstruct;" Do Until GUIGetMsg() = $GUI_EVENT_CLOSE Func WM_INPUT($hWnd, $iMsg, $wParam, $lParam) #forceref $iMsg, $wParam Switch $hWnd Case $g_hForm Local $tRIM = DllStructCreate($tagRAWINPUTHEADER) If _WinAPI_GetRawInputData($lParam, $tRIM, DllStructGetSize($tRIM), $RID_HEADER) Then ; Retrieves the raw input from the specified device $devType = DllStructGetData($tRIM, 'Type') $devSize = DllStructGetData($tRIM, 'Size') Else ConsoleWrite ("Device Header Retrieval Failed" & @CRLF) Return EndIf ; Now use the handle to the device to get it's name Local $tText = DllStructCreate('wchar[256]') If _WinAPI_GetRawInputDeviceInfo(DllStructGetData($tRIM, 'hDevice'), $tText, 256, $RIDI_DEVICENAME) Then $devName = DllStructGetData($tText, 1) Else ConsoleWrite ("Device Name Retrieval Failed" & @CRLF) EndIf if $devType = $RIM_TYPEHID Then $tRIM = DllStructCreate($structWACOM_PEN_DATA) If _WinAPI_GetRawInputData($lParam, $tRIM, DllStructGetSize($tRIM), $RID_INPUT) Then if DllStructGetData($tRIM, 8) = 32 Then ; filter for ring key down (G-Mode) Switch DllStructGetData($tRIM, 9) ; if in g mode assign the Keys G9 to G20 to autoit functions. case 0 Return ; key released case 1 g1() case 2 g2() case 4 g3() case 8 g4() case 16 g5() case 32 g6() case 64 g7() case 128 g8() case 256 g9() case 512 g10() case 1024 g11() case 2048 g12() Case Else ;;; ConsoleWrite(DllStructGetData($tRIM, 9) & @CRLF) EndSwitch EndIf EndIf EndIf EndSwitch sleep(50) Return $GUI_RUNDEFMSG EndFunc ;==>WM_INPUT Func g1() ;~ Send("2") ConsoleWrite("g1" & @CRLF) ;~ _CommSendString("KEY_TAB" & @LF) ;~ Sleep(random(50,100,1)) ;~ _CommSendString("KEY_RELEASE_ALL" & @LF) ;~ Sleep(random(50,100,1)) ;~ _CommSendString("KEY_2" & @LF) ;~ Sleep(random(50,100,1)) ;~ _CommSendString("KEY_RELEASE_ALL" & @LF) EndFunc Func g2() ConsoleWrite("g2" & @CRLF) EndFunc Func g3() ConsoleWrite("g3" & @CRLF) EndFunc Func g4() ConsoleWrite("g4" & @CRLF) EndFunc Func g5() ConsoleWrite("g5" & @CRLF) EndFunc Func g6() ConsoleWrite("g6" & @CRLF) EndFunc Func g7() ConsoleWrite("g7" & @CRLF) EndFunc Func g8() ConsoleWrite("g8" & @CRLF) EndFunc Func g9() ConsoleWrite("g9" & @CRLF) EndFunc Func g10() ConsoleWrite("g10" & @CRLF) EndFunc Func g11() ConsoleWrite("g11" & @CRLF) EndFunc Func g12() ConsoleWrite("g12" & @CRLF) EndFunc