method for interaction with hidden window(s)

[garbb]

Here is a method for running a windows application (that may create any number of windows) completely hidden while still being able to interact with it with autoit (controlsend(), controlclick(), etc...) and get a screencap of any of the windows. Basically this is accomplished by running the application on a hidden desktop (credit to Decipher) and then running a separate instance of autoit also on the hidden desktop to interact with the application. This is kind of a goofy hack, but it seems to work.

hiddenDesktopInteract.au3:

expandcollapse popup

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseUpx=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <WinAPI.au3>
#include <WinAPIEx.au3>
#include <GDIPlus.au3>

;1st command line param that will launch seperate instance
Const $hdiParam = 'hiddenDesktopInteract'

;######DO NOT PUT ANY OTHER CODE ABOVE THIS THAT CALLS A 2ND INSTANCE OF THE SCRIPT OR ELSE YOU MAY END UP CALLING INFINITE INSTANCES AND CRASHING WINDOWS!!!!!!
;seperate process for hidden desktop interaction
;1st param should be 'hiddenDesktopInteract' and 2nd param is name of func to call for interaction code
;ex: "autoit.exe script.au3 hiddenDesktopInteract interactionfunc" or "compiledscript.exe hiddenDesktopInteract interactionfunc"
if $CmdLine[0] = 2 AND $CmdLine[1] = $hdiParam Then EXIT Call($CmdLine[2])

;credit to Decipher in thread https://www.autoitscript.com/forum/topic/152515-start-a-process-hidden/#comment-1095173 for code for running process on a hidden desktop
Func _hiddenDesktopInteract_run($sProgram, $sInteractionFunc, $sCommand = '')
    ;if the interaction function does not exist then don't do anything
    If Not IsFunc(Execute($sInteractionFunc)) Then
        ConsoleWrite('error: interaction function "' & $sInteractionFunc & '" for "' & $sProgram & '" does not exist' & @CRLF)
        Return
    EndIf

    ;create structs
    Local $tStartupInfo = DllStructCreate($tagStartupInfo)
    DllStructSetData($tStartupInfo, "Size", DllStructGetSize($tStartupInfo))
    Local $tProcessInfo_targetApp = DllStructCreate($tagPROCESS_INFORMATION)
    Local $tProcessInfo_interactionScript = DllStructCreate($tagPROCESS_INFORMATION)

    ; Create Desktop
    Local $hDesktop = _WinAPI_CreateDesktop('AutoItHidden', BitOR($DESKTOP_CREATEWINDOW, $DESKTOP_SWITCHDESKTOP))
    If Not $hDesktop Then
        MsgBox(0, 'Error', 'Unable to create desktop.')
        Exit
    EndIf

    ; Prep Process Info
    Local $nSuccess = DllStructSetData($tStartupInfo, "Desktop", _WinAPI_CreateString("AutoItHidden"))

    ;run target program on hidden desktop and get PID
    _WinAPI_CreateProcess('', '"' & $sProgram & '"' & ($sCommand ? ' ' & $sCommand : ''), 0, 0, 0, 0x00000200, 0, 0, DllStructGetPtr($tStartupInfo), DllStructGetPtr($tProcessInfo_targetApp))
    Local $aPID_targetApp = DllStructGetData($tProcessInfo_targetApp, 'ProcessID')
    ConsoleWrite('!>target app PID:' & $aPID_targetApp & @CRLF)

    ;run instance of this script on hidden desktop to interact with target program
    Local $sParams = '"' & @ScriptFullPath & '" ' & $hdiParam & ' ' & $sInteractionFunc
    Switch @Compiled
        case True
            Local $sAppName = @ScriptFullPath
            Local $sCommandLine = $sParams
        case False
            Local $sAppName = @AutoItExe
            Local $sCommandLine = '"' & @AutoItExe & '" ' & $sParams
    EndSwitch
    _WinAPI_CreateProcess('', $sCommandLine, 0, 0, 0, 0x00000200, 0, 0, DllStructGetPtr($tStartupInfo), DllStructGetPtr($tProcessInfo_interactionScript))

    ;and get PID of interaction instance so we can wait until it is finished
    Local $iPID_interactionScript = DllStructGetData($tProcessInfo_interactionScript, 'ProcessID')
    ConsoleWrite('!>Interaction Script PID: ' & $iPID_interactionScript & @CRLF)

    ;wait until interaction script instance is finished and get exit code (which is return value from interaction function)
    ProcessWaitClose($iPID_interactionScript)
    Local $exitcode_interactionScript = @extended
    ConsoleWrite("!>Interaction Script Exit Code: " & $exitcode_interactionScript & @CRLF)
    Local $sOutput = StdoutRead($iPID_interactionScript)
    ConsoleWrite("!>Interaction Script Stdout: " & $sOutput & @CRLF)

    ;close hidden desktop
    Local $aRet = DllCall("User32.dll", "int", "CloseDesktop", "handle", $hDesktop)
    ConsoleWrite("!>Close Desktop: " & $aRet[0] & @CRLF) ; Non-Zero is successfull!

    Return $exitcode_interactionScript
EndFunc

Func _hiddenDesktopInteract_cap($title, $imgfile, $Left = 0, $Top = 0, $Right = -1, $Bottom = -1)
    _GDIPlus_Startup()
    $hWnd = WinGetHandle($title)
    $iWidth = _WinAPI_GetWindowWidth($hWnd)
    $iHeight = _WinAPI_GetWindowHeight($hWnd)
    $hDDC = _WinAPI_GetDC($hWnd)
    $hCDC = _WinAPI_CreateCompatibleDC($hDDC)
    $hBMP = _WinAPI_CreateCompatibleBitmap($hDDC, $iWidth, $iHeight)
    _WinAPI_SelectObject($hCDC, $hBMP)
    DllCall("User32.dll", "int", "PrintWindow", "hwnd", $hWnd, "hwnd", $hCDC, "int", 0)
    _WinAPI_ReleaseDC($hWnd, $hDDC)
    _WinAPI_DeleteDC($hCDC)
    $hBMPclone = _GDIPlus_BitmapCreateFromHBITMAP($hBMP)
    if $Right = -1 then
        $iX = _GDIPlus_ImageGetWidth($hBMPclone) - $Left
    Else
        $iX = $Right - $Left
    EndIf
    if $Bottom = -1 then
        $iY = _GDIPlus_ImageGetHeight($hBMPclone) - $Top
    Else
        $iY = $Bottom - $Top
    EndIf
    ;convert from 32bit bitmap to 24bit bitmap b/c 32bit bitmap cannot display correctly in autoit GUI for some reason
    $hClone = _GDIPlus_BitmapCloneArea($hBMPclone, $Left, $Top, $iX, $iY, $GDIP_PXF24RGB)
    _GDIPlus_ImageSaveToFile($hClone, $imgfile)
    _WinAPI_DeleteObject($hBMP)
    _WinAPI_DeleteObject($hBMPclone)
    _WinAPI_DeleteObject($hClone)
    _GDIPlus_Shutdown()
EndFunc

 

hiddenDesktopInteract_EXAMPLES.au3:

#include "hiddenDesktopInteract.au3"

;test function for interacting with notepad app on hidden desktop
func notepadtest()
    opt('winwaitdelay', 0)  ;to make it run a little bit faster
    WinWait('[CLASS:Notepad]')
    $hwin = WinGetHandle('[CLASS:Notepad]')
    ControlSend($hwin, '', '[CLASS:Edit; INSTANCE:1]', 'hello{ENTER}')
    WinMenuSelectItem($hwin, '', '&Edit', 'Time/&Date')
    _hiddenDesktopInteract_cap($hwin, @DesktopDir & '\notepad_cap.jpg')
    ControlSend($hwin, '', '[CLASS:Edit; INSTANCE:1]', '^a')
    ControlSend($hwin, '', '[CLASS:Edit; INSTANCE:1]', '{DELETE}')
    WinClose($hwin)
EndFunc

;test function for interacting with cmd.exe on hidden desktop
func cmdtest()
    opt('winwaitdelay', 0)  ;to make it run a little bit faster
    WinWait('[CLASS:ConsoleWindowClass]')
    $hwin = WinGetHandle('[CLASS:ConsoleWindowClass]')
    ControlSend($hwin, '', '', 'dir{ENTER}')
    sleep(500)
    _hiddenDesktopInteract_cap($hwin, @DesktopDir & '\cmd_cap.jpg')
    ControlSend($hwin, '', '', 'exit{ENTER}')
EndFunc

_hiddenDesktopInteract_run("notepad.exe", 'notepadtest')
_hiddenDesktopInteract_run("cmd.exe", 'cmdtest')

The example script demonstrates running a hidden instance of notepad and cmd.exe and interacting with it via controlclick() and controlsend() and then capturing a screencap of the window before closing it.

I made this because some applications ignore the @SW_HIDE parameter when launching them and show a window anyways, and also because I wanted to be able to screencap windows while making them hidden so they wouldn't steal focus.

To use this you must first write a function that will interact with the target application/window and then call _hiddenDesktopInteract_run() with the name of the application as the first parameter and the name of the function as the second parameter. You can also use _hiddenDesktopInteract_cap() within your function to store an image of the hidden window to a file. Because the way that this works is that ONLY the function that you specify will be run on the hidden desktop, you must make sure that your function does not depend on any variables/code that is outside of the function or it will fail.

I was also trying to find some way to transfer information from the autoit instance that runs on the hidden desktop to the main instance of the script using consolewrite() and getting the stdout stream of the other instance but I couldn't figure it out. (I was looking at the code here but I couldn't get it to work) Maybe someone knows how to get it working... Right now the exit code of the hidden desktop autoit instance is used to transfer the return code of the function but this only works with integers and so isn't really a good method.

Anyways, I hope this will be useful for someone. Or maybe someone knows a better/less hacky way to achieve the same thing...

Tested with autoit 3.3.14.2 on Windows 7 64bit SP1 and Windows 10 64bit 1511 (OS Build 10586.164)

hiddenDesktopInteract.au3

hiddenDesktopInteract_EXAMPLES.au3

Edited July 16, 2016 by garbb
updated to work on windows 10

[dmob]

Windows 10 Pro, AutoIt 3.1.14.2, ran script uncompiled:

 

[garbb]

Thanks for testing it on windows 10. I have updated the script and tested it successfully on windows 10 now. Also for some reason the windows calculator app won't work with this on windows 10 so I changed the example script to use notepad and cmd.exe as examples.

Let me know if it works for you now.

Edited July 16, 2016 by garbb

[masvil]

It works great on Windows 10 Pro - AutoIt 3.1.14.2, compiled and uncompiled. Good job!

Any test on Windows 8.x?

[Mixalhadas]

Hi, I'm new to autoit, but I've managed to build some simple scripts to personal use. I've stumbled upon this udf and I must say that I'm amazed, great job! 

 

I'm having some trouble running a pixel color search loop inside it, but I can probably bypass it if I can get a full screen shot or the size of the desktop created.

Any advice would be great.

 

[rajeevrra]

Greetings,
  I tried the below code: , I am getting  : #include "hiddenDesktopInteract.au3" - Error opening File. Please help me resolve if possible. As I am relatively new in AuToIT. Thanks and take care

<snip>

 

Edited June 6, 2021 by Melba23
Removed code

[Melba23]

rajeevrra,

Answer my question in your own thread before you go posting all over the place.

M23