DllCall: how to pass pointer to handle for UpdateProcThreadAttribute

[qwerty12]

AutoIt 3.3.16.0 x64 (Portable) on Windows 11

As part of trying to leverage the Windows functionality of letting me select a parent for a process, I'm trying to call UpdateProcThreadAttribute with a pointer to handle. The most obvious way I can think of doing it ("ptr*", $hParentProcess) doesn't seem to work. Not that I really expected anything else, but neither does substituting "ptr*" for "handle*". UpdateProcThreadAttribute does return True, however. But CreateProcess fails with 6: the handle is invalid. The script in its entirety:

expandcollapse popup

#include <ProcessConstants.au3>
#include <WinAPIProc.au3>

Local Const $iParentPID = _WinAPI_GetParentProcess()
Local Const $tStartup = DllStructCreate("dword Size;ptr Reserved1;ptr Desktop;ptr Title;dword X;dword Y;dword XSize;dword YSize;dword XCountChars;dword YCountChars;dword FillAttribute;dword Flags;word ShowWindow;word cbReserved2;ptr Reserved2;handle StdInput;handle StdOutput;handle StdError;ptr AttributeList")
DllStructSetData($tStartup, "Size", DllStructGetSize($tStartup))
Local $iCreateProcFlags = 0
Local $tAttribBuf = 0
Local $ptAttribBuf = 0
Local $hParentProcess = 0
If $iParentPID <> 0 Then
    $hParentProcess = _WinAPI_OpenProcess($PROCESS_CREATE_PROCESS, False, $iParentPID)
    If $hParentProcess Then
        Local $iSize = DllCall("kernel32.dll", "boolean", "InitializeProcThreadAttributeList", "ptr", 0, "dword", 1, "dword", 0, "ulong_ptr*", 0)[4]
        If $iSize <> 0 Then
            Local Const $PROC_THREAD_ATTRIBUTE_PARENT_PROCESS = 0x00020000
            Local Const $EXTENDED_STARTUPINFO_PRESENT = 0x00080000
            $tAttribBuf = DllStructCreate("char[" & $iSize & "]")
            $ptAttribBuf = DllStructGetPtr($tAttribBuf)
            $x = DllCall("kernel32.dll", "boolean", "InitializeProcThreadAttributeList", "ptr", $ptAttribBuf, "dword", 1, "dword", 0, "ulong_ptr*", $iSize)
            $x = DllCall("kernel32.dll", "boolean", "UpdateProcThreadAttribute", "ptr", $ptAttribBuf, "dword", 0, "dword_ptr", $PROC_THREAD_ATTRIBUTE_PARENT_PROCESS, "ptr*", $hParentProcess, "ulong_ptr", BinaryLen($hParentProcess), "ptr", 0, "Ptr", 0)
            MsgBox(0, $x[0], $x[0])
            DllStructSetData($tStartup, "AttributeList", $ptAttribBuf)
            $iCreateProcFlags = $EXTENDED_STARTUPINFO_PRESENT
        EndIf
    EndIf
EndIf

Local Const $tProcess = DllStructCreate($tagPROCESS_INFORMATION)
If _WinAPI_CreateProcess('', @SystemDir & '\notepad.exe', 0, 0, False, $iCreateProcFlags, 0, 0, $tStartup, $tProcess) Then
    Local Const $hProcess = DllStructGetData($tProcess, "hProcess")
    Local Const $hThread = DllStructGetData($tProcess, "hThread")
    _WinAPI_CloseHandle($hThread)
    _WinAPI_CloseHandle($hProcess)
Else
    _WinAPI_ShowLastError()
EndIf

If $ptAttribBuf Then
    DllCall("kernel32.dll", "none", "DeleteProcThreadAttributeList", "ptr", $ptAttribBuf)
EndIf

If $hParentProcess Then
    _WinAPI_CloseHandle($hParentProcess)
EndIf

 

If, however, I add the handle to a struct and then use DllStructGetPtr to get a pointer to that handle, the script works fine - the process starts and does choose the right parent process:

--- 
+++ 
@@ -12,16 +12,17 @@
    $hParentProcess = _WinAPI_OpenProcess($PROCESS_CREATE_PROCESS, False, $iParentPID)
    If $hParentProcess Then
        Local $iSize = DllCall("kernel32.dll", "boolean", "InitializeProcThreadAttributeList", "ptr", 0, "dword", 1, "dword", 0, "ulong_ptr*", 0)[4]
        If $iSize <> 0 Then
            Local Const $PROC_THREAD_ATTRIBUTE_PARENT_PROCESS = 0x00020000
            Local Const $EXTENDED_STARTUPINFO_PRESENT = 0x00080000
-           $tAttribBuf = DllStructCreate("char[" & $iSize & "]")
-           $ptAttribBuf = DllStructGetPtr($tAttribBuf)
+           $tAttribBuf = DllStructCreate("char buf[" & $iSize & "];handle h")
+           $ptAttribBuf = DllStructGetPtr($tAttribBuf, "buf")
+           DllStructSetData($tAttribBuf, "h", $hParentProcess)
            $x = DllCall("kernel32.dll", "boolean", "InitializeProcThreadAttributeList", "ptr", $ptAttribBuf, "dword", 1, "dword", 0, "ulong_ptr*", $iSize)
-           $x = DllCall("kernel32.dll", "boolean", "UpdateProcThreadAttribute", "ptr", $ptAttribBuf, "dword", 0, "dword_ptr", $PROC_THREAD_ATTRIBUTE_PARENT_PROCESS, "ptr*", $hParentProcess, "ulong_ptr", BinaryLen($hParentProcess), "ptr", 0, "Ptr", 0)
+           $x = DllCall("kernel32.dll", "boolean", "UpdateProcThreadAttribute", "ptr", $ptAttribBuf, "dword", 0, "dword_ptr", $PROC_THREAD_ATTRIBUTE_PARENT_PROCESS, "ptr", DllStructGetPtr($tAttribBuf, "h"), "ulong_ptr", BinaryLen($hParentProcess), "ptr", 0, "Ptr", 0)
            MsgBox(0, $x[0], $x[0])
            DllStructSetData($tStartup, "AttributeList", $ptAttribBuf)
            $iCreateProcFlags = $EXTENDED_STARTUPINFO_PRESENT
        EndIf
    EndIf
 EndIf

Is there any way I can avoid copying the handle to a struct and go with my original approach? Thanks.

Edited August 16, 2022 by qwerty12

[LarsJ]

Replace "ptr*", $hParentProcess with "ptr", $hParentProcess. The fact that a data type is provided with an asterisk means that it's an output parameter. Without an asterisk, it's an input parameter. Since the current parameter here is an input parameter, the data type must be without an asterisk.

The reason why some of your code works isn't because $hParentProcess is stored in a DllStruct. It's because the parameter data type is specified correctly as an input parameter.

[qwerty12]

Thank you for your post. That was one of the first things I tried. AutoIt crashes when I use "ptr", $hParentProcess which I'm not surprised at because as far I know, UpdateProcThreadAttribute wants a pointer to the variable, like

"HANDLE hParentProcess = OpenProcess(PROCESS_CREATE_PROCESS, FALSE, iParentPID); UpdateProcThreadAttribute(ptAttribBuf, 0, PROC_THREAD_ATTRIBUTE_PARENT_PROCESS, &hParentProcess, sizeof(hParentProcess), NULL, NULL);"

I was hoping "ptr*" would be the equivalent of &hParentProcess in C, as per the DllCall help ("Add * to the end of another type to pass it by reference. For example "int*" passes a pointer to an "int" type."), but CreateProcess fails. The DllStruct stuff does work with DllStructGetPtr returning a pointer to a variable (which is why I know to pass that as "ptr" to the UpdateProcThreadAttribute DllCall) and I'm OK sticking with that as a workaround, but as someone who knows nothing about AutoIt, I was hoping for something quicker.

Edited August 17, 2022 by qwerty12