CreateTimerQueueTimer

[KaFu]

Quote

AutoIt is not thread-safe, i.e. functions called in parallel threads (like by CreateTimerQueueTimer) can lead to crashes or undefined behavior.

There are cases with work-arounds using opcode execution in separate threads, e.g. transex gif animation UDF. I think this RegisterSyncCallback hack might be a good workaround at least for simple callbacks in separate threads.

[UEZ]

@KaFu what do you want to achieve? 

[KaFu]

Hi UEZ,

initially my request was about CreateTimerQueueTimer, triggered by this comment.

Quote

https://www.codeproject.com/KB/system/timers_intro.aspx
If you want a high-precision, low-overhead, non-blocking timer that will work on Windows 2000 and later, use the queue timer.

By jugadors comment I learned of the opcode hack with RegisterSyncCallback, as described by the AHK guys.

Quote

https://www.autohotkey.com/boards/viewtopic.php?f=6&t=21223
RegisterSyncCallback provides a limited solution to the problem by creating a callback which does not call the script's function directly, but instead synchronises with the script's main thread by sending a window message.

Also jugador tested it with _WinHttpSetStatusCallback and EnumWindows here.

So now it's only about making RegisterSyncCallback x64 compatible, to have a hack in the quiver for any functions using callbacks, which will crash with DllCallbackRegister. Besides the mentioned ones I think there are more functions out there, where this can be used. I think the callback in CopyFileEx also crashed with DllCallbackRegister, maybe it works with RegisterSyncCallback?

 

[jugador]

@KaFu op code worked as it is just need to add volatile

volatile Func _CallBackFunction($lParam, $TimerOrWaitFired)
    ConsoleWrite("_CallBackFunction fired = " & TimerInit() & @TAB & $lParam & @TAB & $TimerOrWaitFired & @CRLF)
EndFunc   ;==>_CallBackFunction

 

[KaFu]

Hi jugador, thanks for the feedback. The x86 code worked without the volatile for me, but I've added it to be on the safe side 👍.

The only open topic for me would be to make the opcode x64 compatible.

Just ported SMF to x64, and I made it a habit to try to make all my code x86 and x64 compatible.

Edited Monday at 09:55 AM by KaFu

[Nine]

2 hours ago, jugador said:

op code worked as it is just need to add volatile

If you mean the first attempt from @KaFu. No its not enough.  It may take a few attempts but it will break eventually. Like I said before in this thread :

On 5/30/2025 at 3:09 PM, Nine said:

I thought that using Volatile would help, but it does not.

Here a revised version of the code including x64 :

expandcollapse popup

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <Memory.au3>
#include <WindowsConstants.au3>
#include <WinAPIError.au3>
#include <WinAPIHObj.au3>
#include <WinAPISys.au3>

Global Const $WT_EXECUTELONGFUNCTION = 0x00000010 ; The callback function can perform a long wait. This flag helps the system to decide if it should create a new thread.
Global Const $WT_EXECUTEINTIMERTHREAD = 0x00000020 ; The callback function is invoked by the timer thread itself. This flag should be used only for short tasks or it could affect other timer operations.
Global Const $WT_EXECUTEINPERSISTENTTHREAD = 0x00000080 ; The callback function is queued to a thread that never terminates.

Local $a_h_CreateTimerQueue = DllCall("kernel32.dll", "handle", "CreateTimerQueue")
ConsoleWrite("CreateTimerQueue = " & $a_h_CreateTimerQueue[0] & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Local $i_TimerQueue_Start_after = 100
Local $i_TimerQueue_Repeat_after = 100

Local $pCallback = _RegisterSyncCallback(_CallBackFunction)

; https://msdn.microsoft.com/en-us/library/windows/desktop/ms682485(v=vs.85).aspx
Local $a_hCall = DllCall("kernel32.dll", "bool", "CreateTimerQueueTimer", _
    "handle*", 0, _
    "handle", $a_h_CreateTimerQueue[0], _
    "ptr", $pCallback, _
    "ptr", 678, _
    "dword", $i_TimerQueue_Start_after, _
    "dword", $i_TimerQueue_Repeat_after, _
    "ulong", $WT_EXECUTEINTIMERTHREAD)
Local $phNewTimer = $a_hCall[1]
ConsoleWrite("CreateTimerQueueTimer = " & $a_hCall[0] & @TAB & $phNewTimer & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Local $timer = TimerInit()
While Sleep(10)
  ConsoleWrite("+ " & TimerDiff($timer) & @CRLF)
  If TimerDiff($timer) > 5000 Then
    ConsoleWrite("! fire Exitloop event" & @CRLF)
    ExitLoop
  EndIf
WEnd

Local $a_hCall = DllCall("kernel32.dll", "bool", "DeleteTimerQueueTimer", _
    "handle", $a_h_CreateTimerQueue[0], _
    "handle", $phNewTimer, _
    "handle", 0)
ConsoleWrite("DeleteTimerQueueTimer = " & $a_hCall[0] & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

ConsoleWrite("_WinAPI_CloseHandle($phNewTimer) = " & _WinAPI_CloseHandle($phNewTimer) & @CRLF)

Local $a_hCall = DllCall("kernel32.dll", "bool", "DeleteTimerQueueEx", "handle", $a_h_CreateTimerQueue[0], "handle", 0)
ConsoleWrite("DeleteTimerQueueEx = " & $a_hCall[0] & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Func _CallBackFunction($lParam, $TimerOrWaitFired)
  ConsoleWrite("_CallBackFunction fired = " & $lParam & @TAB & $TimerOrWaitFired & @CRLF)
EndFunc   ;==>_CallBackFunction

Func _RegisterSyncCallback($Function)
  Local Static $hGUI
  Local Static $pSendMessage
  Local Static $iMsg

  If Not $hGUI Then
    $hGUI = GUICreate("RegisterSyncCallback_Msg", 300, 200)
    $iMsg = $WM_APP
    GUIRegisterMsg($iMsg, RegisterSyncCallback_Msg)
    $pSendMessage = _WinAPI_GetProcAddress(_WinAPI_GetModuleHandle("user32.dll"), "SendMessageW")
    ConsoleWrite("$hGUI: " & Hex($hGUI) & @CRLF)
    ConsoleWrite("$pSendMessage: " & Hex($pSendMessage) & @CRLF)
    ConsoleWrite("$iMsg: " & Hex($iMsg) & @CRLF)
  EndIf

  Local $pRemoteCode = _MemVirtualAlloc(0, 96, $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
  If Not $pRemoteCode Then MsgBox(0, "Error", "_MemVirtualAlloc :(")
  Local $tCodeBuffer = DllStructCreate("byte[96]", $pRemoteCode)
  Local $hHandle = DllCallbackRegister($Function, "none", "int;int")

  If @AutoItX64 Then
    Local $sOPCode = "0x48894C240848895424104C894424184C894C24204883EC284989C8" & _
      "48C7C1" & SwapEndian($hGUI, False) & _
      "48C7C2" & SwapEndian($iMsg, False) & _
      "49B9" & SwapEndian(DllCallbackGetPtr($hHandle)) & _
      "48B8" & SwapEndian($pSendMessage) & _
      "FFD04883C428C3"
  Else
    Local $sOPCode = "0x68" & SwapEndian(DllCallbackGetPtr($hHandle)) & _
      "FF74240868" & SwapEndian($iMsg) & "68" & SwapEndian($hGUI) & _
      "B8" & SwapEndian($pSendMessage) & "FFD0C20800"
  EndIf

  DllStructSetData($tCodeBuffer, 1, $sOPCode)
  ConsoleWrite("$tCodeBuffer: " & DllStructGetData($tCodeBuffer, 1) & @CRLF)

  Return $pRemoteCode
EndFunc   ;==>_RegisterSyncCallback

Func RegisterSyncCallback_Msg($hWnd, $iMsg, $wParam, $lParam)
  ConsoleWrite(">RegisterSyncCallback_Msg Called" & @CRLF)
  ;ConsoleWrite("$hWnd: " & Hex($hWnd) & @CRLF)
  ;ConsoleWrite("$iMsg: " & Hex($iMsg) & @CRLF)
  ;ConsoleWrite("$wParam: " & Hex($wParam) & @CRLF)
  ;ConsoleWrite("$lParam: " & Hex($lParam) & @CRLF)
  DllCallAddress("none", $lparam, "int", $wParam, "int", 1)
EndFunc   ;==>RegisterSyncCallback_Msg

Func SwapEndian($uInt, $b64 = True)
  Local $iLen = (@AutoItX64 And $b64) ? 16 : 8, $sHex = Hex($uInt, $iLen), $sRes
  For $i = $iLen - 2 To 0 Step -2
    $sRes &= StringMid($sHex, $i + 1, 2)
  Next
  Return $sRes
EndFunc

 

[KaFu]

Looks good on my side for x64 🙂👍.

You've removed the parameters from this example, I would think they are required for the callbacks to fully work. I would also assume that the parameter type is relevant for the distinction between x86 and x64, so an explicit variable for the callback parameter structure should be part of the _RegisterSyncCallback() function call (they need to be passed to the DllCallbackRegister() call for $hHandle too?).

Local $t_Callback_Parameters = "ptr lpParameter; bool TimerOrWaitFired" ; WaitOrTimerCallback callback function > https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms687066(v=vs.85)
Local $pCallback = _RegisterSyncCallback(_CallBackFunction, $t_Callback_Parameters)

I also propose to create the opcode memory only after the definition of the opcode itself, with something like this.

Local $pRemoteCode = _MemVirtualAlloc(0, BinaryLen($sOPCode), $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
If Not $pRemoteCode Then MsgBox(0, "Error", "_MemVirtualAlloc :(")
Local $tCodeBuffer = DllStructCreate("byte[" & BinaryLen($sOPCode) & "]", $pRemoteCode)

 

 

Edited Monday at 01:46 PM by KaFu

[MattyD]

Thank you for this, I'll probably do some testing as well through the week.

On 6/1/2025 at 8:32 AM, Nine said:

Nevertheless, I do not trust the proposed code ( that I gave you ) is a solution with AutoIt.  The ASM code simply calls (in a few nano secs) the SendMessageW API

Yeah I tend to agree.  my understanding is:

Autoit's message pump should pick the window messages up whenever its ready, just the same as other GuiRegisterMessage() funcs. So that part is probably OK. I think we're also safe while we're processing a callback. Because SendMessageW is synchronous, that ASM function cannot return until the window message is processed.  But I'm not sure how protected we are when a callback hits while we're in the middle of a normal routine...

PS: just a side note -  volatile (callback) funcs don't block the message pump while they are being processed. So I may be misunderstanding things, but I'd imagine they'd generally make scripts more likely to crash!

Edited Monday at 02:22 PM by MattyD

[Nine]

TimerOrWaitFired is always True under our circumstances.  So why complexify the code with it...

[KaFu]

This is true for this specific CreateTimerQueue callback, but for other callbacks the return parameters will change. It would be great to make _RegisterSyncCallback() as generic and flexible as possible.

Edited Monday at 02:33 PM by KaFu

[KaFu]

This seems to work for two parameters. Are there callbacks with more than two parameters? I guess this hack is limited to two parameters anyhow, because it relies on sendmessage (with only two parameters)? Though the clean-up after the main loop is still messed, but that's for tomorrow.

Although I broke it again for x64 🙄 with another hack, that currently works for x86 (without this hack it worked under x64 too). 

In the CreateTimerQueueTimer call I pass a ptr for $t_Data_Send_to_Callback, which is fed to to callback via $lParam. Under x86 I can read the buffer in the callback, under x64 it currently crashes.

expandcollapse popup

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
; based on
; https://www.autoitscript.com/forum/topic/212917-createtimerqueuetimer/page/2/#findComment-1543584
; by nine

#include <Memory.au3>
#include <WindowsConstants.au3>
#include <WinAPIError.au3>
#include <WinAPIHObj.au3>
#include <WinAPISys.au3>

Global Const $WT_EXECUTELONGFUNCTION = 0x00000010 ; The callback function can perform a long wait. This flag helps the system to decide if it should create a new thread.
Global Const $WT_EXECUTEINTIMERTHREAD = 0x00000020 ; The callback function is invoked by the timer thread itself. This flag should be used only for short tasks or it could affect other timer operations.
Global Const $WT_EXECUTEINPERSISTENTTHREAD = 0x00000080 ; The callback function is queued to a thread that never terminates.

Local $a_h_CreateTimerQueue = DllCall("kernel32.dll", "handle", "CreateTimerQueue")
ConsoleWrite("CreateTimerQueue = " & $a_h_CreateTimerQueue[0] & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Local $i_TimerQueue_Start_after = 100
Local $i_TimerQueue_Repeat_after = 100

Local $pCallback = _RegisterSyncCallback("_CallBackFunction", "none", "ptr;bool") ; WaitOrTimerCallback callback function > https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms687066(v=vs.85)
Func _CallBackFunction($lParam, $TimerOrWaitFired)
    ConsoleWrite("_CallBackFunction fired - 0 = " & TimerInit() & @CRLF)
    ConsoleWrite("_CallBackFunction fired - 1 = " & $lParam & @TAB & $TimerOrWaitFired & @CRLF)
    ConsoleWrite("_CallBackFunction fired - 2 = " & VarGetType($lParam) & @TAB & VarGetType($TimerOrWaitFired) & @CRLF)
    Local $t_Data_Received = DllStructCreate("wchar[256]", $lParam)
    ConsoleWrite(DllStructGetData($t_Data_Received, 1) & @CRLF)
EndFunc   ;==>_CallBackFunction

Local $t_Data_Send_to_Callback = DllStructCreate("wchar[256]")
DllStructSetData($t_Data_Send_to_Callback, 1, "Update from main @ " & TimerInit())
; https://msdn.microsoft.com/en-us/library/windows/desktop/ms682485(v=vs.85).aspx
Local $a_hCall = DllCall("kernel32.dll", "bool", "CreateTimerQueueTimer", _
        "handle*", 0, _
        "handle", $a_h_CreateTimerQueue[0], _
        "ptr", $pCallback, _
        "ptr", DllStructGetPtr($t_Data_Send_to_Callback), _
        "dword", $i_TimerQueue_Start_after, _
        "dword", $i_TimerQueue_Repeat_after, _
        "ulong", $WT_EXECUTEINTIMERTHREAD)

Local $phNewTimer = $a_hCall[1]
ConsoleWrite("CreateTimerQueueTimer = " & $a_hCall[0] & @TAB & $phNewTimer & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Local $timer = TimerInit()
While Sleep(10)
    ConsoleWrite("+ " & TimerDiff($timer) & @CRLF)
    DllStructSetData($t_Data_Send_to_Callback, 1, "Update from main @ " & TimerInit())
    If TimerDiff($timer) > 2000 Then
        ConsoleWrite("! fire Exitloop event" & @CRLF)
        ExitLoop
    EndIf
WEnd

Local $a_hCall = DllCall("kernel32.dll", "bool", "DeleteTimerQueueTimer", _
        "handle", $a_h_CreateTimerQueue[0], _
        "handle", $phNewTimer, _
        "handle", 0)
ConsoleWrite("DeleteTimerQueueTimer = " & $a_hCall[0] & @CRLF & @CRLF)
If _WinAPI_GetLastError() = 997 Then ; ERROR_IO_PENDING = 997
    ConsoleWrite("If there are outstanding callback functions and CompletionEvent is NULL, the function will fail and set the error code to ERROR_IO_PENDING (997). " & @CRLF _
            & "This indicates that there are outstanding callback functions. Those callbacks either will execute or are in the middle of executing. " & @CRLF _
            & "The timer is cleaned up when the callback function is finished executing." & @CRLF & @CRLF)
EndIf
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

ConsoleWrite("_WinAPI_CloseHandle($phNewTimer) = " & _WinAPI_CloseHandle($phNewTimer) & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Local $a_hCall = DllCall("kernel32.dll", "bool", "DeleteTimerQueueEx", "handle", $a_h_CreateTimerQueue[0], "handle", 0)
ConsoleWrite("DeleteTimerQueueEx = " & $a_hCall[0] & @CRLF & @CRLF)
ConsoleWrite(_WinAPI_GetLastError() & @TAB & _WinAPI_GetLastErrorMessage() & @CRLF & @CRLF)

Exit

Func _RegisterSyncCallback($sFunction, $sReturnType, $sParams)
    Local Static $hGUI
    Local Static $pSendMessage
    Local Static $iMsg

    If Not $hGUI Then
        $hGUI = GUICreate("RegisterSyncCallback_Msg", 100, 100)
        $iMsg = $WM_APP
        GUIRegisterMsg($iMsg, RegisterSyncCallback_Msg)
        $pSendMessage = _WinAPI_GetProcAddress(_WinAPI_GetModuleHandle("user32.dll"), "SendMessageW")
        ConsoleWrite("$hGUI: " & Hex($hGUI) & @CRLF)
        ConsoleWrite("$pSendMessage: " & Hex($pSendMessage) & @CRLF)
        ConsoleWrite("$iMsg: " & Hex($iMsg) & @CRLF)
    EndIf

    Local $hHandle = DllCallbackRegister($sFunction, $sReturnType, $sParams)
    ConsoleWrite("$hHandle = " & $hHandle & @CRLF)

    If @AutoItX64 Then

        Local $sOPCode = "0x" _
                & "48894C2408488954" _
                & "24104C894424184C" _
                & "894C24204883EC28" _
                & "4989C8" _
                & "48C7C1" _
                & SwapEndian($hGUI, False) _
                & "48C7C2" _
                & SwapEndian($iMsg, False) _
                & "49B9" _
                & SwapEndian(DllCallbackGetPtr($hHandle)) _
                & "48B8" _
                & SwapEndian($pSendMessage) _
                & "FFD04883C428C3"
    Else

        Local $sOPCode = "0x" _
                & "68" _
                & SwapEndian(DllCallbackGetPtr($hHandle)) _
                & "FF74240868" _
                & SwapEndian($iMsg) _
                & "68" _
                & SwapEndian($hGUI) _
                & "B8" _
                & SwapEndian($pSendMessage) _
                & "FFD0C20800"

    EndIf

    Local $pRemoteCode = _MemVirtualAlloc(0, BinaryLen($sOPCode), $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
    If Not $pRemoteCode Then MsgBox(0, "Error", "_MemVirtualAlloc :(")
    Local $tCodeBuffer = DllStructCreate("byte[" & BinaryLen($sOPCode) & "]", $pRemoteCode)

    DllStructSetData($tCodeBuffer, 1, $sOPCode)
    ConsoleWrite("$tCodeBuffer: " & DllStructGetData($tCodeBuffer, 1) & @CRLF)

    Return $pRemoteCode
EndFunc   ;==>_RegisterSyncCallback

Func RegisterSyncCallback_Msg($hWnd, $iMsg, $wParam, $lParam)
    ConsoleWrite(">RegisterSyncCallback_Msg Called" & @CRLF)
    ; ConsoleWrite("$hWnd: " & Hex($hWnd) & @CRLF)
    ; ConsoleWrite("$iMsg: " & Hex($iMsg) & @CRLF)
    ; ConsoleWrite("$wParam: " & Hex($wParam) & @CRLF)
    ; ConsoleWrite("$lParam: " & Hex($lParam) & @CRLF)
    DllCallAddress("none", $lParam, "int", $wParam, "int", 1)
EndFunc   ;==>RegisterSyncCallback_Msg

Func SwapEndian($uInt, $b64 = True)
    Local $iLen = (@AutoItX64 And $b64) ? 16 : 8, $sHex = Hex($uInt, $iLen), $sRes
    For $i = $iLen - 2 To 0 Step -2
        $sRes &= StringMid($sHex, $i + 1, 2)
    Next
    Return $sRes
EndFunc   ;==>SwapEndian

 

 

Edited Tuesday at 12:15 AM by KaFu

[mLipok]

On 6/1/2025 at 12:32 AM, Nine said:

Nevertheless, I do not trust the proposed code ( that I gave you ) is a solution with AutoIt.  The ASM code simply calls (in a few nano secs) the SendMessageW API.  So why does it seem to work ? You ask me.  Well it is not obvious to answer since I do not own the core AutoIt code.  But my take on it, will go those few options :

1- the latency between AutoIt statement is larger than the time it takes to execute the ASM code

2- for "not that I know of" reason, the ASM routine is uninterruptible state

3- Windows execute ASM functions into a separate thread

Anyway, it was fun to look deeper into this challenging thread...Good luck !

Maybe @jpm can say more about.

I hope so.

[KaFu]

For x64:

Quote

Data Ptr in Main = 0x000001BC8B916A10 > binarylen(ptr) = 8
Data Ptr in Callback = 0x000000008B916A10 > binarylen(ptr) = 4

There's a conversion error, is the Int datatype hardcoded in the x64 opcode?

The best template from the AHK guys for variable parameters in the opcode seems to be in this post: https://www.autohotkey.com/boards/viewtopic.php?p=594011#p594011

Edit: Would it be a quick (and dirty) solution to set both parameters as ptr in the opcode? By default 4-byte in x86 and 8-byte in x64?

Edited Tuesday at 08:18 AM by KaFu

[Nine]

It is because the code is using int to call the callback while, in your case, it should be ptr.

DllCallAddress("none", $lParam, "ptr", $wParam, "int", 1)

After some readings, it became quite clear what is the issue when an APC is executed inside AutoIt.  We have to remember that AutoIt (I tend to forget about it) is interpreted. Which means that it is the AutoIt3.exe engine that executes our script lines.  So when an APC is launched, Windows expect that it will be using its own stack.  But since the callback address is referring to one managed by the interpreter which has a different stack, corruption is doomed to happen.

By using ASM code, the interpreter is not involved in any way in the callback.  We then communicate thru a message, which is perfectly handled by AutoIt.

So bottom line, it is very safe to use the ASM code and I do not see anyway it could break.