Deadlock with Subclassed Button and Mouse Hook

[wraithdu]

I posted this in the GUI forum but didn't get much response. After more investigating, I'm thinking it might be a problem in AutoIt. In the below repro script, when the button is subclassed, sending a message to the button from the mouse hook is triggering an infinite loop by reposting the same mouse button press to the mouse hook. I have no idea why this is happening. It DOES NOT happen if the button is NOT subclassed, so it is not a logical error. The subclassed button procedure is just a dummy procedure that passes any messages onto the original procedure.

There are other functions that produce a finite loop, such as GUICtrlSetStyle or _WinAPI_SetWindowLong. I have no idea why the loop ends when these functions are used, but not with SendMessage. Lastly, switching to PostMessage seems to solve this particular deadlock, but that is of no use to the other noted functions.

Ideas?

expandcollapse popup

#NoTrayIcon
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <WinAPI.au3>
#include <Constants.au3>
#include <WindowsConstants.au3>

OnAutoItExitRegister("_Exit")

Global $gui = GUICreate("Deadlock", 200, 75)
Global $b = GUICtrlCreateButton("Click to Deadlock", 10, 10, 110)
Global $hb = GUICtrlGetHandle($b)

; set mouse hook
Global $MouseProc = DllCallbackRegister("_MouseProc", "lresult", "int;wparam;lparam")
Global $MouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE, DllCallbackGetPtr($MouseProc), 0, _WinAPI_GetCurrentThreadId())
; subclass button
Global $BtnProc = DllCallbackRegister("_BtnProc", "lresult", "hwnd;uint;wparam;lparam")
Global $OldBtnProc = _WinAPI_SetWindowLong($hb, $GWL_WNDPROC, DllCallbackGetPtr($BtnProc))

GUISetState()

Do
Until GUIGetMsg() = -3

Exit

Func _MouseProc($nCode, $wParam, $lParam)
    If $nCode < 0 Then Return _WinAPI_CallNextHookEx($MouseHook, $nCode, $wParam, $lParam)
    Local $tagMHS, $MHS, $hwnd
    Switch $wParam
        Case $WM_LBUTTONDOWN
            $tagMHS = "long;long;hwnd;uint;ulong_ptr"
            $MHS = DllStructCreate($tagMHS, $lParam)
            $hwnd = DllStructGetData($MHS, 3)
            If $hwnd = $hb Then
                ConsoleWrite("down" & @CRLF)
                Return 1
            EndIf
        Case $WM_LBUTTONUP
            $tagMHS = "long;long;hwnd;uint;ulong_ptr"
            $MHS = DllStructCreate($tagMHS, $lParam)
            $hwnd = DllStructGetData($MHS, 3)
            If $hwnd = $hb Then
                ConsoleWrite("up" & @CRLF)
                ; sending this message will deadlock AutoIt if the button is subclassed
                _SendMessage($hb, 0xF3, 1) ; BM_SETSTATE
                Return 1
            EndIf
    EndSwitch
    Return _WinAPI_CallNextHookEx($MouseHook, $nCode, $wParam, $lParam)
EndFunc

Func _BtnProc($hWnd, $Msg, $wParam, $lParam)
    Return _WinAPI_CallWindowProc($OldBtnProc, $hWnd, $Msg, $wParam, $lParam)
EndFunc

Func _Exit()
    _WinAPI_UnhookWindowsHookEx($MouseProc)
    DllCallbackFree($MouseHook)
    _WinAPI_SetWindowLong($hb, $GWL_WNDPROC, $OldBtnProc)
    DllCallbackFree($BtnProc)
EndFunc

[Valik]

SendMessage() blocks waiting for the message to be handled. SendMessage() is called inside a callback function which in turn causes another callback function to be invoked. This leads to a deadlock, only one callback function can be executed at a time. Use PostMessage() or SendMessageTimeout().

[wraithdu]

What does that have to do with the button being subclassed? And why is the mouse hook triggered in a loop? Running the example you see the "up" message displayed in an infinite loop, meaning extra WM_LBUTTONUP messages are being created somehow. I could understand what you are describing, but this seems the opposite. AutoIt is not waiting indefinitely for something to happen, it gets stuck in an infinite loop.

The problem disappears if the button is not subclassed. That's the part that's really got me. The button proc isn't doing anything special but passing the messages onto the original procedure.

[Valik]

What does it have to do with subclassing? You subclassed the button with a callback function in AutoIt (_BtnProc()). You have another callback function (_MouseProc()) which attempts to invoke _BtnProc() via the callback mechanism. This is a deadlock and it has everything to do with _BtnProc() being the subclassed callback for the button. You must use PostMessage() in this situation so that _MouseProc() can return and give _BtnProc() a chance to execute. This is your problem, don't get side-tracked by other weird things going on. Fix this and your deadlock issue goes away.

Edited February 12, 2010 by Valik

[wraithdu]

Thanks again for the clarification. I appreciate it.