Deadlock with Subclassed Button and Mouse Hook

[wraithdu]

Can anyone shed some light as to why the following code deadlocks when pressing the button? I'm trying to write a toggle button library... it's way harder than it sounds. I think to get everything working I need to subclass the buttons to handle some messages, but whenever I subclass the button and click, the app deadlocks. Without subclassing, it does not. Nevermind the very incomplete code, it's just a quick reproducer. Tested with 3.3.5.3 on Win7 32-bit.

expandcollapse popup

#NoTrayIcon
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include <Constants.au3>

Global $pressed = 0
Global $gui = GUICreate("Test", 400, 400)
Global $b1 = GUICtrlCreateButton("Test Button 1", 10, 10)
Global $hb1 = GUICtrlGetHandle($b1)

; subclass the button
Global $ButtonProc = DllCallbackRegister("_ButtonProc", "long_ptr", "hwnd;uint;wparam;lparam")
Global $defproc = _WinAPI_SetWindowLong($hb1, $GWL_WNDPROC, DllCallbackGetPtr($ButtonProc))
; set mouse hook
Global $MouseProc = DllCallbackRegister("_MouseProc", "long_ptr", "int;wparam;lparam")
Global $NextHook = _WinAPI_SetWindowsHookEx($WH_MOUSE, DllCallbackGetPtr($MouseProc), 0, _WinAPI_GetCurrentThreadId())

GUISetState()

Do
Until GUIGetMsg() = -3

_Exit()

Func _Exit()
    _WinAPI_UnhookWindowsHookEx($NextHook)
    _WinAPI_SetWindowLong($hb1, $GWL_WNDPROC, $defproc)
    DllCallbackFree($ButtonProc)
    DllCallbackFree($MouseProc)
    Exit
EndFunc

Func _ButtonProc($hWnd, $Msg, $wParam, $lParam)
    Return _WinAPI_CallWindowProc($defproc, $hWnd, $Msg, $wParam, $lParam)
EndFunc

Func _MouseProc($nCode, $wParam, $lParam)
    Local $ret = 0, $point, $hwnd, $tagMHS = "long;long;hwnd;uint;ulong_ptr"

    If $nCode < 0 Then Return _WinAPI_CallNextHookEx($NextHook, $nCode, $wParam, $lParam)
    Switch $wParam
        Case $WM_LBUTTONDOWN
            ; get hovered control hwnd
            $point = DllStructCreate($tagMHS, $lParam)
            $hwnd = DllStructGetData($point, 3)
            If $hwnd = $hb1 Then
                _SendMessage($hb1, 0xF3, 1) ; pressed state
                Return 1 ; block normal processing
            EndIf
        Case $WM_LBUTTONUP
            ; get hovered control hwnd
            $point = DllStructCreate($tagMHS, $lParam)
            $hwnd = DllStructGetData($point, 3)
            If $hwnd = $hb1 Then
                $pressed = Number(Not $pressed) ; toggle
                _SendMessage($hb1, 0xF3, $pressed) ; set new state
                Return 1
            EndIf
    EndSwitch
    Return _WinAPI_CallNextHookEx($NextHook, $nCode, $wParam, $lParam)
EndFunc

Edited February 10, 2010 by wraithdu

[Melba23]

wraithdu,

I hesitate to offer advice, but..........

If you are looking to use the MouseProc to toggle the button state, do you not have to block the actual button press by the mouse within the subclassed ButtonProc? So you would need something like this:

Func _ButtonProc($hWnd, $Msg, $wParam, $lParam)
    If $Msg = ####### Then ; eat the actual button press by the mouse
        Return 0
    Else
        Return _WinAPI_CallWindowProc($defproc, $hWnd, $Msg, $wParam, $lParam)
    EndIf
EndFunc

Quite what it is you would have to trap I am not sure - what do buttons send when they are pressed? WM_COMMAND? WM_NOTIFY?

Or am I talking complete spherical objects as I usually do when it gets this deep?

M23

[wraithdu]

LOL, a little. The BN_CLICKED notification is sent to the parent window via WM_COMMAND. But by that point it's too late. What I'm dealing with here is the button's own window procedure that controls the drawing state. The mouse hook blocks the button from getting clicks and manually sets the pushed state via the BM_SETSTATE message.

All of this can be done with just the mouse hook. The subclassing is needed to handle keyboard navigation, specifically the button receiving WM_KILLFOCUS. That's right, the button gets WM_KILLFOCUS, the parent window gets BN_KILLFOCUS via WM_COMMAND. Notice the difference, WM = Window Message (does something), BN = Button Notification (something has happened, too late for you to do anything about it).

So yeah, I've got to get inside the button to stop it from screwing with the view state. The problem is all the things that the WM_KILLFOCUS message does -

Removes the focus rectangle from a button. For push buttons and default push buttons, the focus rectangle is invalidated. If the button has the mouse capture, the capture is released, the button is not clicked, and any push state is removed.

I've tried handling BN_KILLFOCUS and BN_CLICKED in WM_COMMAND to reset the pushed state based on saved values, but tabbing through a control that is in the pushed state first sends a BN_CLICKED notification before the BN_KILLFOCUS notification, thus resetting any saved values I might have had for the state of that button. By the time I got the BN_KILLFOCUS, I had no idea what the previous state was. So, I have to get at it earlier in the button's window procedure.

Or... I have to install a keyboard hook as well, and handle Enter/Space/Tab in some appropriate way. I'd be much better off if it didn't deadlock for whatever reason. But you're essentially right, I have to block stuff in the button window procedure. But as you can see, simply passing along the messages is deadlocking the app. This has to be solved first.

Edited February 9, 2010 by wraithdu

[Melba23]

wraithdu,

Only small cubes, rather than spheres then.....I am getting better!

M23