Set areas of interest on screen and capture their coords by drag-drawing rectangles

mavygr · October 6, 2014

Hi,

I have made a script that tracks an object in a video using imagesearcharea in the main tracking function. I would like to add areas of interest on the video, and then gather some information about each area (e.g. time the object appears in each area, distance travelled in each area etc). To define the areas, I've made a button on the main GUI that allows the user to create rectangles on the screen and then the script takes the coords from the start and end points of the rectangle and puts them in an array. This is how it looks (the rectangle making code is modified from a script by UEZ I found here):

Func SetArea()
    GUICtrlSetState($SetAreaButton, $GUI_DISABLE) ; disable this button
    GUICtrlSetState($TrackButton, $GUI_DISABLE) ; disable a different button in the gui
    Local $aMouse_Pos, $hMask, $hMaster_Mask, $iTemp
    Local $UserDLL = DllOpen("user32.dll")
    $AreaNumber = GUICtrlRead($InputAreaNumber) ; number of areas in total
    ; Create transparent GUI with Cross cursor
    $hCross_GUI = GUICreate("Test", @DesktopWidth, @DesktopHeight - 20, 0, 0, $WS_POPUP, $WS_EX_TOPMOST)
    WinSetTrans($hCross_GUI, "", 8)
    GUISetState(@SW_SHOW, $hCross_GUI)
    GUISetCursor(3, 1, $hCross_GUI)
    For $i = 1 To $AreaNumber
        Global $hRectangle_GUI = GUICreate("", @DesktopWidth, @DesktopHeight, 0, 0, $WS_POPUP, $WS_EX_TOOLWINDOW + $WS_EX_TOPMOST)
        GUISetBkColor($RedRGB) ; red
        ; Wait until mouse button pressed
        While Not _IsPressed("01", $UserDLL)
            Sleep(10)
            GUICtrlSetData($InputCursorX, MouseGetPos(0)) ; update mouse cursor coords
            GUICtrlSetData($InputCursorY, MouseGetPos(1))
            GUICtrlSetData($InputCurrentAction, "Setting area of interest" & $i) ; update main gui current action input
        WEnd
        ; Get first mouse position
        $aMouse_Pos = MouseGetPos()
        $iX1 = $aMouse_Pos[0]
        $iY1 = $aMouse_Pos[1]
        ; Draw rectangle while mouse button pressed
        While _IsPressed("01", $UserDLL)
            $aMouse_Pos = MouseGetPos()
            $hMaster_Mask = _WinAPI_CreateRectRgn(0, 0, 0, 0)
            $hMask = _WinAPI_CreateRectRgn($iX1, $aMouse_Pos[1], $aMouse_Pos[0], $aMouse_Pos[1] + 1) ; Bottom of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($iX1, $iY1, $iX1 + 1, $aMouse_Pos[1]) ; Left of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($iX1 + 1, $iY1 + 1, $aMouse_Pos[0], $iY1) ; Top of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($aMouse_Pos[0], $iY1, $aMouse_Pos[0] + 1, $aMouse_Pos[1]) ; Right of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            ; Set overall region
            _WinAPI_SetWindowRgn($hRectangle_GUI, $hMaster_Mask)
            If WinGetState($hRectangle_GUI) < 15 Then GUISetState()
            GUICtrlSetData($InputCursorX, MouseGetPos(0)) ; update mouse cursor coords
            GUICtrlSetData($InputCursorY, MouseGetPos(1))
            GUICtrlSetData($InputCurrentAction, "Setting area of interest...")
            Sleep(10)
        WEnd
        ; Get second mouse position
        $iX2 = $aMouse_Pos[0]
        $iY2 = $aMouse_Pos[1]
        ; Set in correct order if required
        If $iX2 > $iX1 Then
            $Areas[UBound($Areas) - 1][0] = $iX1
            $Areas[UBound($Areas) - 1][2] = $iX2
        Else
            $Areas[UBound($Areas) - 1][0] = $iX2
            $Areas[UBound($Areas) - 1][2] = $iX1
        EndIf
        If $iY2 > $iY1 Then
            $Areas[UBound($Areas) - 1][1] = $iY1
            $Areas[UBound($Areas) - 1][3] = $iY2
        Else
            $Areas[UBound($Areas) - 1][1] = $iY2
            $Areas[UBound($Areas) - 1][3] = $iY1
        EndIf
        ReDim $Areas[UBound($Areas) + 1][4]
    Next
    GUIDelete($hCross_GUI)
    DllClose($UserDLL)
    GUICtrlSetState($SetAreaButton, $GUI_ENABLE)
    GUICtrlSetState($TrackButton, $GUI_ENABLE)
    _ArrayDisplay($Areas) ; to check if it's working
EndFunc   ;==>SetArea

At the moment, it can capture the coords and place them in the array. However, I can't get the drawn rectangles to stay on screen after I finish with defining the areas and also the "for - next" part doesnt seem to work... Generally it doesn't work smoothly... Any ideas on how to improve this?

Thanks in advance!

M

Edited October 6, 2014 by mavygr

jaberwacky · October 6, 2014

Which video is it? Is it on youtube for us to watch? It would help us to help you better.

Geir1983 · October 6, 2014

You have a while inside your for loop, what is the intention of $AreaNumber? Are you supposed to draw multiple rectangles from this one function?

For $i = 1 To $AreaNumber
        Global $hRectangle_GUI = GUICreate("", @DesktopWidth, @DesktopHeight, 0, 0, $WS_POPUP, $WS_EX_TOOLWINDOW + $WS_EX_TOPMOST)
        GUISetBkColor($RedRGB) ; red
        ; Wait until mouse button pressed
        While Not _IsPressed("01", $UserDLL) ;     <-    This will prevent your For loop to continue...

mavygr · October 6, 2014

Hi, thanks for the replies.

@ jaberwacky: The script is intended to be used for tracking a fish (!) inside an experimental tank (I am a PhD student doing behavioural experiments ). I would like to make it work with different videos.

Something very similar to this:

So my code does what is shown in the video (track the fish and then prints he trajectory in a png file) but I also want to include areas of interest (e.g. two rectangles on both ends of the tank) and get info for them (e.g time the fish spent in each of them).

@Geir1983: Yes the $areanumber will be declared by an input in the main gui and will be the total number of areas that will be defined with the function. So the for loop won't work with the while loop inside... I can't think of a better way to be able to draw multiple rectangles (and get their coords) with this function... It sort of works, but probably needs complete rewriting to improve it...

Edited October 6, 2014 by mavygr

Gianni · October 6, 2014

interesting project,
I'm a little off topic, but hope also useful
I think that could also be usefull for your project this >opencv udf (I've never used it.)
here a video that shows what can be done with opencv.

Fish tracking

http://www.youtube.com/watch?v=pWs8t16g0VQ

JohnOne · October 6, 2014

I can't think of a better way to be able to draw multiple rectangles (and get their coords) with this function

Forgive me is I've missed something, but if you are drawing rectangles you must already know their "coords".

mavygr · October 6, 2014

@Chimp: the video shows exactly what I'm trying to do (without the OpenCV engine - just autoit). I'm not sure I have enough knowledge to effectively use opencv on this project....

@JohnOne: I would like to have a user-friendly way to draw the rectangles on screen (in this case, pressing mouse and dragging across the screen). So the user would have to know where the rectangles need to be drawn (these are the areas of interest), but the script should be able to get the coords of the drawn rectangles. In other words, I don't want the user to type in the coords but rather input them by drawing the rectangle him/herself

JohnOne · October 6, 2014

Ok, so it is your script that these rectangles will be drawn with?

Still stands, your script cannot draw a rectangle without first knowing where to draw it.

Make sense?

Edited October 6, 2014 by JohnOne

mavygr · October 6, 2014

If you look at the code in the first post, the script starts drawing the rectangle after the left mouse-click is pressed (using these initial coords as left/top). Then while it is still pressed it continuously re-draws and re-shapes the rectangle according to where the mouse is dragged at. Then in the end, when the mouse-click is not pressed any more, it finalizes the rectangle by using the last set of coords as right/bottom. So this is how the rectangle is made. The problem is how to expand this, and make it possible to draw more than one rectangles (that remain on screen after they're drawn) with this function. Secondly, how to save these rectangles' coords in an array (after they're drawn), this is my attempt in the last part on the code...

JohnOne · October 6, 2014

Ok, I just thought you were having a problem knowing where those rectangles where.

Your problem with keeping the graphics on the screen I don't know much about even though I did do something similar a while back.

It could have something to do with how often a region of whatever the rect is over redraws itself, in a browser could be quite often meaning your graphics would be wiped.

EDIT:

Maybe a function ran every fraction of a second could redraw your final rects.

Edited October 6, 2014 by JohnOne

Gianni · October 6, 2014

your function it seems working here..... I am not sure to understand what's the problem

#include <GDIPlus.au3>
#include <WindowsConstants.au3>
#include <Misc.au3>
#include <array.au3>

Global $Areas[1][5], $AreaNumber = 5
SetArea()
MsgBox(0, "", "Rectangles goes away when program finish" & @CRLF & "click OK to finish")
Func SetArea()
    ; GUICtrlSetState($SetAreaButton, $GUI_DISABLE) ; disable this button
    ; GUICtrlSetState($TrackButton, $GUI_DISABLE) ; disable a different button in the gui
    Local $aMouse_Pos, $hMask, $hMaster_Mask, $iTemp
    Local $UserDLL = DllOpen("user32.dll")
    ; $AreaNumber = GUICtrlRead($InputAreaNumber) ; number of areas in total
    ; Create transparent GUI with Cross cursor
    $hCross_GUI = GUICreate("Test", @DesktopWidth, @DesktopHeight - 20, 0, 0, $WS_POPUP, $WS_EX_TOPMOST)
    WinSetTrans($hCross_GUI, "", 8)
    GUISetState(@SW_SHOW, $hCross_GUI)
    GUISetCursor(3, 1, $hCross_GUI)
    For $i = 1 To $AreaNumber
        Global $hRectangle_GUI = GUICreate("", @DesktopWidth, @DesktopHeight, 0, 0, $WS_POPUP, $WS_EX_TOOLWINDOW + $WS_EX_TOPMOST)
        GUISetBkColor(0xFF0000) ; $RedRGB) ; red
        ; Wait until mouse button pressed
        While Not _IsPressed("01", $UserDLL)
            Sleep(10)
            ; GUICtrlSetData($InputCursorX, MouseGetPos(0)) ; update mouse cursor coords
            ; GUICtrlSetData($InputCursorY, MouseGetPos(1))
            ; GUICtrlSetData($InputCurrentAction, "Setting area of interest" & $i) ; update main gui current action input
        WEnd
        ; Get first mouse position
        $aMouse_Pos = MouseGetPos()
        $iX1 = $aMouse_Pos[0]
        $iY1 = $aMouse_Pos[1]
        ; Draw rectangle while mouse button pressed
        While _IsPressed("01", $UserDLL)
            $aMouse_Pos = MouseGetPos()
            $hMaster_Mask = _WinAPI_CreateRectRgn(0, 0, 0, 0)
            $hMask = _WinAPI_CreateRectRgn($iX1, $aMouse_Pos[1], $aMouse_Pos[0], $aMouse_Pos[1] + 1) ; Bottom of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($iX1, $iY1, $iX1 + 1, $aMouse_Pos[1]) ; Left of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($iX1 + 1, $iY1 + 1, $aMouse_Pos[0], $iY1) ; Top of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            $hMask = _WinAPI_CreateRectRgn($aMouse_Pos[0], $iY1, $aMouse_Pos[0] + 1, $aMouse_Pos[1]) ; Right of rectangle
            _WinAPI_CombineRgn($hMaster_Mask, $hMask, $hMaster_Mask, 2)
            _WinAPI_DeleteObject($hMask)
            ; Set overall region
            _WinAPI_SetWindowRgn($hRectangle_GUI, $hMaster_Mask)
            If WinGetState($hRectangle_GUI) < 15 Then GUISetState()
            ; GUICtrlSetData($InputCursorX, MouseGetPos(0)) ; update mouse cursor coords
            ; GUICtrlSetData($InputCursorY, MouseGetPos(1))
            ; GUICtrlSetData($InputCurrentAction, "Setting area of interest...")
            Sleep(10)
        WEnd
        ; Get second mouse position
        $iX2 = $aMouse_Pos[0]
        $iY2 = $aMouse_Pos[1]
        ; Set in correct order if required
        If $iX2 > $iX1 Then
            $Areas[UBound($Areas) - 1][0] = $iX1
            $Areas[UBound($Areas) - 1][2] = $iX2
        Else
            $Areas[UBound($Areas) - 1][0] = $iX2
            $Areas[UBound($Areas) - 1][2] = $iX1
        EndIf
        If $iY2 > $iY1 Then
            $Areas[UBound($Areas) - 1][1] = $iY1
            $Areas[UBound($Areas) - 1][3] = $iY2
        Else
            $Areas[UBound($Areas) - 1][1] = $iY2
            $Areas[UBound($Areas) - 1][3] = $iY1
        EndIf
        ReDim $Areas[UBound($Areas) + 1][4]
    Next
    GUIDelete($hCross_GUI)
    DllClose($UserDLL)
    ; GUICtrlSetState($SetAreaButton, $GUI_ENABLE)
    ; GUICtrlSetState($TrackButton, $GUI_ENABLE)
    _ArrayDisplay($Areas) ; to check if it's working
EndFunc   ;==>SetArea

Edited October 6, 2014 by Chimp

jaberwacky · October 7, 2014

Here is a shot in the dark. Is this what you're describing?

Edit: To use, click and hold down the left mouse button and let go when the rectangle is where you want it.

Edit2: There is an issue. For best results start as close to coord (0,0) as possible...

#region ; Includes
#include <APISysConstants.au3>

#include <GUIMenu.au3>

#include <Array.au3>

#include <GDIPlus.au3>

#include <WinAPIProc.au3>

#include <WinAPIGdi.au3>

#include <WinAPISys.au3>

#include <WindowsConstants.au3>

#include <BorderConstants.au3>

#include <WinAPIsysinfoConstants.au3>

#include <WindowsConstants.au3>
#EndRegion

Const $monitor_count = _WinAPI_GetSystemMetrics($SM_CMONITORS)

Const $monitor_height = _get_monitor_heights($monitor_count)

Const $monitor_width = _get_monitor_widths($monitor_count)

Const $desktop_width = _WinAPI_GetSystemMetrics($SM_CXVIRTUALSCREEN)

Const $tallest_monitor = _ArrayMax($monitor_height)

Const $desktop_left = _get_desktop_rect()[0]

Const $half_desktop_width = $desktop_width / 2

Const $user32_dll = DllOpen("User32.dll") 

Const $marker_board = GUICreate('', $desktop_width, $tallest_monitor, 0, 0, $WS_POPUP, BitOR($WS_EX_LAYERED, $WS_EX_TOPMOST, $WS_EX_TRANSPARENT))

GUISetState(@SW_SHOWNORMAL, $marker_board)

Global $DesktopDC = _WinAPI_GetDC($marker_board)
Global $DC        = _WinAPI_CreateCompatibleDC($DesktopDC)
Global $Bitmap    = _WinAPI_CreateCompatibleBitmap($DesktopDC, $desktop_width, $tallest_monitor)
  
_WinAPI_SelectObject($DC, $Bitmap)  

_GDIPlus_Startup()

Global $Graphics  = _GDIPlus_GraphicsCreateFromHDC($DC)
Global $Pen       = _GDIPlus_PenCreate(0xFFFF0000, 3)

Const $mouse_proc_callback = _RegisterLLMouseHook(_ll_mouse_proc)

Global $top_left[2], $bottom_right[2], $active = False

OnAutoItExitRegister(_on_exit)

Do
  Sleep(1000)
Until False

Volatile Func _ll_mouse_proc($code, $w_param, $l_param)
  Switch $code >= 0
    Case True
      Switch $w_param
        Case $wm_mousemove 
          If $active Then
            $bottom_right = MouseGetPos()
            
            DrawRectangle($top_left[0], $top_left[1], $bottom_right[0], $bottom_right[1])
            
            ClearRectangle()
          EndIf
          
        Case $wm_lbuttondown  
          $active = True
          
          $top_left = MouseGetPos()
          
        Case $wm_lbuttonup     
          $active = False
          
          DrawRectangle($top_left[0], $top_left[1], $bottom_right[0], $bottom_right[1])
      EndSwitch
  EndSwitch

  Return _WinAPI_CallNextHookEx($mouse_proc_callback, $code, $w_param, $l_param)
EndFunc

Func _RegisterLLMouseHook(Const $func)
  Local Const $h_func = DllCallbackRegister($func, "long", "int;wparam;lparam")

  Local Const $p_func = DllCallbackGetPtr($h_func)

  Local Const $h_mod = _WinAPI_GetModuleHandle(0)

  Return _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, $p_func, $h_mod)
EndFunc

Func _get_desktop_rect()
  Local Const $tWorkArea = DllStructCreate($tagRECT)
  
  _WinAPI_SystemParametersInfo($SPI_GETWORKAREA, 0, DllStructGetPtr($tWorkArea))
  
  Local $rect[4]
  
  $rect[0] = DllStructGetData($tWorkArea, "Left")
  $rect[1] = DllStructGetData($tWorkArea, "Top")
  $rect[2] = DllStructGetData($tWorkArea, "Right") - $rect[0]
  $rect[3] = DllStructGetData($tWorkArea, "Bottom") - $rect[1]
  
  Return $rect
EndFunc

Func _get_monitor_info(Const $monitor_number)
  If $monitor_number <= 0 Then 
    Return SetError(1, 0, False)
  EndIf

  Local Const $monitor = _WinAPI_EnumDisplayMonitors()[$monitor_number][0]

  Return _WinAPI_GetMonitorInfo($monitor)[0]
EndFunc

Func _get_monitor_heights(Const $count)  
  Local $height[($count + 1)]
  
  $height[0] = $count
  
  For $i = 1 To $height[0]
    $height[$i] = _get_monitor_info($i).Bottom
  Next
  
  Return $height
EndFunc

Func _get_monitor_widths(Const $count)  
  Local $width[($count + 1)]
  
  $width[0] = $count
  
  For $i = 1 To $width[0]    
    $width[$i] = _get_monitor_info($i).Right - _get_monitor_info($i).Left
  Next
  
  Return $width
EndFunc

Func DrawRectangle(Const $left, Const $top, Const $right, Const $bottom)  
  _GDIPlus_GraphicsDrawRect($Graphics, $left,  $top, $right, $bottom, $Pen)

  UpdateLayeredWindow()
EndFunc

Func ClearRectangle()
  _GDIPlus_GraphicsClear($Graphics, 0xFF000000)
  
  UpdateLayeredWindow()
EndFunc

Func UpdateLayeredWindow()
  Local Static $tSource = DllStructCreate($tagPOINT)

  Local Static $tSize = DllStructCreate($tagSIZE)    
  DllStructSetData($tSize, 1, $desktop_width)    
  DllStructSetData($tSize, 2, $tallest_monitor) 

  Local Static $tBlend = DllStructCreate($tagBLENDFUNCTION)    
  DllStructSetData($tBlend, "Alpha", 255)    
  DllStructSetData($tBlend, "Format", 1)  

  DllCall($user32_dll,  "bool", "UpdateLayeredWindow", _
                        "hwnd",    $marker_board,      _
                        "handle",  $DesktopDC,         _
                        "ptr",     0,                  _
                        "struct*", $tSize,             _
                        "handle",  $DC,                _
                        "struct*", $tSource,           _
                        "dword",   0,                  _
                        "struct*", $tBlend,            _
                        "dword",   $ULW_ALPHA)
EndFunc

Volatile Func _on_exit()  
  _GDIPlus_PenDispose($Pen)
  _GDIPlus_GraphicsDispose($Graphics)
  _GDIPlus_Shutdown()
  
  _WinAPI_DeleteObject($Bitmap)
  _WinAPI_DeleteDC($DC)
  
  DllClose($user32_dll)
  
  DllCallbackFree($mouse_proc_callback)
EndFunc

Edited October 7, 2014 by jaberwacky

mavygr · October 7, 2014

Thanks for the help guys.

your function it seems working here..... I am not sure to understand what's the problem

Yes your version seems to work better! Perhaps it doesn't work very well with my gui (because I use a button to call this function)? I'll have to check better and come back at this.

Thanks for all the interest guys

Edited October 7, 2014 by mavygr

Set areas of interest on screen and capture their coords by drag-drawing rectangles

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members