Opencv UDF

Lion66 · July 13, 2021

I really appreciate your help!

I removed the _cvNormalize and the _cvThreshold, and now $tmaxval.max shows the correct results.

In a printout I see the maximum value at scale 2, which corresponds to reality: the picture is increased by 2 times.

Scale: 1.5  0.263792783021927
Scale: 1.75 0.21775709092617
Scale: 2    0.272976666688919
Scale: 2.25 0.223066240549088
Scale: 2.5  0.226864993572235

But rectangles are not displayed correctly 🙄.

Something is wrong with the parameters of the $tmaxloc.x and $tmaxloc.y ?

I can not understand. Could you see? Thank you.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <GDIplus.au3>
#include <Memory.au3>
#include <GUIConstantsEx.au3>
#include <Array.au3>
#include <emgucv-autoit-bindings\cve_extra.au3>

;start dll opencv
_GDIPlus_Startup()
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")

Local $matEmpty = _cveMatCreate()

#Region Template
;Load template
Local $ptemp = _cveImreadAndCheck("Template.png") ;Opencv function to load image
;// Create a destination image to hold output
;//smooth and converting the original image into grayscale
Local $TemplateGray = _cveMatCreate()
;_cveSmoothMat($ptemp, $ptemp, $CV_GAUSSIAN, 3, 3) ;
_cveCvtColorMat($ptemp, $TemplateGray, $CV_BGR2GRAY) ;
_cveCannyMat($TemplateGray, $TemplateGray, 50, 200) ; kontyre
#EndRegion Template

#Region Image
;Load image from file
Local $pimg = _cveImreadAndCheck("bridge200p.jpg") ;Opencv function to load image
;// Create a destination image to hold output
;//smooth and converting the original image into grayscale
Local $pimgGrayScale = _cveMatCreate()
;_cveSmoothMat($pimg, $pimg, $CV_GAUSSIAN, 3, 3) ;
_cveCvtColorMat($pimg, $pimgGrayScale, $CV_BGR2GRAY) ;
_cveCannyMat($pimgGrayScale, $pimgGrayScale, 50, 200) ; kontyre
;_cvEqualizeHistMat( $pimgGrayScale, $pimgGrayScale ) ; osvetlit'
#EndRegion Image

;// Create some CV and Autoit windows to show the input
;_cveNamedWindow("Example")
;// show input image
;_cveImshowMat("Example", $ptemp) ;

Local $width2 = _cveMatGetWidth($TemplateGray)
Local $height2 = _cveMatGetHeight($TemplateGray)

Local $pimgResized, $scale, $width, $height, $rw, $rh, $presult
Local $CurMaxVal = 0, $CurLocX = 0, $CurLocY = 0
Local $tBlackColor = _cvScalar(0)
Local $tRedColor = _cvRGB(255, 0, 0)
Local $tWhiteColor = _cvRGB(255, 255, 255)
Local $tDsize = _cvSize(0, 0)

For $i = 1.5 To 2.5 Step 0.25
    ;//Resize image
    $scale = $i
    $pimgResized = _cveMatCreate()
    $tDsize.width = _cveMatGetWidth($pimgGrayScale) / $scale
    $tDsize.height = _cveMatGetHeight($pimgGrayScale) / $scale
    _cveResizeMat($pimgGrayScale, $pimgResized, $tDsize, 0, 0, $CV_INTER_LINEAR)
    _cveCannyMat($pimgResized, $pimgResized, 50, 200) ; kontyre

    $width = _cveMatGetWidth($pimgResized)
    $height = _cveMatGetHeight($pimgResized)

    If ($width < $width2) Or ($height < $height2) Then
        ExitLoop
    EndIf

    $rw = $width - $width2 + 1
    $rh = $height - $height2 + 1
    ;// Create Opencv matrix object 32 bit floating
    $presult = _cveMatCreate()
    _cveMatCreateData($presult, $rh, $rw, $CV_32FC1)

    ;// Template matching
    _cveMatchTemplateMat($pimgResized, $TemplateGray, $presult, 5, $matEmpty)

    ;// Create minmax variables to pass to opencv
    Local $tmaxloc = DllStructCreate($tagCvPoint)
    Local $tminloc = DllStructCreate($tagCvPoint)
    Local $tmaxval = DllStructCreate("double max;")
    Local $tminval = DllStructCreate("double min;")

    ;// create mask to find multiple matches
    Local $pmask = _cveMatCreate()
    _cveMatOnes($rh, $rw, $CV_8UC1, $pmask)

    ;// Find location of maximum
    _cveMinMaxLocMat($presult, $tminval, $tmaxval, $tminloc, $tmaxloc, $pmask)
    ;// Mask it to find others if exists and draw red rectangle on input image
    Local $tMaskRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 5, $height2 + 5)
    _cveRectangleMat($pmask, $tMaskRect, $tBlackColor, -1, $CV_LINE_8, 0)
    $tMaskRect = 0

    Local $tImgRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 15, $height2 + 15)
    ;Local $tImgRect = _cvRect($tmaxloc.x, $tmaxloc.y, $width2, $height2)
    _cveRectangleMat($pimgResized, $tImgRect, $tWhiteColor, -1, $CV_LINE_8, 0)
    $tImgRect = 0

    ;// Update input image
    _cveImshowMat("", $pimgResized)
    Sleep(100)

    If $CurMaxVal < $tmaxval.max Then
        $CurMaxVal = $tmaxval.max
        $CurScale = $scale
        $CurLocX = $tmaxloc.x * $scale
        $CurLocY = $tmaxloc.y * $scale
        MsgBox(0, "", "Changed. Scale: " & $scale)
    EndIf

    ConsoleWrite("Scale: " & $scale & " " & $tmaxval.max & @CRLF)
    Sleep(1000)

Next

;// Mask it to find others if exists and draw red rectangle on input image
;_cveRectangle($pmask, _cvPoint($CurLocX - 5, $CurLocY - 5), _cvPoint($CurLocX + $width2, $CurLocY + $height2), $tBlackColor, -1, 8, 0)
Local $tImgRect = _cvRect($CurLocX - 5, $CurLocY - 5, $width2 + 15, $height2 + 15)
_cveRectangleMat($pimg, $tImgRect, $tRedColor, 1, 8, 0)
$tImgRect = 0

;// Update input image
;_cveImshowMat("Example-Resize", $pimgResized)
_cveImshowMat("", $pimg)

_cveWaitKey()

$tDsize = 0
$tWhiteColor = 0
$tRedColor = 0
$tBlackColor = 0

_cveMatRelease($ptemp)
_cveMatRelease($TemplateGray)
_cveMatRelease($pimg) ;
_cveMatRelease($pimgResized) ;
_cveMatRelease($pimgGrayScale) ;
_cveMatRelease($matEmpty) ;
_cveDestroyAllWindows()

_Opencv_DLLClose()
_GDIPlus_Shutdown()
Exit

smbape · July 13, 2021

@Lion66 you are welcome.

Great catch on the removal of the _cveNormalize.

With $CV_TM_CCOEFF_NORMED (5) as matchMethod, an optimum value ($tmaxval.max in this case) which is less than 0.75 can be considered as a false match.

Your match is wrong for at least 2 reasons:

Canny on the image + Resize + Canny rarely gives expected results
The match rectangle has incoherent values. Why - 5 ? Why +15?

Here is a modified version that gives expected result.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <GDIplus.au3>
#include <Memory.au3>
#include <GUIConstantsEx.au3>
#include <Array.au3>
#include <emgucv-autoit-bindings\cve_extra.au3>

;start dll opencv
_GDIPlus_Startup()
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")

Local $matEmpty = _cveMatCreate()

#Region Template
;Load template
Local $ptemp = _cveImreadAndCheck("Template.png") ;Opencv function to load image
;// Create a destination image to hold output
;//smooth and converting the original image into grayscale
Local $TemplateGray = _cveMatCreate()
;_cveSmoothMat($ptemp, $ptemp, $CV_GAUSSIAN, 3, 3) ;
_cveCvtColorMat($ptemp, $TemplateGray, $CV_BGR2GRAY) ;
_cveCannyMat($TemplateGray, $TemplateGray, 50, 200) ; kontyre
#EndRegion Template

#Region Image
;Load image from file
Local $pimg = _cveImreadAndCheck("bridge200p.jpg") ;Opencv function to load image
;// Create a destination image to hold output
;//smooth and converting the original image into grayscale
Local $pimgGrayScale = _cveMatCreate()
;_cveSmoothMat($pimg, $pimg, $CV_GAUSSIAN, 3, 3) ;
_cveCvtColorMat($pimg, $pimgGrayScale, $CV_BGR2GRAY) ;
; Canny + Resize + Canny will give wrong matches
; _cveCannyMat($pimgGrayScale, $pimgGrayScale, 50, 200) ; kontyre
;_cvEqualizeHistMat( $pimgGrayScale, $pimgGrayScale ) ; osvetlit'
#EndRegion Image

; show input image
; _cveImshowMat("Image", $pimgGrayScale)

; show template image
; _cveImshowMat("Template", $TemplateGray)

Local $width2 = _cveMatGetWidth($TemplateGray)
Local $height2 = _cveMatGetHeight($TemplateGray)

Local $pimgResized, $scale, $width, $height, $rw, $rh, $presult
Local $CurMaxVal = 0, $CurLocX = 0, $CurLocY = 0, $CurScale = 1
Local $tBlackColor = _cvScalar(0)
Local $tRedColor = _cvRGB(255, 0, 0)
Local $tWhiteColor = _cvRGB(255, 255, 255)
Local $tDsize = _cvSize(0, 0)

For $i = 1.5 To 2.5 Step 0.25
    ;//Resize image
    $scale = $i

    $tDsize.width = _cveMatGetWidth($pimgGrayScale) / $scale
    $tDsize.height = _cveMatGetHeight($pimgGrayScale) / $scale
    If ($tDsize.width < $width2) Or ($tDsize.height < $height2) Then
        ExitLoop
    EndIf

    $pimgResized = _cveMatCreate()
    _cveResizeMat($pimgGrayScale, $pimgResized, $tDsize, 0, 0, $CV_INTER_LINEAR)
    _cveCannyMat($pimgResized, $pimgResized, 50, 200) ; Canny after resize will give better results

    $width = _cveMatGetWidth($pimgResized)
    $height = _cveMatGetHeight($pimgResized)

    $rw = $width - $width2 + 1
    $rh = $height - $height2 + 1
    ;// Create Opencv matrix object 32 bit floating
    $presult = _cveMatCreate()
    _cveMatCreateData($presult, $rh, $rw, $CV_32FC1)

    ;// Template matching
    _cveMatchTemplateMat($pimgResized, $TemplateGray, $presult, $CV_TM_CCOEFF_NORMED, $matEmpty)

    ;// Create minmax variables to pass to opencv
    Local $tmaxloc = DllStructCreate($tagCvPoint)
    Local $tminloc = DllStructCreate($tagCvPoint)
    Local $tmaxval = DllStructCreate("double max;")
    Local $tminval = DllStructCreate("double min;")

    ;// create mask to
    ;// Find location of maximum
    _cveMinMaxLocMat($presult, $tminval, $tmaxval, $tminloc, $tmaxloc, $matEmpty)

    ; Local $tImgRect = _cvRect($tmaxloc.x - 5, $tmaxloc.y - 5, $width2 + 15, $height2 + 15)
    Local $tImgRect = _cvRect($tmaxloc.x, $tmaxloc.y, $width2, $height2)
    _cveRectangleMat($pimgResized, $tImgRect, $tWhiteColor, 1, $CV_LINE_8, 0)
    $tImgRect = 0

    ;// Update input image
    _cveImshowMat("", $pimgResized)

    ConsoleWrite("Scale: " & $scale & " " & $tmaxval.max & @CRLF)

    If $CurMaxVal < $tmaxval.max Then
        $CurMaxVal = $tmaxval.max
        $CurScale = $scale
        $CurLocX = $tmaxloc.x * $scale
        $CurLocY = $tmaxloc.y * $scale
        MsgBox(0, "", "Changed. Scale: " & $scale)
    EndIf

    Sleep(1000)
    _cveMatRelease($presult) ;
    _cveMatRelease($pimgResized) ;
Next

;// Mask it to find others if exists and draw red rectangle on input image
;_cveRectangle($pmask, _cvPoint($CurLocX - 5, $CurLocY - 5), _cvPoint($CurLocX + $width2, $CurLocY + $height2), $tBlackColor, -1, 8, 0)
Local $tImgRect = _cvRect($CurLocX, $CurLocY, $width2 * $CurScale, $height2 * $CurScale)
_cveRectangleMat($pimg, $tImgRect, $tRedColor, 1, 8, 0)
$tImgRect = 0

;// Update input image
;_cveImshowMat("Example-Resize", $pimgResized)
_cveImshowMat("", $pimg)

_cveWaitKey()

$tDsize = 0
$tWhiteColor = 0
$tRedColor = 0
$tBlackColor = 0

_cveMatRelease($ptemp)
_cveMatRelease($TemplateGray)
_cveMatRelease($pimg) ;
_cveMatRelease($pimgGrayScale) ;
_cveMatRelease($matEmpty) ;
_cveDestroyAllWindows()

_Opencv_DLLClose()
_GDIPlus_Shutdown()
Exit

Edited July 13, 2021 by smbape

Lion66 · July 13, 2021

Perfect! Thank you!
It does exactly what I need!
I see you made a lot of changes. Later I study it.
Now technical question:
What files must be left for work from two packages (dll and au3)?

Now they are too much. I would like to limit the minimum files.

Why - 5 ? Why +15?

This is a few pixels on the width of the frame lines so that the frame is outside the sample found.

Not essential.

Edited July 13, 2021 by Lion66

smbape · July 13, 2021

I can't tell. Those files where generated by a script.

The dll is needed for sure.

cve_extra.au3 has some convinient functions that may not be needed for you.
It also includes cve_world.au3 which includes all the generated files.

To know the files that really need to be included, remove #include <emgucv-autoit-bindings\cve_extra.au3> and repeat the next process

Run the script
Find the file which define the missing functions
Include it
Repeat until there are not more missing functions

You can also create your own .au3 which contains only the needed functions

Edited July 13, 2021 by smbape

Lion66 · July 14, 2021

@ smbape

Вопрос был более относится к DLL , и это хорошо , что один достаточно.

Для моих примеров был 21 файл AU3 😃

Вы проделали большую интересную работу и полезный UDF.

Думаю, вам нужно сделать новую тему и рассказать на примерах своего проекта. Так ваш проект увидит больше людей.

Может попросить администратора перенести наши сообщения в новую тему. И из этой темы сделать ссылку.

На мой взгляд, ваш проект заслуживает того, чтобы его добавили в Wiki UDF .

Удачи в будущем!

Edited July 15, 2021 by Lion66

smbape · July 14, 2021

@Lion66

Thanks for the encouragement.

I will make a new topic when I am ready.

Lion66 · July 15, 2021

@ smbape

A new question: Is there in you project a function to search on the screen?

As in the file OpenCV-Match_UDF.au3 in this post :

; Name ..........: _MatchPicture
; Description ...: Searches for a picture on screen or on a specific area of the screen and returns the coordinates where the picture has been found.

Thank you.

smbape · July 15, 2021

There is no function to search for an image.

However there is _cveMatchTemplate that can be used for this purpose.

For screen search, you can load the screen as a matrix, and from there use any opencv functions.

Here is how to load a screen capture as a matrix

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "emgucv-autoit-bindings\cve_extra.au3"

Local $tDesktopScreenRect = _WinAPI_GetDesktopScreenRect()
ConsoleWrite("$tDesktopScreenRect.left: " & $tDesktopScreenRect.left & @CRLF)
ConsoleWrite("$tDesktopScreenRect.top: " & $tDesktopScreenRect.top & @CRLF)
ConsoleWrite("$tDesktopScreenRect.right: " & $tDesktopScreenRect.right & @CRLF)
ConsoleWrite("$tDesktopScreenRect.bottom: " & $tDesktopScreenRect.bottom & @CRLF)
ConsoleWrite(@CRLF)

Local $iLeft = $tDesktopScreenRect.left
Local $iTop = $tDesktopScreenRect.top
Local $biWidth = $tDesktopScreenRect.right - $tDesktopScreenRect.left
Local $biHeight = $tDesktopScreenRect.bottom - $tDesktopScreenRect.top
Local $iChannels = 4

Local $hWnd = _WinAPI_GetDesktopWindow()
Local $hDesktopDC = _WinAPI_GetDC($hWnd)
Local $hMemoryDC = _WinAPI_CreateCompatibleDC($hDesktopDC) ;create compatible memory DC

Local $tBIHDR = DllStructCreate($tagBITMAPINFO)
$tBIHDR.biSize = DllStructGetSize($tBIHDR)
$tBIHDR.biWidth = $biWidth
$tBIHDR.biHeight = -$biHeight
$tBIHDR.biPlanes = 1
$tBIHDR.biBitCount = $iChannels * 8

Local $aDIB = DllCall("gdi32.dll", "ptr", "CreateDIBSection", "hwnd", 0, "struct*", $tBIHDR, "uint", 0, "ptr*", 0, "ptr", 0, "dword", 0)
_WinAPI_SelectObject($hMemoryDC, $aDIB[0])

Local $tBits = DllStructCreate('byte[' & $biWidth * $biHeight * $iChannels & ']', $aDIB[4])
_WinAPI_BitBlt($hMemoryDC, 0, 0, $biWidth, $biHeight, $hDesktopDC, $iLeft, $iTop, $SRCCOPY)

_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")
Local $matImg = _cveMatCreateWithData($biHeight, $biWidth, $CV_8UC4, $tBits, 0)
_cveImshowMat("Screen capture", $matImg)
_cveWaitKey(0)

_WinAPI_DeleteObject($aDIB[0])
$tBIHDR = 0
_WinAPI_DeleteDC($hMemoryDC)
_WinAPI_ReleaseDC($hWnd, $hDesktopDC)

_cveMatRelease($matImg)
_cveDestroyWindow("Screen capture")

_Opencv_DLLClose()

Func _WinAPI_GetDesktopScreenRect()
    Local $iRight, $iBottom, $aRetrun

    Local $tRect = DllStructCreate($tagRECT)
    $tRect.Left = 0
    $tRect.Top = 0
    $tRect.Right = -1
    $tRect.Bottom = -1

    Local Const $tagDISPLAY_DEVICE = "dword Size;wchar Name[32];wchar String[128];dword Flags;wchar ID[128];wchar Key[128]"
    Local $tDisplayDevice = DllStructCreate($tagDISPLAY_DEVICE)
    $tDisplayDevice.Size = DllStructGetSize($tDisplayDevice)

    Local $tDisplaySettings = DllStructCreate($tagDEVMODE_DISPLAY)
    $tDisplaySettings.Size = DllStructGetSize($tDisplaySettings)

    Local $iDevNum = 0
    While 1
        ; _WinAPI_EnumDisplayDevices("", $iDevNum)
        $aRetrun = DllCall("user32.dll", "int", "EnumDisplayDevicesW", "ptr", 0, "dword", $iDevNum, "struct*", $tDisplayDevice, "dword", 1)
        If Not $aRetrun[0] Then ExitLoop
        $iDevNum += 1

        If BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MIRRORING_DRIVER) Or Not BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_ATTACHED_TO_DESKTOP) Then
            ContinueLoop
        EndIf

        If BitAND($_cve_debug, 1) Then
            ConsoleWrite($tDisplayDevice.Name & @TAB & "Attached to desktop: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_ATTACHED_TO_DESKTOP) <> 0) & @CRLF)
            ConsoleWrite($tDisplayDevice.Name & @TAB & "Primary: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_PRIMARY_DEVICE) <> 0) & @CRLF)
            ConsoleWrite($tDisplayDevice.Name & @TAB & "Mirroring driver: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MIRRORING_DRIVER) <> 0) & @CRLF)
            ConsoleWrite($tDisplayDevice.Name & @TAB & "VGA compatible: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_VGA_COMPATIBLE) <> 0) & @CRLF)
            ConsoleWrite($tDisplayDevice.Name & @TAB & "Removable: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_REMOVABLE) <> 0) & @CRLF)
            ConsoleWrite($tDisplayDevice.Name & @TAB & "More display modes: " & (BitAND($tDisplayDevice.Flags, $DISPLAY_DEVICE_MODESPRUNED) <> 0) & @CRLF)
            ConsoleWrite(@CRLF)
        Endif

        ; _WinAPI_EnumDisplaySettings($tDisplayDevice.Name, $ENUM_CURRENT_SETTINGS)
        Local $sDevice = $tDisplayDevice.Name
        Local $sTypeOfDevice = 'wstr'
        If Not StringStripWS($sDevice, $STR_STRIPLEADING + $STR_STRIPTRAILING) Then
            $sTypeOfDevice = 'ptr'
            $sDevice = 0
        EndIf
        $aRetrun = DllCall("user32.dll", "bool", "EnumDisplaySettingsW", $sTypeOfDevice, $sDevice, "dword", $ENUM_CURRENT_SETTINGS, "struct*", $tDisplaySettings)
        If Not $aRetrun[0] Then ContinueLoop

        If $tRect.Left > $tDisplaySettings.X Then $tRect.Left = $tDisplaySettings.X
        If $tRect.Top > $tDisplaySettings.Y Then $tRect.Left = $tDisplaySettings.Y

        $iRight = $tDisplaySettings.X + $tDisplaySettings.PelsWidth
        If $tRect.Right < $iRight Then $tRect.Right = $iRight

        $iBottom = $tDisplaySettings.Y + $tDisplaySettings.PelsHeight
        If $tRect.Bottom < $iBottom Then $tRect.Bottom = $iBottom
    WEnd

    $tDisplaySettings = 0
    $tDisplayDevice = 0

    Return $tRect
EndFunc   ;==>_WinAPI_GetDesktopScreenRect

Edited July 15, 2021 by smbape

Lion66 · July 16, 2021

Thank you.

With your example, I was able to make a script to find icon by template on the desktop and click by mouse.

I am sure that you offered the best option.

But I am curious, can I use the _ScreenCapture_Capture function for these purposes ?

In such style:

_cveImreadAndCheck(_ScreenCapture_Capture("", 0, 0, -1, -1, false))

smbape · July 16, 2021

There may be a way to use screen capture, but I don't know how.

_ScreenCapture_Capture("", 0, 0, -1, -1, false)

returns a bitmap as in

Local $hBMP = _WinAPI_CreateCompatibleBitmap($hDDC, $iW, $iH)

If you cat get from the bitmap something equivalent to what you get from

Local $tBits = DllStructCreate('byte[' & $biWidth * $biHeight * $iChannels & ']', $aDIB[4])
_WinAPI_BitBlt($hMemoryDC, 0, 0, $biWidth, $biHeight, $hDesktopDC, $iLeft, $iTop, $SRCCOPY)

then you can use it.

A less efficient alternative will be: save the screen capture into a file, the load it with _cveImread

Lion66 · July 17, 2021

On 11.07.2021 at 22:40, smbape said:

Keep in mind that cv::matchTemplate is reliable only if you know that the template is present in the bigger image.
If not, it has to be combined with something that tells you how good the match is, for example cv::compareHist
Look at _cveFindTemplate in cve_extra.au3 for a "work in progress" way to reliably find a template in a bigger image.

Now I've come to this, and I don't know where and how to apply it.

Please show me how it's done.

smbape · July 19, 2021

I was busy finding a satisfiying way to do that.
I ended up hidding all the complicated code in _cveFindTemplate

With the new udf version below there is an example of how you can find matches of a template.
For your own images, adjust parameters, usually threshold is enough, until you get a satisfying result.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include "emgucv-autoit-bindings\cve_extra.au3"

_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")

Local $tDesktopScreenRect = _WinAPI_GetDesktopScreenRect()
ConsoleWrite("$tDesktopScreenRect.left: " & $tDesktopScreenRect.left & @CRLF)
ConsoleWrite("$tDesktopScreenRect.top: " & $tDesktopScreenRect.top & @CRLF)
ConsoleWrite("$tDesktopScreenRect.right: " & $tDesktopScreenRect.right & @CRLF)
ConsoleWrite("$tDesktopScreenRect.bottom: " & $tDesktopScreenRect.bottom & @CRLF)
ConsoleWrite(@CRLF)

Local $tBits = _cveGetDesktopScreenBits($tDesktopScreenRect)
Local $matImg = _cveMatCreateWithData($tDesktopScreenRect.bottom - $tDesktopScreenRect.top, $tDesktopScreenRect.right - $tDesktopScreenRect.left, $CV_8UC4, $tBits, 0)
Local $matTempl = _cveImreadAndCheck("mario_coin.png")

Local $w = _cveMatGetWidth($matTempl)
Local $h = _cveMatGetHeight($matTempl)

; we compute the histogram from the B G R channels
;Local $aChannels[3] = [0, 1, 2]
Local $aChannels = _cveDefaultBGRChannels()

; Quantize each color to 32 levels
; The higher the value, the closer to the template the matched color is
; Local $aHistSize[3] = [32, 32, 32]
Local $aHistSize = _cveDefaultBGRHistSize()

; Each color varies from 0 to 255
; Local $aRanges[6] = [0, 256, 0, 256, 0, 256]
Local $aRanges = _cveDefaultBGRRanges()

; The higher the value, the higher the match is exact
Local $fThreshold = 0.8

; Ignore the alpha channel when matching the template => $CV_COLOR_BGRA2BGR
; Only consider gray scale colors when matching the template => $CV_COLOR_BGRA2GRAY
; using cveCanny and/or gray scale can significantly reduce the matching time but also reduce the matching accuracy
Local $hTimer = TimerInit()
Local $aMatches = _cveFindTemplate($matImg, $matTempl, $fThreshold, $CV_COLOR_BGRA2GRAY)
Local $fDiff = TimerDiff($hTimer)
ConsoleWrite("Finding matches took " & $fDiff & "ms" & @CRLF)

Local $iMatches = UBound($aMatches)
ConsoleWrite("Number of matches: " & $iMatches & @CRLF)

Local $tRedColor = _cvRGB(255, 0, 0)
Local $tMatchRect = _cvRect(0, 0, $w, $h)

For $i = 0 To $iMatches - 1
    $tMatchRect.x = $aMatches[$i][0]
    $tMatchRect.y = $aMatches[$i][1]

    ; Draw a red rectangle around the matched position
    _cveRectangleMat($matImg, $tMatchRect, $tRedColor, 1, $CV_LINE_8, 0)
Next

_cveImshowMat("Source Image", $matImg)
_cveWaitKey(0)

$tMatchRect = 0
$tRedColor = 0

_cveMatRelease($matTempl)
_cveMatRelease($matImg)
$tBits = 0

_cveDestroyAllWindows()

_Opencv_DLLClose()

emgucv-autoit-bindings.zip

Lion66 · July 20, 2021

Fun picture. And the example pleased.

Again, a different approach. I can't keep up with your creativity!

Thank you.

Lion66 · July 27, 2021

Hi @smbape

Is it possible with you UDF make a detect the key points using SURF detector, or others: AKAZE, SIFT?
As in this example.
I found your respective functions, but I myself can not create the logic.
Can you show an example?
Thanks.

smbape · July 27, 2021

Hi @Lion66

At the present time I can't give you an example.

SURF is a patented algorithm.

Try to replace SURF with ORB and FLANNBASED (1) with BRUTEFORCE_HAMMING (5)

There should be an UDF with ORB, detectAndCompute, knnMatch.

If not, I will try when I have more time

Lion66 · July 28, 2021

Your sentence doesn't mention AKAZE, KAZE, SIFT.
Does that mean they're patented, too?

smbape · July 29, 2021

Hi @Lion66

I managed to make a functionnal example.
It was hard. I had to look into the function signatures to know the expected parameters.
I had to modify the UDF.

AKAZE, KAZE, SIFT algorithms are free

Here is an example with ORB working with the new UDF version

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

Opt("MustDeclareVars", 1)

#include <ButtonConstants.au3>
#include <ComboConstants.au3>
#include <EditConstants.au3>
#include <File.au3>
#include <FileConstants.au3>
#include <GDIPlus.au3>
#include <GuiComboBox.au3>
#include <GUIConstantsEx.au3>
#include <Math.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "emgucv-autoit-bindings\cve_extra.au3"

;~ Sources:
;~     https://docs.opencv.org/4.5.2/d7/dff/tutorial_feature_homography.html
;~     https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/feature_homography/SURF_FLANN_matching_homography_Demo.cpp

Local Const $OPENCV_SAMPLES_DATA_PATH = @ScriptDir

#Region ### START Koda GUI section ### Form=
Local $FormGUI = GUICreate("Features2D + Homography to find a known object", 1000, 707, 192, 95)

Local $InputObject = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box.png", 230, 16, 449, 21)
Local $BtnObject = GUICtrlCreateButton("Object", 689, 14, 75, 25)

Local $InputScene = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box_in_scene.png", 230, 52, 449, 21)
Local $BtnScene = GUICtrlCreateButton("Scene", 689, 50, 75, 25)

Local $LabelMatchType = GUICtrlCreateLabel("Match type", 414, 92, 79, 20)
GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif")
Local $ComboMatchType = GUICtrlCreateCombo("", 502, 92, 177, 25, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE))
GUICtrlSetData(-1, "BruteForce|BruteForce-L1|BruteForce-Hamming|BruteForce-HammingLUT|BruteForce-Hamming(2)|BruteForce-SL2")

Local $BtnExec = GUICtrlCreateButton("Execute", 832, 48, 75, 25)

Local $LabelMatches = GUICtrlCreateLabel("Good Matches & Object detection", 377, 144, 245, 20)
GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif")
Local $GroupMatches = GUICtrlCreateGroup("", 20, 166, 958, 532)
Local $PicMatches = GUICtrlCreatePic("", 25, 177, 948, 516)
GUICtrlCreateGroup("", -99, -99, 1, 1)

GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

Local $aMatchTypes[6] = [ _
    $CV_NORM_L2, _
    $CV_NORM_L1, _
    $CV_NORM_HAMMING, _
    $CV_NORM_HAMMING, _
    $CV_NORM_HAMMING2, _
    $CV_NORM_L2SQR _
]
_GUICtrlComboBox_SetCurSel($ComboMatchType, 2)

_GDIPlus_Startup()
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")

Local $img_object, $img_scene
Local $nMsg
Local $sObject, $sScene
Local $tBackgroundColor = _cvRGB(0xF0, 0xF0, 0xF0)

Main()

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit
        Case $BtnObject
            $sObject = ControlGetText($FormGUI, "", $InputObject)
            $sObject = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sObject)
            If @error Then
                $sObject = ""
            Else
                ControlSetText($FormGUI, "", $InputObject, $sObject)
            EndIf
        Case $BtnScene
            $sScene = ControlGetText($FormGUI, "", $InputScene)
            $sScene = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sScene)
            If @error Then
                $sScene = ""
            Else
                ControlSetText($FormGUI, "", $InputScene, $sScene)
            EndIf
        Case $ComboMatchType
            Detect()
        Case $BtnExec
            Clean()
            Main()
    EndSwitch
WEnd

_Opencv_DLLClose()
_GDIPlus_Shutdown()

Func Main()
    ;;! [load_image]
    ;;/ Load object and scene
    $sObject = ControlGetText($FormGUI, "", $InputObject)
    $img_object = _cveImreadAndCheck($sObject, $CV_IMREAD_GRAYSCALE)
    If @error Then
        $sObject = ""
        Return
    EndIf

    $sScene = ControlGetText($FormGUI, "", $InputScene)
    $img_scene = _cveImreadAndCheck($sScene, $CV_IMREAD_GRAYSCALE)
    If @error Then
        _cveMatRelease($img_object)
        $sObject = ""
        $sScene = ""
        Return
    EndIf
    ;;! [load_image]


    Detect()
EndFunc   ;==>Main

Func Clean()
    If $sObject == "" Then Return

    _cveMatRelease($img_object)
    _cveMatRelease($img_scene)

    $sObject = ""
EndFunc   ;==>Clean

Func Detect()
    Local $match_type = $aMatchTypes[_GUICtrlComboBox_GetCurSel($ComboMatchType)]

    ;;-- Step 1: Detect the keypoints using ORB Detector, compute the descriptors
    Local $numberOfFeatures = 500 ;
    Local $tFeature2DPtr = DllStructCreate("ptr value")
    Local $tSharedPtr = DllStructCreate("ptr")
    _cveOrbCreate($numberOfFeatures, 1.2, 8, 31, 0, 2, $CV_ORB_HARRIS_SCORE, 31, 20, $tFeature2DPtr, $tSharedPtr)
    Local $detector = $tFeature2DPtr.value
    Local $keypoints_object = _VectorOfKeyPointCreate()
    Local $keypoints_scene = _VectorOfKeyPointCreate()
    Local $descriptors_object = _cveMatCreate()
    Local $descriptors_scene = _cveMatCreate()
    _CvFeature2DDetectAndComputeMat($detector, $img_object, _cveNoArrayMat(), $keypoints_object, $descriptors_object, False) ;
    _CvFeature2DDetectAndComputeMat($detector, $img_scene, _cveNoArrayMat(), $keypoints_scene, $descriptors_scene, False) ;

    ;;-- Step 2: Matching descriptor vectors with a BruteForce based matcher
    ;; Since ORB is a floating-point descriptor NORM_L2 is used
    Local $tMatcherPtr = DllStructCreate("ptr value")
    Local $bf_matcher = _cveBFMatcherCreate($match_type, False, $tMatcherPtr) ;
    Local $matcher = $tMatcherPtr.value
    Local $knn_matches = _VectorOfVectorOfDMatchCreate() ;
    _cveDescriptorMatcherKnnMatch1Mat($matcher, $descriptors_object, $descriptors_scene, $knn_matches, 2, _cveNoArrayMat(), False) ;

    ;;-- Filter matches using the Lowe's ratio test
    Local $ratio_thresh = 0.75 ;
    Local $good_matches = _VectorOfDMatchCreate() ;
    Local $tVectorDMatchPtr = DllStructCreate("ptr value")
    Local $tDMatchPtr0 = DllStructCreate("ptr value")
    Local $tDMatchPtr1 = DllStructCreate("ptr value")

    For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1
        _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $i, $tVectorDMatchPtr)

        _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 0, $tDMatchPtr0)
        Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value)

        _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 1, $tDMatchPtr1)
        Local $tDMatch1 = DllStructCreate($tagDMatch, $tDMatchPtr1.value)

        If $tDMatch0.distance < $ratio_thresh * $tDMatch1.distance Then
            _VectorOfDMatchPush($good_matches, $tDMatch0)
        EndIf
    Next

    ;;-- Draw matches
    Local $img_matches = _cveMatCreate() ;
    Local $matchesMask = _VectorOfByteCreate()
    _drawMatchedFeatures1Mat($img_object, $keypoints_object, $img_scene, $keypoints_scene, $good_matches, $img_matches, _cvScalarAll(-1), _
            _cvScalarAll(-1), $matchesMask, $CV_DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS) ;

    ;;-- Localize the object
    Local $obj = _VectorOfPointFCreate() ;
    Local $scene = _VectorOfPointFCreate() ;
    Local $tObjectKeyPointPtr = DllStructCreate("ptr value")
    Local $tSceneKeyPointPtr = DllStructCreate("ptr value")

    For $i = 0 To _VectorOfDMatchGetSize($good_matches) - 1
        ;;-- Get the keypoints from the good matches
        _VectorOfDMatchGetItemPtr($good_matches, $i, $tDMatchPtr0)
        Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value)

        _VectorOfKeyPointGetItemPtr($keypoints_object, $tDMatch0.queryIdx, $tObjectKeyPointPtr)
        Local $tObjectKeyPoint = DllStructCreate($tagKeyPoint, $tObjectKeyPointPtr.value)
        _VectorOfPointFPush($obj, $tObjectKeyPoint)

        _VectorOfKeyPointGetItemPtr($keypoints_scene, $tDMatch0.trainIdx, $tSceneKeyPointPtr)
        Local $tSceneKeyPoint = DllStructCreate($tagKeyPoint, $tSceneKeyPointPtr.value)
        _VectorOfPointFPush($scene, $tSceneKeyPoint)
    Next

    Local $H = _cveMatCreate()
    Local $i_arr_H = _cveInputArrayFromMat($H)
    Local $o_arr_H = _cveOutputArrayFromMat($H)
    Local $i_arr_obj = _cveInputArrayFromVectorOfPointF($obj)
    Local $i_arr_scene = _cveInputArrayFromVectorOfPointF($scene)
    Local $resultMask = _cveMatCreate()
    Local $o_arr_resultMask = _cveOutputArrayFromMat($resultMask)
    _cveFindHomography($i_arr_obj, $i_arr_scene, $o_arr_H, $CV_RANSAC, 3, $o_arr_resultMask) ;
    _cveOutputArrayRelease($o_arr_resultMask)
    _cveMatRelease($resultMask)

    If Not _cveMatIsEmpty($H) Then
        ;;-- Get the corners from the image_1 ( the object to be "detected" )
        Local $img_object_size = _cvSize()
        _cveMatGetSize($img_object, $img_object_size)

        Local $obj_corners = _VectorOfPointFCreate()
        _VectorOfPointFPush($obj_corners, _cvPoint2f(0, 0))
        _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, 0))
        _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, $img_object_size.height))
        _VectorOfPointFPush($obj_corners, _cvPoint2f(0, $img_object_size.height))

        Local $scene_corners = _VectorOfPointFCreateSize(4)

        Local $i_arr_obj_corners = _cveInputArrayFromVectorOfPointF($obj_corners)
        Local $o_arr_scene_corners = _cveOutputArrayFromVectorOfPointF($scene_corners)

        _cvePerspectiveTransform($i_arr_obj_corners, $o_arr_scene_corners, $i_arr_H) ;

        Local $tPointFPtr = DllStructCreate("ptr value")

        _VectorOfPointFGetItemPtr($scene_corners, 0, $tPointFPtr)
        Local $scene_corners_0 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

        _VectorOfPointFGetItemPtr($scene_corners, 1, $tPointFPtr)
        Local $scene_corners_1 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

        _VectorOfPointFGetItemPtr($scene_corners, 2, $tPointFPtr)
        Local $scene_corners_2 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

        _VectorOfPointFGetItemPtr($scene_corners, 3, $tPointFPtr)
        Local $scene_corners_3 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

        ;;-- Draw lines between the corners (the mapped object in the scene - image_2 )
        _cveLineMat($img_matches, _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _
                _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _cvScalar(0, 255, 0), 4) ;

        _cveLineMat($img_matches, _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _
                _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _cvScalar(0, 255, 0), 4) ;

        _cveLineMat($img_matches, _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _
                _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _cvScalar(0, 255, 0), 4) ;

        _cveLineMat($img_matches, _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _
                _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _cvScalar(0, 255, 0), 4) ;

        _cveOutputArrayRelease($o_arr_scene_corners)
        _cveInputArrayRelease($i_arr_obj_corners)
        _VectorOfPointFRelease($scene_corners)
        _VectorOfPointFRelease($obj_corners)
    EndIf

    _cveInputArrayRelease($i_arr_scene)
    _cveInputArrayRelease($i_arr_obj)
    _cveOutputArrayRelease($o_arr_H)
    _cveInputArrayRelease($i_arr_H)
    _cveMatRelease($H)
    _VectorOfPointFRelease($scene)
    _VectorOfPointFRelease($obj)

    ;-- Show detected matches
    ; _cveImshowMat("Good Matches & Object detection", $img_matches) ;
    _cveImshowControlPic($img_matches, $FormGUI, $PicMatches, $tBackgroundColor)

    _VectorOfByteRelease($matchesMask)
    _cveMatRelease($img_matches)
    _VectorOfDMatchRelease($good_matches)
    _VectorOfVectorOfDMatchRelease($knn_matches)
    _cveBFMatcherRelease($bf_matcher)
    _cveMatRelease($descriptors_scene)
    _cveMatRelease($descriptors_object)
    _VectorOfKeyPointRelease($keypoints_scene)
    _VectorOfKeyPointRelease($keypoints_object)
    _cveOrbRelease($tSharedPtr)
EndFunc   ;==>Detect

emgucv-autoit-bindings.zip

Edited July 29, 2021 by smbape

Lion66 · July 30, 2021

Hi smbape

I went and found a short working code on Python, from which I wanted to start:

import cv2
from matplotlib import pyplot as plt
img1 = cv2.imread('box.png', cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread('box_in_scene.png', cv2.IMREAD_GRAYSCALE)
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1, des2)
matches = sorted(matches, key=lambda x: x.distance)
img3 = cv2.drawMatches(img1, kp1, img2, kp2, matches[:20], img2, flags=2)
cv2.imshow("Matches", img3)
cv2.waitKey()

During this time, you wrote a huge piece of code!

I can not imagine how it is possible!

I'm shocked and thanks for your work!

I would like to explain the result I want to get:

In the picture of the scene, find the center of the rectangle of the found object to click on the mouse.

Now about your code:

I like the concept of your example, but some pairs of pictures does not accept, the program closes.

I will attach such files, and when you have the opportunity to test them.

Thanks.

smbape · July 30, 2021

2 hours ago, Lion66 said:

In the picture of the scene, find the center of the rectangle of the found object to click on the mouse.

Matching homography to find a known object can be tedious and I am not well informed in the subject.
If the object always has the same size and orientation in the scene, I would have used matchTemplate like a previous example.
If the object always has the same orientation but different size in a known scale range, I would have used multiscale matching like a previous example.
Otherwise, I have no clue.

2 hours ago, Lion66 said:

I like the concept of your example, but some pairs of pictures does not accept, the program closes.

There is an error at _cveFindHomography.

That is because there is a need for at least 4 point correspondences to calculate Homography.

Cases where _VectorOfDMatchGetSize($good_matches) < 4 will crash.

Here is an updated version handling that case

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

Opt("MustDeclareVars", 1)

#include <ButtonConstants.au3>
#include <ComboConstants.au3>
#include <EditConstants.au3>
#include <File.au3>
#include <FileConstants.au3>
#include <GDIPlus.au3>
#include <GuiComboBox.au3>
#include <GUIConstantsEx.au3>
#include <Math.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include "emgucv-autoit-bindings\cve_extra.au3"

;~ Sources:
;~     https://docs.opencv.org/4.5.2/d7/dff/tutorial_feature_homography.html
;~     https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/feature_homography/SURF_FLANN_matching_homography_Demo.cpp

Local Const $OPENCV_SAMPLES_DATA_PATH = @ScriptDir

#Region ### START Koda GUI section ### Form=
Local $FormGUI = GUICreate("Features2D + Homography to find a known object", 1000, 707, 192, 95)

Local $InputObject = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box.png", 230, 16, 449, 21)
Local $BtnObject = GUICtrlCreateButton("Object", 689, 14, 75, 25)

Local $InputScene = GUICtrlCreateInput($OPENCV_SAMPLES_DATA_PATH & "\box_in_scene.png", 230, 52, 449, 21)
Local $BtnScene = GUICtrlCreateButton("Scene", 689, 50, 75, 25)

Local $LabelMatchType = GUICtrlCreateLabel("Match type", 414, 92, 79, 20)
GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif")
Local $ComboMatchType = GUICtrlCreateCombo("", 502, 92, 177, 25, BitOR($GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE))
GUICtrlSetData(-1, "BruteForce|BruteForce-L1|BruteForce-Hamming|BruteForce-HammingLUT|BruteForce-Hamming(2)|BruteForce-SL2")

Local $BtnExec = GUICtrlCreateButton("Execute", 832, 48, 75, 25)

Local $LabelMatches = GUICtrlCreateLabel("Good Matches & Object detection", 377, 144, 245, 20)
GUICtrlSetFont(-1, 10, 800, 0, "MS Sans Serif")
Local $GroupMatches = GUICtrlCreateGroup("", 20, 166, 958, 532)
Local $PicMatches = GUICtrlCreatePic("", 25, 177, 948, 516)
GUICtrlCreateGroup("", -99, -99, 1, 1)

GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

Local $aMatchTypes[6] = [ _
    $CV_NORM_L2, _
    $CV_NORM_L1, _
    $CV_NORM_HAMMING, _
    $CV_NORM_HAMMING, _
    $CV_NORM_HAMMING2, _
    $CV_NORM_L2SQR _
]
_GUICtrlComboBox_SetCurSel($ComboMatchType, 2)

_GDIPlus_Startup()
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.2.4673\libs\x64\cvextern.dll")

Local $img_object, $img_scene
Local $nMsg
Local $sObject, $sScene
Local $tBackgroundColor = _cvRGB(0xF0, 0xF0, 0xF0)

Main()

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit
        Case $BtnObject
            $sObject = ControlGetText($FormGUI, "", $InputObject)
            $sObject = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sObject)
            If @error Then
                $sObject = ""
            Else
                ControlSetText($FormGUI, "", $InputObject, $sObject)
            EndIf
        Case $BtnScene
            $sScene = ControlGetText($FormGUI, "", $InputScene)
            $sScene = FileOpenDialog("Select an image", $OPENCV_SAMPLES_DATA_PATH, "Image files (*.bmp;*.jpg;*.jpeg;*.png;*.gif)", $FD_FILEMUSTEXIST, $sScene)
            If @error Then
                $sScene = ""
            Else
                ControlSetText($FormGUI, "", $InputScene, $sScene)
            EndIf
        Case $ComboMatchType
            Detect()
        Case $BtnExec
            Clean()
            Main()
    EndSwitch
WEnd

_Opencv_DLLClose()
_GDIPlus_Shutdown()

Func Main()
    ;;! [load_image]
    ;;/ Load object and scene
    $sObject = ControlGetText($FormGUI, "", $InputObject)
    $img_object = _cveImreadAndCheck($sObject, $CV_IMREAD_GRAYSCALE)
    If @error Then
        $sObject = ""
        Return
    EndIf

    $sScene = ControlGetText($FormGUI, "", $InputScene)
    $img_scene = _cveImreadAndCheck($sScene, $CV_IMREAD_GRAYSCALE)
    If @error Then
        _cveMatRelease($img_object)
        $sObject = ""
        $sScene = ""
        Return
    EndIf
    ;;! [load_image]

    Detect()
EndFunc   ;==>Main

Func Clean()
    If $sObject == "" Then Return

    _cveMatRelease($img_object)
    _cveMatRelease($img_scene)

    $sObject = ""
EndFunc   ;==>Clean

Func Detect()
    Local $match_type = $aMatchTypes[_GUICtrlComboBox_GetCurSel($ComboMatchType)]

    ;;-- Step 1: Detect the keypoints using ORB Detector, compute the descriptors
    Local $numberOfFeatures = 500 ;
    Local $tFeature2DPtr = DllStructCreate("ptr value")
    Local $tSharedPtr = DllStructCreate("ptr")
    _cveOrbCreate($numberOfFeatures, 1.2, 8, 31, 0, 2, $CV_ORB_HARRIS_SCORE, 31, 20, $tFeature2DPtr, $tSharedPtr)
    Local $detector = $tFeature2DPtr.value
    Local $keypoints_object = _VectorOfKeyPointCreate()
    Local $keypoints_scene = _VectorOfKeyPointCreate()
    Local $descriptors_object = _cveMatCreate()
    Local $descriptors_scene = _cveMatCreate()
    _CvFeature2DDetectAndComputeMat($detector, $img_object, _cveNoArrayMat(), $keypoints_object, $descriptors_object, False) ;
    _CvFeature2DDetectAndComputeMat($detector, $img_scene, _cveNoArrayMat(), $keypoints_scene, $descriptors_scene, False) ;

    ;;-- Step 2: Matching descriptor vectors with a BruteForce based matcher
    ;; Since ORB is a floating-point descriptor NORM_L2 is used
    Local $tMatcherPtr = DllStructCreate("ptr value")
    Local $bf_matcher = _cveBFMatcherCreate($match_type, False, $tMatcherPtr) ;
    Local $matcher = $tMatcherPtr.value
    Local $knn_matches = _VectorOfVectorOfDMatchCreate() ;
    _cveDescriptorMatcherKnnMatch1Mat($matcher, $descriptors_object, $descriptors_scene, $knn_matches, 2, _cveNoArrayMat(), False) ;

    ;;-- Filter matches using the Lowe's ratio test
    Local $ratio_thresh = 0.75 ;
    Local $good_matches = _VectorOfDMatchCreate() ;
    Local $tVectorDMatchPtr = DllStructCreate("ptr value")
    Local $tDMatchPtr0 = DllStructCreate("ptr value")
    Local $tDMatchPtr1 = DllStructCreate("ptr value")

    For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1
        _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $i, $tVectorDMatchPtr)

        _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 0, $tDMatchPtr0)
        Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value)

        _VectorOfDMatchGetItemPtr($tVectorDMatchPtr.value, 1, $tDMatchPtr1)
        Local $tDMatch1 = DllStructCreate($tagDMatch, $tDMatchPtr1.value)

        If $tDMatch0.distance < $ratio_thresh * $tDMatch1.distance Then
            _VectorOfDMatchPush($good_matches, $tDMatch0)
        EndIf
    Next

    ;;-- Draw matches
    Local $img_matches = _cveMatCreate() ;
    Local $matchesMask = _VectorOfByteCreate()
    _drawMatchedFeatures1Mat($img_object, $keypoints_object, $img_scene, $keypoints_scene, $good_matches, $img_matches, _cvScalarAll(-1), _
            _cvScalarAll(-1), $matchesMask, $CV_DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS) ;

    ;;-- Need at least 4 point correspondences to calculate Homography
    If _VectorOfDMatchGetSize($good_matches) >= 4 Then
        ;;-- Localize the object
        Local $obj = _VectorOfPointFCreate() ;
        Local $scene = _VectorOfPointFCreate() ;
        Local $tObjectKeyPointPtr = DllStructCreate("ptr value")
        Local $tSceneKeyPointPtr = DllStructCreate("ptr value")

        For $i = 0 To _VectorOfDMatchGetSize($good_matches) - 1
            ;;-- Get the keypoints from the good matches
            _VectorOfDMatchGetItemPtr($good_matches, $i, $tDMatchPtr0)
            Local $tDMatch0 = DllStructCreate($tagDMatch, $tDMatchPtr0.value)

            _VectorOfKeyPointGetItemPtr($keypoints_object, $tDMatch0.queryIdx, $tObjectKeyPointPtr)
            Local $tObjectKeyPoint = DllStructCreate($tagKeyPoint, $tObjectKeyPointPtr.value)
            _VectorOfPointFPush($obj, $tObjectKeyPoint)

            _VectorOfKeyPointGetItemPtr($keypoints_scene, $tDMatch0.trainIdx, $tSceneKeyPointPtr)
            Local $tSceneKeyPoint = DllStructCreate($tagKeyPoint, $tSceneKeyPointPtr.value)
            _VectorOfPointFPush($scene, $tSceneKeyPoint)
        Next

        Local $H = _cveMatCreate()
        Local $i_arr_H = _cveInputArrayFromMat($H)
        Local $o_arr_H = _cveOutputArrayFromMat($H)
        Local $i_arr_obj = _cveInputArrayFromVectorOfPointF($obj)
        Local $i_arr_scene = _cveInputArrayFromVectorOfPointF($scene)
        Local $resultMask = _cveMatCreate()
        Local $o_arr_resultMask = _cveOutputArrayFromMat($resultMask)
        _cveFindHomography($i_arr_obj, $i_arr_scene, $o_arr_H, $CV_RANSAC, 3, $o_arr_resultMask) ;
        _cveOutputArrayRelease($o_arr_resultMask)
        _cveMatRelease($resultMask)

        If Not _cveMatIsEmpty($H) Then
            ;;-- Get the corners from the image_1 ( the object to be "detected" )
            Local $img_object_size = _cvSize()
            _cveMatGetSize($img_object, $img_object_size)

            Local $obj_corners = _VectorOfPointFCreate()
            _VectorOfPointFPush($obj_corners, _cvPoint2f(0, 0))
            _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, 0))
            _VectorOfPointFPush($obj_corners, _cvPoint2f($img_object_size.width, $img_object_size.height))
            _VectorOfPointFPush($obj_corners, _cvPoint2f(0, $img_object_size.height))

            Local $scene_corners = _VectorOfPointFCreateSize(4)

            Local $i_arr_obj_corners = _cveInputArrayFromVectorOfPointF($obj_corners)
            Local $o_arr_scene_corners = _cveOutputArrayFromVectorOfPointF($scene_corners)

            _cvePerspectiveTransform($i_arr_obj_corners, $o_arr_scene_corners, $i_arr_H) ;

            Local $tPointFPtr = DllStructCreate("ptr value")

            _VectorOfPointFGetItemPtr($scene_corners, 0, $tPointFPtr)
            Local $scene_corners_0 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

            _VectorOfPointFGetItemPtr($scene_corners, 1, $tPointFPtr)
            Local $scene_corners_1 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

            _VectorOfPointFGetItemPtr($scene_corners, 2, $tPointFPtr)
            Local $scene_corners_2 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

            _VectorOfPointFGetItemPtr($scene_corners, 3, $tPointFPtr)
            Local $scene_corners_3 = DllStructCreate($tagCvPoint2f, $tPointFPtr.value)

            ;;-- Draw lines between the corners (the mapped object in the scene - image_2 )
            _cveLineMat($img_matches, _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _
                    _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _cvScalar(0, 255, 0), 4) ;

            _cveLineMat($img_matches, _cvPoint($scene_corners_1.x + $img_object_size.width, $scene_corners_1.y), _
                    _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _cvScalar(0, 255, 0), 4) ;

            _cveLineMat($img_matches, _cvPoint($scene_corners_2.x + $img_object_size.width, $scene_corners_2.y), _
                    _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _cvScalar(0, 255, 0), 4) ;

            _cveLineMat($img_matches, _cvPoint($scene_corners_3.x + $img_object_size.width, $scene_corners_3.y), _
                    _cvPoint($scene_corners_0.x + $img_object_size.width, $scene_corners_0.y), _cvScalar(0, 255, 0), 4) ;

            _cveOutputArrayRelease($o_arr_scene_corners)
            _cveInputArrayRelease($i_arr_obj_corners)
            _VectorOfPointFRelease($scene_corners)
            _VectorOfPointFRelease($obj_corners)
        EndIf

        _cveInputArrayRelease($i_arr_scene)
        _cveInputArrayRelease($i_arr_obj)
        _cveOutputArrayRelease($o_arr_H)
        _cveInputArrayRelease($i_arr_H)
        _cveMatRelease($H)
        _VectorOfPointFRelease($scene)
        _VectorOfPointFRelease($obj)
    EndIf

    ;-- Show detected matches
    ; _cveImshowMat("Good Matches & Object detection", $img_matches) ;
    _cveImshowControlPic($img_matches, $FormGUI, $PicMatches, $tBackgroundColor)

    _VectorOfByteRelease($matchesMask)
    _cveMatRelease($img_matches)
    _VectorOfDMatchRelease($good_matches)
    _VectorOfVectorOfDMatchRelease($knn_matches)
    _cveBFMatcherRelease($bf_matcher)
    _cveMatRelease($descriptors_scene)
    _cveMatRelease($descriptors_object)
    _VectorOfKeyPointRelease($keypoints_scene)
    _VectorOfKeyPointRelease($keypoints_object)
    _cveOrbRelease($tSharedPtr)
EndFunc   ;==>Detect

Edited July 30, 2021 by smbape

Lion66 · July 31, 2021

Thank you, very good turned out an example!
And after I checked my pairs of images on examples in Python, I realized that there was not enough match.
But if I understand correctly, there are two problems here:
1. Not enough coincidences.
2. No 4 points for drawing a frame.
I compared the search for ORB points with search in FAST.

ORB:

import cv2 as cv
imageread = cv.imread('2-border.png')
ORB_object = cv.ORB_create()
keypoints = ORB_object.detect(imageread)
keypoints, descriptors = ORB_object.compute(imageread, keypoints)
imageresult = cv.drawKeypoints(imageread, keypoints, None, color=(255,0,0), flags=0)
cv.imshow('ORB_image', imageresult)
cv.waitKey()

FAST:

import cv2
img = cv2.imread('2-border.png',0)
fast = cv2.FastFeatureDetector_create(threshold=25)
kp = fast.detect(img,None)
img2 = cv2.drawKeypoints(img, kp, None,color=(255,0,0))
cv2.imshow('fast_true',img2),cv2.waitKey()

https://docs.opencv.org/4.5.3/df/d0c/tutorial_py_fast.html#gsc.tab=0

And I see that Fast finds much more points!
Is it possible to use this point search method with any comparison method, and will it work for turning images or with a resizing?

Regarding the second problem:
I would instead of a frame of four corners, would draw a circle with the center in the accumulation of correspondence points. If there are tools for this.
I would be happy to see such a example if you have an interest and there is time.

Thanks.

P.S. In new version of UDF missing $CV_BGR2GRAY=6 in file cv_constants.au3.

Old examples do not work. This is done on purpose?

Edited July 31, 2021 by Lion66

Opencv UDF

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

mylise

junkew

mylise

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members