Jump to content

OpenCV v4 UDF


smbape
 Share

Recommended Posts

Congratulations on the birth of the theme!
Interesting and necessary direction.

I looked at your example matching_homography_Demo.au3 and found some problems:
1. If you try to select an algorithm, the script crashes with a message:

"Error: The image \box.png could not be loaded." , 

If not exist file $OPENCV_SAMPLES_DATA_PATH & "\box.png". Even if another file is selected!

2. I propose move

;-- Show detected matches
    ; _cveImshowMat("Good Matches & Object detection", $img_matches) ;
    _cveImshowControlPic($img_matches, $FormGUI, $PicMatches)

to the end of the cycle

;;-- Need at least 4 point correspondences to calculate Homography
    If _VectorOfDMatchGetSize($good_matches) >= 4 Then


And add an "ELSE" that clears the picture in the GUI and changes the message

"Good Matches && Object detection" to "no matches found."

 


3. Testing with standard "box_in_scene.png" files does not cause any problems.
But when I try to work with custom files, I can see the difference.
I checked about 10 different sets of images and came to the conclusion that the ORB algorithm has low ability to find points.
In the same script (python) when replacing ORB with KAZE and sometimes with BRISK, I get a positive result.
I'm attaching a couple of pictures and a code on the python that recognizes them.
Unfortunately, your script using the same KAZE algorithm cannot do this!
I can't compare the difference in scripts, but if you try to repeat all the steps used in the python, then it should work.

4. It may be worth adding the ability for the user to modify the MIN_MATCH_COUNT to cut off false matches.

Thanks and success in working on the project!

dog.jpg

dog-scene.jpg

KAZE-DetectAndFrame.py

Edited by Lion66
Link to comment
Share on other sites

Hi @Lion66

Thanks for the feedback and the encouragement.

I though that KAZE did not have detectAndCompute implemented, but it is NORM_HAMMING and NORM_HAMMING2 that are not compatible with KAZE. I will update the example to take that into account.

I was testing with BRISK and I had no matches, while with KAZE I have similar results as your python script. See the attached file.

 

3 hours ago, Lion66 said:

1. If you try to select an algorithm, the script crashes with a message:

I don't have that error.
Can you check your value of $OPENCV_SAMPLES_DATA_PATH ?
The examples are designed to work inside the repository.
As  describe in these instructions, can you clone the whole repo, and try again ?

 

3 hours ago, Lion66 said:

2. I propose move

;-- Show detected matches
    ; _cveImshowMat("Good Matches & Object detection", $img_matches) ;
    _cveImshowControlPic($img_matches, $FormGUI, $PicMatches)

to the end of the cycle

;;-- Need at least 4 point correspondences to calculate Homography
    If _VectorOfDMatchGetSize($good_matches) >= 4 Then

 

Are you talking about the file matching_homography_Demo.au3 ?
If yes, it is already at the end of the cycle.
Are you refering to another file?

 

3 hours ago, Lion66 said:

And add an "ELSE" that clears the picture in the GUI and changes the message

"Good Matches && Object detection" to "no matches found."

The message label can be changed to "no matches found." in case of no matches.
However I don't understand the need of clearing then GUI.
$img_matches is always a new image.
Can you describe a test case where clearing the GUI might help?

160732-opencv-udf.au3

Link to comment
Share on other sites

About file 160732-opencv-udf.au3.

Great implementation! Finally, custom pictures are recognized.

1 hour ago, smbape said:

I was testing with BRISK and I had no matches

BRISK applicable to other images. I checked both algorithms. Both show the expected result.

A few questions:

1. Could you add an ORB algorithm here just in case 😉.

2. When there are dots, but they are scattered, and the object is not recognized (there is no red frame in the "found" window), is it possible to track this and issue a message?

3. Is there a way to move the mouse to the center of the red frame in the "found" window?

 

Less important, but still: about matching_homography_Demo.au3:

As a user, I assume that I will use the script with my images and without having a "\samples\data" folder.
Now this causes an emergency closure.

That is, the GUI must be opened without any file.
And select an algorithm without having a specified file should not cause an emergency closure.

Many thanks.

Link to comment
Share on other sites

21 minutes ago, Lion66 said:

1. Could you add an ORB algorithm here just in case 😉.

You can do it by adding the following lines where needed (I prefer not to upload the whole file again)

_cveOrbCreate(500, 1.2, 8, 31, 0, 2, $CV_ORB_HARRIS_SCORE, 31, 20, $tFeature2DPtr, $tSharedPtr)
$destructor = "_cveOrbRelease"

 

23 minutes ago, Lion66 said:

2. When there are dots, but they are scattered, and the object is not recognized (there is no red frame in the "found" window), is it possible to track this and issue a message?

That is the case when UBound($dmatches) < 4 or _cveFindHomography did not find any homography.
I will add a ConsoleWriteError in the example

 

30 minutes ago, Lion66 said:

3. Is there a way to move the mouse to the center of the red frame in the "found" window?

The red frame not being a rectangle, the center is not well defined.
You can define the center as the crossing point of the 2 diagonals from the corners.
The four corners points are $dst_0, $dst_1, $dst_2, $dst_3.
The diagonals are [$dst_0, $dst_2] and [$dst_1, $dst_3].
If you remember you mathematical courses, you know how to get the crossing point of two segments.
If not, use your favorite search engine. I don't remember.
If you have hard time finding an answer, ask again in the post.

 

38 minutes ago, Lion66 said:

Now this causes an emergency closure.

I really don't reproduce it. I copied and pasted the coded in an isolated location and I had no crash.
The GUI showed up, but I had a bunch of errors in the console

Error while loading unkinwn.dll
Error - cveMatCreate: unable to use the DLL file
Error - cveStringCreateFromStr: unable to use the DLL file
Error - cveImread: unable to use the DLL file
Error - cveStringRelease: unable to use the DLL file
Error - cveInputArrayFromMat: unable to use the DLL file
Error - cveInputArrayIsEmpty: unable to use the DLL file
Error - cveInputArrayRelease: unable to use the DLL file
!>Error: The image \box.png could not be loaded.

 

Link to comment
Share on other sites

@Lion66

I found a way to get the center of a polygons

; draw a circle at the center
Local $moments = _cveMomentsCreate()
_cveMoments($i_arr_dst_32, False, $moments)
Local $cx = _cveMomentsGetM10($moments) / _cveMomentsGetM00($moments)
Local $cy = _cveMomentsGetM01($moments) / _cveMomentsGetM00($moments)
_cveCircleMat($img2, _cvPoint($cx, $cy), 5, _cvScalar(255, 255, 255), $CV_FILLED)
_cveMomentsRelease($moments)

The code with ORB added and the above code added

 

160732-opencv-udf.au3

Edited by smbape
Link to comment
Share on other sites

Hi @smbape

Thank you for your answers and examples.

My mouse now tends to the center of the polygon. 😄

I added exit to the last script if number of object key point or scene key point  is zero. There's no point in continuing.

For the ORB algorithm, need to change too the match type to $CV_NORM_HAMMING. Otherwise, the frame is not drawn correctly.

And for others, match type $CV_NORM_L2 works better than $CV_NORM_L1.

If $algorithm = "ORB" Then
    $bf_matcher = _cveBFMatcherCreate($CV_NORM_HAMMING, False, $tMatcherPtr)
Else
    $bf_matcher = _cveBFMatcherCreate($CV_NORM_L2, False, $tMatcherPtr)
EndIf

I do not yet understand whether it is possible to track behavior when there were matches, but they are false and the frame is not drawn?

For example, with these pictures and the ORB algorithm, I get 61 matches, but there is no frame.

frame-obj.png

frames-scene.png

Link to comment
Share on other sites

28 minutes ago, Lion66 said:

For example, with these pictures and the ORB algorithm, I get 61 matches, but there is no frame.

The number of matches does not indicate a correlation in images.
It indicates how many descriptors of the object are matched in the descriptors of the scene.
However, it doesn't say if there is an homography that can be applied to transform those matched descriptors of object into the matched descriptors of the scene.
In other word, if object can be transformed and then found in scene.

It will be hard to find a combination that works for every set of images.
The script I provided does not allow to easily test different combinations.
Using matching_homography_Demo.au3 , and with the provided object and scene, the following combinations find the object in the scene:

  • Brisk + ($CV_NORM_L2, $CV_NORM_L1, $CV_NORM_HAMMING, $CV_NORM_HAMMING2, $CV_NORM_L2SQR)
  • KAZE + ($CV_NORM_L2, $CV_NORM_L1, $CV_NORM_L2SQR)
  • AKAZE + ($CV_NORM_L2, $CV_NORM_L1, $CV_NORM_HAMMING, $CV_NORM_HAMMING2, $CV_NORM_L2SQR)
Edited by smbape
Link to comment
Share on other sites

5 hours ago, Lion66 said:

Visually we see this when the object is not detected, and there is no solution for the script to understand this?

I don't understand the question.
What do we see when the object is not detected?

Link to comment
Share on other sites

Ok, I understand the question.

There are 2 cases when you will not see the red frame:

  • After _cveDescriptorMatcherMatch1Mat, If UBound($dmatches) < 4 Then there is not enough matches, which will lead to no red frame
  • After _cveFindHomography, If _cveMatIsEmpty($M) Then there no homography, which will lead to no red frame

If what I said is not clear, post your code and I will modify it.

Edited by smbape
Link to comment
Share on other sites

Take your code 160732-opencv-udf.au3.

Take two previously incompatible pictures: frame-obj.png (attached above ) and box_in_scene.png.

We successfully pass the test "If UBound($dmatches) >= 4 Then". Ok.

But I was expecting an event "else" in loop "If Not _cveMatIsEmpty($M) Then" (I added debug message in Else).

However, the "Else" block is not activated, but there is no red frame.

Link to comment
Share on other sites

There is a red frame.
However, the points are so close that it displays a red line.
Look at the window "found" to see it.

Here are the coordinates of the points

$dst_0 [ 144.033889770508, 192.807693481445 ]
$dst_1 [ 147.521987915039, 180.503540039063 ]
$dst_2 [ 142.773986816406, 197.195739746094 ]
$dst_3 [ 155.550369262695, 152.486877441406 ]

 

Edited by smbape
Link to comment
Share on other sites

That's good. Let's clean up points with close coordinates, say 10 pixels away.

In the hope that this will not harm in other cases.

I think it might be right for that "groupRectangles".

Will you try? 😉

(edit) Or is it easier to compare mathematically?

 

Edited by Lion66
Link to comment
Share on other sites

The distance between 2 points (x1, y1), (x2, y2) is given by sqrt((x1 - x2)² + (y1 - y2)²)
Use it to compare the distance between the 4 corners.
If two corners are too close, do what you want

Local $mindist = 10
Local $points[4]= [$dst_0, $dst_1, $dst_2, $dst_3]
For $i = 0 To 3
    For $j = $i + 1 To 3
        If Sqrt((($points[$i].x - $points[$j].x) ^ 2) + (($points[$i].y - $points[$j].y) ^ 2)) < $mindist Then
            ConsoleWrite("Points are too close" & @CRLF)
            ExitLoop 2
        EndIf
    Next
Next

 

Edited by smbape
Link to comment
Share on other sites

Hi

Beautiful solution and helped improve the code.

Although I prefer to cut only those cases where all 4 points are located close.

Often, displaying a frame as a line also gives a good result. I use it like this:

Local $mindist = 10
Local $points[4] = [$dst_0, $dst_1, $dst_2, $dst_3]
Local $Expression1 = Sqrt((($points[0].x - $points[1].x) ^ 2) + (($points[0].y - $points[1].y) ^ 2))
Local $Expression2 = Sqrt((($points[1].x - $points[2].x) ^ 2) + (($points[1].y - $points[2].y) ^ 2))
Local $Expression3 = Sqrt((($points[2].x - $points[3].x) ^ 2) + (($points[2].y - $points[3].y) ^ 2))
Local $Expression4 = Sqrt((($points[0].x - $points[3].x) ^ 2) + (($points[0].y - $points[3].y) ^ 2))

If (Execute("$Expression1") < $mindist) And (Execute("$Expression2") < $mindist) And (Execute("$Expression3") < $mindist)   And (Execute("$Expression4") < $mindist) Then

    MsgBox(0, "", "Points are too close")
    ;Return
EndIf

The problem remains only when the frame profile is concave.

I see that you are well versed in mathematics 😀.

I will ask, without much hope, if you know way to identify that the profile, drawn in the example, is closed and concave at the same time?

Thank you.

Edited by Lion66
Link to comment
Share on other sites

Hi @Lion66

1 hour ago, Lion66 said:

The problem remains only when the frame profile is concave.

Sorry, that is beyond my knowledge, and I am not sure that you need to go in that direction.
It has to be confirmed, but I theorized that, if an homography where found, then a closed path will remain closed after applying the homography.

 

1 hour ago, Lion66 said:

I will ask, without much hope, if you know way to identify that the profile, drawn in the example, is closed and concave at the same time?

Are they cases where my provided code gives wrong results?
Can you give an example of coordinates representing a case?

Regarding your code, there things that bother me.

First, the equivalent without the loop is a check of 6 distances with an "Or" expression not an "And".
The 6 distances to check are:

  • $dst_0 <-> $dst_1
  • $dst_0 <-> $dst_2
  • $dst_0 <-> $dst_3
  • $dst_1 <-> $dst_2
  • $dst_1 <-> $dst_3
  • $dst_2 <-> $dst_3

You are only checking 4 distances which will most likely miss cases.

You are also doing a "And" not an "Or".
That means the 4 following distances have to be close: $dst_0 <-> $dst_1, $dst_1 <-> $dst_2, $dst_2 <-> $dst_3, $dst_0 <-> $dst_1
However, as long as 2 points are close, you will not have a rectangle.
You will have a triangle (only 2 points are close), a line (2 groups of 2 points are close or 3 points are close), or a point (all the 4 points are close)

Second, why are you using Execute("$Expression1") < $mindist instead of $Expression1 < $mindist ?
Execute could have been usefull in the loop by doing

Execute("Sqrt((($dst_" & $i & ".x - $dst_" & $j & ".x) ^ 2) + (($dst_" & $i & ".y - $dst_" & $j & ".y) ^ 2))")

 

Third, since you are splitting the loop in individual check, there is no need for an array. Use $dst_0 instead of $points[0]

 

Edited by smbape
Link to comment
Share on other sites

You're right. This is superfluous: Execute and $points[4].

Initially, I planned to leave cases of triangle and line. Therefore, I used AND and therefore it is enough to check 4 distances.

But now I have changed my mind, because such cases of detection are not quite accurate.

I'll see again, how your method works. But still there is a case with a strange form of frame.

They're always concave. I would like to filter the result by this.

Pic3: (similar you can see by file 160732-opencv-udf.au3, algorithm BRISK).

$dst_0 [ 97.4931182861328, 130.25163269043 ]
$dst_1 [ -130.481994628906, 344.5224609375 ]
$dst_2 [ 234.231307983398, 278.392486572266 ]
$dst_3 [ 124.302398681641, 250.157531738281 ]
Distance:
312.865210508152, 370.66017672785, 113.497038616658, 122.866440258512

But in the last file "matching_homography_Demo.au3" from GIThub, this does not happen. I'm completely confused 🙄

Pic3.jpg

Link to comment
Share on other sites

Maybe you noticed that there is one point outside the view box.

$dst_1 [ -130.481994628906, 344.5224609375 ]

 

1 hour ago, Lion66 said:

But in the last file "matching_homography_Demo.au3" from GIThub, this does not happen. I'm completely confused

This is what I was refering to in a previous post.
All the matches are used to find the homagraphy,  which will most likely lead to an innacurrate homography.

In 160732-opencv-udf.au3, if you ReDim dmatches after populating and sorting it, you will have a more accurate homography

ReDim $dmatches[20]

matching_homography_Demo.au3 does not have this problem for 2 reasons:

  • _cveDescriptorMatcherKnnMatch1Mat is used instead of _cveDescriptorMatcherMatch1Mat thus, removing the need of sorting matches
  • Only good matches are kept. This is done with the condition "$tDMatch0.distance < $ratio_thresh * $tDMatch1.distance"

Regarding the detection of concave polygones, I theorize 2 ways to do it:

  • Calculate the angle at each corner and if there is an angle greater than 180°, then it is convex.
    This will definitly work, however it involves reminding how to calculate an angle between 2 vectors.
  • Use cv::convexHull to find the convex hull of the polygone.
    If the convex hull is also the polygone, then the polygone is convex, otherwise it is concave.
    It is in theory easier to implement.
Edited by smbape
Link to comment
Share on other sites

  • smbape changed the title to OpenCV v4 UDF

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...