Jump to content

Recommended Posts

Hi again @Lion66

The theory was correct.
Using cv::convexHull, it is possible to determine if a polygone is concave.

Local $hull = _VectorOfPointCreate()
Local $o_arr_hull = _cveOutputArrayFromVectorOfPoint($hull)
_cveConvexHull($i_arr_dst_int32, $o_arr_hull)
For $i = 0 To _VectorOfPointGetSize($dst_int32) - 1
    _VectorOfPointGetItemPtr($dst_int32, $i, $tPointFPtr)
    Local $dst_int32_i = DllStructCreate($tagCvPoint, $tPointFPtr.value)

    _VectorOfPointGetItemPtr($hull, $i, $tPointFPtr)
    Local $hull_i = DllStructCreate($tagCvPoint, $tPointFPtr.value)

    If $hull_i.x <> $dst_int32_i.x Or $hull_i.y <> $dst_int32_i.y Then
        ConsoleWrite("Concave contour" & @CRLF)
        ExitLoop
    EndIf
Next
_cveOutputArrayRelease($o_arr_hull)
_VectorOfPointRelease($hull)


See the attached code.

160732-opencv-udf.au3

Edited by smbape
Link to post
Share on other sites
  • Replies 150
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

I wanted to use OpenCV v4+ in AutoIt. I found Opencv UDF on the forum, but there was no support for OpenCV v4+ This UDF provides support for OpenCV v4+ Update There is a new implem

Hi @malcev The environment variable is intended to override the embeded manifest. It is not needed for scripting languages, but for compiled languages. That being said, I kept it just in case

Update opencv to 4.7.0 COM Registration is no longer required

Posted Images

I have added your UDF to the wiki :)

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to post
Share on other sites

Hello @ smbape

Congratulations! 👏 👏 👏 Now you are in a wiki (as I said!). It's worthy of the job you're doing!

I found this example. He draws the frame so beautifully.

I see that here the method of finding the contour is applied here and after that boundingRect.
I wanted to try, but I can't handle it myself.
Perhaps you will have the opportunity to show how to do this.

Attached is an example on the python.

Thank you so much.

 

Result.png

card.py

 

 

 

 

 

Edited by Lion66
Link to post
Share on other sites

Hi @Lion66

Sorry for the late response, I was on another project.
I read the python code.

The detection is done in the following steps:

  • Detect objects of a scene using cv2.findContours.
    To improve the countour detection, the image is first blured with cv2.GaussianBlur, thus removing noises,
    then thresholded with cv2.threshold to enhance seperation of areas.
  • For each countour, pass it through a bank of images (images/cards/sample/)
  • The type of image that matches the most will be choosed as the type of the countour.

This technique will work if the background and the object are very contrasted.
If not, threshold will either blacked everything or whited everything, thus making countour detection impossible.

On 8/18/2021 at 9:07 PM, Lion66 said:

I wanted to try, but I can't handle it myself.

What do you  want to try?

  • Countour detection?
  • GaussianBlur?
  • Threshold?
  • knnMatch ?
  • A combination of those?

What have you tried that was not working?

Do you have a python/autoit example of your try?

Link to post
Share on other sites

A very important part when using your UDF is to correctly declare the type of variables and structures before applying the function.
I have little understanding of the structure of the DLL, and I lack instructions for translating the syntax from docs.opencv.org to this UDF.
My plan is:

_cveGaussianBlurMat($gray_obj, $Gaus_obj, $ksize, 0, 0, $CV_BORDER_DEFAULT)
_cveThresholdMat($gray_obj, $gray_obj, $thresh, $maxval, $type)
_cveFindContoursMat($matImage, $matContours, $matHierarchy, $mode, $method, $offset = _cvPoint())
_cveBoundingRectangleMat($matPoints, $boundingRect)

And already the first function fails.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
Opt("MustDeclareVars", 1)
#include "emgucv-autoit-bindings\cve_extra.au3"
_OpenCV_DLLOpen("libemgucv-windesktop-4.5.3.4721\libs\x64\cvextern.dll")

Local $img_obj = _cveImreadAndCheck("pic1\box.png", $CV_IMREAD_COLOR)   ; query image (small object)
Local $gray_obj = _cveMatCreate()
_cveCvtColorMat($img_obj, $gray_obj, $CV_COLOR_BGR2GRAY)

;_cveGaussianBlurMat($matSrc, $matDst, $ksize, $sigmaX, $sigmaY = 0, $borderType = $CV_BORDER_DEFAULT)
Local $Gaus_obj = _cveMatCreate()
Local $ksize = "(3,3)"
_cveGaussianBlurMat($gray_obj, $Gaus_obj, $ksize, 3, 0, $CV_BORDER_DEFAULT)

;_cveThresholdMat($gray_obj, $gray_obj, $thresh, $maxval, $type)
;_cveFindContoursMat($matImage, $matContours, $matHierarchy, $mode, $method, $offset = _cvPoint())
;_cveBoundingRectangleMat($matPoints, $boundingRect)

_cveImshowMat("", $Gaus_obj)
_cveWaitKey()
_cveDestroyAllWindows()
_Opencv_DLLClose()

And another question:
Can I process the error?
In the compiled script, if possible to show the message to the user instead of emergency completion?

Edited by Lion66
Link to post
Share on other sites

Hi @Lion66

3 hours ago, Lion66 said:

A very important part when using your UDF is to correctly declare the type of variables and structures before applying the function.

You are right. I am aware of the high level of knwoledge needed in order to use the UDF correctly.
I planned on making a documentation for rules of translation from c++ to the UDF, however is it hard teach it correctly.

This is what I do to convert to the UDF:

  • Look at the udf definition without "Mat" at the end. For cv2.gaussianBlur it will be _cveGaussianBlur
  • The first line of the function is the c++ signature of the function, which is
    CVAPI(void) cveGaussianBlur(cv::_InputArray* src, cv::_OutputArray* dst, CvSize* ksize, double sigmaX, double sigmaY, int borderType);

     

  • The "Mat" version is a function where every cv::_InputArray, _OutputArray, _InputOutArray are derived from a matrix
  • CvSize* is a _cvSize() element. In this case Local $ksize = _cvSize(3,3)
  • When in the c++ exemple one of the cv::_*Array is a cv::Scalar, the "Mat" version cannot be used.
    You have to first create a cv::Scalar with _cveScalarCreate, then use _cve*ArrayFromScalar to convert it to the expected type before using it in the function
  • There are cases where the function has extra parameters required. Those parameters take their default value if not specified.
    If the UDF is not aware of those default values, you have to put them manually.
    In that case, you have to look at the documentation at opencv.
    For exemple _cveKAZEDetectorCreate(False, False, 0.001, 4, 4, $CV_KAZE_DIFF_PM_G2, $tFeature2DPtr, $tSharedPtr) comes from cv::KAZE::create

With the previous instructions, can you try again?

Not that I do not want to help, but I want to see if my instructions are understandable and correct.

Edited by smbape
Link to post
Share on other sites
3 hours ago, Lion66 said:

Can I process the error?

When calling a dll, AutoIt doesn't provide a way for it. It just crashes.
The best things you can do are:

  • Tools > "Trace: Add Trace Lines"
  • Set $_cve_debug = 1
    This will print all successfully executed functions.
    With the last executed function, you can manage to find where it crashes.
    Crashes are usually due to incorrect types or Null pointers.
Edited by smbape
Link to post
Share on other sites

Thank you for your answers.

The topic is very extensive. Sorry, if I to be importunate.

I advance with "$ksize = _cvSize(3,3)", but I did not understand why it was told about:

2 hours ago, smbape said:

You have to first create a cv::Scalar with _cveScalarCreate, then use _cve*ArrayFromScalar to convert it to the expected type before usin it in the function

Now I'm stuck on the "$matHierarchy" (third parameter in _cveFindContoursMat)

Local $img_scene = _cveImreadAndCheck("pic1\cards.JPG", $CV_IMREAD_COLOR)   ; query image - scene
Local $gray_scene = _cveMatCreate()
_cveCvtColorMat($img_scene, $gray_scene, $CV_COLOR_BGR2GRAY)

Local $Gaus_scene = _cveMatCreate()
_cveGaussianBlurMat($gray_scene, $Gaus_scene, _cvSize(3,3), 0, 0)
_cveThresholdMat($Gaus_scene, $Gaus_scene, 200, 255, $CV_THRESH_BINARY)


Local $H = _cveMatCreate()
Local $o_arr_H = _cveOutputArrayFromMat($H)
_cveFindContoursMat($Gaus_scene, $Gaus_scene, $o_arr_H, $CV_RETR_EXTERNAL, $CV_CHAIN_APPROX_SIMPLE)

 

Edited by Lion66
Link to post
Share on other sites
1 hour ago, Lion66 said:

The topic is very extensive. Sorry, if I to be importunate.

I don't feel that way. It is a pleasure to have someone interested in my project.
That is why I am answering every question you have.

 

1 hour ago, Lion66 said:

but I did not understand why it was told about

It doesn't apply to this case.
You missed the first line, "When in the c++ exemple one of the cv::_*Array is a cv::Scalar, the "Mat" version cannot be used.".
For example, in Threshold_inRange.cpp#L89 , the method inRange takes a Scalar as a parameter. I was talking about those kind of cases.
The are also case where there are vectors like in SURF_FLANN_matching_homography_Demo.cpp#L73 . Again, _cve*ArrayFrom* should be used like in matching_homography_Demo.au3#L301

 

1 hour ago, Lion66 said:

Now I'm stuck on the "$matHierarchy":

It is more than that.
I you look at the documentation of cv::findContours, it is written

Quote
contours Detected contours. Each contour is stored as a vector of points (e.g. std::vector<std::vector<cv::Point> >).
hierarchy Optional output vector (e.g. std::vector<cv::Vec4i>), containing information about the image topology. It has as many elements as the

For contours, it is not a matrix, but a vector of points (std::vector<std::vector<cv::Point> >), that means, the "Mat" version cannot be used.
For hierarchy, it is an output vector (std::vector<cv::Vec4i>). Vec4i is a matrix. Therefore, hierarchy is a cv::_OutputArray from a vector of matrix.
It will then be

$contoursVect = _VectorOfVectorOfPointCreate()
$contours = _cveOutputArrayFromVectorOfVectorOfPoint($contourVect)

$hierarchyVect = _VectorOfMatCreate()
$hierarchy = _cveOutputArrayFromVectorOfMat($hierarchyVect)

; rest of the code here

; Then loop through the VectorOfVectorOfPoint like in
; https://github.com/smbape/node-emgucv-autoit-generator/blob/master/samples/tutorial_code/features2D/feature_homography/matching_homography_Demo.au3#L232

It is true that it needs c++ understanding.
If my directions are not enough to help you have a working script, come back to me

Edited by smbape
Link to post
Share on other sites

I do not have knowledge in C++.

For academic purposes, your UDF is very good.

For practical purposes, I would transfer the creation of all auxiliary variables (necessary for the function itself) into UDF.

And the user would be given simple functions with user arguments.

But this is another story!

I have to admit I don't understand your explanation at this level. I'm sorry. :(

I myself would like to understand how to get a ready-made code.

But this is not my level and we are unlikely to advance like that.

It doesn't work again. :mad:

Local $contoursVect = _VectorOfVectorOfPointCreate()
Local $contours = _cveOutputArrayFromVectorOfVectorOfPoint($contoursVect)

Local $hierarchyVect = _VectorOfMatCreate()
Local $hierarchy = _cveOutputArrayFromVectorOfMat($hierarchyVect)

_cveFindContoursMat($Gaus_scene, $contours, $hierarchy, $CV_RETR_EXTERNAL, $CV_CHAIN_APPROX_SIMPLE)

 

Link to post
Share on other sites
1 hour ago, Lion66 said:

For practical purposes, I would transfer the creation of all auxiliary variables (necessary for the function itself) into UDF.

And the user would be given simple functions with user arguments.

I agree with you.
However, OpenCV has more that 3000 functions.
It is impossible, without spending months, to translate everything.
And supposed I am done. Tracking updates from the point I started translation will be tedious.
Like every big c/c++ library out there, people just give up on using them with AutoIt, or write dlls, which requires c/c++ knowledge.

 

1 hour ago, Lion66 said:

But this is not my level and we are unlikely to advance like that.

It doesn't work again.

I understand your frustration. I will stop here.
Can you post your current file and image.
I will fix it for you.

Edited by smbape
Link to post
Share on other sites

Like in cards.py, boundingRect has to be called on each countour.
The documentation of boundingRect only has one parameter and returns a cv::Rect.
Usually the UDF has no return value and the last parameter will receive the returned value.

Changes have been made in SearchContureDraw-v2.au3

Following cards.py, to crop an image based on a cv::Rect do:

$boundingRect.x -= 15
$boundingRect.y -= 15
$boundingRect.width += 15
$boundingRect.height += 15
$cropped = _cveMatCreateFromRect($img_scene, $boundingRect)

SearchContureDraw-v2.au3

Edited by smbape
Link to post
Share on other sites

Hi @smbape

First thanks for your help and for your patience!
In the script, I want to recognize only one image, and changing the size of the frame is not important.
I don't know how to properly link the result after Filter matches and the frame.
I came up with an option, but for some reason the loop on line 71 makes only one turn.

 

SearchContureDraw-v5.au3

Link to post
Share on other sites

You used the same variable in two different loops

; #71
For $i = 0 To _VectorOfVectorOfPointGetSize($contoursVect) - 1

; #134
For $i = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1

; change it to
For $j = 0 To _VectorOfVectorOfDMatchGetSize($knn_matches) - 1
        _VectorOfVectorOfDMatchGetItemPtr($knn_matches, $j, $tVectorDMatchPtr)

 

Edited by smbape
Link to post
Share on other sites
  • smbape changed the title to OpenCV v4 UDF

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By JoGa
      Greetings,
      here is a Autoit3 v3.3.14.5, 64 Bit installation on windows 10 with current updates.
      In a simple script I'm Dllcall'ing a 64 bit dll.
      # T1.au3 Local $name = "T1DLL.dll" if (FileExists($name)) Then ConsoleWrite("DLL '" &$name &"' exists" &@CRLF) EndIf Local $DLL = DllOpen($name) if (@ERROR OR $DLL = -1) then ConsoleWrite("DllOpen ERROR=" &@ERROR &" DLL=" &$DLL &@CRLF) Else ConsoleWrite("DllOpen Success" &@CRLF) endif DllCall($DLL, "none:cdecl", "SomeFunction", "str", "DLL Call from T1.au3") if (@ERROR) then ConsoleWrite("DllCall ERROR=" &@ERROR &" DLL=" &$DLL &@CRLF) Else ConsoleWrite("DllCall Success" &@CRLF) endif The C code contains one 64 bit OpenCV call:    cv::destroyAllWindows();.

      Excuting T1.au3 with SciTE gives:

      If the c code is compiled *without* the OpenCV call T1.au3 runs successful:
       
      I checked T1DLL.dll, it's definitely a 64 bit dll.
      What could cause the problem?
      Any hint would be very much appreciated.
      Thanks
      Wolf
       
       
       
       
    • By BBs19
      I needed a function to automate programs at work that can't be fully automated via Autoits built in functions.
      For example a virtual machine running on your physical machine, meaning you would need to run an extra script within the virtual machine (if it is even running Windows) in order to automate everything.
      I came across OpenCV which allows matching/finding a picture in another picture. This would also allow searching for a button/text on the screen in order to press the exact position. Fortunately @mylise already translated all the required OpenCV functions to Autoit, I just had to remove all unnecessary functions to make the script as small as possible.
      The problem:
      Using this method, you will never be able to fully automate everything dynamically, as it will only work on the machine with same resolution/dpi settings, same theme etc.. This is only a last resort for programs that can't be automated using the built in Autoit functions.
      Features:
      Find a given picture on the entire screen (all monitors) or a certain area on the screen and execute mouse clicks on this position. Adjust the threshold so that the picture doesn't have to match 100%. Debugging options like logging and marking the screen where the picture was found etc. Includes a Snapshot-Tool that creates snapshots of a certain area(buttons, text etc.) on the screen and generates the code that is required for the matching. It can also be used to get the coordinates to a marked area on the screen in order to check only on a certain area for the match picture. Example:
      Note: The example will probably not work on your computer, depending on the display resolution and dpi settings, as the picture has to match the exact same size on the screen. Please use the included Snapshot-Tool to generate new match pictures and code very easily.
      #AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work #include "OpenCV-Match_UDF.au3" _OpenCV_Startup();loads opencv DLLs _OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output. ;Please note that these examples might not work as the match pictures have to be found with the exact same size on your screen. ;Example 1 ShellExecute("http://www.tv.com/");Open Website tv.com $Match1 = _MatchPicture(@ScriptDir&"\Match\1.png", 0.70,False,10,500);Try to find the match picture on the screen. Number of tries: 10, Sleep between each try: 500ms. If Not @error Then _MarkMatch($Match1) ;Debugging: Draws a rect on the screen/coordinates of the match to show the user where the match was found Sleep(100) _ClickMouse($Match1, "left",1) ;Calculates the center of the match and clicks the left mouse once on click position EndIf Sleep(1000) ;Example 2, matching on a specific area of the screen ShellExecute("notepad.exe");open nodepad WinWait("[CLASS:Notepad]","",5) WinMove("[CLASS:Notepad]","",0,0,500,500) Local $sCoords[4] = [0, 0, 500,500] $Match2 = _MatchPicture(@ScriptDir&"\Match\2.png", 0.80,$sCoords,3,500) If Not @error Then _MarkMatch($Match2) Sleep(100) _ClickMouse($Match2, "left", 1) EndIf _OpenCV_Shutdown();Closes DLLs So basically, all you need to do is provide a path to the match picture and the function will return you the coordinates (x1,y1,x2,y2) of where the picture has been found on the screen. With these, you can either calculate an exact position for the mouse click or use the "_ClickMouse" function which will execute a mouse click on the center of the coordinates where the picture was found. 
      Credits:
      @mylise for the OpenCV UDF
       
      Download:
      Includes the required .DLL files of OpenCV. You can also manually download them on the website of OpenCV (Version 3.x doesn't include these anymore, you need to download 2.x).
      OpenCV_Match.zip
×
×
  • Create New...