Jump to content

Simple automation using OpenCVs match feature


BBs19
 Share

Recommended Posts

  • 2 years later...
  • 9 months later...

@mylise

@BBs19

I have installed OpenCV and am trying to automate a Java x64 app on windows 10 64-bit. My script is not working.

I am simply trying to get the cursor to move to screen coordinates and click. for now.

Quote

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "test.au3"


_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output.


HotKeySet( "{SPACE}", "MyFunction")
HotKeySet( "{ESC}", "CloseScript")

Local $x = 342 ; x is first value of mouse position
Local $y = 74 ; y is second value of mouse position

While 1
   sleep(10)
Wend

Func MyFunction()
MouseMove($x, $y, 2)                ; moves the mouse pointer do mouse position with speed 2
Sleep(500)                        ; wait 500 ms half of second ( delay )
MouseClick("left", $x, $y, 1, 2)    ; now click left mouse key on mouse position, 1 click with speed 2
EndFunc

Func CloseScript()
   Exit
EndFunc

Error:include depth exceeded

Link to comment
Share on other sites

3 hours ago, FredMP said:

@mylise

@BBs19

I have installed OpenCV and am trying to automate a Java x64 app on windows 10 64-bit. My script is not working.

I am simply trying to get the cursor to move to screen coordinates and click. for now.

Error:include depth exceeded

did you try:

?

Signature beginning:
Please remember: "AutoIt"..... *  Wondering who uses AutoIt and what it can be used for ? * Forum Rules *
ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Codefor other useful stuff click the following button:

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 *

 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskSchedulerIE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related:How to get reference to PDF object embeded in IE * IE on Windows 11

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

I also encourage you to check awesome @trancexx code:  * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuffOnHungApp handlerAvoid "AutoIt Error" message box in unknown errors  * HTML editor

winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2023-04-24

Link to comment
Share on other sites

On 9/3/2020 at 4:16 AM, FredMP said:

I am simply trying to get the cursor to move to screen coordinates and click. for now.

Error:include depth exceeded

you are not calling any of the OpenCV-Match_UDF functions in your script
what does the test.au3 file you included at the beginning of your script contain?
... I think if you remove it your snippet will work ....

 

image.jpeg.9f1a974c98e9f77d824b358729b089b0.jpeg Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Link to comment
Share on other sites

  • 1 month later...

Hello Guys 

I want to find pictures in (  WinGetHandle )

Can Someone please fix it for me ?

You can view this code from  " OpenCV-Match_UDF.au3 "

 

Code below i tried to fix it , but i can't find the image in " WinGetHandle "

  "Example.au3" 

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "OpenCV-Match_UDF.au3"
_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output.
Global $Test = WinGetHandle("Program Manager")
$Match1 = _MatchPicture($Test,@ScriptDir&"\Match\4.png", 0.70,False,10,500);Try to find the match picture on the screen. Number of tries: 10, Sleep between each try: 500ms.
If Not @error Then
    _MarkMatch($Match1) ;Debugging: Draws a rect on the screen/coordinates of the match to show the user where the match was found
    Sleep(100)
    MouseMove($Match1[0],$Match1[1])
EndIf

 

  " OpenCV-Match_UDF.au3 "

GDIPlus_Startup()
Func _MatchPicture($Handle,$Match_Pic, $Threshold = 2, $CustomCords = False, $LoopCount = 2, $LoopWait = 2000)
     Local $rect_window = WinGetPos($Handle)
    Local $hDC = _WinAPI_GetWindowDC($Handle)
    Local $hDestDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBitmap = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hDestSv = _WinAPI_SelectObject($hDestDC, $hBitmap)
    Local $hSrcDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBmp = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hSrcSv = _WinAPI_SelectObject($hSrcDC, $hBmp)

    _WinAPI_PrintWindow($Handle, $hSrcDC, True)

    If Not FileExists($Match_Pic) Then
        _Internal_ErrorLogger("Match Image not found: " & $Match_Pic)
        Return SetError(1)
    EndIf
    If $LoopCount <= 0 Then
        ConsoleWrite("Error: $LoopCount Variable set to 0. At least 1 is required.")
        Return SetError(1)
    EndIf
    Local $hBitmap

    ;Performance Counters
    Local $Perf = TimerInit()
    ;Load Match Image
    Local $hMatch_Pic = _cvLoadImage($Match_Pic)
    Local $hMatch_Size = _cvGetSize($hMatch_Pic)
    Local $width2 = DllStructGetData($hMatch_Size, "width")
    Local $height2 = DllStructGetData($hMatch_Size, "height")
    Local $ScreenSize = _ScreenSize()

    For $iTries = 1 To $LoopCount Step +1
        ;Capature Screen / Load as Main image
        If Not IsArray($CustomCords) Then

            $hBitmap = _ScreenCapture_Capture("", 0, 0, $ScreenSize[0], $ScreenSize[1], False)
        Else
            $hBitmap = _ScreenCapture_Capture("", $CustomCords[0], $CustomCords[1], $CustomCords[2], $CustomCords[3], False)
        EndIf
        Local $Bitmap = _GDIPlus_BitmapCreateFromHBITMAP($hBitmap)

        Local $hMain_Pic = _Opencv_BMP2IPL($Bitmap)
        Local $hMain_Size = _cvGetSize($hMain_Pic)
        Local $width = DllStructGetData($hMain_Size, "width")
        Local $height = DllStructGetData($hMain_Size, "height")
        Local $rw = $width - $width2 + 1
        Local $rh = $height - $height2 + 1
        Local $tmaxloc = DllStructCreate( "int x;" & "int y;")
        Local $tminloc = DllStructCreate( "int x;" & "int y;")
        Local $tmaxval = DllStructCreate("double max;")
        Local $tminval = DllStructCreate("double min;")
        Local $pmaxloc = DllStructGetPtr($tmaxloc)
        Local $pminloc = DllStructGetPtr($tminloc)
        Local $pmaxval = DllStructGetPtr($tmaxval)
        Local $pminval = DllStructGetPtr($tminval)

        ;Search for Match
        Local $presult = _cvCreateMat($rh, $rw, 5)
        _cvMatchTemplate($hMain_Pic, $hMatch_Pic, $presult, 5)
        _cvThreshold($presult, $presult, $Threshold, 1, 0)
        _cvMinMaxLoc($presult, $pminval, $pmaxval, $pminloc, $pmaxloc, Null)

        ;Set coordinates
        Local $Coordinates[4]
        If IsArray($CustomCords) Then
            $Coordinates[0] = DllStructGetData($tmaxloc, "x") +2+ $CustomCords[0] ; x
            $Coordinates[1] = DllStructGetData($tmaxloc, "y") +2+ $CustomCords[1] ; y
            $Coordinates[2] = DllStructGetData($tmaxloc, "x") +2+ $width2 + $CustomCords[0] ;x2
            $Coordinates[3] = DllStructGetData($tmaxloc, "y") +2+ $height2 + $CustomCords[1] ;y2
        Else
            $Coordinates[0] = DllStructGetData($tmaxloc, "x") ; x
            $Coordinates[1] = DllStructGetData($tmaxloc, "y") ; y
            $Coordinates[2] = DllStructGetData($tmaxloc, "x") + $width2 ;x2
            $Coordinates[3] = DllStructGetData($tmaxloc, "y") + $height2 ;y2
        EndIf

        ;Release Resources
        _cvReleaseMat($presult)
        _cvReleaseImage($hMain_Pic)
        _GDIPlus_BitmapDispose($Bitmap)
        _WinAPI_DeleteObject($hBitmap)
       _WinAPI_SelectObject($hDestDC, $hDestSv)
       _WinAPI_SelectObject($hSrcDC, $hSrcSv)
       _WinAPI_ReleaseDC($Handle, $hDC)
       _WinAPI_DeleteDC($hDestDC)
       _WinAPI_DeleteDC($hSrcDC)
       _WinAPI_DeleteObject($hBmp)
        ;Check if found
        If Not (DllStructGetData($tmaxloc, "x") = 0 And DllStructGetData($tmaxloc, "y") = 0 And $width2 = DllStructGetData($tmaxloc, "x") + $width2 And $height2 = DllStructGetData($tmaxloc, "y") + $height2) Then
            _cvReleaseImage($hMatch_Pic)
            _Internal_MatchLogger("Match found at: " & $Coordinates[0] & "|" & $Coordinates[1] & "|" & $Coordinates[2] & "|" & $Coordinates[3] & " // Loop counter: " & $iTries & " // Threshold: " & $Threshold & " // Total check time: " & Round(TimerDiff($Perf), 0) & " ms")
            Return $Coordinates
        EndIf
        Sleep($LoopWait)
    Next
    _cvReleaseImage($hMatch_Pic)
    _Internal_ErrorLogger("No match found. Loop counter: " & $LoopCount & ". // Total check time: " & Round(TimerDiff($Perf), 0) & " ms // Wait-Time per loop: " & $LoopWait & " ms. // Threshold: " & $Threshold & " // Match Image: " & $Match_Pic)
    Return SetError(2)

EndFunc   ;==>_MatchPicture

 

Link to comment
Share on other sites

  • Jos locked and unlocked this topic
  • 4 weeks later...

@Chimp

Finally got around to fixing this and your suggestion worked, and my 1st script finally works, yesss!

Now, I have two questions:

1.  I need this script to be on standby the whole time the TTS app is active. So, if I press the hotkey (SPACE) the script will run moving the mouse and clicking at the coordinates and then end. But the next time I press SPACE it should do it again. My question would be is it ok to leave the script running on standby like this for hours on end or is there a more efficient way for the script to run? 

2. I need the (SPACE) hotkey to only work in TTS app. So if i go over to a browser window and type SPACE I do not want the script to activate there, only in TTS app. How do I write this into the script?

Here is the current script, I welcome any suggestions!

#AutoIt3Wrapper_UseX64=n ; In order for the x86 DLLs to work
#include "OpenCV-Match_UDF.au3"
_OpenCV_Startup();loads opencv DLLs
_OpenCV_EnableLogging(True,True,True) ;Logs matches, errors in a log file and autoit console output.

HotKeySet( "{SPACE}", "MyFunction")
HotKeySet( "{ESC}", "CloseScript")

Local $x = 342 ; x is first value of mouse position
Local $y = 74 ; y is second value of mouse position

While 1
   sleep(10)
Wend

Func MyFunction()
   #RequireAdmin
if winexists("Main@TTS") Then
    winactivate("Main@TTS")
    EndIf
MouseMove($x, $y, 2)                ; moves the mouse pointer do mouse position with speed 2
Sleep(500)                        ; wait 500 ms half of second ( delay )
MouseClick("left", $x, $y, 2, 2)    ; now click left mouse key on mouse position, 2 click with speed 2
EndFunc

Func CloseScript()
   Exit
EndFunc

 

Link to comment
Share on other sites

@FredMP

I'm not very clear about your goal, however,
using the hotkey in that way will always activate the function every time you press it, even if the focus is on another window (the browser).
You can limit hotkey activation only when the active window is the "Main@TTS" window using something like:

Func MyHotKeyFunc()
    Local $HotKeyPressed = @HotKeyPressed
    HotKeySet($HotKeyPressed) ; Disable hotkey to prevent looping

    If WinActive("Main@TTS") Then ; hotkey works only if this window is active
        ; do whatever you need
    Else
        Send($HotKeyPressed) ; send key to destination
    EndIf

    HotKeySet($HotKeyPressed, "MyHotKeyFunc") ; enable hotkey again
EndFunc   ;==>MyHotKeyFunc

but in your hot function i see you force focus on "Main@TTS". therefore I assume you want the hotkey to work always, even when "Main@TTS" has no focus (?).
Or, if you want it not to work only when the browser (or another window) is active then you could use some condition like this:

Func MyHotKeyFunc()
    Local $HotKeyPressed = @HotKeyPressed
    HotKeySet($HotKeyPressed) ; Disable hotkey to prevent looping

    If Not WinActive('Browser_hwd') Then ; <-- note the Not operator to exclude the browser
        ; do whatever you need
    Else
        Send($HotKeyPressed) ; send key to destination (the browser)
    EndIf
    HotKeySet($HotKeyPressed, "MyHotKeyFunc")     ; enable hotkey again
EndFunc   ;==>MyHotKeyFunc

 

 

image.jpeg.9f1a974c98e9f77d824b358729b089b0.jpeg Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Link to comment
Share on other sites

  • 2 months later...

 

@Chimp

I want background images or overlays to be detected, how should I fix them? Can someone help me please?

 

 

Func Image_Search( $Match_Pic, $Left, $Top, $Right, $Bottom, $Threshold = 0.75)
    Local $hBitmap
    ;Performance Counters
    Local $Perf = TimerInit()
    ;Load Match Image
    Local $hMatch_Pic = _cvLoadImage($Match_Pic)
    Local $hMatch_Size = _cvGetSize($hMatch_Pic)
    Local $width2 = DllStructGetData($hMatch_Size, "width")
    Local $height2 = DllStructGetData($hMatch_Size, "height")
    Local $ScreenSize = _ScreenSize()

    $hBitmap = _ScreenCapture_Capture("", $Left, $Top,$Right, $Bottom, False)
    Local $Bitmap = _GDIPlus_BitmapCreateFromHBITMAP($hBitmap)

    Local $hMain_Pic = _Opencv_BMP2IPL($Bitmap)
    Local $hMain_Size = _cvGetSize($hMain_Pic)
    Local $width = DllStructGetData($hMain_Size, "width")
    Local $height = DllStructGetData($hMain_Size, "height")
    Local $rw = $width - $width2 + 1
    Local $rh = $height - $height2 + 1
    Local $tmaxloc = DllStructCreate("int x;" & "int y;")
    Local $tminloc = DllStructCreate("int x;" & "int y;")
    Local $tmaxval = DllStructCreate("double max;")
    Local $tminval = DllStructCreate("double min;")
    Local $pmaxloc = DllStructGetPtr($tmaxloc)
    Local $pminloc = DllStructGetPtr($tminloc)
    Local $pmaxval = DllStructGetPtr($tmaxval)
    Local $pminval = DllStructGetPtr($tminval)

    ;Search for Match
    Local $presult = _cvCreateMat($rh, $rw, 5)
    _cvMatchTemplate($hMain_Pic, $hMatch_Pic, $presult, 5)
    _cvThreshold($presult, $presult, $Threshold, 1, 0)
    _cvMinMaxLoc($presult, $pminval, $pmaxval, $pminloc, $pmaxloc, Null)

    ;Set coordinates
    Local $Coordinates[6]
    $Coordinates[0] = DllStructGetData($tmaxloc, "x") + $Left         ; x
    $Coordinates[1] = DllStructGetData($tmaxloc, "y") + $Top         ; y
    $Coordinates[2] = DllStructGetData($tmaxloc, "x") + $width2 + $Left         ;x2
    $Coordinates[3] = DllStructGetData($tmaxloc, "y") + $height2 + $Top         ;y2
    $Coordinates[4] = ($Coordinates[0] + $Coordinates[2]) / 2
    $Coordinates[5] = ($Coordinates[1] + $Coordinates[3]) / 2
    ;Release Resources
    _cvReleaseMat($presult)
    _cvReleaseImage($hMain_Pic)
    _GDIPlus_BitmapDispose($Bitmap)
    _WinAPI_DeleteObject($hBitmap)

    ;Check if found
    If Not (DllStructGetData($tmaxloc, "x") = 0 And DllStructGetData($tmaxloc, "y") = 0 And $width2 = DllStructGetData($tmaxloc, "x") + $width2 And $height2 = DllStructGetData($tmaxloc, "y") + $height2) Then
        _cvReleaseImage($hMatch_Pic)
        Return $Coordinates
    EndIf
    for $i = 0 to 5
        $Coordinates[$i] = 0
    Next
    Return $Coordinates

EndFunc   ;==>Image_Search

 

Link to comment
Share on other sites

  • 2 years later...

Hi all,

Would it be possible that this function needs a GPU to works fine ?

It seems that on old PC or on VM without GPU, we encounter 'Failed to load DLL' error.

I'm trying to figure out by checking the open CV website but maybe one of you already knows the answer 🙂

Thx in advance 😉

Link to comment
Share on other sites

  • 2 weeks later...

I'm a bit more far in my investigation.  On 2 of my 3 PC, it works fine.

On the 3rd one, the OPENDLL always return me "-1" as result.  I was wondering if it was not link to any Visual studio install (got that from other forum) but, asexcpected, this does not solve the problem.

I checked the file location, everything is okay (even if I change the dll and put it in the same locaton as the script) but DllOpen("opencv_core2413.dll") always return me -1.

Any idea ?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...