Jump to content
kylejustknows

A Simple Native "ImageSearch" Function (find pic on screen) made by combining "image to array" and "search 2D-array in 2D-array"

Recommended Posts

The code is solid and simple, it can almost explain itself.

This is the native autoit way to do the "imagesearch", no 3rd party .dll needed.

 

It gets "your.bmp", and "screenshot.bmp" ----> Convert the  .bmp files into 2D-Arrays (Malkey's function)  ----> Compare the 2D-arrays, return the matched position.

 

Tested on: Windows 7; Windows server 2008R2; Windows 10 1809.

 

Pros:

It is native. No extra .dll needed

It is super robust. (I used to have lots of funny results using other imagesearch libs).

It gets screenshot the same you get your screenshot crop, so it always gets a solid result, and 100% accurate.

The code is very simple and friendly, all level users can understand and use it.

 

Cons:

It is slow to convert your.big.screen.bmp into a 2D-array, and may consume 200+MB of memory and may take 5 - 20 seconds to return the result. (the actual search in an array is fast, but the conversion from .bmp to array is slow. The speed depends on your CPU speed and your screen size).

Correct: now optimized,  it's ~5 seconds and ~ 70MB ram usage.

It is a pixel-by-pixel color-code strict comparison in the "array-in-array" search, so you have to use the 24-bit BMP file, no "Tolerance" allowed.

 

2019-Jun-11: script update:

Same day updated: Update example;  Optimize the algorithm for performance, now most computers can get the result in ~5 seconds, using ~70MB temporary memory, for the 1920x1080 resolution screen.

 

2019-Jun-12 script update:

It now uses "PrintScreen" hotkey to save the screenshot.bmp (restores the user's old clipboard content after it is done) ~This is the only way to make sure the screenshot matches exactly what the user is seeing, after doing dozens of harsh tests.

The reason: The UDF "ScreenCapture" and "ImageSearch.dll"  are not reliable for an unknown reason. Some window/dialogue special drawings are "invisible" in their screenshots.

But the "PrintScreen" key -> Clipboard -> screenshot.bmp, this method always catches exact things showing on the screen.

 

#include <GDIPlus.au3>
#include <ClipBoard.au3>

;Sinple Example.================== the 1.bmp is what you want to find on your screen
$result = _ScreenSearchBmp("1.bmp")

if $result[0] = 0 Then
    MsgBox(0,"","not found")
Else
    MouseMove($result[0],$result[1],20) ;move mouse to the result
EndIf
;Example End.================== You can "include" this file after you remove this "Example" part here.



;===============================================================================
;
; Description:      Main Function.  Find the position of an image on the desktop
; Parameter(s):
;                   $center = 1 - Set where the returned x,y location of the image is.
;                   default 1 means center, 0 means top-left
;
; Return Value(s):  On Success - Returns the array of matched position [x,y] on your screen.
;                   On Failure - Returns array [0,0] (BTW, there is no position 0,0 on a screen, so it means error)
;
; Note: Warning:  The BMP file must be a 24-bit BMP (windows default)
;
;===============================================================================

Func _ScreenSearchBmp($file,$center=1)

    local $pixelarray,$screenarray

    ;get both your image.bmp and screenshot.bmp into pixel-by-pixel 2D arrays
    _FileImageToArray($file, $pixelarray)
    _Clip_screenshot(@TempDir & "\screenshot.bmp")
    _FileImageToArray(@TempDir & "\screenshot.bmp",$screenarray)
    FileDelete(@TempDir & "\screenshot.bmp")

    ;compare the 2 2D-arrays
    local $result = _2darray_in_2darray($screenarray,$pixelarray)

    ;result tidy up, for if $center=1, and for if not found.
    Local $aresult[2]
    $aresult[0] = $result[0]
    $aresult[1] = $result[1]
    if $aresult[0] = 0 then Return $aresult ;if not found , return 0 0 here
    if $center = 1 then $aresult[0] = $result[0]+ Round(UBound($pixelarray,1)/2)
    if $center = 1 then $aresult[1] = $result[1]+ Round(UBound($pixelarray,2)/2)
    Return $aresult ;if ALL GOOD, and $center=1 then return the center of the image here.

EndFunc



;===============================================================================
; Code by Malkey, converts .bmp into 2D array pixal by pixal. : thanks man!
;===============================================================================
Func _FileImageToArray($filename, ByRef $aArray)
    Local $Reslt, $stride, $format, $Scan0, $iW, $iH, $hImage
    Local $v_Buffer, $width, $height
    Local $i, $j

    _GDIPlus_Startup()
    $hImage = _GDIPlus_ImageLoadFromFile($filename)
    $iW = _GDIPlus_ImageGetWidth($hImage)
    $iH = _GDIPlus_ImageGetHeight($hImage)
    $Reslt = _GDIPlus_BitmapLockBits($hImage, 0, 0, $iW, $iH, $GDIP_ILMREAD, $GDIP_PXF32ARGB)

    ;Get the returned values of _GDIPlus_BitmapLockBits ()
    $width = DllStructGetData($Reslt, "width")
    $height = DllStructGetData($Reslt, "height")
    $stride = DllStructGetData($Reslt, "stride")
    $format = DllStructGetData($Reslt, "format")
    $Scan0 = DllStructGetData($Reslt, "Scan0")

    Dim $aArray[$width][$height]
    For $i = 0 To $iW - 1
        For $j = 0 To $iH - 1
            $aArray[$i][$j] = DllStructGetData(DllStructCreate("dword", $Scan0 + ($j * $stride) + ($i * 4)), 1)
        Next
    Next
    _GDIPlus_BitmapUnlockBits($hImage, $Reslt)
    _GDIPlus_ImageDispose($hImage)
    _GDIPlus_Shutdown()
    Return
EndFunc   ;==>_FileImageToArray


;===============================================================================
;
; Description:
; My code, search a 2D array inside another 2d array
; If found, return the positon of first element
; If error or not found, return array [0,0]. Because the very first match would be [1,1], "0" means something wrong.
; eg. search a 2d array
; [1,2,3,4]
; [5,6,7,8]
; [9,0,1,2]
;    for:
; [7,8]
; [1,2]
; You will get result [2,3] (means, matched, first element position is row 2, colunm 3)
;
; Parameter(s):
;
; Return Value(s):  On Success - Returns the array of matched [x,y], the top-left element position in the source.
;                   On Failure - Returns [0,0]
;
;
;===============================================================================

Func _2darray_in_2darray($source,$search)

    ;get the size of the both arrays
    local $sourcerow = UBound($source,1)
    Local $sourcecol = UBound($source,2)
    local $searchrow = UBound($search,1)
    Local $searchcol = UBound($search,2)

    ;error input cheching, if error return position 0,0
    if $sourcerow = 0 or $sourcecol = 0 or $searchrow = 0 or $searchcol = 0 then
        Local $aPeople[2]
        $aPeople[0] = 0
        $aPeople[1] = 0
        Return $aPeople
    EndIf


    ; A crazy 4-for-loops, compare every x,y of search array in every x,y in source array
    for $ssr = 1 to $sourcerow - $searchrow +1
        for $ssc = 1 to $sourcecol - $searchcol +1
            for $sr = 1 to $searchrow
                for $sc = 1 to $searchcol
                    ;if an element not match, go back, search for next
                    if $search[$sr-1][$sc-1] <> $source[$ssr+$sr-2][$ssc+$sc-2] then ContinueLoop 3
                Next
            Next

            ;if the loop passed all elements test,  made it here, means the result is found! congress! lets return the result:
            Local $aPeople[2]
            $aPeople[0] = $ssr
            $aPeople[1] = $ssc
            Return $aPeople

        Next
    Next

    ;all the loops finished, no result found. return [0,0]
    Local $aPeople[2]
    $aPeople[0] = 0
    $aPeople[1] = 0
    Return $aPeople

EndFunc


; #FUNCTION# ====================================================================================================================
; Name ..........: _Clip_screenshot
; Description ...: This get a screenshot.bmp using "Print Screen" key, so the image is EXACT same image you use "Print Screen" key, to avoid funny results.
; Syntax ........: _Clip_screenshot($file)
; Parameters ....: $file - The location of the screen shot .bmp file you want it to save
; Return values .: None
; Author ........: Kyle
; ===============================================================================================================================
Func _Clip_screenshot($file)
    local $tempdata = _ClipBoard_GetData() ;save current user's clipboard
    Send("{PRINTSCREEN}")
    sleep(200)
    If _ClipBoard_IsFormatAvailable($CF_BITMAP) Then
        _ClipBoard_Open(0)
        $hClipboardImage = _ClipBoard_GetDataEx($CF_BITMAP)
        _ClipBoard_Close()
        _GDIPlus_Startup()
        $hBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hClipboardImage)
        Local $iX = _GDIPlus_ImageGetWidth($hBitmap)
        Local $iY = _GDIPlus_ImageGetHeight($hBitmap)
        Local $hClone = _GDIPlus_BitmapCloneArea($hBitmap, 0, 0, $iX, $iY, $GDIP_PXF24RGB) ;make sure its 24bit bmp
        _GDIPlus_ImageDispose($hBitmap)
        $hBitmap = $hClone
        $sCLSID = _GDIPlus_EncodersGetCLSID("BMP")
        _GDIPlus_ImageSaveToFileEx($hBitmap, $file, $sCLSID, 0)
        _GDIPlus_BitmapDispose($hBitmap)
        _GDIPlus_Shutdown()
    EndIf
    _ClipBoard_SetData($tempdata) ; restore user clipboard
EndFunc

Remove the "example" part then you can include this code as a file.

Edited by kylejustknows

Share this post


Link to post
Share on other sites
On 12/4/2019 at 5:24 AM, legend said:

First one i tried that actually works, unfortunately it's super slow finding the image, any way to improve the speed?

unfortunately, this is the fastest the AutoIt can do. As an advanced script language, Autoit is awfully slow at native heavy computing (image process). Unless someone can provide me a more "advanced algorithm" to do the faster array-in-array search.

Share this post


Link to post
Share on other sites

Func mouse_move()
    $enemy = _ScreenSearchBmp("Enemy.bmp")
    While 1
    If _IsPressed(20) Then
    ElseIf $enemy = 0 Then
        MsgBox(0,"","no enemy visable")
    Else
        MouseMove($enemy[0],$enemy[1])
    EndIf
    WEnd

EndFunc

 

Can you help me it isn't working

Share this post


Link to post
Share on other sites

Skamleinad,

Welcome to the AutoIt forum.

Unfortunately you appear to have missed the Forum rules on your way in. Please read them now - particularly the bit about not discussing game automation - and then you will understand why you will get no help here.

See you soon with a legitimate question I hope.

M23


Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By zuladabef
      All my scripts were working fine and now I am getting this error.  How do I resolve it?  Which folder(s) do the DLLs need to be in?
      ! Dll not found or Call Dll error !  
    • By Synaps3
      I've recently been annoyed by how google images works. You can't click an image and see the full res image. You have to go to the website and find the image on the page and occasionally, it won't allow you to view the image easily. This script makes it easy to view any image in google images in full resolution immediately. You just press Ctrl + Q while hovering over the image you want and it'll open full res in a new tab. This is only tested in firefox, but it shouldn't be hard to modify for another browser.
      Here it is:
      #include <Misc.au3> #include <MsgBoxConstants.au3> #include <AutoItConstants.au3> #include <Clipboard.au3> #include <Array.au3> #include <String.au3> Local $clipB Local $urlArray Local $theLink HotKeySet("^q", "OpenImage") Func DecodeUrl($src) Local $i Local $ch Local $buff ;Init Counter $i = 1 While ($i <= StringLen($src)) $ch = StringMid($src, $i, 1) ;Correct spaces If ($ch = "+") Then $ch = " " EndIf ;Decode any hex values If ($ch = "%") Then $ch = Chr(Dec(StringMid($src, $i + 1, 2))) $i += 2 EndIf ;Build buffer $buff &= $ch ;Inc Counter $i += 1 WEnd Return $buff EndFunc ;==>DecodeUrl Func OpenImage() MouseClick($MOUSE_CLICK_RIGHT) Send("A") Sleep(100) $clipB = _ClipBoard_GetData($CF_TEXT) $theLink = DecodeUrl($clipB) $urlArray = _StringBetween($theLink, "=", "&") If StringInStr($urlArray[0], "?") <> 0 Then $urlArray = _StringBetween($theLink, "=", "?") EndIf ShellExecute($urlArray[0]) EndFunc While 1 Wend  
    • By Schuster
      Hey there,
      i need your help guys. I am working in the QA department of my company. A little part of or software produces hints (like little popups) which needs to be tested every release. I wrote scripts to trigger those hints and i also implemented the imagesearch.dll to find those hints on the screen. My goal now is it to let the machine work over night, so when i come back to work i only get a report which says 130/140 hints were found successfully. This already works, aslong as my pc is not locked, but i want/need to lock my pc because of security concerns.
      My general question. Is it possible to run my scripts on my locked PC and still find an image that i provide which will open sooner or later in the background?
       
      Thanks in advance
    • By kuhicop
      Hello, I need to find an image on screen and return it's position left, top, right, botton.
      I'm using the ImageSearch function but it only returns 1 or 0.
      Any ideas? Thanks!
    • By Errious
      Hello,
      maybe i am just tired or i did not understand exactly how to use ImageSearch for different pictures, but i tried now a few things with my example script and only the first part is working.
      Explaining what i try to do, at work we have a programm running which has three different states, they are visible but sometimes the statuses of the process is not running and to avoid a long time without progress of the programm i searched to get a way to have a better visible notification as existing, so there is grey as work in progress, orange for not running and red for aborted, now when the states are changing i would like to receive a little window that is just telling the actual status.
      #include <ImageSearch2015.au3> #include <MsgBoxConstants.au3> #include <AutoItConstants.au3> $x1=0 $y1=0 While 1 sleep(100) $image1 = _ImageSearch("red.png", 1,$x1,$y1,0) If $image1 = 1 then SplashTextOn ( "", "RED !" , 100 , 50 , 1800 , 220 , $DLG_TEXTLEFT, "Arial" , 12 , 500 ) Sleep(1000) SplashOff() sleep(100) EndIf $image2 = _ImageSearch("orange.png", 1,$x1,$y1,0) If $image2 = 1 Then SplashTextOn ( "", "Orange !" , 100 , 50 , 1800 , 220 , $DLG_TEXTLEFT, "Arial" , 12 , 500 ) Sleep(1000) SplashOff() sleep(100) EndIf $image3 = _ImageSearch("grey.png", 1,$x1,$y1,0) If $image3 = 1 Then SplashTextOn ( "", "Grey !" , 100 , 50 , 1800 , 220 , $DLG_TEXTLEFT, "Arial" , 12 , 500 ) Sleep(1000) SplashOff() EndIf WEnd As i said before, part one to receive the state for aborted is working, so i get the splash window for this, but what do i miss for the other two?
      I put the While statement in for continuously running as long it is needed when the program is in use, but i believe i missed something.
×
×
  • Create New...