BmpSearch - Search for Bitmap within Bitmap - Assembly Version

[InnI]

2 hours ago, pubeoutros said:

It's possible to add some tolerance to this function?

Image search is in the binary code that Beege wrote. Ask him.

[pubeoutros]

On 1/27/2014 at 5:54 AM, Beege said:

Heres a function for searching for a bitmap within another bitmap. The heart of it is written assembly (source included) and working pretty quick I feel. I have included an example which is pretty basic and should be easily enough for anyone to get the concept. 

You will be given a small blue window that will take a screencapture of that size:

 

It will then take a full screenshot and highlight all locations that it found

Please let me know if you have any issues or questions. Thanks!

BmpSearch.zip

 

GAMERS - Asking for help with ANY kind of game automation is against the forum rules. DON'T DO IT.

Hi!

It's possible to add some tolerance to this function? Sometimes it doesn't find the image just because of a simple tolerance.

Thanks!

[Beege]

This has been updated and rewrote with fasmg.  I got everything separated out into two folders. The source folder contains everything needed with fasmg if you want to modify it. When it comes to tolerance I need some samples to work with. I feel like this is more a change in display or bit count and not a small difference. I noticed that this completely fails if the bmps are 16 bit which may happen if you are running in a lesser graphics mode like I was the other day using RDP.  I updated the bmp dump function to use GetDIBits instead of  GetBitmapBits so I could force 32bit for now. 

[totta_fadil]

thank you very much for this.....I've some question though if I may ask

if I run this inside a loop I always get this error after some minutes

Variable must be of type "Object".:
Local $iRowInc = ($tSizeSource.X - $tSizeFind.X) * 4
Local $iRowInc = ($tSizeSource^ ERROR

my main code search for a simple Picture in the background using ImagesearchEX UDF

#AutoIt3Wrapper_UseX64=NO
#include <imageSearchEX.au3>
#include "_IMGSearch_Debug.au3"
pic1()
Func pic1()
    $ImgPath = @ScriptDir & "\duplicatepic.bmp"
    $Title = "File Explorer"
    Sleep(500)
    ControlClickWindowsByImage($Title, $ImgPath)
Call("pic1")
EndFunc

Func ControlClickWindowsByImage($Title, $ImgPath)
    $p = imageSearchEX($Title, $ImgPath)
    If $p = 0 Then
        Return SetError(3)
    EndIf
    ControlClick($Title, "", "", "",1 , $p[0], $p[1])
EndFunc

another question is.....can I use this on other computers with the same Pictures ? I mean without having to change the Pictures themselves using tolerence or something ?

if so how can I do it please.....

thanks

[Beege]

19 hours ago, totta_fadil said:

my main code search for a simple Picture in the background using ImagesearchEX UDF

pic1()
Func pic1()
    $ImgPath = @ScriptDir & "\duplicatepic.bmp"
    $Title = "File Explorer"
    Sleep(500)
    ControlClickWindowsByImage($Title, $ImgPath)
    Call("pic1") <<<=======================================================
EndFunc

 

Thanks @totta_fadil. Your current issue is due to how your calling your function from within your function. Thats called recursion and there is a limit of how many times a function can recurse itself . Thats also why it works for a little bit then stops.

https://www.autoitscript.com/autoit3/docs/appendix/LimitsDefaults.htm (MAXCALLRECURSE)

Put your function in a loop with one of these statements:

https://www.autoitscript.com/autoit3/docs/intro/lang_loops.htm

There is currently no tolerance put in place. Its a exact match or not so if that exact bmp is there on the other pc it would, but with screenshots the image can change a if the monitor dimensions are different so keep that in mind.  

 

[totta_fadil]

23 hours ago, Beege said:

Thanks @totta_fadil. Your current issue is due to how your calling your function from within your function. Thats called recursion and there is a limit of how many times a function can recurse itself . Thats also why it works for a little bit then stops.

https://www.autoitscript.com/autoit3/docs/appendix/LimitsDefaults.htm (MAXCALLRECURSE)

Put your function in a loop with one of these statements:

https://www.autoitscript.com/autoit3/docs/intro/lang_loops.htm

There is currently no tolerance put in place. Its a exact match or not so if that exact bmp is there on the other pc it would, but with screenshots the image can change a if the monitor dimensions are different so keep that in mind.  

 

thanks again.

I tried with Do..until

and While...Wend

but still same error ? it's an error from the UDF

"C:\Program Files (x86)\AutoIt3\Include\BmpSearch.au3" (46) : ==> Variable must be of type "Object".:
Local $iRowInc = ($tSizeSource.X - $tSizeFind.X)*4
Local $iRowInc = ($tSizeSource^ ERROR
>Exit code: 1    Time: 116.2

and the code is

#AutoIt3Wrapper_UseX64=NO
#include <imageSearchEX.au3>
#include "_IMGSearch_Debug.au3"

$S=0
Do
    $ImgPath = @ScriptDir & "\duplicatepic.bmp"
    $Title = "File Explorer"
    Sleep(500)
    ControlClickWindowsByImage($Title, $ImgPath)
Until $S=1

Func ControlClickWindowsByImage($Title, $ImgPath)
    $p = imageSearchEX($Title, $ImgPath)
    If $p = 0 Then
        Return SetError(3)
    EndIf
    ControlClick($Title, "", "", "",1 , $p[0], $p[1])
    
Endfunc

 

Edited October 27, 2019 by totta_fadil
mistake in code

[Beege]

Your missing a EndFunc statement in your function. That specific error your getting means _WinAPI_GetBitmapDimension($hSource) failed to get the dimensions of the bitmap getting passed to it. I didn't write imageSearchEx() so I'm not sure what happens in between.  

[totta_fadil]

1 hour ago, Beege said:

Your missing a EndFunc statement in your function. That specific error your getting means _WinAPI_GetBitmapDimension($hSource) failed to get the dimensions of the bitmap getting passed to it. I didn't write imageSearchEx() so I'm not sure what happens in between.  

sorry I copied it manually so I missed the EndFunc in the code but I corrected it ? this is the ImagesearchEX link.....and if the problem is with this ImagesearchEX, can I use some alternative way to do it ? I mean an Image search in the background ?

[Beege]

23 minutes ago, totta_fadil said:

sorry I copied it manually so I missed the EndFunc in the code but I corrected it ?

Yes the loop is valid now and no more recursion.

24 minutes ago, totta_fadil said:

this is the ImagesearchEX link.....and if the problem is with this ImagesearchEX, can I use some alternative way to do it ? I mean an Image search in the background ?

Sorry I don't know what you mean when you say "image search in the background" but either way we are getting out of scope for what my function does and Im not going to troubleshoot someone else's code for you because I would still be on a wild goose chase. 

All my function does is take two inputs which will be handles to a big bitmap and (presumably) a little bitmap and searches the big for existence of the little.  The core loop of the function is honestly no different then searching for a string within in a string (it literally uses CMPSD--Compare String instruction) which is the main reason there's no tolerance. How one goes about getting those bitmaps, weather screencapture or some other app creating them, and then also processes those bitmaps via loading and getting handles, converting, clean up (big key here*)  before sending to my function is a much different scope. 

I don't think my function is the issue here but if it was I would need the source and search bmps to troubleshoot. That's the only way to trouble shoot a function like this. In your case I belive you need to dig in and learn a bit more of what the code is doing and play with debugging it. Try and get an understanding of what every variable being passed to every function is suppose to be and confirm it is what it should be. Make "ALT-D" key shortcut your friend. _arraydisplay() for looking at anything passing an array. 

Your getting stuck at _WinAPI_GetBitmapDimension(). Open the documentation and then open the example it has. Make sure you understand not only how that function works, but also each other supporting function in the example. Then modify the example and substitute your bmp with the one used in the example. See what you can learn from just playing with that and then go back to your code and see if you can confirm all the requirements for _WinAPI_GetBitmapDimension() are getting met with whatever bmps your working with. 

 

[faustf]

hi guys , anyone  have  some  example for find  shape  with color with tollerance ???  thankz

[ScrapeYourself]

Hello Beege,

 

First I would like to thank you for bringing up this topic and developing a very nice tool to do it.

I am working on automating deductions processing for a website, and this tool seems like it could really help me as the 2nd part of the automation involves clicking in a flash website to lookup invoice numbers, and downloading it, then uploading it to our website under the matching number.

Which tool do we use to supply the images to search for?
 

Edited November 26, 2019 by ScrapeYourself

[lakazai]

On 10/28/2019 at 4:01 AM, Beege said:

Yes the loop is valid now and no more recursion.

Sorry I don't know what you mean when you say "image search in the background" but either way we are getting out of scope for what my function does and Im not going to troubleshoot someone else's code for you because I would still be on a wild goose chase. 

All my function does is take two inputs which will be handles to a big bitmap and (presumably) a little bitmap and searches the big for existence of the little.  The core loop of the function is honestly no different then searching for a string within in a string (it literally uses CMPSD--Compare String instruction) which is the main reason there's no tolerance. How one goes about getting those bitmaps, weather screencapture or some other app creating them, and then also processes those bitmaps via loading and getting handles, converting, clean up (big key here*)  before sending to my function is a much different scope. 

I don't think my function is the issue here but if it was I would need the source and search bmps to troubleshoot. That's the only way to trouble shoot a function like this. In your case I belive you need to dig in and learn a bit more of what the code is doing and play with debugging it. Try and get an understanding of what every variable being passed to every function is suppose to be and confirm it is what it should be. Make "ALT-D" key shortcut your friend. _arraydisplay() for looking at anything passing an array. 

Your getting stuck at _WinAPI_GetBitmapDimension(). Open the documentation and then open the example it has. Make sure you understand not only how that function works, but also each other supporting function in the example. Then modify the example and substitute your bmp with the one used in the example. See what you can learn from just playing with that and then go back to your code and see if you can confirm all the requirements for _WinAPI_GetBitmapDimension() are getting met with whatever bmps your working with. 

 

sorry, i same error when calling bmpsearch every second for a long time.
 

Quote

"C:\Program Files (x86)\AutoIt3\Include\BmpSearch.au3" (46) : ==> Variable must be of type "Object".:
Local $iRowInc = ($tSizeSource.X - $tSizeFind.X)*4
Local $iRowInc = ($tSizeSource^ ERROR

 

[blackeye87]

Hello Beege

Im newbie in Autoit

Quote

Local $hFind = _ScreenCapture_Capture('', $aPos[0], $aPos[1], $aPos[0] + $aPos[2], $aPos[1] + $aPos[3])

How can i load an bitmap from path then input into $hFind?

I tried "_GDIPlus_BitmapCreateFromFile",  "_WinAPI_LoadImage" but i get error  BmpSearch.au3" (46) : ==> Variable must be of type "Object".

Can you help me? Thank you.

[Beege]

On 1/31/2020 at 3:05 PM, blackeye87 said:

Hello Beege

Im newbie in Autoit

How can i load an bitmap from path then input into $hFind?

I tried "_GDIPlus_BitmapCreateFromFile",  "_WinAPI_LoadImage" but i get error  BmpSearch.au3" (46) : ==> Variable must be of type "Object".

Can you help me? Thank you.

This below works; however, I had to adjust the window capture height to 12. _GDIPlus_BitmapCreateFromMemory gives me an error code of 3 for some reason when height is 10 from the example.

_GDIPlus_Startup() ;initialize GDI+
Local $hHighlight_Capture = GUICreate('', 36, 12, -1, -1, $WS_POPUPWINDOW, $WS_EX_CONTROLPARENT)
;...
_ScreenCapture_Capture(@ScriptDir & '\find.bmp', $aPos[0], $aPos[1], $aPos[0] + $aPos[2], $aPos[1] + $aPos[3])
Local $hFind = _GDIPlus_BitmapCreateFromMemory(Binary(FileRead(@ScriptDir & '\find.bmp')), True)
If @error Then Exit (ConsoleWrite('_GDIPlus_BitmapCreateFromMemory ERROR: ' & @error & @CRLF))

 

[careca]

Hi, im getting an error in the example with the line 37:

Local $aCords = _BmpSearch($hSource, $hFind)

Script just hangs for a bit, and fails:

!>05:50:22 AutoIt3.exe ended.rc:-1073741819

This is with the fasm in the first post. Any tips?

AutoIt 3.3.14.5

AutoIt3Wrapper 19.1127.1402.0

[InnI]

@careca This UDF contains compiled code for x86 only. Make sure that you don't use AutoIt3_x64.exe

[Sw4rM]

I try to make a function to search multi-pixels with shade variation.

I replaced _FindFirstDiff with this code.

Func _FindFirstDiff($tPix)

    ;####### (BinaryStrLen = 106) ########################################################################################################################
    Static Local $Opcode = '0xC80000008B5D0C8B1383C3048B4D103913750C83C304E2F7B800000000EB118B5508FF338F028B451029C883C002EB00C9C20C00'
    Static Local $aMemBuff = DllCall("kernel32.dll", "ptr", "VirtualAlloc", "ptr", 0, "ulong_ptr", BinaryLen($Opcode), "dword", 4096, "dword", 64)
    Static Local $tMem = DllStructCreate('byte[' & BinaryLen($Opcode) & ']', $aMemBuff[0])
    Static Local $fSet = DllStructSetData($tMem, 1, $Opcode)
    ;#####################################################################################################################################################

    Local $iMaxLoops = (DllStructGetSize($tPix) / 4) - 1
    If $iMaxLoops Then
        Local $aRet = DllCallAddress('dword', DllStructGetPtr($tMem), 'dword*', 0, 'struct*', $tPix, 'dword', $iMaxLoops)
        Return $aRet
    Else
        Local $aRet[] = [1, DllStructGetData($tPix, 1, 1)]
        Return $aRet
    EndIf
EndFunc   ;==>_FindFirstDiff

And wrote new code below.

expandcollapse popup

#include <WindowsConstants.au3>
#include <WinAPISysWin.au3>
#include <GDIPlus.au3>
#include <Color.au3>
#include <BmpSearch.au3>

Func Terminate()
    Exit
EndFunc   ;==>Terminate
HotKeySet("{ESC}", "Terminate")

Local $ihWnd = _WinAPI_GetDesktopWindow()

Local $bCords, $iTime
While 1
    $bCords = iSearchPixel(0, 0, 960, 600, 16777215, 5, 5000, $ihWnd)
    If Not @error Then
        $iTime = @extended
        ToolTip($bCords[0][0] & ' matches found. Time = ' & $iTime & ' ms')
        ;_ArrayDisplay($bCords)
    Else
        ToolTip('No Matches Found!')
    EndIf

    Sleep(200)
WEnd

;############################ FUNCTION ##################################
;iSearchPixel ( left, top, right, bottom, color [, shade-variation = 0 [, max search = 5000 [, hwnd]]] )
;#######################################################################
Func iSearchPixel($iLeft, $iTop, $iRight, $iBottom, _
        $iColorSearch, $iColorShade = 0, $iMax = 5000, $WinHandle = _WinAPI_GetDesktopWindow())

    Local $iTime = TimerInit()

    Local $hBMP_Width = $iRight - $iLeft + 1
    Local $hBMP_Height = $iBottom - $iTop + 1

    Local $Start_X = $iLeft
    Local $Start_Y = $iTop

    $iColorSearch = Hex($iColorSearch, 6)

    if $iColorShade <> 0 Then $iColorShade = 20 + ($iColorShade * 2)

    Local $RGBcolor = _ColorGetRGB(Dec($iColorSearch))

    Local $upperR = $RGBcolor[0] + $iColorShade
    If $upperR > 255 Then $upperR = $RGBcolor[0]
    Local $upperRcolor = Hex($upperR, 2) & Hex($RGBcolor[1], 2) & Hex($RGBcolor[2], 2)

    Local $upperG = $RGBcolor[1] + $iColorShade
    If $upperG > 255 Then $upperG = $RGBcolor[1]
    Local $upperGcolor = Hex($RGBcolor[0], 2) & Hex($upperG, 2) & Hex($RGBcolor[2], 2)
    Local $upperRGcolor = Hex($upperR, 2) & Hex($upperG, 2) & Hex($RGBcolor[2], 2)

    Local $upperB = $RGBcolor[2] + $iColorShade
    If $upperB > 255 Then $upperB = $RGBcolor[2]
    Local $upperBcolor = Hex($RGBcolor[0], 2) & Hex($RGBcolor[1], 2) & Hex($upperB, 2)
    Local $upperRBcolor = Hex($upperR, 2) & Hex($RGBcolor[1], 2) & Hex($upperB, 2)
    Local $upperGBcolor = Hex($RGBcolor[0], 2) & Hex($upperG, 2) & Hex($upperB, 2)

    Local $upperRGBcolor = Hex($upperR, 2) & Hex($upperG, 2) & Hex($upperB, 2)

    Local $lowerR = $RGBcolor[0] - $iColorShade
    If $lowerR < 0 Then $lowerR = $RGBcolor[0]
    Local $lowerRcolor = Hex($lowerR, 2) & Hex($RGBcolor[1], 2) & Hex($RGBcolor[2], 2)

    Local $lowerG = $RGBcolor[1] - $iColorShade
    If $lowerG < 0 Then $lowerG = $RGBcolor[1]
    Local $lowerGcolor = Hex($RGBcolor[0], 2) & Hex($lowerG, 2) & Hex($RGBcolor[2], 2)
    Local $lowerRGcolor = Hex($lowerR, 2) & Hex($lowerG, 2) & Hex($RGBcolor[2], 2)

    Local $lowerB = $RGBcolor[2] - $iColorShade
    If $lowerB < 0 Then $lowerB = $RGBcolor[2]
    Local $lowerBcolor = Hex($RGBcolor[0], 2) & Hex($RGBcolor[1], 2) & Hex($lowerB, 2)
    Local $lowerRBcolor = Hex($lowerR, 2) & Hex($RGBcolor[1], 2) & Hex($lowerB, 2)
    Local $lowerGBcolor = Hex($RGBcolor[0], 2) & Hex($lowerG, 2) & Hex($lowerB, 2)

    Local $lowerRGBcolor = Hex($lowerR, 2) & Hex($lowerG, 2) & Hex($lowerB, 2)

    Local $bBinaryPic = '0x424D3A000000000000003600000028000000010000000100000001001800000000000400000000000000000000000000000000000000' _ ;Bmp size 1x1
             & Hex($RGBcolor[2], 2) & Hex($RGBcolor[1], 2) & Hex($RGBcolor[0], 2) _ ;'& 00FF00' _ ;Color --> Blue Green Red
             & '00'

    _GDIPlus_Startup()

    Local $hDDC = _WinAPI_GetDC($WinHandle)
    Local $hCDC = _WinAPI_CreateCompatibleDC($hDDC)
    Local $hBMP = _WinAPI_CreateCompatibleBitmap($hDDC, $hBMP_Width, $hBMP_Height)

    _WinAPI_SelectObject($hCDC, $hBMP)
    _WinAPI_BitBlt($hCDC, 0, 0, $hBMP_Width, $hBMP_Height, $hDDC, $Start_X, $Start_Y, $SRCCOPY)

    ;Local $BMP = _GDIPlus_BitmapCreateFromHBITMAP($hBMP)

    If $iColorShade > 0 Then

        Local $BMP = _GDIPlus_BitmapCreateFromHBITMAP($hBMP)

        Local $COLOR_tPalette = _GDIPlus_PaletteInitialize(15, $GDIP_PaletteTypeOptimal, 15, False, $BMP)
        $COLOR_tPalette.ARGB((1)) = '0xFF' & $iColorSearch

        $COLOR_tPalette.ARGB((2)) = '0xFF' & $upperRcolor
        $COLOR_tPalette.ARGB((3)) = '0xFF' & $upperGcolor
        $COLOR_tPalette.ARGB((4)) = '0xFF' & $upperRGcolor
        $COLOR_tPalette.ARGB((5)) = '0xFF' & $upperBcolor
        $COLOR_tPalette.ARGB((6)) = '0xFF' & $upperRBcolor
        $COLOR_tPalette.ARGB((7)) = '0xFF' & $upperGBcolor
        $COLOR_tPalette.ARGB((8)) = '0xFF' & $upperRGBcolor

        $COLOR_tPalette.ARGB((9)) = '0xFF' & $lowerRcolor
        $COLOR_tPalette.ARGB((10)) = '0xFF' & $lowerGcolor
        $COLOR_tPalette.ARGB((11)) = '0xFF' & $lowerRGcolor
        $COLOR_tPalette.ARGB((12)) = '0xFF' & $lowerBcolor
        $COLOR_tPalette.ARGB((13)) = '0xFF' & $lowerRBcolor
        $COLOR_tPalette.ARGB((14)) = '0xFF' & $lowerGBcolor
        $COLOR_tPalette.ARGB((15)) = '0xFF' & $lowerRGBcolor

        ;msgbox(0, $iColorSearch, $upperRGBcolor & ', ' & $lowerRGBcolor)

        _GDIPlus_BitmapConvertFormat($BMP, $GDIP_PXF04INDEXED, $GDIP_DitherTypeSolid, $GDIP_PaletteTypeOptimal, $COLOR_tPalette)
        _WinAPI_DeleteObject($COLOR_tPalette)

        _WinAPI_DeleteObject($hBMP)

        local $hBMP = _GDIPlus_BitmapCreateHBITMAPFromBitmap($BMP)

    EndIf

    #cs
        ; Draw bitmap to GUI
        local $hGUI = GUICreate("hFind", $hBMP_Width, $hBMP_Height)
        GUISetState(@SW_SHOW)
        local $hGraphic = _GDIPlus_GraphicsCreateFromHWND($hGUI)
        _GDIPlus_GraphicsDrawImage($hGraphic, $BMP, 0, 0)
        MsgBox($MB_TOPMOST, 'TEST', "Press OK")
        Sleep(500)
        GUIDelete($hGUI)
    #ce

    Local $hBitmap = _GDIPlus_BitmapCreateFromMemory(Binary($bBinaryPic), True)

    ;#######################################################
    Local $aCords = _BmpSearch($hBMP, $hBitmap)
    ;#######################################################
    If Not @error Then
        _WinAPI_ReleaseDC($WinHandle, $hDDC)
        _WinAPI_DeleteDC($hCDC)
        _WinAPI_DeleteObject($hBMP)
        _WinAPI_DeleteObject($hBitmap)
        ;_GDIPlus_GraphicsDispose($hGraphic)
        If IsDeclared('BMP') Then _GDIPlus_BitmapDispose($BMP)

        _GDIPlus_Shutdown()

        Return SetExtended(TimerDiff($iTime), $aCords)
    Else
        _WinAPI_ReleaseDC($WinHandle, $hDDC)
        _WinAPI_DeleteDC($hCDC)
         _WinAPI_DeleteObject($hBMP)
        _WinAPI_DeleteObject($hBitmap)
        ;_GDIPlus_GraphicsDispose($hGraphic)
        If IsDeclared('BMP') Then _GDIPlus_BitmapDispose($BMP)

        _GDIPlus_Shutdown()

        Return SetError(1)
    EndIf

EndFunc   ;==>iSearchPixel

If the screen is a freeze frame (not motion), it works well all the time.

But I have a problem. When the desktop has many motion pictures on the screen such as a video, this script can't work for long run and has fatal error (-1073741819) in the end.

How can I fix it?

Best Regards

Edited October 25, 2020 by Sw4rM