Sign in to follow this  
Followers 0

Check image on screen and get closest one from library

3 posts in this topic

Hello all.

I'm trying to do something simple, but I can't manage to find where to start since I can have multiple possibilities and not sure what is the best one :

  1. Taking screen capture (I can do that)
  2. On this capture, I'll look for 10 particular regions that will give me 10 images of 40px * 40px (I can do that)
  3. Then what I want is for each of these regions, find on my library of ~140 images which one is the closest one to my region (= the match)


What do you think is the best approach ? Ican easily do step 1 and 2, however for the third step :

  • I can't use memcmp because they won't be exactly the same (original library can differ a very little bit from what I'll get from my screen capture)
  • There is no text, only drawing
  • Should I do a pixelSum of what I get on screen and do the same via GDIPlus for each of my images in the library?


Thanks a lot,



Share this post

Link to post
Share on other sites

#2 ·  Posted (edited)

Tried this ;

And this :

Without a lot of success, bot scripts fail for different reason and are quite old.

Edited by timmalos

Share this post

Link to post
Share on other sites
1 hour ago, timmalos said:

I'm trying to do something simple

Errm, no, not at all. Image recognition and interpretation is computationally very hard, otherwise companies wouldn't be spending hundreds of millions of dollars on developing better algorithms (self-learning, AI, PCA) to do something you take for granted such as distinguishing between the faces of your family members, for example.

That said, here are some simple suggestions, or you could use ImageMagick's compare functionality.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • SimTheNo1
      By SimTheNo1
      I am busy with building a solution for change monitoring of VOIP call program and to be properly automated means among others need for some limited OCR functionality. Current works of others are way too much overkill for this case what makes the need to build it myself. But to do it properly I significantly have to increase my knowledge about digital graphics management.
      For now I already have discovered here and there some mind blowing  Autoit miracles what can be achieved with Windows own possibilities to manipulate that what is output to the monitor. And as far I can judge there are 2 options to process graphics without use of any external libraries like ImageMagick, FreeImage and so on. These are:
       WinAPI  GDIPlus It is for me quite obvious to have various holes in general understanding of graphics and it is once more very clear what advantages gives consistent general study in 1 or another official institutions like University.  Cause there you are introduced into certain domain of knowledge in a way which usually has been perfected over long period of time. So you are not overloaded by stuff which requires a certain amount of information to be initially clear for you. For example, before starting to solve physics you first learn to read, count and so on and then move to subjects like physics.
      Though in my case opportunity to study in such educational system I had only for 8 years, from my 7th to 15th year of age in the country that was falling apart now Ukraine but used to be USSR, was all what it was. After have immigrated to Netherlands possibilities to study further haven't occur. And this therefor causes often various implications when going deep in that or another field of practical knowledge acquiring for any needed physical result, like programming to perform enormous amount of tasks. In this particular case automating VOIP call program.
      Anyway, right now I think the best direction to move is to concentrate on as basic as possible image management and if someone would maybe explain in general what is a pixel will definitely help. Particularly I am very curious about how to do picture manipulations in Autoit. Especially would help a lot to produce eventually following functions:
      createImage($imageFileName, $width, $height, $color) readImagePixel($imageFileName, $x, $y)  writeImagePixel($imageFileName, $x, $y) I do not know exactly how image is handled in computer but preferably above mentioned functions should deal with so far possible origin of graphics creation on computer. But nevertheless I definitely would love to hear any proposition for solution.
      The problem with explaining screenshots:
       VOIPConnect full Window  Part of Window with control to monitor for changes  Exact location of area where actual changes occur and have to be processed  It comes down to a rectangle of approximately 51 pixel wide and 7 pixel high. In fact if I get to learn as far as to be able exactly read, write and compare 2 images consistently across different computers I could narrow down then even further the area to watch as little as a square of 2-5 pixels wide. 
      To finish here is last detail about particularly no need for ultra fast solution at all. This because it is needed only once when it is first run on a new computer and to have to wait few minutes while it is being set instead of just a few seconds make no sence.
      This is it and what I too think to do beside this very particular case is to purify out beautiful generic Autoit functions for core image manipulation by using WinAPI or/and GDIPlus. 
    • corgano
      By corgano
      What I am trying to accomplish is to simplify a .PNG image and convert it to use only a specified pallet of colors. Eg if I have a pallet of 0xFF0000, 0x00FF00, 0x0000FF, it would change every pixel in the image to one of those three colors by whichever color is closest.
      The simplest method I have tried is simply converting the image to a 16 bit bitmap, and that does simplify the pallet of the image, but I want to be able to specify what colors it specifies to. The other problem is the resulting saved .BMP file is much larger (file size) than the original, and I don't know if it's possible to make a 16 bit jpg.
      It looks like _GDIPlus_BitmapConvertFormat is the solution but the help file doesn't go into how to use the pallet and I couldn't find an example that did what I needed. Can anyone help me?
    • timmalos
      By timmalos
      Hello all . i'm trying to build a tool to generate some pictures/graphs automatically based on some variables. I have completed the first part and I'm struggling now on what I thought would be the easiest part :
      In my tool there is a "preview" button, which helps to display on a new children GUI the generated picture. What I do in simplified :
      Global $g_hGraphics = _GDIPlus_GraphicsCreateFromHWND($g_hGUI) ;Then, I'll add multiple objects to this graphic, for example $Temp = _GDIPlus_ImageLoadFromFile($temp[1]) _GDIPlus_GraphicsDrawImage($g_hGraphics,$temp)  
      I'm then able to have a preview GUI showing all my elements perfectly. This works but now I want to be able to really generate it and have a ".png" file written to disk.
      What I tried without success:
      $g_hBitmap = _GDIPlus_BitmapCreateFromGraphics($previewWidth, $previewHeight, $g_hGraphics) _GDIPlus_ImageSaveToFile($g_hBitmap,$OUTPUTDIR & "\generated\test-"&@HOUR&@MIN&@SEC&".png") But what I write to file is a transparent PNG without anything.
      I'm pretty sure I'm missing something obvious there, I need your help
      NB : If this change your answer, my goal will be to generate transparent png, meaning I'll add multiple objects/pictures/texts on a transparent png background and if there are some "transparent holes" I need them to be transparent in the generated png.
      Thanks a lot for any help !
    • TheNorwegianUser
      By TheNorwegianUser
      I don't get why the images I take turns up like this, but it seems to me like the screenshot doesn't go from the actual positions specified. It's weird, cause what happends is that when I for example move the mouse over the number 25 on SciTE, the image shows that it's over 19... Anyone knows what I'm doing wrong?
      #include <screencapture.au3> #include <WindowsConstants.au3> Hotkeyset("{ESC}", "Terminate") $gui = GUICreate("", 100, 800, 100, 0, $WS_POPUP, $WS_EX_TOPMOST) _GDIPlus_Startup() $graphics = _GDIPlus_GraphicsCreateFromHWND($gui) GUISetState(@SW_SHOW, $gui) While 1 $scr = _ScreenCapture_Capture("", 0, 0, 100, 800) $bitmap = _GDIPlus_BitmapCreateFromHBITMAP($scr) _WinAPI_DeleteObject($scr) _GDIPlus_GraphicsDrawImage($graphics, $bitmap, 0, 0) WEnd Func Terminate() _GDIPlus_GraphicsDispose($graphics) _GDIPlus_BitmapDispose($bitmap) _GDIPlus_Shutdown() Exit EndFunc  
    • LegitStack
      By LegitStack
      Been struggling with this one for a while. 
      when I do a _screencapture_capture call on a high resolution monitor (like my surface book) it gives me an image that has 2 problems:
      1. its in the wrong location on the screen
      2. it gives me a picture that is larger than the area of the screen I selected, though it only has the content of what I selected. 
      I was able to easily fix problem #1 by manually adjusting the x y coordinates to compensate for the amount of DPI scale I have.
      for instance if I'm 200% zoomed in the code looks like this:
      Local $bmp = _ScreenCapture_Capture("", $iX1*2, $iY1*2, $iX2*2, $iY2*2, false)  it's problem #2 that is the big problem. I'd like to attach a screen shot of what I'm talking about (see capture.png)

      Now I basically understand why this is happening. ScreenCapture grabs each pixel of the screen. This screen, being a high resolution, when its zoomed in adds up several pixels to make one on the screen. 
      This is a problem for me because I'm taking images of the screen and later looking for those exact images on the screen. if everything is blown up by an indeterminate amount (in my case 2x) then those images can't be found later on. 
      Does anyone know what I can do? 
      I tried resizing the images back down to no avail. 
      _GDIPlus_ImageResize and _GDIPlus_ImageScale don't work because they don't compress the pixels correctly. quality is lost. and the exact image isn't preserved, so I can't search for it later. (see capture1.png)

      Anyway, I'm about to give up, been on this problem for too long! does anyone know what I can do? Seems to me that the ideal solution would be to eventually have autoit add an argument to _screencapture_capture that lets you specify a DPI scale amount or something.  that can be pulled from the registry at HKEY_CURRENT_USER\control panel\desktop\windowmetrics\appliedDPI.
      But in the meantime, does anyone have any suggestions for how I can make my program compatible with 4k resolution monitors? I either need to take the screen capture like normal, then scale it down appropriately without losing quality, or I need to capture the screen in the first place like the human sees it. But I don't know how to do that either. 
      I'll post my relevant code here: (in my project I call ScreenCapture_DPI_Aware)
      #include <Security.au3> Func _GetAppliedDPI()   Local $aArrayOfData = _Security__LookupAccountName(@UserName)   If IsArray($aArrayOfData) Then   ;msgbox(64, "SID String = ", $aArrayOfData[0] & @CRLF)   ;msgbox(64, "Domain name = ", $aArrayOfData[1] & @CRLF)   ;msgbox(64, "SID type = ", _Security__SidTypeStr($aArrayOfData[2]) & @CRLF)     ;Local $AppliedDPI = RegRead("HKEY_USERS\" & $aArrayOfData[0] & "\Control Panel\Desktop\WindowMetrics", "AppliedDPI") Local $AppliedDPI = RegRead("HKEY_CURRENT_USER\Control Panel\Desktop\WindowMetrics", "AppliedDPI")     return $AppliedDPI   EndIf EndFunc Func GetScale()   $applied = _GetAppliedDPI()   if $applied == "" then     return 1   else     return $applied / 96   EndIf EndFunc Func ScreenCapture_Capture_DPI_Aware($sBMP_Path, $iX1, $iY1, $iX2, $iY2, $bool)   $R = GetScale() ;Raito   Local $bmp = _ScreenCapture_Capture($sBMP_Path, $iX1*$R, $iY1*$R, $iX2*$R, $iY2*$R, $bool) ;Scaling didn't work: ;_ScaleImage($bmp, $sBMP_Path, abs($iX2 - $iX1), abs($iY2 - $iY1), $R)   ;return _ScreenCapture_Capture($sBMP_Path, $iX1*$R, $iY1*$R, $iX2*$R, $iY2*$R, $bool) EndFunc ;Func _ScaleImage($bmp, $outimage, $w, $h, $scale) ; _GDIPlus_Startup() ;Get the encoder of to save the resized image in the format you want. ; Local $Ext = StringUpper(StringMid($outimage, StringInStr($outimage, ".", 0, -1) + 1)) ; $CLSID = _GDIPlus_EncodersGetCLSID($Ext) ; code found here : ; Local $sImgCLSID = _GDIPlus_EncodersGetCLSID("png") ;create CLSID for a JPG image file type ; Local $tGUID = _WinAPI_GUIDFromString($sImgCLSID) ;convert CLSID GUID to binary form and returns $tagGUID structure ; Local $tParams = _GDIPlus_ParamInit(1) ;initialize an encoder parameter list and return $tagGDIPENCODERPARAMS structure ; Local $tData = DllStructCreate("int Quality") ;create struct to set JPG quality setting ; DllStructSetData($tData, "Quality", 100) ;quality 0-100 (0: lowest, 100: highest) ; Local $pData = DllStructGetPtr($tData) ;get pointer from quality struct ; _GDIPlus_ParamAdd($tParams, $GDIP_EPGQUALITY, 1, $GDIP_EPTLONG, $pData) ;add a value to an encoder parameter list ; Local $gbmp = _GDIPlus_BitmapCreateFromHBITMAP($bmp) ; _WinAPI_DeleteObject($bmp) ; Local $gsbmp = _GDIPlus_ImageResize($gbmp, $w * $scale, $h * $scale) ;Local $ext = _GDIPlus_EncodersGetCLSID("PNG") ; _GDIPlus_ImageSaveToFileEx($gsbmp, $outimage, $sImgCLSID) ; _GDIPlus_BitmapDispose($gbmp) ; _GDIPlus_BitmapDispose($gsbmp) ; _GDIPlus_Shutdown() ;EndFunc
      Thanks for any help you can give me!!!