Lasker

A new image search solution

18 posts in this topic

#1 ·  Posted (edited)

Hello,

Im a new AutoIt user, im from Spain and i would like to share here my own image search detection method.

What's about it?

It's not perfect, but with it you don't need to use any DLL file. You don't need to distribute the image file with your final AU3 Script, or Exe if compiled. The image size does not affect to the detection speed. It's bug free (i believe).

You can find here as an attach file, a distro package with all you need, and it includes a simple how-to guide.

Im going to paste here the how-to guide:

How to use this AutoIt image detector:

Point 1:

Install JRE (Java Runtime Environment), without this you can't execute Java Applications (.jar files). Open the Java tool named AISD with double click on it, or "java -jar aisd.jar" in the cmd. In AISD, open the image you want to detect in your script. Remember: the upper left pixel of the image, MUST be a not common color in the background if you want a fast detection. Then you will get a script file if you want, or only a clipboard copied version, full ready to paste into your main script because this one comes with extra help codes. Anyway, the generated code is the $sign definition of the image (calm down, point 3 explains what is the $sign and it's easy).

Point 2:

Paste the mentioned code in your main script, and try to follow the sample_usage script. If you know how to write au3 codes, it should be very easy. In general terms, all you need is to include image_search.au3, and then you can use the func image_search($sign, $width, $height). You need also the get_image_sign() function, that calculates the first parameter for image_search(). The get_image_sign() function is the one you have generated at point 1.

Point 3:

image_search() returns the center point of the target image. The func needs the $sign from the image, width and height. This $sign is the "footprint" of the image, in the form of an array of 9 integers. In most of the cases the $sign should be enough to identify the image, this is useful because using the $sign instead of the real image let you get a small script and a fast detection. You can also delete the image once you get the $sign. This $sign has always the same size (those 9 integers, it does not matter if the original image is small or big). How can you define the $sign for a certain image?, you don't need to generate the $sign, AISD does (point 1). Follow the sample_usage script and you will get it.

Files in this distro package:

* AISD.jar: The already explained Java app.

* image_search.au3: The library or module with the resources that lets you detect images.

* sample_usage.au3: The mentioned sample_usage script. It contains the smallest example, but i think it should be ok for learning purposes.

* sample_usage.exe: The sample_usage.au3 script compiled (au3 -> .exe). You can execute this to see the image_search() func working.

* sample_target.png: The image to be detected in the sample_usage. You can delete this file and the script will be fine.

* OffsetCalc.jar: It's only a Java tool to calculate the offset to transform an XY point into another one. It's useful if you want to use apply_offset function (from image_search.au3). Both things are only necessary when it's hard to find a version of the target image with upper left pixel in the way mentioned. As you can imagine, the idea is to detect another image and then apply the offset to get the real coordinates of the original target image.

Happy Detections

Lasker image_search.zip

Edited by Lasker
2 people like this

Share this post


Link to post
Share on other sites



Hi @Lasker Thank You very much for you sharing I tried it and found it very useful but why i cant detect another created "PNG" images I'm able to detect example image only !

 

Share this post


Link to post
Share on other sites

Thank you very much it's working now I forgot to paste copied code ..

It's very easy to use thank you again for making Image search Easier but is this working in other Pc resolution ?

Share this post


Link to post
Share on other sites

@Lasker - just popped in to thank for sharing your solution. Well documented, easy to use and lightning fast. Good job mate!

Share this post


Link to post
Share on other sites

@Lasker - thank you for this, I tested and it's clearly working.

Two questions please:

1. Does your image search support color variations? (e.g. imagesearch.au3 has smth like this - it can find image even if not perfectly same colors, maybe big image is from lower quality jpg etc).

2. Is it possible to find different sizes of the same image/pattern? This would be indeed great, imagesearch.au3 only solution is to have multiple target images.

I assume your 9 signs could be multiplied by the number of sizes to search, but not sure of speed then.

Thanks anyway

1 person likes this

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Hola :) Nice, going to test, I would like to have image search which can scale images, so no worry about browser zoom  or different screen resolutions :)
Also like autog ask, tolerance would be great :)
What about images with transparent parts, where part of image can be a bit different as they have ex animated background ?
Thx anyway, great idea and good job man, best greetings from Poland :)

Edited by WiValdiBB

Share this post


Link to post
Share on other sites

Ok first of all it does not support dual screens. i used 2 screens with extrended view and it could not find the logo.I suspected it was because of 2 screens so i ddisconnected one and now it wiorks. But its kinda slow the image i used was a red button and it was a black background so I thought it would go faster if i had a red    pixel in the left uper corner but NO still slow..

 

 

Is it possible to scan an determined surface??

Share this post


Link to post
Share on other sites

Is this checked by anyone for effectivity (greater than imagesearch library?) Thanks

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

On ‎18‎/‎05‎/‎2016 at 2:24 PM, Lasker said:

 

Can you direct me on where to do some research on methods to find "uncommon", or even better, "unique" pixel in an image sample? Thank you in advance.

Edited by Melba23
Huge quote removed

Share this post


Link to post
Share on other sites

#10 ·  Posted

r4sc4l,

Welcome to the AutoIt forums.

But when you post in future, please use the "Reply to this topic" button at the top of the thread or the "Reply to this topic" editor at the bottom rather than the "Quote" button - others know what they wrote and it just pads the thread unnecessarily.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#11 ·  Posted

this doesn't work for me.

Share this post


Link to post
Share on other sites

#12 ·  Posted

@dwaynek "It's broke" doesn't help anyone help you with your issue. How about a detailed explanation as to what you tried, and what is or is not working, including screenshots, error codes, etc.?


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

Hi,

I certainly can't get this working at all. I'm on Windows 10 and it throws up saying it failed to find the image.

I've tried it with the website open on every screen and I've also replaced the image with a fresh copy of the logo yet it still fails to find it... findBMP & ImageSearch both work just I want to try this one to see how much faster/slower it is and how resource intensive it is.

Edited by Hyflex

Share this post


Link to post
Share on other sites

#14 ·  Posted

2 hours ago, Hyflex said:

findBMP & ImageSearch both work just I want to try this one to see how much faster/slower it is and how resource intensive it is.

If it doesn't work, then I guess it won't work for you to compare against, try something else or figure out why this one doesn't work.

No one is going to be able to help you because you've given absolutely zero information. Imagine you were trying to figure out how to help you with the total lack of information received, then you will realize what your post actually tells us.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

#15 ·  Posted

2 minutes ago, BrewManNH said:

If it doesn't work, then I guess it won't work for you to compare against, try something else or figure out why this one doesn't work.

No one is going to be able to help you because you've given absolutely zero information. Imagine you were trying to figure out how to help you with the total lack of information received, then you will realize what your post actually tells us.

How can I possibly give more information... I've told you everything... what else would you require?

I gave you my OS, I told you that I tried the example script, I told you that I tried the example script with a fresh image. I told you what didn't work / what the result was.

Share this post


Link to post
Share on other sites

#16 ·  Posted

You told us, you're on Windows 10 and that it didn't find the image.

You said nothing about which script you used to find the image, or what error messages (if any) were returned. NOW you're telling us that you used the example script. Did you install the JRE?


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

#17 ·  Posted

12 hours ago, BrewManNH said:

You told us, you're on Windows 10 and that it didn't find the image.

You said nothing about which script you used to find the image, or what error messages (if any) were returned. NOW you're telling us that you used the example script. Did you install the JRE?

I told you there error it throws...

15 hours ago, Hyflex said:

and it throws up saying it failed to find the image.

From this you can see that it has failed an if/else argument and that also tells you that JRE is installed otherwise it would have thrown an exception which also tells you that JRE is installed otherwise it would have thrown an exception.

I also told you that I double checked the images were correct by replacing the image with a new copy just incase it was unable to find due to having slighty different desktop resolution/brightness... etc.

I also told you that I am capable of using both findBMP & ImageSearch therefore I probably checked most of the basic reasons that it might be failing...

Share this post


Link to post
Share on other sites

#18 ·  Posted

Great share. But this doesn't work for me when trying the other images

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • DeathChicken
      By DeathChicken
      If possible please add or edit the comments to explain how this works.
       
      ;includes functions from other things
      #include <GDIPlus.au3>
      #include <ScreenCapture.au3>
      ;hotkeys
      HotKeySet("{ESC}", _exit)
      HotKeySet("{F1}", _scan)
      ;global variables
      Global $win_title                    ;name of the window
      Global $area_x                        ;
      Global $area_y                        ;
      Global $area_w                       ;
      Global $area_h                        ;
      Global $cursor                         ;
      Global $rect_file                      ;
      Global $hbmpscreen              ;
      Global $i=  1043                      ;moves the rect
      Global $ii=    378                      ;moves the rect
      Global $x=1044                        ;inner rect offset
      Global $y=501                          ;inner rect offset
      Global $hbmprect                    ;image inside rect
      Global $hscreen                        ;image whole screen
                                         
      _GDIPlus_Startup()                    ;?
      ;display hotkeys on screen
      ToolTip("Press F1 to scan | Press ESC to Exit",0,0)
                                          ;infinite loop to keep prog running
      While 1
          Sleep(100)
      WEnd
      Func _scan()
      ;~ reads your screen area:
      _read()
      ;~ converts screen captured into bmp
      _convert()
      ;~ .  Loads converted bmp to be read
      _loadBMP()
      ;~ compares the bmp of your scanned screen to the actual screen that is being displayed
      _compare()
      EndFunc
                                          ;reads screen
      Func _read()
          $hscreen = _ScreenCapture_CaptureWnd("", WinGetHandle($win_title), $area_x, $area_y, $area_x + $area_w, $area_y + $area_h, $cursor)
      EndFunc
                                          ;converts screen into bmp
      Func _convert()
          _GDIPlus_BitmapCreateFromHBITMAP($hscreen)
      EndFunc
                                          ;loads converted screen
      Func _loadBMP()
      _GDIPlus_BitmapCreateFromFile($rect_file)
      EndFunc
                                          ;compares savedBMP to current screen
      Func _compare()
      ;If(                              below code                                     ) = (                   below code           ) Then
      If _GDIPlus_BitmapGetPixel($hbmpscreen, ($i + $x) - $area_x, ($ii + $y) - $area_y) = _GDIPlus_BitmapGetPixel($hbmprect, $x, $y) Then
      ;display message box titled found with a message of found
      MsgBox("","","found")
      ;if above is not correct then
      EndIf
      EndFunc
                                          ;exit func
      Func _exit()
          Exit
      EndFunc
    • GordonFreeman
      By GordonFreeman
      Hi, i searched some topics but cannot found a simple way to do that:
      #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> $Form1 = GUICreate("Form1", 615, 438, 192, 124) GUISetState(@SW_SHOW) ; I tried this but user reported that it work in 2006, not seems work actually $filemenu = GUICtrlCreateMenu ("&File") $fileitem = GUICtrlCreateMenuitem ("Open",$filemenu) GuiCtrlSetImage($fileitem, "shell32.dll", 4) While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit EndSwitch WEnd Also tried:
      #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> $Form1 = GUICreate("Form1", 615, 438, 192, 124) GUISetState(@SW_SHOW) $filemenu = TrayCreateMenu("&File") $fileitem = TrayCreateItem("Open",$filemenu) ;GuiCtrlSetImage($fileitem, "shell32.dll", 4) ;GuiCtrlSetImage(TrayItemGetHandle($fileitem), "shell32.dll", 4) While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit EndSwitch WEnd Well.. thanks in advance!
       
      Ok. I found this working example from Yashied. If i cannot find a simple then i will go with it
      #Include <GUIConstantsEx.au3> #Include <GUIMenu.au3> #Include <Constants.au3> #Include <WinAPI.au3> #Include <WindowsConstants.au3> Opt('MustDeclareVars', 1) Global $hMenu, $hForm, $hFile = 1000, $idNew, $idExit $hForm = GUICreate('Menu', 400, 300) $hFile = _GUICtrlMenu_CreateMenu() _GUICtrlMenu_InsertMenuItem ($hFile, 0, ' &Favorites', $idNew) _GUICtrlMenu_InsertMenuItem ($hFile, 1, '', 0) _GUICtrlMenu_InsertMenuItem($hFile, 2, ' E&xit', $idExit) $hMenu = _GUICtrlMenu_CreateMenu() _GUICtrlMenu_InsertMenuItem($hMenu, 0, '&File', 0, $hFile) _GUICtrlMenu_SetMenu($hForm, $hMenu) _GUICtrlMenu_SetItemBmp($hFile, 0, _CreateBitmapFromIcon(_WinAPI_GetSysColor($COLOR_MENU), @SystemDir & '\shell32.dll', 43, 16, 16)) _GUICtrlMenu_SetItemBmp($hFile, 2, _CreateBitmapFromIcon(_WinAPI_GetSysColor($COLOR_MENU), @SystemDir & '\shell32.dll', 27, 16, 16)) GUISetState() Do Until GUIGetMsg() = $GUI_EVENT_CLOSE Func _CreateBitmapFromIcon($iBackground, $sIcon, $iIndex, $iWidth, $iHeight) Local $hDC, $hBackDC, $hBackSv, $hIcon, $hBitmap $hDC = _WinAPI_GetDC(0) $hBackDC = _WinAPI_CreateCompatibleDC($hDC) $hBitmap = _WinAPI_CreateSolidBitmap(0, $iBackground, $iWidth, $iHeight) $hBackSv = _WinAPI_SelectObject($hBackDC, $hBitmap) $hIcon = _WinAPI_PrivateExtractIcon($sIcon, $iIndex, $iWidth, $iHeight) If Not @error Then _WinAPI_DrawIconEx($hBackDC, 0, 0, $hIcon, 0, 0, 0, 0, $DI_NORMAL) _WinAPI_DestroyIcon($hIcon) EndIf _WinAPI_SelectObject($hBackDC, $hBackSv) _WinAPI_ReleaseDC(0, $hDC) _WinAPI_DeleteDC($hBackDC) Return $hBitmap EndFunc ;==>_CreateBitmapFromIcon Func _WinAPI_PrivateExtractIcon($sIcon, $iIndex, $iWidth, $iHeight) Local $hIcon, $tIcon = DllStructCreate('hwnd'), $tID = DllStructCreate('hwnd') Local $Ret = DllCall('user32.dll', 'int', 'PrivateExtractIcons', 'str', $sIcon, 'int', $iIndex, 'int', $iWidth, 'int', $iHeight, 'ptr', DllStructGetPtr($tIcon), 'ptr', DllStructGetPtr($tID), 'int', 1, 'int', 0) If (@error) Or ($Ret[0] = 0) Then Return SetError(1, 0, 0) EndIf $hIcon = DllStructGetData($tIcon, 1) If ($hIcon = Ptr(0)) Or (Not IsPtr($hIcon)) Then Return SetError(1, 0, 0) EndIf Return $hIcon EndFunc ;==>_WinAPI_PrivateExtractIcon Ok, i noticed that is a pain create submenu with _GUICtrlMenu_CreatePopup to do like in Tray, and Tray better located, then i need images/icon with Tray functions, some workaround?? 
    • algiuxas
      By algiuxas
      Hello everybody, so I might found a bug in _ArraySearch:
       
      #include <Array.au3> Local $abc[2] = ["b", "b"] $h = "+---------------------------------+" & @CRLF ConsoleWrite($h) For $i = 0 To 1 $abc[0] = 0 out() $abc[0] = 1 out() $abc[0] = -1 out() $abc[0] = "Abc" out() $abc[1] = "Hello" Next Func txt() Return "$abc = [" & $abc[0] & ", " & $abc[1] & "]" & @CRLF & _ "> _ArraySearch($abc,""Hello"") = " EndFunc ;==>txt Func shouldoutput($ans) Return "_ArraySearch Should return: " & ($ans ? 1 : -1) EndFunc ;==>shouldoutput Func out() ConsoleWrite("> " & txt() & _ArraySearch($abc, "Hello") & @CRLF & "> " & shouldoutput($abc[1] == "Hello") & @CRLF & $h) EndFunc ;==>out Exit Sorry for this messy script
    • SimTheNo1
      By SimTheNo1
      I am busy with building a solution for change monitoring of VOIP call program and to be properly automated means among others need for some limited OCR functionality. Current works of others are way too much overkill for this case what makes the need to build it myself. But to do it properly I significantly have to increase my knowledge about digital graphics management.
      For now I already have discovered here and there some mind blowing  Autoit miracles what can be achieved with Windows own possibilities to manipulate that what is output to the monitor. And as far I can judge there are 2 options to process graphics without use of any external libraries like ImageMagick, FreeImage and so on. These are:
       WinAPI  GDIPlus It is for me quite obvious to have various holes in general understanding of graphics and it is once more very clear what advantages gives consistent general study in 1 or another official institutions like University.  Cause there you are introduced into certain domain of knowledge in a way which usually has been perfected over long period of time. So you are not overloaded by stuff which requires a certain amount of information to be initially clear for you. For example, before starting to solve physics you first learn to read, count and so on and then move to subjects like physics.
      Though in my case opportunity to study in such educational system I had only for 8 years, from my 7th to 15th year of age in the country that was falling apart now Ukraine but used to be USSR, was all what it was. After have immigrated to Netherlands possibilities to study further haven't occur. And this therefor causes often various implications when going deep in that or another field of practical knowledge acquiring for any needed physical result, like programming to perform enormous amount of tasks. In this particular case automating VOIP call program.
      Anyway, right now I think the best direction to move is to concentrate on as basic as possible image management and if someone would maybe explain in general what is a pixel will definitely help. Particularly I am very curious about how to do picture manipulations in Autoit. Especially would help a lot to produce eventually following functions:
      createImage($imageFileName, $width, $height, $color) readImagePixel($imageFileName, $x, $y)  writeImagePixel($imageFileName, $x, $y) I do not know exactly how image is handled in computer but preferably above mentioned functions should deal with so far possible origin of graphics creation on computer. But nevertheless I definitely would love to hear any proposition for solution.
      The problem with explaining screenshots:
       VOIPConnect full Window  Part of Window with control to monitor for changes  Exact location of area where actual changes occur and have to be processed  It comes down to a rectangle of approximately 51 pixel wide and 7 pixel high. In fact if I get to learn as far as to be able exactly read, write and compare 2 images consistently across different computers I could narrow down then even further the area to watch as little as a square of 2-5 pixels wide. 
      To finish here is last detail about particularly no need for ultra fast solution at all. This because it is needed only once when it is first run on a new computer and to have to wait few minutes while it is being set instead of just a few seconds make no sence.
      This is it and what I too think to do beside this very particular case is to purify out beautiful generic Autoit functions for core image manipulation by using WinAPI or/and GDIPlus.