Sign in to follow this  
Followers 0
SaeidN

ImageSearch and MouseDrag in background window

30 posts in this topic

Hi,

I saw a couple topics about both mousedrag and imagesearch in background or minimized window, but none of them has the solution, and they were old topics.

So I wanted to ask whether there is a way to mouse drag without the cursor to move, (I want to use mouse when mousedrag is working on a window). I don't know whether I can mix ControlMove with some other stuff or any other way?

And also for the image search, is there any way to search for image (or pixel) in a background or minimized window? I thought maybe there is a au3 or dll or ... file for doing this.

Another question I have: Any way to pinch zoom (two fingers) a window like "windows photo viewer" to zoom in or out? I saw something almost the same here, but not what I want.

I appreciate it.

Share this post


Link to post
Share on other sites



If I remember correctly there's an alternate ImageSearch function (in the dll) that takes a handle to a bitmap, the bitmap you can get by using CaptureScreen. Then you can use SendMessage to send

  1. $WM_LBUTTONDOWN,
  2. $WM_MOUSEMOVE
  3. $WM_LBUTTONUP
#RequireAdmin
Func CaptureWindow($iLeft = 0, $iTop = 0, $iWidth = -1, $iHeight = -1, Const $bSaveImage = False, Const $hWnd = WinGetHandle("[Active]"))
    Local $rect_window = WinGetPos($hWnd)
    Local $hDC = _WinAPI_GetWindowDC($hWnd)
    Local $hDestDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBitmap = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hDestSv = _WinAPI_SelectObject($hDestDC, $hBitmap)
    Local $hSrcDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBmp = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hSrcSv = _WinAPI_SelectObject($hSrcDC, $hBmp)

    _WinAPI_PrintWindow($hWnd, $hSrcDC, True)

    ScreenToClient($hWnd, $iLeft, $iTop)

    If ($iWidth > 0 and $iHeight > 0) Then
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $iWidth, $iHeight, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    Else
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $rect_window[2] - 16 - $iLeft, $rect_window[3] - 38 - $iTop, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    EndIf

    _WinAPI_SelectObject($hDestDC, $hDestSv)
    _WinAPI_SelectObject($hSrcDC, $hSrcSv)
    _WinAPI_ReleaseDC($hWnd, $hDC)
    _WinAPI_DeleteDC($hDestDC)
    _WinAPI_DeleteDC($hSrcDC)
    _WinAPI_DeleteObject($hBmp)

    If ($image_search_tolerance < 50) Then $image_search_tolerance += 50
    If ($image_search_tolerance > 80) Then $image_search_tolerance = 80

    If ($bSaveImage) Then _ScreenCapture_SaveImage(@ScriptDir & "\CaptureWindow.jpg", $hBitmap, False)

    Return $hBitmap
EndFunc   ;==>CaptureWindow
DllCall("ImageSearchDll.dll", "str", "ImageSearchExt", "int", $iLeft, "int", $iTop, "int", $iRight, "int", $iBottom, "int", $iTolerance, "ptr", $hHbitmapImageToFind, "ptr", $hHbitmapImageToSearch)

Should get you started. Just put the DLLCall in your ImageSearch function. $hHBitmapImageToSearch is gonna be your CaptureWindow and $hHbitmapImage is a handle to a bitmap of the image to look for

1 person likes this

Share this post


Link to post
Share on other sites
9 hours ago, InunoTaishou said:

If I remember correctly there's an alternate ImageSearch function (in the dll) that takes a handle to a bitmap, the bitmap you can get by using CaptureScreen. Then you can use SendMessage to send

  1. $WM_LBUTTONDOWN,
  2. $WM_MOUSEMOVE
  3. $WM_LBUTTONUP
#RequireAdmin
Func CaptureWindow($iLeft = 0, $iTop = 0, $iWidth = -1, $iHeight = -1, Const $bSaveImage = False, Const $hWnd = WinGetHandle("[Active]"))
    Local $rect_window = WinGetPos($hWnd)
    Local $hDC = _WinAPI_GetWindowDC($hWnd)
    Local $hDestDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBitmap = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hDestSv = _WinAPI_SelectObject($hDestDC, $hBitmap)
    Local $hSrcDC = _WinAPI_CreateCompatibleDC($hDC)
    Local $hBmp = _WinAPI_CreateCompatibleBitmap($hDC, $rect_window[2] - 16, $rect_window[3] - 38)
    Local $hSrcSv = _WinAPI_SelectObject($hSrcDC, $hBmp)

    _WinAPI_PrintWindow($hWnd, $hSrcDC, True)

    ScreenToClient($hWnd, $iLeft, $iTop)

    If ($iWidth > 0 and $iHeight > 0) Then
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $iWidth, $iHeight, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    Else
        _WinAPI_BitBlt($hDestDC, $iLeft, $iTop, $rect_window[2] - 16 - $iLeft, $rect_window[3] - 38 - $iTop, $hSrcDC, $iLeft, $iTop, $MERGECOPY)
    EndIf

    _WinAPI_SelectObject($hDestDC, $hDestSv)
    _WinAPI_SelectObject($hSrcDC, $hSrcSv)
    _WinAPI_ReleaseDC($hWnd, $hDC)
    _WinAPI_DeleteDC($hDestDC)
    _WinAPI_DeleteDC($hSrcDC)
    _WinAPI_DeleteObject($hBmp)

    If ($image_search_tolerance < 50) Then $image_search_tolerance += 50
    If ($image_search_tolerance > 80) Then $image_search_tolerance = 80

    If ($bSaveImage) Then _ScreenCapture_SaveImage(@ScriptDir & "\CaptureWindow.jpg", $hBitmap, False)

    Return $hBitmap
EndFunc   ;==>CaptureWindow
DllCall("ImageSearchDll.dll", "str", "ImageSearchExt", "int", $iLeft, "int", $iTop, "int", $iRight, "int", $iBottom, "int", $iTolerance, "ptr", $hHbitmapImageToFind, "ptr", $hHbitmapImageToSearch)

Should get you started. Just put the DLLCall in your ImageSearch function. $hHBitmapImageToSearch is gonna be your CaptureWindow and $hHbitmapImage is a handle to a bitmap of the image to look for

I included <ScreenCapture.au3> and <WinAPIEx.au3>. But still giving me couple error here.

I appreciate if you would show me where to get imagesearch dll file.

And also is $WM_LBUTTONDOWN and 2 others for mousedrag?

error.JPG

Share this post


Link to post
Share on other sites
#include <ScreenCapture.au3>
#include <WinAPI.au3>
#include <WindowsConstants.au3>

Func ScreenToClient(Const ByRef $hWnd, ByRef $iX, ByRef $iY)
    Local $tReturn = DLLStructCreate("int iX;int iY")
    DllStructSetData($tReturn, 1, $iX)
    DllStructSetData($tReturn, 2, $iY)
    
    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $hWnd, "ptr", DllStructGetPtr($tReturn))
    
    $iX = DllStructGetData($tReturn, 1)
    $iY = DllStructGetData($tReturn, 2)
    $tReturn = Null
EndFunc   ;==>ClientToScreen

Func ClientToScreen(Const ByRef $hWnd, ByRef $iX, ByRef $iY)
    Local $tReutrn = DllStructCreate("int iX;int iY")
    DllStructSetData($tReturn, 1, $iX)
    DllStructSetData($tReturn, 2, $iY)
    
    DllCall("user32.dll", "int", "ClientToScreen", "hwnd", $hWnd, "ptr", DllStructGetPtr($tReturn))
    
    $iX = DllStructGetData($tReturn, 1)
    $iY = DllStructGetData($tReturn, 2)
    $tReturn = Null
EndFunc   ;==>ClientToScreen

Those windows messages (WM_*) variables send a message to the window telling it that the mouse has done that action (left button down, move, left button up) which is a mouse drag

Share this post


Link to post
Share on other sites
9 hours ago, InunoTaishou said:
#include <ScreenCapture.au3>
#include <WinAPI.au3>
#include <WindowsConstants.au3>

Func ScreenToClient(Const ByRef $hWnd, ByRef $iX, ByRef $iY)
    Local $tReturn = DLLStructCreate("int iX;int iY")
    DllStructSetData($tReturn, 1, $iX)
    DllStructSetData($tReturn, 2, $iY)
    
    DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $hWnd, "ptr", DllStructGetPtr($tReturn))
    
    $iX = DllStructGetData($tReturn, 1)
    $iY = DllStructGetData($tReturn, 2)
    $tReturn = Null
EndFunc   ;==>ClientToScreen

Func ClientToScreen(Const ByRef $hWnd, ByRef $iX, ByRef $iY)
    Local $tReutrn = DllStructCreate("int iX;int iY")
    DllStructSetData($tReturn, 1, $iX)
    DllStructSetData($tReturn, 2, $iY)
    
    DllCall("user32.dll", "int", "ClientToScreen", "hwnd", $hWnd, "ptr", DllStructGetPtr($tReturn))
    
    $iX = DllStructGetData($tReturn, 1)
    $iY = DllStructGetData($tReturn, 2)
    $tReturn = Null
EndFunc   ;==>ClientToScreen

Those windows messages (WM_*) variables send a message to the window telling it that the mouse has done that action (left button down, move, left button up) which is a mouse drag

I'm sorry, I'm not getting it. I know what it means to send message to a window to do a mouse action, but I don't know how to do it. Would you please give me a simple example (maybe with paint?), so I read it and understand what's happening?

Thank you

Share this post


Link to post
Share on other sites

There are plenty of examples of how to use SendMessage on the forums and in the help document
 

You can check that topic to see how they used SendMessage to click on a window. Modify it to do what you need.

Share this post


Link to post
Share on other sites
3 hours ago, InunoTaishou said:

There are plenty of examples of how to use SendMessage on the forums and in the help document
 

You can check that topic to see how they used SendMessage to click on a window. Modify it to do what you need.

I have checked https://www.autoitscript.com/autoit3/docs/libfunctions/_SendMessage.htm and it helped me understand better, but I don't know what are the codes in front of variables in the example and how can I obtain them. Like these two: 

 Local Const $WM_SYSCOMMAND = 274
    Local Const $SC_MONITORPOWER = 61808

So far I have checked two other topics and no result yet, but I'll search more.

Share this post


Link to post
Share on other sites

Okay. I read topics below . I really do not know what's wrong with the code in that topic. There were two other topics that doesn't give any clue at all. The codes in these topics don't give any error. I wish it was giving me error so I know what to fix.

I'm so tired of searching the whole net for almost 10 days. I also found this file, which is a good resource to send key and mouse message to a window, but again.

Doesn't work. Maybe because it's old or my win is 64bit.

 

 

 

PostMessage.au3

Share this post


Link to post
Share on other sites

I changed

$result = DllCall($h_ImageSearchDLL, "str", "ImageSearch", "int", $x1, "int", $y1, "int", $right, "int", $bottom, "str", $findImage)

to

DllCall("ImageSearchDll.dll", "str", "ImageSearchExt", "int", $iLeft, "int", $iTop, "int", $iRight, "int", $iBottom, "int", $iTolerance, "ptr", $hHbitmapImageToFind, "ptr", $hHbitmapImageToSearch)

I also have my screencapture as $hHbitmapImageToFind and the image to search as $hHbitmapImageToSearch.

But didn't work, anything else I need to change?

I wish I could have a simple example, so I could look and see what's happening. I literally searched the whole net, and didn't find an example, even in other languages other than Eng.

How can I know the difference between ImageSearchEx and ImageSearchExt, and ImageSearch. How Do I know how many parameters they should have?

What's the difference between str and ptr, Are they string and pointer? Why do we need pointer?

Share this post


Link to post
Share on other sites

#10 ·  Posted

Ayone knows?

I'm tired of searching and reading the tutorials. It's been two weeks now I'm looking for a solution and every day I'm spending more than 6 hours to find something, and no result yet. Please help.

Share this post


Link to post
Share on other sites

#11 ·  Posted

Most serious apps don't need ImageSearch, 99,9% of them can be automated with better AutoIt funcs or UDF's. So what's the name of the app. And also please post a runable (reproducer) script showing what you alreadey tried and showing the issues.

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

1 hour ago, AutoBert said:

Most serious apps don't need ImageSearch, 99,9% of them can be automated with better AutoIt funcs or UDF's. So what's the name of the app. And also please post a runable (reproducer) script showing what you alreadey tried and showing the issues.

I already wrote about 900 lines of codes, and it works perfect. I just want to do the imagesearch for inactive windows too. I wanna write some programs and scripts for our club and other events in univesity. The imageseach and mousedrag that I have now, which is only for active windows is not an efficient way to present as a project. What other UDFs or funcs works better for doing these in inactive windows?

If I see a small example in Paint or any other app for both imagesearch and mousedrag, then I can implement it in my script.

I appreciate your help.

Edited by SaeidN

Share this post


Link to post
Share on other sites

#13 ·  Posted

12 minutes ago, SaeidN said:

I already wrote about 900 lines of codes, and it works perfect.

Fine than mark as solved :)

But seems ImageSearch make problems so name the app to find a better way.

Share this post


Link to post
Share on other sites

#14 ·  Posted

Just now, AutoBert said:

Fine than mark as solved :)

But seems ImageSearch make problems so name the app to find a better way.

I didn't get them to work in inactive window, so it's not solved.

It's not a commercial app or something, I wrote it and it doesn't have any name. I just need an example which I don't think it would be more than 20 lines.

Share this post


Link to post
Share on other sites

#15 ·  Posted

Is it written with AutoIt? Than it's very easy to find a solution.

Share this post


Link to post
Share on other sites

#16 ·  Posted

29 minutes ago, AutoBert said:

Is it written with AutoIt? Than it's very easy to find a solution.

Yes, it's in Autoit. I wish I knew how to do it. :(

Share this post


Link to post
Share on other sites

#17 ·  Posted

Use Au3Info and the control* functions.

Share this post


Link to post
Share on other sites

#18 ·  Posted

1 minute ago, AutoBert said:

Use Au3Info and the control* functions.

But that's just for click or send keys. i don't know how is it for drag and searchimage an inactive window.

Share this post


Link to post
Share on other sites

#19 ·  Posted

You can also use ControlMove to move the controls in your background GUI.

Share this post


Link to post
Share on other sites

#20 ·  Posted

12 minutes ago, AutoBert said:

You can also use ControlMove to move the controls in your background GUI.

But it doesnt move the mouse, click, drag in an inactive window. and also imagesearh.

It if does mousedrag and imagesearch in an inactive window so please give me an example to see how it is working.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • Miliardsto
      ImageSearch Dll not working on 32bit
      By Miliardsto
      Hello i got problem with ImageSearchDll.dll
      It works for me perfect always I made program which one use a lot of ImageSearch funcs on 64bit.
      I got 64bit and i copied to c:/Windows - ImageSearchDll.dll 64bit version
      But I encountered some friends got 32bit OS and then if I compile program for 32bit then they put ImageSearchDll.dll 32bit ver on windows dir and program crash when function is call.
      it happens always :/
       
      How to run this func on 32bit?
    • computergroove
      ImageSearch is giving me an error message when I try to compile
      By computergroove
      I am trying to integrate imagesearch into a script I am making. When I try to compile the script I get the following inside imagesearch.au3 under all 4 of the imagesearch.au3 defined functions:
      "Error: Missing separator character before keyword"
      I tried the beta version of Autoit and it is doing the same thing. I am using the x32 version of the imagesearch.au3 found here - https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiUmsrEsLzPAhWs7YMKHS9MClQQFggfMAA&url=https%3A%2F%2Fwww.autoitscript.com%2Fforum%2Ftopic%2F148005-imagesearch-usage-explanation%2F&usg=AFQjCNFwpFnZMxVK9H4cDASRnjiB-65gOw&bvm=bv.134495766,d.amc
      I have put the imagesearchDLL.dll in both the includes directory and the system32 directory. I am using x64 Windows 8.1 Pro. I cant see anything wrong. Please help.
    • Reizvoller
      ImageSearch Script Closing w/ Error
      By Reizvoller
      Hello,
      I am back once again to ask for assistance. I am just trying to fool around with the "ImageSearch" function in order to detect when certain elements appear on my active window to then respond to. To begin testing this I simply cut out a small segment of my background, saved it as a BMP, and used that as the bmp for the search function. I had hoped for it to just run through quickly and say it found it and spit out the X Y coords of where it was found. Instead I am getting this : http://screencast.com/t/MD9aOzPhETE
      Here's my script thus far :
      #Include <msgBoxConstants.au3> #Include <ImageSearch.au3> global $x = 0 global $y = 0 CheckForImage () Func checkForImage () Local $Search = _ImageSearchArea ('ninjaCheck.bmp', 1, 0, 0, 1366, 768, $x, $y, 100) If $Search = 1 Then MsgBox ($IDOK,"Found",$resultPosition) EndIf EndFunc While 1 Sleep (10) WEnd What I would like assistance with is how do I "Debug" this in order to see where/why it is failing.
      I've reviewed https://www.autoitscript.com/wiki/FAQ#How_can_I_debug_my_script.3F but I am not entirely sure how to apply the information. If I could get a return on "Hey, this Func or This Line is where I died" , that would be awesome.
      Any suggestions on how I can tackle this issue?
      Thank you for your time!
      -Reiz
       
      checkNinja.bmp is " " around it now and it has the same result 
    • FMS
      error in returning valeus
      By FMS
      Hello,
      I'm rather new at Autoit (picked it up again afther a couple of years) but i dont know what I'm doing wrong here.
      I hope somebody can help me making this script "better readeble" and explain what I'm doing wrong here.
      The thing that i want to do here is making a tool that helps me whit doing mine work.
      Also want to do the "ping" command in the background whit only a msgbox if ok or not.
      In the code I'm writing now I'm using Send commands but know that there is a better way for this.
      At this point there is a error also whish i do not udnerstand where it is comming from.
      the error i get is:
      MsgBox($MB_SYSTEMMODAL, "Error", " Error no IP or name is filled in")
      MsgBox(^ ERROR
      Could somebody help me whit looking at this code?
       
      #include <ButtonConstants.au3> #include <EditConstants.au3> #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> $Form1 = GUICreate("Form1", 385, 112, 192, 124) $iIP = GUICtrlCreateInput("", 80, 16, 209, 21) $btn_ping = GUICtrlCreateButton("Ping", 24, 56, 89, 25) $btn_vnc = GUICtrlCreateButton("VNC", 136, 56, 89, 25) $btn_evr = GUICtrlCreateButton("Eventvieuwer", 248, 56, 89, 25) GUISetState(@SW_SHOW) Func check_input() If GUICtrlRead($iIP) = "" Then Return False Else Return True Endif EndFunc While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $btn_ping $check = check_input() If ($check = False) Then Run ("cmd.exe") WinWaitActive("C:\WINDOWS\system32\cmd.exe") Send ("ping " & GUICtrlRead($iIP) & "{ENTER}") WinClose ("C:\WINDOWS\system32\cmd.exe") ;if ping OK then msgbox OK else NOK Else MsgBox($MB_SYSTEMMODAL, "Error", " Error no IP or name is filled in") EndIf Case $btn_vnc $check = check_input() If ($check = False) Then Run ("cmd.exe") WinWaitActive("C:\WINDOWS\system32\cmd.exe") Send ("eventvwr.exe " & GUICtrlRead($iIP) & "{ENTER}") WinClose ("C:\WINDOWS\system32\cmd.exe") Else MsgBox($MB_SYSTEMMODAL, "Error", " Error no IP or name is filled in") EndIf Case $btn_evr $check = check_input() If ($check = False) Then Run ("cmd.exe") WinWaitActive("C:\WINDOWS\system32\cmd.exe") Send ("vncviewer " & GUICtrlRead($iIP) & "{ENTER}") WinClose ("C:\WINDOWS\system32\cmd.exe") Else MsgBox($MB_SYSTEMMODAL, "Error", " Error no IP or name is filled in") EndIf EndSwitch WEnd  
    • avidovi
      ImageSearch error
      By avidovi
      Hello friends,
      i need your help and advice regarding the image search function.
      I need to click an image on a web site. after using the image search function i am getting the following error, on the image search function
      if $result[0]="0" then return 0 if $result^ ERROR I am using the x64 configuration with the following files attached
      thanks for helping..
      ImageSearch 64 bit.rar