BotIt FindAndClick

[ScrapeYourself 0]

I didn't like the search time of Simple Native Image Search, and being on windows 10 64bit, I couldn't get ImageSearchDll.dll to work properly.

So I started researching image search routines and found this excellent post and set of replies find-a-bitmap-within-another-bitmap.

I really liked the pattern that the Simple Native Image Search used, the clipboard usage and the method of searching. Although I think it could be improved by using some short circuit techniques to return sooner, like consecutive matched > 65% return | matched total > 85% return) , and I wanted the function to manage the click on the found image as well.

So I just did a bit more research and a few trips to MSDN and stackoverflow, these two snippets allow me to replicate KyleJustKnows code, and click.  Another feature is that it also saves the image it captures to disk, so that if the image is not found you can check what was captured, and alternatively cut out a new search image to use.

private void PrintScreen()
        {
            keybd_event(VK_SNAPSHOT, 0, KEYEVENTF_EXTENDEDKEY, 0);
            keybd_event(VK_SNAPSHOT, 0, KEYEVENTF_KEYUP, 0);
        }

        public Bitmap CaptureScreenPrtSc()
        {

            PrintScreen();
            if (Clipboard.ContainsImage())
            {
                using (Image img = Clipboard.GetImage())
                {
                    img.Save("ClipBoard.PNG", ImageFormat.Png);
                    return new Bitmap(img);
                }
            }
            return default;
        }

        [DllImport("user32.dll")]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool GetCursorPos(out MousePoint lpMousePoint);

        [DllImport("user32.dll")]
        private static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo);

        private MousePoint GetCursorPosition()
        {
            var gotPoint = GetCursorPos(out MousePoint currentMousePoint);
            if (!gotPoint) { currentMousePoint = new MousePoint(0, 0); }
            return currentMousePoint;
        }

        private void MouseEvent(MouseEvents value)
        {
            MousePoint position = GetCursorPosition();

            mouse_event
                ((int)value,
                 position.X,
                 position.Y,
                 0,
                 0)
                ;
        }

So after managing to compile the dll with COM support, with much reading of this forum and many posts from paulpmeier, ptrex, LarsJ, and others about loading .net, I managed to get this all working.

Here is the BotIt Core:

;   BotIt Core


Global $sPath = "BotIt.dll"

Global $RegAsmPath = "C:\Windows\Microsoft.NET\Framework\v4.0.30319\RegAsm.exe"

Func BotIt_StartUp()
    RegisterBotIt()
    OnAutoItExitRegister("UnregisterBotIt")
EndFunc   ;==>BotIt_StartUp

Func RegisterBotIt()
    RunWait($RegAsmPath & " /register /codebase /tlb " & $sPath, @ScriptDir, @SW_HIDE)
EndFunc   ;==>RegisterBotIt

Func UnregisterBotIt()
    FileDelete("Step.txt")
    RunWait($RegAsmPath & " /unregister " & $sPath, @ScriptDir, @SW_HIDE)
EndFunc   ;==>UnregisterBotIt

Func ActivateAndSearch($sTitle, $sImgPath, $bClick = True)
    WinActivate($sTitle)
    Sleep(1000)
    $oBotIt = ObjCreate("BotIt.DetectImageAndClick")
    ConsoleWrite($sImgPath & @CRLF & $bClick & @CRLF)
    $bRet = $oBotIt.FindAndClick($sImgPath, $bClick)
    Return $bRet
EndFunc   ;==>ActivateAndSearch

 

Usage:

Do
        Sleep(500)
    Until ActivateAndSearch("Window Title", "PathToFile")

I hope you enjoy!

Regards,

ScrapeYourself
 

 

BotIt.cs

Edited December 25, 2019 by ScrapeYourself
spelling / wording

[Musashi 721]

This thread will be closed 99% anyway (forum rules), but that is not the reason for my contribution.

Warning : Never download and execute a .dll from an unknown source.

[ScrapeYourself 0]

After reading the rules this is not about botting a game, or has anything to do with a game.

rules as I see them.

• Malware of any form - trojan, virus, keylogger, spam tool, "joke/spoof" script, etc.
• Bypassing of security measures - log-in and security dialogs, CAPTCHAs, anti-bot agents, software activation, etc.
• Automation of software/sites contrary to their EULA (see Reporting bullet below).
• Launching, automation or script interaction with games or game servers, regardless of the game.
• Running or injecting any code (in any form) intended to alter the original functionality of another process.
• Decompilation of AutoIt scripts or details of decompiler software

This tool is not used for anything more then what it is intended to do..Find and click a image on the screen.

I provided source code for self compilation in first post.

 

 

Edited December 25, 2019 by ScrapeYourself

[Musashi 721]

 

9 minutes ago, ScrapeYourself said:

After reading the rules this is not about botting a game, or has anything to do with a game.

Fortunately, that's not my decision. I just wanted to inform (especially inexperienced) users not to start executables, if the origin is possibly uncertain .

[ScrapeYourself 0]

I still want to understand further, why you think 99 % it will be closed  

what can i do to change that?

[Musashi 721]

13 minutes ago, ScrapeYourself said:

I still want to understand further, why you think 99 % it will be closed  
what can i do to change that?

It is best to wait and see how a moderator judges this issue. If there are no objections, the thread will not be closed . But I have experienced in the past that threads about "Pixelsearch", "Imagesearch" and similar operations are considered very strictly.

Edited December 25, 2019 by Musashi

[JLogan3o13 1,761]

Locked while reviewed.

[Jos 2,811]

We leave this thread open for now, but make sure it will adhere to our forum rules.  

Jos

[ScrapeYourself 0]

Thank you Jos,

 

To clear any misunderstanding in my names of images searched, I am working on a deductions processing website as a side job. All parties are aware of automation.

One of the issues I experience isI search these 2 images in sequence to ensure I end on the SIS inquery tab and search.
 

If not Automation Scripts\SISInquiry.PNG then
    If Automation Scripts\PaymentInquiry.PNG then
        Automation Scripts\SISInquiry.PNG
    EndIf
EndIf

Automation Scripts\RegisterBotIt.au3" (56) : ==> Variable must be of type "Object".:

$bRet = $oBotIt.FindAndClick($sImgPath,    $Region, $bClick)
$bRet = $oBotIt^ ERROR

But it occassionally fails on the 2nd search with the error above, how can i detect that before it occurs, would IsObj Work?

 

My next question is what pattern could we use to compress the searched for image to a small set of coords and values to identify?

Edited December 30, 2019 by ScrapeYourself
spelling / wording

[water 2,703]

You need to do much more error checking!
Means: When you call a function that creates an object make sure, the result is an object. Most of the time this is being done by implementing a COM error handler (see ObjEvent in the help file).

For examples check the Excel or Word UDF.

[ScrapeYourself 0]

I will check it out, thank you.