Jump to content

comunicating with the screen


Recommended Posts

all i know about macros i learned from actool.

in actool there is a thing called an object. an object is a large bulk of code that is used to describe a graphical thing.

objects are used in actool in order to make sure you're not clicking an empty space and the button you're supposed to be clicking is indeed there.

i already sort of mastered the moving mouse, clicking, sleep, so now i feel i'm ready for the less noobish stuff.

is there a thing in autoit that's similar to the actool object, and if there is, how do i use it?

Edited by GodForsakenSoul
Link to comment
Share on other sites

all i know about macros i learned from actool.

in actool there is a thing called an object. an object is a large bulk of code that is used to describe a graphical thing.

objects are used in actool in order to make sure you're not clicking an empty space and the button you're supposed to be clicking is indeed there.

i already sort of mastered the moving mouse, clicking, sleep, so now i feel i'm ready for the less noobish stuff.

is there a thing in autoit that's similar to the actool object, and if there is, how do i use it?

In AC Tools it looks like an object is just a definition of some pixel colours at various relative coordinates. The same thing can be done using PixelSearch in AutoIt.

Serial port communications UDF Includes functions for binary transmission and reception.printing UDF Useful for graphs, forms, labels, reports etc.Add User Call Tips to SciTE for functions in UDFs not included with AutoIt and for your own scripts.Functions with parameters in OnEvent mode and for Hot Keys One function replaces GuiSetOnEvent, GuiCtrlSetOnEvent and HotKeySet.UDF IsConnected2 for notification of status of connected state of many urls or IPs, without slowing the script.
Link to comment
Share on other sites

In AC Tools it looks like an object is just a definition of some pixel colours at various relative coordinates. The same thing can be done using PixelSearch in AutoIt.

pixelsearch sounds really nice, but it raises another question: can i define an "object" using pixel by pixel method?

because, for an example, i need to click a colorfull text. however, that text may change into something else, which means i'll need two objects, one for one text and one for the other.

is it possible to make an object by defining each pixel manually?

Link to comment
Share on other sites

pixelsearch sounds really nice, but it raises another question: can i define an "object" using pixel by pixel method?

because, for an example, i need to click a colorfull text. however, that text may change into something else, which means i'll need two objects, one for one text and one for the other.

is it possible to make an object by defining each pixel manually?

If you want to recognise an area of the screen as being a certain object then PixelCheckSum might be what you want.

$object1 = PixelCheckSum($left,$top,$right,$Bottom);record the value for some region, maybe using a different script.

$object2 = PixelCheckSum($left,$top,$right,$Bottom);record the value at a different time
.
.
.

if PixelCheckSum($left,$top,$right,$Bottom) = $object1 then
 $object1 = $hereagain
Serial port communications UDF Includes functions for binary transmission and reception.printing UDF Useful for graphs, forms, labels, reports etc.Add User Call Tips to SciTE for functions in UDFs not included with AutoIt and for your own scripts.Functions with parameters in OnEvent mode and for Hot Keys One function replaces GuiSetOnEvent, GuiCtrlSetOnEvent and HotKeySet.UDF IsConnected2 for notification of status of connected state of many urls or IPs, without slowing the script.
Link to comment
Share on other sites

but that means i'll need to record that object every time in order for the script to work.

here's an example of what i want to do (written in actool though...)

object testobject

enormous bulk of code defining the object pixel by pixel

end

isobject testobject at xxx, yyy //if object testobject exists on xxx, yyy

bulk of code to execute if the object is found

end

else

bulk of code to execute if the object is not found

end

recording the object, unfortunately, is not an option.

Link to comment
Share on other sites

but that means i'll need to record that object every time in order for the script to work.

here's an example of what i want to do (written in actool though...)

object testobject

enormous bulk of code defining the object pixel by pixel

end

isobject testobject at xxx, yyy //if object testobject exists on xxx, yyy

bulk of code to execute if the object is found

end

else

bulk of code to execute if the object is not found

end

recording the object, unfortunately, is not an option.

It might be that I misunderstand, but you don't need to record the object every time. If recording the object is not an option then how do you test your code and how do you know how the create the 'object' in the first place? You only need to record it once and then save it somewhere, maybe in an ini file, then use it again. You can write a script to get the pixelchecksum at some region when you press a key.

Serial port communications UDF Includes functions for binary transmission and reception.printing UDF Useful for graphs, forms, labels, reports etc.Add User Call Tips to SciTE for functions in UDFs not included with AutoIt and for your own scripts.Functions with parameters in OnEvent mode and for Hot Keys One function replaces GuiSetOnEvent, GuiCtrlSetOnEvent and HotKeySet.UDF IsConnected2 for notification of status of connected state of many urls or IPs, without slowing the script.
Link to comment
Share on other sites

Get the checksum(PixelChecksum) of the grapic you want to identify, then store the checksum as a constant in your script(no need to make the checksum each time, as the checksum does not change(unless the background is transparent, which makes it ... impossible to use PixelChecksum)).

Link to comment
Share on other sites

It might be that I misunderstand, but you don't need to record the object every time. If recording the object is not an option then how do you test your code and how do you know how the create the 'object' in the first place? You only need to record it once and then save it somewhere, maybe in an ini file, then use it again. You can write a script to get the pixelchecksum at some region when you press a key.

yes, that's what i want to do!

put an object in a file, which i'll access when i need it.

however, i don't know how how to do what you said.

all i have is basic knowledge.

all the commands i sort of mastered are

MouseMove

MouseClick

Send

Sleep

Beep

well, i'm not sure about send yet, because i didn't poke it till it's full potential, but i do know some commands.

however, recording or doing... anything that is beyond moving the mouse and clicking is what i'm trying to learn XD

suggestions?

also, i'd like to put the object into an ini file. or any other file for that matter, but a different file.

@freefry: i'm very unfamiliar with the commands you mentioned...

no clue how to use them. =(

elaborate?

Edited by GodForsakenSoul
Link to comment
Share on other sites

Well, do a pixel checksum of the place of the screen you want to "remember"(it works just the MouseMove/Click functions, X and Y of where to start and end the checksum, an area basically), then when you have the checksum you could save it as a file, or just save it as a Constant in yours script.

Example(taken from the helpfile):

; Wait until something changes in the region 0,0 to 50,50

; Get initial checksum
$checksum = PixelChecksum(0,0, 50,50)

; Wait for the region to change, the region is checked every 100ms to reduce CPU load
While $checksum = PixelChecksum(0,0, 50, 50)
  Sleep(100)
WEnd

MsgBox(0, "", "Something in the region has changed!")oÝ÷ Ù8ZµìZ^Â¥uÈ^rKa{)¥²)íë©¥êåyûh*.®Ç+yéÚÙz!yÉ"­º{b[(ëax!jÇ!jxv)í«Z­æ«r¢ì(ºW]¢+,j÷©®Ún¬¢g­)à)¶¬jëh×6MsgBox(0, "CheckSum", PixelChecksum(0,0, 50,50))oÝ÷ Ø   ݶ§±«Þ¶§ºfޮƧvËh­æ¬iÊ'²Ö§µ«­¢+Ù¥´
½¹ÍÐÀÌØí
¡­ÍÕ´ôÄÈÌÐÔØÜàoÝ÷ Ù8^*.q©Üç$j|­æ¨*.®Ç+yéí¢Çûay»­¶è¯ZµëÞ®
ڦʡiا)ÝÊÞjYr¶«xLZ^jëh×6#include <Misc.au3>
Dim $Checksum = 123456789 ; Previous, stored checksum. - note: not a real checksum, just to demonstrate

While 1
    If _IsPressed("01") Then
        $mPos = MouseGetPos()
        If PixelCheckSum($mPos[0]-5, $mPos[1]-5, $mPos[0]+5, $mPos[1]+5) = $CheckSum Then
            ; .. do whatever here
        EndIf
    EndIf
    Sleep(10)
WEnd
Edited by FreeFry
Link to comment
Share on other sites

Well, do a pixel checksum of the place of the screen you want to "remember"(it works just the MouseMove/Click functions, X and Y of where to start and end the checksum, an area basically), then when you have the checksum you could save it as a file, or just save it as a Constant in yours script.

Example(taken from the helpfile):

; Wait until something changes in the region 0,0 to 50,50

; Get initial checksum
$checksum = PixelChecksum(0,0, 50,50)

; Wait for the region to change, the region is checked every 100ms to reduce CPU load
While $checksum = PixelChecksum(0,0, 50, 50)
  Sleep(100)
WEnd

MsgBox(0, "", "Something in the region has changed!")oÝ÷ Ù8ZµìZ^Â¥uÈ^rKa{)¥²)íë©¥êåyûh*.®Ç+yéÚÙz!yÉ"­º{b[(ëax!jÇ!jxv)í«Z­æ«r¢ì(ºW]¢+,j÷©®Ún¬¢g­)à)¶¬jëh×6MsgBox(0, "CheckSum", PixelChecksum(0,0, 50,50))oÝ÷ Ø   ݶ§±«Þ¶§ºfޮƧvËh­æ¬iÊ'²Ö§µ«­¢+Ù¥´
½¹ÍÐÀÌØí
¡­ÍÕ´ôÄÈÌÐÔØÜàoÝ÷ Ù8^*.q©Üç$j|­æ¨*.®Ç+yéí¢Çûay»­¶è¯ZµëÞ®
ڦʡiا)ÝÊÞjYr¶«xLZ^jëh×6#include <Misc.au3>
Dim $Checksum = 123456789 ; Previous, stored checksum. - note: not a real checksum, just to demonstrate

While 1
    If _IsPressed("01") Then
        $mPos = MouseGetPos()
        If PixelCheckSum($mPos[0]-5, $mPos[1]-5, $mPos[0]+5, $mPos[1]+5) = $CheckSum Then
            ; .. do whatever here
        EndIf
    EndIf
    Sleep(10)
WEnd
ok, this already sounds like the thing i want....

however, i have one last question:

"checksum" is "check" and "sum" combined. that means autoit checks the sum of the... something of the pixels and uses it.

is it possible that two different objects have identical checksum?

Link to comment
Share on other sites

You shouldn't referr to parts of the screen as objects, as they're not.

PixelChecksum gathers up all pixels in an area and makes a CheckSum of them, which is just a bunch of numbers, which is unique to the pixel colors in the area it is used on.

So, yes it can appear on another part of the scren if the same graphic/button or w/e exist there..

Link to comment
Share on other sites

You shouldn't referr to parts of the screen as objects, as they're not.

PixelChecksum gathers up all pixels in an area and makes a CheckSum of them, which is just a bunch of numbers, which is unique to the pixel colors in the area it is used on.

So, yes it can appear on another part of the scren if the same graphic/button or w/e exist there..

no, i mean, two different buttons.

can two different buttons have the same checksum?

also, just making sure that i understand how to use checksum the way i want to:

1) find the coordinates of the button

2) msgbox (0. "checksum", pixelchecksum (xxx, yyy, aaa, bbb)

where xxx, yyy, are the coordinates of the top-left pixel of the checksum box thing

and aaa, bbb, is the size of the box.

3) save the numbers of the checksum as a constant. dim whatever.

now...

dim pcs = 1234567890

while 1=1
  if pixelchecksum (xxx, yyy, aaa, bbb) = $pcs
     execute code if button found
  else
     execute code if button not found

is this correct?

Link to comment
Share on other sites

Yes, that is correct.

Just a side note so you got it really straight:

It's not the buttons you're checksumming, it's the pixels on your screen(PixelChecksum doesn't know, nor care about buttons or whatever, all it does it check the pixels in the area specified).

Two exactly identical buttons, in size and properties could have the same checksum. But it's not really probable.

It's just a matter of how they look, as that is what pixels makes up.

Link to comment
Share on other sites

  • 3 weeks later...

I don't post here much but I ran across this in a search. I wanted to add a bit that I hope will be helpful.

The "object" the original poster was referring to is an array / matrix that was defined as a special datatype in AC Tool. It contained an x,y offset and a max color value for each pixel (I assume it didn't store every r,g,b value for sake of computation). So if you had a 2 by 2 blue square (RGB of 10 10 235) surrounded by white, a 4 by 4 object of that area would look like:

Object Square

255=0,0|255=1,0|255=2,0|255=3,0|

255=0,1|235=1,1|235=2,1|255=3,1|

255=0,2|235=1,2|235=2,2|255=3,2|

255=0,3|255=1,3|255=2,3|255=3,3|

End Object

The 255 is white (rgb 255 255 255) and the 235 is the square. The format is MaxColorValue=XOffset,YOffset and then | as a delimiter.

AC Tool can be found at http://www.actool.net/

The source code is there and I believe it's in pascal.

Doing a similar thing is entirely possible with AutoIt. Just as a quick fix most people use the PixelGetColor function and search for one or a few pixels at a predetermined location. I think to properly write a similar function for AutoIt you would need to use the GDI32.dll to grab the window image and poll it for color values. I'm not sure if PixelGetColor grabs the image everytime called but the concern is CPU usage. If you're searching for a 30 x 30 image over 100 possible locations you might have problems with those 90000 PixelGetColor calls.

Anyway use PixelGetColor for now. PixelCheckSum works as well if you know the background doesn't change and you have no transparency. You can even do multiple PixelGetColors in an If state to get similar behavior ie

If PixelGetColor(10, 10) = 0x000000 AND PixelGetColor(11, 11) = 0x000000 AND PixelGetColor(10,11) = 0x000000 AND PixelGetColor(11, 10) = 0x000000 Then
    MsgBox(0, "Successful", "Found object")
Else
    MsgBox(0, "Failure","Did not find object")
EndIf

0x000000 = black so above code looks for 2 x 2 black square with top left origin of 10,10 (I have black desktop background)

Anyway I'll cut on email notification of reply so if you're still struggling I'll come back to help.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...