derKrieger

Getting data with the mouse moves?

23 posts in this topic

Hello everyone. I am a new member of this forum as well as the autoit itself. I have a question which is possibly has no answer. But if that is available in autoit will aid me really a lot .

Can anyone tell me if i can get the data from the website by only moving the mouse onto it. I mean for example could i get the all contents of this post which i have just written by moving the mouse through it. Why i intend to do is if i could do this with any of autoit command i did not have to deal with the source code of the web pages and spare time much .

If it is not possibly any suggest will be appreciated which is alternative .

Hopefully that was clear.

Thanks anyone in advance  :)

Share this post


Link to post
Share on other sites



I had feeling that i have to explain more. :)  Please imagine i am trying to code something . And This code will execute this  as shown below

What i see is while writing this post is a "Submit Reply". I know i can click on it by GetObjectbyIdName etc. But i dont want to do this with that because the tag names or id names can be edited by the owner of the page(at least theoretically). Or maybe i had to compare the datas on the website , but we have a lot of data which could be hard coding with only using the source code.That's why the autoit will move the cursor all the webpage along and when it detects the name that we want will continue to do other stuffs

 

I hope this is more clarified.

 

Thanks for everything in advance

Share this post


Link to post
Share on other sites

any suggestion ;(

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

I realized It did not have to be with mouse. Something simply scan through all the writings on the page and do other stuffs after detecting specified value.

 

Thank you all in advance

Edited by derKrieger

Share this post


Link to post
Share on other sites

Is there noone want to help ?

Share this post


Link to post
Share on other sites

Hi @derKrieger

.... not sure what exactly you want achieve, anyway here is a script I've posted here (https://www.autoitscript.com/forum/topic/173741-highlight-element-in-my-web-page/?do=findComment&comment=1259988 ) some days ago that I have a little modified so that it can print on the ConsoleWrite the text of the control on the web page behind the mouse cursor.

#include <IE.au3>
#include <WinAPI.au3>
#include <GUIConstantsEx.au3>
; #include <WinAPIEx.au3>
#include <WindowsConstants.au3>

Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
Local $oDoc = _IEDocGetObj($oIE)
Local $oIE_hwnd = _IEPropertyGet($oIE, "hwnd")
Local $oElement, $oElement_save, $aMousePos
Local $aBrowserPos[6] ; x , y , width , height , x right , y bottom
Local $aOverlayPos[4] ; coordinates of the overlay frame
; Create a Phantom window with a visible frame
Local $aParentPos, $AlphaKey = 0xABABAB
Local $iBorderColor = 0x00FF00, $iBorderThickness = 5
$hOverlay = GUICreate("", -50, -50, 10, 10, $WS_POPUPWINDOW, $WS_EX_LAYERED + $WS_EX_TOPMOST) ; transparent
$Panel = GUICreate("", 48, 48, 10, 10, $WS_CHILD + $WS_VISIBLE, -1, $hOverlay) ; this new window becomes a child of $hOverlay window
GUISetBkColor($iBorderColor, $hOverlay)
GUISetBkColor($AlphaKey, $Panel)
_WinAPI_SetLayeredWindowAttributes($hOverlay, $AlphaKey, 0, $LWA_COLORKEY)
GUISetState(@SW_SHOW, $hOverlay)

While WinExists($oIE_hwnd)
    $aMousePos = MouseGetPos()
    ; Now we get reference to the DOM object behind the mouse pointer
    $oElement = $oDoc.elementFromPoint($aMousePos[0] - $oIE.document.parentwindow.screenLeft, $aMousePos[1] - $oIE.document.parentwindow.screenTop)
    If IsObj($oElement) And $oElement <> $oElement_save Then
        $oElement_save = $oElement
        $aBrowserPos[0] = Int($oIE.document.parentwindow.screenLeft) ;      X position on the desktop of the left edge of the browser
        $aBrowserPos[1] = Int($oIE.document.parentwindow.screenTop) ;       Y position on the desktop of the top border of the browser
        $aBrowserPos[2] = Int($oIE.document.documentElement.clientWidth) ;  Width of browser's client area
        $aBrowserPos[3] = Int($oIE.document.documentElement.clientHeight) ; Height of browser's client area
        $aBrowserPos[4] = $aBrowserPos[0] + $aBrowserPos[2] ;               X position on the desktop of the right border of the browser
        $aBrowserPos[5] = $aBrowserPos[1] + $aBrowserPos[3] ;               Y position on the desktop of the bottom border of the browser
        ;
        $oRect = $oElement.getBoundingClientRect() ; coordinates of the DOM object within the client area of the browser
        ;
        $aOverlayPos[0] = Int($oRect.left) + $aBrowserPos[0] ; X position of the left overlay border (absolute desktop coordinates)
        If $aOverlayPos[0] < $aBrowserPos[0] Then $aOverlayPos[0] = $aBrowserPos[0] ; if left border of the DOM obj is out of browser, overlay border stays anyway within the screen
        $aOverlayPos[1] = Int($oRect.Top) + $aBrowserPos[1] ; Y  position of the top overlay border
        If $aOverlayPos[1] < $aBrowserPos[1] Then $aOverlayPos[1] = $aBrowserPos[1] ; if top border of obj is out of browser, overlay border stays anyway within the screen
        $aOverlayPos[2] = Int($oRect.right - $oRect.left) ; width of the obj
        If $aOverlayPos[0] + $aOverlayPos[2] > $aBrowserPos[4] Then $aOverlayPos[2] = $aBrowserPos[4] - $aOverlayPos[0] ; if obj's width is out of bounds, force the edge within the browser
        $aOverlayPos[3] = Int($oRect.bottom - $oRect.top) ; Height of the obj
        If $aOverlayPos[1] + $aOverlayPos[3] > $aBrowserPos[5] Then $aOverlayPos[3] = $aBrowserPos[5] - $aOverlayPos[1] ; stay within browser's bounds
        ;
        WinMove($hOverlay, "", $aOverlayPos[0], $aOverlayPos[1], $aOverlayPos[2], $aOverlayPos[3])
        $aParentPos = WinGetPos($hOverlay) ; x, y, width, height
        WinMove($Panel, "", $iBorderThickness - 1, $iBorderThickness - 1, Int($aParentPos[2] - $iBorderThickness * 2), Int($aParentPos[3] - $iBorderThickness * 2))

        ConsoleWrite($oElement.OuterText & @CRLF) ; <---- modified here to print the text of control

    EndIf
WEnd

Have it a try and adapt it to your needs if is what you are searching.


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Thank you very much Chimp, i will tell you the result as fast as i can. 

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Hello Chimp. May i ask something? I had no much knowledge of Autoit only the basics. Can you tell me if i have understood right ? It is the reason i am novice in autoit i did not understand a lot about your script . But i had a try and what i see is :

1-Run the script

2-Locate the cursor somewhere of the screen

3-And the code will get the data where the cursor is on (actually get the data inside the green oblong shape)

 

Am i right ?

Thank you again Chimp.

 

 

Is there something i have not seen ?

 

Edited by derKrieger

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

Hi derKriger,

The above script do more things than what is needed for your purpose.
Sorry, I have lazily adapted that script mainly written for a different purpose, anyway here is a semplified version that will focus only on the text retrieval from the web page.
when you press the F2 key you will get the text behind the cursor.
The web page used is that opened by the _IeCreate() function, but if you need to read text from an already opened web page, than you have to use the _IeAttach() function instead. Have a look to the help on that function to see how to use it.
I added some comments on the script to try to explain what each instruction do.

#include <IE.au3>

; this will open a web page to a specified address.
Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
; Use the  _IEAttach() function instead if you want to attach to an already running instance of Iexplorer

HotKeySet("{ESC}", "Terminate") ; press esc to end this program
HotKeySet("{F2}", "ShowText") ; press F2 to get the text behind the cursor

While WinExists(_IEPropertyGet($oIE, "hwnd"))
    Sleep(1000) ; idle around waiting for user actions
WEnd

Func ShowText() ; This function is executed when you press the F2 key
    Local $sText = GetObjText($oIE) ; retrieve the text behind the cursor from the web page
    If $sText <> "" Then MsgBox(0, 0, $sText, 2) ; show text in an MsgBox for 2 seconds
EndFunc   ;==>ShowText

Func GetObjText(ByRef $oIE)
    ; If WinActive(_IEPropertyGet($oIE, "hwnd")) Then ; if the browser is the active window
    Local $oDoc = _IEDocGetObj($oIE)
    Local $aMousePos = MouseGetPos() ; get the coordinates of the mouse pointer
    Local $aBrowserPos[6]
    $aBrowserPos[0] = Int($oIE.document.parentwindow.screenLeft) ;      X position on the desktop of the left edge of the browser
    $aBrowserPos[1] = Int($oIE.document.parentwindow.screenTop) ;       Y position on the desktop of the top border of the browser
    $aBrowserPos[2] = Int($oIE.document.documentElement.clientWidth) ;  Width of browser's client area
    $aBrowserPos[3] = Int($oIE.document.documentElement.clientHeight) ; Height of browser's client area
    $aBrowserPos[4] = $aBrowserPos[0] + $aBrowserPos[2] ;               X position on the desktop of the right border of the browser
    $aBrowserPos[5] = $aBrowserPos[1] + $aBrowserPos[3] ;               Y position on the desktop of the bottom border of the browser

    ; check if mouse pointer is within the browser bounds
    If $aMousePos[0] >= $aBrowserPos[0] And $aMousePos[0] <= $aBrowserPos[4] And $aMousePos[1] >= $aBrowserPos[1] And $aMousePos[1] <= $aBrowserPos[5] Then

        ; Now we get reference to the DOM object behind the mouse pointer within the browser
        Local $oElement = $oDoc.elementFromPoint($aMousePos[0] - $aBrowserPos[0], $aMousePos[1] - $aBrowserPos[1])
        If IsObj($oElement) Then
            Return $oElement.OuterText ; <---- here we get the text of the DOM object behind the mouse pointer and return it to caller
        EndIf
    EndIf
    ; EndIf
    Return ""
EndFunc   ;==>GetObjText

Func Terminate()
    Exit 0
EndFunc   ;==>Terminate

 

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Hi Chimp,

 

Firstly, i want to thank you. It worked perfect and easier to understand for the novices such as me. However while i was dealing with your script i thought something. I tried to code something for it but unfortunately failed. Could we do this thing without using the mouse?

This time i know the data but i need the position of it .For example, in this page i have posted before this "Thank you very much Chimp, i will tell you the result as fast as i can. "

Can i get the position of that sentence or maybe words of the sentence which those i have declared before ?

I tried to do this by creating somewhat of searchbox like when we press Ctrl+F in chrome, and when that box detected the word, it would give me the position without using the mouse.  But failed.

 

Thanks for everything you teach me Chimp.

 

Share this post


Link to post
Share on other sites

Is there anyway to do that ?

Share this post


Link to post
Share on other sites

I realized It did not have to be with mouse. Something simply scan through all the writings on the page and do other stuffs after detecting specified value.

you could use tesseract.au3 ocr in autoit it will scan the page for you and convert it to text and will do the job for you but it is not that much dependable (but it is always an option)

Tesseract.au3 here: https://www.autoitscript.com/forum/topic/89542-tesseract-screen-ocr-udf/


No matter whatever the challenge maybe control on the outcome its on you its always have been.

MY UDF: Transpond UDF (Sent vriables to Programs) , Utter UDF (Speech Recognition)

Share this post


Link to post
Share on other sites

Hi Chimp,

 

Firstly, i want to thank you. It worked perfect and easier to understand for the novices such as me. However while i was dealing with your script i thought something. I tried to code something for it but unfortunately failed. Could we do this thing without using the mouse?

This time i know the data but i need the position of it .For example, in this page i have posted before this "Thank you very much Chimp, i will tell you the result as fast as i can. "

Can i get the position of that sentence or maybe words of the sentence which those i have declared before ?

I tried to do this by creating somewhat of searchbox like when we press Ctrl+F in chrome, and when that box detected the word, it would give me the position without using the mouse.  But failed.

 

Thanks for everything you teach me Chimp.

 

?? not clear to me what you want achieve, you say "... it would give me the position ..." what do you mean for position? x and y coordinates on the screen? and what to do if the searched phrase is outside the screen? or what else you mean, maybe the position of the phrase within the whole text of the web page?... (not clear)
anyway if you want retrieve the whole text  of the web page you can use the _IEBodyReadText() function (it returns raw text). here is a little example

 

#include <IE.au3>

; this will open a web page to a specified address.
Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
; Use the  _IEAttach() function instead if you want to attach to an already running instance of Iexplorer

Local $sText = _IEBodyReadText($oIE); retrieve the whole text from the web page
ConsoleWrite($sText & @CRLF) ; print the text to consol

 


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Thank you Surya, it seems to work but can you suggest me a link to see the functions of the tesseract. Because i need to understand how to use it.

 

Thank you for you too Chimp. Let me clear what i intend to do for you.i will give you more concrete example. Imagine there is a website. For instance let us think about the word 'pencil' . I want to get the positions of the every word which is 'pencil' on the website (i mean X,Y coordinates with position. ) And when i get the coordinates i will use the Mouseclickdrag function.It will highlight the word then rightclick and copy and paste somewhere. The script must do the same process for every word which is 'pencil'. I need to scan through the all page because i cant know where the words are located in order to transfer data somewhere. In our example, i dont know where the words of 'pencil' and their location. The script will scan the the page , will find the word and do other things. :)   It did not have to be 'pencil' everytime , I only gave an example. By the way i didnt get what you mean by this 'what to do if the searched phrase is outside the screen?'

 

Gratitude for your aid.

 

 

,

Share this post


Link to post
Share on other sites

... still not clear
suppose you search the word "pencil", well I don't get the logic of searching that word, copy it and "transfer data somewhere" then search again the same word and  "transfer data somewhere" again and repeat this till there are no more "pencil".... ?
At the end of this process you will have a "place" full of as many "pencil" as many  there are in the text.
If I understand well, maybe what you need is count how many times that word appears in the text....?


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

Hi Chimp, 

Thank you for answer. You completely understand right what i intend to do . Just the logic of that is not clear. Let me clear the main purpose.  The copy paste process is not the point. I only gave it as an example. The main point is to reach the word 'pencil' .   I added an image to clarify for you. In this image, i tried to show something.(sorry for my drawings)  . Imagine that is a website. That includes the words and the websites under the words. I want to get access only one website which is "www.pencil.com" But. unfortunately, this webpage is refreshed for every 20 seconds. And when it is refreshed, the new words with its websites are replaced the previous ones as shown in the image file named "Unbenannt2". And even though the new refreshed webpage contain the word i desire their locations will be different as shown in the image named "Unbenannt3" . So i can not know when the webpage which contains the word i have desire to access will be shown and its coordinates.If the webpage wasnt refreshed, i would directly use the Mouseclick function to click on it because i would have the certain coordinates.  So , i thought maybe i could create a while loop, and it will search for the word 'pencil' for every 20  seconds. When it detect the word that means  the webpage is refreshed again and the word 'pencil' is visible now. So when it detect the word it will give me the coordinates and will exit the while loop.Now, I  know the coordinates of the word 'pencil' . Finally i can use the Mouseclick function which will click on the website link that is a bit under the location of the word 'pencil'.  However i dont know how to write the script.

 

I am not sure whether it is complicated, but if you suggest me the way to do it or maybe any easier way, it will be greatly appreciated.

 

Gratitude for your help. 

 

 

 

 

Unbenannt.png

Unbenannt2.png

Unbenannt3.png

Edited by derKrieger

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

I would go in a way more programmatical than visual. That is:
Instead of trying to hit a link that changes its location on the web page, I would suggest to try using _IE* functions, in this way you should go without caring about the phisical position of the link on the page.:
1) one way could be to try to use the _IELinkClickByText() function
if this fails
2) enumerate all the links of the web page, then look for the word in the href property of the link, and if you found it, fire the "click"  method on that link.
or as an alternative
3) enumerate all the links of the web page, then search the word in the OuterText property of the link, and if you found it, fire the "click"  method on that link. (but this should be the equivalent as the method 1).

Of course, since I have no access to the source of the web page,  those are only wild guesses.

Would be better to have access to the source of the web page you want automate to see what's in there....

here 3 examples of what i'm talking about (....good luck):

method 1): try using the _IELinkClickByText() function

#include <IE.au3>

Local $sTarget = "pencil" ; searched word

; this will open a web page to a specified address.
Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
; Use the  _IEAttach() function instead if you want to attach to an already running instance of Iexplorer

While 1
    ; if you are lucky then this function should click on the link containing searched word
    _IELinkClickByText($oIE, $sTarget)

    If Not @error Then ExitLoop ; if no errors then click was OK

    Sleep(60000) ; pause for a minute (use Sleep(1200000) to pause for 20 minutes)
    _IEAction($oIE, "refresh") ; Refreshes the current document.
    Sleep(5000) ; wait 5 seconds to give time to web page to reload
WEnd

method 2): search text in the URL of the link

#include <IE.au3>

Local $sTarget = "pencil" ; searched word

; this will open a web page to a specified address.
Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
; Use the  _IEAttach() function instead if you want to attach to an already running instance of Iexplorer

Local $GotIt = False, $oLinks = ""
Do
    $oLinks = _IELinkGetCollection($oIE) ; get all links

    For $oLink In $oLinks
        ; search text in the url
        If StringInStr($oLink.href, $sTarget) Then
            _IEAction($oLink, "click")
            $GotIt = True
            ExitLoop
        EndIf
    Next
    If Not $GotIt Then
        Sleep(60000) ; pause for a minute (use Sleep(1200000) to pause for 20 minutes)
        _IEAction($oIE, "refresh") ; Refreshes the current document.
        Sleep(5000) ; wait 5 seconds to give some time to web page to reload
    EndIf
Until Not $GotIt

method 3): search text in the Outertext of the link

#include <IE.au3>

Local $sTarget = "pencil" ; searched word

; this will open a web page to a specified address.
Local $oIE = _IECreate("https://www.autoitscript.com/forum/topic/176026-getting-data-with-the-mouse-moves/")
; Use the  _IEAttach() function instead if you want to attach to an already running instance of Iexplorer

Local $GotIt = False, $oLinks
Do
    $oLinks = _IELinkGetCollection($oIE) ; get all links

    For $oLink In $oLinks
        ; search text in the Outertext of the link
        If StringInStr($oLink.OuterText, $sTarget) Then
            _IEAction($oLink, "click")
            $GotIt = True
            ExitLoop
        EndIf
    Next
    If Not $GotIt Then
        Sleep(60000) ; pause for a minute (use Sleep(1200000) to pause for 20 minutes)
        _IEAction($oIE, "refresh") ; Refreshes the current document.
        Sleep(5000) ; wait 5 seconds to give some time to web page to reload
    EndIf
Until Not $GotIt

 

Edited by Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites

Thank you Chimp. I will try them as soon as possible.

Share this post


Link to post
Share on other sites

Hello Chimp,

 

If i understand right, you recommend me to use source codes rather than these 3 option . Actually before starting this thread, i used those source codes to write my script. But  as i see ( not sure i am right) , i can only get the data from the source codes by their tag, name and id with autoit functions. What about timers embedded into span,  ul , div ,image, td and so on. I could not get the values  of them. That's why i attempted to consult this forum in order to get their values by another way. 

 

I am sorry Chimp if i ask very silly questions.

 

I am really thanful

,

 

 

 

Share this post


Link to post
Share on other sites

Hello Chimp, 

 

i had searched a lot before this thread, but now i realised i missed something at that time. I think as you recommend i can get value of everything by IETagNameGetCollection.

Right?

Really am i able to get all values or it has exceptions?

 

Thank you Chimp, ...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now