Jump to content
Sign in to follow this  
Danp2

Get browser element under mouse cursor

Recommended Posts

Danp2

I am attempting to use the elementFromPoint method to retrieve the text below the mouse cursor. In testing, it seems like the Y coordinates are off by approx the width of the menu, toolbars, and tabs. Is this normal? If so, how should I adjust for this?

#include <IE.au3>
#Include <GUIConstantsEx.au3>

Opt("MouseCoordMode", 2)  

HotKeySet("^+a", "Test")

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit

    EndSwitch
WEnd

Func Test()
Local $oIE, $pos, $element, $t

$pos = MouseGetPos()

; get topmost browser tab
$oIE = _IEAttach ("", "instance", 1)

If @error = $_IEStatus_Success Then
; get window title 
    $hwnd = _IEPropertyGet($oIE, "hwnd")
    $title = WinGetTitle($hwnd)

    $oIE = _IEAttach($title, "windowtitle")
EndIf

;MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

$element = $oIE.document.elementFromPoint($pos[0], $pos[1])

If IsObj($element) Then
    MsgBox(0,$pos[0] & ' ' & $pos[1], _IEPropertyGet($element, 'innertext'))
EndIf

EndFunc

Also, I think I found an issue with the _IEPropertyGet function. To see, uncomment the line

MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

Dan

Share this post


Link to post
Share on other sites
Authenticity

#include <IE.au3>
#include <WinAPI.au3>

$_IEErrorNotify = False

Dim $oIE = _IECreate('http://www.google.com')
_IEErrorHandlerRegister()
Dim $hWnd = ControlGetHandle(_IEPropertyGet($oIE, 'hwnd'), '', 'Internet Explorer_Server1')
Dim $tPt
;_IEHeadInsertEventScript($oIE, 'document', 'onmousedown', 'alert("clientX: " + event.clientX + " clientY: " + event.clientY + "\noffsetX: " + event.offsetX + " offsetY: " + event.offsetY + "\nsrcElement: " + event.srcElement + " srcElementId: " + event.srcElement.id + " srcElementName: " + event.srcElement.name, 0, "Info");')


While _IEPropertyGet($oIE, 'visible')
    Sleep(800)
    $tPt = _WinAPI_GetMousePos(True, $hWnd)
    $oElem = _elementFromPoint(DllStructGetData($tPt, 1), DllStructGetData($tPt, 2))
    ConsoleWrite($oElem.tagName & ': ' & $oElem.id & ', ' & $oElem.name & @LF)
WEnd

Func _elementFromPoint($iX, $iY)
    Return $oIE.document.elementFromPoint($iX, $iY)
EndFunc

Maybe you can use something like _IEHeadInsertEventScript() with document event 'onmousemove' and use IEEval() (search the forum).

Share this post


Link to post
Share on other sites
Danp2

That's some interesting code, but I don't think I can use it to address my particular need.

In the example I posted, I'm using a hotkey to allow the user to execute a routine after they've positioned the mouse over the desired element. I need to be able to determine the element the mouse is positioned over.

Share this post


Link to post
Share on other sites
DaleHohm

I was actually reluctant to implement the coordinate routines in _IEPropertyGet because they have a history of being calculated incorrectly and changing from IE version to IE version. These did work when they were first released. What browser version are you testing with?

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
Danp2

I was actually reluctant to implement the coordinate routines in _IEPropertyGet because they have a history of being calculated incorrectly and changing from IE version to IE version. These did work when they were first released. What browser version are you testing with?

Dale

IE8

Share this post


Link to post
Share on other sites
DaleHohm

I see that I didn't read your post carefully enough. Thought you were using the _IEPropertyGet coordinate functions, be you were not.

The error you report with _IEPropertyGet($oIE, "browsery") is because I am neglecting to trap the error and send back bad object type (what you are asking for is invalid). The browserX and browserY can only be used on a document element, not on the browser itself as you attempted to do here.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
DaleHohm

Based on your experience, is there a better way to identify the element below the mouse cursor?

No, this would be the most straight forward method. Once you figure out the geometry it should work reliably.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
Danp2

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)

Using the above code, the event object's clientX & clientY properties are used to retrieve the element below the mouse cursor. In my limited testing, I have yet to find a scenario where this code doesn't work.

Thoughts?

Dan

Share this post


Link to post
Share on other sites
Danp2

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)
I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?

Share this post


Link to post
Share on other sites
AdmiralAlkex

I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?

I don't know. Perhaps the FF UDF. Did you look if it has something like that?

Share this post


Link to post
Share on other sites
Danp2

OK... here's what I've come up with thus far using FF UDF:

Local $pos = MouseGetPos(), $element

_FFConnect(Default, Default, 3000)
_FFObjNew("mouseevent", "null")

_FFCmd("FFau3.mouseevent=null;")
_FFCmd("var listener = function (event) {FFau3.mouseevent = event};")
_FFCmd("content.document.addEventListener('mousemove',listener, false);")

; Force minimal mouse movement
MouseMove($pos[0]+1, $pos[1])
MouseMove($pos[0]-1, $pos[1])

_FFCmd("content.document.removeEventListener('mousemove',listener, false);")

$element = _FFCmd("content.document.elementFromPoint(FFau3.mouseevent.clientX, FFau3.mouseevent.clientY)")

Share this post


Link to post
Share on other sites
Danp2

Here is my revised version, which doesn't require the MouseMove trick:

Func _FFGetMouseElement($aPoint)
Local $aFFPos

$aFFPos = _ScreenToClient(_FFWindowGetHandle(), $aPoint[0], $aPoint[1])

; Adjust Y value for height of Tabs, Toolbars, etc
$aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

_FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")
EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

Local $t_xy_coords = DllStructCreate("int x; int y")
DllStructSetData($t_xy_coords, "x", $i_x)
DllStructSetData($t_xy_coords, "y", $i_y)

DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
Return $a_ret_coords
EndFunc

Share this post


Link to post
Share on other sites
chachew

So this works very well for me except if i move my cursor out of the IE browser window. I get the error that the

Variable must be of type "Object".:
ConsoleWrite($oElem.tagName & ': ' & $oElem.id & $oElem.name & @LF)
ConsoleWrite($oElem^ ERROR

Is there a way to ignore the error while i dont have the cursor in the IE window?

Edited by chachew

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.