Get browser element under mouse cursor

[Danp2]

I am attempting to use the elementFromPoint method to retrieve the text below the mouse cursor. In testing, it seems like the Y coordinates are off by approx the width of the menu, toolbars, and tabs. Is this normal? If so, how should I adjust for this?

expandcollapse popup

#include <IE.au3>
#Include <GUIConstantsEx.au3>

Opt("MouseCoordMode", 2)  

HotKeySet("^+a", "Test")

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit

    EndSwitch
WEnd

Func Test()
Local $oIE, $pos, $element, $t

$pos = MouseGetPos()

; get topmost browser tab
$oIE = _IEAttach ("", "instance", 1)

If @error = $_IEStatus_Success Then
; get window title 
    $hwnd = _IEPropertyGet($oIE, "hwnd")
    $title = WinGetTitle($hwnd)

    $oIE = _IEAttach($title, "windowtitle")
EndIf

;MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

$element = $oIE.document.elementFromPoint($pos[0], $pos[1])

If IsObj($element) Then
    MsgBox(0,$pos[0] & ' ' & $pos[1], _IEPropertyGet($element, 'innertext'))
EndIf

EndFunc

Also, I think I found an issue with the _IEPropertyGet function. To see, uncomment the line

MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

Dan

[Authenticity]

#include <IE.au3>
#include <WinAPI.au3>

$_IEErrorNotify = False

Dim $oIE = _IECreate('http://www.google.com')
_IEErrorHandlerRegister()
Dim $hWnd = ControlGetHandle(_IEPropertyGet($oIE, 'hwnd'), '', 'Internet Explorer_Server1')
Dim $tPt
;_IEHeadInsertEventScript($oIE, 'document', 'onmousedown', 'alert("clientX: " + event.clientX + " clientY: " + event.clientY + "\noffsetX: " + event.offsetX + " offsetY: " + event.offsetY + "\nsrcElement: " + event.srcElement + " srcElementId: " + event.srcElement.id + " srcElementName: " + event.srcElement.name, 0, "Info");')


While _IEPropertyGet($oIE, 'visible')
    Sleep(800)
    $tPt = _WinAPI_GetMousePos(True, $hWnd)
    $oElem = _elementFromPoint(DllStructGetData($tPt, 1), DllStructGetData($tPt, 2))
    ConsoleWrite($oElem.tagName & ': ' & $oElem.id & ', ' & $oElem.name & @LF)
WEnd

Func _elementFromPoint($iX, $iY)
    Return $oIE.document.elementFromPoint($iX, $iY)
EndFunc

Maybe you can use something like _IEHeadInsertEventScript() with document event 'onmousemove' and use IEEval() (search the forum).

[Danp2]

That's some interesting code, but I don't think I can use it to address my particular need.

In the example I posted, I'm using a hotkey to allow the user to execute a routine after they've positioned the mouse over the desired element. I need to be able to determine the element the mouse is positioned over.

[Danp2]

FYI, I finally got this working by subtracting the value of $oIE.document.parentwindow.screenTop from the vertical position returned by MouseGetPos().

[DaleHohm]

I was actually reluctant to implement the coordinate routines in _IEPropertyGet because they have a history of being calculated incorrectly and changing from IE version to IE version. These did work when they were first released. What browser version are you testing with?

Dale

[Danp2]

I was actually reluctant to implement the coordinate routines in _IEPropertyGet because they have a history of being calculated incorrectly and changing from IE version to IE version. These did work when they were first released. What browser version are you testing with?

Dale

IE8

[DaleHohm]

I see that I didn't read your post carefully enough. Thought you were using the _IEPropertyGet coordinate functions, be you were not.

The error you report with _IEPropertyGet($oIE, "browsery") is because I am neglecting to trap the error and send back bad object type (what you are asking for is invalid). The browserX and browserY can only be used on a document element, not on the browser itself as you attempted to do here.

Dale

[Danp2]

Based on your experience, is there a better way to identify the element below the mouse cursor?

[DaleHohm]

Based on your experience, is there a better way to identify the element below the mouse cursor?

No, this would be the most straight forward method. Once you figure out the geometry it should work reliably.

Dale

[Danp2]

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)

Using the above code, the event object's clientX & clientY properties are used to retrieve the element below the mouse cursor. In my limited testing, I have yet to find a scenario where this code doesn't work.

Thoughts?

Dan

[Danp2]

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)

I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?

[AdmiralAlkex]

I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?

I don't know. Perhaps the FF UDF. Did you look if it has something like that?

[Danp2]

Yes, I'm using the FF UDF to interact with the website. Since my earlier post, I've been playing around with using the _FFCmd function to temporarily adding a listener for the mousemove event.

[Danp2]

OK... here's what I've come up with thus far using FF UDF:

Local $pos = MouseGetPos(), $element

_FFConnect(Default, Default, 3000)
_FFObjNew("mouseevent", "null")

_FFCmd("FFau3.mouseevent=null;")
_FFCmd("var listener = function (event) {FFau3.mouseevent = event};")
_FFCmd("content.document.addEventListener('mousemove',listener, false);")

; Force minimal mouse movement
MouseMove($pos[0]+1, $pos[1])
MouseMove($pos[0]-1, $pos[1])

_FFCmd("content.document.removeEventListener('mousemove',listener, false);")

$element = _FFCmd("content.document.elementFromPoint(FFau3.mouseevent.clientX, FFau3.mouseevent.clientY)")

[Danp2]

Here is my revised version, which doesn't require the MouseMove trick:

Func _FFGetMouseElement($aPoint)
Local $aFFPos

$aFFPos = _ScreenToClient(_FFWindowGetHandle(), $aPoint[0], $aPoint[1])

; Adjust Y value for height of Tabs, Toolbars, etc
$aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

_FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")
EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

Local $t_xy_coords = DllStructCreate("int x; int y")
DllStructSetData($t_xy_coords, "x", $i_x)
DllStructSetData($t_xy_coords, "y", $i_y)

DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
Return $a_ret_coords
EndFunc

[chachew]

So this works very well for me except if i move my cursor out of the IE browser window. I get the error that the

Variable must be of type "Object".:

ConsoleWrite($oElem.tagName & ': ' & $oElem.id & $oElem.name & @LF)
ConsoleWrite($oElem^ ERROR

Is there a way to ignore the error while i dont have the cursor in the IE window?

Edited May 27, 2014 by chachew

[JohnOne]

Yes.

Use IsObj() function before trying to use an object.