Jump to content

Get browser element under mouse cursor


Recommended Posts

I am attempting to use the elementFromPoint method to retrieve the text below the mouse cursor. In testing, it seems like the Y coordinates are off by approx the width of the menu, toolbars, and tabs. Is this normal? If so, how should I adjust for this?

#include <IE.au3>
#Include <GUIConstantsEx.au3>

Opt("MouseCoordMode", 2)  

HotKeySet("^+a", "Test")

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit

    EndSwitch
WEnd

Func Test()
Local $oIE, $pos, $element, $t

$pos = MouseGetPos()

; get topmost browser tab
$oIE = _IEAttach ("", "instance", 1)

If @error = $_IEStatus_Success Then
; get window title 
    $hwnd = _IEPropertyGet($oIE, "hwnd")
    $title = WinGetTitle($hwnd)

    $oIE = _IEAttach($title, "windowtitle")
EndIf

;MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

$element = $oIE.document.elementFromPoint($pos[0], $pos[1])

If IsObj($element) Then
    MsgBox(0,$pos[0] & ' ' & $pos[1], _IEPropertyGet($element, 'innertext'))
EndIf

EndFunc

Also, I think I found an issue with the _IEPropertyGet function. To see, uncomment the line

MsgBox(0,'yoffset', _IEPropertyGet($oIE, "browsery"))

Dan

Link to comment
Share on other sites

#include <IE.au3>
#include <WinAPI.au3>

$_IEErrorNotify = False

Dim $oIE = _IECreate('http://www.google.com')
_IEErrorHandlerRegister()
Dim $hWnd = ControlGetHandle(_IEPropertyGet($oIE, 'hwnd'), '', 'Internet Explorer_Server1')
Dim $tPt
;_IEHeadInsertEventScript($oIE, 'document', 'onmousedown', 'alert("clientX: " + event.clientX + " clientY: " + event.clientY + "\noffsetX: " + event.offsetX + " offsetY: " + event.offsetY + "\nsrcElement: " + event.srcElement + " srcElementId: " + event.srcElement.id + " srcElementName: " + event.srcElement.name, 0, "Info");')


While _IEPropertyGet($oIE, 'visible')
    Sleep(800)
    $tPt = _WinAPI_GetMousePos(True, $hWnd)
    $oElem = _elementFromPoint(DllStructGetData($tPt, 1), DllStructGetData($tPt, 2))
    ConsoleWrite($oElem.tagName & ': ' & $oElem.id & ', ' & $oElem.name & @LF)
WEnd

Func _elementFromPoint($iX, $iY)
    Return $oIE.document.elementFromPoint($iX, $iY)
EndFunc

Maybe you can use something like _IEHeadInsertEventScript() with document event 'onmousemove' and use IEEval() (search the forum).

Link to comment
Share on other sites

That's some interesting code, but I don't think I can use it to address my particular need.

In the example I posted, I'm using a hotkey to allow the user to execute a routine after they've positioned the mouse over the desired element. I need to be able to determine the element the mouse is positioned over.

Link to comment
Share on other sites

  • 2 weeks later...

I was actually reluctant to implement the coordinate routines in _IEPropertyGet because they have a history of being calculated incorrectly and changing from IE version to IE version. These did work when they were first released. What browser version are you testing with?

Dale

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Link to comment
Share on other sites

I see that I didn't read your post carefully enough. Thought you were using the _IEPropertyGet coordinate functions, be you were not.

The error you report with _IEPropertyGet($oIE, "browsery") is because I am neglecting to trap the error and send back bad object type (what you are asking for is invalid). The browserX and browserY can only be used on a document element, not on the browser itself as you attempted to do here.

Dale

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Link to comment
Share on other sites

Based on your experience, is there a better way to identify the element below the mouse cursor?

No, this would be the most straight forward method. Once you figure out the geometry it should work reliably.

Dale

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Link to comment
Share on other sites

  • 5 months later...

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)

Using the above code, the event object's clientX & clientY properties are used to retrieve the element below the mouse cursor. In my limited testing, I have yet to find a scenario where this code doesn't work.

Thoughts?

Dan

Link to comment
Share on other sites

  • 2 years later...

I've been struggling with this for a while now, and I think that I finally found a reliable way to get the mouse coordinates within the client region by using the createEventObject() function:

$evt = $oIE.document.createEventObject()
$element = $oIE.document.elementFromPoint($evt.clientX, $evt.clientY)
I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?
Link to comment
Share on other sites

I've been successfully using this method with IE for several years. Now I want to do the same thing with Firefox. Can this be done without using the onmousemove event handler?

I don't know. Perhaps the FF UDF. Did you look if it has something like that?
Link to comment
Share on other sites

OK... here's what I've come up with thus far using FF UDF:

Local $pos = MouseGetPos(), $element

_FFConnect(Default, Default, 3000)
_FFObjNew("mouseevent", "null")

_FFCmd("FFau3.mouseevent=null;")
_FFCmd("var listener = function (event) {FFau3.mouseevent = event};")
_FFCmd("content.document.addEventListener('mousemove',listener, false);")

; Force minimal mouse movement
MouseMove($pos[0]+1, $pos[1])
MouseMove($pos[0]-1, $pos[1])

_FFCmd("content.document.removeEventListener('mousemove',listener, false);")

$element = _FFCmd("content.document.elementFromPoint(FFau3.mouseevent.clientX, FFau3.mouseevent.clientY)")
Link to comment
Share on other sites

Here is my revised version, which doesn't require the MouseMove trick:

Func _FFGetMouseElement($aPoint)
Local $aFFPos

$aFFPos = _ScreenToClient(_FFWindowGetHandle(), $aPoint[0], $aPoint[1])

; Adjust Y value for height of Tabs, Toolbars, etc
$aFFPos[1] -= _FFCmd("window.content.mozInnerScreenY - window.mozInnerScreenY")

_FFCmd("FFau3.xpath=FFau3.WCD.elementFromPoint(" & $aFFPos[0] & ", " & $aFFPos[1] & ");")
EndFunc

Func _ScreenToClient($h_wnd, $i_x, $i_y); Returns an array, [0] = x, [1] = y
If IsString($h_wnd) Then $h_wnd = WinGetHandle($h_wnd)

Local $t_xy_coords = DllStructCreate("int x; int y")
DllStructSetData($t_xy_coords, "x", $i_x)
DllStructSetData($t_xy_coords, "y", $i_y)

DllCall("user32.dll", "int", "ScreenToClient", "hwnd", $h_wnd, "ptr", DllStructGetPtr($t_xy_coords))

Local $a_ret_coords[2] = [DllStructGetData($t_xy_coords, "x"), DllStructGetData($t_xy_coords, "y")]
Return $a_ret_coords
EndFunc
Link to comment
Share on other sites

  • 1 year later...

So this works very well for me except if i move my cursor out of the IE browser window. I get the error that the

Variable must be of type "Object".:
ConsoleWrite($oElem.tagName & ': ' & $oElem.id & $oElem.name & @LF)
ConsoleWrite($oElem^ ERROR

Is there a way to ignore the error while i dont have the cursor in the IE window?

Edited by chachew
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...