IUIAutomation MS framework automate chrome, FF, IE, ....

FaridAgl · December 29, 2013

@junkew

Thank you.

@Melba23

OK, you are the boss.

Melba23 · December 29, 2013

D4RKON3,

Thanks.

M23

dlow · January 1, 2014

Greetingz All

First off Thanks to everyone for their contribution.

@ junkew You are brilliant & thanks for sharing your code, knowledge, & time.

I have been working with your FF & IE examples. When I run the code I get a error in the UIAWrapper.

"C:\Program Files (x86)\AutoIt3\Include\UIAWrappers.au3" (363) : ==> Variable must be of type "Object".:
$obj.GetCurrentPropertyValue($Id,$tVal)
$obj^ ERROR

Well I am new to AutoIt & though I have tried to figure this out I don't know how to fix this. Could someone please point me in the right direction.

Thanks & all the best,

d

junkew · January 1, 2014

In an update I will put this function as a fix

replace the function in UIA_Wrappers with below and script will at least continue (but still not do what you want as the object itself is not found)

func _UIA_getPropertyValue($obj, $id)
    local $tval
    local $tStr

    if not isobj($obj) Then
        seterror(1,0,0)
        return "** NO PROPERTYVALUE DUE TO NONEXISTING OBJECT"
    EndIf
    
    $obj.GetCurrentPropertyValue($Id,$tVal)
    $tStr=""
    if isarray($tVal) Then
        for $i=0 to ubound($tval)-1
            $tStr=$tStr & $tVal[$i]
            if $i <> ubound($tVal)-1 Then
                $tStr=$tStr & ";"
            endif
        Next
        return $tStr
    endIf
    return $tVal
EndFunc

dlow · January 1, 2014

Okay thanks.

In my quest to use the JAB I came across this post. The answer seems logical from my limited programing knowledge. I am not sure if you are doing anything like this in your code. Or know/don't know, I just wanted to put it out there in my effort to further help the development of the project.

Thanks again. Now I am off to try & put this wonderful code you wrote to work.

All the best,

dirty

topten · January 2, 2014

Strange thing: When I run the script with admin rights, I get error "The requested action with this object has failed". But when I run it in normal mode- everything works fine. What can be the reason for that? Thanx and best regards, and Happy New Year!!!

junkew · January 2, 2014

@topTen: do you have this behavior also with simple spy source? I assume it can be related to UAC setting of windows

topten · January 2, 2014

I've checked it. I started simple spy with admin rights- and it works ok. You are right- the problem is different: IE 10 doesnt allow to create IE object with admin rights!!!!

Thanx a lot!!!!

topten · January 6, 2014

Hi!

1. Possible bug: I found that in some cases ^a - simply won't work (for example I have multi language keyboard layout- and I was having something like a

avarvarvar instead of varvarvar, so I made a small change

case "setvalue using keys"
           $obj.setfocus()
           ;send("^a")
          send($p1)
          sleep($UIA_DefaultWaitTime)

Now works ok -perhaps will be usefull for anyone- but again it can be different on different machines of course

2. Is it possible to use Controlsend() function instead of send () - so that I could work with application in background

case "setvalue using keys"
           $obj.setfocus()
           ;send("^a")
           Controlsend($p1)
           sleep($UIA_DefaultWaitTime)

If the answer "yes" how can I find controlID?

ControlSend ( "title", "text", controlID, "string" [, flag = 0] )

Thanx in advance!!!!!!

junkew · January 6, 2014

not sure what in multi language the shortcut is for select all (Ctrl+A keycombination)

you could try to replace with

a. send("^(a)")

b. send("^{a}")

c. make a small testapplication against notepad and see how it behaves with setvalue actions

try to use the action setvalue instead of setvalue using keys. Setvalue using keys is already a workaround for those textboxes that do not deal well with setvalue action itself

controlsend is the native AutoIT way of doing it so check within the helpfile of autoit the controlsend syntax

in general the simple spy automationid will be equal to the ID property of the AU3INF.EXE result(s) but not all controls are identifiable by a controlid.

Use the AU3INF.exe files to see your result(s)

This could be an alternate viewer for controls '?do=embed' frameborder='0' data-embedContent>>

topten · January 8, 2014

Thank you very much for your help and support!!!!

if IsObj($IE) Then
$sText="search"; indexrelative:=1" ;Is equal to
$oButton=_UIA_getFirstObjectOfElement($IE,"name:=" & $sText, $treescope_subtree)
EndIf

Is there any method to calculate the x,y position of found element on the window or on the screen?

Thanx in advance!!!

LarsJ · January 11, 2014

I have made some tests with event handler interfaces. For a start just to verify whether I could get some events at all.

I succeeded to get events for IUIAutomationEventHandler, IUIAutomationFocusChangedEventHandler and UIAutomationStructureChangedEventHandler but not for IUIAutomationPropertyChangedEventHandler.

Update 2014-01-13

Added the proper function to create custom interface methods to all 4 examples. See post #115. This means that all examples are working including the example with IUIAutomationPropertyChangedEventHandler.

IUIAutomationEventHandler

Run this code in Scite and you'll see window and menu open/close events in the console when you open/close a program e.g. Calculator and when you open/close a menu.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationEventHandler, $oIUIAutomationEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationEventHandler = ObjectFromTag( "oIUIAutomationEventHandler_", $dtagIUIAutomationEventHandler, $tIUIAutomationEventHandler, True )
  If Not IsObj( $oIUIAutomationEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  ; Window open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_Window_WindowClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  ; Menu open/close events
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuOpenedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit
  If $oUIAutomation.AddAutomationEventHandler( $UIA_MenuClosedEventId, $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationEventHandler_HandleAutomationEvent( $pSelf, $pSender, $iEventId ) ; Ret: long  Par: ptr;int
  ConsoleWrite( "oIUIAutomationEventHandler_HandleAutomationEvent: " & $iEventId & @CRLF )
  If $iEventId <> $UIA_Window_WindowClosedEventId Then
    Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    $oSender.AddRef()
    ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                  "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                  "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                  "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                  "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                  "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  EndIf
  Return $S_OK
EndFunc

Func oIUIAutomationEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationEventHandler Then
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: IUIAutomationEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationFocusChangedEventHandler

If you test with Calculator you'll get events when you click a number button and the focus changes between the button and the edit box.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationFocusChangedEventHandler, $oIUIAutomationFocusChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationFocusChangedEventHandler = ObjectFromTag( "oIUIAutomationFocusChangedEventHandler_", $dtagIUIAutomationFocusChangedEventHandler, $tIUIAutomationFocusChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationFocusChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  If $oUIAutomation.AddFocusChangedEventHandler( 0, $oIUIAutomationFocusChangedEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationFocusChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationFocusChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent( $pSelf, $pSender ) ; Ret: long  Par: ptr
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent: " & $pSender & @CRLF )
  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef()
  ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  Return $S_OK
EndFunc

Func oIUIAutomationFocusChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationFocusChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: IUIAutomationFocusChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationFocusChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationFocusChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationFocusChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationStructureChangedEventHandler

Switch between Standard and Scientific view in Calculator or switch folder in Windows Explorer and you'll get lots of events.

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationStructureChangedEventHandler, $oIUIAutomationStructureChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationStructureChangedEventHandler = ObjectFromTag( "oIUIAutomationStructureChangedEventHandler_", $dtagIUIAutomationStructureChangedEventHandler, $tIUIAutomationStructureChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationStructureChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  If $oUIAutomation.AddStructureChangedEventHandler( $pUIElement, $TreeScope_Subtree, 0, $oIUIAutomationStructureChangedEventHandler ) Then Exit

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(10)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationStructureChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationStructureChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $tVal
  $obj.GetCurrentPropertyValue( $id, $tVal )
  Return $tVal
EndFunc

Func oIUIAutomationStructureChangedEventHandler_HandleStructureChangedEvent( $pSelf, $pSender, $iChangeType, $pRuntimeId ) ; Ret: long  Par: ptr;long;ptr
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_HandleStructureChangedEvent: " & $pSender & " " & $iChangeType & @CRLF )
  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef()
  ConsoleWrite( "Title     = " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _
                "Class     = " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _
                "Ctrl type = " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _
                "Ctrl name = " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _
                "Value     = " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF & _
                "Handle    = " & Hex( _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & @CRLF )
  Return $S_OK
EndFunc

Func oIUIAutomationStructureChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationStructureChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: IUIAutomationStructureChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationStructureChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationStructureChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationStructureChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

IUIAutomationPropertyChangedEventHandler

I have tested the code below with Internet Explorer and I have expected to get $UIA_ValueValuePropertyId change events when I switch between different URLs. But I get no events. I have tried with several other programs but I simply get no events.

Is there anyone who has managed to record these events? In that case, I'd like to see the code.

To test the code you must change the description for AddPropertyChangedEventHandlerNativeArray in CUIAutomation2.au3 to this:

"AddPropertyChangedEventHandlerNativeArray hresult(ptr;long;ptr;ptr;struct*;int);"

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationPropertyChangedEventHandler, $oIUIAutomationPropertyChangedEventHandler

Global $oUIAutomation

MainFunc()



Func MainFunc()

  $oIUIAutomationPropertyChangedEventHandler = ObjectFromTag( "oIUIAutomationPropertyChangedEventHandler_", $dtagIUIAutomationPropertyChangedEventHandler, $tIUIAutomationPropertyChangedEventHandler, True )
  If Not IsObj( $oIUIAutomationPropertyChangedEventHandler ) Then Return

  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return

  Local $pUIElement
  $oUIAutomation.GetRootElement( $pUIElement ) ; Desktop
  If Not $pUIElement Then Return

  Local $tPropertyArray = DllStructCreate( "int[2]" )
  DllStructSetData( $tPropertyArray, 1, $UIA_ValueValuePropertyId, 1 )
  DllStructSetData( $tPropertyArray, 1, $UIA_ToggleToggleStatePropertyId, 2 )
  $oUIAutomation.AddPropertyChangedEventHandlerNativeArray( $pUIElement, $TreeScope_Descendants, 0, $oIUIAutomationPropertyChangedEventHandler, $tPropertyArray, 2 )

  HotKeySet( "{ESC}", "Quit" )

  While Sleep(100)
  WEnd

EndFunc

Func Quit()
  $oIUIAutomationPropertyChangedEventHandler = 0
  DeleteObjectFromTag( $tIUIAutomationPropertyChangedEventHandler )
  Exit
EndFunc



Func _UIA_getPropertyValue( $obj, $id )
  Local $vVal
  $obj.GetCurrentPropertyValue( $id, $vVal )
  Return $vVal
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent( $pSelf, $pSender, $iPropertyId, $newValue ) ; Ret: long  Par: ptr;int;variant
  ConsoleWrite( @CRLF & "oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = " & $iPropertyId & ", $newValue = " & $newValue & @CRLF )

  Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  $oSender.AddRef(  )

  ConsoleWrite( "Handle    " & _UIA_getPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) & @CRLF )
  ConsoleWrite( "Name      " & _UIA_getPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF )
  ConsoleWrite( "Class     " & _UIA_getPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF )
  ConsoleWrite( "Ctrl type " & _UIA_getPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF )
  ConsoleWrite( "Ctrl name " & _UIA_getPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF )
  ConsoleWrite( "Value     " & _UIA_getPropertyValue( $oSender, $UIA_LegacyIAccessibleValuePropertyId ) & @CRLF )

  Return $S_OK
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_QueryInterface( $pSelf, $pRIID, $pObj ) ; Ret: long  Par: ptr;ptr*
  Local $sIID = StringFromGUID( $pRIID )
  If $sIID = $sIID_IUnknown Then
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    Return $S_OK
  ElseIf $sIID = $sIID_IUIAutomationPropertyChangedEventHandler Then
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler" & @CRLF )
    DllStructSetData( DllStructCreate( "ptr", $pObj ), 1, $pSelf )
    Return $S_OK
  Else
    ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_QueryInterface: " & $sIID & @CRLF )
    Return $E_NOINTERFACE
  EndIf
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_AddRef( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_AddRef" & @CRLF )
  Return 1
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_Release( $pSelf ) ; Ret: ulong
  ConsoleWrite( "oIUIAutomationPropertyChangedEventHandler_Release" & @CRLF )
  Return 1
EndFunc



Func StringFromGUID( $pGUID )
  Local $aResult = DllCall( "ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40 )
  If @error Then Return SetError( @error, @extended, "" )
  Return SetExtended( $aResult[0], $aResult[2] )
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $fPrint = False, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        If $fPrint Then
            Local $iPar = StringInStr( $sTagPart, ";", 2 ), $t
            If $iPar Then
                $t = "Ret: " & StringLeft( $sTagPart, $iPar - 1 ) & "  " & _
                     "Par: " & StringRight( $sTagPart, StringLen( $sTagPart ) - $iPar )
            Else
                $t = "Ret: " & $sTagPart
            EndIf
            Local $s = "Func " & $sMethod & _
                "( $pSelf ) ; " & $t & @CRLF & _
                "EndFunc" & @CRLF
            ConsoleWrite( $s )
        EndIf
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

Update 2014-01-13

Added the proper function to create custom interface methods to all 4 examples. See post #115. This means that all examples are working including the example with IUIAutomationPropertyChangedEventHandler.

Edited January 13, 2014 by LarsJ

LarsJ · January 12, 2014

I didn't succeed to get events for the IUIAutomationPropertyChangedEventHandler in the post above. As a quick test I have put the code in a DLL. And now I get events (printed in a text file).

This is the code in the DLL:

#define WIN32_LEAN_AND_MEAN

#include <stdio.h>
#include <windows.h>
#include <oleauto.h>
#include <UIAutomation.h>

class CEventHandler : public IUIAutomationPropertyChangedEventHandler
{
  public: 

    CEventHandler();
    ~CEventHandler();

    HRESULT RegisterEventHandler();

    IFACEMETHODIMP QueryInterface(__in REFIID riid, __out PVOID *ppvObj);
    IFACEMETHODIMP_(ULONG) AddRef();
    IFACEMETHODIMP_(ULONG) Release();

    IFACEMETHODIMP HandlePropertyChangedEvent(IUIAutomationElement *sender, PROPERTYID propertyId, VARIANT newValue);

  private:

    void  UIAPrintProperty( IUIAutomationElement *pUIAutomationElement, PROPERTYID propertyId, char *text );
    ULONG _cRef;
    FILE  *f;
};

CEventHandler  *pEventHandler = NULL;


extern "C" __declspec(dllexport) int __cdecl TstFunc()
{
  pEventHandler = new CEventHandler();
  if ( pEventHandler == NULL ) return 1;

  HRESULT hr = pEventHandler->RegisterEventHandler();
  if ( SUCCEEDED(hr) < 0 ) return 1;

  return 0;
}


CEventHandler::CEventHandler() : _cRef(1)
{
  CoInitialize( NULL );
  f = fopen( "UIAEventHandler.txt", "w" );
}

CEventHandler::~CEventHandler()
{
  fclose(f);
  CoUninitialize();
}

HRESULT CEventHandler::RegisterEventHandler()
{
  HRESULT              hr = E_FAIL;
  IUIAutomation        *pUIAutomation;
  IUIAutomationElement *pUIAutomationElement;
  PROPERTYID           properties[] = {UIA_ValueValuePropertyId,
                                       UIA_ToggleToggleStatePropertyId};

  hr = CoCreateInstance(CLSID_CUIAutomation, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pUIAutomation));
  if (SUCCEEDED(hr))
  {
    hr = pUIAutomation->GetRootElement( &pUIAutomationElement );
    if (SUCCEEDED(hr))
    {
      hr = pUIAutomation->AddPropertyChangedEventHandlerNativeArray( pUIAutomationElement, TreeScope_Descendants, NULL, this, properties, sizeof(properties)/sizeof(properties[0]) );
      if (SUCCEEDED(hr)) {
        fprintf( f, "AddPropertyChangedEventHandlerNativeArray\n" );
      }
    }
  }

  return hr;
}

IFACEMETHODIMP CEventHandler::HandlePropertyChangedEvent(IUIAutomationElement *sender, PROPERTYID propertyId, VARIANT newValue)
{
  fprintf( f, "\nHandlePropertyChangedEvent\n" );
  UIAPrintProperty( sender, UIA_NativeWindowHandlePropertyId,     "Handle   " );
  UIAPrintProperty( sender, UIA_NamePropertyId,                   "Name     " );
  UIAPrintProperty( sender, UIA_ClassNamePropertyId,              "Class    " );
  UIAPrintProperty( sender, UIA_ControlTypePropertyId,            "Ctrl type" );
  UIAPrintProperty( sender, UIA_LocalizedControlTypePropertyId,   "Ctrl name" );
  UIAPrintProperty( sender, UIA_LegacyIAccessibleValuePropertyId, "Value    " );
  return S_OK;
}

IFACEMETHODIMP CEventHandler::QueryInterface(__in REFIID riid, __out PVOID *ppvObj)
{
  if (ppvObj == NULL)
  {
    return E_POINTER;
  }

  *ppvObj = NULL;

  HRESULT hr = E_NOINTERFACE;

  if ( riid == __uuidof(IUnknown) )
  {
    fprintf( f, "QueryInterface: IUnknown\n" );
    *ppvObj = static_cast<IUIAutomationPropertyChangedEventHandler*>(this);
    AddRef();
    hr = S_OK;
  }
  else if ( riid == __uuidof(IUIAutomationPropertyChangedEventHandler) )
  {
    fprintf( f, "QueryInterface: IUIAutomationPropertyChangedEventHandler\n" );
    *ppvObj = static_cast<IUIAutomationPropertyChangedEventHandler*>(this);
    AddRef();
    hr = S_OK;
  }
  else
  {
    LPOLESTR lpolestr;
    StringFromCLSID( riid, &lpolestr );
    fwprintf( f, L"QueryInterface: %s\n", lpolestr );
    CoTaskMemFree( lpolestr );
  }

  return hr;
}

IFACEMETHODIMP_(ULONG) CEventHandler::AddRef()
{
  return InterlockedIncrement(&_cRef);
}

IFACEMETHODIMP_(ULONG) CEventHandler::Release()
{
  ULONG cRef = InterlockedDecrement(&_cRef);
  if (cRef == 0)
  {
    delete this;
  }

  return cRef;
}

void CEventHandler::UIAPrintProperty( IUIAutomationElement *pUIAutomationElement, PROPERTYID propertyId, char *text )
{
  VARIANT v;
  pUIAutomationElement->GetCurrentPropertyValue( propertyId, &v );
  if ( v.vt == VT_I4 )
    if ( strcmp( text, "Handle   " ) == 0 )
      fprintf( f, "%s: 0x%p\n", text, v.lVal);
    else
      fprintf( f, "%s: %d\n", text, v.lVal);
  else if ( v.vt == VT_BSTR ) {
    fprintf( f, "%s: ", text );
    fwprintf( f, L"%s\n", v.bstrVal ); }
}

The AutoIt GUI:

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>

Opt( "MustDeclareVars", 1 )

MainFunc()

Func MainFunc()
  Local $dllTest, $aRet
  If @AutoItX64 Then
    $dllTest = DllOpen( "UIAEventHandler-x64.dll" )
  Else
    $dllTest = DllOpen( "UIAEventHandler.dll" )
  EndIf
  GUICreate( "UIAEventHandler", 200, 100, 200, 100 )
  GUISetState( @SW_SHOW )
  $aRet = DllCall( $dllTest, "int:cdecl", "TstFunc" )
  ;ConsoleWrite( "@error = " & @error & @CRLF )
  ;ConsoleWrite( "TstFunc = " & $aRet[0] & @CRLF )
  While GUIGetMsg() <> $GUI_EVENT_CLOSE
  WEnd
  GUIDelete()
  DllClose( $dllTest )
  Exit
EndFunc

And this is the events when I open Internet Explorer, change the URL to AutoIt home, open a text file with Wordpad and types "Test":

QueryInterface: {0000001B-0000-0000-C000-000000000046}
QueryInterface: {00000003-0000-0000-C000-000000000046}
AddPropertyChangedEventHandlerNativeArray
QueryInterface: {0000001B-0000-0000-C000-000000000046}
QueryInterface: IUnknown
QueryInterface: {00000018-0000-0000-C000-000000000046}
QueryInterface: {00000019-0000-0000-C000-000000000046}
QueryInterface: {4C1E39E1-E3E3-4296-AA86-EC938D896E92}
QueryInterface: IUIAutomationPropertyChangedEventHandler
QueryInterface: {1C733A30-2A1C-11CE-ADE5-00AA0044773D}
QueryInterface: IUIAutomationPropertyChangedEventHandler

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : about:blank

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : about:blank

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/site/

HandlePropertyChangedEvent
Handle   : 0x00000000000D0506
Name     : Address and search using Google
Class    : Edit
Ctrl type: 50004
Ctrl name: edit
Value    : http://www.autoitscript.com/site/

HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    :  
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : T 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Te 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Tes 
HandlePropertyChangedEvent
Handle   : 0x00000000000C0432
Name     : Rich Text Window
Class    : RICHEDIT50W
Ctrl type: 50004
Ctrl name: edit
Value    : Test

DLL files:

DLLfiles.7z

junkew · January 12, 2014

@topten:

if IsObj($IE) Then
$sText="search"; indexrelative:=1" ;Is equal to
$oButton=_UIA_getFirstObjectOfElement($IE,"name:=" & $sText, $treescope_subtree)
EndIf

above is functionally equal but technically differently implemented

_UIA_getFirstObjectOfElement will use the MS UIA library straightforward to find on a single property the first object and as such syntax with indexrelative will not work

The UIA_Wrappers add some functionality to walk thru all elements and add enhancements like

a. indexrelative, index, instance properties

b. match on regular expression

c. match easier on multiple properties combined (opposed to the complex logica of and/or conditions the MS UIA has)

Is there any method to calculate the x,y position of found element on the window or on the screen?

_UIA_getPropertyValue($oElement, $UIA_BoundingRectanglePropertyId)

@LarsJ: Regarding eventhandling I will post another example (without dll)

Edited January 12, 2014 by junkew

trancexx · January 12, 2014

LarsJ, I have rewritten that event handling function (several times LOL), so your non-working example should look more like this:

#include "CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Global Const $S_OK = 0x00000000
Global Const $E_NOINTERFACE = 0x80004002
Global Const $sIID_IUnknown = "{00000000-0000-0000-C000-000000000046}"

Global $tIUIAutomationPropertyChangedEventHandler, $oIUIAutomationPropertyChangedEventHandler

Global $oUIAutomation

MainFunc()


Func MainFunc()

    $oIUIAutomationPropertyChangedEventHandler = ObjectFromTag("oIUIAutomationPropertyChangedEventHandler_", $dtagIUIAutomationPropertyChangedEventHandler, $tIUIAutomationPropertyChangedEventHandler)
    If Not IsObj($oIUIAutomationPropertyChangedEventHandler) Then Return

    $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return

    Local $pUIElement
    $oUIAutomation.GetRootElement($pUIElement) ; Desktop
    If Not $pUIElement Then Return

    Local $tPropertyArray = DllStructCreate("int[2]")
    DllStructSetData($tPropertyArray, 1, $UIA_ValueValuePropertyId, 1)
    DllStructSetData($tPropertyArray, 1, $UIA_ToggleToggleStatePropertyId, 2)
    Local $x = $oUIAutomation.AddPropertyChangedEventHandlerNativeArray($pUIElement, $TreeScope_Descendants, 0, $oIUIAutomationPropertyChangedEventHandler, $tPropertyArray, 2)


    HotKeySet("{ESC}", "Quit")

    While Sleep(100)
    WEnd

EndFunc   ;==>MainFunc

Func Quit()
    Exit
EndFunc   ;==>Quit



Func _UIA_getPropertyValue($obj, $id)
    Local $vVal
    $obj.GetCurrentPropertyValue($id, $vVal)
    Return $vVal
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent($pSelf, $pSender, $iPropertyId, $newValue) ; Ret: long  Par: ptr;int;variant
    ConsoleWrite(@CRLF & "oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = " & $iPropertyId & ", $newValue = " & $newValue & @CRLF)

    Local $oSender = ObjCreateInterface($pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    $oSender.AddRef()

    ConsoleWrite("Handle    " & _UIA_getPropertyValue($oSender, $UIA_NativeWindowHandlePropertyId) & @CRLF)
    ConsoleWrite("Name      " & _UIA_getPropertyValue($oSender, $UIA_NamePropertyId) & @CRLF)
    ConsoleWrite("Class     " & _UIA_getPropertyValue($oSender, $UIA_ClassNamePropertyId) & @CRLF)
    ConsoleWrite("Ctrl type " & _UIA_getPropertyValue($oSender, $UIA_ControlTypePropertyId) & @CRLF)
    ConsoleWrite("Ctrl name " & _UIA_getPropertyValue($oSender, $UIA_LocalizedControlTypePropertyId) & @CRLF)
    ConsoleWrite("Value     " & _UIA_getPropertyValue($oSender, $UIA_LegacyIAccessibleValuePropertyId) & @CRLF)

    Return $S_OK
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_QueryInterface($pSelf, $pRIID, $pObj) ; Ret: long  Par: ptr;ptr*
    Local $sIID = StringFromGUID($pRIID)
    If $sIID = $sIID_IUnknown Then
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown" & @CRLF)
        DllStructSetData(DllStructCreate("ptr", $pObj), 1, $pSelf)
        Return $S_OK
    ElseIf $sIID = $sIID_IUIAutomationPropertyChangedEventHandler Then
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler" & @CRLF)
        DllStructSetData(DllStructCreate("ptr", $pObj), 1, $pSelf)
        Return $S_OK
    Else
        ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_QueryInterface: " & $sIID & @CRLF)
        Return $E_NOINTERFACE
    EndIf
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_AddRef($pSelf) ; Ret: ulong
    ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_AddRef" & @CRLF)
    Return 1
EndFunc

Func oIUIAutomationPropertyChangedEventHandler_Release($pSelf) ; Ret: ulong
    ConsoleWrite("oIUIAutomationPropertyChangedEventHandler_Release" & @CRLF)
    Return 1
EndFunc



Func StringFromGUID($pGUID)
    Local $aResult = DllCall("ole32.dll", "int", "StringFromGUID2", "struct*", $pGUID, "wstr", "", "int", 40)
    If @error Then Return SetError(@error, @extended, "")
    Return SetExtended($aResult[0], $aResult[2])
EndFunc



Func ObjectFromTag($sFunctionPrefix, $tagInterface, ByRef $tInterface, $bIsUnknown = Default, $sIID = "{00000000-0000-0000-C000-000000000046}") ; last param is IID_IUnknown by default
    If $bIsUnknown = Default Then $bIsUnknown = True
    Local $sInterface = $tagInterface ; copy interface description
    Local $tagIUnknown = "QueryInterface hresult(ptr;ptr*);" & _
            "AddRef dword();" & _
            "Release dword();"
    ; Adding IUnknown methods
    If $bIsUnknown Then $tagInterface = $tagIUnknown & $tagInterface
    ; Below line is really simple even though it looks super complex. It's just written weird to fit in one line, not to steal your attention
    Local $aMethods = StringSplit(StringReplace(StringReplace(StringReplace(StringReplace(StringTrimRight(StringReplace(StringRegExpReplace(StringRegExpReplace($tagInterface, "\w+\*", "ptr"), "\h*(\w+)\h*(\w+\*?)\h*(\((.*?)\))\h*(;|;*\z)", "$1\|$2;$4" & @LF), ";" & @LF, @LF), 1), "object", "idispatch"), "hresult", "long"), "bstr", "ptr"), "variant", "ptr"), @LF, 3)
    Local $iUbound = UBound($aMethods)
    Local $sMethod, $aSplit, $sNamePart, $aTagPart, $sTagPart, $sRet, $sParams, $hCallback
    ; Allocation
    $tInterface = DllStructCreate("int RefCount;int Size;ptr Object;ptr Methods[" & $iUbound & "];int_ptr Callbacks[" & $iUbound & "];ulong_ptr Slots[16]") ; 16 pointer sized elements more to create space for possible private props
    If @error Then Return SetError(1, 0, 0)
    For $i = 0 To $iUbound - 1
        $aSplit = StringSplit($aMethods[$i], "|", 2)
        If UBound($aSplit) <> 2 Then ReDim $aSplit[2]
        $sNamePart = $aSplit[0]
        $sTagPart = $aSplit[1]
        $sMethod = $sFunctionPrefix & $sNamePart
        $aTagPart = StringSplit($sTagPart, ";", 2)
        $sRet = $aTagPart[0]
        $sParams = StringReplace($sTagPart, $sRet, "", 1)
        $sParams = "ptr" & $sParams
        $hCallback = DllCallbackRegister($sMethod, $sRet, $sParams)
        ConsoleWrite(@error & @CRLF)
        DllStructSetData($tInterface, "Methods", DllCallbackGetPtr($hCallback), $i + 1) ; save callback pointer
        DllStructSetData($tInterface, "Callbacks", $hCallback, $i + 1) ; save callback handle
    Next
    DllStructSetData($tInterface, "RefCount", 1) ; initial ref count is 1
    DllStructSetData($tInterface, "Size", $iUbound) ; number of interface methods
    DllStructSetData($tInterface, "Object", DllStructGetPtr($tInterface, "Methods")) ; Interface method pointers
    Return ObjCreateInterface(DllStructGetPtr($tInterface, "Object"), $sIID, $sInterface, $bIsUnknown) ; pointer that's wrapped into object
EndFunc

Func DeleteObjectFromTag(ByRef $tInterface)
    For $i = 1 To DllStructGetData($tInterface, "Size")
        DllCallbackFree(DllStructGetData($tInterface, "Callbacks", $i))
    Next
    $tInterface = 0
EndFunc

...That's almost exact replica of c++ dll code you posted. Edited January 12, 2014 by trancexx

junkew · January 12, 2014

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\Elwin\Documents\UIA\ex12_propertyevent.au3" /UserParams    
+>19:12:27 Starting AutoIt3Wrapper v.2.1.0.33    Environment(Language:0413  Keyboard:00020409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64)
>Running AU3Check (3.3.10.2)  from:C:\Program Files (x86)\AutoIt3
+>19:12:27 AU3Check ended.rc:0
>Running:(3.3.10.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\Elwin\Documents\UIA\ex12_propertyevent.au3"    
--> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
0
0
0
0
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown
oIUIAutomationPropertyChangedEventHandler_Release
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {0000001B-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000003-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {0000001B-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUnknown
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000018-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {00000019-0000-0000-C000-000000000046}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {4C1E39E1-E3E3-4296-AA86-EC938D896E92}
oIUIAutomationPropertyChangedEventHandler_Release
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler
oIUIAutomationPropertyChangedEventHandler_AddRef
oIUIAutomationPropertyChangedEventHandler_QueryInterface: {1C733A30-2A1C-11CE-ADE5-00AA0044773D}
oIUIAutomationPropertyChangedEventHandler_QueryInterface: IUIAutomationPropertyChangedEventHandler

oIUIAutomationPropertyChangedEventHandler_HandlePropertyChangedEvent: $iPropertyId = 30045, $newValue = 0x001D0000
Handle    1901574
Name      Adres en zoeken met SecureSearch
Class     Edit
Ctrl type 50004
Ctrl name bewerken
Value     http://www.google.nl/

On the example of trancexx I only get one time output (and all I can do is press escape and program exits)

steps i did

1. start example in Scite

2. open Internet Explorer and see above output coming

3. clicking around, closing IE and reopening gives no additonal output (not expected)

4. press escape and program exits (as expected)

trancexx · January 12, 2014

Yes, I forgot to add reference for the sender inside event function.

But I have edited the post 18 minutes ago.

junkew · January 12, 2014

thx did just a minute ago myself indeed this line

Local $oSender = ObjCreateInterface($pSender, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

$oSender.addref()

trancexx · January 12, 2014

So it works for you correctly now?

I also left consolewrite line inside ObjectFromTag() in case there are more types that aren't converted from COM types to callback type correctly because I'm not sure if all the types inside your UDF are covered. It should always print zeroes.

You had some invalid types last time I checked the UDF, but it was really long time ago.

Edited January 12, 2014 by trancexx

junkew · January 12, 2014

yes it works now correctly.

It still could be there are invalid types in the UDF but I think most are fixed (only complaints came from LarsJ since november)

none definitions changed to correct type
LarsJ identified this to be fixed in CUIAutomation2.au3 (lines 532 - 537 in the current version):

"TreeScope hresult(long*);" & _
"TreeScope hresult(long);" & _
"TreeFilter hresult(ptr*);" & _
"TreeFilter hresult(ptr);" & _
"AutomationElementMode hresult(long*);" & _
"AutomationElementMode hresult(long);"

with these lines:

"get_TreeScope hresult(long*);" & _
"put_TreeScope hresult(long);" & _
"get_TreeFilter hresult(ptr*);" & _
"put_TreeFilter hresult(ptr);" & _
"get_AutomationElementMode hresult(long*);" & _
"put_AutomationElementMode hresult(long);"
and LarsJ gave now back some issue on SAFEARRAY currently its in the UDF probably an int and should be a struct*
AddPropertyChangedEventHandlerNativeArray is an example of where safearray was defined incorrectly

IUIAutomation MS framework automate chrome, FF, IE, ....

Recommended Posts

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

junkew

junkew

junkew

Posted Images

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content