Help sending clicks to window (_WinAPI_PostMessage)

[MaximusCZ]

Hello all.
I have a window "Simulation diagram", (that is subwindow of main program).

I succesfully open the drop down and select line I need:

ControlCommand("Simulation diagram", "", "ComboBox1", "ShowDropDown")
$hWnd = WinGetHandle("[CLASS:ComboLBox]")

_WinAPI_PostMessage($hWnd, $WM_MOUSEMOVE, 0, _MakeLong(10, 60))
_WinAPI_PostMessage($hWnd, $WM_LBUTTONDOWN, 0, _MakeLong(10, 60)) ;Click Acceleration - absolute
_WinAPI_PostMessage($hWnd, $WM_LBUTTONUP, 0, _MakeLong(10, 60))   ;Click Acceleration - absolute

That works. But then I need to click the small button A1 next to the drop down.
When I do 

$hWnd = ControlGetHandle("Simulation diagram", "", "ABSActiveBarDockWnd1")
_WinAPI_PostMessage($hWnd, $WM_MOUSEMOVE, 0, _MakeLong(295, 19))

I can see the button indicates a mouse is over it 
But clicking doesnt work.. Unless, I actually move my real mouse over it!

while 1
    _WinAPI_PostMessage($hWnd, $WM_LBUTTONDOWN, 0, _MakeLong(295, 19)) ;
    Sleep(50)
    _WinAPI_PostMessage($hWnd, $WM_LBUTTONUP, 0, _MakeLong(295, 19))   ;
    Sleep(500)
WEnd

When I let it run, but position the mouse NOT on "A1" button, I can see the button being hovered for 50ms (changes to half sec hover when I change sleep 50 to 500), but it never gets clicked. If I, while the loop is running, move the mouse over the button, it gets clicked every half a second, as it should. But I need the actual clicking to occur even when I am not actually mouse hovering over the button...


I tried to add MOUSEMOVE message above, MOUSEHOVER above, neither helps, even if I play with sleeps (or delete them).
Anyone got any idea why the click gets trough only when I actually hover over the button, but fails when I am not hovering the button? (even after I send "MOUSEHOVER" right before clicking)..

When I change "L"Button to "R"Button, the menu opens and closes at the real mouse coordinate, not where the PostMessage is specifying.

This works exactly the same. Works when mouse is over it, not when mouse is elsewhere..

ControlClick("Simulation diagram", "", "ABSActiveBarDockWnd1", "left", 1, 295, 19)

 

 

For completeness: 

Func _MakeLong($LoWord, $HiWord)
    Return BitOR($HiWord * 0x10000, BitAND($LoWord, 0xFFFF))
EndFunc   ;==>_MakeLong

 

 

Edited March 10, 2022 by MaximusCZ

[junkew]

try iuiautomation see faq 31

 

[MaximusCZ]

Thanks for pointer.


I tried to identify the object with given tools:
Simplespy: (or IUIautomation)
The CTRL+W action on a button of interest highlights the control of the entire bar the button is part of, same as default AU3info.exe (shown in pic 1)
The output can be seen in spoiler under, when trying to highlight/setfocus oP3, 4 or 5 the whole application crashes.
I dont think its targeting the control in more speccific way compared to AU3info anyways.

Spoiler

expandcollapse popup

Mouse position is retrieved 1740-421
At least we have an element title: [DockTop] class: [ABSActiveBarDockWnd]

Having the following values for all properties: 
Title is: <DockTop> Class   := <ABSActiveBarDockWnd>    controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    1450;402;757;27
*** Parent Information top down ***
4: Title is: <EcoScreen 3DOnSite>   Class   := <ThunderRT6MDIForm>  controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    1346;262;874;614
"Title:=EcoScreen 3DOnSite;controltype:=UIA_WindowControlTypeId;class:=ThunderRT6MDIForm""  
3: Title is: <Simulation diagram>   Class   := <ABSABFloatWin>  controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    1436;365;786;640
"Title:=Simulation diagram;controltype:=UIA_WindowControlTypeId;class:=ABSABFloatWin""  
2: Title is: <#frmSimChart> Class   := <ThunderRT6FormDC>   controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    1447;399;763;595
"Title:=#frmSimChart;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6FormDC""   
1: Title is: <> Class   := <ThunderRT6UserControlDC>    controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    1450;402;757;589
"Title:=;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6UserControlDC""    
0: Title is: <> Class   := <ABSDynaBarCls>  controltype:= <UIA_PaneControlTypeId>   ,<50033>    , (0000C371)    1450;402;757;589
"Title:=;controltype:=UIA_PaneControlTypeId;class:=ABSDynaBarCls""  


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=EcoScreen 3DOnSite;controltype:=UIA_WindowControlTypeId;class:=ThunderRT6MDIForm")    ;EcoScreen 3DOnSite
_UIA_setVar("oP2","Title:=Simulation diagram;controltype:=UIA_WindowControlTypeId;class:=ABSABFloatWin")    ;Simulation diagram
_UIA_setVar("oP3","Title:=#frmSimChart;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6FormDC") ;#frmSimChart
_UIA_setVar("oP4","Title:=;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6UserControlDC")  ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_PaneControlTypeId;class:=ABSDynaBarCls")    ;

_UIA_setVar("DockTop.mainwindow","title:=DockTop;classname:=ABSActiveBarDockWnd")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")

_UIA_action("DockTop.mainwindow","setfocus")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=EcoScreen 3DOnSite;controltype:=UIA_WindowControlTypeId;class:=ThunderRT6MDIForm", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=Simulation diagram;controltype:=UIA_WindowControlTypeId;class:=ABSABFloatWin", $treescope_children)    
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=#frmSimChart;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6FormDC", $treescope_children) 
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_PaneControlTypeId;class:=ThunderRT6UserControlDC", $treescope_children)  
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=ABSDynaBarCls", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("DockTop.mainwindow","title:=DockTop;classname:=ABSActiveBarDockWnd")
_UIA_action("DockTop.mainwindow","setfocus")


*** Detailed properties of the highlighted element ***
UIA_title:= <DockTop>
UIA_text:= <DockTop>
UIA_regexptitle:= <DockTop>
UIA_class:= <ABSActiveBarDockWnd>
UIA_regexpclass:= <ABSActiveBarDockWnd>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <656128>
UIA_RuntimeId:= <42;656128>
UIA_BoundingRectangle:= <1450;402;757;27>
UIA_ProcessId:= <11612>
UIA_ControlType:= <50033>
UIA_LocalizedControlType:= <pane>
UIA_Name:= <DockTop>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_ClassName:= <ABSActiveBarDockWnd>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <656128>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Win32>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <DockTop>
UIA_LegacyIAccessibleRole:= <10>
UIA_LegacyIAccessibleState:= <1048576>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:11060,providerId:0xA0300 Main:Nested [pid:11612,providerId:0xA0300 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>
UIA_OptimizeForVisualContent:= <False>
UIA_IsObjectModelPatternAvailable:= <False>
UIA_AnnotationAnnotationTypeId:= <60000>
UIA_IsAnnotationPatternAvailable:= <False>
UIA_IsTextPattern2Available:= <False>
UIA_StylesStyleId:= <0>
UIA_StylesFillColor:= <0>
UIA_StylesFillPatternColor:= <0>
UIA_IsStylesPatternAvailable:= <False>
UIA_IsSpreadsheetPatternAvailable:= <False>
UIA_IsSpreadsheetItemPatternAvailable:= <False>
UIA_Transform2CanZoom:= <False>
UIA_IsTransformPattern2Available:= <False>
UIA_LiveSetting:= <0>
UIA_IsTextChildPatternAvailable:= <False>
UIA_IsDragPatternAvailable:= <False>
UIA_DragIsGrabbed:= <False>
UIA_IsDropTargetPatternAvailable:= <False>
UIA_Transform2ZoomLevel:= <1>
UIA_Transform2ZoomMinimum:= <1>
UIA_Transform2ZoomMaximum:= <1>
UIA_IsTextEditPatternAvailable:= <False>
UIA_IsPeripheral:= <False>
UIA_IsCustomNavigationPatternAvailable:= <False>
UIA_PositionInSet:= <0>
UIA_SizeOfSet:= <0>
UIA_Level:= <0>
UIA_LandmarkType:= <0>
UIA_FillType:= <0>
UIA_VisualEffects:= <0>
UIA_IsSelectionPattern2Available:= <False>
UIA_Selection2ItemCount:= <0>
UIA_HeadingLevel:= <80050>
UIA_IsDialog:= <False>

 

CV: 
Again, focuses the whole bar, doesnt recognize the buttons themselves.

The control info shows basically the same info as AU3info

Showing/hiding the individual controls, theres one that hides/shows the whole bar, but none to handle it with more granularity
 

UIASpy:
Again, captures the control of the bar, doesnt show any more childrens

The whole output here:

Spoiler

JavaUI gives me error of Java not installed, but I would bet my socks that java isnt what this program is written in anyway

 

 

Edited March 15, 2022 by MaximusCZ

[InnI]

@MaximusCZ

Try to send $WM_SETCURSOR before $WM_LBUTTONDOWN and $WM_LBUTTONUP.

Or use _MouseLeftClick() from this post

 

[junkew]

With freeware then you most likely will not come further then to calculate position's  either absolute or some custom logic to click(x%,y%) relative to size of the element clicked

Advanced tricks can be to move slowly with a mousemove and see if there is a tooltip event flowing around you can catch and recognize but all no standard logic.

With commercial tools I would advice to try testcomplete to see if it recognizes more

 

[junkew]

Try also inspect.exe in msaa mode. Inspect.exe is part of windows sdk

[MaximusCZ]

@InnI Sadly, works exactly the same as all the tries above. Works fine when actual mouse is over the button, but not when mouse is elsewhere

@junkew I am clicking the button trough window-relative coordinates already, the button is at the same place everytime, so thats not the issue. 
Could you elaborate on catching tooltip event?

When I tried the inspect.exe, I can get it to draw rectangles (and provide info) for when I click different areas of the window (like the drop down, slider, graph area). 
But clicking the buttons of interest it doesnt redraw the rectangle at all, giving me this info: (notice its found by mousemove, not mouseclick, even when I try clicking..)

[junkew]

Welcome to the powerfull world 😉  of UIA automation me and @LarsJ made many UIA examples in the forum

Your controls are probably from the VB6 world and can be hard to deal with (worst case is detecting by moving mouse and observing pixels/bitmap area is changing so you know when you reach a new button) unfortunately I do not remember where I stored a function to do exactly that.

This monitor will help you to see if there are any tooltip or other events from UIA you can catch when hovering over a button (still then you are unable to determine easily the button width/height)

Something like below can help (combined with UIA element size information) to be less dependent on screenwidth

testpercentagemove("50%","50%")

func testpercentagemove($x,$y)
    if stringinstr($x,"%") Then
        $realX=stringreplace($x,"%","") * (@DesktopWidth / 100)
        $realy=stringreplace($y,"%","") * (@DesktopHeight / 100)
        
        mousemove($realX ,$realy)
    EndIf
EndFunc

Maybe this helps?

 

 

Edited March 16, 2022 by junkew

[MaximusCZ]

I tried UI Automation events, and ran both MTWEvents and UIAEHEvents. 
The MTW records me opening menus to get into my window of interest, but not any changes inside of it.

I ran UIAEHEvents with all windows, all event handlers, event IDs and Property IDs selected. Tried separately for every Windows version option under "Options" menu. It is giving me a lot of information when changing focus and playing with the dropdown, but not a single message is fired when I mousehover/mouseclick on any button of interest.. The program still responds by showing the button is hovered and can be clicked to toggle, but UIAEHEvents reports back nothing. 

Regarding Find BMP in another BMP: I have no trouble finding the button, its always in the same position relative to "Simulation Diagram" window, and while hardcoding the click coordinates isnt ideal, I would argue its okay for my usecase. The issue lies in my unability to click the button without moving mouse cursor there..

Edited March 16, 2022 by MaximusCZ

[junkew]

Maybe hiding it and moving it back to where your mouse was coming from?

https://www.autoitscript.com/autoit3/docs/libfunctions/_WinAPI_ShowCursor.htm

[MaximusCZ]

Oh yes. I was trying to find a way to interface with the window in a more controlled way.
Over the years I tried saving bmps of buttons of interest, which breaks down once resolution/scaling changes.
I tried going pixel-by-pixel to look for different color indicating window delimeter, and hardcoding a path to arrive at the button, but that broke in more ways I can remember.
In the end, window/control relative coordinates with commands to set focus to certain controls (to make sure my target is actually visible), proved to be the most robust, but even that I have to manually adapt every once in a while. 

+ I was hoping the macro do certain tasks in background, having to have a window on screen and highjacking mouse kinda sucks.

Any ideas on how to control this without pointing real mouse cursor over it is highly appreciated

Edited March 17, 2022 by MaximusCZ

[junkew]

Hooking functions could be an option but normally is not worth the effort. 

Google for easyhook and deviare com

[MaximusCZ]

Thanks for suggestion, altho from a quick scan it looks way above my ability.
If anyone with knowhow and a idea of how to capture and replicate the event feels like trying, we could arrange some cooperation.
This is the screen of process exe in Deviare

Spoiler

 

In the meantime, if anyone has other ideas, please share.

 

Edited March 18, 2022 by MaximusCZ

[MaximusCZ]

Fiddled a bit with Deviare
Managed to catch an event when clicking on two buttons marked in red.

Left and Right button in red are mutually exclusive - selecting one deselects the other.
Left button button fires the event as shown and right button fires identical one but with "uCheck 0x8"

Some of following buttons are just regular buttons like "reset camera"(the house one) and those are not catched.
The buttons im interested are in subwindow thats not shown here, also acting as regular buttons (more than one can be selected at once) and are also not catched in this hook.

From time to time my hooked application crashes.
After restart, this CheckMenuItem still works (on those two buttons only)
The HMENU - hMenu - value changed, but UINT - uIDCheckItem and UINT - uCheck values are the same
also noteworthy that the event is fired even when using a hotkey (default in program, one of few) to switch between those two items

BUT theres also a button that swaps orthographic/perspective camera (single button that changes on click, the "cube" on the right on screenshot), and clicking this button fires Two events, identical to the one shown in picture, first with 0x0 and second with 0x8 as their "uCheck" value. The resulting events from single click on "change perspective" button are identical to two clicks on "select cursor" and "select drag"... 😕

 

Edited March 18, 2022 by MaximusCZ

[Danyfirex]

Can I have access to the application? is it free? could you give me remote access. Send me a PM.

 

Saludos

[junkew]

Maybe if you send the messages from a toolbar

https://docs.microsoft.com/en-us/windows/win32/controls/bumper-toolbar-control-reference-messages

Then I would suggest to first get TB_BUTTONCOUNT message working

$buttonCount=DLLCall("user32.dll","int","SendMessage", "hwnd", $hWnd, "int", $TB_BUTTONCOUNT, "int", 0, "int", 0)

And not sure but I think you have to use openprocess to be inside the other process of the other .exe running. before you can do things like this

DLLCall("user32.dll","int","SendMessage", "hwnd", $toolbarHwnd, "int", $TB_GETITEMRECT, "int", $index, "ptr",$lpData[0])

 

All doable but in general I feel then switch to a commercial automation tool that supports this logic.

Over here some example code to how this potentially can be done

 

[ad777]

@maximus

dose this works?

_WinAPI_PostMessage($hWnd, $WM_MOUSEMOVE, 0, _MakeLong(295, 19))
while 1
    _WinAPI_PostMessage($hWnd, $WM_LBUTTONDBLCLK, 0, _MakeLong(295, 19))
    Sleep(100)
    _WinAPI_PostMessage($hWnd, $WM_LBUTTONUP, 0, _MakeLong(295, 19))
    Sleep(100)
WEnd

Func _MakeLong($LoWord, $HiWord)
    Return BitOR($HiWord * 0x10000, BitAND($LoWord, 0xFFFF))
EndFunc   ;==>_MakeLong

 

Edited March 19, 2022 by ad777

[MaximusCZ]

@junkew Sadly the buttonCount variable is empty after calling the DLL. I added $TB_BUTTONCOUNT value of 0x0418 to WindowsConstants.au3.
Isnt the GETITEMRECT basically the same way all those spy tools getting their own data about controls?
Also since the buttons were not really visible until now, I have a bad feeling those buttons might be custom coded instead of using stubs like "toolbar" windows would expect..
@ad777 Works to the same effect as my gif in very first post. The button is showing itself being hovered over for 100ms and then not for another 100ms. Moving my mouse over it makes it click as needed, but once the mouse leaves the button, it goes back to the hovering effect. 

@Danyfirex While giving you the installation package would probably be fine, the app needs a hardware bound licencekey and backup of data to load correctly, for which I could get into big trouble if shared. The closest I can get you to it would be remote desktop - is teamviewer ok?

[junkew]

• Did you try inspect.exe in MSAA mode? It could be that its then shown with its childs.

Your application seems its origin in VB6 (ThunderRT6* controls like ThunderRT6CommandButton). And yes in general getitemrect is the generic logic most tools do this. Maybe this book can give you some insights https://books.google.nl/books/about/Subclassing_and_Hooking_with_Visual_Basi.html

But more advanced tools inject dll into the process of the running application and as such can reveal more on the inner workings by directly getting access to the properties of the object created. You could try Testcomplete trial version to see what they reveal on this application.

Maybe its potentially possible to sendmessage a WM_GETOBJECT so you get a reference to a com/activex object of some kind of toolbar activex control and then you can read it directly.
Not sure if there are examples around for AutoIt but here you have one for VBA https://www.mrexcel.com/board/threads/referecing-textbox-from-another-workbook.1131943/

 

 

Edited March 21, 2022 by junkew

[ad777]

@MaximusCZ

maybe the program detect mouse cursor.(for sure)

if mouse .... then click else....

Edited March 21, 2022 by ad777