Mouse doesn't interact with window

[marko001]

Hi all,

i'm developing a tool to manage my stocks. I use a professional tool given by the bank to check and see variations of my earnings.

I'm not satisfied about datas I can manipulate from the tool so I decided to manually extract datas to manage by myself.

Here is the first and big problem: I can't interact with the GUI:

>>>> Window <<<<
Title:  BNL Professional Stock Manager
Class:  WindowsForms10.Window.8.app.0.22b0d8d
Position:   0, 0
Size:   1920, 1018
Style:  0x170B0000
ExStyle:    0x00050000
Handle: 0x0000000000040680

>>>> Control <<<<
Class:  
Instance:   
ClassnameNN:    
Name:   
Advanced (Class):   
ID: 
Text:   
Position:   
Size:   
ControlClick Coords:    
Style:  
ExStyle:    
Handle: 

>>>> Mouse <<<<
Position:   1325, 734
Cursor ID:  5
Color:  0x767063

>>>> StatusBar <<<<

>>>> ToolsBar <<<<

>>>> Visible Text <<<<

So neither mousemove, mouseclick work inside the programa and pixelchecksum/pixelsearch returns me different values if I relaunch the program.

I tried to look around the forums and, related to games mainly, I found that a possible matter could be the "transparency" of the main window that will forbid me to interact trough Auto-It.

Is there a way to interact to the window, any UDF, or tool that can exactly get it and let me do Control-click or mouseclicks?

BTW Processexist it does work:

$check_1= ProcessExists ( "BNL_PSM.exe" )
Return me the correct handle

But on the contrart Controlgethandle returns me 0

$handle = ControlGetHandle("BNL Professional Stock Manager", "", "[Class:WindowsForms10.Window.8.app.0.22b0d8d]")

Thanks to all the community for the help

[bogQ]

To confirm the gui transparency of the main gui, easier way is to do is to look with autoit win tool magnifier to see if on magnifier you see the gui or the desktop pixels (or window behinde the transparent gui), if you see it and you cant detect gui with magnifier than youl know that your problem is trans.

Eather way, trans shud not influence the gui control detection so ever if you find your way around trans problem (if that is the problem) you will still not have gui controls with autoit win info.

I know what should i do to try to solve control-mouse click problem, and trans problem if it's trans problem.

But i think that you are asking to do something to the program that is apparently designed so that you cant automate it.

and your using ControlGetHandle wrong, if you dont have control ID, how can you use win class as control class?!!

[marko001]

Thats'it, you're right about ControlGetHandle, and I knew it. That shows that the control is not targettable directly from Autoit.

But I think, anyway, that there must be a way (and I hope to with Auto-It) that can let me interact with "theorically" not interact-able cotrols.

And yes, the problem is double: transparency and no-control_ID

Maybe a solution could be memory access, but I know really few about pointers and memory access.

What you think about?

M.

[bogQ]

I think that

If program is constructed to be transparent and not to have readable controls and not to react on other program clicks, there si probably something at his Tos-Eula that you did not read that forbids automating it, for your own or the company protection it doesnt mater.

If that's the situation, you'll probably not get any help from the forum users heare.

[Zedna]

Look here

You can search this forum for WindowsForms to see more related topics.

[marko001]

Well, I tried also using Ranorex Spy to understand which are the controls but I found none.

I tried using the DotNetIdentification.au3 UDF buy without luck.

global $test_hndl
sleep(5000)
$windowTitle = "BNL Professional Stock Manager"
$test_hndl = GetClassNameNNforWindow( $windowTitle, $WindowText = "" )
msgbox(0,"..",$testhdnl)

also using

dim $test_hndl
$windowTitle = "BNL Professional Stock Manager"
$test_hndl = GetClassNameNNforWindow( $windowTitle, $WindowText = "" )
_arraydisplay(test_hndl)

So damn! I can't believe is it so hard to just move mouse and do some clicks there!

I looked around for everything related Windowsform and .Net applications but still seems hard to understand.

The only thing in saw in forum threads a bit similar to this is about a game, Magic I think it's the name, but I have neither time and interest on installing neither trying it.

Any suggestion?

Thanks,

M.

[marko001]

@Zedna

Mate, probably you could help me a bit more to manage it:

I tried almost everything now:

#Include <WinAPI.au3>
_WINAPI_SHOWCURSOR(True)
$hWnd = WinActive("BNL Professional Stock Manager")
$value = _WinAPI_EnableWindow($hWnd)
$class = _WinAPI_GetClassName($hWnd)
_WinAPI_SetFocus($hWnd)
_WinAPI_SetLayeredWindowAttributes($hWnd,0x000000,0)
$process = _WinAPI_GetCurrentProcessID()
WinActivate("[CLASS:WindowsForms10.Window.8.app.0.22b0d8d]")
Opt("MouseCoordMode", 1)
MouseMove(100,100,10)
MouseClick("left",208,606,1,0)

Mouse won't move and won't click but the window comes in foreground.

What's is still missing? Transparency? (that's the second problem I'll manage once I get in touch with the window) I don't think.

Cheers,

M.

[Zedna]

I don't think transparency is source of problems (important is focus) but I'm not sure.

Try these my functions (instead of MouseClick/MouseMove):

Func _MouseClickFast($x, $y)
    $x = $x*65535/@DesktopWidth
    $y = $y*65535/@DesktopHeight

    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_MOVE), $x, $y)
    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_LEFTDOWN), $x, $y)
    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_LEFTUP), $x, $y)
EndFunc 

Func _MouseMoveFast($x, $y)
    $x = $x*65535/@DesktopWidth
    $y = $y*65535/@DesktopHeight

    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_MOVE), $x, $y)
EndFunc

Edited May 29, 2011 by Zedna

[marko001]

Nope, mouse stands still.

Yes I know, transparency comes after, now the problem is to give focus to the window.

[Zedna]

Nope, mouse stands still.

Yes I know, transparency comes after, now the problem is to give focus to the window.

MouseClick/MouseMove don't depend on type of classname/controls, so that's weird.

Is your target application fullscreen? In such case it's source of problems.

[marko001]

Not necessary full screen, I can set a windowed mode, with scrollbars but it's not a problem. If I can get the focus on the window I can use mouseclick&drag to solve scrollbars.

Seems like the application brings a transparent screen in foreground so that clicks can't pass.

[Zedna]

Seems like the application brings a transparent screen in foreground so that clicks can't pass.

If your app has some areas (holes) transparent then if really probably will not work in these transparent areas

But it should work at non transparent areas of application.

You can test it on this my example (from helpfile) for window's regions - there is transparent option too.

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>

; get height of window title and width of window frame - may be different when XP theme is ON/OFF
Global $htit = _WinAPI_GetSystemMetrics($SM_CYCAPTION)
Global $frame = _WinAPI_GetSystemMetrics($SM_CXDLGFRAME)

Local $gui = GUICreate("Test Windows regions", 350, 210)
Local $btn_default = GUICtrlCreateButton("Default region", 100, 30, 150)
Local $btn_round = GUICtrlCreateButton("Round region", 100, 60, 150)
Local $btn_buble = GUICtrlCreateButton("Buble region ", 100, 90, 150)
Local $btn_transparent = GUICtrlCreateButton("Transparent region", 100, 120, 150)
Local $btn_exit = GUICtrlCreateButton("Exit", 100, 150, 150)
GUISetState(@SW_SHOW)

Local $pos = WinGetPos($gui) ; get whole window size (no client size defined in GUICreate)
Global $width = $pos[2]
Global $height = $pos[3]

Local $msg, $rgn
While 1
    $msg = GUIGetMsg()
    Select
        Case $msg = $GUI_EVENT_CLOSE Or $msg = $btn_exit
            ExitLoop

        Case $msg = $btn_default
            $rgn = _WinAPI_CreateRectRgn(0, 0, $width, $height)
            _WinAPI_SetWindowRgn($gui, $rgn)

        Case $msg = $btn_round
            $rgn = _WinAPI_CreateRoundRectRgn(0, 0, $width, $height, $width / 3, $height / 3)
            _WinAPI_SetWindowRgn($gui, $rgn)

        Case $msg = $btn_buble
            Local $rgn1 = _WinAPI_CreateRoundRectRgn(0, 0, $width / 2, $height / 2, $width / 2, $height / 2) ; left-top
            Local $rgn2 = _WinAPI_CreateRoundRectRgn($width / 2, 0, $width, $height / 2, $width / 2, $height / 2) ; right-top
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn(0, $height / 2, $width / 2, $height, $width / 2, $height / 2) ; left-bottom
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn($width / 2, $height / 2, $width, $height, $width / 2, $height / 2) ; right-bottom
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn(10, 10, $width - 10, $height - 10, $width, $height) ; middle
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            _WinAPI_SetWindowRgn($gui, $rgn1)

        Case $msg = $btn_transparent
            _GuiHole($gui, 40, 40, 260, 170)

    EndSelect
WEnd

; make inner transparent area but add controls
Func _GuiHole($h_win, $i_x, $i_y, $i_sizew, $i_sizeh)
    Local $outer_rgn, $inner_rgn, $combined_rgn

    $outer_rgn = _WinAPI_CreateRectRgn(0, 0, $width, $height)
    $inner_rgn = _WinAPI_CreateRectRgn($i_x, $i_y, $i_x + $i_sizew, $i_y + $i_sizeh)
    $combined_rgn = _WinAPI_CreateRectRgn(0, 0, 0, 0)
    _WinAPI_CombineRgn($combined_rgn, $outer_rgn, $inner_rgn, $RGN_DIFF)
    _WinAPI_DeleteObject($outer_rgn)
    _WinAPI_DeleteObject($inner_rgn)
    _AddCtrlRegion($combined_rgn, $btn_default)
    _AddCtrlRegion($combined_rgn, $btn_round)
    _AddCtrlRegion($combined_rgn, $btn_buble)
    _AddCtrlRegion($combined_rgn, $btn_transparent)
    _AddCtrlRegion($combined_rgn, $btn_exit)
    _WinAPI_SetWindowRgn($h_win, $combined_rgn)
EndFunc   ;==>_GuiHole

; add control's area to given region
; respecting also window title/frame sizes
Func _AddCtrlRegion($full_rgn, $ctrl_id)
    Local $ctrl_pos, $ctrl_rgn

    $ctrl_pos = ControlGetPos($gui, "", $ctrl_id)
    $ctrl_rgn = _WinAPI_CreateRectRgn($ctrl_pos[0] + $frame, $ctrl_pos[1] + $htit + $frame, _
            $ctrl_pos[0] + $ctrl_pos[2] + $frame, $ctrl_pos[1] + $ctrl_pos[3] + $htit + $frame)
    _WinAPI_CombineRgn($full_rgn, $full_rgn, $ctrl_rgn, $RGN_OR)
    _WinAPI_DeleteObject($ctrl_rgn)
EndFunc   ;==>_AddCtrlRegion

Edited May 29, 2011 by Zedna

[marko001]

I'm at the beach atm, reading this trough iphone. Back to you in a few hrs once at home. Big thanks for the great effort anyway.

M.

[marko001]

Ok, Back and I tried.

Well since I saw, as I told you, so many threads of a card game (?!) with analog problems I decide to see what is that.

First I tried contacting customer care of the bank but on Sunday it's hard, I'll retry tomorrow morning to see if they can help me a bit tech-side.

From the programming side well, I downloaded that game (i'm not able to create an account but I think it's not important) and launched it.

Oh! Same thing as in my Trade Exchange software (and same of my Vodafone Mobile Connect Application).

So, since i can't provide you the stock exchange software (unless you are italian and have a BNL bank account, but in that case you can find the file to DL inside your personal stocking page) you can try and see what i mean using that login screen of that Magic program (http://www.wizards.com/Magic/Digital/MagicOnline.aspx). Just DL and run it. Unless you like that game, forget about registration, just try to movemouse inside that login form and you'll understand my problems.

Later mate,

M.

[Zedna]

Games automatisation is forbidden here! Read forum rules!

So we can help you only with your stock manager.

[marko001]

Infact I read the 1st post here, didn't know about that and it's not my interest. Let's just go on on the Stock Manager.

[marko001]

Partially solved it. It's enough to Run the application inside the main autoit program.

I have, as thought at beginning, the problem of Pixelchecksum changing time to time for same region.

Maybe this is due to "semitrasparent" window?

How can this be solved?

M.

[Zedna]

You can change transparent value of target window by your script to zero = non transparent

and then do other stuff.

Look at WinSetTrans()

[marko001]

Maybe not necessary, I just need a technical info, that could explain me several things:

Pixelcheckusm(bla,bla,bla,bla) returns a value. This value is "something like" a Dynamic Address? I mean if I relaunch X times the program, for same section I will get different values? Or they MUST be same value unless something change?

M.

[Zedna]

Or they MUST be same value unless something change?

Yes.