Sign in to follow this  
Followers 0
marko001

Mouse doesn't interact with window

20 posts in this topic

Hi all,

i'm developing a tool to manage my stocks. I use a professional tool given by the bank to check and see variations of my earnings.

I'm not satisfied about datas I can manipulate from the tool so I decided to manually extract datas to manage by myself.

Here is the first and big problem: I can't interact with the GUI:

>>>> Window <<<<
Title:  BNL Professional Stock Manager
Class:  WindowsForms10.Window.8.app.0.22b0d8d
Position:   0, 0
Size:   1920, 1018
Style:  0x170B0000
ExStyle:    0x00050000
Handle: 0x0000000000040680

>>>> Control <<<<
Class:  
Instance:   
ClassnameNN:    
Name:   
Advanced (Class):   
ID: 
Text:   
Position:   
Size:   
ControlClick Coords:    
Style:  
ExStyle:    
Handle: 

>>>> Mouse <<<<
Position:   1325, 734
Cursor ID:  5
Color:  0x767063

>>>> StatusBar <<<<

>>>> ToolsBar <<<<

>>>> Visible Text <<<<

So neither mousemove, mouseclick work inside the programa and pixelchecksum/pixelsearch returns me different values if I relaunch the program.

I tried to look around the forums and, related to games mainly, I found that a possible matter could be the "transparency" of the main window that will forbid me to interact trough Auto-It.

Is there a way to interact to the window, any UDF, or tool that can exactly get it and let me do Control-click or mouseclicks?

BTW Processexist it does work:

$check_1= ProcessExists ( "BNL_PSM.exe" )
Return me the correct handle

But on the contrart Controlgethandle returns me 0

$handle = ControlGetHandle("BNL Professional Stock Manager", "", "[Class:WindowsForms10.Window.8.app.0.22b0d8d]")

Thanks to all the community for the help

Share this post


Link to post
Share on other sites



To confirm the gui transparency of the main gui, easier way is to do is to look with autoit win tool magnifier to see if on magnifier you see the gui or the desktop pixels (or window behinde the transparent gui), if you see it and you cant detect gui with magnifier than youl know that your problem is trans.

Eather way, trans shud not influence the gui control detection so ever if you find your way around trans problem (if that is the problem) you will still not have gui controls with autoit win info.

I know what should i do to try to solve control-mouse click problem, and trans problem if it's trans problem.

But i think that you are asking to do something to the program that is apparently designed so that you cant automate it.

and your using ControlGetHandle wrong, if you dont have control ID, how can you use win class as control class?!!


TCP server and client - Learning about TCP servers and clients connection
Au3 oIrrlicht - Irrlicht project
Au3impact - Another 3D DLL game engine for autoit. (3impact 3Drad related)



460px-Thief-4-temp-banner.jpg
There are those that believe that the perfect heist lies in the preparation.
Some say that it’s all in the timing, seizing the right opportunity. Others even say it’s the ability to leave no trace behind, be a ghost.

 

Share this post


Link to post
Share on other sites

Thats'it, you're right about ControlGetHandle, and I knew it. That shows that the control is not targettable directly from Autoit.

But I think, anyway, that there must be a way (and I hope to with Auto-It) that can let me interact with "theorically" not interact-able cotrols.

And yes, the problem is double: transparency and no-control_ID

Maybe a solution could be memory access, but I know really few about pointers and memory access.

What you think about?

M.

Share this post


Link to post
Share on other sites

I think that

If program is constructed to be transparent and not to have readable controls and not to react on other program clicks, there si probably something at his Tos-Eula that you did not read that forbids automating it, for your own or the company protection it doesnt mater.

If that's the situation, you'll probably not get any help from the forum users heare.


TCP server and client - Learning about TCP servers and clients connection
Au3 oIrrlicht - Irrlicht project
Au3impact - Another 3D DLL game engine for autoit. (3impact 3Drad related)



460px-Thief-4-temp-banner.jpg
There are those that believe that the perfect heist lies in the preparation.
Some say that it’s all in the timing, seizing the right opportunity. Others even say it’s the ability to leave no trace behind, be a ghost.

 

Share this post


Link to post
Share on other sites

Well, I tried also using Ranorex Spy to understand which are the controls but I found none.

I tried using the DotNetIdentification.au3 UDF buy without luck.

global $test_hndl
sleep(5000)
$windowTitle = "BNL Professional Stock Manager"
$test_hndl = GetClassNameNNforWindow( $windowTitle, $WindowText = "" )
msgbox(0,"..",$testhdnl)

also using

dim $test_hndl
$windowTitle = "BNL Professional Stock Manager"
$test_hndl = GetClassNameNNforWindow( $windowTitle, $WindowText = "" )
_arraydisplay(test_hndl)

So damn! I can't believe is it so hard to just move mouse and do some clicks there!

I looked around for everything related Windowsform and .Net applications but still seems hard to understand.

The only thing in saw in forum threads a bit similar to this is about a game, Magic I think it's the name, but I have neither time and interest on installing neither trying it.

Any suggestion?

Thanks,

M.

Share this post


Link to post
Share on other sites

@Zedna

Mate, probably you could help me a bit more to manage it:

I tried almost everything now:

#Include <WinAPI.au3>
_WINAPI_SHOWCURSOR(True)
$hWnd = WinActive("BNL Professional Stock Manager")
$value = _WinAPI_EnableWindow($hWnd)
$class = _WinAPI_GetClassName($hWnd)
_WinAPI_SetFocus($hWnd)
_WinAPI_SetLayeredWindowAttributes($hWnd,0x000000,0)
$process = _WinAPI_GetCurrentProcessID()
WinActivate("[CLASS:WindowsForms10.Window.8.app.0.22b0d8d]")
Opt("MouseCoordMode", 1)
MouseMove(100,100,10)
MouseClick("left",208,606,1,0)

Mouse won't move and won't click but the window comes in foreground.

What's is still missing? Transparency? (that's the second problem I'll manage once I get in touch with the window) I don't think.

Cheers,

M.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

I don't think transparency is source of problems (important is focus) but I'm not sure.

Try these my functions (instead of MouseClick/MouseMove):

Func _MouseClickFast($x, $y)
    $x = $x*65535/@DesktopWidth
    $y = $y*65535/@DesktopHeight

    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_MOVE), $x, $y)
    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_LEFTDOWN), $x, $y)
    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_LEFTUP), $x, $y)
EndFunc 

Func _MouseMoveFast($x, $y)
    $x = $x*65535/@DesktopWidth
    $y = $y*65535/@DesktopHeight

    _WinAPI_Mouse_Event(BitOR($MOUSEEVENTF_ABSOLUTE, $MOUSEEVENTF_MOVE), $x, $y)
EndFunc
Edited by Zedna

Share this post


Link to post
Share on other sites

Nope, mouse stands still.

Yes I know, transparency comes after, now the problem is to give focus to the window.

Share this post


Link to post
Share on other sites

Nope, mouse stands still.

Yes I know, transparency comes after, now the problem is to give focus to the window.

MouseClick/MouseMove don't depend on type of classname/controls, so that's weird.

Is your target application fullscreen? In such case it's source of problems.

Share this post


Link to post
Share on other sites

Not necessary full screen, I can set a windowed mode, with scrollbars but it's not a problem. If I can get the focus on the window I can use mouseclick&drag to solve scrollbars.

Seems like the application brings a transparent screen in foreground so that clicks can't pass.

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Seems like the application brings a transparent screen in foreground so that clicks can't pass.

If your app has some areas (holes) transparent then if really probably will not work in these transparent areas

But it should work at non transparent areas of application.

You can test it on this my example (from helpfile) for window's regions - there is transparent option too.

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>

; get height of window title and width of window frame - may be different when XP theme is ON/OFF
Global $htit = _WinAPI_GetSystemMetrics($SM_CYCAPTION)
Global $frame = _WinAPI_GetSystemMetrics($SM_CXDLGFRAME)

Local $gui = GUICreate("Test Windows regions", 350, 210)
Local $btn_default = GUICtrlCreateButton("Default region", 100, 30, 150)
Local $btn_round = GUICtrlCreateButton("Round region", 100, 60, 150)
Local $btn_buble = GUICtrlCreateButton("Buble region ", 100, 90, 150)
Local $btn_transparent = GUICtrlCreateButton("Transparent region", 100, 120, 150)
Local $btn_exit = GUICtrlCreateButton("Exit", 100, 150, 150)
GUISetState(@SW_SHOW)

Local $pos = WinGetPos($gui) ; get whole window size (no client size defined in GUICreate)
Global $width = $pos[2]
Global $height = $pos[3]

Local $msg, $rgn
While 1
    $msg = GUIGetMsg()
    Select
        Case $msg = $GUI_EVENT_CLOSE Or $msg = $btn_exit
            ExitLoop

        Case $msg = $btn_default
            $rgn = _WinAPI_CreateRectRgn(0, 0, $width, $height)
            _WinAPI_SetWindowRgn($gui, $rgn)

        Case $msg = $btn_round
            $rgn = _WinAPI_CreateRoundRectRgn(0, 0, $width, $height, $width / 3, $height / 3)
            _WinAPI_SetWindowRgn($gui, $rgn)

        Case $msg = $btn_buble
            Local $rgn1 = _WinAPI_CreateRoundRectRgn(0, 0, $width / 2, $height / 2, $width / 2, $height / 2) ; left-top
            Local $rgn2 = _WinAPI_CreateRoundRectRgn($width / 2, 0, $width, $height / 2, $width / 2, $height / 2) ; right-top
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn(0, $height / 2, $width / 2, $height, $width / 2, $height / 2) ; left-bottom
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn($width / 2, $height / 2, $width, $height, $width / 2, $height / 2) ; right-bottom
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            $rgn2 = _WinAPI_CreateRoundRectRgn(10, 10, $width - 10, $height - 10, $width, $height) ; middle
            _WinAPI_CombineRgn($rgn1, $rgn1, $rgn2, $RGN_OR)
            _WinAPI_DeleteObject($rgn2)
            _WinAPI_SetWindowRgn($gui, $rgn1)

        Case $msg = $btn_transparent
            _GuiHole($gui, 40, 40, 260, 170)

    EndSelect
WEnd

; make inner transparent area but add controls
Func _GuiHole($h_win, $i_x, $i_y, $i_sizew, $i_sizeh)
    Local $outer_rgn, $inner_rgn, $combined_rgn

    $outer_rgn = _WinAPI_CreateRectRgn(0, 0, $width, $height)
    $inner_rgn = _WinAPI_CreateRectRgn($i_x, $i_y, $i_x + $i_sizew, $i_y + $i_sizeh)
    $combined_rgn = _WinAPI_CreateRectRgn(0, 0, 0, 0)
    _WinAPI_CombineRgn($combined_rgn, $outer_rgn, $inner_rgn, $RGN_DIFF)
    _WinAPI_DeleteObject($outer_rgn)
    _WinAPI_DeleteObject($inner_rgn)
    _AddCtrlRegion($combined_rgn, $btn_default)
    _AddCtrlRegion($combined_rgn, $btn_round)
    _AddCtrlRegion($combined_rgn, $btn_buble)
    _AddCtrlRegion($combined_rgn, $btn_transparent)
    _AddCtrlRegion($combined_rgn, $btn_exit)
    _WinAPI_SetWindowRgn($h_win, $combined_rgn)
EndFunc   ;==>_GuiHole

; add control's area to given region
; respecting also window title/frame sizes
Func _AddCtrlRegion($full_rgn, $ctrl_id)
    Local $ctrl_pos, $ctrl_rgn

    $ctrl_pos = ControlGetPos($gui, "", $ctrl_id)
    $ctrl_rgn = _WinAPI_CreateRectRgn($ctrl_pos[0] + $frame, $ctrl_pos[1] + $htit + $frame, _
            $ctrl_pos[0] + $ctrl_pos[2] + $frame, $ctrl_pos[1] + $ctrl_pos[3] + $htit + $frame)
    _WinAPI_CombineRgn($full_rgn, $full_rgn, $ctrl_rgn, $RGN_OR)
    _WinAPI_DeleteObject($ctrl_rgn)
EndFunc   ;==>_AddCtrlRegion
Edited by Zedna

Share this post


Link to post
Share on other sites

I'm at the beach atm, reading this trough iphone. Back to you in a few hrs once at home. Big thanks for the great effort anyway.

M.

Share this post


Link to post
Share on other sites

Ok, Back and I tried.

Well since I saw, as I told you, so many threads of a card game (?!) with analog problems I decide to see what is that.

First I tried contacting customer care of the bank but on Sunday it's hard, I'll retry tomorrow morning to see if they can help me a bit tech-side.

From the programming side well, I downloaded that game (i'm not able to create an account but I think it's not important) and launched it.

Oh! Same thing as in my Trade Exchange software (and same of my Vodafone Mobile Connect Application).

So, since i can't provide you the stock exchange software (unless you are italian and have a BNL bank account, but in that case you can find the file to DL inside your personal stocking page) you can try and see what i mean using that login screen of that Magic program (http://www.wizards.com/Magic/Digital/MagicOnline.aspx). Just DL and run it. Unless you like that game, forget about registration, just try to movemouse inside that login form and you'll understand my problems.

Later mate,

M.

Share this post


Link to post
Share on other sites

Infact I read the 1st post here, didn't know about that and it's not my interest. Let's just go on on the Stock Manager.

Share this post


Link to post
Share on other sites

Partially solved it. It's enough to Run the application inside the main autoit program.

I have, as thought at beginning, the problem of Pixelchecksum changing time to time for same region.

Maybe this is due to "semitrasparent" window?

How can this be solved?

M.

Share this post


Link to post
Share on other sites

Maybe not necessary, I just need a technical info, that could explain me several things:

Pixelcheckusm(bla,bla,bla,bla) returns a value. This value is "something like" a Dynamic Address? I mean if I relaunch X times the program, for same section I will get different values? Or they MUST be same value unless something change?

M.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0