Jump to content
junkew

IUIAutomation MS framework automate chrome, FF, IE, ....

Recommended Posts

On ‎2016‎年‎11‎月‎11‎日 at 4:38 AM, junkew said:
On ‎2014‎年‎1‎月‎18‎日 at 9:12 PM, LarsJ said:

; Click (invoke) Save button       Local $pInvoke, $oInvoke       $oSave.GetCurrentPattern( $UIA_InvokePatternId, $pInvoke )       $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )       If Not IsObj( $oInvoke ) Then Return $S_OK       $oInvoke.Invoke()       ConsoleWrite( "Save button clicked" & @CRLF & @CRLF )

sometimes,the save is ok,but sometimes it has click the dropdown button and it has not save

 

Share this post


Link to post
Share on other sites

It's the example in post 126.

I've just tried the example 10 times in a row with inspect.rar here. The file was saved automatically every time. You can try the same.

What files do not work? Are there any specific characteristics for the files or the web sites? Do you have any examples of files that failed, so I can try myself?

Share this post


Link to post
Share on other sites

junkew, I think the problem here is that Invoke actually works. The Save button is clicked, the dialog box disappears, but the file is not saved anyway.

The standard solution is to insert a Sleep() eg. Sleep(200) immediately before the code section with the Save button click. This gives the download a little bit more time to finish before the Save button is clicked.

Edited by LarsJ
Last sentence

Share this post


Link to post
Share on other sites

I need help.
What do I need to do?
I need to be able to get the text of a table.  I started testing with "ControlListView" but it returned "0".  Was when I found "IUIAutomation MS framework automates Chrome, FF, IE, ...." I decided to try.

try with the inspect that I will follow @TheDcoder and it is much more complete. it extracts almost all the necessary information, but still I am still unable to access the text of the table.
log2.PNG


I would like to know if these values represent any problem or if the contents of the table can still be accessed.

Grid.ColumnCount:        [Not supported]
Grid.RowCount:        [Not supported]
GridItem.Column:        [Not supported]
GridItem.ColumnSpan:    [Not supported]
GridItem.ContainingGrid:[Not supported]
GridItem.Row:            [Not supported]
GridItem.RowSpan:        [Not supported]

In the log.txt file there more info. 

 

log.txt

Edited by Jasp402

Share this post


Link to post
Share on other sites

Does anyone know of an issue with simplespy.au3 giving an error during running:

Line 4 (File "MSAccessibility.au3");

#include <autoitconstants.au3>

Error: Error opening the file.

 

Any idea? 

Thanks.

Share this post


Link to post
Share on other sites

your AutoIt installation may be corrupt. the file lives here by default. Check if it is there.

C:\Program Files (x86)\AutoIt3\Include\AutoItConstants.au3

worst case you have to uninstall and reinstall latest AutoIt installation?

Edited by Earthshine

My resources are limited. You must ask the right questions

 

Share this post


Link to post
Share on other sites
6 minutes ago, nassausky said:

That was exactly it. Thanks @Earthshine

you may want to schedule a disk check

at a command prompt type

Chkdsk c: /s

then reboot and let it fix up the drive. Sometimes files go missing when the OS hangs and it gets hard booted


My resources are limited. You must ask the right questions

 

Share this post


Link to post
Share on other sites

 

Hi @junkew

For anyone interested I figured out what my problem was with a newer version of chrome. Anyone who is an expert at this would have figured this out in a couple minutes.

I was getting an error running the ex5_chrome example.

D:\EXAMPLES_V0_64\ex5_chrome.au3" (116) : ==> Variable must be of type "Object".:
$oValueP.CurrentValue($myText)
$oValueP^ ERROR

Chrome is version  Version 63.0.3239.84 (Official Build) (64-bit) 

This is the dump log file. I've been looking at a couple examples all day trying to see how to traverse the elements using this threads technique.  Wish there was an easier way.

<?xml version="1.0" encoding="UTF-8"?>
<log space="preserve">
<logline timestamp="20171212-193701870" level="5">
<information> Information _UIA_VersionInfo version: T0.6-5; Release date: 20160909; OS Version: WIN_10</information>
</logline>
<logline timestamp="20171212-193701870" level="5"> Matching class transformed to propertynumber: 30012: for having value Chrome_WidgetWin_1</logline>
<logline timestamp="20171212-193701932" level="5"> UIA found the element</logline>
<logline timestamp="20171212-193702120" level="0">
<treedump>
<treeheader>***** Dumping tree *****</treeheader>
<elementinfo>Title is: <> Class := <Intermediate D3D Window> controltype:= <UIA_PaneControlTypeId> ,<50033> , (0000C371) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <> Class := <> controltype:= <UIA_TitleBarControlTypeId> ,<50037> , (0000C375) , acceleratorkey:= <> , automationid:= <> </elementinfo>
<elementinfo>Title is: <Browser container> Class := <> controltype:= <UIA_CustomControlTypeId> ,<50025> , (0000C369) , acceleratorkey:= <> , automationid:= <> </elementinfo>
</treedump>
</logline>
</log>

Full day lesson.  Figured out for the chrome toolbar the matchstring should be:

switch @OSLang
          case 0409 ;OS Language = en-US
              ;-$MatchString="start" ;- Doesn't work form Chrome v63.0.3239.84
              $MatchString= "controltype:=" & $UIA_CustomControlTypeId ;

For the chrome address bar

local $oChromeAddressBar=_UIA_getFirstObjectOfElement($oChromeToolbar,"controltype:=" & $UIA_EditControlTypeId , $treescope_subtree) ;works in chrome 29

So far I think I'm liking this automation but oh man it's way harder than ie.au3. I didn't even get to the part of traversing the DOM to find html elements. I need more lessons :-O

Edited by nassausky
Typo

Share this post


Link to post
Share on other sites

Its much more powerfull then ie.au3 and in that sense maybe more complicated but just let me know what you find easier on ie.au3 so I can improve the uiawrappers

  • Chhrome examples I will fix with next update.
  • Variable must be of type object I will make library more robust  but indeed in the end means incorrect identification properties for object.

Some tips

  1.  Understand your UI Tree with the different spying tools around.  I am using
    inspect.exe but sometimes freezes,
    VisualUIAVerify.exe cannot see them all (especially popups that dissapear when you want to inspect),
    Simplespy made by myself to tweak more into what I need in my day to day job (and I frequently add some small parts to identify specific scenario's that are hard to generalize)
  2. Understand visible and non visible grouping controls in the parent hierarchy
  3. Understand the difference between finding in children and grandchildren (number of objects searching through)

Share this post


Link to post
Share on other sites

Is there some way to speed up _UIA_getObjectByFindAll()? See gif...it's taking nearly 2 seconds to identify each element 

I'm running the following code in a loop that identifies each of the checkboxes. The $subsection variable is as close as I can get to the direct parent of those elements when using simplespy.

$element = _UIA_getObjectByFindAll($subsection, 'title:=' & $consultation_of.Keys[$i] & ';classname:=WindowsForms10.BUTTON.app.0.202c666', $treescope_subtree)

Here's the simplespy output for the "Shoulder" checkbox:

   https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/?do=findComment&comment=1156373 
At least we have an element title: [Shoulder] class: [WindowsForms10.BUTTON.app.0.202c666]

Having the following values for all properties: 
Title is: <Shoulder>    Class   := <WindowsForms10.BUTTON.app.0.202c666>    controltype:= <UIA_ButtonControlTypeId>    ,<50000>    , (0000C350)    1420;531;86;17
*** Parent Information top down ***
14: Title is: <Note Authoring>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    951;72;935;936
"Title:=Note Authoring;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
13: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    955;119;927;885
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
12: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    955;119;927;885
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
11: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    955;119;927;885
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
10: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    955;178;927;826
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
9: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    955;178;927;826
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
8: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_TabControlTypeId>    ,<50018>    , (0000C362)    965;188;907;806
"Title:=;controltype:=UIA_TabControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
7: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    966;209;905;784
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
6: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    966;209;905;784
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
5: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    966;209;905;784
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
4: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1168;241;703;752
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
3: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1168;274;703;585
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
2: Title is: <twNoteInputTemplate1>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1168;274;703;585
"Title:=twNoteInputTemplate1;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
1: Title is: <History of Present Illness>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1180;286;663;3007
"Title:=History of Present Illness;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    
0: Title is: <>    Class   := <WindowsForms10.Window.8.app.0.202c666>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    1190;312;643;2981
"Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666""    


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Note Authoring;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;Note Authoring
_UIA_setVar("oP2","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP3","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP6","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP7","Title:=;controltype:=UIA_TabControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP8","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP9","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP10","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP11","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP12","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;
_UIA_setVar("oP13","Title:=twNoteInputTemplate1;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;twNoteInputTemplate1
_UIA_setVar("oP14","Title:=History of Present Illness;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;History of Present Illness
_UIA_setVar("oP15","Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666")    ;

_UIA_setVar("Shoulder.mainwindow","title:=Shoulder;classname:=WindowsForms10.BUTTON.app.0.202c666")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
_UIA_Action("oP6","setfocus")
;~_UIA_Action("oP7","highlight")
_UIA_Action("oP7","setfocus")
;~_UIA_Action("oP8","highlight")
_UIA_Action("oP8","setfocus")
;~_UIA_Action("oP9","highlight")
_UIA_Action("oP9","setfocus")
;~_UIA_Action("oP10","highlight")
_UIA_Action("oP10","setfocus")
;~_UIA_Action("oP11","highlight")
_UIA_Action("oP11","setfocus")
;~_UIA_Action("oP12","highlight")
_UIA_Action("oP12","setfocus")
;~_UIA_Action("oP13","highlight")
_UIA_Action("oP13","setfocus")
;~_UIA_Action("oP14","highlight")
_UIA_Action("oP14","setfocus")
;~_UIA_Action("oP15","highlight")
_UIA_Action("oP15","setfocus")

_UIA_action("Shoulder.mainwindow","setfocus")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP14=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Note Authoring;controltype:=UIA_WindowControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP14,"setfocus")
Local $oP13=_UIA_getObjectByFindAll($oP14, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP13,"setfocus")
Local $oP12=_UIA_getObjectByFindAll($oP13, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP12,"setfocus")
Local $oP11=_UIA_getObjectByFindAll($oP12, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP11,"setfocus")
Local $oP10=_UIA_getObjectByFindAll($oP11, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP10,"setfocus")
Local $oP9=_UIA_getObjectByFindAll($oP10, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP9,"setfocus")
Local $oP8=_UIA_getObjectByFindAll($oP9, "Title:=;controltype:=UIA_TabControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP8,"setfocus")
Local $oP7=_UIA_getObjectByFindAll($oP8, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP7,"setfocus")
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP6,"setfocus")
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=twNoteInputTemplate1;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=History of Present Illness;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=WindowsForms10.Window.8.app.0.202c666", $treescope_children)    
_UIA_Action($oP0,"setfocus")
_UIA_setVar("Shoulder.mainwindow","title:=Shoulder;classname:=WindowsForms10.BUTTON.app.0.202c666")
_UIA_action("Shoulder.mainwindow","setfocus")


*** Detailed properties of the highlighted element ***
UIA_title:= <Shoulder>
UIA_text:= <Shoulder>
UIA_regexptitle:= <Shoulder>
UIA_class:= <WindowsForms10.BUTTON.app.0.202c666>
UIA_regexpclass:= <WindowsForms10.BUTTON.app.0.202c666>
UIA_iaccessiblechildId:= <0>
UIA_id:= <TextFinding361>
UIA_handle:= <788282>
UIA_RuntimeId:= <42;788282>
UIA_BoundingRectangle:= <1420;531;86;17>
; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

 

2018-01-10_13-48-04.gif

Share this post


Link to post
Share on other sites

So quick question.. SimpleSpy works perfectly and is able to click what I couldn't 'see' before with window info tool.. And using the 'click' command works great. But... Is there any way to speed it up? Is it getting stuck somewhere or is that just how long the deep search function takes? I even changed the "UIAWrappers.au3" file to where the sleep function was 20ms and it didn't help.. I only have these three lines in my code, but it still takes 12.54seconds to complete on an i7 processor.. is something wrong (ie is it getting stuck somewhere) or is that realistic? Is there anything I can do to speed it up? 

 

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_action("Title:=Analytical;controltype:=UIA_RadioButtonControlTypeId;class:=","click")

 

Share this post


Link to post
Share on other sites

The purpose of the wrapper functions is first and foremost to make UI Automation as easy as possible, avoiding advanced functions like ObjCreateInterface. This function seems to be an insurmountable barrier for 99.99% of all AutoIt users. Not only in connection with UI Automation but generally.

The wrapper functions contains a lot of code to make it easy and to implement as much functionality as possible. As usual you'll get nothing for free. The cost here is especially the speed.

Once you've got the wrapper functions to work, you can optimize the speed by using the UI Automation objects and methods directly (without using the wrapper functions). See some of my examples for details.

The example above in post #617 can probably be optimized to about 200 milliseconds.

Share this post


Link to post
Share on other sites

@u547899 @BatMan22 yes, the uia of ms in itself is very fast the wrappers indeed sacrifice some speed but as most teams are coming from manual testing that takes hours or days to execute they are very happy as its now blazingly fast in minutes or seconds running during lunch or overnight. But as said 99.99% of the people are a little afraid to use the objcreateinterface logic.

Basically with 1 call of findall and filtercondition on type checkbox you can retrieve all checkboxes and then you have directly a array with reference to all of them. These examples are in examples.zip.

The wrappers do this

  1. Get all childs unfiltered from parent
  2. Loop thru the returned array and filter with regular expression logic and capability to jump n elements back or forward which is not possible with uia interfaces out of the box

So above 2 steps happens for every uiaction wrapper function which slows down especially if you know that all checkboxes are already in the array.

Share this post


Link to post
Share on other sites
2 hours ago, junkew said:

@u547899 @BatMan22 yes, the uia of ms in itself is very fast the wrappers indeed sacrifice some speed but as most teams are coming from manual testing that takes hours or days to execute they are very happy as its now blazingly fast in minutes or seconds running during lunch or overnight. But as said 99.99% of the people are a little afraid to use the objcreateinterface logic.

Basically with 1 call of findall and filtercondition on type checkbox you can retrieve all checkboxes and then you have directly a array with reference to all of them. These examples are in examples.zip.

The wrappers do this

  1. Get all childs unfiltered from parent
  2. Loop thru the returned array and filter with regular expression logic and capability to jump n elements back or forward which is not possible with uia interfaces out of the box

So above 2 steps happens for every uiaction wrapper function which slows down especially if you know that all checkboxes are already in the array.

Thanks dude.. I'll try and figure out how all this works >.<, either way I appreciate your work, it really helps for idiots like me :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By bobmcrae
      I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?
      #include "UIAWrappers.au3" _UIA_setVar("Global.Debug", False) _UIA_setVar("Global.Debug.File", False) _UIA_setVar("Global.Highlight", False) _UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window") _UIA_action('DPN','setFocus') _UIA_setVar("DPN.firstName","AutomationId:=txtFirstName") _UIA_setVar("DPN.lastName", "AutomationId:=txtLastName") _UIA_Action('DPN.lastName','setvalue','last name') _UIA_setVar("DPN.ID", "AutomationId:=txtSubjectId") _UIA_setVar("DPN.DOB", "AutomationId:=PART_TextBox") _UIA_setVar("DPN.Ft", "AutomationId:=txtSubjectHeight") _UIA_setVar("DPN.In", "AutomationId:=txtSubjectHeight2") _UIA_Action('DPN.firstName','setvalue','first name') _UIA_Action('DPN.ID','setvalue','ID012345') _UIA_Action('DPN.DOB','setvalue','1/31/1932') _UIA_Action('DPN.Ft','setvalue','6') _UIA_Action('DPN.In','setvalue','1') SimpleSpy output:
      ;~ *** Standard code *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children) Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children) Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) ;~ First find the object in the parent before you can do something ;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) _UIA_action($oUIElement,"click")  
    • By jackchen
      OK,I've made my "Chrome_KeepLastTab.au3" work.This script adds some features to Chrome browser:
      1. Double click on a tab to close the tab.
      2. Keep last tab:This script monitors your mouse clicks and hotkeys,if you are about to close the last tab within Chrome(click close button on the last tab, middle click/double click on the last tab or press Ctrl + w or Ctrl + {F4}), a new tab will be open and then the old tab be closed.
      #include <WindowsConstants.au3> #include <WinAPI.au3> #include "CUIAutomation2.au3" #AutoIt3Wrapper_UseX64=Y ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os ConsoleWrite("@OSArch: " & @OSArch & ", @AutoItX64: " & @AutoItX64 & @CRLF) Global $DoubleClickTime = 500 Global $UIA_oUIAutomation ; The main library core CUI automation reference Global $hMouseEvent, $hMouseHook Global $aMouseEvent[2] Global $KeepLastTab = True ; settings from ini file If $KeepLastTab Then ;The main object with acces to the windows automation api 3.0 $UIA_oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation) If IsObj($UIA_oUIAutomation) Then HotKeySet("^w", "Hotkey_Event") ; Ctrl + w to close tab HotKeySet("^{F4}", "Hotkey_Event") ; Ctrl + {F4} to close tab $DoubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")[0] OnAutoItExitRegister("UnhookMouse") ; Register mouse events callback $hMouseEvent = DllCallbackRegister("Mouse_Event", "int", "int;ptr;ptr") $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) EndIf EndIf While 1 Sleep(100) WEnd ToolTip("") ; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/ Func Mouse_Event($nCode, $wParam, $lParam) Local $info, $mouseData, $time, $timeDiff If $nCode < 0 Then Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf $tPoint = _WinAPI_GetMousePos() $hWnd = _WinAPI_WindowFromPoint($tPoint) ; if mouse is on the widget window(class: Chrome_RenderWidgetHostHWND), ; use $hWnd = _WinAPI_GetParent($hWnd) to get the parent Chrome window If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_") Then ; Chrome_WidgetWin_1: Chrome window ; Chrome_WidgetWin_2: Chrome menu ; ignore non Chrome window Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) EndIf ToolTip($hWnd & " - " & _WinAPI_GetClassName($hWnd)) ;$tagPOINT = "struct;long X;long Y;endstruct" Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo" $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam) $mouseData = DllStructGetData($info, 3) $time = DllStructGetData($info, 5) $timeDiff = $time - $aMouseEvent[1] Local $block Switch $wParam Case $WM_LBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "LClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "LDClick" Else $aMouseEvent[0] = "LClick" EndIf $block = KeepLastTab($hWnd, $aMouseEvent[0]) Case $WM_MBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "MClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "MDClick" Else $aMouseEvent[0] = "MClick" $block = KeepLastTab($hWnd, $aMouseEvent[0]) EndIf EndSwitch If Not $block Then _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf EndFunc ;==>Mouse_Event Func UnhookMouse() _WinAPI_UnhookWindowsHookEx($hMouseHook) $hMouseHook = 0 DllCallbackFree($hMouseEvent) $hMouseEvent = 0 EndFunc ;==>UnhookMouse Func Hotkey_Event() Local $block Local $hWnd = WinGetHandle(WinGetTitle("[ACTIVE]")) If StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_1") Then ; Chrome_WidgetWin_1, Chrome window $block = KeepLastTab($hWnd, "Hotkey") EndIf If Not $block Then HotKeySet("^w") Send(@HotKeyPressed) HotKeySet("^w", "Hotkey_Event") EndIf EndFunc Func KeepLastTab($hWnd, $action = "LClick") ; Possible $action value: LClick, LDClick, MClick, Hotkey ConsoleWrite(@CRLF & $action & " on a Chrome window: " & $hWnd & @CRLF) Local $aMousePos = MouseGetPos() Local $pChrome, $oChrome $UIA_oUIAutomation.ElementFromHandle($hWnd, $pChrome) ; Window $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oChrome) Then ConsoleWrite("Failed to get Chrome object from hWnd." & @CRLF) Return EndIf ;$UIA_ControlTypePropertyId = 30003 $oChromeTabs = UIA_getFirstElement($oChrome, $UIA_ControlTypePropertyId, $UIA_TabControlTypeId, $treescope_subtree) If Not IsObj($oChromeTabs) Then ConsoleWrite("Failed to get Chrome tab bar object." & @CRLF) Return EndIf Local $t $oChromeTabs.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1]) Then ; Mouse not on tabs bar ConsoleWrite("Mouse is not on the tab bar. Ignore and return..." & @CRLF) Return EndIf Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition) $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition) If Not IsObj($oCondition) Then Return $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements) $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray) If Not IsObj($oAutomationElementArray) Then ConsoleWrite("Failed to find all Chrome tabs. " & @CRLF) Return EndIf $oAutomationElementArray.Length($iLength) Local $UIA_pUIElement, $oTab2 Local $iTabs = $iLength - 1 If $iTabs > 1 Then ; more than one tab If $action = "LDClick" Then ConsoleWrite("There are " & $iTabs & " tabs within Chrome window. " & @CRLF) For $i = 1 To $iTabs $oAutomationElementArray.GetElement($i, $UIA_pUIElement) $oTab2 = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab2) Then ContinueLoop $oTab2.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then ContinueLoop If $aMousePos[0] >= $t[0] And $aMousePos[0] <= $t[2]+$t[0] And $aMousePos[1] >= $t[1] And $aMousePos[1] <= $t[3]+$t[1] Then ConsoleWrite("You double clicked on one of " & $iTabs & " tabs. Close the tab and return..." & @CRLF) HotKeySet("^w") Send("^w") HotKeySet("^w", "Hotkey_Event") Return True EndIf Next EndIf Return EndIf ConsoleWrite("There is ONLY one tab within Chrome window. " & @CRLF) $oAutomationElementArray.GetElement(1, $UIA_pUIElement) $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab) Then ConsoleWrite("Failed to get the last tab object." & @CRLF) Return EndIf Local $rtTab $oTab.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $rtTab) If UBound($rtTab) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $rtTab[0] Or $aMousePos[0] > $rtTab[2]+$rtTab[0] Or $aMousePos[1] < $rtTab[1] Or $aMousePos[1] > $rtTab[3]+$rtTab[1]) Then ; Mouse not on the last tab ConsoleWrite("Mouse is not on the last tab. Ignore and return..." & @CRLF) Return EndIf If $action = "LClick" Then $oTabClose = UIA_getFirstElement($oTab, $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $treescope_subtree) If Not IsObj($oTabClose) Then ConsoleWrite("Failed to get the last tab close object." & @CRLF) Return EndIf $oTabClose.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1] Then ; Mouse not on the tab close button Return EndIf EndIf ; open a new tab within chrome ConsoleWrite("The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing." & @CRLF) Send("^t") ConsoleWrite("A new tab created!" & @CRLF) Sleep(100) ConsoleWrite("Close the old tab and return..." & @CRLF) ;~ Local $pPattern ;~ $oTabClose.GetCurrentPattern($UIA_InvokePatternId, $pPattern) ;~ $oPattern = ObjCreateInterface($pPattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern) ;~ If IsObj($oPattern) Then ;~ ConsoleWrite("Invoke to close the tab..." & @CRLF) ;~ $oTabClose.SetFocus() ;~ $oPattern.Invoke() ;~ EndIf _WinAPI_UnhookWindowsHookEx($hMouseHook) $aMousePos = MouseGetPos() If $aMousePos[0] >= $rtTab[0] And $aMousePos[0] <= $rtTab[2]+$rtTab[0] And $aMousePos[1] >= $rtTab[1] And $aMousePos[1] <= $rtTab[3]+$rtTab[1] Then MouseClick("middle", $aMousePos[0], $aMousePos[1], 1, 0) Else MouseClick("middle", $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2, 1, 0) ; close the tab MouseMove($aMousePos[0], $aMousePos[1], 0) ; move mouse back to previous position EndIf ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEDOWN, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) ;~ Sleep(100) ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEUP, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) Return True ; to block mouse click/hot key EndFunc Func UIA_getFirstElement($obj, $propertyID, $tval, $treeScope) Local $pCondition, $oCondition $UIA_oUIAutomation.CreatePropertyCondition($propertyID, $tval, $pCondition) $oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition) Local $UIA_oUIElement, $UIA_pUIElement $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement) $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If IsObj($UIA_oUIElement) Then Return $UIA_oUIElement Return SetError(1, 0, "") EndFunc ;==>UIA_getFirstElement My question is related to #AutoIt3Wrapper_UseX64 when run this script on 64-bit Win 7.
      No mater I set #AutoIt3Wrapper_UseX64=Y or #AutoIt3Wrapper_UseX64=N, this script works very well on hotkey event, while mouse clicks sometimes works if #AutoIt3Wrapper_UseX64=N and sometimes works on Y. Can some one test this and finger out what's wrong?
      Info from SciTe if #AutoIt3Wrapper_UseX64=Y :
      @OSArch: X64, @AutoItX64: 1 Hotkey on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... Info if #AutoIt3Wrapper_UseX64=N or comment out this line:
      @OSArch: X64, @AutoItX64: 0 Hotkey on a Chrome window: 0x00140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x00140330 Failed to get Chrome tab bar object. Hotkey events and mouse click events share the same function KeepLastTab($hWnd, $action = "LClick"),Why this function triggered by hotkey works on both 32-bit and 64-bit while that triggered by mouse events failed on 32-bit autoit?
×
×
  • Create New...