Jump to content

IUIAutomation MS framework automate chrome, FF, IE, ....


Recommended Posts

  • Replies 855
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Automate all windows and browser applications with one UDF function library. Based on the microsoft automation API this library high level supports Recognition of conttrols from EDGE, Chrome, FF

Quickstart  Unzip the 2 zips (UIA*.zip and examples*.zip) of post #1 Start simplespy.au3 Hover your mouse over the element you want to handle Press ctrl+w Copy / past

@TheSaint no problem, bear in mind I am working (slowly) on a new spy to integrate all different spying tools so far seen. Will have different tabs with more/les information, more different ways

Posted Images

On 9/20/2020 at 7:31 PM, NassauSky said:

Hi @junkew,

I've used this UDF a few years ago and it was nice.  I'm trying to run the Example #5 "Chrome example" as is and I'm seeing a unique error in the UDF:

Here is the resulting output and error:

address: 
We have 2 tabs
Title is: <New Tab>    Class   := <>    controltype:= <50019>    
Title is: <New Tab>    Class   := <>    controltype:= <50000>    
"T:\Cool\Autoit-T\Scripts\STATScraper\ScrapingOptionsWebDriverOrIUI\UIA_V0_70\UIAWrappers.au3" (1673) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR

I see the UDF highlight the address bar in red then the program crashes with the above error.  

I'm running the following Chrome:

Google Chrome 
Version 85.0.4183.102 (Official Build) (64-bit)
 
Any tips, tricks, ideas or suggestions would be great.
 
Thanks for all ! 🙂

I am facing this problem too.

I am not able to tell why it happens. The code works for hours, and this randomly happens.

Link to post
Share on other sites
  • 2 weeks later...

Do you have an example of how to stop Chrome from Loading in order to be able to recognize 'Select a Certificate' popup. I am using AutoiTX and works for IE and Edge but Chrome is a problem for this particular scenario.  I have found no solution and have looked all over for anyone who has been able to solve this.  Thank you!

Link to post
Share on other sites
  • 1 month later...

junkew, your work is excellent.

I'm trying to use it for a script that attempts to log into a program that's limited to a certain number of user seats.  If login fails, the program closes.  The script attempts the login every minute until logged in.  I'm using a simple loop for this.  The code below contains the function I'm using for the program launch and login attempt.

When it first launches the program, it recognizes the login window and successfully attempts the login.  If the login fails, the script successfully relaunches the program but then it doesn't appear to get the new login window because it doesn't fill out the username and password fields.  Any ideas where I'm going wrong?  I tried to clear out the oP1 and oUIElement variables but it's not making any difference.

Any help is greatly appreciated.

Func LaunchProg()
    if WinExists("Shop Manager") then Exit
    Run('\\NAS1\Shop\Login\Shop.exe')
    WinWait("Shop Manager")

    ;~ *** UIA Standard code maintainable ***
    _UIA_setVar("oP1","Title:=Shop Manager;controltype:=UIA_WindowControlTypeId;class:=Window") ;Managed Shop Manager sign-in window
    _UIA_setVar("oUIElement1","Title:=;controltype:=UIA_EditControlTypeId;class:=TextBox") ;Username field
    _UIA_setVar("oUIElement2","Title:=;controltype:=UIA_EditControlTypeId;class:=PasswordBoxEdit") ;Password box
    _UIA_setVar("oUIElement3","Title:=Login;controltype:=UIA_ButtonControlTypeId;class:=Button") ;Login button

    _ArrayDisplay($UIA_Vars)

    Sleep(700)
    _UIA_Action("oP1","setfocus") ;Window focus
    _UIA_Action("oUIElement1","setvalue", $temparray[1]) ;Set Username
    ;MsgBox(0, "Debug", "Username should be set")
    _UIA_Action("oUIElement2","setvalue", $temparray[2]) ;Set Password
    ;MsgBox(0, "Debug", "Password should be set")
    ;_UIA_Action("oUIElement3","click") ;Click Login
    ;_UIA_Action("oUIElement3","invoke") ;Click Login (invoke avoids moving the mouse)

    _UIA_setVar("oP1","")
    _UIA_setVar("oUIElement1","")
    _UIA_setVar("oUIElement2","")
    _UIA_setVar("oUIElement3","")
    _ArrayDisplay($UIA_Vars)
    ;MsgBox(0, "DEBUG", "END FUNC")
EndFunc

 

Edited by gi_jimbo
typo

Jimbo

Using AutoIt v3.3.14.5 and SciTE version 4.2.0

Link to post
Share on other sites

Internally it caches the variables with an rti prefix to find 2nd time quicker but thats not working correctly so removing those will most likely help.

In general I allways try to highlight first in a loop to get windows and elements stable recognized and then start doing actual actions.

Small sleeps of 50 100 help and there is one default set if you look in the wrappers.

Link to post
Share on other sites
  • 6 months later...

junkew,

Thanx a lot for your great library.

Could you give me a hint

In firefox I am finding elements this way:

 

#include "UIAWrappers.au3"

AutoItSetOption("MustDeclareVars", 1)

Local $oP10=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Announcements and Site News - AutoIt Forums — Mozilla Firefox;controltype:=UIA_WindowControlTypeId;class:=MozillaWindowClass", $treescope_children)
_UIA_Action($oP10,"setfocus")
Local $oP9=_UIA_getObjectByFindAll($oP10, "Title:=;controltype:=UIA_GroupControlTypeId;class:=", $treescope_children)
Local $oP8=_UIA_getObjectByFindAll($oP9, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP7=_UIA_getObjectByFindAll($oP8, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP6=_UIA_getObjectByFindAll($oP7, "Title:=Announcements and Site News - AutoIt Forums;controltype:=UIA_DocumentControlTypeId;class:=", $treescope_children)
ConsoleWrite ("ok" &$oP6)
Local $oP5=_UIA_getObjectByFindAll($oP6, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Announcements and Site News;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
ConsoleWrite ("ok" &$oP0)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("AnnouncementsandSiteNews.mainwindow", "title:=Announcements and Site News;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Announcements and Site News;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

Is it possible to get the contents of the browser window into var?

Link to post
Share on other sites

yes, that is possible although its getting more "complicated" by using the bookmarklet trick thru the addressbar.

Approach 1 just by setting the addresbar value followed by enter with a small javascript

javascript:alert(document.body.innerHTML);

you get a messagebox with all the information you can read out with the regular autoit or UIA functions and then click the ok button.

Approach 2: Still thru addressbar add a textbox and read that out with the uia functions

to see what happens just go to www.google.com and copy/paste below in addressbar (make sure you type javascript: in front ) and press ok. 

Look then at bottom of the page and you should see a new textbox with all your information

javascript:var tb = document.createElement("input");tb.value = document.body.innerHTML;document.body.appendChild(tb);  

Approach 3: Continue on above and use eval from javascript to inject as much as you want to your HTML

approach 2 can make the textbox which you can fill with ui with javascript coding which you then finally can do eval(tb.value). This approach I have used in the past to automate "non automatable browsers"  or "closed" company browsers where you cannot install addons.

Approach 4 Probably easiest is having a addon you communicate with (never had time to do that)

         

Link to post
Share on other sites
  • 2 months later...

Just another simple spy in powershell that helps in automating and trying out different patterns

Usage

  1. Open powershell ISE (5.1 is fine)
  2. Run the script it will extend the ISE environment with a new menu (but more importantly a hotkey)
  3. Hover your mouse to your object of interest
  4. Press ctrl+shift+w
  5. You should see some output in your ISE window
  6. Play with your $ae automation element from the cmdline
  7. Extend the get-elementInfo function to your own ideas (run only function as it will reload directly)
#region load the most important GUI assemblies
Add-Type –AssemblyName UIAutomationClient
Add-Type –AssemblyName UIAutomationTypes
Add-Type –AssemblyName UIAutomationProvider
Add-Type –AssemblyName UIAutomationClientsideProviders

Add-Type -AssemblyName System.Windows.Forms
[void] [System.Reflection.Assembly]::LoadWithPartialName("'Microsoft.VisualBasic")
#endregion

#region addons
$spyName='Spy'
$MyEntry = $psISE.CurrentPowerShellTab.AddOnsMenu.Submenus | ?{$_.DisplayName -eq $spyName}
if ($myEntry -ne $null) {$psISE.CurrentPowerShellTab.AddOnsMenu.Submenus.Remove($MyEntry)}

$psISE.CurrentPowerShellTab.AddOnsMenu.Submenus.Add(
$spyName,
{
   get-elementInfo 
},
"CTRL+SHIFT+W")

#endregion

function get-elementInfo() {
    $Mouse = [System.Windows.Forms.Cursor]::Position
    $global:ae = [System.Windows.Automation.AutomationElement]::FromPoint([system.windows.point]::new($mouse.x,$mouse.y))
    $ae.current
}

So you could hover over your browser addressbar to get it "selected" 

And then play around with the different patterns from the blue cmdline ise window

$ae.getcurrentpattern([Windows.Automation.ValuePattern]::Pattern).setvalue("Hello world")

and to click if its a button

$ae.GetCurrentPattern([Windows.Automation.InvokePattern]::Pattern).Invoke()
Link to post
Share on other sites
  • 3 months later...

I'm trying to check if a table cell on a webpage (chrome) contains a certain text.  How can I specify which grid row (UIA_GridItemRow) I want to check?
Atm I'm doing this: 

Local $oUIRowElement = _UIA_getObjectByFindAll($oP2, "title:=Sample Text;ControlType:=UIA_DataItemControlTypeId", $treescope_subtree)
If $oUIRowElement <> "" Then
    ...

The text appears at most once per row and only in column one. This snippet finds the first appearance of the searched string. But what to do if I want to check in a specific row? Simplespy shows that the UIA_GridItemRow property is different for each row, however I don't know how to use this.
Properties of highlighted element (1st row):

*** Detailed properties of the highlighted element ***
UIA_title:= <Sample Text>
UIA_text:= <Sample Text>
UIA_regexptitle:= <Sample Text>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_RuntimeId:= <42;855218;4;-452608>       ;~ differs
UIA_BoundingRectangle:= <905;581;485;81>    ;~ differs
UIA_ProcessId:= <15860>
UIA_ControlType:= <50029>
UIA_LocalizedControlType:= <item>
UIA_Name:= <Sample Text>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <False>
UIA_IsEnabled:= <True>
UIA_Culture:= <9>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_FrameworkId:= <Chrome>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <True>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <False>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <True>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <True>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <1>                       ;~ differs
UIA_GridItemColumn:= <1>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Sample Text>
UIA_LegacyIAccessibleRole:= <29>
UIA_LegacyIAccessibleState:= <0>
UIA_IsDataValidForForm:= <True>
UIA_ProviderDescription:= <[pid:15860,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (IAccessible2) (unmanaged:UIAutomationCore.DLL)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>
UIA_OptimizeForVisualContent:= <False>
UIA_IsObjectModelPatternAvailable:= <False>
UIA_AnnotationAnnotationTypeId:= <60000>
UIA_IsAnnotationPatternAvailable:= <False>
UIA_IsTextPattern2Available:= <False>
UIA_StylesStyleId:= <0>
UIA_StylesFillColor:= <0>
UIA_StylesFillPatternColor:= <0>
UIA_IsStylesPatternAvailable:= <False>
UIA_IsSpreadsheetPatternAvailable:= <False>
UIA_IsSpreadsheetItemPatternAvailable:= <False>
UIA_Transform2CanZoom:= <False>
UIA_IsTransformPattern2Available:= <False>
UIA_LiveSetting:= <0>
UIA_IsTextChildPatternAvailable:= <True>
UIA_IsDragPatternAvailable:= <False>
UIA_DragIsGrabbed:= <False>
UIA_IsDropTargetPatternAvailable:= <False>
UIA_Transform2ZoomLevel:= <1>
UIA_Transform2ZoomMinimum:= <1>
UIA_Transform2ZoomMaximum:= <1>
UIA_IsTextEditPatternAvailable:= <False>
UIA_IsPeripheral:= <False>
UIA_IsCustomNavigationPatternAvailable:= <False>
UIA_PositionInSet:= <0>
UIA_SizeOfSet:= <0>
UIA_Level:= <0>
UIA_LandmarkType:= <0>
UIA_FillType:= <0>
UIA_VisualEffects:= <0>
UIA_IsSelectionPattern2Available:= <False>
UIA_Selection2ItemCount:= <0>
UIA_HeadingLevel:= <80050>
UIA_IsDialog:= <False>

Three properties are different in 2nd row:

...
UIA_RuntimeId:= <42;855218;4;-452637>
UIA_BoundingRectangle:= <905;661;485;81>
...
UIA_GridItemRow:= <2>
...

Thanks. :)

Link to post
Share on other sites

For that the wrappers will not be much of help. You have to build your own logic either with findall or a treewalker

  • Study the findall functionality
  • Give it a combined condition (with and- ing them together).
  • Then iterate over the array of elements given back by the findall (assumption that if you want to do partial match on text you have to do that yourself as thats not possible with the conditions out of the box)

The UIA wrappers where made to be more "friendly" in reading and filtering on partial text not for speed or for all functionality thats in the UIA framework then you have to call the direct functions, patterns, .....

Maybe this thread gives some indication 

 

 

Link to post
Share on other sites

Thanks for your response! I've took a look at the link you provided and at the Grid Control Pattern. Seems quiet complicated to me😅. Also I've encountered a problem with other items later on in the table that appear in a tooltip when you hover over certain cells. It seems I can not distinguish these items by their properties with the UIA framework.

I'm atm short on time so I've decided not to take a in depth look into the MS IUIAutomation framework. Instead I'll just use the UIA framework to download all pages with tables and then process the html files to get the desired data. (The pages are not static and need javascript in order to be evaluated, therefore a simple InetGet does not work and downloading them from the browser with UIA works fine)

Link to post
Share on other sites

What you potentially can do is to use the addressbar to type in small javascript bits and pieces. 

Search for bookmarklets on the internet but basically its a find addressbar, activate/focus it and do a sendkeys

sendkeys("javascript:<your short code>")

example

sendkeys("javascript:alert(document.body.innerText);")

and there are almost no limits on what you can retrieve this way from an html page

  • You can add a dummy textbox on the webpage first and fill that with javascript and retrieve it with UIA commands to read the text if you do not like to get it from an alert box.
  • You can put in the textbox some code and use the eval from javascript to interpret it as code (If your javascript is lenghty to get thru addressbar this is your workaround)
  • Its a handy approach on systems where you only have VBA or Powershell to give small POC on testautomation on "locked" down computers

attention point

  • It doesn't work when you choose new tab (Seems you have to be on an url before it works)

 

Link to post
Share on other sites
  • 3 weeks later...
  • 3 months later...
4 hours ago, junkew said:

There are many ways but No, type is sending it with activating window first.

You can use send

I think there is no command calling "send" with this IUIAuto. There is only one" sendkeys", which works same as type which is only for active windows

I've tried SendControl from au3, and returns 1, but doesn't work on this chromium based app.

Link to post
Share on other sites
6 hours ago, junkew said:

You can set textvalues directly but then you have to study how this uia model works. Its a huge model and there are many ways to get values into a textbox completely depending on its class. Why do you not want to activate a window?

Because it's a bot that I want to run in background and I use other softwares.

I might be able to get the class of textbox using inspect elements. I'll check it later.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By bobmcrae
      I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?
      #include "UIAWrappers.au3" _UIA_setVar("Global.Debug", False) _UIA_setVar("Global.Debug.File", False) _UIA_setVar("Global.Highlight", False) _UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window") _UIA_action('DPN','setFocus') _UIA_setVar("DPN.firstName","AutomationId:=txtFirstName") _UIA_setVar("DPN.lastName", "AutomationId:=txtLastName") _UIA_Action('DPN.lastName','setvalue','last name') _UIA_setVar("DPN.ID", "AutomationId:=txtSubjectId") _UIA_setVar("DPN.DOB", "AutomationId:=PART_TextBox") _UIA_setVar("DPN.Ft", "AutomationId:=txtSubjectHeight") _UIA_setVar("DPN.In", "AutomationId:=txtSubjectHeight2") _UIA_Action('DPN.firstName','setvalue','first name') _UIA_Action('DPN.ID','setvalue','ID012345') _UIA_Action('DPN.DOB','setvalue','1/31/1932') _UIA_Action('DPN.Ft','setvalue','6') _UIA_Action('DPN.In','setvalue','1') SimpleSpy output:
      ;~ *** Standard code *** #include "UIAWrappers.au3" AutoItSetOption("MustDeclareVars", 1) Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children) _UIA_Action($oP4,"setfocus") Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children) Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children) Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children) ;~ First find the object in the parent before you can do something ;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree) _UIA_action($oUIElement,"click")  
    • By jackchen
      OK,I've made my "Chrome_KeepLastTab.au3" work.This script adds some features to Chrome browser:
      1. Double click on a tab to close the tab.
      2. Keep last tab:This script monitors your mouse clicks and hotkeys,if you are about to close the last tab within Chrome(click close button on the last tab, middle click/double click on the last tab or press Ctrl + w or Ctrl + {F4}), a new tab will be open and then the old tab be closed.
      #include <WindowsConstants.au3> #include <WinAPI.au3> #include "CUIAutomation2.au3" #AutoIt3Wrapper_UseX64=Y ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os ConsoleWrite("@OSArch: " & @OSArch & ", @AutoItX64: " & @AutoItX64 & @CRLF) Global $DoubleClickTime = 500 Global $UIA_oUIAutomation ; The main library core CUI automation reference Global $hMouseEvent, $hMouseHook Global $aMouseEvent[2] Global $KeepLastTab = True ; settings from ini file If $KeepLastTab Then ;The main object with acces to the windows automation api 3.0 $UIA_oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation) If IsObj($UIA_oUIAutomation) Then HotKeySet("^w", "Hotkey_Event") ; Ctrl + w to close tab HotKeySet("^{F4}", "Hotkey_Event") ; Ctrl + {F4} to close tab $DoubleClickTime = DllCall("user32.dll", "uint", "GetDoubleClickTime")[0] OnAutoItExitRegister("UnhookMouse") ; Register mouse events callback $hMouseEvent = DllCallbackRegister("Mouse_Event", "int", "int;ptr;ptr") $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) EndIf EndIf While 1 Sleep(100) WEnd ToolTip("") ; https://www.autoitscript.com/forum/topic/103362-monitoring-mouse-events/ Func Mouse_Event($nCode, $wParam, $lParam) Local $info, $mouseData, $time, $timeDiff If $nCode < 0 Then Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf $tPoint = _WinAPI_GetMousePos() $hWnd = _WinAPI_WindowFromPoint($tPoint) ; if mouse is on the widget window(class: Chrome_RenderWidgetHostHWND), ; use $hWnd = _WinAPI_GetParent($hWnd) to get the parent Chrome window If Not StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_") Then ; Chrome_WidgetWin_1: Chrome window ; Chrome_WidgetWin_2: Chrome menu ; ignore non Chrome window Return _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) EndIf ToolTip($hWnd & " - " & _WinAPI_GetClassName($hWnd)) ;$tagPOINT = "struct;long X;long Y;endstruct" Local Const $MSLLHOOKSTRUCT = $tagPOINT & ";dword mouseData;dword flags;dword time;ulong_ptr dwExtraInfo" $info = DllStructCreate($MSLLHOOKSTRUCT, $lParam) $mouseData = DllStructGetData($info, 3) $time = DllStructGetData($info, 5) $timeDiff = $time - $aMouseEvent[1] Local $block Switch $wParam Case $WM_LBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "LClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "LDClick" Else $aMouseEvent[0] = "LClick" EndIf $block = KeepLastTab($hWnd, $aMouseEvent[0]) Case $WM_MBUTTONUP $aMouseEvent[1] = $time If $aMouseEvent[0] = "MClick" And ($timeDiff) < $DoubleClickTime Then $aMouseEvent[0] = "MDClick" Else $aMouseEvent[0] = "MClick" $block = KeepLastTab($hWnd, $aMouseEvent[0]) EndIf EndSwitch If Not $block Then _WinAPI_CallNextHookEx($hMouseHook, $nCode, $wParam, $lParam) ; Continue processing EndIf EndFunc ;==>Mouse_Event Func UnhookMouse() _WinAPI_UnhookWindowsHookEx($hMouseHook) $hMouseHook = 0 DllCallbackFree($hMouseEvent) $hMouseEvent = 0 EndFunc ;==>UnhookMouse Func Hotkey_Event() Local $block Local $hWnd = WinGetHandle(WinGetTitle("[ACTIVE]")) If StringInStr(_WinAPI_GetClassName($hWnd), "Chrome_WidgetWin_1") Then ; Chrome_WidgetWin_1, Chrome window $block = KeepLastTab($hWnd, "Hotkey") EndIf If Not $block Then HotKeySet("^w") Send(@HotKeyPressed) HotKeySet("^w", "Hotkey_Event") EndIf EndFunc Func KeepLastTab($hWnd, $action = "LClick") ; Possible $action value: LClick, LDClick, MClick, Hotkey ConsoleWrite(@CRLF & $action & " on a Chrome window: " & $hWnd & @CRLF) Local $aMousePos = MouseGetPos() Local $pChrome, $oChrome $UIA_oUIAutomation.ElementFromHandle($hWnd, $pChrome) ; Window $oChrome = ObjCreateInterface($pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oChrome) Then ConsoleWrite("Failed to get Chrome object from hWnd." & @CRLF) Return EndIf ;$UIA_ControlTypePropertyId = 30003 $oChromeTabs = UIA_getFirstElement($oChrome, $UIA_ControlTypePropertyId, $UIA_TabControlTypeId, $treescope_subtree) If Not IsObj($oChromeTabs) Then ConsoleWrite("Failed to get Chrome tab bar object." & @CRLF) Return EndIf Local $t $oChromeTabs.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1]) Then ; Mouse not on tabs bar ConsoleWrite("Mouse is not on the tab bar. Ignore and return..." & @CRLF) Return EndIf Local $pTrueCondition, $pElements, $iLength, $oAutomationElementArray $UIA_oUIAutomation.CreateTrueCondition($pTrueCondition) $oCondition = ObjCreateInterface($pTrueCondition, $sIID_IUIAutomationCondition, $dtagIUIAutomationCondition) If Not IsObj($oCondition) Then Return $oChromeTabs.FindAll($treescope_children, $oCondition, $pElements) $oAutomationElementArray = ObjCreateInterface($pElements, $sIID_IUIAutomationElementArray, $dtagIUIAutomationElementArray) If Not IsObj($oAutomationElementArray) Then ConsoleWrite("Failed to find all Chrome tabs. " & @CRLF) Return EndIf $oAutomationElementArray.Length($iLength) Local $UIA_pUIElement, $oTab2 Local $iTabs = $iLength - 1 If $iTabs > 1 Then ; more than one tab If $action = "LDClick" Then ConsoleWrite("There are " & $iTabs & " tabs within Chrome window. " & @CRLF) For $i = 1 To $iTabs $oAutomationElementArray.GetElement($i, $UIA_pUIElement) $oTab2 = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab2) Then ContinueLoop $oTab2.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then ContinueLoop If $aMousePos[0] >= $t[0] And $aMousePos[0] <= $t[2]+$t[0] And $aMousePos[1] >= $t[1] And $aMousePos[1] <= $t[3]+$t[1] Then ConsoleWrite("You double clicked on one of " & $iTabs & " tabs. Close the tab and return..." & @CRLF) HotKeySet("^w") Send("^w") HotKeySet("^w", "Hotkey_Event") Return True EndIf Next EndIf Return EndIf ConsoleWrite("There is ONLY one tab within Chrome window. " & @CRLF) $oAutomationElementArray.GetElement(1, $UIA_pUIElement) $oTab = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If Not IsObj($oTab) Then ConsoleWrite("Failed to get the last tab object." & @CRLF) Return EndIf Local $rtTab $oTab.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $rtTab) If UBound($rtTab) < 4 Then Return If $action <> "Hotkey" And ($aMousePos[0] < $rtTab[0] Or $aMousePos[0] > $rtTab[2]+$rtTab[0] Or $aMousePos[1] < $rtTab[1] Or $aMousePos[1] > $rtTab[3]+$rtTab[1]) Then ; Mouse not on the last tab ConsoleWrite("Mouse is not on the last tab. Ignore and return..." & @CRLF) Return EndIf If $action = "LClick" Then $oTabClose = UIA_getFirstElement($oTab, $UIA_ControlTypePropertyId, $UIA_ButtonControlTypeId, $treescope_subtree) If Not IsObj($oTabClose) Then ConsoleWrite("Failed to get the last tab close object." & @CRLF) Return EndIf $oTabClose.GetCurrentPropertyValue($UIA_BoundingRectanglePropertyId, $t) If UBound($t) < 4 Then Return If $aMousePos[0] < $t[0] Or $aMousePos[0] > $t[2]+$t[0] Or $aMousePos[1] < $t[1] Or $aMousePos[1] > $t[3]+$t[1] Then ; Mouse not on the tab close button Return EndIf EndIf ; open a new tab within chrome ConsoleWrite("The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing." & @CRLF) Send("^t") ConsoleWrite("A new tab created!" & @CRLF) Sleep(100) ConsoleWrite("Close the old tab and return..." & @CRLF) ;~ Local $pPattern ;~ $oTabClose.GetCurrentPattern($UIA_InvokePatternId, $pPattern) ;~ $oPattern = ObjCreateInterface($pPattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern) ;~ If IsObj($oPattern) Then ;~ ConsoleWrite("Invoke to close the tab..." & @CRLF) ;~ $oTabClose.SetFocus() ;~ $oPattern.Invoke() ;~ EndIf _WinAPI_UnhookWindowsHookEx($hMouseHook) $aMousePos = MouseGetPos() If $aMousePos[0] >= $rtTab[0] And $aMousePos[0] <= $rtTab[2]+$rtTab[0] And $aMousePos[1] >= $rtTab[1] And $aMousePos[1] <= $rtTab[3]+$rtTab[1] Then MouseClick("middle", $aMousePos[0], $aMousePos[1], 1, 0) Else MouseClick("middle", $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2, 1, 0) ; close the tab MouseMove($aMousePos[0], $aMousePos[1], 0) ; move mouse back to previous position EndIf ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEDOWN, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) ;~ Sleep(100) ;~ _WinAPI_Mouse_Event($MOUSEEVENTF_MIDDLEUP, $rtTab[0]+$rtTab[2]/2, $rtTab[1]+$rtTab[3]/2) $hMouseHook = _WinAPI_SetWindowsHookEx($WH_MOUSE_LL, DllCallbackGetPtr($hMouseEvent), _WinAPI_GetModuleHandle(0)) Return True ; to block mouse click/hot key EndFunc Func UIA_getFirstElement($obj, $propertyID, $tval, $treeScope) Local $pCondition, $oCondition $UIA_oUIAutomation.CreatePropertyCondition($propertyID, $tval, $pCondition) $oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition) Local $UIA_oUIElement, $UIA_pUIElement $t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement) $UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement) If IsObj($UIA_oUIElement) Then Return $UIA_oUIElement Return SetError(1, 0, "") EndFunc ;==>UIA_getFirstElement My question is related to #AutoIt3Wrapper_UseX64 when run this script on 64-bit Win 7.
      No mater I set #AutoIt3Wrapper_UseX64=Y or #AutoIt3Wrapper_UseX64=N, this script works very well on hotkey event, while mouse clicks sometimes works if #AutoIt3Wrapper_UseX64=N and sometimes works on Y. Can some one test this and finger out what's wrong?
      Info from SciTe if #AutoIt3Wrapper_UseX64=Y :
      @OSArch: X64, @AutoItX64: 1 Hotkey on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x0000000000140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... Info if #AutoIt3Wrapper_UseX64=N or comment out this line:
      @OSArch: X64, @AutoItX64: 0 Hotkey on a Chrome window: 0x00140330 There is ONLY one tab within Chrome window. The last tab is about to be closed, so we have to open a new tab to prevent Chrome window from closing. A new tab created! Close the old tab and return... LClick on a Chrome window: 0x00140330 Failed to get Chrome tab bar object.Hotkey events and mouse click events share the same function KeepLastTab($hWnd, $action = "LClick"),Why this function triggered by hotkey works on both 32-bit and 64-bit while that triggered by mouse events failed on 32-bit autoit?
×
×
  • Create New...