Silverlight, Cannot find Controls

[redrum]

Hello again,

I believe I need to be able to find the element via the "automationid", as that is the only element that is unique for each of the element I need to find.

Application:  freestockcharts.com - this is a free version of the full version I am using.  I re-wrote the script to access freestockcharts.com to make sure the issues I am facing are the same.

So, you can access the application web page by:

-go to freestockchart.com

-click on:  Start FreeStockCharts

-click on: View, and turn off Scrolling Ticker and select "Show Full Screen Chart"  (This makes the app dump smaller and almost identical to the app in my full version)

-click on the yellow symbol (whatever the symbol is) and change it to "HON"  as the script is presently set to look for Title=HON, which actually gets the wrong element, an earlier one

When the script is run it gets to the parent of the desired element, and then finds an element with Title=HON.  I cannot use this as a search (as this changes always to the stock symbol selected for display, the active symbol). I have noted in the dumps the symbol element that I am trying to get to, which is the yellow element on the chart.

All properties of the element I want are common with many other elements - as the attached dumps show.

The only unique property is "automationid"

So, the question is can I find the element I need using the automationid property in the search string?  I have tried and it does not return an object.

If not, is there another possible approach?

Attachments are:  "FSS Script", "FSS_HON_Spy", which is Spy data w/cursor over the yellow "HON" on chart, "FSS_HON_dump" of Parent0, and "FSS_TPLM_dump" of Parent0 just to show a different symbol

 

 

FSS Script.au3

FSS_HON_Spy.doc

FSS_HON_dump.doc

FSS_TPLM_dump.doc

[redrum]

Hello,

I have made significant progress using the "ui integration" to automate this Silverlight application, and much appreciate the prior help that got me on a workable path.  I am now having some, quite consistent, App Crashes when accessing an element title (which is actually the text data that I need) at a cursor position.  I used some code from SimpleSpy to create this method of accessing the data.  The code works fine and allows me to get displayed data on the screen, by placing cursor over desired data.  As discussed above in this thread, I cannot search/find the element I need due to no unique fields to search for - so using cursor position instead.

A short Hot Key function code is attached.  This code executes and reliably gets the title data that I am looking for.  However, it randomly causes an Apps Crash.  I have attached the Hot Key function code, and the Crash Data.  I have searched forum and not found anything similar to this,

Could someone help with some direction on how to solve this?

Thanks,

Doug

 

Hot Key Function that Crashes.au3

CRASH DATA.doc

[junkew]

Actually it can be as simple as below

1. have a chrome browser started

2. run below, the trick is in indexrelative with a negative value 

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_Action("Title:=FreeStockCharts.*;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", "setfocus")  
_UIA_Action("Title:=;controltype:=Pane;class:=WrapperNativeWindowClass", "setfocus")    
_UIA_Action("Title:=Silverlight Control;controltype:=Window;class:=NativeWindowClass", "setfocus")  
_UIA_Action("title:=Add Indicator;ControlType:=Text;indexrelative:=-1","click")

[redrum]

TC Test Script.au3ExecutionResults.txtYes, I ran the standard code, as above, using the indexrelative and works, but took about 1min to execute.  I also created a similar script using standard code and indexrelative for my non-browser version which also works, but takes about 7min to execute.  That is way beyond what will be acceptable for interactive control.  I compiled the non-browser version, both Beta x64 and Beta x86 and they both took about 7min.

The script I have working is fast (I guess because I use the cursor position of the element I need) - so it is fast, like SimpleSpy is.  So at this point:

Is it possible to search for the element and get results fast enough for interactive control (needs to be <2-3 sec for sure)?

Is there a solution for the AppCrash I am facing in my "SimpleSpy-like" script?

"Standard code" script w/indexrelative for TC2000 attached, execution results attached(7min)

Regards,

Doug

[junkew]

I think its slow because

a. the extensive debugging that is default written to log.txt
    turn the logging off (check _UIA_DEBUG function as I am not at my development workstation to tell exact details).

b. the _UIA_Action is trying to determine the context to search in.

If you need that control frequently you can write

$quickOBJ=_UIA_ACTION(<your reference>,"getobject") or

$quickOBJ=_UIA_ACTION(<your reference>,"attach")

and later on you do _UIA_Action($quickOBJ, "click")

if you read the log.txt you can actually see how many objects its trying before it found the needed object(s)

the appcrash is due to 10 and 9 in the simplespy for the array where parents are stored in. Make it 50 and 49 and it should work better.

Later this week I will post a new version (am testing now all the examples on regression issues)

[redrum]

OK, thanks a lot, I will work on this approach.

[redrum]

If you need that control frequently you can write

$quickOBJ=_UIA_ACTION(<your reference>,"getobject") or

$quickOBJ=_UIA_ACTION(<your reference>,"attach")

 

and later on you do _UIA_Action($quickOBJ, "click")

*************

Hello,

I interpreted the above statements to mean that once I fined an element (which takes awhile) I can store that as an "object variable"?, and then I can use that to click on (or get a property of) that element.  Is that correct?  I have been trying to do this and it seems that when I try to use the "quickOBJ" variable my code does not recognize it as an object.  I tried both "getobject" and "attach", both statement execute without an error, but I cannot use the resulting $quickOBJ, as I get an indication that it is not an object.

[junkew]

sorry for that its currently under work in progress on my desktop your assumption is correct but I think I did not add that version yet to first post

at some case statements in UIA_Wrappers in function _UIA_action

Func _UIA_action($obj_or_string, $strAction, $p1 = 0, $p2 = 0, $p3 = 0, $p4 = 0)

Case "getobject", "object"
            Return $obj2ActOn
        Case "attach"
            Return $obj2ActOn
        case "capture","screenshot"
            $tRect = StringSplit(_UIA_getPropertyValue($obj2ActOn, $UIA_BoundingRectanglePropertyId), ";")
;~          _UIA_DrawRect($t[1], $t[3] + $t[1], $t[2], $t[4] + $t[2])

            consolewrite($p1 & ";" & $tRect[1] & ";" & ($tRect[3] + $tRect[1]) & ";" &  $tRect[2]& ";" &  ($tRect[4] +  $tRect[2]))
            _ScreenCapture_Capture($p1, $tRect[1], $tRect[2], $tRect[3] + $tRect[1], $tRect[4] + $tRect[2])

Hard to put a new Work In Progress zip to first post as I am rewriting some stuff

Speed is relative but

1. build stuff in C/C++ and you will get speed

2. The UIAWrappers have as main benefit

a. it supports regular expressions

b. it simplifies stuff but hides the complex part a little and speed is not of main concern (although I do not recognize the speed you gave in minutes)

c. it abstracts even more with the _UIA_action finding dynamically the right context (which slows sometimes down stuff if it does not directly knows where to search in tree)

3. Go to the direct UIA functions of the treewalkers in the first few post explaining how you can use the treewalkers

You could find for example an element based on its name (which normally is quick) and with treewalker you sible to the one you want (assuming the element itself does not have a unique id / name)

Edited September 7, 2014 by junkew

[redrum]

Ok, many thanks!  I am making good progress here on making an acceptable script.  Being able to store the object found and click on it repetitively will be, I think, a big help.  So, please let me know when you provide a new version so I can update.  For now, I have acceptable workarounds.

[redrum]

Hello,

I have been using the UI Automation functions, on-going since I started this post awhile back, and they are working well for me.  The specific way I'm using the functions is by using the coordinates of a screen element to retrieve the control object of that element, then assign a variable to that object so I can use it throughout my script for control, etc.

I have some screen elements (horizontal and vertical line positions for example) that move to different locations on the screen making it impractical to use coordinates to retrieve the controls.

Question:  Is there a method of finding controls for screen elements (control elements and displayed-value elements)  without the need to start with the screen position coordinates?

Example:  I have a graph on a screen that has a horizontal line. The specific Y value of that line is displayed at the right end of the line.  This line, and it's displayed "Y" value, assume different positions up/down on the screen.  I need to find the control of this displayed value so I can read it, but the coordinates of the screen position are always changing.  

For all other (fixed) elements on the screen that I need, I identify the coordinates and store them as constants.  This allows retrieval of the control objects for those.

Thanks in advance for any suggestions that may help me resolve this.

Doug

 

 

[junkew]

What is simplespy giving

[redrum]

Hello,

Attached is text file of what Simplespy returned.  I tried to run the standard code to access the value of the element.  Std code run time is long, and does not appear to find the element (red box does not appear on element).

Also, simplespy would only run with my Beta version of AutoIt - I suspect this is normal?

I'm trying to access the value of this element that is displayed on the screen.  The positioning of this displayed element moves to different positions on the screen.  I'm hoping that if I can get an object for this element, that the object will remain valid as the element moves to different positions on the screen.

Not sure what to try at this point.

Thanks

 

PriceSpyTradeFire.txt

[junkew]

Hard to tell can you try thru internet explorer. No idea how well silverlight works with chrome.

[redrum]

Here is the Simplespy data for the same element using IE - attached file

I now trying the standard code using IE

PriceSpyIEbrowser.txt

[junkew]

I am not an expert on silverlight but it seems not to be implemented on that websit. https://msdn.microsoft.com/en-us/windows/desktop/gg591270.aspx

[redrum]

OK, thanks.  Std code from simplespy does not find element (at least it is not identified in red box).  It does put red box around the webpage window.  I will research silverlight more to try make progress.  Accessing controls that have fixed screen coordinates has been working fine for that website.  Thanks for the ms link, I'll start there.