[Solved] Help and Guidance using Javascript & DOM with AutoIt

[icu]

Dear AutoIt Community,

I'm really struggling with automating a screen scrape of a website that uses extensive Javascript.

Normally I would fire up IE-Builder, grab the source code from there and then code the automation from the Html Elements, Anchor Tags or the Form Elements.

When IE-Builder loads the source code of the page it does not recognize the page's Html Elements, Anchor Tags or Form Elements as it seems as if the whole website is in Javascript and somehow outside of the normal tags that IE-Builder parses. Obviously this means I can't get an object to refer to in the AutoIt script.

Tricks like getting all the Tags and looping through them until I find what I need and then getting AutoIt to click it (somewhat like the following code below) don't work either probably due to the same problem IE-Builder is having.

$o_Elements = _IETagNameAllGetCollection($o_IE)
For $o_Element In $o_Elements
MsgBox(0, "Element Info", "Tagname: " & $o_Element.tagname & @CR & "innerText: " & $o_Element.innerText)
Next

I'm pretty sure it is still possible to automate the navigation of pages and the scraping of data in AutoIt as it has DOM abilities but I'm a newbie at DOM and don't have a clue how to use it with AutoIt. Google searches for info on DOM and AutoIt don't give a clear understanding of how to do things and really only point to specific forum questions that I can't glean any knowledge from.

Can anyone give general help and guidance in this regard? Specifically:

1) Are there references that will clearly explain how to use Javascript/DOM with AutoIt?

2) What are the abstract ideas for coding AutoIt automation when facing Javascript? Like, "should I grab the page source and loop through things and then create and object to click"... that sort of thing.

I'm certain others will be having this same problem and any help here would be of great benefit to others down the road.

Many thanks for the help.

Edited November 16, 2012 by icu

[SmOke_N]

Sounds like you're working in the main DOM, and the objects you're trying to reach are in a frame. Download debugbar and you'll see what I'm talking about.

[icu]

Hi SmOke_N,

Thank you for your reply.

Can you please point the way to using the info from debugbar? Or said in a different way, how do I extract the objects from the frame for web automation with AutoIt?

Also where would I go for learning information about DOM and AutoIt?

Edited November 12, 2012 by icu

[SmOke_N]

Look at the DOM area of the debugbar, then using the treepane, find the frame your dealing with.

Most times it has a name ( eg. mainframe ), use that with _IEFrameGetObjByName(), if not, use the index of the frame starting with 0 until your have yours.

Use the frame object as you would your browser object ( remember it's a window though ).

Trial and error with the functions you'd expect to work with the $oIE browser object with the $oFrame object window.

DebugBar makes it easy to get the id's and such that you need with treepane type navigation, cutting your work down considerably.

There's plenty of information on here about where to learn DOM, msdn has your main methods/properties/etc info.

[icu]

@SmOke_N: Thanks for your reply, but the internet is a vast place and quality of learning materials vary wildly. Even searching this forum for 'DOM Help' brings back many search results and while some are informative, none that I have seen point to a starting place for learning DOM and then applying it with AutoIt.

Yes, I know about MSDN which is a huge repository of information that I can't make much sense of--because I don't know how to use this stuff yet. And while you are right about trial and error being an important part of learning how to program I simply don't know what I don't know--which makes it hard to find a starting point. Take for example AuotIt, I started learning it from and without these two learning materials I wouldn't have had a starting point and I wouldn't have learnt AutoIT. However I now need to do some advanced stuff with IE and I can't seem to find any learning material for that except that using DOM is the right direction.

Edited November 13, 2012 by icu

[icu]

@SmOke_N: You were right, everything was in a frame.

$o_Frames = _IEFrameGetCollection ($o_IE)
$i_NumFrames = @extended
If $i_NumFrames > 0 Then
    If _IEIsFrameSet ($o_IE) Then
        MsgBox(0, "Frame Info", "Page contains " & $i_NumFrames & " frames in a FrameSet")
    Else
        MsgBox(0, "Frame Info", "Page contains " & $i_NumFrames & " iFrames")
    EndIf
Else
    MsgBox(0, "Frame Info", "Page contains no frames")
EndIf
For $i = 0 to ($i_NumFrames - 1)
    $o_Frame = _IEFrameGetCollection ($o_IE, $i)
    MsgBox(0, "Frame Info", _IEPropertyGet ($o_Frame, "locationurl"))
MsgBox(0, "Frame Info", _IEPropertyGet ($o_Frame, "innertext"))
Next

The above code confirms this. I now need to work out how to get the objects out of the frame but I imagine I can with _IEFrameGetObjByName. Can you please tell me if the frame object is destroyed every time the page is reloaded or a new page navigated to?