Jump to content
Seminko

How to grab javascript generated data?

Recommended Posts

Seminko

Is there a way to grab non-hardcoded but rather javascript generated data from a webpage?

Tried a get request as well as _IEBodyReadHTML but both seem to grab the code without the javascript generated data.

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
$oHTTP.Open("GET", "link", False)
$oHTTP.Send()
$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status
Global $DataArray[10][5]

If $oStatusCode <> 200 Then
    Exit
    MsgBox(1, "Error", "Status Code <> 200")
EndIf

FileWrite(@ScriptDir & "\output.txt", $oReceived)

; ////////

#include <IE.au3>

Local $FullLink = "link"
Local $oIE = _IECreate($FullLink, 0, 0)
_IELoadWait($oIE)
Local $sText = _IEBodyReadHTML($oIE)
FileWrite(@ScriptDir & "\output.txt", $sText)

 

Share this post


Link to post
Share on other sites
Seminko

So as it turns out _IEBodyReadHTML does what I'm looking for, but unfortunately the site seems to have trouble with Internet Explorer - it runs smoothly in Chrome, but in IE it totally bugs out.

I tried the WebDriver UDF but haven't been able to find the option to create a hidden window just like _IECreate does.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • SkysLastChance
      By SkysLastChance
       
      WinActivate("MEDITECH - Internet Explorer") Sleep (500) $oIE = _IEAttach("MEDITECH") $oDiv1 = _IEGetObjById($oIE, "sysmenu-searchbarbutton") _IEAction($oDiv1, "click") I am just trying to click the little magnifying glass, next to the gear button with no luck. I was hoping someone might have an idea why this is not working?
       

    • guinness
      By guinness
      Just trying out the latest version of AutoIt and thinking more functional
      #include <Array.au3> ; Example ; An example of filtering, mapping and reducing arrays, using a function reference. ; This is similiar to how it would be done in the likes of JavaScript ; i.e. more functional (declarative) than procedural (imperative) ; Filter example Local $aiFilteredBefore[] = [1, 2, 3, 50, 30, 40, 20, 30] Local $aiFilteredAfter = _ArrayFilter($aiFilteredBefore, GtrThan30) _ArrayDisplay($aiFilteredAfter, '_ArrayFilter::') ; Map example Local $aiMappedBefore[] = [1, 2, 3, 4, 5, 6, 7, 8, 9] Local $aiMappedAfter = _ArrayMap($aiMappedBefore, MultiplyByTwo) _ArrayDisplay($aiMappedAfter, '_ArrayMap::') ; Reduce example ; Sum all values in the array Local $aiReducedBefore[] = [1, 2, 3, 50, 30, 40, 20, 30] ConsoleWrite('_ArrayReduce:: ' & _ArrayReduce($aiReducedBefore, SumValues) & @CRLF) ; Passing an empty array, will return the initial value; otherwise, sets @error to 4 ; if no initial value is defined Local $aEmpty[] = [] ConsoleWrite('_ArrayReduce:: ' & _ArrayReduce($aEmpty, SumValues, 0) & @CRLF) ; Array callback functions (for the examples only) Func GtrThan30($iValue) Return $iValue > 30 EndFunc ;==>GtrThan30 Func MultiplyByTwo($iValue, $iIndex, $aiArray) ; Notice how the function is called with the optional arguments "index" and "original array" ConsoleWrite('Index:: ' & $iIndex & ', Array:: ' & _ArrayToString($aiArray) & @CRLF) Return $iValue * 2 EndFunc ;==>MultiplyByTwo Func SumValues($a, $b) Return $a + $b EndFunc ;==>SumValues ; Functions ; The callback function is invoked with fn(value, [index, [array]]) Func _ArrayFilter($avArray, $hFunc) If Not IsArray($avArray) Then ; Null is more appropriate than returning the likes of -1 or an empty array Return SetError(1, 0, Null) EndIf If Not IsFunc($hFunc) Then Return SetError(2, 0, Null) EndIf Local Const $iLength = UBound($avArray) Local $avFiltered[$iLength] If $iLength = 0 Then Return $avFiltered EndIf Local $iIndex = 0 For $i = 0 To $iLength - 1 Local $bIsFiltered = __ArrayCall($hFunc, 3, $avArray[$i], $i, $avArray) If @error Then Return SetError(@error, @extended, Null) ElseIf $bIsFiltered Then $avFiltered[$iIndex] = $avArray[$i] $iIndex += 1 EndIf Next ReDim $avFiltered[$iIndex] Return $avFiltered EndFunc ;==>_ArrayFilter ; The callback function is invoked with fn(value, [index, [array]]) Func _ArrayMap($avArray, $hFunc) If Not IsArray($avArray) Then ; Null is more appropriate than returning the likes of -1 or an empty array Return SetError(1, 0, Null) EndIf If Not IsFunc($hFunc) Then Return SetError(2, 0, Null) EndIf Local Const $iLength = UBound($avArray) Local $avMapped[$iLength] If $iLength = 0 Then Return $avMapped EndIf For $i = 0 To $iLength - 1 $avMapped[$i] = __ArrayCall($hFunc, 3, $avArray[$i], $i, $avArray) If @error Then Return SetError(@error, @extended, Null) EndIf Next Return $avMapped EndFunc ;==>_ArrayMap ; The callback function is invoked with fn(current, value, [index, [array]]) Func _ArrayReduce($avArray, $hFunc, $vInitial = Default) If Not IsArray($avArray) Then ; Null is more appropriate than returning the likes of -1 or an empty array Return SetError(1, 0, Null) EndIf If Not IsFunc($hFunc) Then Return SetError(2, 0, Null) EndIf Local $bHasInitial = @NumParams >= 3 Local $iLength = UBound($avArray) If $iLength = 0 Then If Not $bHasInitial Then Return SetError(4, 0, Null) EndIf Return $vInitial EndIf For $i = 0 To $iLength - 1 If $bHasInitial Then $vInitial = __ArrayCall($hFunc, 3, $vInitial, $avArray[$i], $i, $avArray) If @error Then Return SetError(@error, @extended, Null) EndIf Else $bHasInitial = True $vInitial = $avArray[$i] EndIf Next Return $vInitial EndFunc ;==>_ArrayReduce Func __ArrayCall($hFunc, $iError, $vArg1 = Default, $vArg2 = Default, $vArg3 = Default, $vArg4 = Default) Local Const $CALL_ERROR = 0xDEAD Local Const $CALL_EXTENDED = 0xBEEF Local $vRet = Call($hFunc, $vArg1) If @error = $CALL_ERROR And @extended = $CALL_EXTENDED Then $vRet = Call($hFunc, $vArg1, $vArg2) If @error = $CALL_ERROR And @extended = $CALL_EXTENDED Then $vRet = Call($hFunc, $vArg1, $vArg2, $vArg3) If @error = $CALL_ERROR And @extended = $CALL_EXTENDED Then $vRet = Call($hFunc, $vArg1, $vArg2, $vArg3, $vArg4) If @error = $CALL_ERROR And @extended = $CALL_EXTENDED Then ; The function exists, but there is no appropriate function signature Return SetError($iError, 0, Null) EndIf EndIf EndIf EndIf Return SetError(@error, @extended, $vRet) EndFunc ;==>__ArrayCall  
    • Burgs
      By Burgs
      Hello,
        I have a website with a Google Map I setup using the Google Map API.  It works and displays just fine.  However to make it useful to me I need to be able to dynamically change the map to display different areas by sending new Latitude and Longitude coordinates.  I am having difficulty making this happen.  Here is my code thus far:
      #include <IE.au3> $oIE3 = _IECreate("http://my_sample_website.html") ;just an example, not an actual site... _IELoadWait($oIE3) $s_word = "lat:" $oInputs = _IETagNameAllGetCollection($oIE3) if @error <> 0 Then MsgBox($MB_SYSTEMMODAL, "ERROR", "Error is: " & @error) EndIf ;@error For $oInput In $oInputs if Number($iPos) == -1 Then $iPos = StringInStr($oInput.innerHTML, String($s_word)) if (Number($iPos) > 0) AND (@error == 0) Then ConsoleWrite("I FOUND IT...! " & String($s_word) & @CRLF) $sHTML = _IEBodyReadHTML($oIE3) $_lat_look = 0 $_lng_look = 0 $_end_look = 0 ;default $_lat_look = StringInStr(String($sHTML), "lat:") if Number($_lat_look) <> 0 Then $_lng_look = StringInStr(String($sHTML), "lng:") if Number($_lng_look) <> 0 Then $_end_look = StringInStr(String($sHTML), "}") if Number($_end_look) <> 0 Then ConsoleWrite("HTML BODY: " & $sHTML & @CRLF) $_old_lat = String(StringMid(String($sHTML), $_lat_look, ($_lng_look - $_lat_look))) $_old_lng = String(StringMid(String($sHTML), $_lng_look, ($_end_look - $_lng_look))) ConsoleWrite("$_old_lat: " & $_old_lat & @CRLF) ConsoleWrite("$_old_lng: " & $_old_lng & @CRLF) $_new_lat = "lat: " & String("-34.397") & ", " $_new_lng = "lng: " & String("150.644") & "}; " ConsoleWrite("...new lat is: " & String($_new_lat) & " new lng is: " & String($_new_lng) & @CRLF) $_LOOK = StringReplace($_old_lat, 1, String($_new_lat)) $_LOOK2 = StringReplace($_old_lng, 1, String($_new_lng)) ConsoleWrite("$_LOOK: " & $_LOOK & "$_LOOK2: " & $_LOOK2 & @CRLF) EndIf ;'$_end_look' NOT "0"... $iPos = -1 EndIf ;'String($s_word)' was found in the collection '$oInputs' EndIf ;'$iPos' is "-1" Next  
        I am having trouble trying to replace the line in the HTML ($sHTML variable in my example) that contains the "lat:" and "lng:" information.  I figure if I can replace that line everything else remains the same, and in theory, the map should cycle to display a map with the new latitude and longitude coordinates...I hope. 
        I have attempted to write the $sHTML to a text document and then use '_IEBodyWriteHTML' to read it back into the webpage HTML however that is not working.  There must be an easier method to accomplish this...what am I missing here...?  Any thoughts greatly appreciated.  Regards.       
    • Chimp
      By Chimp
      An example on how to inject jQuery into a web page
      It can be useful to manage the page from AutoIt using jQuery.
      Idea from here: http://www.learningjquery.com/2009/04/better-stronger-safer-jquerify-bookmarklet
      Suggestions and improvements are welcome
      #include <ie.au3> Example() Func Example() Local $oIE = _IECreate("www.google.com") Local $jQuery = _jQuerify($oIE) MsgBox(0, "Version", "jQuery version: " & $jQuery.fn.jquery) MsgBox(0, "Example", "click ok to exit." & @CRLF & "Google logo will fade out by jQuery...") $jQuery('#hplogo').fadeOut(3000) ; jQuery will fade out the google logo EndFunc ;==>Example ; #FUNCTION# ==================================================================================================================== ; Name ..........: _jQuerify ; Description ...: ; Syntax ........: _jQuerify(Byref $oIE) ; Parameters ....: $oIE - Object variable of an InternetExplorer.Application. ; Return values .: an object variable pointing to the jQuery library ; Author ........: Chimp ; Modified ......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: ; =============================================================================================================================== Func _jQuerify(ByRef $oIE) Local $jsEval, $jQuery, $otherlib = False ; create a reference to the javascript eval() function $oIE.document.parentWindow.setTimeout('document.head.eval = eval', 0) Do Sleep(250) $jsEval = Execute('$oIE.Document.head.eval') Until IsObj($jsEval) ; if jQuery is not already loaded then load it If $jsEval("typeof jQuery=='undefined'") Then ; check if the '$' (dollar) name is already in use by other library If $jsEval("typeof $=='function'") Then $otherlib = True Local $oScript = $oIE.document.createElement('script'); $oScript.type = 'text/javascript' ; If you want to load jQuery from a disk file use the following statement ; where i.e. jquery-1.9.1.js is the file containing the jQuery source ; (or also use a string variable containing the whole jQuery listing) ;~ $oScript.TextContent = FileRead(@ScriptDir & "\jquery-1.9.1.js") ; <--- from a file ; If you want to download jQuery from the web use this statement $oScript.src = 'https://code.jquery.com/jquery-latest.min.js' ; <--- from an url $oIE.document.getElementsByTagName('head').item(0).appendChild($oScript) Do Sleep(250) Until $jsEval("typeof jQuery == 'function'") EndIf Do Sleep(250) $jQuery = $jsEval("jQuery") Until IsObj($jQuery) If $otherlib Then $jsEval('jQuery.noConflict();') Return $jQuery EndFunc ;==>_jQuerify  
    • XaelloNegative
      By XaelloNegative
      Hi there.
      Im having some kind of a coder block here, 
      <td class="treeview ctl00_CenterContent_objTreeView_2" style="white-space:nowrap;"> <input type="checkbox" name="ctl00_CenterContent_objTreeViewn1CheckBox" id="ctl00_CenterContent_objTreeViewn1CheckBox"> <span class="ctl00_CenterContent_objTreeView_0 treeview ctl00_CenterContent_objTreeView_1" id="ctl00_CenterContent_objTreeViewt1" style="border-style:none;font-size:1em;"> <i class="fa fa-file-code-o fa-fw"> </i> <span data-placement="bottom" data-toggle="tooltip" title="" data-original-title="Tool tip for said part">REFUNDS</span></span></td> this is a code directly from the website im working with. I was thinking of inserting a javascript that would get the "REFUNDS" part. Any ideas on how to do so?
      Any thoughts will be highly appreciated.
×