## Recommended Posts

Hello,

Before Broadcom took over Symantec, I was able to use the following code as the base to scrape daily definition downloads from the web. Since the pages were moved over to the Broadcom web servers, I get a page full of what I believe may be JavaScript instead of the fully rendered page that lies behind it. Does anyone have any suggestions as to how I can read the full rendered webpage using AutoIt?

#include <IE.au3>
$oIE_DEFS = _IECreate("https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14", 0, 0, 1, 1)$sString_DEFS = _IEBodyReadHTML($oIE_DEFS) MsgBox(0, "https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14",$sString_DEFS)

The above code will show the JavaScript, but if you go to the URL in a browser that has JavaScript enabled, you will see the fully rendered page that I would like to access. I hope my question makes sense, and I would appreciate any suggestions to get this working again.

All the best,
-Mike

##### Share on other sites

if  you  find a link use stringregrexp and  match all href

##### Share on other sites

Thank you for the suggestion, but unfortunately, the links to the files don't appear to be within the JavaScript...they only appear after the page is fully rendered.

##### Share on other sites

scroll the page

and after find a link use stringregrexp and  match all href

or  i remember  the  big company have also FTP anonymous  for download tools or other , try to look if symantec  have  , google is our friend , bye

##### Share on other sites

<SNIP>

When I ran my example, I thought I saw the rendered HTML but I appear to have been mistaken.  So I removed it.

Edited by TheXman
Removed example since it was incorrect.

"Any fool can know.  The point is to understand."  -Albert Einstein

##### Share on other sites
On 5/12/2020 at 3:29 PM, mdwerne said:

Hello,

Before Broadcom took over Symantec, I was able to use the following code as the base to scrape daily definition downloads from the web. Since the pages were moved over to the Broadcom web servers, I get a page full of what I believe may be JavaScript instead of the fully rendered page that lies behind it. Does anyone have any suggestions as to how I can read the full rendered webpage using AutoIt?

#include <IE.au3>
$oIE_DEFS = _IECreate("https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14", 0, 0, 1, 1)$sString_DEFS = _IEBodyReadHTML($oIE_DEFS) MsgBox(0, "https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14",$sString_DEFS)

The above code will show the JavaScript, but if you go to the URL in a browser that has JavaScript enabled, you will see the fully rendered page that I would like to access. I hope my question makes sense, and I would appreciate any suggestions to get this working again.

All the best,
-Mike

When I am writing a script to scrape a website, there are several ways that I go about it:

1) If the html is fully available, then that is the easiest way to go about it

then I use the wget tool and have it handle the redirections and force the file download into the file format that I know it is supposed to be in.  So, using the above example, I know the file is supposed to be a zip file, so I would run the wget command

Oh, also, wget is native to linux operating systems, but there is a very decent windows port of wget available here

##### Share on other sites

Thanks for the reply, @MattHiggs. This is something new to try as I have been unable to get at the files any other way. I haven't played with WGET much, but I'll give it a shot. Manually downloading a handful of different definition sets, every night, from this site: https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14
is getting kinda old.

-Mike

P.S. Part of the issue is also that the download files names change a few times a day...which is why I need to scrape the fully rendered Javascript (HTML) page each time.

Edited by mdwerne
Forgot some info...

## Create an account

Register a new account

• ### Similar Content

• By Gianni
This script is an alternative (improvement) to the script published at this link (www.autoitscript.com/forum/topic/186225-interesting-effect-with-transparency). In that post a flash animation was used and therefore you need the flash player to work, while here only javascript is used.
Here, too, the aim is to show an interesting effect that can be obtained by exploiting transparency, but can also be used as it is just for fun.
The animation consists of some fish wandering around the screen. You can continue working on the computer by letting them swim.
The javascript source was not written by me, but I got it from this link: https://github.com/lrusso/Aquarium. I shrunk it down a bit and tweaked it so that fish can swim across the entire screen even with different sized monitors.
The fish images and html and javascript sources have been bundled into a single "monolithic" AutoIt script, so it's easier to run.
I hope you find it relaxing.
Have fun
fishMonolitic.zip
• By UEZ
This project has been discontinued!

Here a small tool I wrote to update my Sysinternal tools collection without the need to download always the whole package or visiting the site to check for updates. I know that there are several tools available (also some tools written in AutoIt) but here another one for the collection. It was good exercise for me to code it.

Some files from the live web site cannot be downloaded although they are visible!

-=> Requires AutoIt version 3.3.13.20 or higher / tested on Win8.1 real machine and some VMs: Win7 / Vista / Win10

Compiled exe only: @MediaFire

Just select the Sysinternal Tools folder or create one and press the synchronize button to download the selected items. Click on AutoIt label (near to left upper corner) to open menu.

Special thanks to LarsJ, Melba23 and mesale0077 for their help.

I've still some ideas to implement which are more gimmick related, so it is not finished yet...

Please report any bug or if you have any suggestions.

The language of the tool tip from each of the executable in the left list view were automatically created using Google translator and weren't checked for correctness.

Br,
UEZ
• By Gianni
Making the integration between AutoIt and the BrowserControl easier may facilitate the development of interesting programs. Just think of all those (fantastic) libraries and frameworks available for javascript that could be integrated and exploited in an AutoIt program. For example, to graphically present the results of an AutoIt processing in the GUI, ... and many other possibilities. Providing the basic functions to implement this synergistic interaction is the purpose of this post (if for no other reason, just even to collect some tests done over time here and there, so as not to leave them scattered and easily find them in just one place)
In this UDF there are only a few functions (or better if called wrappers), but I hope to add more as I go and, even better, to receive suggestions and advice on new features and techniques to improve and expand it.
_WebBrowser_GUICtrlCreate Create an Internet Explorer 'Browser Control' _WebBrowser_SetHTML Set a new HTML listing in the BrowserControl _WebBrowser_CSS_Inject Creates into the html document a CSS node element with embedded the passed CSS _WebBrowser_JS_Inject Creates into the html document a javascript node element with embedded the passed javascript _WebBrowser_JS_Eval Evaluates a passed string as JavaScript code and executes it _WebBrowser_JS_setTimeout Calls a javascript function or executes a javascript code snippet [option after a specified delay] _WebBrowser_JS_ElementGetRef Retrieves a reference to an element suitable to be used from AutoIt _WebBrowser_ExecuteDotNotation Get a reference to an object's child member or the value of a property, by means of a dotted path _WebBrowser_BasicHTML Returns a basic html page listing (a little enhanced than about:blank) The three examples provided in the attached ziped file are a bit 'improvised' and do not take advantage of all the possibilities offered by the underlying javascript libraries used. They are just three "hello world" scripts to test the ABC of the interaction with the "BrowserControl".
(ToDo: Interaction with javascript custom events)
Bug reports, creative criticisms and suggestions (particularly regarding the interaction with javascript custom events) are welcome
I hope you can have fun with the Browser Control
BrowserControl.zip
• By Gianni
This is a little experiment that makes use of a "Browser Control" embedded in a GUI in order to be able to use AutoIt, HTML, JavaScript and CSS all together.
This little toy will only work on systems with IE11.
The purpose is to drag all the names of the scientists & drop on the right ones. (among scientists it has also infiltrated an intruder). I hope you find it entertaining.
I've posted it here in the hope to have some suggestions on how to improve it all (I mean the interaction between Autoit Javascript html and css). Thanks
; this works on systems with IE11 ; ------------------------------- #include <GUIConstantsEx.au3> #include <array.au3> Global $oIE,$oDocument, $ohJS,$sDroping Global $iCorrect = 0,$iGoal = 11 Example() Exit Func Example() Local $aScientists[12][2] = _ [["Schrodinger", "Schrodinger"],["Planck", "Planck"],["Pauli", "Pauli"],["Einstein", "Einstein"], _ ["Chimp", "Chimp"],["Dirac", "Dirac"],["Heisenberg", "Heisenberg"],["Born", "Born"], _ ["De Broglie", "De_Broglie"],["Bohr", "Bohr"],["Sommerfeld", "Sommerfeld"],["", "empty"]] Local$oIE = ObjCreate("Shell.Explorer.2") ; Create a BrowserControl Local $hGUI = GUICreate("", 660, 600, 30, 30) GUICtrlCreateObj($oIE, 0, 0, 660, 600) ; Place BrowserControl on the GUI GUISetState() ;Show GUI $oIE.navigate('about:blank') ; file:///' & @ScriptDir & '\WhoIsWho.html') While Not String($oIE.readyState) = 'complete' ; wait for about:blank Sleep(100) WEnd ; this waits till the document is ready to be used (portion of code from IE.au3) While Not (String($oIE.readyState) = "complete" Or$oIE.readyState = 4) Sleep(100) WEnd While Not (String($oIE.document.readyState) = "complete" Or$oIE.document.readyState = 4) Sleep(100) WEnd $oIE.document.Write(_GetHTML()) ; inject lising directly to the HTML document:$oIE.document.close() ; close the write stream $oIE.document.execCommand("Refresh") While Not String($oIE.readyState) = 'complete' ; wait for readyState after a refresh Sleep(100) WEnd ; https://msdn.microsoft.com/en-us/library/52f50e9t(v=vs.94).aspx ; $ohJS is a reference to the javascript Global Obj ; -------------------------------------------------$ohJS = $oIE.document.parentwindow.JSglobal$oDocument = $oIE.document Local$oTable1 = $ohJS.table1 Local$oTable2 = $ohJS.table2 _Randomize($aScientists, $oTable1) ; --- Setup events ------------ ; https://msdn.microsoft.com/en-us/library/hh801967(v=vs.85).aspx Local$aoEventObject[2] $aoEventObject[0] = ObjEvent($oTable1, "IEEvent_", "HTMLTableEvents2") $aoEventObject[1] = ObjEvent($oTable2, "IEEvent_", "HTMLTableEvents2") ; ----------------------------- ; Loop until the user exits. While 1 Switch GUIGetMsg() Case $GUI_EVENT_CLOSE ExitLoop EndSwitch If$iCorrect = $iGoal Then _Goal() _Randomize($aScientists, $oTable1)$iCorrect = 0 EndIf WEnd ; the end For $i = 0 To UBound($aoEventObject) - 1 $aoEventObject[$i] .stop Next $aoEventObject = 0 ; Kill the Event Object$oIE = 0 ; Remove IE from memory (not really necessary). GUIDelete($hGUI) ; Remove GUI EndFunc ;==>Example ; --- event management zone --- ; following functions are fired by events occurred in the browser control Volatile Func IEEvent_onDragstart($oEvent) ; we save the ID of the dragged obj into the $sDroping variable ; for a later use in the drop function$sDroping = $oEvent.srcElement.ID EndFunc ;==>IEEvent_onDragstart Volatile Func IEEvent_onDragOver($oEvent) $ohJS.event.preventDefault() EndFunc ;==>IEEvent_onDragOver Volatile Func IEEvent_onDrop($oEvent) $ohJS.event.preventDefault() If$sDroping <> "" Then If $oDocument.getElementById($sDroping).innerText <> $oEvent.srcElement.innerText Then If$oEvent.srcElement.ClassName = $oDocument.getElementById($sDroping).ClassName Then $oEvent.srcElement.innerText =$oDocument.getElementById($sDroping).innerText$oDocument.getElementById($sDroping).innerText = ""$oDocument.getElementById($sDroping).draggable = False$oDocument.getElementById($sDroping).setAttribute("style") = "background-color: #80ff80;"$iCorrect += 1 Else For $i = 1 To 3$oDocument.getElementById($sDroping).setAttribute("style") = "background-color: #ff0000;" Sleep(125)$oDocument.getElementById($sDroping).setAttribute("style") = "background-color: #ffffff;" Sleep(125) Next EndIf EndIf$sDroping = "" EndIf EndFunc ;==>IEEvent_onDrop Func _Randomize(ByRef $aScientists, ByRef$oTable) Local $iRows = ($oTable.rows.length) - 1 Local $iCols = ($oTable.rows.item(0).cells.length) - 1 Local $index _ArrayShuffle($aScientists) For $y = 0 To$iRows For $x = 0 To$iCols $index = ($y * ($iCols + 1)) +$x $oTable.rows.item($y).cells.item($x).innerText =$aScientists[$index][0] ; text$oTable.rows.item($y).cells.item($x).className = $aScientists[$index][1] ; class $oTable.rows.item($y).cells.item($x).draggable =$aScientists[$index][0] <> "" Next Next EndFunc ;==>_Randomize Func _Goal() Local$oTable1 = $ohJS.table1 ; names Local$oTable2 = $ohJS.table2 ; photos Local$iRows = ($oTable1.rows.length) Sleep(250) Local$iCols = 6 ; ($oTable1.rows.item(0).cells.length) Local$aIndex[$iRows *$iCols], $sTemp For$i = 0 To UBound($aIndex) - 1$aIndex[$i] =$i ; + 1 Next _ArrayShuffle($aIndex) For$i = 0 To UBound($aIndex) - 1$oTable2.rows.item(Int($aIndex[$i] / $iCols)).cells.item(Mod($aIndex[$i],$iCols)).innerText = "" $oTemp0 =$oTable2.rows $oTemp1 =$oTemp0.item(Int($aIndex[$i] / $iCols)).cells$oTemp2 = $oTemp1.item(Mod($aIndex[$i],$iCols)).getAttribute("style") $oTable2.rows.item(Int($aIndex[$i] /$iCols)).cells.item(Mod($aIndex[$i], $iCols)).setAttribute("style") = "background-color: " & _rndColor() Sleep(100); MsgBox(0,"Debug",$sTemp) $oTable2.rows.item(Int($aIndex[$i] /$iCols)).cells.item(Mod($aIndex[$i], $iCols)).setAttribute("style") =$oTemp2 Next For $x = 1 To 2 For$i = 0 To UBound($aIndex) - 1$oTable1.rows.item(Int($aIndex[$i] / $iCols)).cells.item(Mod($aIndex[$i],$iCols)).setAttribute("style") = "background-color: " & _rndColor() Sleep(100) $oTable1.rows.item(Int($aIndex[$i] /$iCols)).cells.item(Mod($aIndex[$i], $iCols)).setAttribute("style") = "background-color: #ffffff;" Next Next EndFunc ;==>_Goal Func _rndColor() Return String("#" & Hex(Random(0, 255, 1), 2) & Hex(Random(0, 255, 1), 2) & Hex(Random(0, 255, 1), 2) & ";") EndFunc ;==>_rndColor Func _GetHTML() Local$sHTML = _ "<!DOCTYPE HTML>" & @CRLF & _ "<html>" & @CRLF & _ "<head>" & @CRLF & _ "<meta http-equiv=""X-UA-Compatible"" content=""IE=edge"" />" & @CRLF & _ " <script type=""text/javascript"">" & @CRLF & _ " var JSglobal = (1,eval)(""this"");" & @CRLF & _ " </script>" & @CRLF & _ "</head>" & @CRLF & _ "<body>" & @CRLF & _ "<h2>Who is who?</h2>" & @CRLF & _ "<p>Drag&Drop names on the right scientist</p>" & @CRLF & _ "<img src=""https://i.imgur.com/STWql7w.jpg""" & @CRLF & _ "height=""394"" width=""640""" & @CRLF & _ "style=""position: absolute; left: 10px; top: 100px;"">" & @CRLF & _ "" & @CRLF & _ "<style>" & @CRLF & _ ".target td {width: 100px; height: 160px; text-align: center; color: white; font-weight: bold; vertical-align: bottom; border: 0px solid grey;}" & @CRLF & _ ".source td {width: 100px; height: 30px; text-align: center; border: 1px solid red;}" & @CRLF & _ "</style>" & @CRLF & _ "" & @CRLF & _ "<table class=""target"" style=""position: absolute; left: 25px; top: 100px;"" id=""table2"">" & @CRLF & _ " <tr><td class=""Schrodinger""></td><td class=""Planck""></td><td class=""Pauli""></td><td class=""Einstein""></td><td class=""Chimp""></td><td class=""Dirac""></td></tr>" & @CRLF & _ " <tr><td class=""Heisenberg""></td><td class=""Born""></td><td class=""De_Broglie""></td><td class=""Bohr""></td><td class=""Sommerfeld""></td><td class=""empty""></td></tr>" & @CRLF & _ "</table>" & @CRLF & _ "" & @CRLF & _ "<table class=""source"" style=""position: absolute; left: 10px; top: 504px;"" id=""table1"">" & @CRLF & _ " <tr><td ID=""td1""></td><td ID=""td2""></td><td ID=""td3""></td><td ID=""td4"" ></td><td ID=""td5"" ></td><td ID=""td6"" ></td></tr>" & @CRLF & _ " <tr><td ID=""td7""></td><td ID=""td8""></td><td ID=""td9""></td><td ID=""td10""></td><td ID=""td11""></td><td ID=""td12""></td></tr>" & @CRLF & _ "</table>" & @CRLF & _ "</body>" & @CRLF & _ "</html>" Return $sHTML EndFunc ;==>_GetHTML • By Phil1991 Hi, I'm working on an interface and I have a problem with zoom management. In desktop mode it works and in android cell mode it seems to work fine too. However, on a Windows tablet, it seems to have a problem ... Basically what I'm trying to do is display a modal in bootstrap 4. But when on the tablet I zoom, my modal s really big and out of my screen. On the android mobile, I remove the zoom before displaying the modal but on the Windows tablet I am unable to reproduce the same behavior. My toggle function to reset zoom before display modal ( Yeah it maybe sketches but I'm not find other solutions) function resetZooming() { if (zoomOut) {$('meta[name=viewport]').attr('content', 'width=device-width, initial-scale=1.0, maximum-scale=1'); setTimeout(resetZooming, 1000); zoomOut = false; }else{ $('meta[name=viewport]').attr('content', 'width=device-width, initial-scale=1.0'); zoomOut = true; } } And I call this function under my event$("#modalCreationForm").modal("show"); resetZooming();
If you have any ideas
Thank you
×

• Wiki

• Back

• #### Beta

• Git
• FAQ
×
• Create New...