Scrape Javascript Rendered Directory

[NassauSky]

Hi, 

Working on getting the latest chrome driver  from https://chromedriver.storage.googleapis.com/index.html but the page is pure javascript that gets generated after the user visits it. Standard scraping isn't working. For example:

#include <IE.au3>

$Url = "https://chromedriver.storage.googleapis.com/index.html"
$oIE = _IECreate($Url)
$oRows = _IETagnameGetCollection($oIE, "tr") 
ConsoleWrite("oRows:" & $oRows.Length & @CRLF)

Results in the following output:

oRows:0
>Exit code: 0    Time: 0.5149

 

Thanks for any ideas and help 🙂

 

 

Edited December 21, 2019 by NassauSky
Added resulting output

[NassauSky]

So far this. Seems maybe I had to add a sleep before scraping but this code works slow as molasses

 

#include <IE.au3>

$Url = "https://chromedriver.storage.googleapis.com/index.html"
$oIE = _IECreate($Url)
Sleep(2000) ; So far sleeping 2 seconds here gets results
$oRows = _IETagnameGetCollection($oIE, "tr")
ConsoleWrite("oRows:" & $oRows.Length & @CRLF)
For $oRow in $oRows
    $innerString = ''
    If StringLen($oRow.innerText) > 0 And $oRow.innerText <> "-" Then
      $aChromeVer = StringSplit($oRow.innerText,".")
      If StringIsDigit ( $aChromeVer[1] ) And Number($aChromeVer[1]) > 2 Then ;
         ConsoleWrite("->" & $aChromeVer[1] & @CRLF)
      EndIf
      $oDatas = _IETagnameGetCollection($oRow, "td") ;<--this slowed up the process looking for the subset of data
    EndIf
 Next
_IEQuit($oIE)

Takes so much longer than loading the page to extract some data. Is there a way to speed this up?

 

EDIT UPDATE: I found out that the next step $oData = _IETagnameGetCollection was slowing down the scraping.

Edited December 21, 2019 by NassauSky
Removed code not yet necessary to operate

[Danp2]

@NassauSky Did you review the code posted by @CYCho?

P.S. I'm already working on turning this into a helper function that works with Chrome and Firefox.

[NassauSky]

@Danp2 Excellent! Don't know how I missed that. 

Thanks!

[Nine]

@NassauSky or you could have used table functions like this :

#include <IE.au3>
#include <Array.au3>

$Url = "https://chromedriver.storage.googleapis.com/index.html"
$oIE = _IECreate($Url)
Sleep(1000) ; So far sleeping 1 second here gets results

$oTable = _IETableGetCollection($oIE,0)
$aTable = _IETableWriteToArray($oTable, True)
_ArrayDisplay ($aTable)
_IEQuit($oIE)

Instant fast 

[NassauSky]

Thanks @Nine

That makes sense too.

[Subz]

You could also just use childNodes.item(index) as well, which is useful for getting attributes.

#include <IE.au3>

$Url = "https://chromedriver.storage.googleapis.com/index.html"
$oIE = _IECreate($Url)
Sleep(2000) ; So far sleeping 2 seconds here gets results
$oRows = _IETagNameGetCollection($oIE, "tr")
For $oRow In $oRows
    If $oRow.childNodes.Length = 5 Then
        If Int($oRow.childNodes.item(1).innerText) > 2 Then
            ConsoleWrite(Int($oRow.childNodes.item(1).innerText) & " - " & $oRow.childNodes.item(1).childNodes.item(0).href & @CRLF)
        EndIf
    EndIf
Next

 

Edited December 23, 2019 by Subz

[NassauSky]

Not bad, yes I like.

 

Thanks @Subz