Jump to content
Sign in to follow this  
wisem2540

Help scraping data from webpage

Recommended Posts

wisem2540

I am appealing to my fellow autoit'ers for help.  I have tried to parse the sourcecode and the data that I am looking for appears to not be in plain text.  

 

I am looking for all of the relevant data, Hashrate, Last Round Reward and so on,  I apologize for not posting an example code but this is beyond my skill level with autoit.

here is the site.

 

https://eggpool.net/index.php?miner=55713816bf48b85cc1b03db521c0142f99402925d05e726b476a67a4&action=miner&submit=Show

Share this post


Link to post
Share on other sites
jdelaney

I'm not a fan of clicking links.   Send F12 at your browser, select the spy tool, click what you want to grab. ..copy the html, and paste it back here.   Or, wait for someone that doesn't mind clicking links from strangers :)


IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites
wisem2540

Thanks for the reply.  As I said, the text I want is not in plain view.  Its a query of some sort.  All the website is, is a Cryptomining stats page.  I have close to 400 posts here.  I am hardly a stranger and  I would not direct this community to some place malicious

Share this post


Link to post
Share on other sites
jdelaney

Not sure what to tell you...for a website to display anything requires html source, and that can be manipulated/read...even if it's just a file's text that is being displayed, you can still grab it, and do a regularexpression to parse out your data..  I'll defer to another that will work with the external links.


IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites
wisem2540

@jdelaney

Here is a C&P from the Miner Info section I am most interested in.  I hope this helps

 

 

</div>
  <div class="col-lg-3 col-xs-12">
   
  <div class="box box-info">
  <div class="box-header">
  <h3 class="box-title">Miner info</h3>
  </div>
  <!-- /.box-header -->
  <div class="box-body no-padding">
  <table class="table table-striped">
  <tr><td>Min. Payout</td><td><span id="minp">50</span></td><td>BIS</td></tr>
  <tr><td>Avg HR this round</td><td><span id="hr2">??</span></td><td>GH/s</td></tr>
  <tr><td>Avg HR last round</td><td><span id="lasthr">??</span></td><td>GH/s</td></tr>
  <tr><td>Reward last round</td><td><span id="immature2">??</span></td><td>BIS</td></tr>
  <tr><td>Est. round reward</td><td>(soon)</td><td>BIS</td></tr>
  <tr><td>Est. BIS/Day</td><td>(soon)</td><td>BIS</td></tr>
  <tr><td>Total Paid</td><td><span id="totalpaid">??</span></td><td>BIS</td></tr>
  </table>
  </div>
  <!-- /.box-body -->
  </div>
  <!-- /.box -->
  <!-- /div>

Share this post


Link to post
Share on other sites
jdelaney

Perfect, so it's a table that has lots of data...use this one:

_IETableWriteToArray

Look at the helpfile, try to get it working, and then you can loop through the returned array, and grab your data.

Helpfile example:

; Open a browser with the table example, get a reference to the first table
; on the page (index 0) and read its contents into a 2-D array

#include <Array.au3>
#include <IE.au3>

Local $oIE = _IE_Example("table")
Local $oTable = _IETableGetCollection($oIE, 0)
Local $aTableData = _IETableWriteToArray($oTable)

_ArrayDisplay($aTableData)

_IEQuit($oIE)

Example of looping through all the tables:

#include <Array.au3>
#include <IE.au3>

$oIE = _IECreate("https://eggpool.net/index.php?miner=55713816bf48b85cc1b03db521c0142f99402925d05e726b476a67a4&action=miner&submit=Show")
$oTables = _IETableGetCollection($oIE)
For $oTable In $oTables
    $aTableData = _IETableWriteToArray($oTable)
    _ArrayDisplay($aTableData)
Next

 

Edited by jdelaney
  • Like 1

IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites
wisem2540

@jdelaney

 

Holy crap that worked!  I will admit I had my doubts, but I think that is exactly what I needed.

 

Thanks!!

Share this post


Link to post
Share on other sites
jdelaney

A little less generic a script:

$sHTML = '<div class="box-body no-padding">     <table class="table table-striped">     <tr><td>Min. Payout</td><td><span id="minp">50</span></td><td>BIS</td></tr>     <tr><td>Avg HR this round</td><td><span id="hr2">??</span></td><td>GH/s</td></tr>   <tr><td>Avg HR last round</td><td><span id="lasthr">??</span></td><td>GH/s</td></tr>    <tr><td>Reward last round</td><td><span id="immature2">??</span></td><td>BIS</td></tr>  <tr><td>Est. round reward</td><td>(soon)</td><td>BIS</td></tr>  <tr><td>Est. BIS/Day</td><td>(soon)</td><td>BIS</td></tr>   <tr><td>Total Paid</td><td><span id="totalpaid">??</span></td><td>BIS</td></tr>     </table>    </div>'

#include <Array.au3>
#include <IE.au3>

$oIE = _IECreate("")
_IEBodyWriteHTML($oIE,$sHTML)
$oTables = _IETableGetCollection($oIE)
For $oTable In $oTables
    $aTableData = _IETableWriteToArray($oTable,True)
    For $i = 0 To UBound($aTableData)-1
        If $aTableData[$i][0] = "Min. Payout" Then
            ConsoleWrite("Minimum payout = " & $aTableData[$i][1] & " " & $aTableData[$i][2] & @CRLF)
        EndIf
    Next
    _ArrayDisplay($aTableData)
Next

You can add all the if statements you want.  Also, I'm lazy, so didn't do any error handling.

Edited by jdelaney

IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×