Jump to content
Sign in to follow this  
KimberlyJillPereira

Extract table from website until the end

Recommended Posts

#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]

$oIE = _IECreate("http://www.timeview2.net/alarms", 1)
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    $oIE = _IECreate("http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

Edited by Subz
Added Table Header + Removed unwanted spaces from table results

Share this post


Link to post
Share on other sites

Thanks it works but then I want to type in username and password automatically then click submit button and then click on Alarms button to extract all the data but then it is already logged in and then extracts all the data. I copied my code to it and it only automatically types in username and password but then it doesnt click submit button. I used _IEFormSubmit($oDivs) and it doesnt work and it shows warning from function _IEAttach, $_IESTATUS_NoMatch even clicking on alarms button also doesnt work. I used this code _IELinkClickByText($oIE, "Alarms") and it also shows warning Warning from function _IELinkClickByText, $_IESTATUS_NoMatch by the way I used these codes in a seperate program and it worked but when I use my codes together with yours it doesnt work. Please help me. Thanks.

Share this post


Link to post
Share on other sites

Try this, just replace values for $sUserName and $sPassWord.

#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]
Local $oUserName, $oPassWord
Local $sUserName = "user"
Local $sPassWord = "demo"

$oIE = _IECreate("http://www.timeview2.net/", 1)
$oForms = _IETagNameGetCollection($oIE, "form")
For $oForm In $oForms
    If $oForm.id = "UserLoginForm" Then
        $oUserName = _IEFormElementGetObjByName($oForm, "data[User][username]")
        _IEFormElementSetValue($oUserName, $sUserName)
        $oPassWord = _IEFormElementGetObjByName($oForm, "data[User][password]")
        _IEFormElementSetValue($oPassWord, $sPassWord)
        _IEFormSubmit($oForm)
    EndIf
Next

$oIE = _IECreate("http://www.timeview2.net/alarms", 1)
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    $oIE = _IECreate("http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

Share this post


Link to post
Share on other sites
#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]
Local $oUserName, $oPassWord
Local $sUserName = "user"
Local $sPassWord = "demo"

$oIE = _IECreate("http://www.timeview2.net/", 1)
$oForms = _IETagNameGetCollection($oIE, "form")
For $oForm In $oForms
    If $oForm.id = "UserLoginForm" Then
        $oUserName = _IEFormElementGetObjByName($oForm, "data[User][username]")
        _IEFormElementSetValue($oUserName, $sUserName)
        $oPassWord = _IEFormElementGetObjByName($oForm, "data[User][password]")
        _IEFormElementSetValue($oPassWord, $sPassWord)
        _IEFormSubmit($oForm)
    EndIf
Next

_IENavigate($oIE, "http://www.timeview2.net/alarms")
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    _IENavigate($oIE, "http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

Share this post


Link to post
Share on other sites

If you look at the results.csv file it already excludes the Actions column, the _ArrayDisplay shows the full table but I only wrote column 0 to 5 in the csv, see code after :~ Save to CSV

Share this post


Link to post
Share on other sites

You can actually remove _ArrayDisplay line it doesn't really mean anything, I just wanted to show that it had grabbed the table, the results that are saved to csv are only Col 0 to Col 5.  If you want to keep _ArrayDisplay then just use _ArrayColDelete($aResults, 6) before _ArrayDisplay

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By jitendriya
      Hi every one .
      I want to read a pdf file and write into a excel using autoit , so how can i do this with out using third party server please tell me .
      Thank you..
    • By JackER4565
      Hi, first of all thanks to all the guys who always help people in the forums, I wouldn't be able to do anything if wasn't for your help, even if I don't ask it myself.
       
      I've created this code to get some info on a monitoring network on my work. It relays on _IETableGetCollection and _IETableWriteToArray.
      It works well, but take around 3:25 minutes to get the info from 28 pages (some of them are large and take longer to load, but most of them are small and fast).
      My question is if you see a way to get the program to go faster...
       
      I've tried to make it easy for you to understand and edited somethings with sensitive info.
      (Some of the pages doesn't have the black divider with MIRA in the end, so I need to search if it is there or not.)
       
      #include <IE.au3> #include <array.au3> Local $oIE = _IECreate("about:blank", 0, 0) Local $paginas[28] = [89, 90, 91, 92, 93, 96, 105, 113, 119, 125, 126, 129, 131, 133, 135, 137, 139, 140, 141, 144, 145, 146, 148, 149, 150, 151, 158, 159] Local $Datos_array[0][2] Local $oTable Local $tabla Local $aux_x = 1 Local $ar = 1 Local $Numtables_datos = 0 MsgBox(0, "asd", "asd") For $pag = 0 To UBound($paginas) - 1 Step 1 _IENavigate($oIE, "<WEBSITE URL>" & $paginas[$pag]) ; <<< the pages to load are always the same except for the last digits. _ArrayAdd($Datos_array, $paginas[$pag] & "|" & "Entrante", 0, "|") ; <<<<<<<<<<<<<<<< adds the page number toarray [0, 0] ;############################################ START counts amount of tables with traffic $oTable = _IETableGetCollection($oIE) Local $iNumTables = @extended For $i = 3 To $iNumTables - 2 Step 1 $oTable = _IETableGetCollection($oIE, $i) $nomb_tabla2 = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY. $string2 = StringStripWS($nomb_tabla2[1][0], 8) If $string2 <> "MIRA" Then $Numtables_datos = $Numtables_datos + 1 Next $tabla_End = $iNumTables - $Numtables_datos ;############################################ FIN $tabla_Start = 4 $tabla_trafico = 2 For $for = 1 To $Numtables_datos Step 1 $oTable = _IETableGetCollection($oIE, $tabla_Start - 1) ; <<<<<<<<<<< NAME OF THE TABLE; row2 = mira $nomb_tabla = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY ;########################################### ADDS the traffic number into the row $string = StringStripWS($nomb_tabla[1][0], 8) If $string == "MIRA" Then ;si o si pasa por aca 1 vez _ArrayAdd($Datos_array, $nomb_tabla[0][0]) $nomb_aux = $nomb_tabla[0][0] $aux_x = 1 $tabla_trafico = $tabla_trafico + 2 Else ;esto deberia ser por row _ArrayAdd($Datos_array, $nomb_aux & " " & $aux_x) $aux_x = $aux_x + 1 $tabla_trafico = $tabla_trafico + 1 EndIf $oTable = _IETableGetCollection($oIE, $tabla_trafico) Local $aTableData = _IETableWriteToArray($oTable) $bps = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 0, 0) $bps = StringRight($bps, 5) $bps = StringLeft($bps, 4) $trafico_actual = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 2, 2) If $bps == "Gbps" Then $trafico_actual = $trafico_actual * 1000 If $bps == "Kbps" Then $trafico_actual = $trafico_actual / 1000 $Datos_array[$ar][1] = $trafico_actual $ar = $ar + 1 If $string == "MIRA" Then $tabla_Start = $tabla_Start + 2 Else $tabla_Start = $tabla_Start + 1 EndIf Next $ar = $ar + 1 ;~ ############# CAÍDA ############ ;~ If $actual_entrante = 0 Then ;~ $xxx = 0 ;~ Do ;~ MsgBox(0, "Tráfico Caído", $paginas[$i], 5) ;~ $xxx = $xxx + 1 ;~ Until $xxx = 10 ;~ EndIf ;~ ############# CAÍDA ############. Local $Numtables_datos = 0 Next _ArrayDisplay($Datos_array, "Array display") _IEQuit($oIE) Thanks!! 


      monitoria.html
    • By lavascript
      I have a Word document containing a 9-column table where row 1 is the column headers. My goal is to read the table into a 2d array, remove some rows, update some fields, and add a few rows to the end. The resulting array will likely be a different length. Next, I want to write the data back into the table. If it's easier, I can write the data to a new document from a template containing the same table header with a blank 2nd row.
      Here's my early attempt:
      Local $oWord = _Word_Create() Local $oDoc = _Word_DocOpen($oWord, $sFile) Local $aData = _Word_DocTableRead($oDoc, 1) $aData[3][5] = "Something else" Local $oRange = _Word_DocRangeSet($oDoc, 0) $oRange = _Word_DocRangeSet($oDoc, $oRange, $wdCell, 9) _Word_DocTableWrite($oRange,$aData) This, unfortunately, writes the entire array into the first cell of row 2. What am I doing wrong?
       
    • By nacerbaaziz
      Hello Members of this best Forum
      i have a question please
      for example if i have a long string
      and i want to extract a text between two tag
      what i can do to make that?
      note :
      i know that there is a
      StringRegExp function
      it's do that work
      but it result is be as an array
      i want the result to be a string
      is there any function on autoit can do that?
      Thanks in advance.
    • By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1 - 4.5 (on this version Microsoft destroy old rules) System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
×
×
  • Create New...