Jump to content
KimberlyJillPereira

Extract table from website until the end

Recommended Posts

KimberlyJillPereira

I could only extract the first 20 from table into Microsoft Excel by using Array Extract but I want to extract until the end what I mean is until the second page. How to do it? Please revert. Thanks.:):)

image1.jpg

image2.jpg

image3.png

Share this post


Link to post
Share on other sites
KimberlyJillPereira

since there is total of 33 I want to extract from row 0 to row 32. How to do it? please help. Thanks.

Share this post


Link to post
Share on other sites
KimberlyJillPereira

@water   @MaoMao  please help

Edited by KimberlyJillPereira

Share this post


Link to post
Share on other sites
Subz
#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]

$oIE = _IECreate("http://www.timeview2.net/alarms", 1)
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    $oIE = _IECreate("http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

Edited by Subz
Added Table Header + Removed unwanted spaces from table results
  • Like 1

Share this post


Link to post
Share on other sites
KimberlyJillPereira

Thanks it works but then I want to type in username and password automatically then click submit button and then click on Alarms button to extract all the data but then it is already logged in and then extracts all the data. I copied my code to it and it only automatically types in username and password but then it doesnt click submit button. I used _IEFormSubmit($oDivs) and it doesnt work and it shows warning from function _IEAttach, $_IESTATUS_NoMatch even clicking on alarms button also doesnt work. I used this code _IELinkClickByText($oIE, "Alarms") and it also shows warning Warning from function _IELinkClickByText, $_IESTATUS_NoMatch by the way I used these codes in a seperate program and it worked but when I use my codes together with yours it doesnt work. Please help me. Thanks.

Share this post


Link to post
Share on other sites
Subz

Can you post your code without username and password and I'll take a look although probably later this evening.

  • Like 1

Share this post


Link to post
Share on other sites
Subz

Try this, just replace values for $sUserName and $sPassWord.

#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]
Local $oUserName, $oPassWord
Local $sUserName = "user"
Local $sPassWord = "demo"

$oIE = _IECreate("http://www.timeview2.net/", 1)
$oForms = _IETagNameGetCollection($oIE, "form")
For $oForm In $oForms
    If $oForm.id = "UserLoginForm" Then
        $oUserName = _IEFormElementGetObjByName($oForm, "data[User][username]")
        _IEFormElementSetValue($oUserName, $sUserName)
        $oPassWord = _IEFormElementGetObjByName($oForm, "data[User][password]")
        _IEFormElementSetValue($oPassWord, $sPassWord)
        _IEFormSubmit($oForm)
    EndIf
Next

$oIE = _IECreate("http://www.timeview2.net/alarms", 1)
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    $oIE = _IECreate("http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

  • Like 1

Share this post


Link to post
Share on other sites
KimberlyJillPereira

Thanks it works :):)but the problem is that there is too many popup window when its on Alarms page then it shows full table extraction. How to avoid too many popup windows? thanks.

Share this post


Link to post
Share on other sites
Subz
#include <Array.au3>
#include <IE.au3>
#include <String.au3>

Local $oIE
Local $oDivs
Local $aPageCount, $iPageCount
Local $aTable, $oTables
Local $aResults[0][7]
Local $oUserName, $oPassWord
Local $sUserName = "user"
Local $sPassWord = "demo"

$oIE = _IECreate("http://www.timeview2.net/", 1)
$oForms = _IETagNameGetCollection($oIE, "form")
For $oForm In $oForms
    If $oForm.id = "UserLoginForm" Then
        $oUserName = _IEFormElementGetObjByName($oForm, "data[User][username]")
        _IEFormElementSetValue($oUserName, $sUserName)
        $oPassWord = _IEFormElementGetObjByName($oForm, "data[User][password]")
        _IEFormElementSetValue($oPassWord, $sPassWord)
        _IEFormSubmit($oForm)
    EndIf
Next

_IENavigate($oIE, "http://www.timeview2.net/alarms")
$oDivs = _IETagNameGetCollection($oIE, "div")
For $oDiv In $oDivs
    If $oDiv.ClassName = "alarms index" Then
        $aPageCount = _StringBetween($oDiv.InnerText, "Page 1 of ", ", showing ")
        If @error Then Exit
        $iPageCount = Number($aPageCount[0])
    EndIf
Next

For $i = 1 To $iPageCount
    _IENavigate($oIE, "http://www.timeview2.net/alarms/index/group:All%20Groups/page:" & $i, 1)
    $oTables = _IETagNameGetCollection($oIE, "table")
    For $oTable In $oTables
        If $oTable.ClassName = "pageindex" Then
            $nTable = $oTable.NextElementSibling
            $aTable = _IETableWriteToArray($nTable, True)
            If UBound($aTable) - 1 > 0 And UBound($aTable, 2) = 7 Then
                If $i = 1 Then
                    _ArrayConcatenate($aResults, $aTable, 0)
                Else
                    _ArrayConcatenate($aResults, $aTable, 1)
                EndIf
                ExitLoop
            EndIf
        EndIf
    Next
Next
_ArrayDisplay($aResults)
;~ Save to CSV
Local $hFileOpen = FileOpen(@ScriptDir & "\Results.csv", 2)
For $i = 0 To UBound($aResults) - 1
    FileWrite($hFileOpen, '"' & _
    StringStripWS($aResults[$i][0], 7) & '","' & _
    StringStripWS($aResults[$i][1], 7) & '","' & _
    StringStripWS($aResults[$i][2], 7) & '","' & _
    StringStripWS($aResults[$i][3], 7) & '","' & _
    StringStripWS($aResults[$i][4], 7) & '","' & _
    StringStripWS($aResults[$i][5], 7) & '"' & @CRLF)
Next
FileClose($hFileOpen)

 

  • Like 1

Share this post


Link to post
Share on other sites
KimberlyJillPereira

No more popups. It worked. Thank you so much!:):):)

Share this post


Link to post
Share on other sites
KimberlyJillPereira

@Subz , Sorry to disturb you again I just only want to extract from column 0 to 5 that means excluding the column 'Actions'. Please help me. Thanks.:)

Share this post


Link to post
Share on other sites
Subz

If you look at the results.csv file it already excludes the Actions column, the _ArrayDisplay shows the full table but I only wrote column 0 to 5 in the csv, see code after :~ Save to CSV

Share this post


Link to post
Share on other sites
KimberlyJillPereira

It doesnt have column but it has rows. I dont want 'View'. how to remove rows too for the Column 'Action' Please help. Thanks.:)

jjjj.png

Share this post


Link to post
Share on other sites
Subz

You can actually remove _ArrayDisplay line it doesn't really mean anything, I just wanted to show that it had grabbed the table, the results that are saved to csv are only Col 0 to Col 5.  If you want to keep _ArrayDisplay then just use _ArrayColDelete($aResults, 6) before _ArrayDisplay

  • Like 1

Share this post


Link to post
Share on other sites
KimberlyJillPereira

Thank you so much!!! It worked.:):D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • lavascript
      By lavascript
      I have a Word document containing a 9-column table where row 1 is the column headers. My goal is to read the table into a 2d array, remove some rows, update some fields, and add a few rows to the end. The resulting array will likely be a different length. Next, I want to write the data back into the table. If it's easier, I can write the data to a new document from a template containing the same table header with a blank 2nd row.
      Here's my early attempt:
      Local $oWord = _Word_Create() Local $oDoc = _Word_DocOpen($oWord, $sFile) Local $aData = _Word_DocTableRead($oDoc, 1) $aData[3][5] = "Something else" Local $oRange = _Word_DocRangeSet($oDoc, 0) $oRange = _Word_DocRangeSet($oDoc, $oRange, $wdCell, 9) _Word_DocTableWrite($oRange,$aData) This, unfortunately, writes the entire array into the first cell of row 2. What am I doing wrong?
       
    • nacerbaaziz
      By nacerbaaziz
      Hello Members of this best Forum
      i have a question please
      for example if i have a long string
      and i want to extract a text between two tag
      what i can do to make that?
      note :
      i know that there is a
      StringRegExp function
      it's do that work
      but it result is be as an array
      i want the result to be a string
      is there any function on autoit can do that?
      Thanks in advance.
    • Ascer
      By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1 - 4.5 (on this version Microsoft destroy old rules) System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
    • islandspapand
      By islandspapand
      Hi all
      i am currently trying to click on an element in a HTML Table, but just can get it to work.
      i am able to click the top of the table so it changes to sort  but just can't click on the element in the table.
      an i need to click on element to continue in the site.
      i have attached the code so far and pictures of the table  element want to click plus the source of the table.
      i am able to get data in the table with $oTable = _IETableGetCollection($oIE, 2) but not able to click on them.
       
      Help is very much appreciated
       
      #cs ---------------------------------------------------------------------------- AutoIt Version: 3.3.14.2 Author: myName Script Function: Template AutoIt script. #ce ---------------------------------------------------------------------------- ; Script Start - Add your code below here #include <IE.au3> #include "DOM.au3" #include <Array.au3> #include <MsgBoxConstants.au3> Global $oIE = _IECreate("*") _IELoadWait($oIE) Sleep(2000) _PageLogin($oIE) _PageLoadWait() _PageNewReq($oIE) _PageLoadWait() _InputModelInf($oIE) _PageLoadWait() Sleep(1000) $aTableLink = BGe_IEGetDOMObjByXPathWithAttributes($oIE, "//table/tbody/tr/td[.='Name Of user']", 2000) ;~ $aTableLink = BGe_IEGetDOMObjByXPathWithAttributes($oIE, "//table/tbody/tr", 2000) ;~ _ArrayDisplay($aTableLink,"$aTableLink") If IsArray($aTableLink) Then ConsoleWrite("Able to BGe_IEGetDOMObjByXPathWithAttributes($oIE, //table/tbody/tr/td[.='Name Of user'])" & @CRLF) For $i = 0 To UBound($aTableLink)-1 ConsoleWrite(" OuterHTML : " & $aTableLink[$i].outerHTML & @CRLF) ConsoleWrite(" Parentnode : " & $aTableLink[$i].parentnode & @CRLF) ConsoleWrite(" Parentnode.click : " & $aTableLink[$i].parentnode.fireEvent("onclick","click") & @CRLF) $objClick = $aTableLink[$i].parentnode ;~ _IEAction($aTableLink[$i] , "focus") _IEAction($objClick , "focus") ;~ If _IEAction($aTableLink[$i], "click") Then If _IEAction($objClick, "click") Then ConsoleWrite("Able to _IEAction($aForumLink[0], 'click')" & @CRLF) _IELoadWait($oIE) Else ConsoleWrite("UNable to _IEAction($aForumLink[0], 'click')" & @CRLF) Exit 3 EndIf Next Else ConsoleWrite("Unable to BGe_IEGetDOMObjByXPathWithAttributes($oIE, //table/tbody/tr/td[.='Name Of user'])" & @CRLF) Exit 2 EndIf _PageLoadWait() Func _InputModelInf($oTmpIE) ; Add Var for Model & Serial in Func $oModelInput = _IEGetObjById($oTmpIE,"model") _IEAction($oModelInput,"focus") _IEDocInsertText($oModelInput, "*") $oSerialInput = _IEGetObjById($oTmpIE,"serial") _IEAction($oModelInput,"focus") _IEDocInsertText($oSerialInput, "*") $links = $oTmpIE.document.getElementsByClassName("btn btn-primary ng-scope") For $link In $links If $link.innertext = "Søg" Or $link.innertext = "Search" Then $link.click() ExitLoop EndIf Next Return True EndFunc Func _PageNewReq($oTmpIE) $links = $oTmpIE.document.getElementsByClassName("ng-scope k-link") For $link In $links If $link.innertext = "Send ny fejlmelding" Or $link.innertext = "Submit a New Service Request" Then $link.click() ExitLoop EndIf Next Return True EndFunc Func _PageLogin($oTmpIE) $oUserInput = _IEGetObjById($oTmpIE,"loginid") _IEDocInsertText($oUserInput, "*") $oPasswordInput = _IEGetObjById($oTmpIE,"password") _IEDocInsertText($oPasswordInput, "*") $links = $oTmpIE.document.getElementsByClassName("btn btn-primary login ng-scope") For $link In $links If $link.innertext = "Sign in" Then $link.click() ExitLoop EndIf Next Return True EndFunc Func _PageLoadWait() Local $PageLoadWait = False ;~ nav navbar-nav navbar-right ng-hide ;~ nav navbar-nav navbar-right $tags = $oIE.document.GetElementsByTagName("ul") For $tag in $tags $class_value = $tag.GetAttribute("class") If $class_value = "nav navbar-nav navbar-right" Then ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : Webpage loading :) ' & @CRLF) ;### Debug Console $PageLoadWait = True ExitLoop EndIf Next Do sleep(250) For $tag in $tags $class_value = $tag.GetAttribute("class") If $class_value = "nav navbar-nav navbar-right ng-hide" Then ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : Webpage load finished :)'& @CRLF) ;### Debug Console $PageLoadWait = False ExitLoop EndIf Next Until $PageLoadWait = False EndFunc  
      Thanks in advance
       
       


    • Gowrisankar
      By Gowrisankar
      Hello everyone,
      I'm trying to pass values to elements in a website.
      The elements are present within a table, which is again present within a table, which is inside a form.
      I tried to read the form, tables, etc., but with no results. It appears to me that the elements, tables, form, etc., were not read at all.
      The following is what I tried. Please guide me.
      ;I tried the following to read the tables into arrays #include <IE.au3> #include <MsgBoxConstants.au3> Local $oIE = _IECreate() _IENavigate($oIE, "---- URL HERE ----") _IELoadWait($oIE) $o_Table = _IETableGetCollection ($oIE) $i_NumTables = @extended For $i = 0 To $i_NumTables - 1 Step 1 $o_Table_Temp2 = _IETableGetCollection ($oIE, $i) $a_TableData = _IETableWriteToArray ($o_Table_Temp2) _ArrayDisplay($a_TableData) Next  
      ;I tried the following code to pass value to the field #include <IE.au3> #include <MsgBoxConstants.au3> Local $oIE = _IECreate() _IENavigate($oIE, "---- URL HERE ----", 0) _IELoadWait($oIE) Local $oForm = _IEFormGetObjByName($oIE, "default") Local $oField = _IEFormElementGetObjByName($oForm, "tGroup") _IEFormElementSetValue($oField, "---- VALUE HERE ----")  
      The following is the html view of the website and the highlighted field is the one that I want to pass values to. Since this is an official website, I can't share the exact url.

×