Jump to content
JackER4565

Getting info from a website with arrays and table read (Optimize my code)

Recommended Posts

Posted (edited)

Hi, first of all thanks to all the guys who always help people in the forums, I wouldn't be able to do anything if wasn't for your help, even if I don't ask it myself.

 

I've created this code to get some info on a monitoring network on my work. It relays on _IETableGetCollection and _IETableWriteToArray.

It works well, but take around 3:25 minutes to get the info from 28 pages (some of them are large and take longer to load, but most of them are small and fast).

My question is if you see a way to get the program to go faster...

 

I've tried to make it easy for you to understand and edited somethings with sensitive info.

(Some of the pages doesn't have the black divider with MIRA in the end, so I need to search if it is there or not.)

 

#include <IE.au3>
#include <array.au3>

Local $oIE = _IECreate("about:blank", 0, 0)
Local $paginas[28] = [89, 90, 91, 92, 93, 96, 105, 113, 119, 125, 126, 129, 131, 133, 135, 137, 139, 140, 141, 144, 145, 146, 148, 149, 150, 151, 158, 159]
Local $Datos_array[0][2]
Local $oTable
Local $tabla
Local $aux_x = 1
Local $ar = 1
Local $Numtables_datos = 0
MsgBox(0, "asd", "asd")
For $pag = 0 To UBound($paginas) - 1 Step 1
    _IENavigate($oIE, "<WEBSITE URL>" & $paginas[$pag]) ; <<< the pages to load are always the same except for the last digits.


    _ArrayAdd($Datos_array, $paginas[$pag] & "|" & "Entrante", 0, "|") ; <<<<<<<<<<<<<<<< adds the page number toarray  [0, 0] 

    ;############################################ START counts amount of tables with traffic
    $oTable = _IETableGetCollection($oIE)
    Local $iNumTables = @extended
    For $i = 3 To $iNumTables - 2 Step 1
        $oTable = _IETableGetCollection($oIE, $i)
        $nomb_tabla2 = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY.
        $string2 = StringStripWS($nomb_tabla2[1][0], 8)
        If $string2 <> "MIRA" Then $Numtables_datos = $Numtables_datos + 1
    Next
    $tabla_End = $iNumTables - $Numtables_datos
    ;############################################ FIN

    $tabla_Start = 4
    $tabla_trafico = 2
    For $for = 1 To $Numtables_datos Step 1

        $oTable = _IETableGetCollection($oIE, $tabla_Start - 1) ; <<<<<<<<<<< NAME OF THE TABLE; row2 = mira

        $nomb_tabla = _IETableWriteToArray($oTable) ; <<<<<<<< TABLE TO ARRAY

        ;########################################### ADDS the traffic number into the row
        $string = StringStripWS($nomb_tabla[1][0], 8)
        If $string == "MIRA" Then
            ;si o si pasa por aca 1 vez
            _ArrayAdd($Datos_array, $nomb_tabla[0][0])
            $nomb_aux = $nomb_tabla[0][0]
            $aux_x = 1
            $tabla_trafico = $tabla_trafico + 2
        Else
            ;esto deberia ser por row
            _ArrayAdd($Datos_array, $nomb_aux & " " & $aux_x)
            $aux_x = $aux_x + 1
            $tabla_trafico = $tabla_trafico + 1
        EndIf

        $oTable = _IETableGetCollection($oIE, $tabla_trafico)
        Local $aTableData = _IETableWriteToArray($oTable)

        $bps = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 0, 0)
        $bps = StringRight($bps, 5)
        $bps = StringLeft($bps, 4)
        $trafico_actual = _ArrayToString($aTableData, "|", 0, 0, @CRLF, 2, 2)
        If $bps == "Gbps" Then $trafico_actual = $trafico_actual * 1000
        If $bps == "Kbps" Then $trafico_actual = $trafico_actual / 1000
        $Datos_array[$ar][1] = $trafico_actual
        $ar = $ar + 1

        If $string == "MIRA" Then

            $tabla_Start = $tabla_Start + 2
        Else
            $tabla_Start = $tabla_Start + 1
        EndIf
    Next
    $ar = $ar + 1




;~ ############# CAÍDA ############
;~          If $actual_entrante = 0 Then
;~              $xxx = 0
;~              Do

;~              MsgBox(0, "Tráfico Caído", $paginas[$i], 5)
;~                  $xxx = $xxx + 1
;~              Until $xxx = 10
;~          EndIf
;~ ############# CAÍDA ############.
    Local $Numtables_datos = 0
Next
_ArrayDisplay($Datos_array, "Array display")
_IEQuit($oIE)

Thanks!! 

array output.png

WEB SCREENSHOT2.png

monitoria.html

Edited by JackER4565

Share this post


Link to post
Share on other sites
16 hours ago, Danp2 said:

You could likely shave off some time by loading some of the slower pages in a background tab and then using _IEAttach to select them as needed.

good idea, will try it and share the results!

 

Thanks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By WoodGrain
      Hi All,
      I've tried to create a script to browse to a web page in IE, but it only works for some sites, I've tried to simplify the script as much as possible, hoping you can point me in the right direction - thanks.
      This one going to Google does work:
      #include <IE.au3> $oIE = _IECreate("https://www.google.com.au/", 0, 1, 1) _IELoadWait($oIE) _IEQuit($oIE) But this one going to OneDrive opens the IE page, but fails immediately returning the errors, and the IE window is not closed:
      --> IE.au3 T3.0-2 Error from function _IELoadWait, $_IESTATUS_ClientDisconnected (-2147023174, Browser has been deleted prior to operation.)
      --> IE.au3 T3.0-2 Error from function _IELoadWait, $_IESTATUS_InvalidObjectType ()
      --> IE.au3 T3.0-2 Error from function _IEQuit, $_IESTATUS_InvalidObjectType
      #include <IE.au3> $oIE = _IECreate("https://abccorp-my.sharepoint.com/", 0, 1, 1) _IELoadWait($oIE) _IEQuit($oIE) Even if I put a sleep timer in for 10 seconds after the _IECreate it still doesn't work
      Thanks all!
    • By nacerbaaziz
      good morning everybody.
      today i liked to share an small example with you
      which it an function to read the registry values as an array
      the result array is 2d array witch
      $a_array[n][0] = value name
      $a_array[n][1] = value Data
      $a_array[0][0] = values count
      here's the function

      #include <Array.au3> #include <WinAPIReg.au3> #include <APIRegConstants.au3> Local $a_array = _RegReadToArray("HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run") If @error Then     MsgBox(16, "error", @error)     Exit EndIf _ArrayDisplay($a_array) Func _RegReadToArray($s_RegKey)     Local $a_KeySplitInfo = StringSplit($s_RegKey, "\\", 2)     If UBound($a_KeySplitInfo) <= 1 Then         $a_KeySplitInfo = StringSplit($s_RegKey, "\", 2)         If UBound($a_KeySplitInfo) <= 1 Then Return (1, 1, 0)     EndIf     Local $H_KeyInfo = "", $s_RegKeyInfo = ""     Switch $a_KeySplitInfo[0]         Case "hklm", "HKEY_LOCAL_MACHINE", "hklm64", "HKEY_LOCAL_MACHINE64"             $H_KeyInfo = $HKEY_LOCAL_MACHINE         Case "hkCu", "HKEY_CURRENT_USER", "hkCU64", "HKEY_CURRENT_USER64"             $H_KeyInfo = $HKEY_CURRENT_USER         Case "hkCr", "HKEY_CLASSES_ROOT", "HKCR64", "HKEY_CLASSES_ROOT64"             $H_KeyInfo = $HKEY_CLASSES_ROOT         Case "HKU", "HKEY_USERS", "HKU64", "HKEY_USERS64"             $H_KeyInfo = $HKEY_USERS         Case Else             Return SetError(2, 2, 0)     EndSwitch     _ArrayDelete($a_KeySplitInfo, 0)     $s_RegKeyInfo = _ArrayToString($a_KeySplitInfo, "\")     Local $H_KeyInfoOpen = _WinAPI_RegOpenKey($H_KeyInfo, $s_RegKeyInfo, $KEY_READ)     Local $A_KeyInfo = _WinAPI_RegQueryInfoKey($H_KeyInfoOpen)     If @error Then Return SetError(1, 1, 0)     _WinAPI_RegCloseKey($H_KeyInfoOpen)     Local $A_RegVal[$A_KeyInfo[2] + 1][2]     Local $iV = 1, $s_RegRead = ""     While 1         $s_RegVal = RegEnumVal($s_RegKey, $iV)         If @error <> 0 Then ExitLoop         $s_RegRead = RegRead($s_RegKey, $s_RegVal)         If Not (@error) Then             $A_RegVal[$iV][0] = $s_RegVal             $A_RegVal[$iV][1] = $s_RegRead         EndIf         $iV += 1     WEnd     $A_RegVal[0][0] = UBound($A_RegVal) - 1     If $A_RegVal[0][0] >= 1 Then         Return $A_RegVal     Else         Return SetError(3, 3, 0)     EndIf EndFunc   ;==>_RegReadToArray
      i hope you benefit from it
      with my greetings
    • By uncommon
      So I have been reading on how to use OAuth2 Service accounts(https://developers.google.com/identity/protocols/OAuth2ServiceAccount) and ran into an issue with autoit I can not solve. To keep this short to communicate to the Google API I need to use a JSON Web signature containing
      {Base64url encoded header}.{Base64url encoded claim set}.

      The Header and claim set work fine but I do not know how to convert them to a byte array using autoit. I have looked around on the forums and website but have not found anything that seems to work. Here is my code.
      #include-once #include <Array.au3> #include <Constants.au3> #include <Debug.au3> #include <File.au3> #include "Json.au3" #include <Process.au3> #include <ProgressConstants.au3> #include <String.au3> #include <WindowsConstants.au3> #include "WinHttp.au3" #include <UnixTime.au3> #include <StringConstants.au3> Global $UnixTime = _TimeGetStamp() $sJWTheader = '{"alg":"RS256","typ":"JWT"}' $sJWTclaimset = '{"iss":"Removed@forprivacy.com","scope":"https://www.googleapis.com/auth/spreadsheets","aud":"https://www.googleapis.com/oauth2/v4/token","exp":' & $UnixTime + 3600 & ',"iat":' & $UnixTime & '}' Local $taiData = DllStructCreate("BYTE[256]") Local $sText = _base64($sJWTheader)&'.'&_base64($sJWTclaimset) DllStructSetData($taiData, 1, StringToBinary($sText, 4)) Local $bvResult = DllStructGetData($taiData, 1) $sJWTSigature = $bvResult $sJWT = _base64($sJWTheader)&'.'&_base64($sJWTclaimset)&'.'&_base64($sJWTSigature) $POSTHeader = "Content-Type: application/x-www-form-urlencoded" $hOpen = _WinHttpOpen() $hConnect = _WinHttpConnect($hOpen, "https://www.googleapis.com/") $sRead = _WinHttpSimpleSSLRequest($hConnect, "POST", "oauth2/v4/token", Default, "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=" & $sJWT) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) ConsoleWrite($sRead & @LF) The whole bytes thing is beyond with what I know about auotit. I can't tell what I need to fix.
       
       
    • By Dent
      https://rmplusportal.rightmove.co.uk/
      On that page is a login form requiring username, password and then the clicking of the button (or pressing enter but I'm not a fan of using Send)
      Here's the section of code from the page that specifically deals with the form...
      <form> <div class="error-message-login" data-test="error-message"></div> <label> <span>User email</span> <div class="ui input"> <input id="email" type="text" placeholder="Your email"> </div> </label> <label> <span>Password</span> <div class="ui input"> <input id="password" type="password" placeholder="Password"> </div> </label> <button class="ui primary button" role="button" data-test="login">Login</button> <p> <a class="forgot-your-password" href="https://www.rightmove.co.uk/forgottenPasswordForm.html">Forgot your password?</a> </p> </form> Specifically the <button class...></button> is causing me problems as I can't find a way to click it. The 'User email' and 'Password' objects have an 'id' so I can reference these in the AutoIT code and interact with them and the button also previously had the id of 'login' but I guess in a change to this page that was altered.
      Oddly, if I get the code to show the instance of IE that is created and manually click Login or press Enter the page gives me an error saying 'Please enter your email address' even though it is there. I have to manually go into each field and make a change i.e. adding a character to the end of each field then deleting it before it will let me in by either clicking Login or pressing enter. So maybe there's so code elsewhere on the page that knows the fields haven't been physically typed in?
      Any help on this would be much appreciated.
    • By Dent
      Here is the URL to get to the page with the code as shown in the screenshot https://www.rightmove.co.uk/house-prices/detail.html?country=england&locationIdentifier=POSTCODE^1603593&searchLocation=SW1V+1AA&year=1&referrer=listChangeCriteria
      But how do I select the highlighted text, the address?
      I've tried variations of the below code but to no avail so far
      Local $oClassReturn = $oIE.document.getElementsByClassName("soldList") If Not IsObj($oClassReturn) Then Exit ConsoleWrite("Error in $oClassReturn" & @CRLF) Local $oTagReturn = $oClassReturn.Item(0).getElementsByTagName("div") If Not IsObj($oTagReturn) Then Exit ConsoleWrite("Error in $oTagReturn" & @CRLF) MsgBox(0, "TEST", $oTagReturn.Item(0))  

×
×
  • Create New...