plasmon

Wait until part of web page loads

7 posts in this topic

Hi All,

I am trying to run a script that goes through a number of webpages by clicking and entering values. The problem is that when I get to one of the pages, the page loads, but a section on the page does not load; it just shows a circular loading sign in this section and after 5 minutes or so the data in finally populates. I found a related post called 'While Loop confusion', but it seems the problem was not fully solved. 

I have tried while loops myself to try to make the code sleep in increments until the text appears, but could not get it to work. Could someone please advise me on how to wait for a web page to fully load before continuing with the script? The time to load can vary, which is why sleep commands are not guaranteed to always work.

Thanks in advance.

 

 

 

 

 

Share this post


Link to post
Share on other sites



plasmon,

Welcome to the AutoIt forums.

The only reliable way I have found to deal with these slow loads is to determine something visual that changes only when the section has finally loaded and then loop until you can detect that change. So what could that clue be in your case? A change in colour for part of the page? A suddenly visible unique text string? Over to you.

M23

 

 


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

A trick can be to use javascript favlet in addressbar to add something to the html dom content. It will only be added when the page is finished loading.

Interesting.
Can you show any example?

 

mLipok


Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

On google you can find bookmarklets and favlets

As you can reach javascript thru the addressbar in any browser (opera, safari, chrome, firefox, internet explorer, microsoft edge) there are no limits to remotely controlling browsers as javascript can reach the dom. Below example is manually typing but its easy to do controlsend to most browsers addressbar (otherwise use IUIAutomation to reach addressbar)

1.  Type in your addressbar:     javascript:alert('hello');void(0);

javascript: you have to type other part can be copy pasted

void(0); is there to prevent reloading of the page

2.a  Next step is to inject some javascript as a oneliner

\ is escaping character for \/ and \"

javascript:var js = document.createElement('script');js.type = "text\/javascript";js.text = 'function testme(){alert(\"hello\");}';var first=document.getElementsByTagName('script')[0];first.parentNode.insertBefore(js, first);void(0);

2.b. Type in your addressbar:     javascript:testme();void(0);

3.   Type in your addressbar:

javascript:var input = document.createElement('input'); input.type = "text";input.value=document.readyState;document.body.appendChild(input);void(0);

You should see a textbox at the bottom of your page including the state of your document (see examples section on IUIAutomation on how to reach the contents of that textbox)

4.  probably easiest to get status (as you can read statusbar or statusbar as regular windows controls)

javascript:window.status=document.readyState;void(0);

or javascript:window.history.replaceState('Object', 'Title', '/'+document.readyState);void(0);

5. And from this part on its any imagination you have with html5 like injecting websocket and talk tcp/ip from AutoIT to your injected websocket javascript

javascript:var wsUri = "ws://127.0.0.1:65432/";function testWebSocket() {websocket = new WebSocket(wsUri);websocket.onopen = function(evt) {onOpen(evt)};websocket.onclose = function(evt) {onClose(evt)};  websocket.onmessage = function(evt) { onMessage(evt) };websocket.onerror = function(evt) { onError(evt) };} function onOpen(evt) { alert("CONNECTED");} function onClose(evt) { alert("DISCONNECTED");} function onMessage(evt) { alert("Message" + evt.data);} function onError(evt) { alert("ERROR:" + evt.data); } ;void(0);

and then run this script

#include <MsgBoxConstants.au3>

; I am the server, start me first! (Start in second the TCPConnect example script).

Example()

Func Example()
    TCPStartup() ; Start the TCP service.

    ; Register OnAutoItExit to be called when the script is closed.
    OnAutoItExitRegister("OnAutoItExit")

    ; Assign Local variables the loopback IP Address and the Port.
    Local $sIPAddress = "127.0.0.1" ; This IP Address only works for testing on your own computer.
    Local $iPort = 65432 ; Port used for the connection.

    ; Bind to the IP Address and Port specified with a maximum of 100 pending connexions
    ;(Take a look at the example of this function for further details).
    Local $iListenSocket = TCPListen($sIPAddress, $iPort, 100)
    Local $iError = 0

    ; If an error occurred display the error code and return False.
    If @error Then
        ; Someone is probably already listening on this IP Address and Port (script already running?).
        $iError = @error
        MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Could not listen, Error code: " & $iError)
        Return False
    EndIf

    ; Assign Local variable to be used by Listening and Client sockets.
    Local $iSocket = 0
    Do ; Wait for someone to connect (Unlimited).
        ; Accept incomming connexions if present (Socket to close when finished; one socket per client).
        $iSocket = TCPAccept($iListenSocket)

        ; If an error occurred display the error code and return False.
        If @error Then
            $iError = @error
            MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Could not accept the incoming connection, Error code: " & $iError)
            Return False
        EndIf
    Until $iSocket <> -1 ;if different from -1 a client is connected.

    ; Close the Listening socket to allow afterward binds.
    TCPCloseSocket($iListenSocket)

    MsgBox($MB_SYSTEMMODAL, "", "Client Connected.")

    ; Close the socket.
    TCPCloseSocket($iSocket)
EndFunc   ;==>Example

Func OnAutoItExit()
    TCPShutdown() ; Close the TCP service.
EndFunc   ;==>OnAutoItExit

and then type in your addressbar:  

javascript:testWebSocket();void(0);

6. As you have a socket you can exchange strings and dynamically eval these in javascript and as such completely remote control any browser just like ff.au3 is doing internally with a lot of javascript calls but then its an addin.

 

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

I see that I have something to read, search, test ... for the next few days.
Thanks.

mLipok

 

EDIT:

I see this is much related feature to this:

http://slayeroffice.com/tools/modi/v2.0/modi_help.html

 

Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

Hi All,

 

Sorry, I did not receive notifications that I had replies. Many thanks for your responses, I have since developed my own technique wherein I search for text on the loading screen, I put this search in a loop and and when the text disappears, the loop exits and the rest of the code is executed. 

 

Thanks again.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now