Cannot determine when page is fully loaded

[jasontj]

Hello all.  I have many scripts that I run against website to check for criminal records.  The one that is giving me trouble is the national sex offender database.  I cannot determine when the page is fully loaded.  It looks like they are using something like AJAX to load data from the backend and populate the page.   The looping won't work on the inserted code because I normally use an Excel spreadsheet.  Any help would be appreciated at determining when the page is fully loaded.  I put 5000ms on the "LoadWait", and also checked the HTML for specific text values.  Nothing works properly.

 

Thanks for taking a look.

Jason

 

expandcollapse popup

#Region - include files
; non standard UDF's
#include "wd_helper.au3"
#include "wd_capabilities.au3"
#include <File.au3>
#include <Array.au3>
#EndRegion - include files

; Register a customer error handler
; _IEErrorHandlerRegister("MyErrFunc")

; If you want to download/update driver the next line should be uncommented'  Does it get latest verion, or just version to match your browser version?
_WD_UpdateDriver('chrome')

#Region - Global's declarations
Global Const $sElementSelector = "//input[@name='q']"
Global Const $aBrowsers[][2] = _
        [ _
        ["Chrome", SetupChrome] _
        ]

Global Const $aDebugLevel[][2] = _
        [ _
        ["None", $_WD_DEBUG_None], _
        ["Error", $_WD_DEBUG_Error], _
        ["Full", $_WD_DEBUG_Info] _
        ]

Global $sSession
Global $__g_idButton_Abort
#EndRegion - Global's declarations
$URLstr = "https://www.nsopw.gov/search-public-sex-offender-registries"
Global $sCheck


; RUN SEARCHES BELOW
Run_Pre_Screen($_WD_DEBUG_None, SetupChrome)

Exit

Func Run_Pre_Screen($idDebugging, $idBrowsers)
    ; Set debug level
    $_WD_DEBUG = $_WD_DEBUG_Info

    ; Execute browser setup routine for user's browser selection
    Local $sDesiredCapabilities = Call($idBrowsers)

    _WD_Startup()
    If @error <> $_WD_ERROR_Success Then Return

    $sSession = _WD_CreateSession($sDesiredCapabilities)
   _WD_Navigate($sSession, $URLstr)
   _WD_LoadWait($sSession, 1000)


    ; if Click Confirm button to dismiss dialog
    msgbox (0, "CLICK BUTTON", "Click the 'Continue' button if it appeared, and then click 'OK'")

; normally I would loop though values in my Excel file
   $INDEX = 1
   $sCellValueL = "SMITH"
   $sCellValueF = "JOHN"
   $RecordNumber = "XYZ"

   do
      ; Enter First Name Value
      $sElement = _WD_GetElementById($sSession, "firstname")
      If @error = $_WD_ERROR_Success Then
        _WD_ElementAction($sSession, $sElement, 'value', $sCellValueF)
      EndIf

      ; Enter Last Name Value
        $sElement = _WD_GetElementById($sSession, "lastname")
        If @error = $_WD_ERROR_Success Then
            _WD_ElementAction($sSession, $sElement, 'value', $sCellValueL)
        EndIf

      ; Click "Search" button
      $sButton = _WD_GetElementById($sSession, "searchbynamezip")


      _WD_ElementAction($sSession, $sButton, 'click')
      _WD_LoadWait($sSession, 5000)

      $sHTML = _WD_GetSource($sSession)
      $testStr3 = StringRegExp($sHTML, "Loading data. Please wait")

      if $testStr3 = 1 Then
        msgBox(0,"waiting", "Search taking too long.  Press 'OK' to continue")
        Sleep(10000)
      EndIf

    $sHTML = _WD_GetSource($sSession)
    $testStr3 = StringRegExp($sHTML, "Loading data. Please wait")

    if $testStr3 = 1 Then
        msgBox(0,"waiting", "Search taking too long.  Press 'OK' to continue")
        Sleep(10000)
    EndIf

    $sHTML = _WD_GetSource($sSession)
    $testStr1 = StringRegExp($sHTML, "Showing 0 to 0 of 0 entries")
    $testStr2 = StringRegExp($sHTML, "No data available in table")
    $testStr3 = StringRegExp($sHTML, "time out")

     if $testStr1 = 1 and $testStr2 = 1 Then
        ; Mark item 'clear' in my database


    Else
        _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, "POSSIBLE RECORD", "G" & $INDEX)
        $LastPossible = $RecordNumber
    EndIf


; Expand to search again  (COMMENTED OUT FOR NOW BECAUSE I AM TRYING A NEW ROUTE OF LOADING A WHOLE NEW PAGE TO SEE IF THIS HELPS
;        $sButton = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "/html/body/div/div[2]/main/div/div/div/div/article/div/div[2]/div/div/div/div/div[1]/h2/button")
;        $sButton = _WD_GetElementById($sSession, "nsopwsearchfilter")
;        If @error = $_WD_ERROR_Success Then
;           _WD_ElementAction($sSession, $sButton, 'click')
;           _WD_LoadWait($sSession, 2000)
;        EndIf

; Search from new page (LOAD WHOLE NEW PAGE)
      _WD_Navigate($sSession, $URLstr)
      _WD_LoadWait($sSession, 1000)

      $INDEX = $INDEX + 1
      $sCellValueL = "JOHNSON"
      $sCellValueF = "MARK"
      $RecordNumber = "XYZ-123"
      
      if INDEX = 2 then ; blank out cell value which will cause this to exit for loop (this is for testing only)
        $sCellValueL = ""
      EndIf

   until $sCellValueL = ""


if $LastPossible <> "" Then
    ; deal with last record
EndIf


_WD_Shutdown()
msgBox(0,"Finished", "All searches have been processed.")

EndFunc

 

[Danp2]

The first thing I would try is to use _WD_WaitElement to make sure the target element exists and that it is visible and enabled.

[jasontj]

@Danp2, thanks for replying.  I tried to use your suggestions, but I cannot determine what element to check for.  I opened the page (without performing a search), viewed the source of the page, saved it to a text file.  Then I performed a search with a random name that came back with 0 records, and ran a "FC" to compare the 2 text files.  I got "no differences".  Then I ran a search that came back with many records on a name of "ROBERT SMITH" and did the same thing.  The files still had no differences.  So, how do I determine where these results are being pulled into on this page?  Is it a frame of some sort?  I am at a loss.  Thanks.

C:\temp>fc sor1.txt sor2.txt
Comparing files SOR1.txt and SOR2.TXT
FC: no differences encountered
 

Jason

[Danp2]

It sounds like you are using _WD_GetSource to retrieve the page's source code. This won't reflect any dynamically loaded elements. If you examine the source  code within the browser's Dev Tools (press F12 to open), then you should see the complete source.

I can see a Div element that appears to contain the spinning visual that is present when a search is active. Try using _WD_WaitElement to check for an xpath of "//div[@id='loaderGif']". Be sure to use the $_WD_OPTION_Hidden option so that it will only return once it is found and it isn't visible.

[mLipok]

If you have frames try to use _WD_FrameList it will check each document (each separate frame) and will wait for they load.

[jasontj]

Thanks @Danp2 and @mLipok,  I will give these suggestions a try today and post back the results.  I really appreciate you guys taking time to help me out.  

[jasontj]

Thanks @Danp2 and @mLipok,  I will give these suggestions a try today and post back the results.  I really appreciate you guys taking time to help me out.  

Here is an update.  This worked for me.  I did not have the "$_WD_OPTION_Hidden option".  I may need to update to get this function.  

_WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, "//div[@id='loaderGif']", 100, 2000, 1)
      $sElement1 = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//div[@id='loaderGif']")
      $lIsVisible1 = _WD_ElementAction($sSession, $sElement1, 'displayed')

      $lIsVisible = _WD_ElementAction($sSession, $sElement1, 'displayed')
      do
         $lIsVisible = _WD_ElementAction($sSession, $sElement1, 'displayed')
      until $lIsVisible = False

I stored this variable ($lIsVisible1) immediately after I submit because sometimes the page has an error, and if $lIsVisible1 is false, then the spinner was never visible.  If this is the case, then I run the search again.  Overall, it seems to work.  I appreciate your help @Danp2.

 

[Danp2]

3 hours ago, jasontj said:

I did not have the "$_WD_OPTION_Hidden option".  I may need to update to get this function.

That option has been around for over a year, so it sounds like you are way out of date. 🤨

[jasontj]

I'm pretty bad about keeping things up to date.  If it's not broken, no need to download a new version..hahah..  I will download it tomorrow.  Thanks!