_IENavigate() Breaks With Variable?

[IAMK]

I have the following code which is supposed to navigate through pages on a website.

Local $currentPage = 1

While($currentPage < $totalPages)
    If($currentPage > 1) Then
        _IENavigate($ie, "http://lang-8.com/MYIDREMOVED/journals?page=" & $currentPage)
    EndIf
    ToolTip("1")

For page 1, the code after the If statement works properly (because before Local $currentPage = 2 is executed, I have a _IENavigate() call to a plain URL (no $currentPage variable used as page 1 is always the same).

However, when the code for page 1 finishes, I increment $currentPage to 2, and then the If statement never gets exited. It seems something inside _IENavigate() is not being handled properly when I use $currentPage in the URL.
Note: The browser correctly navigates to page 2 but then just hangs.

I have also tried to set $currentPage = 2, and the hang happens at the same spot, so the issue is not with doing the loop a second time.

Does anyone know what could possibly be the issue and how I could fix this?

[FrancescoDiMuro]

@IAMK
Could you please post the entire script, since what you posted is just a part of it?

[IAMK]

@FrancescoDiMuro

No, it contains my password, etc.
The most I can give is:

expandcollapse popup

#include <IE.au3>
#include <Array.au3>

Local $ie = _IECreate("https://lang-8.com/login?from=header") ;The Browser
Local $totalPages = 60

Func runScript()
   _IELoadWait($ie)
   Sleep(2000)

   login() ;Not showing this function

   _IENavigate($ie, "http://lang-8.com/NotshowingmyID/journals/")

   Local $linkArray[1] = ["0"]
   Local $linkCount = 1

   Local $currentPage = 1

   While($currentPage < $totalPages)
      If($currentPage > 1) Then
         _IENavigate($ie, "http://lang-8.com/NotshowingmyID/journals?page=" & $currentPage)
      EndIf
ConsoleWrite("1")
      While(UBound($linkArray) > 1) ;Reset the array.
         _ArrayDelete($linkarray, 2)
      WEnd
ConsoleWrite("2")
      Local $oH3Tags = _IETagNameGetCollection($ie, "H3")

      For $oH3Tag In $oH3Tags
         If($oH3Tag.ClassName = "journal_title") Then
            $oLinks = _IETagNameGetCollection($oH3Tag, "a")
            For $oLink In $oLinks
                  _ArrayAdd($linkArray, $oLink.href)
            Next
         EndIf
      Next
ConsoleWrite("3")
      $linkArray[0] = UBound($linkArray) - 1
      _ArrayDisplay($linkArray)
ConsoleWrite("4")
      For $i = 1 To $linkArray[0]
         _IENavigate($ie, $linkArray[$i]) ;Go to the journal.

         Local $divTags = _IETagNameGetCollection($ie, "div")

         For $divTag In $divTags
            If($divTag.id = "body_show_ori") Then
               $fout = FileOpen("lang8journals.txt", 1)
               FileWrite($fout, _IEPropertyGet($divTag, "InnerHTML") & @CRLF & "--------------------------------------------------" & @CRLF)
               FileClose($fout)
            ElseIf($divTag.id = "body_show_mo") Then
               $fout = FileOpen("lang8journals.txt", 1)
               FileWrite($fout, _IEPropertyGet($divTag, "InnerHTML") & @CRLF & @CRLF)
               FileClose($fout)
            EndIf
         Next
      Next
ConsoleWrite("5")
      $currentPage = $currentPage + 1
   WEnd

   _IEQuit($ie)
EndFunc

runScript()

The only part you need is the part in post 1, as the issue is from $currentPage being >1 and not returning from _IENavigate().

Edited February 16, 2019 by IAMK
added tag

[FrancescoDiMuro]

@IAMK
Why don't you add a little of error checking in your script, so you do know what it is doing?
At least after you _IENavigate() when you use the variable Current Page.

[IAMK]

@FrancescoDiMuro

Sorry, I should have mentioned before that the only reason I added ConsoleWrite/Tooltip was to see if the script ever reached those places. The only other thing I can do is Consolewrite("http://lang-8.com/MYID/journals?page=" & $currentPage) before the If statement, but that is the correct link format (and the website navigates to the correct page as expected).

[Nine]

 

2 hours ago, IAMK said:

However, when the code for page 1 finishes, I increment $currentPage to 2, and then the If statement never gets exited. It seems something inside _IENavigate() is not being handled properly when I use $currentPage in the URL.

By default _IENavigate wait for the page to be loaded. Try using no wait just to see if it breaks or not.  Maybe on page 2 some part never loads completely...

Edited February 16, 2019 by Nine

[Danp2]

Try sticking to https. You start there, but then switch to standard http.

[IAMK]

@Nine Ah, that will help me with debugging I think. I've been playing around for a while now, just doing random things. At one point I received an error when trying to navigate to the second page, but I can't reproduce that. The confusing thing is that the browser navigates correctly and does finish loading (because the icon stops spinning). Now, it works... I don't know why...

@Danp2 Good observation. I just copied the links as they are on the website at those pages.

Anyways, upon removing the wait as @Nine suggested, I found another issue in my code, correction in caps:

While(UBound($linkArray) > 1) ;Reset the array.
         _ArrayDelete($linkArray, 2) ;THIS SHOULD HAVE BEEN 1
      WEnd

Now, my code works correctly, but that's because I've changed the following code as well, as suggested by @Nine:

If($currentPage > 1) Then
         _IENavigate($ie, "http://lang-8.com/MYID/journals?page=" & $currentPage) ;ADDED 0 AS THE 3RD PARAMETER TO NOT WAIT FOR LOAD.
      EndIf
      ;ADDED HERE: Sleep(6000)

I'm still not convinced though. I want to know why not setting WAIT to false breaks. Any ideas?
Okay... just as I wrote the line above, I went to double check and changed the code back to set WAIT to true, and it works now...

I am seriously confused now, because the first change I made of setting _ArrayDelete to delete element 1 is called AFTER my ConsoleWrite(), which was not getting reached on page 2, which was directly after _IENavigate().

Anyone have any clue?

[Nine]

Might be just circumstantial. As I said, maybe some part of page 2 were not fully loading.   If this ever happens again, check if the circle still spins.  But it might be a good idea to remove the wait on _IENavigate () and add a _IELoadWait with timeout (and warning message in case of timeout).  Would make your script more robust.

[IAMK]

@Nine Thanks, I will try that. Also, the load did finish in my case (stopped spinning almost instantly).