Extracting data by looping through a website

[jasontj]

Hi all.  I hope to get a little help on this since we're all cooped up at home.   Any help would be appreciate.

I'm trying to extract some data from an old, internal website, by looping through links, 3 layer deep.  The first pass works fine.  Then it tries to loop again and it loses track of my arrays of links.

 

Getting this error.

"C:\Users\Jason\Documents\test2.au3" (19) : ==> The requested action with this object has failed.:
if StringInStr ( $oLink2.href, $sSearch2, 0) > 0 Then
if StringInStr ( $oLink2^ ERROR

 

 

#include <IE.au3>

$oIE = _IECreate("https://www.myinternalweb.com/organizations", 1)
WinSetState(_IEPropertyGet($oIE, "title"), "", @SW_MAXIMIZE)
$sSearch = "/organizations/"
$sSearch2 = "/profile/"
$sSearch3 = "mailto:"
$oLinks = _IELinkGetCollection($oIE)
If IsObj($oLinks) Then
   For $oLink In $oLinks
       if StringInStr ( $oLink.href, $sSearch, 0) > 0 Then
         $sLinkText = _IEPropertyGet($oLink, "innerText")
;         MsgBox(0, "test", $oLink.href)
;         MsgBox(0, "Inner", $sLinkText)
         _IENavigate($oIE, $oLink.href)
         $oLinks2 = _IELinkGetCollection($oIE)
         If IsObj($oLinks2) Then
            For $oLink2 In $oLinks2
                  if StringInStr ( $oLink2.href, $sSearch2, 0) > 0 Then
                  $sLinkText2 = _IEPropertyGet($oLink2, "innerText")
;                  MsgBox(0, "test", $oLink2.href)
                  _IENavigate($oIE, $oLink2.href)

                  $oLinks3 = _IELinkGetCollection($oIE)
                  If IsObj($oLinks3) Then
                     For $oLink3 In $oLinks3
;                        msgbox(0, "test 3", $oLink3.href)
                        if StringInStr ( $oLink3.href, $sSearch3, 0) > 0 Then
                           $sLinkText3 = _IEPropertyGet($oLink3, "innerText")
                           MsgBox(0, "test", $oLink3.href)
                           MsgBox(0, "Inner", $sLinkText3)

                           $oLinks4 = _IETagNameGetCollection($oIE, "h3")
                           For $oLink4 In $oLinks4
                               If $oLink4.ClassName = "mt-1 mb-0" Then
;                                  msgbox(0, "test", $oLink4.ClassName)
                                   MsgBox(4096, "href attribute", $oLink4.outerText)
                               EndIf
                           Next
                        endif
                     Next
                  EndIf
               endif
            Next
         EndIf
      endif
   Next
EndIf

[Subz]

You could test if the $oLink2 is an object before referencing it, you can also try using _IEAttach to re-attach $oIE to the current page, based on the previous link.  Another way is to capture all .hrefs into an array then browse to each of the array items and add those to an array etc...

[jasontj]

I'm going 3 levels deep, so using the _IEAttach will probably lose track of things.  I figured someone would say to put these into an array, so that's what I'm writing right now.  Thanks for the quick response.  I just wanted to see if I was missing something simple.

[Nine]

The  _IENavigate() is scrambling the initial _IECreate.  What I would suggest is to create 3 $oIE instances or use tab navigation for each of your level down, so you can go back without loosing the previous level of objects.

Please when you post code, use this tool.

Edited March 24, 2020 by Nine

[Colduction]

Hi @jasontj, please read this git, may be helpful for your future projects:

https://github.com/SAP/openui5/issues/1697