WebDriver UDF - Help & Support (II)

Danp2 · July 9, 2020

@nooneclose This is Troubleshooting 101 --

Do you have an existing instance of Firefox open?
Was it launched with Marionette enabled?
Is the correct Marionette port being used?

You can check the last two items under the About:Config page.

Also, I just noticed this in your post --

Quote

_WDStartup: WinHTTP:    1.6.4.1

See troubleshooting section in the Wiki -- https://www.autoitscript.com/wiki/WebDriver#Troubleshooting

Also this --

Quote

_WDStartup: WD.au3: 0.2.0.9 (Update available)

You are multiple versions behind the latest release. 🙄

nooneclose · July 9, 2020

@Danp2 how do I check if Marionette is enabled?

yes I did have an existing instance of Firefox open.

I just finished updating both the Webdriver UDF and the WinHTTP UDF

Danp2 · July 9, 2020

@nooneclose Perhaps you missed this from my earlier post --

Quote

You can check the last two items under the About:Config page

Go to About:Config, click the Accept button, then filter by "Marionette"

nooneclose · July 9, 2020

@Danp2 It works now with the existing Firefox browser. Though it doesn't work every time? It acts like it doesn't get the session every run.

To better test that statement I tried to run the code 15 times back to back: Worked 11 times, Did not work: 4 times.

When it doesn't work the output says:

_WDStartup: Existing instance of geckodriver.exe detected!

When it doesn't say that it works.

Thank you very much for your help.

Edited July 9, 2020 by nooneclose

Danp2 · July 9, 2020

@nooneclose Glad you got it working, even if it's only intermittently. FWIW, I don't recall having this fail, so it's likely something on your end. 😛

I don't think geckodriver supports multiple sessions, so maybe it's failing because you haven't completed the prior session. No way for me to know for sure without more details.

nooneclose · July 9, 2020

@Danp2 maybe its do to my code messing up 😝. Im fixing another issue with excel opening a file so I keep killing my code and running it. Most likely the brakes are my fault.

Thanks again though. you were a huge help.

crazycrash · July 11, 2020

On 7/9/2020 at 12:11 AM, Danp2 said:

@crazycrash Here's a revised version that works AFAICS --

#include "wd_core.au3"
#include "wd_helper.au3"

Local $sDesiredCapabilities, $sSession, $sElement
Local $sImageXpath = "//a[@rel='theater' and @data-ploi]"
Local $sContentXpath = "//ancestor::div[@class='_3x-2']/preceding-sibling::div[contains(@class,'userContent')]"

SetupChrome()
_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error = $_WD_ERROR_Success Then
    ConsoleWrite("Starting extraction: " & @CRLF)
    _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/")
    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sImageXpath, 2000)
    $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sImageXpath, Default, True) ; get image elements
    For $i = 0 To UBound($aElements, 1) - 1
        $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath) ; get user content elements

        If @error = $_WD_ERROR_Success Then
            $sComment = _WD_ElementAction($sSession, $sElement, 'attribute', 'innerText')
            ConsoleWrite($sComment & @crlf & "===============" & @CRLF)
            Sleep(350)
        EndIf
    Next

EndIf

_WD_DeleteSession($sSession)
_WD_Shutdown()

Func SetupChrome()
_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--verbose --log-path="' & @ScriptDir & '\chrome.log"')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}'
EndFunc

I can confirm that on Chrome everything works just fine. It seems the "innertext" issue is then definitely related to Firefox. I also tried "textContent" without luck on Firefox. "textContent" also works on Chrome.

Also, could you explain how the relative Xpath to the previous element works? Can one specify the element to which the xpath should be relative to? Maybe it's possible that this doesn't behave the same way as it does in Firefox?

For Example:

//ancestor::div[@class='_5pcr userContentWrapper']//span[contains(@class,'timestampContent')]

always returns the same timestamp, the first one in the blog.

Edit: I just realized that also in Chrome it always returns the same text and date. Hence it seems the relative path is not related to the current element being processed.

Cheers

Edited July 11, 2020 by crazycrash

Danp2 · July 11, 2020

@crazycrash Seems like innerText is a property, not an attribute. Try this and let me know how it works for you --

#include "wd_core.au3"
#include "wd_helper.au3"

Local $sDesiredCapabilities, $sSession, $sElement
;~ Local $sImageXpath = "//a[@rel='theater' and @data-ploi]"
;~ Local $sContentXpath = "//ancestor::div[@class='_3x-2']/preceding-sibling::div[contains(@class,'userContent')]"
Local $sContentXpath2 = "//div[@data-testid='post_message' and contains(@class, '_5pbx')]"
Local $sJavascript = "window.scrollTo(0, document.body.scrollHeight)"
SetupGecko()
_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error = $_WD_ERROR_Success Then
    ConsoleWrite("Starting extraction: " & @CRLF)
    _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/")

    ; force page to load dynamic content
    For $i = 1 To 3
        _WD_ExecuteScript($sSession, $sJavascript)
    Next

    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath2, 2000)

    $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sContentXpath2, Default, True)
    For $i = 0 To UBound($aElements, 1) - 1
        $sComment = _WD_ElementAction($sSession, $aElements[$i], 'property', 'innerText')
        ConsoleWrite($sComment & @crlf & "===============" & @CRLF)
        Sleep(350)
    Next

EndIf

_WD_DeleteSession($sSession)
_WD_Shutdown()

Func SetupChrome()
_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--verbose --log-path="' & @ScriptDir & '\chrome.log"')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}'
EndFunc

Func SetupGecko()
_WD_Option('Driver', 'geckodriver.exe')
_WD_Option('DriverParams', '--log trace')
_WD_Option('Port', 4444)

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts":true}}}'
EndFunc

crazycrash · July 12, 2020

Thanks a lot for the guidance. Everything works beautifully on any browser.

crazycrash · July 12, 2020

It seems I still fail to find an element within or related to a previously found element. Could you show me an example of how to use _WD_FindElement to search within another or from that element? I tried the following to get the images related to the comments in the posts:

Local $sXpathPosts = "//div[@data-testid='post_message' and contains(@class, '_5pbx')]"
Local $sJavascript = "window.scrollTo(0, document.body.scrollHeight)"
Local $XpathPostTimeStamp = "//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"
Local $XpathPostImages = "//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"

$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error = $_WD_ERROR_Success Then
    ConsoleWrite("Starting extraction: " & @CRLF)
    _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/")
    ; force page to load dynamic content
    For $i = 1 To 3
        _WD_ExecuteScript($sSession, $sJavascript)
    Next
    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, 2000)
    Sleep(2000)
    $aElements = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, Default, True)
    For $i = 0 To UBound($aElements, 1) - 1
        $sComment = _WD_ElementAction($sSession, $aElements[$i], 'property', 'innerText')
        ConsoleWrite($sComment & @CRLF)
        $sTime = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $XpathPostTimeStamp, $aElements[$i], False)
        $sUtimestamp = _WD_ElementAction($sSession, $sTime, 'attribute', 'data-utime')
        ConsoleWrite("Timestamp extracted: " & $sUtimestamp & @CRLF)
        $aImages = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $XpathPostImages, $aElements[$i], True)
        ConsoleWrite("Images: " & UBound($aImages, 1) - 1 & @CRLF)
        ConsoleWrite(@CRLF & "==============Done with current post ================" & @CRLF)
        Sleep(350)
    Next
EndIf
_WD_DeleteSession($sSession)
_WD_Shutdown()

Console out:

__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/url; $sData={"url":"https://www.facebook.com/AstrophysicsAndAstronomy/"}
__WD_Post: StatusCode=200; ResponseText={"value":null}...
_WD_Navigate: {"value":null}
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]}
__WD_Post: StatusCode=200; ResponseText={"value":null}...
_WD_ExecuteScript: {"value":null}...
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]}
__WD_Post: StatusCode=200; ResponseText={"value":null}...
_WD_ExecuteScript: {"value":null}...
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/execute/sync; $sData={"script":"window.scrollTo(0, document.body.scrollHeight)", "args":[]}
__WD_Post: StatusCode=200; ResponseText={"value":null}...
_WD_ExecuteScript: {"value":null}...
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element; $sData={"using":"xpath","value":"//div[@data-testid='post_message' and contains(@class, '_5pbx')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"}}
_WD_WaitElement ==> Success
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//div[@data-testid='post_message' and contains(@class, '_5pbx')]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"8d023ade-5572-457a-813f-fbc2d9613fbb"},{"element-6066-11e4-a52e-4f735466cecf":"32260115-9a73-4e25-bc49-8366caa8f00d"},{"element-6066-11e4-a52e-4f735466cecf":"2f8952dc-a35c-49b8-97f4-3e4254a04b3b"},{"element-6066-11e4-a52e-4f735466cecf":"19be5389-6d9e-47e7-9807-052ffaeb214d"},{"element-6066-11e4-a52e-4f735466cecf":"a99ae5ca-7045-4bc1-9dbd-d45580e2a22f"}]}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/8d023ade-5572-457a-813f-fbc2d9613fbb/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"I do have to admit that although I have seen the published data plots, a true understandin...
_WD_ElementAction: {"value":"I do have to admit that although I have seen the published data plots, a true understandin...
I do have to admit that although I have seen the published data plots, a true understanding of the Higgs boson and some of the high energy particle physics done at CERN has eluded me, so I'm listening to and sharing this series on the discovery of the Higgs boson this week. This first episode describes the utility of colliding things at high energy in order to generate new particles and exchange energy for mass in the process.
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/8d023ade-5572-457a-813f-fbc2d9613fbb/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}...
_WD_ElementAction: {"value":"1594561734"}...
Timestamp extracted: 1594561734
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]}
Images: 4

==============Done with current post ================
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/32260115-9a73-4e25-bc49-8366caa8f00d/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"Comparing the Gas Giants\n\nOur solar system has 4 large planets made up mostly of gas: Ju...
_WD_ElementAction: {"value":"Comparing the Gas Giants\n\nOur solar system has 4 large planets made up mostly of gas: Ju...
Comparing the Gas Giants

Our solar system has 4 large planets made up mostly of gas: Jupiter, Saturn, Uranus, and Neptune. Each of them have some things in common; large numbers of moons, rings present, gaseous outer layers in their atmosphere, and it is thought each likely has some sort of rocky core around which the gases originally were collected as the planets formed.

However, the type of material in the planet’s interior depends on what it is made of and how big the plan...

See More
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/32260115-9a73-4e25-bc49-8366caa8f00d/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}...
_WD_ElementAction: {"value":"1594561734"}...
Timestamp extracted: 1594561734
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]}
Images: 4

==============Done with current post ================
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/2f8952dc-a35c-49b8-97f4-3e4254a04b3b/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Cen...
_WD_ElementAction: {"value":"Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Cen...
Meet the nearest star systems to Earth - Proxima Centauri and 2 companion stars, Alpha Centauri A and B, 4.2, 4.35, and 4.37 light years away, respectively. At least one planet is thought to exist there. What else might we find out about this solar system?
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/2f8952dc-a35c-49b8-97f4-3e4254a04b3b/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}...
_WD_ElementAction: {"value":"1594561734"}...
Timestamp extracted: 1594561734
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]}
Images: 4

==============Done with current post ================
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/19be5389-6d9e-47e7-9807-052ffaeb214d/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"An incredible view\n\nThis image was taken at 3060m above sea level (10,000ft), on the mou...
_WD_ElementAction: {"value":"An incredible view\n\nThis image was taken at 3060m above sea level (10,000ft), on the mou...
An incredible view

This image was taken at 3060m above sea level (10,000ft), on the mountain peak of Cerro Armazones in Chile. The high number of star trails is testament to the dark, clear site. This is one of the reasons why this mountain top will host the upcoming European Extremely Large Telescope (E-ELT).

The E-ELT, nicknamed the “world's biggest eye on the sky” will have a 39m (128ft) primary mirror and will constitute the largest optical/near-infrared telescope on Eart...

See More
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/19be5389-6d9e-47e7-9807-052ffaeb214d/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}...
_WD_ElementAction: {"value":"1594561734"}...
Timestamp extracted: 1594561734
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]}
Images: 4

==============Done with current post ================
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/a99ae5ca-7045-4bc1-9dbd-d45580e2a22f/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"This is what it is like to start off with an overview of our night sky and zoom in on the ...
_WD_ElementAction: {"value":"This is what it is like to start off with an overview of our night sky and zoom in on the ...
This is what it is like to start off with an overview of our night sky and zoom in on the Andromeda Galaxy.
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/a99ae5ca-7045-4bc1-9dbd-d45580e2a22f/element; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//abbr[contains(@class,'livetimestamp')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"b2b1d6b9-0d93-49ea-b682-5304746b835f"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/element/b2b1d6b9-0d93-49ea-b682-5304746b835f/attribute/data-utime
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"1594561734"}...
_WD_ElementAction: {"value":"1594561734"}...
Timestamp extracted: 1594561734
__WD_Post: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803/elements; $sData={"using":"xpath","value":"//ancestor::div[@class='_5pcr userContentWrapper']//a[@rel='theater' and @data-ploi]"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"0ccfdaf3-287e-4dde-8816-377acc7dac09"},{"element-6066-11e4-a52e-4f735466cecf":"f8cbe335-a54f-4bdf-ae24-6d23ed85c66d"},{"element-6066-11e4-a52e-4f735466cecf":"6ed740e5-d5ec-4231-9224-2be581254609"},{"element-6066-11e4-a52e-4f735466cecf":"e836156d-dda6-4a14-8ae3-4023ac15722d"},{"element-6066-11e4-a52e-4f735466cecf":"9d9c7bc0-9692-44e1-83bd-49a97e9c6189"}]}
Images: 4

==============Done with current post ================
__WD_Delete: URL=HTTP://127.0.0.1:9515/session/2216519c55948f8047423c429cf61803
__WD_Delete: StatusCode=200; ResponseText={"value":null}...

AS you can see I always get the 4 Images, same elements. An example to only search within another element in java taken from seleniumjava:

WebElement result = driver.findElement(RESULT_BOX_XPATH);
WebElement title = result.findElement(TITLE_XPATH);

I'm going to try to search for a child element like that afterwards, will post an update in a few hours. Thanks

Edit: Added xpath's and console out 😃

Edited July 12, 2020 by crazycrash

Danp2 · July 12, 2020

@crazycrash It would help if you includ the xpath strings needed to run your code. 😃

Also, should your results from Scite output panel.

crazycrash · July 12, 2020

I fixed the post above trying to get the xpath relative to another previously found element. Below my attempt at getting a child element of a previously found element. (Similar to the javascribt above). However, this is also failing.

Local $sXpathPosts = "//div[@class='_5pcr userContentWrapper']"
Local $sXpathUserContent = "//div[contains(@class, 'userContent')]" ;xpath within element $PostContent to usercontent
Local $sXpathPostTimeStamp = "//div[contains(@class, '_6a')]//abbr[contains(@class, '5ptz')]" ;xpath within element $PostContent to timestamp
Local $sXpathPostImages = "//a[@rel='theater' and @data-ploi]" ;xpath within element $PostContent to images

$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error = $_WD_ERROR_Success Then
    ConsoleWrite("Starting extraction: " & @CRLF)
    _WD_Navigate($sSession, "https://www.facebook.com/AstrophysicsAndAstronomy/")
    ; force page to load dynamic content
    For $i = 1 To 3
        _WD_ExecuteScript($sSession, $sJavascript)
    Next
    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, 2000)
    Sleep(2000)
    $ePosts = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathPosts, Default, True)
    For $i = 0 To UBound($ePosts, 1) - 1
        $eUserContent = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, $sXpathUserContent, $ePosts[$i], False)
        If @error = $_WD_ERROR_Success Then ;if there is a user content
        $sUserContent = _WD_ElementAction($sSession, $eUserContent, 'property', 'innerText')
        ConsoleWrite($ePosts & @CRLF)
        EndIf
        ConsoleWrite(@CRLF & "==============Done with current post ================" & @CRLF)
        Sleep(350)
    Next
EndIf
_WD_DeleteSession($sSession)
_WD_Shutdown()

Console out (always finds the same element for the post content, as where I would like to find the usercontent child element of the post element)

__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element; $sData={"using":"xpath","value":"//div[@class='_5pcr userContentWrapper']"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
_WD_WaitElement ==> Success
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/elements; $sData={"using":"xpath","value":"//div[@class='_5pcr userContentWrapper']"}
__WD_Post: StatusCode=200; ResponseText={"value":[{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"},{"element-6...
_WD_FindElement: {"value":[{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"},{"element-6066-11e4-a52e-4f735466cecf":"cbc1ba94-029a-49cd-80c4-3cf5718859b2"},{"element-6066-11e4-a52e-4f735466cecf":"13fb39a9-629b-449c-b30b-a3cd6497d659"},{"element-6066-11e4-a52e-4f735466cecf":"8f37bd3e-0e7e-4a1d-9394-b67e2189a5da"},{"element-6066-11e4-a52e-4f735466cecf":"0a87bf1b-3c98-43a9-8c88-60eac244bcff"},{"element-6066-11e4-a52e-4f735466cecf":"81360c07-65d4-4f7d-8fbc-befecf52ff26"},{"element-6066-11e4-a52e-4f735466cecf":"f0221fa5-e46f-41cc-8012-2a34bbfea96a"},{"element-6066-11e4-a52e-4f735466cecf":"bdc6a3dd-42ba-4c48-8302-14b081e443e6"}]}
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/cbc1ba94-029a-49cd-80c4-3cf5718859b2/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/13fb39a9-629b-449c-b30b-a3cd6497d659/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/8f37bd3e-0e7e-4a1d-9394-b67e2189a5da/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/0a87bf1b-3c98-43a9-8c88-60eac244bcff/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/81360c07-65d4-4f7d-8fbc-befecf52ff26/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/f0221fa5-e46f-41cc-8012-2a34bbfea96a/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Post: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/bdc6a3dd-42ba-4c48-8302-14b081e443e6/element; $sData={"using":"xpath","value":"//div[contains(@class, 'userContent')]"}
__WD_Post: StatusCode=200; ResponseText={"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}...
_WD_FindElement: {"value":{"element-6066-11e4-a52e-4f735466cecf":"d78c4a5b-abc5-42a6-8639-54c074c90799"}}
__WD_Get: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995/element/d78c4a5b-abc5-42a6-8639-54c074c90799/property/innerText
__WD_Get: StatusCode=200; $iResult = 0; $sResponseText={"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...
_WD_ElementAction: {"value":"The Universe\n2 hrs · \n\nSKY FROM THE EQUATOR\n\nThis long exposure image, by Kwon O Chul...

==============Done with current post ================
__WD_Delete: URL=HTTP://127.0.0.1:9515/session/f66d3889be40c7a89ab174e769101995
__WD_Delete: StatusCode=200; ResponseText={"value":null}...
_WD_DeleteSession: {"value":null}

So basically I always get the same element for the usercontent element instead of the usercontent child element of the postelement. The reason for all this is that sometimes there is text, sometimes not, sometimes there is one or multiple images. Hence I need the relationship between those elements. Thanks

Edited July 13, 2020 by crazycrash

cramaboule · July 13, 2020

Hi,

I have an internal website like: https://mywebsite
Chrome, obviously is warning me because of certificate, that obviously I do not have.

How to bypass, this warning ?

Cramaboule

Danp2 · July 13, 2020

@cramaboule Have you tried adding the following to your DesiredCapabilities?

"args":["--ignore-certificate-errors"]

cramaboule · July 13, 2020

9 minutes ago, Danp2 said:
@cramaboule Have you tried adding the following to your DesiredCapabilities?
"args":["--ignore-certificate-errors"]

__WD_Post: URL=HTTP://127.0.0.1:9515/session; $sData={"capabilities":{"args":["--ignore-certificate-errors"]}}
__WD_Post: StatusCode=200; ResponseText={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"83.0....

It dosn't work !!! 😞
C.

*EDITED the response code

Edited July 13, 2020 by cramaboule

Danp2 · July 13, 2020

@cramaboule I don't think you implemented it correctly. It should look like this --

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}'

Note: I didn't actually test this, so YMMV. 😉

If that still doesn't work, then I know for a fact that this does --

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"acceptInsecureCerts": true, "goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}'

cramaboule · July 13, 2020

1 minute ago, Danp2 said:
@cramaboule I don't think you implemented it correctly. It should look like this --
$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}'
Note: I didn't actually test this, so YMMV. 😉

If that still doesn't work, then I know for a fact that this does --
$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"acceptInsecureCerts": true, "goog:chromeOptions": {"w3c": true, "excludeSwitches": [ "enable-automation"], "useAutomationExtension": false }}}}'

Quote

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--ignore-certificate-errors"]}}}}'

Works fine for me...

Thanks a lot

Cramaboule.

crazycrash · July 13, 2020

I found what I d like to do in the official documentation of selenium:

selenium.dev -> Find Element(s) From Element

If you could show me how to do that in AutoIt it would answer my previous questions 😃

Thanks

Edited July 13, 2020 by crazycrash

Danp2 · July 13, 2020

@crazycrash That is what _WD_FindElement already does, but it isn't working correctly for some reason. I'll have to review the code to see if it's something that I've done wrong.

cal · July 14, 2020

On 7/3/2020 at 3:55 PM, cal said:

On to the actual issue.

I just started looking into the webdriver. I've hit a snag that I'm not seeing talked about. On a whim I tried to run the same exact code at work and its fine.

All I'm doing is opening chrome, moving to another page and closing. I'm just seeing how things work. When I try and open chrome using my existing profile, two of my extensions are removed by the webdriver. I've been hunting for the issue for a couple days when I tried running it at work and it does not have the same issue

Responding to my own post....

A clean test profile had the same issues. It fine at work fine but removes lastpass at home. Same code. Today I decided to look into this deeper. After completely redoing Chrome a couple times and playing around a lot with nothing fixing the problem I found the issue.

Turns out the issue was the way that lastpass was installed. At least I think thats the case. I have no issues with lastpass so was not looking at it as the source of the problem. I had one odd behavior that I thought was chrome syncing extensions. Lastpass would be removed by the webdriver. Then after starting chrome normally the extension was automatically put back. I thought this was chrome doing a sync and reinstalling the missing extension.

But during my test today I had a clean blank chrome install with no accounts logged in or syncing. .... and the lastpass extension installed itself without me doing so on a fresh blank install. Since I was not logged into chrome and I could see the sync was off, where did it come from?

I don't know for sure but I believe its because I had done a full Lastpass install on my home computer and it was running as a service in the background. This service is what reinstalled it. Removing this caused lastpass to no longer self install when I opened chrome. Likely I installed lastpass at work via the extensions and never did the full thing. Its always synced just fine between Firefox/chrome both at home and work. All 4 browsers stayed in sync.

My guess is that the full lastpass service was seeing chrome start using the webdriver and stepping in and removing the lastpass extension. Then when chrome was started without the webdriver, it restored the extension.

Removing the full lastpass install and using just the extension install method has fixed the issue. Lastpass is no longer being removed when the webdriver starts.

Cal.

WebDriver UDF - Help & Support (II)

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

SamsonSlice

Danp2

Jos

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Recently Browsing 0 members