WebDriver UDF - Help & Support (II)

[Danp2]

You should be able to grab the links with the correct xpath. Try this --

expandcollapse popup

#include "wd_helper.au3"
#include "wd_core.au3"

OnAutoItExitRegister("_onExit")

Local $sDesiredCapabilities

$_WD_DEBUG = $_WD_DEBUG_None ; You could also use $_WD_DEBUG_Error
;~ You can also control the visibility of the console with the function _WD_ConsoleVisible.

_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--log-path=' & @ScriptDir & '\chrome.log')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--user-data-dir=C:\\Users\\' & @UserName & '\\AppData\\Local\\Google\\Chrome\\User Data\\", "--profile-directory=Default"]}}}}'


_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)

If @error = $_WD_ERROR_Success Then
    _WD_Navigate($sSession, "https://www.amazon.com/s?k=guide+to+investing+crypto&i=digital-text&page=2&qid=1587374511&ref=sr_pg_2")
    _WD_LoadWait($sSession)

    $oEls = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//div[@data-asin]//h2/a[contains(@href,'/dp/')]", Default, true)

    For $i=0 to UBound($oEls)-1
            $sHref = _WD_ElementAction($sSession, $oEls[$i], "Attribute", "href")
            ConsoleWrite("@@@@@@@@@@@@@@ " & $oEls[$i] & " ___" & $sHref & @CRLF)
    Next
EndIf

Func _onExit()
    _WD_DeleteSession($sSession)
    _WD_Shutdown()
EndFunc

 

[frank10]

Thank you, it works.

[HJL]

Hi Dan,
is there a function with which I can determine whether a session still exists? All functions that I have tested and that have $sSession as parameters take a long time to generate an error message that I can use to recognize that there is no window for the session.
Many thanks.

[Danp2]

@HJL Not that I can think of, but we could probably create something if you want to provide some additional details. How is the session getting "lost"? Is the webdriver console getting closed? Or the web browser? etc

[HJL]

11 hours ago, Danp2 said:

@HJL Not that I can think of, but we could probably create something if you want to provide some additional details. How is the session getting "lost"? Is the webdriver console getting closed? Or the web browser? etc

The user can open multiple browserpages from a menu. The prerequisite is that the same profile is always used because of the plugins and cookies. So we can't open Chrome with a new session. If the user stupidly closes all the windows, which he shouldn't, the session is gone. We would and could then start a new session. But to do this, I need a query to see if the current session is still running or not in the way like "If _wd_sessionexists($sSession) Then". It works with the errorcodes of any function already, but the time to return the errorcode is very long. 

 

Edited April 29, 2020 by HJL

[seadoggie01]

@HJL Would a simple WinExists work here? Sorry to suggest something simple, but sometimes it works

[Danp2]

@HJL I think that I understand the issue and have some ideas on a solution, but some additional details would help. What error code are you receiving in this scenario? Can you post a brief example that demonstrates the issue?

[HJL]

On 4/29/2020 at 2:33 PM, Danp2 said:

@HJL I think that I understand the issue and have some ideas on a solution, but some additional details would help. What error code are you receiving in this scenario? Can you post a brief example that demonstrates the issue?

expandcollapse popup

#include "wd_core.au3"
#include "wd_helper.au3"

#cs ====================================================================
Function opens file "user.txt", reads URL, opens URL and deletes "user.txt"
waits for new file user.txt with new URL
:loop
Test: create "user.txt", open in editor, write URL and save
editor remains open, write other URL and save
"ESC"  ==>  end
Problem:
User can close all tabs without problems
but if he closes the window => error because of wrong $sSession
Needed:
function _WD_SessionExists($sSession) to start an new session if old one was closed
Reason:
Start a new session for each URL => Firefox error at the second attempt
because of using same profile
#ce ===================================================================

$_WD_DEBUG = False
HotKeySet("{ESC}", "_Terminate")
Local $sDesiredCapabilities, $sSession, $sURL
Local Const $sFilePath = @ScriptDir & "\user.txt"

ReadLoop()  ; Exit with "Esc"

Func ReadLoop()
    While 1
        If FileExists ($sFilePath) Then
            Local $hFileHandle = FileOpen($sFilePath, $FO_READ)
            $sURL = FileReadLine ($hFileHandle)
            FileClose ($hFileHandle)
            FileDelete($sFilePath)
            If $sURL > "" Then
                ; only first time or session deleted because window was closed: create session
                If $sSession = "" Or _WD_SessionExists($sSession) = False Then
                    SetUpGecko()
                    _WD_Startup()
                    $sSession = _WD_CreateSession($sDesiredCapabilities)
                    _WD_LoadWait($sSession, 1000)
                    _WD_Navigate($sSession, $sURL)
                Else
                    If StringLeft($sURL,4) <> "http" Then   ; error?
                        $sURL = "http://" & $sURL
                    EndIf
;                   _WD_NewTab($sSession, True, -1, $sURL, "width=1200, height=800") ; new Window
                    _WD_NewTab($sSession, True, -1, $sURL) ; new tab
                EndIf
            EndIf
        EndIf
        sleep(2000)
    WEnd
EndFunc  ;==> ReadLoop

Func _WD_SessionExists($sSession)
    Return True ; test if session still exists with quick response
EndFunc   ; ==>_WD_SessionExists

Func _Terminate()
    _WD_DeleteSession($sSession)
    _WD_Shutdown()
    Exit
EndFunc   ;==>_Terminate

Func SetupGecko()
    _WD_Option('Driver', 'geckodriver.exe')
    _WD_Option('DriverParams', '--log trace --marionette-port 2828')
    _WD_Option('Port', 4444)
    $sDesiredCapabilities = '{"capabilities":{"alwaysMatch": {"moz:firefoxOptions": {"args": ["-profile", "C:\\webdriver\\FF_WD_Profile"]}}}}'
EndFunc

This is only a short example to show the problem. Actually the Script has more than 1.000 lines and the problem is, that we have to use always the same firefox profile and firefox reports an error when opening a second session. It would not help to get FF to open that second ore more sessions because of memory and other problems.

What I have noticed while running that demo: 1. URL = http://ndr.de 2. URL ndr.de leads to error 404 because URL to open is http://ndr.de/ndr.de. Thats why I check the "http" in the URL and add "http://" if not there.

Thanks for your help.

 

Needed Func _WD_SessionExists($sSession).

Edited April 30, 2020 by HJL

[Danp2]

@HJL You previously stated this --

Quote

But to do this, I need a query to see if the current session is still running or not in the way like "If _wd_sessionexists($sSession) Then". It works with the errorcodes of any function already, but the time to return the errorcode is very long. 

I'm not seeing this behavior. How long is "very long" in your case?

What might be helpful is if you change your example so that $_WD_DEBUG = $_WD_DEBUG_Info and then add some ConsoleWrites so that we can track start / end times for a given command.

[HJL]

My example has no "very long time" because I always return "True" in my "Dummy"-Function 

Func _WD_SessionExists($sSession)
    Return True ; test if session still exists with quick response
EndFunc   ; ==>_WD_SessionExists

I don't test "If $sSession still exist" because I don't know how to test it in a reasonable kind.

Edited April 30, 2020 by HJL

[frank10]

I need to make a jQuery  ajax call with a JS function on success and it works. Something like this, but more complex:

local $jqueryCommand = "jQuery($.ajax({ url:'" & $urlAm & "',success:function(result){$('.sg-col-inner').eq(2).prepend(result);}}))"
    _WD_ExecuteScript($sSession, $jqueryCommand )

How can I access the result var that contains the page loaded to parse it in my autoit script ?

Edited April 30, 2020 by frank10

[HJL]

2 hours ago, Danp2 said:

@HJL You previously stated this --

I'm not seeing this behavior. How long is "very long" in your case?

What might be helpful is if you change your example so that $_WD_DEBUG = $_WD_DEBUG_Info and then add some ConsoleWrites so that we can track start / end times for a given command.

Solved:

After some attempts I found a reliable and fast solution:

Func _WD_SessionExists($sSession)
    If _WD_Timeouts($sSession) = "0" Then
        Return False
    Else
        Return True
    EndIf
EndFunc   ; ==>_WD_SessionExists

I use the result of _WD_Timeouts($sSession)  to determine if the session $sSession still exists or not. The delay is very low. If result = "0" the session doesn't exist any more.

Before I tried to use the errorcode of other functions like WD_navigate , but they took a long time to respond.

[Danp2]

1 hour ago, HJL said:

Solved: After some attempts I found a reliable and fast solution:

I'm glad that you found something that works for you. However, it's still unclear which commands that you tried before that resulted in a long pause. To be honest, I don't see why _WD_Timeouts would be faster that other commands, such as _WD_Status, _WD_Action, etc.

Also, you would be better off checking the value of @error since the value returned during an error condition is subject to change in the future.

[Danp2]

@frank10 _WD_ExecuteScript supports the Execute Async Script option from the W3C specs, but it isn't something that I've personally tested. Suggest that you come up with a short, runnable script that we can run if you want us to look into it further.

Edit: Maybe you could execute the javascript and then use _WD_WaitElement to detect when the page update was complete

Edited April 30, 2020 by Danp2

[frank10]

Ok thank you.

This is the previous script adapted with the ajax call:

#include "wd_helper.au3"
#include "wd_core.au3"

OnAutoItExitRegister("_onExit")

Local $sDesiredCapabilities

$_WD_DEBUG = True

_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--log-path=' & @ScriptDir & '\chrome.log')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--user-data-dir=C:\\Users\\' & @UserName & '\\AppData\\Local\\Google\\Chrome\\User Data\\", "--profile-directory=Default"]}}}}'

_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)

If @error = $_WD_ERROR_Success Then
    _WD_Navigate($sSession, "https://www.amazon.com/s?k=guide+to+investing+crypto&i=digital-text&page=2&qid=1587374511&ref=sr_pg_2")
    _WD_LoadWait($sSession)
    _WD_jQuerify($sSession) 
    
    local $jqueryCommand = "jQuery($.ajax({ url:'www.amazon.com/dp/B07H71XXGY',success:function(result){console.log('loaded');}}))"
    _WD_ExecuteScript($sSession, $jqueryCommand )
    
    MsgBox(0,'',"where's my html page loaded by jquery ajax into var result?")

EndIf

Func _onExit()
    _WD_DeleteSession($sSession)
    _WD_Shutdown()
EndFunc

 

EDIT

Or another option  could be to create a global JS var, fill it with the success func of ajax  and then access it from autoit... but how?

Edited April 30, 2020 by frank10

[Danp2]

9 minutes ago, frank10 said:

Or another option  could be to create a global JS var, fill it with the success func of ajax  and then access it from autoit... but how?

Something like this should work, but you would potentially need to call it in a loop until you get the desired result --

_WD_ExecuteScript($sSession, "return yourJSVariable;")

 

[frank10]

Ok I did it:

expandcollapse popup

#include "wd_helper.au3"
#include "wd_core.au3"

OnAutoItExitRegister("_onExit")

Local $sDesiredCapabilities

$_WD_DEBUG =  True ; You could also use $_WD_DEBUG_Error
;~ You can also control the visibility of the console with the function _WD_ConsoleVisible.

_WD_Option('Driver', 'chromedriver.exe')
_WD_Option('Port', 9515)
_WD_Option('DriverParams', '--log-path=' & @ScriptDir & '\chrome.log')

$sDesiredCapabilities = '{"capabilities": {"alwaysMatch": {"goog:chromeOptions": {"w3c": true, "args":["--user-data-dir=C:\\Users\\' & @UserName & '\\AppData\\Local\\Google\\Chrome\\User Data\\", "--profile-directory=Default"]}}}}'


_WD_Startup()

$sSession = _WD_CreateSession($sDesiredCapabilities)

If @error = $_WD_ERROR_Success Then
    _WD_Navigate($sSession, "https://www.amazon.com/s?k=guide+to+investing+crypto&i=digital-text&page=2&qid=1587374511&ref=sr_pg_2")
    _WD_LoadWait($sSession)
    _WD_jQuerify($sSession)

    local $jqueryCommand = "jQuery($('head').append('<script  type=" & 'text/javascript' & "> var myPage;</script>'))"
    _WD_ExecuteScript($sSession, $jqueryCommand )

    local $jqueryCommand = "jQuery($.ajax({ url:'www.amazon.com/dp/B07H71XXGY',success:function(result){myPage = result; console.log('loaded');}}))"
    _WD_ExecuteScript($sSession, $jqueryCommand )

    while 1
        local $answer = _WD_ExecuteScript($sSession, "return myPage;")
        if StringLen($answer) > 1000 Then ExitLoop
        Sleep(10)
    WEnd

    MsgBox(0,'',"where's my result page loaded by jquery ajax?")
EndIf

Func _onExit()
    _WD_DeleteSession($sSession)
    _WD_Shutdown()
EndFunc

I get the page content (even if with only the "<" tag as "\u003C"), anyway, is it possible to transform in some way a text var like this into elements parsable with  xpath or jquery or I can only parse it with StringRegExp?

[Danp2]

@frank10 Not sure that I understand why you chose to implement your script like this. What are you gaining by doing it this way?

[frank10]

Speed.

Some time ago I used IE to load pages in hidden IE tabs and I got loading time about 1s at page, with this ajax loading I get 8ms at page!

If there is another method to retrieve a page in the order of ms, I could use it instead of ajax.

EDIT

But it may be just the HTML rendered code, parsable with DOM, instead of plain text... so it could be that the speed thing.

Edited May 1, 2020 by frank10

[Danp2]

@frank10 Have you tried _INetGetSource instead? If so, does that give you the same results as the ajax method?

Using _INetGetSource should be faster, more reliable, and less complex.