Jump to content

Webdriver Firefox: Login to a website and get data from it. Learning Webdriver


Blaxxun
 Share

Go to solution Solved by Danp2,

Recommended Posts

Hello,

I open a new thread here since i dont want to hijack the thread of @water  called "I'm creating a WebDriver tutorial".
As recommended i started to take a deeper look into the "GDPR fines.au3" example.

I also fount that "au3WebDriver-0.11.0" comes with a nice Help file "Webdriver.chm"

When running GDPR fines.au3 i noticed that it uses a function called:

_WD_GetTable($sSession, $sBaseElement, $iPage)


It has 3 function parameters and is IN the "GDPR fines.au3"

So i got an error: wrong number of args.
Further investigation showed me that this function is already defined in the "wd_helper.au3" which is loaded at the beginnings #include but has only 2 parameters.
So i commented out the function in the "wd_helper.au3"

The _WD_GetTable function ran error free now but the next stop was:

_WD_ElementOptionSelect($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']//option[contains(text(),'" & $aOptions[UBound($aOptions, 1) - 2] & "')]")


Variable subscript badly formatted (Error at -> UBound)
 

I'm not really sure why this happens because i dont really know how to traverse a website yet.
Have to dig deeper and read into it more.

Link to comment
Share on other sites

The GDPR fines.au3 was written long time ago. The WebDriver UDF didn't have a _wd_gettable function at that time. Now it has, but I didn't have the time to update the GDPR example script.
I suggest to not modify the WebDriver UDF as it constantly gets enhanced. Drop my _WD_GetTable and use the one provided with the WebDriver UDF.

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

Okay, i understand. I usually never modify UDF's. But in this case i got confused why there is one version of the _WD_GetTable with 3 and one in the UDF with 2 parameters.
I was able to follow along in the example GDPR until the error.

Is there some similar example that is working and that is not old?

Thanks!

Link to comment
Share on other sites

@BlaxxunHere's a modified version of the GDPR fines example that should work once adjusted for your environment --

; #AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7
#include "wd_core.au3"
#include "wd_helper.au3"
#include "_HtmlTable2Array.au3" ; <== This can be commented out, but the table extraction will be much slower

Global $sDesiredCapabilities, $sSession, $sScriptName = "GDPR fines", $aResult_All
$_WD_DEBUG = $_WD_DEBUG_Error ; Set debug level: $_WD_DEBUG_None (0) = No logging to console, $_WD_DEBUG_Error (1) = Error logging to console, $_WD_DEBUG_Info (2) = Full logging to console (Default)
Global $hSplash = SplashTextOn($sScriptName, "Running ... please be patient!", Default, 45, 0, 0)

; (1, 2) ----------------------------------------------------------------------
; Automate the website using FireFox. Start FF from a different location
; -----------------------------------------------------------------------------
Global $sDriver = "geckodriver.exe" ; <== Please modify this statement to your environment. Must be a local drive, network drives don't work!
; Setup Firefox
_WD_Option("Driver", $sDriver)
If @error Then Exit SetError(1, @error)
_WD_Option('DriverParams', '--log trace')
_WD_Option('Port', 4444)
$sDesiredCapabilities = '{"desiredCapabilities":{"javascriptEnabled":true,"nativeEvents":true,"acceptInsecureCerts":true}}'
; (1, 2) End ------------------------------------------------------------------

_WD_Startup()
If @error Then Exit SetError(2, @error)
$sSession = _WD_CreateSession($sDesiredCapabilities)
If @error Then Exit SetError(3, @error)

; (3) -------------------------------------------------------------------------
; Maximize browser window
; -----------------------------------------------------------------------------
_WD_Window($sSession, "Maximize")
If @error Then Exit SetError(4, @error)
; (3) End ---------------------------------------------------------------------

RetrieveTable($aResult_All)
If @error Then Exit SetError(5, @error)

ControlSetText($hSplash, "", "Static1", "Shutting down WebDriver automation")
_WD_DeleteSession($sSession)
If @error Then Exit SetError(6, @error)
_WD_Shutdown()
If @error Then Exit SetError(7, @error)
SplashOff()
_ArrayDisplay($aResult_All)
MsgBox($MB_ICONINFORMATION, $sScriptName, "Finished!", 5)
Exit

Func RetrieveTable(ByRef $aResult_All)
    Local $aResult, $sElement, $iPage = 0, $sText, $aOptions
    ControlSetText($hSplash, "", "Static1", "Opening website www.enforcementtracker.com")

    ; Open website
    _WD_Navigate($sSession, "http://www.enforcementtracker.com/")
    If @error Then Return SetError(@error)

    ; (4) ---------------------------------------------------------------------
    ; Retrieve the values from the selection list and use the second to last (50)
    ; -------------------------------------------------------------------------
    $sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']")
    $aOptions = _WD_ElementSelectAction($sSession, $sElement, 'options')
    _WD_ElementOptionSelect($sSession, $_WD_LOCATOR_ByXPath, "//select[@name='penalties_length']//option[contains(text(),'" & $aOptions[UBound($aOptions, 1) - 2][0] & "')]")
    If @error Then Return SetError(@error)
    ; (4) End -----------------------------------------------------------------

    ; Wait for initial appearance of navigation button
    _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, "//a[@class='paginate_button next']")

    ; (5) ---------------------------------------------------------------------
    ; Sort table by fine (descending)
    ; -------------------------------------------------------------------------
    $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//th[contains(text(),'Fine [')]") ; Find "Fine" header of the table
    _WD_ElementAction($sSession, $sElement, "click") ; Click twice to sort descending
    _WD_ElementAction($sSession, $sElement, "click")
    ; (5) End -----------------------------------------------------------------

    While 1
        $iPage = $iPage + 1
        ControlSetText($hSplash, "", "Static1", "Loading data from website - page " & $iPage)
        $aResult = _WD_GetTable($sSession, "//table[@id='penalties']")
        If @error Then Return SetError(@error)
        If $iPage = 1 Then
            $aResult_All = $aResult
        Else
            _ArrayConcatenate($aResult_All, $aResult, 2)
        EndIf

        ; (6, 7) --------------------------------------------------------------
        ; Find the "next" button. If found scroll down so the button comes into
        ; view and click the button. Else exit the function
        ; ---------------------------------------------------------------------
        $sElement = _WD_FindElement($sSession, $_WD_LOCATOR_ByXPath, "//a[@class='paginate_button next']") ; Find "Next" button
        If @error Then ExitLoop
        _WD_ExecuteScript($sSession, "arguments[0].scrollIntoView(true);", '{"' & $_WD_ELEMENT_ID & '":"' & $sElement & '"}') ; Scroll down to the link
        If @error Then
            Return SetError(@error)
        Else
            _WD_ElementAction($sSession, $sElement, "click") ; Click on the "Next" button
        EndIf
        ; (6, 7) End ----------------------------------------------------------

    WEnd
EndFunc   ;==>RetrieveTable

 

Link to comment
Share on other sites

@Danp2 Hello Dan, thanks for the working example.

I was able to run it and extract all 33 pages of this website which gave me an array of 1600 rows.
Also the UDF from @Gianni  ( _HtmlTable2Array.au3 ) made the whole process way faster. Thanks for that!
I also made a second version of the _HtmlTable2Array.au3 with the recommended changes from @barbossa .
Works also but i could not feel a speed up. Although i did not measure the time. Maybe it only benefits from larger tables. I dont know.

Cool!

Now i can start to disect and learn about the navigation

Thank you guys!

Link to comment
Share on other sites

Just a hint @Blaxxun, you will also find help regarding WebDriver in the german forum. In case you're feeling more comfortable in your native language (mal abgesehen vom "Schwitzerdütsch" 😅). Out of your english sentences and grammar behavior, I guess it does not matter to you, but I just want to mention it 🤝 .

Besides that you're in good hands (@Danp2, @mLipok and other great guys here) regarding WebDriver.

Best regards
Sven

Stay innovative!

Spoiler

🌍 Au3Forums

🎲 AutoIt (en) Cheat Sheet

📊 AutoIt limits/defaults

💎 Code Katas: [...] (comming soon)

🎭 Collection of GitHub users with AutoIt projects

🐞 False-Positives

🔮 Me on GitHub

💬 Opinion about new forum sub category

📑 UDF wiki list

✂ VSCode-AutoItSnippets

📑 WebDriver FAQs

👨‍🏫 WebDriver Tutorial (coming soon)

Link to comment
Share on other sites

Hello,

So i made it to the website, through the Cookie Popup and the Login.
Also a Tab and a collapse-box.

Now im at the final table for the current day (still need to solve the date selection), but one step after the other.
I have the xpath for the table but i must be doing something wrong with _WD_GetTable and $sBaseElement.

$sElementSelector = "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[11]/div[2]/div/table" ;                  Table
$sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sElementSelector)
ConsoleWrite("Element : " & $sElement & @LF)
$aResult = _WD_GetTable($sSession, $sElement)
_ArrayDisplay($aResult)
Exit
Element : 45892678-1aae-4f91-b275-61b2290a073f
__WD_Post ==> Invalid Expression [11] : HTTP status = 400
_WD_FindElement ==> Invalid Expression [11] : Parameters:   Strategy=xpath   Selector=45892678-1aae-4f91-b275-61b2290a073f   StartNodeID=Default   Multiple=Default   ShadowRoot=Default
_WD_GetTable ==> Invalid Expression [11]

 

I use

$sElement

for

$sBaseElement

like

_WD_GetTable($sSession, $sBaseElement)

Is that wrong? I guess so...

 

Table.png

Link to comment
Share on other sites

So, okay, endgoal.
A 3D array. No problem.

2D=Results of 1 day. 3rd D = every other day.
Pretty simple. Like a book.

So.Now. How do i enter a date into this: "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[6]/div/div/div[4]/div[2]/input"

 

Date2.png

Link to comment
Share on other sites

@Danp2 Hi Dan, YES! That did the trick! Thank you!

$sElementSelector = "/html/body/app-root/div/div/app-consumption/div[3]/div[2]/div[6]/div/div/div[4]/div[2]/input" ;         Date Input
$sElement = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, $sElementSelector)
ConsoleWrite("Element : " & $sElement & @LF)
$sValue = "1.2.2023"
_WD_SetElementValue($sSession, $sElement, $sValue, $_WD_OPTION_Advanced)

 

Link to comment
Share on other sites

Hello, i have another question.

I have this <div> block and i want to delete it.

I just can't find a delete option but only a 'hide' option in

_WD_ElementActionEx($sSession, $sElement, $sCommand)

where $sCommand = 'hide'

<div _ngcontent-spe-c94="" class="col-md-4 d-none d-lg-block">
lalalala...
</div>

The "problem" is more of an cosmetic nature.
In the left block there are only shown customer number at the very top but the rest is empty wasted page space.
In the right block is the chart plus the actual table which iam interested in.

The 'hide' works well with an xpath. But it does not make the right block jump to the left.
If i delete the left div block manually, the ight block fills the whole width of the page as desired.

Thanks!!

 

Link to comment
Share on other sites

Hi @Blaxxun,

I think you have (at least) two options. Either use setAttribute by JavaScript like this:

Local Const $sDivElement = '...'
Local Const $sJavaScript = _
    'var element = arguments[0]; ' & _
    'element.setAttribute("display", "none");'

_WD_ExecuteScript($sSession, $sJavaScript, __WD_JsonElement($sDivElement), Default, Default)

which sets the display attribute of your DIV to none. This should be similar to remove.

 

Or in case it does not work, you can use JavaScript a with element.remove() call, like this:

Local Const $sDivElement = '...'
Local Const $sJavaScript = _
    'var element = arguments[0]; ' & _
    'element.remove();'

_WD_ExecuteScript($sSession, $sJavaScript, __WD_JsonElement($sDivElement), Default, Default)

 

💡 I didn't tested it with AutoIt 😅 . But I did it in the past successfully with Selenium and WebdriverIO .

See _WD_ExecuteScript function and a usage example in wd_demo.au3.

Best regards
Sven

Edited by SOLVE-SMART

Stay innovative!

Spoiler

🌍 Au3Forums

🎲 AutoIt (en) Cheat Sheet

📊 AutoIt limits/defaults

💎 Code Katas: [...] (comming soon)

🎭 Collection of GitHub users with AutoIt projects

🐞 False-Positives

🔮 Me on GitHub

💬 Opinion about new forum sub category

📑 UDF wiki list

✂ VSCode-AutoItSnippets

📑 WebDriver FAQs

👨‍🏫 WebDriver Tutorial (coming soon)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...