Sign in to follow this  
Followers 0
Dizzy

Get values from website to execl

10 posts in this topic

Hi all,

i've tried to get the prices of the different petrol sorts - but i didn't get them.

As aou can see, there are "Diesel", "Super E10", "Super E5", "SuperPlus" and "Autogas".

--> http://www.clever-tanken.de/tankstelle_details/20205

Can someone help me to get them from this site to a excel sheet?

Thanks

Dizzy

Share this post


Link to post
Share on other sites



Hi JohnOne,
 
i've tried to use the _IE options to get the value.
The best result was to read the whole html to text - but this can't be the right way.
 
I use the Firefox inspector to see which name, ID or ... this belongs to.

I think, that all needed information shoud be found around the <div ID "fuel_price_list" - see "Firefox_Inspector.JPG".

But i didn't know how i should look for this "ID" and how to get to 5 different prices into variables (maybe in one array?).

So i stumple around to get this values - using debug and "try&error"  :

I want to read these values (Petrol.jpg) every 30 minutes and store them in a excel sheet.

48 entries / day and over ~ 30 days.

Thanks for helping me.

Dizzy

And : do you know how to change the title? There is a little misspelling there "Excel" ... :ermm:

#include <IE.au3>
#include <MsgBoxConstants.au3>
#include <Debug.au3>

_DebugSetup()

Local $oIE = _IECreate("http://www.clever-tanken.de/tankstelle_details/20205", 0, 0)
_IELoadWait($oIE)

;Local $sText = _IEBodyReadText($oIE)
Local $iNumFrames

$oFrames = _IEFrameGetCollection ($oIE)
;_DebugReport("$oFrames = " & $oFrames)
$iNumFrames = @extended
;_DebugReport("$iNumFrames = " & $iNumFrames)

For $i = 0 to ($iNumFrames - 1)
            $oFrame = _IEFrameGetCollection ($oIE, $i )
            msgBox(1,  " var type " & VarGetType( $oFrame) ,  @error )
            $oForms  = _IEFormGetCollection ($oFrame)
Next

post-83-0-66161900-1419771502.jpg

post-83-0-08008300-1419771519_thumb.jpg

Share this post


Link to post
Share on other sites

When using the Firefox browser you need to use the Firefox UDF as well. The IE UDF only works for the Internet Explorer.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Hi water,

i just use Firefox to "inspect" the website-code.

It is regardless wether using Firefox or IE.

Do you have a first step for me?

Thanks

Dizzy

Share this post


Link to post
Share on other sites

It displays in FF for me, but not in IE  :huh:


Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

It still won't show in my IE, probably a version issue, but try this

_myScrape($oIE)

Func _myScrape(ByRef $oObj)
    
    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
    
    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _IETagNameGetCollection($oObj, "DIV")
    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
    
    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next
    
    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
    
    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _IETagNameGetCollection($aItems[$iItem][0], "DIV")
        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next
    
    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]
    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)
    Next
    Return $aRet
EndFunc

Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Hi SmOke_N,

thanks for your help.

Same for me with your code.

I've inserted same consolewrites as checkpoints

#include <IE.au3>
#include <MsgBoxConstants.au3>

Local $oIE = _IECreate("http://www.clever-tanken.de/tankstelle_details/20205", 0, 0)
_IELoadWait($oIE)

_myScrape($oIE)

_IELoadWait($oIE)

Func _myScrape(ByRef $oObj)
ConsoleWrite("In Func" & @crlf)

    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
ConsoleWrite("No Error_1" & @crlf)

    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _IETagNameGetCollection($oObj, "DIV")
ConsoleWrite("Found oDivs : " & $oDivs & @crlf)

    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
ConsoleWrite("No Error_2" & @crlf)

    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
ConsoleWrite("Found aItems : " & $aItems & @crlf)

    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next

    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
ConsoleWrite("No Error_3" & @crlf)

    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _IETagNameGetCollection($aItems[$iItem][0], "DIV")
ConsoleWrite("Found oDivs_2 : " & $iItem & @crlf)

        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next

    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]

    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)

ConsoleWrite("aRet[" & $i &"][0] :" & $aRet[$i][0] & @crlf)
ConsoleWrite("aRet[" & $i &"][1] :" & $aRet[$i][1] & @crlf)

    Next
    Return $aRet
EndFunc

Result:

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:temp_IE_Test6.au3" /UserParams


+>10:10:30 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0 Keyboard:00000407 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64 Environment(Language:0407)
+> SciTEDir => C:Program Files (x86)AutoIt3SciTE UserDir => C:Program Files (x86)AutoIt3SciTEAutoIt3Wrapper
>Running AU3Check (3.3.12.0) from:C:Program Files (x86)AutoIt3 input:C:temp_IE_Test6.au3
+>10:10:30 AU3Check ended.rc:0
>Running:(3.3.12.0):C:Program Files (x86)AutoIt3autoit3.exe "C:temp_IE_Test6.au3"
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
In Func
No Error_1
Found oDivs :
No Error_2
Found aItems :
No Error_3
Found oDivs_2 : 0
Found oDivs_2 : 1
Found oDivs_2 : 2
Found oDivs_2 : 3
Found oDivs_2 : 4
aRet[0][0] :DieselMTS-K Preis
aRet[0][1] :
aRet[1][0] :Super E10MTS-K Preis
aRet[1][1] :
aRet[2][0] :Super E5MTS-K Preis
aRet[2][1] :
aRet[3][0] :SuperPlus 4 ct.
aRet[3][1] :
aRet[4][0] :Autogas
aRet[4][1] :
+>10:10:34 AutoIt3.exe ended.rc:0
+>10:10:34 AutoIt3Wrapper Finished.
>Exit code: 0 Time: 4.4

 

I've tried to translate this to FF - has taken a moment to see that i have to use mozl ... :ermm:

#include "FF.au3"
;_FFConnect(default,Default,6000)

Local $oFire= _FFStart("http://www.clever-tanken.de/tankstelle_details/20205")

_myScrape($oFire)
_FFQuit()

Func _myScrape(ByRef $oObj)
ConsoleWrite("In Func" & @crlf)

    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
ConsoleWrite("No Error_1" & @crlf)

    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _FFObjGet($oObj, "DIV")
ConsoleWrite("Found oDivs : " & $oDivs & @crlf)

    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
ConsoleWrite("No Error_2" & @crlf)

    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
ConsoleWrite("Found aItems : " & $aItems & @crlf)

    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next

    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
ConsoleWrite("No Error_3" & @crlf)

    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _FFObjGet($aItems[$iItem][0], "DIV")
ConsoleWrite("Found oDivs_2 : " & $iItem & @crlf)

        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next

    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]

    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)

ConsoleWrite("aRet[" & $i &"][0] :" & $aRet[$i][0] & @crlf)
ConsoleWrite("aRet[" & $i &"][1] :" & $aRet[$i][1] & @crlf)

    Next
    Return $aRet
EndFunc

Result:

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:temp_IE_Test6_FF.au3" /UserParams


+>10:04:33 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0 Keyboard:00000407 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64 Environment(Language:0407)
+> SciTEDir => C:Program Files (x86)AutoIt3SciTE UserDir => C:Program Files (x86)AutoIt3SciTEAutoIt3Wrapper
>Running AU3Check (3.3.12.0) from:C:Program Files (x86)AutoIt3 input:C:temp_IE_Test6_FF.au3
+>10:04:33 AU3Check ended.rc:0
>Running:(3.3.12.0):C:Program Files (x86)AutoIt3autoit3.exe "C:temp_IE_Test6_FF.au3"
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
__FFStartProcess: ""C:Program Files (x86)Mozilla Firefoxfirefox.exe" -new-window "http://www.clever-tanken.de/tankstelle_details/20205" "-repl 4242 "
_FFConnect: OS: WIN_7 WIN32_NT 7601 Service Pack 1
_FFConnect: AutoIt: 3.3.12.0
_FFConnect: FF.au3: 0.6.0.1b-14
_FFConnect: IP: 127.0.0.1
_FFConnect: Port: 4242
_FFConnect: Delay: 2ms
_FFConnect: Socket: 528
_FFConnect: Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
__FFSendJavaScripts: Sending functions to FireFox .......... done
_FFLoadWait: . loaded in 10ms
[object HTMLDocument] - {location: {...}, af_6098183008570310744: {...}, location: {...}, getElementsByName: function() {...}, getItems: function() {...}, open: function() {...}, close: function() {...}, ...}
In Func
__FFQuit: Closing FireFox ...
+>10:04:37 AutoIt3.exe ended.rc:0
+>10:04:37 AutoIt3Wrapper Finished.
>Exit code: 0 Time: 4.829

Whats wrong with this code?

You wrote, you'll get it on FF - can you show me how? o:)

I'm not fixed on IE.

Thanks

Dizzy

Edited by Dizzy

Share this post


Link to post
Share on other sites

For starters, you need to switch from using IsObj() to checking @error because the FF UDF doesn't return objects like the IE UDF.

Share this post


Link to post
Share on other sites

I never ran the code, I just wrote it based on what I saw in the javascript spy tool with FF, but I've never used the FF.au3 stuff.

I can't get the data to render in IE at all.

So the fact that it's close, just means you need to check the class object I'm searching for the dollar amount and change them up with one of the others until it works.


Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0