Get values from website to execl

[Dizzy]

Hi all,

i've tried to get the prices of the different petrol sorts - but i didn't get them.

As aou can see, there are "Diesel", "Super E10", "Super E5", "SuperPlus" and "Autogas".

--> http://www.clever-tanken.de/tankstelle_details/20205

Can someone help me to get them from this site to a excel sheet?

Thanks

Dizzy

[JohnOne]

Show what you hae tried.

[Dizzy]

Hi JohnOne,
 
i've tried to use the _IE options to get the value.
The best result was to read the whole html to text - but this can't be the right way.
 
I use the Firefox inspector to see which name, ID or ... this belongs to.

I think, that all needed information shoud be found around the <div ID "fuel_price_list" - see "Firefox_Inspector.JPG".

But i didn't know how i should look for this "ID" and how to get to 5 different prices into variables (maybe in one array?).

So i stumple around to get this values - using debug and "try&error" 

I want to read these values (Petrol.jpg) every 30 minutes and store them in a excel sheet.

48 entries / day and over ~ 30 days.

Thanks for helping me.

Dizzy

And : do you know how to change the title? There is a little misspelling there "Excel" ...

#include <IE.au3>
#include <MsgBoxConstants.au3>
#include <Debug.au3>

_DebugSetup()

Local $oIE = _IECreate("http://www.clever-tanken.de/tankstelle_details/20205", 0, 0)
_IELoadWait($oIE)

;Local $sText = _IEBodyReadText($oIE)
Local $iNumFrames

$oFrames = _IEFrameGetCollection ($oIE)
;_DebugReport("$oFrames = " & $oFrames)
$iNumFrames = @extended
;_DebugReport("$iNumFrames = " & $iNumFrames)

For $i = 0 to ($iNumFrames - 1)
            $oFrame = _IEFrameGetCollection ($oIE, $i )
            msgBox(1,  " var type " & VarGetType( $oFrame) ,  @error )
            $oForms  = _IEFormGetCollection ($oFrame)
Next

[water]

When using the Firefox browser you need to use the Firefox UDF as well. The IE UDF only works for the Internet Explorer.

[Dizzy]

Hi water,

i just use Firefox to "inspect" the website-code.

It is regardless wether using Firefox or IE.

Do you have a first step for me?

Thanks

Dizzy

[SmOke_N]

It displays in FF for me, but not in IE 

[SmOke_N]

It still won't show in my IE, probably a version issue, but try this

expandcollapse popup

_myScrape($oIE)

Func _myScrape(ByRef $oObj)
    
    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
    
    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _IETagNameGetCollection($oObj, "DIV")
    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
    
    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next
    
    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
    
    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _IETagNameGetCollection($aItems[$iItem][0], "DIV")
        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next
    
    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]
    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)
    Next
    Return $aRet
EndFunc

[Dizzy]

Hi SmOke_N,

thanks for your help.

Same for me with your code.

I've inserted same consolewrites as checkpoints

expandcollapse popup

#include <IE.au3>
#include <MsgBoxConstants.au3>

Local $oIE = _IECreate("http://www.clever-tanken.de/tankstelle_details/20205", 0, 0)
_IELoadWait($oIE)

_myScrape($oIE)

_IELoadWait($oIE)

Func _myScrape(ByRef $oObj)
ConsoleWrite("In Func" & @crlf)

    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
ConsoleWrite("No Error_1" & @crlf)

    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _IETagNameGetCollection($oObj, "DIV")
ConsoleWrite("Found oDivs : " & $oDivs & @crlf)

    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
ConsoleWrite("No Error_2" & @crlf)

    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
ConsoleWrite("Found aItems : " & $aItems & @crlf)

    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next

    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
ConsoleWrite("No Error_3" & @crlf)

    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _IETagNameGetCollection($aItems[$iItem][0], "DIV")
ConsoleWrite("Found oDivs_2 : " & $iItem & @crlf)

        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next

    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]

    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)

ConsoleWrite("aRet[" & $i &"][0] :" & $aRet[$i][0] & @crlf)
ConsoleWrite("aRet[" & $i &"][1] :" & $aRet[$i][1] & @crlf)

    Next
    Return $aRet
EndFunc

Result:

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:temp_IE_Test6.au3" /UserParams

+>10:10:30 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0 Keyboard:00000407 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64 Environment(Language:0407)
+> SciTEDir => C:Program Files (x86)AutoIt3SciTE UserDir => C:Program Files (x86)AutoIt3SciTEAutoIt3Wrapper
>Running AU3Check (3.3.12.0) from:C:Program Files (x86)AutoIt3 input:C:temp_IE_Test6.au3
+>10:10:30 AU3Check ended.rc:0
>Running:(3.3.12.0):C:Program Files (x86)AutoIt3autoit3.exe "C:temp_IE_Test6.au3"
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
In Func
No Error_1
Found oDivs :
No Error_2
Found aItems :
No Error_3
Found oDivs_2 : 0
Found oDivs_2 : 1
Found oDivs_2 : 2
Found oDivs_2 : 3
Found oDivs_2 : 4
aRet[0][0] :DieselMTS-K Preis
aRet[0][1] :
aRet[1][0] :Super E10MTS-K Preis
aRet[1][1] :
aRet[2][0] :Super E5MTS-K Preis
aRet[2][1] :
aRet[3][0] :SuperPlus 4 ct.
aRet[3][1] :
aRet[4][0] :Autogas
aRet[4][1] :
+>10:10:34 AutoIt3.exe ended.rc:0
+>10:10:34 AutoIt3Wrapper Finished.
>Exit code: 0 Time: 4.4

 

I've tried to translate this to FF - has taken a moment to see that i have to use mozl ...

expandcollapse popup

#include "FF.au3"
;_FFConnect(default,Default,6000)

Local $oFire= _FFStart("http://www.clever-tanken.de/tankstelle_details/20205")

_myScrape($oFire)
_FFQuit()

Func _myScrape(ByRef $oObj)
ConsoleWrite("In Func" & @crlf)

    If Not IsObj($oObj) Then
        Return SetError(1, 0, 0)
    EndIf
ConsoleWrite("No Error_1" & @crlf)

    ; step 1. <div class="fuel_price_entry ng-scope"
    Local $oDivs = _FFObjGet($oObj, "DIV")
ConsoleWrite("Found oDivs : " & $oDivs & @crlf)

    If Not IsObj($oDivs) Then
        Return SetError(2, 0, 0)
    EndIf
ConsoleWrite("No Error_2" & @crlf)

    ; find matching class objeects
    Local $aItems[0][3], $iDim = 0
ConsoleWrite("Found aItems : " & $aItems & @crlf)

    For $oItm In $oDivs
        If String($oItm.className) = "fuel_price_entry ng-scope" Then
            ReDim $aItems[$iDim + 1][3]
            $aItems[$iDim][0] = $oItm
            $iDim += 1
        EndIf
    Next

    If $iDim = 0 Then
        Return SetError(3, 0, 0) ; nothing found
    EndIf
ConsoleWrite("No Error_3" & @crlf)

    ; we have an array of container objects now
    For $iItem = 0 To $iDim - 1
        ; we need to know the type of fuel
        $oDivs = _FFObjGet($aItems[$iItem][0], "DIV")
ConsoleWrite("Found oDivs_2 : " & $iItem & @crlf)

        For $oItm In $oDivs
            Switch String($oItm.className)
                Case "fuel_type"
                    $aItems[$iItem][1] = StringRegExpReplace($oItm.innerText, "(.+?)(\v*.*?)", "$1")
                Case "price_field price_dirty"
                    $aItems[$iItem][2] = String($oItm.innerText)
            EndSwitch
        Next
    Next

    ; reformat return array to remove objects found
    Local $aRet[$iDim][2]

    For $i = 0 To $iDim - 1
        $aRet[$i][0] = $aItems[$i][1] ; type of fuel
        $aRet[$i][1] = $aItems[$i][2] ; price field/dirty (that just sounds bad)

ConsoleWrite("aRet[" & $i &"][0] :" & $aRet[$i][0] & @crlf)
ConsoleWrite("aRet[" & $i &"][1] :" & $aRet[$i][1] & @crlf)

    Next
    Return $aRet
EndFunc

Result:

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:temp_IE_Test6_FF.au3" /UserParams

+>10:04:33 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0 Keyboard:00000407 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64 Environment(Language:0407)
+> SciTEDir => C:Program Files (x86)AutoIt3SciTE UserDir => C:Program Files (x86)AutoIt3SciTEAutoIt3Wrapper
>Running AU3Check (3.3.12.0) from:C:Program Files (x86)AutoIt3 input:C:temp_IE_Test6_FF.au3
+>10:04:33 AU3Check ended.rc:0
>Running:(3.3.12.0):C:Program Files (x86)AutoIt3autoit3.exe "C:temp_IE_Test6_FF.au3"
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
__FFStartProcess: ""C:Program Files (x86)Mozilla Firefoxfirefox.exe" -new-window "http://www.clever-tanken.de/tankstelle_details/20205" "-repl 4242 "
_FFConnect: OS: WIN_7 WIN32_NT 7601 Service Pack 1
_FFConnect: AutoIt: 3.3.12.0
_FFConnect: FF.au3: 0.6.0.1b-14
_FFConnect: IP: 127.0.0.1
_FFConnect: Port: 4242
_FFConnect: Delay: 2ms
_FFConnect: Socket: 528
_FFConnect: Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
__FFSendJavaScripts: Sending functions to FireFox .......... done
_FFLoadWait: . loaded in 10ms
[object HTMLDocument] - {location: {...}, af_6098183008570310744: {...}, location: {...}, getElementsByName: function() {...}, getItems: function() {...}, open: function() {...}, close: function() {...}, ...}
In Func
__FFQuit: Closing FireFox ...
+>10:04:37 AutoIt3.exe ended.rc:0
+>10:04:37 AutoIt3Wrapper Finished.
>Exit code: 0 Time: 4.829

Whats wrong with this code?

You wrote, you'll get it on FF - can you show me how?

I'm not fixed on IE.

Thanks

Dizzy

Edited December 29, 2014 by Dizzy

[Danp2]

For starters, you need to switch from using IsObj() to checking @error because the FF UDF doesn't return objects like the IE UDF.

[SmOke_N]

I never ran the code, I just wrote it based on what I saw in the javascript spy tool with FF, but I've never used the FF.au3 stuff.

I can't get the data to render in IE at all.

So the fact that it's close, just means you need to check the class object I'm searching for the dollar amount and change them up with one of the others until it works.