Jump to content

pulling select data from a webpage


Recommended Posts

Third day writing script so, I'm still mostly stumbling around but I'm happy. I haven't done any programing since I was a kid but I'm getting reminded of how much fun it is. I guess I'm a dork but I like using logic.

Anyway, I've been trying to pull as much info from searching past posts and reading help files but I'm stuck.

I'm trying to get the program to look at select information from a page and act on that either by clicking on a link or by moving on to the next item on the page. Specifically, I'm looking at a typical page on Amazon. I want the program get one of the books on the page (starting with the first) and get the price for that book. (The specific page and books that will be returned will change depending on search so any of the "get by name" functions won't work.)

If someone can point me in the direction of a help file, sample script or whatever, I'd greatly appreciate.

Thanks.

Link to comment
Share on other sites

Well first I must say that for your first week of scripting you are attempting a very complex task. You'll probaly need to study this at length before you grasp all of the techniques used. It was an interesting exercise however, so I will show you how it can be done. To figure this out I used the IE Developer Toolbar (recently upgraded by the way and much easier to use) - see my sig. The solution requires knowledge of AutoIt, IE.au3 and the DOM/HTML.

#include <IE.au3>
$sURL = "http://www.amazon.com/s/ref=nb_ss_b/102-5993488-0177751?url=search-alias%3Dstripbooks&field-keywords=mystery&Go.x=0&Go.y=0&Go=Go"
$tryAttach = True
$oIE = _IECreate($sURL, $tryAttach)

$oTables = _IETableGetCollection($oIE)
For $oTable in $oTables
    If String($oTable.className) = "searchresults" Then
        $oTDs = _IETagnameGetCollection($oTable, "td")
        For $oTD in $oTDs
            If String($oTD.className) = "searchitem" Then
                $oSpans = _IETagNameGetCollection($oTD, "span")
                For $oSpan in $oSpans
                    Switch String($oSpan.className)
                        Case "srTitle"
                            ConsoleWrite("Title:      " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "listprice"
                            ConsoleWrite("ListPrice:  " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "saleprice"
                            ConsoleWrite("SalePrice:  " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "sr_price"
                            ConsoleWrite("SRPrice:    " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "otherprice"
                            ConsoleWrite("OtherPrice: " & _IEPropertyGet($oSpan, "innertext") & @CR)
                    EndSwitch
                Next
                ConsoleWrite("--------------------------------------------" & @CR)
            EndIf
        Next
    EndIf   
Next

The output generated:

Title:    The Best American Mystery Stories 2006 (The Best American Series (TM))
ListPrice:  $14.00
SalePrice:  $11.20
OtherPrice: $8.52
--------------------------------------------
Title:    On What Grounds (Coffeehouse Mysteries)
SRPrice:    $6.99
OtherPrice: $3.27
--------------------------------------------
Title:    Murder Most Frothy: A Coffeehouse Mystery (Coffeehouse Mysteries)
SRPrice:    $6.99
OtherPrice: $3.42
--------------------------------------------
Title:    Mysteries of the Middle Ages: The Rise of Feminism, Science, and Art from the Cults of Catholic Europe
ListPrice:  $32.50
SalePrice:  $19.11
OtherPrice: $17.50
--------------------------------------------
Title:    The Mystery of Capital: Why Capitalism Triumphs in the West and Fails Everywhere Else
ListPrice:  $16.95
SalePrice:  $11.53
OtherPrice: $9.94
--------------------------------------------
Title:    The Mystery Method: How to Get Beautiful Women Into Bed
ListPrice:  $19.95
SalePrice:  $14.16
--------------------------------------------
Title:    The Mystery Guest: An Account
ListPrice:  $18.00
SalePrice:  $12.24
OtherPrice: $10.79
--------------------------------------------
Title:    The Mysteries of Harris Burdick
ListPrice:  $24.95
SalePrice:  $17.22
OtherPrice: $15.77
--------------------------------------------
Title:    The Jesus Mysteries: Was the "Original Jesus" a Pagan God?
ListPrice:  $14.00
SalePrice:  $11.62
OtherPrice: $8.24
--------------------------------------------
Title:    The Missing Mitten Mystery (Picture Puffin Books (Paperback))
SRPrice:    $6.99
OtherPrice: $3.40
--------------------------------------------
Title:    Two-minute Mysteries (Apple Paperbacks)
SRPrice:    $4.99
OtherPrice: $0.10
--------------------------------------------
Title:    Official Nintendo Pokémon Mystery Dungeon: Blue Rescue Team/Red Rescue Team Player's Guide
SRPrice:    $14.99
OtherPrice: $8.87
--------------------------------------------

Dale

Edited by DaleHohm

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Link to comment
Share on other sites

Well first I must say that for your first week of scripting you are attempting a very complex task. You'll probaly need to study this at length before you grasp all of the techniques used. It was an interesting exercise however, so I will show you how it can be done. To figure this out I used the IE Developer Toolbar (recently upgraded by the way and much easier to use) - see my sig. The solution requires knowledge of AutoIt, IE.au3 and the DOM/HTML.

#include <IE.au3>
$sURL = "http://www.amazon.com/s/ref=nb_ss_b/102-5993488-0177751?url=search-alias%3Dstripbooks&field-keywords=mystery&Go.x=0&Go.y=0&Go=Go"
$tryAttach = True
$oIE = _IECreate($sURL, $tryAttach)

$oTables = _IETableGetCollection($oIE)
For $oTable in $oTables
    If String($oTable.className) = "n2" Then
        $oTDs = _IETagnameGetCollection($oTable, "td")
        For $oTD in $oTDs
            If String($oTD.className) = "searchitem" Then
                $oSpans = _IETagNameGetCollection($oTD, "span")
                For $oSpan in $oSpans
                    Switch String($oSpan.className)
                        Case "srTitle"
                            ConsoleWrite("Title:      " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "listprice"
                            ConsoleWrite("ListPrice:  " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "saleprice"
                            ConsoleWrite("SalePrice:  " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "sr_price"
                            ConsoleWrite("SRPrice:    " & _IEPropertyGet($oSpan, "innertext") & @CR)
                        Case "otherprice"
                            ConsoleWrite("OtherPrice: " & _IEPropertyGet($oSpan, "innertext") & @CR)
                    EndSwitch
                Next
                ConsoleWrite("--------------------------------------------" & @CR)
            EndIf
        Next
    EndIf   
Next

The output generated:

Title:    The Best American Mystery Stories 2006 (The Best American Series (TM))
ListPrice:  $14.00
SalePrice:  $11.20
OtherPrice: $8.52
--------------------------------------------
Title:    On What Grounds (Coffeehouse Mysteries)
SRPrice:    $6.99
OtherPrice: $3.27
--------------------------------------------
Title:    Murder Most Frothy: A Coffeehouse Mystery (Coffeehouse Mysteries)
SRPrice:    $6.99
OtherPrice: $3.42
--------------------------------------------
Title:    Mysteries of the Middle Ages: The Rise of Feminism, Science, and Art from the Cults of Catholic Europe
ListPrice:  $32.50
SalePrice:  $19.11
OtherPrice: $17.50
--------------------------------------------
Title:    The Mystery of Capital: Why Capitalism Triumphs in the West and Fails Everywhere Else
ListPrice:  $16.95
SalePrice:  $11.53
OtherPrice: $9.94
--------------------------------------------
Title:    The Mystery Method: How to Get Beautiful Women Into Bed
ListPrice:  $19.95
SalePrice:  $14.16
--------------------------------------------
Title:    The Mystery Guest: An Account
ListPrice:  $18.00
SalePrice:  $12.24
OtherPrice: $10.79
--------------------------------------------
Title:    The Mysteries of Harris Burdick
ListPrice:  $24.95
SalePrice:  $17.22
OtherPrice: $15.77
--------------------------------------------
Title:    The Jesus Mysteries: Was the "Original Jesus" a Pagan God?
ListPrice:  $14.00
SalePrice:  $11.62
OtherPrice: $8.24
--------------------------------------------
Title:    The Missing Mitten Mystery (Picture Puffin Books (Paperback))
SRPrice:    $6.99
OtherPrice: $3.40
--------------------------------------------
Title:    Two-minute Mysteries (Apple Paperbacks)
SRPrice:    $4.99
OtherPrice: $0.10
--------------------------------------------
Title:    Official Nintendo Pokémon Mystery Dungeon: Blue Rescue Team/Red Rescue Team Player's Guide
SRPrice:    $14.99
OtherPrice: $8.87
--------------------------------------------

Dale

Thanks so much for this. I honestly am almost completely confused about the code but this is a great starting point for me to start checking things out in the help file and I'll go from there.

Again, I really appreciate your generous help. Thanks!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...