Sign in to follow this  
Followers 0
ZacUSNYR

Craigslist Section to Array

4 posts in this topic

#1 ·  Posted (edited)

#include <IE.au3>
#include <INet.au3>
#include <Array.au3>

Const $CRAIGSLISTURL = "craigslist.org"

$vgm = _CLSectionToArray("albany", "vgm", 1)
If Not @error Then
    _ArrayDisplay($vgm, "Craigslist Section")
Else
    ConsoleWrite("Error!")
EndIf

Func _CLSectionToArray($sCity, $sSection, $iPages = 1)
    Local $id = 0, $sIndex, $sURL_Source
    Local $asURL_Parse, $asURL_Listings[1][2]
    Local $sURL = "http://" & $sCity & Chr(46) & $CRAIGSLISTURL & Chr(47) & $sSection & Chr(47)

    For $i = 0 To $iPages - 1
        If $i >= 1 Then
            $id += 100
            $sIndex = "index" & $id & ".html"
        Else
            $sIndex = "index.html"
        EndIf
        $sURL_Source &= _INetGetSource($sURL & $sIndex)
    Next

    $asURL_Parse = StringRegExp($sURL_Source, '<a href="' & $sURL & '(.*?)</a>', 3)
    If @error Then
        SetError(1)
        Return 0
    EndIf

    For $i = 0 To UBound($asURL_Parse) - 1
        ReDim $asURL_Listings[UBound($asURL_Listings) + 1][2]
        $asSplit = StringSplit($asURL_Parse[$i], '">', 3)
        $asURL_Listings[$i + 1][0] = $asSplit[0]
        $asURL_Listings[$i + 1][1] = $asSplit[1]
    Next
    $asURL_Listings[0][0] = (UBound($asURL_Listings) - 1)
    Return $asURL_Listings
EndFunc   ;==>_CLSectionToArray

Function with example. Simple enough.

URL for Craigslist - go to the section you want to read and look at the URL.

_CLSectionToArray("City", "Section", "Pages - default is 1")

So http://phoenix.craigslist.org/wvl/vgm/ would be

_CLSectionToArray("phoenix", "wvl/vgm")

If you wanted the first few pages

_CLSectionToArray("phoenix", "wvl/vgm", 3)

And it'll return a 2 Dimensional array with the page ID in the first field and the description in the second.

Edit: I only designed and tested it in sections I needed it for, so no promises in other classified sections heh

Edit 2: Doesn't work on city's with sub sections, i'll mess around with that. You just need to search each sub section.

Edited by ZacUSNYR

Share this post


Link to post
Share on other sites



I see that using the example I gave (never tested it just chose a city at random lol).

Using Albany, VGM it works fine. Under Phoenix/boston/other large cities it actually sub categorizes.

I'll mess around with that heh

Share this post


Link to post
Share on other sites

Been designing these functions to work as declared. The subsections you just need to declare in the section.

http://phoenix.craigslist.org/wvl/vgm/2186282764.html

Would be

_CLSectionToArray("phoenix", "wvl/vgm", 1)

Not as dynamic but it works :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0