Craigslist Section to Array

[ZacUSNYR 4]

expandcollapse popup

#include <IE.au3>
#include <INet.au3>
#include <Array.au3>

Const $CRAIGSLISTURL = "craigslist.org"

$vgm = _CLSectionToArray("albany", "vgm", 1)
If Not @error Then
    _ArrayDisplay($vgm, "Craigslist Section")
Else
    ConsoleWrite("Error!")
EndIf

Func _CLSectionToArray($sCity, $sSection, $iPages = 1)
    Local $id = 0, $sIndex, $sURL_Source
    Local $asURL_Parse, $asURL_Listings[1][2]
    Local $sURL = "http://" & $sCity & Chr(46) & $CRAIGSLISTURL & Chr(47) & $sSection & Chr(47)

    For $i = 0 To $iPages - 1
        If $i >= 1 Then
            $id += 100
            $sIndex = "index" & $id & ".html"
        Else
            $sIndex = "index.html"
        EndIf
        $sURL_Source &= _INetGetSource($sURL & $sIndex)
    Next

    $asURL_Parse = StringRegExp($sURL_Source, '<a href="' & $sURL & '(.*?)</a>', 3)
    If @error Then
        SetError(1)
        Return 0
    EndIf

    For $i = 0 To UBound($asURL_Parse) - 1
        ReDim $asURL_Listings[UBound($asURL_Listings) + 1][2]
        $asSplit = StringSplit($asURL_Parse[$i], '">', 3)
        $asURL_Listings[$i + 1][0] = $asSplit[0]
        $asURL_Listings[$i + 1][1] = $asSplit[1]
    Next
    $asURL_Listings[0][0] = (UBound($asURL_Listings) - 1)
    Return $asURL_Listings
EndFunc   ;==>_CLSectionToArray

Function with example. Simple enough.

URL for Craigslist - go to the section you want to read and look at the URL.

_CLSectionToArray("City", "Section", "Pages - default is 1")

So http://phoenix.craigslist.org/wvl/vgm/ would be

_CLSectionToArray("phoenix", "wvl/vgm")

If you wanted the first few pages

_CLSectionToArray("phoenix", "wvl/vgm", 3)

And it'll return a 2 Dimensional array with the page ID in the first field and the description in the second.

Edit: I only designed and tested it in sections I needed it for, so no promises in other classified sections heh

Edit 2: Doesn't work on city's with sub sections, i'll mess around with that. You just need to search each sub section.

Edited January 29, 2011 by ZacUSNYR

[JohnOne 1,603]

StringRegExp() errors for me.

[ZacUSNYR 4]

I see that using the example I gave (never tested it just chose a city at random lol).

Using Albany, VGM it works fine. Under Phoenix/boston/other large cities it actually sub categorizes.

I'll mess around with that heh

[ZacUSNYR 4]

Been designing these functions to work as declared. The subsections you just need to declare in the section.

http://phoenix.craigslist.org/wvl/vgm/2186282764.html

Would be

_CLSectionToArray("phoenix", "wvl/vgm", 1)

Not as dynamic but it works