Detecting New Posts

ericnail · September 30, 2011

I am creating a script to check craigslist for new posts based on my search parameters.

I want it to check for new posts and then if any are new then run a function.

Currently I have this:

#include <INet.au3>
Global $postTitle, $postDate, $postPrice, $postLink, $postNumber, $updateTimer, $count = 0
Global $intervalTime = 900000 ;15 Minutes
Global $stackQueue[1] = [0] ;Queue for found ads
Global $stackOverflow[1] = [0] ;Queue for matched ads
Dim $iStack[1][5]
$iStack[0][0] = '<font size=5>Title</font>'
$iStack[0][1] = '<font size=5>Price</font>'
$iStack[0][2] = '<font size=5>Date</font>'
$iStack[0][3] = '<font size=5>Phone Number</font>'
$iStack[0][4] = '<font size=5>URL</font>'
$check_numbers = 0
 
 
GetSearchResults("iPhone 4", "150", "250", 1)
Func GetSearchResults($a, $b, $c, $d = 1)
fillStack($a, $b, $c)
$html = _ArrayToHTML($iStack)
$html = '<html>' & @CRLF & $html & '</html>'
$html = StringReplace($html, "Title", "<b>Title</b>")
$html = StringReplace($html, "Date", "<b>Date</b>")
$html = StringReplace($html, "Price", "<b>Price</b>")
$html = StringReplace($html, "URL", "<b>URL</b>")
$html = StringReplace($html, 'http://atlanta.craigslist.org/', '<a href="http://atlanta.craigslist.org/')
$html = StringReplace($html, '.html', '.html">Link</a>')
If $d = 1 Then FileSaveAndOpen("index.html", $html)
EndFunc
Func fillStack($iQuery, $iMin, $iMax)
 
Local $iData[4][1]
$iQuery = StringReplace($iQuery, " ", "+")
$iURL = "http://atlanta.craigslist.org/search/sss?maxAsk="&$iMax&"&amp;minAsk="&$iMin&"&amp;query="&$iQuery&"&amp;srchType=A"
$source = _INetGetSource($iURL)
$postTitle = StringRegExp($source, '<a href="http://atlanta.craigslist.org/[^0-9][^0-9][^0-9]/[^0-9][^0-9][^0-9]/...........html">(.*?)-</a>', 3)
$postDate = StringRegExp($source, ";</span>\n   [^0-9][^0-9][^0-9] [0-9][0-9] - <a", 3)
$postPrice = StringRegExp($source, '\n[^a-z][^a-z][^a-z](.*?)href="/[^0-9][^0-9][^0-9]/"', 3)
$postLink = StringRegExp($source, '<a href="http://atlanta.craigslist.org/[^0-9][^0-9][^0-9]/[^0-9][^0-9][^0-9]/...........html">', 3)
ReDim $iData[5][UBound($postTitle)]
 
For $i = 0 to UBound($postTitle) -1 Step 1
  $iData[0][$i] = StringTrimRight(StringTrimLeft($postTitle[$i], 0), 0)
Next
For $i = 0 to UBound($postDate) -1 Step 1
  $iData[1][$i] = StringTrimRight(StringTrimLeft($postDate[$i], 17), 5)
Next
For $i = 0 to UBound($postPrice) -1 Step 1
  $iData[2][$i] = "$"&StringTrimLeft(StringLeft($postPrice[$i], 5), 2)
Next
 
For $i = 0 to UBound($postLink) -1 Step 1
  $iData[4][$i] = StringTrimRight(StringTrimLeft($postLink[$i], 9), 2)
Next
 
Local $iTemp_a = UBound($postPrice)-1
Local $iTemp_b = UBound($postTitle)-1
If $check_numbers Then ProgressOn("Eric's Craigslist App", "Collecting Phone Numbers...")
For $i = 0 to $iTemp_b Step 1
  If $check_numbers Then ProgressSet(($i / $iTemp_b) * 100, Round(($i / $iTemp_b) * 100) & "%")
  If $i > $iTemp_a Then ExitLoop
  If StringInStr($postTitle, "want") Then ContinueLoop
  If StringInStr($postTitle, "trade") Then ContinueLoop
  If $postTitle = "" Then ContinueLoop
  AddToStack($iStack, $iData[0][$i], $iData[2][$i], $iData[1][$i], $iData[4][$i], $iQuery)
Next
If $check_numbers Then ProgressOff()
 
Global $Count = 0
 
EndFunc
Func AddToStack(ByRef $iStack, $iTitle, $iPrice, $iDate, $iLink, $iQuery)
 
Local $iNumber = "-"
 
$iQuery = StringReplace($iQuery, "+", " ")
 
If Not StringInStr($iTitle, $iQuery) Then Return
 
;~  If $History Then
;~   If ManageArchive("Check", $iLink) Then Return
;~   ManageArchive("Write", $iLink)
;~  EndIf
 
$count+= 1
 
ReDim $iStack[$count+1][5]
 
If $check_numbers Then
  $src = _INetGetSource($iLink)
  $iTemp = StringRegExp($src, "[0-9][0-9][0-9][^0-9][0-9][0-9][0-9][^0-9][0-9][0-9][0-9][0-9]", 3)
  If IsArray($iTemp) Then
   $iTemp = StringRegExpReplace($iTemp[0], '(\d{3}).(\d{3}).(\d{4})', ' $1-$2-$3 ')
   $iNumber = $iTemp
  Else
   $iTemp = StringRegExpReplace($iTemp, '(\d{3}).(\d{3}).(\d{4})', ' $1-$2-$3 ')
   $iNumber = $iTemp
  EndIf
 
  If $iNumber = 1 Then $iNumber = "-"
EndIf
 
$iStack[$count][0] = $iTitle
$iStack[$count][1] = $iPrice
$iStack[$count][2] = $iDate
$iStack[$count][3] = $iNumber
$iStack[$count][4] = $iLink
 
EndFunc
Func _ArrayToHTML(Const ByRef $avArray, $attrib = 'border="2"', $iStart = 0, $iEnd = 0)
If Not IsArray($avArray) Then Return SetError(1, 0, "")
If UBound($avArray, 0) <> 2 Then Return SetError(2, 0, "")
Local $sResult, $iUBound = UBound($avArray) - 1
Local $row, $sDelimCol = "</td>" & @CRLF, $sDelimRow = '</tr>' & @CRLF
; Bounds checking
If $iEnd < 1 Or $iEnd > $iUBound Then $iEnd = $iUBound
If $iStart < 0 Then $iStart = 0
If $iStart > $iEnd Then Return SetError(3, 0, "")
$sResult = '<table ' & $attrib & '>' & @CRLF
 
; Combine
For $i = $iStart To $iEnd ; rows
    $row = '<tr>' & @CRLF
    For $j = 0 To UBound($avArray,2) - 1 ; columns
        If $j <> 0 Then
   $row &= '<td><font size=2>' & '<center>' & $avArray[$i][$j] & $sDelimCol & '</center>' & '</font>'
  Else
   $row &= '<td><font size=2>' & $avArray[$i][$j] & $sDelimCol & '</font>'
  EndIf
Next
    $sResult &= $row & $sDelimRow
Next
 
Return $sResult & '</table>' & @CRLF
EndFunc
 
Func FileSaveAndOpen($file, $content)
$file = @TempDir & '\' & $file
FileDelete($file)
FileWrite($file, $content)
ShellExecute($file)
EndFunc
 
Func ManageArchive($a, $b)
 
If $a = "Write" Then
  IniWrite("archive.ini", "archived", "viewed", IniRead("archive.ini", "archived", "viewed", "") & "|" & $b)
EndIf
 
If $a = "Remove" Then
  IniWrite("archive.ini", "archived", "viewed", StringReplace(IniRead("archive.ini", "archived", "viewed", ""), $b & ",", ""))
EndIf
 
If $a = "Check" Then
  $iTemp = IniRead("archive.ini", "archived", "viewed", "")
  $iTemp = StringSplit($iTemp, "|")
  For $i = 1 to $iTemp[0] Step 1
   If $b = $iTemp[$i] Then Return True
  Next
 
  Return False
EndIf
EndFunc

Which will search for the terms and find specified info and then make and open an HTML File. I want to be able to make it search for only new posts.

I cannot come up with a way to do this in my head. Maybe there is a simple solution but I have yet to figure it out.

Any help here would be great!

Thanks,

Eric

ericnail · September 30, 2011

Anyone?

Beege · September 30, 2011

When you say "new" do you mean timeline (it was just posted 10 min ago) new or your script has not ever seen this post before "new"?

Edited September 30, 2011 by Beege

ericnail · September 30, 2011

When you say "new" do you mean timeline (it was just posted 10 min ago) new or your script has not ever seen this post before "new"?

Hey,

I mean Timeline.

Beege · October 1, 2011

I looked of the html source for the link in the example and dont see any dates on the search page. I dont see any other way of doing it other than downloading the source of the links you find, then pulling the dates of them. Btw nice script. I like the html table thing you got going. :graduated:

Edit: Looking at the source's of the links deeper, it should really be cake compared to the work you have already done. Of all the links I looked at theres only one date on the page and that date is the same format for all of them. your clearly good with regexpressions so I dont think it will be at all. Unless im missing something else your needing.

Edited October 1, 2011 by Beege

Sign In

Detecting New Posts

Recommended Posts

ericnail

Link to comment

Share on other sites

ericnail

Link to comment

Share on other sites

Beege

Link to comment

Share on other sites

ericnail

Link to comment

Share on other sites

Beege

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta