Jump to content

I want to create an automation script for searching craigslist ads

phatzilla

By phatzilla
in AutoIt General Help and Support

 Share

Recommended Posts

DirtDBaK Universalist

DirtDBaK

Posted
Posted

See if this whet's your whistle with IE.au3

Make it work invisibly by setting the visiblilty parameter to _IECreate to FALSE.

Run this to see how easily you can get the link text and href's

#include <IE.au3>
#include <Array.au3>

$oIE = _IECreate("http://toronto.craigslist.org/ele/")

$oPs = _IETagNameGetCollection($oIE, "p")
$cntPs = @extended

Local $aLinkInfo[$cntPs + 1][3]

$aLinkInfo[0][0] = "Index"
$aLinkInfo[0][1] = "Link Text"
$aLinkInfo[0][2] = "href"

$cnt = 1
For $oP in $oPs
    $oLink = _IETagNameGetCollection($oP, "a", 0)
    $aLinkInfo[$cnt][0] = $cnt - 1
    $aLinkInfo[$cnt][1] = $oLink.innerText
    $aLinkInfo[$cnt][2] = $oLink.href
    $cnt +=1
Next

_ArrayDisplay($aLinkInfo, "LinkInfo")

Dale

Ahh Superb, you've saved me two hours of parsing!

Works better than i could have imaged! Awsome

[center][/center]

  • Replies 51
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

DaleHohm Universalist

DaleHohm

Posted
Posted

Take a look at the function documentation and run the associated examples in the helpfile and ask questions if it doesn't make sense to you.

Dale

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

DirtDBaK Universalist

DirtDBaK

Posted
Posted

Take a look at the function documentation and run the associated examples in the helpfile and ask questions if it doesn't make sense to you.

Dale

Well i've wasted a lot of time wirting a similar routine thats about 350 lines longer.... Wow am i a fool..

[center][/center]

DirtDBaK Universalist

DirtDBaK

Posted
Posted

Ok Almost finished it finds the matches from your criteria, now i need to make the results.ini be a html file...

Hows the smpt file coming along??? Ill pass cmd line parameter it will be the dir and file name of the html file... hopefully you could handle the rest...

[center][/center]

DaleHohm Universalist

DaleHohm

Posted
Posted

Now you understand my first post :)

Yeah, when I saw the response to your suggestion was "Thats not as cool as what i want too do"... I decided we had a "learning opportunity"... ;)

Dale

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

DirtDBaK Universalist

DirtDBaK

Posted
Posted

Done and ready!!!!

2 problems.... it doesn't do this in the background and it doesn't delete the IE window...

shame on me, not too sure how...

#include <IE.au3>
#include <Array.au3>

While 1


$oIE = _IECreate("http://toronto.craigslist.org/ele/") ;not sure how to hide


$oPs = _IETagNameGetCollection($oIE, "p")
$cntPs = @extended

Local $aLinkInfo[$cntPs + 1][3]

$aLinkInfo[0][0] = "Index"
$aLinkInfo[0][1] = "Link Text"
$aLinkInfo[0][2] = "href"

$cnt = 1
For $oP in $oPs
    $oLink = _IETagNameGetCollection($oP, "a", 0)
    $aLinkInfo[$cnt][0] = $cnt - 1
    $aLinkInfo[$cnt][1] = $oLink.innerText
    $aLinkInfo[$cnt][2] = $oLink.href
    $cnt +=1
Next

;_ArrayDisplay($aLinkInfo, "LinkInfo")
$y = 0
$terms = IniReadSection( @SCRIPTDIR & "\Settings.ini", "Terms" )  ;compare and find results
For $i = 1 to $terms[0][0] step 1;to compare for every term
For $x = 1 to $cnt-1 step 1;compare that term to every result
If StringInStr( $aLinkInfo[$x][1], $Terms[$i][1] ) <> 0 Then
    $y=$y+1
    IniWrite( @ScriptDir & "\Results.ini", "Results", $alinkinfo[$x][1], $alinkinfo[$x][2] )
Endif
Next    
Next

$results = IniReadSection( @SCRIPTDIR & "\Results.ini", "Results")
If $y <>0 Then
$file = FileOpen( @ScriptDir & "\Results.htm", 9 )   ;write page header
FileWriteLine( $file, '<h2>AutoIT Search Companion for Craigs List Found <b>' & $y & '</b> results that matched your search criteria!</h2>' & @CRLF )   
For $u = 1 to $results[0][0] step 1     
FileWriteLine( $file, '<p><a href="' & $results[$u][1] & '">' & $results[$u][0] & '</a></p>' & @CRLF ) ;write links and data
Next
FileClose($File)
TrayTip( "Craigs List Results", "Search Found " & $y & " items!", 5000 );results
Sleep(5000)
TrayTip("","",0)
Else
FileClose($File)
TrayTip( "Craigs List Results", "Inconclusive Search (0 Results)", 5000 );for no results
Sleep(5000)
TrayTip("","",0)
EndIf
;ShellExecute( @Scriptdir & "\Smpt.exe", @Scriptdir & "\Results.htm" ) ;UES THIS LINE TO USE SMPT
Sleep((1000*60)*20);20 min Sleep
Wend;start all over

You have to had the Settings.ini There must be at least one term!!!

Format it like so

[Terms]

1=Computer

2=Tv

3=whatever

x=xxx

[center][/center]

DirtDBaK Universalist

DirtDBaK

Posted
Posted

Yeah, when I saw the response to your suggestion was "Thats not as cool as what i want too do"... I decided we had a "learning opportunity"... :)

Dale

Thanks for that though

Its always good to expand my knowledge in AutoIT

[center][/center]

DirtDBaK Universalist

DirtDBaK

Posted
Posted

Great, giving it a whirl now.

The File Results.htm contains the results!!!

its very basic but you can add html in the header part... so it will look better... but i think basic is fine becuz its fast!!

[center][/center]

phatzilla Adventurer

phatzilla

Posted
  • Author
Posted

works great, question though : lets say that i close the script and rerun it, it'll have list the same results twice if they come up. How can i make sure it never records the same 'hits' twice.? Just to keep things organized.

DaleHohm Universalist

DaleHohm

Posted
Posted (edited)

keep in mind it only searchs the first page.... =( I'll have to work on that maybe DaleHohm could help us with that...

Use

_IELinkClickByText($oIE, "next 100 postings")

_IELoadWait($oIE)

and then start over...

Dale

p.s. Check the return status of _IELinkClickByText to see if it found a link for the next 100 or not...

Also, as mentioned earlier, _IECreate has a parameter for invisible. _IEQuit kills the window

Edited by DaleHohm

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y

Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

DirtDBaK Universalist

DirtDBaK

Posted
Posted

I found a SLOW and sloppy solution to our problem...

But it can load as many pages as you tell it, in dayton OH USA there is 9500+ items listed so i wanted it to search my terms using 9500 items which would be 95 pages.... Slow yes but when your at work you wont see it!! and hopefully we can make it work in the background!!!

Anywho here is the new souce:

Change the url to your local url and be sure to change the items and pages to what you need....

Have fun

;DBak & DaleHohm 
;V 0.1


FileDelete( @ScriptDir & "\Results.htm" )
FileDelete( @scriptdir & "\Results.ini" )
#include <IE.au3>
#include <Array.au3>




While 1


$oIE = _IECreate("http://dayton.craigslist.org/sss/") ;not sure how to hide  ;<---CAHNGE TO YOUR URL!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


$oPs = _IETagNameGetCollection($oIE, "p")
$cntPs = @extended

Global $aLinkInfo[9500][3]               ;<---ALLOWS FOR 95 PAGES (9500 ITEMS) !!!!!!!!!!!!!!!! This is Pages * 100

$aLinkInfo[0][0] = "Index"
$aLinkInfo[0][1] = "Link Text"
$aLinkInfo[0][2] = "href"

$cnt = 1
For $oP in $oPs
    $oLink = _IETagNameGetCollection($oP, "a", 0)
    $aLinkInfo[$cnt][0] = $cnt - 1
    $aLinkInfo[$cnt][1] = $oLink.innerText
    $aLinkInfo[$cnt][2] = $oLink.href
    $cnt +=1
Next






GetMore( 95 );gets 9500 items, or 95 page   <-- GETS YOU MORE ITEMS  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Change your Number
;_ArrayDisplay($aLinkInfo, "LinkInfo")




$y = 0
$terms = IniReadSection( @SCRIPTDIR & "\Settings.ini", "Terms" )  ;compare and find results
For $i = 1 to $terms[0][0] step 1;to compare for every term
For $x = 1 to $cnt-1 step 1;compare that term to every result
If StringInStr( $aLinkInfo[$x][1], $Terms[$i][1] ) <> 0 Then
    $y=$y+1
    IniWrite( @ScriptDir & "\Results.ini", "Results", $alinkinfo[$x][1], $alinkinfo[$x][2] )
Endif
Next    
Next


$results = IniReadSection( @SCRIPTDIR & "\Results.ini", "Results")
If $y <>0 Then
$file = FileOpen( @ScriptDir & "\Results.htm", 9 )   ;write page header
FileWriteLine( $file, '<h2>AutoIT Search Companion for Craigs List Found <b>' & $y & '</b> results that matched your search criteria!</h2>' & @CRLF )   
For $u = 1 to $results[0][0] step 1     
FileWriteLine( $file, '<p><a href="' & $results[$u][1] & '">' & $results[$u][0] & '</a></p>' & @CRLF ) ;write links and data
Next
FileClose($File)
TrayTip( "Craigs List Results", "Search Found " & $y & " items!", 5000 );results
Sleep(5000)
TrayTip("","",0)
Else
FileClose($File)
TrayTip( "Craigs List Results", "Inconclusive Search (0 Results)", 5000 );for no results
Sleep(5000)
TrayTip("","",0)
EndIf
;ShellExecute( @Scriptdir & "\Smpt.exe", @Scriptdir & "\Results.htm" ) ;UES THIS LINE TO USE SMPT
Sleep((1000*60)*20);20 min Sleep
Wend;start all over



Func GetMore( $pages )
ProgressOn( "Working...", "Geting Extended Pages!" )
For $i=1 to $pages step 1
ProgressSet( $i/$pages*100, $i )
_IENavigate( $oIE, "http://dayton.craigslist.org/sss/index"& $pages & "00.html") ;YOUR URL !!!!!!!!!!!!!!!!!11 Cahnge before the /index to your url !!!!!
$oPs = _IETagNameGetCollection($oIE, "p")
$cnt = 101
For $oP in $oPs
    $oLink = _IETagNameGetCollection($oP, "a", 0)
    $aLinkInfo[$cnt][0] = $cnt - 1
    $aLinkInfo[$cnt][1] = $oLink.innerText
    $aLinkInfo[$cnt][2] = $oLink.href
    $cnt +=1
Next
Next
ProgressOff()
EndFunc

[center][/center]

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...