Sign in to follow this  
Followers 0
tonyrocks

How to login to Site, crawl list of URLS save each page as HTML

5 posts in this topic

So i've been tasked with logging into a site with IE, and then look at a text file that contains a list of URLs on the site to crawl.  I need to crawl each URL, save the HTML to a specified file name, then go to the next URL and repeat the process.

So far I am able to login to the site in IE with this:

#include <ie.au3>
#include <INet.au3>
#include <MsgBoxConstants.au3>

$uname="tony"
$pwd="tony"

; Get ready to login!
$oIE = _IECreate ("http://localhost/books/login.aspx")
$oForm = _IEFormGetObjByName ($oIE, "form1")
$oQuery1 = _IEFormElementGetObjByName ($oForm, "userNameTextBox")
$oQuery2 = _IEFormElementGetObjByName ($oForm, "passwordTextBox")

; Start sending form values and then simulate a click to login
_IEFormElementSetValue ($oQuery1,$uname)
_IEFormElementSetValue ($oQuery2,$pwd)
$oButton=_IEGetObjById($oIE,"loginButton")
_IEAction ($oButton, "click")
_IELoadWait($oIE,0)

That gets me to login.

My text file (c:urls.txt) looks like this:

http://localhost/books/book1.html
http://localhost/books/book2.html
http://localhost/guies/book1.html

I have to then open IE up to the first URL, then Save file AS something, then go to the next one.  

Any suggestions?

Thanks!
-Tony

Share this post


Link to post
Share on other sites



I'm not sure I really understand your question. Basically, you copy a .HTML file on your computer, with the copy renamed?


- Bruce /*somdcomputerguy */  If you change the way you look at things, the things you look at change.

Share this post


Link to post
Share on other sites

Hello somdcomputerguy,

Yes, I have each page with a specific name.  @jdelaney I'm going to check those out! Thanks!

-Tony

Share this post


Link to post
Share on other sites

Yes, I have each page with a specific name.

That doesn't really answer my question. I probably didn't word it right. Won't using just the FileCopy() function suffice?


- Bruce /*somdcomputerguy */  If you change the way you look at things, the things you look at change.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0