Navigate a website without a browser.

NSearch · December 12, 2005

I am working on a program that navigates to a particular website page. I have to have the web browser open and in the foreground. I was curious if there is anyway to navigate a webpage without bringing up the web browser. I am trying to capture pictures on a website. I am able to capture the pictures, but I would like to be able to do it in the background. Any ideas?

seandisanti · December 12, 2005

I am working on a program that navigates to a particular website page. I have to have the web browser open and in the foreground. I was curious if there is anyway to navigate a webpage without bringing up the web browser. I am trying to capture pictures on a website. I am able to capture the pictures, but I would like to be able to do it in the background. Any ideas?

use dale's IE.au3

you can make your browser session do everything you want it to, even if you use the optional parameter to make the browser window hidden

NSearch · December 12, 2005

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

seandisanti · December 12, 2005

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

if you're not using the beta, get the file IE.au3. Open it up and read the documentation. The function that creates the browser window has an optional paramter which denotes whether the window is visible or not. I think it has to be set to 0 for invisible window...

DaleHohm · December 13, 2005

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

You're right, there is no example of this and it makes a good one. This can readily be done with IE.au3 and adding in INetGet()

This creates a browser window (as cameronsdad points out, pass a parameter of 0 to make it invisible and then be certain to call _IEQuit when done), navigates to a webpage and downloads all images to a local folder using the source image name.

#include <IE.au3>

$sImgDir = "c:\foo\"; Please make certain this folder already exists (silent failure if not)
$sWebPage = "http://www.autoitscript.com/forum/index.php?"; webpage with images

$oIE = _IECreate()
_IENavigate($oIE, $sWebPage)
$oIMGs = _IETagNameGetCollection($oIE.document, "img")

; Loop through all IMG tags and save file to local directory using INetGet
For $oIMG in $oIMGs
    $sImgUrl = $oIMG.src
    $sImgFileName = $oIMG.nameProp
    INetGet($sImgUrl,  $sImgDir & $sImgFileName)
Next

You can find more about the attributes of the IMG tag at MSDN

Dale

Edited December 13, 2005 by DaleHohm

greenmachine · December 13, 2005

Damn. Dale, you're good. I think I might have to use that.

DaleHohm · December 13, 2005

Damn. Dale, you're good. I think I might have to use that.

:"> This has been added as an example in reply 3 of the IE.au3 post

Dale

NSearch · December 13, 2005

This is not exactly what I am looking for. I do not even want the web page to open. I have seen perl scripts which accomplish what I am looking for. I just wonder if it is possible using Autoit. I want the webpage to run completely in the background. I am also navigating through a https. That may cause a problem. One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.

Also, with the code provided I got the following error, and I believe it is caused from trying to run it on a https.

$sImgUrl = $oIMG.src

$sImgUrl = $oIMG.src^ERROR

Error:The requested action with this object failed

t0ddie · December 13, 2005

wont work for firefox browser

DaleHohm · December 13, 2005

This is not exactly what I am looking for. I do not even want the web page to open. I have seen perl scripts which accomplish what I am looking for. I just wonder if it is possible using Autoit. I want the webpage to run completely in the background. I am also navigating through a https. That may cause a problem. One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.
Also, with the code provided I got the following error, and I believe it is caused from trying to run it on a https.
$sImgUrl = $oIMG.src
$sImgUrl = $oIMG.src^ERROR
Error:The requested action with this object failed

No, your error is not caused by https. Once the page is loaded it acts as any other. You do need to be running the beta version of AutoIt (beta 63 or higher) to use IE.au3 Can't tell if that's an issue though since you don't specify. Hopefully you tried the script as written and saw that it worked. If you can reproduce your trouble on a page I can access I'll be happy to take a look.

If you read what I posted, you'll notice the suggestion to use _IECreate(0) instead of _IECreate() so that the browser window never becomes visible (it runs "completely in the background").

Also, I wasn't trying to write a custom script for you that gave you "exactly what you were looking for", but rather to to show you how it could be done. Is it so hard for you to add a conditional statement to the INetGet line to see if the image name is the one you want before you capture it?

Dale

@t0ddie -- how observant. Thanks for sharing...

NSearch · December 13, 2005

I certainly did not mean for my statements to come across the way that they did. I was not asking for you to write a script for me. I was just trying to carry on the conversation. I apprecite the help that you have given everyone that visits this forum.

Thanks again.

Gene · December 13, 2005

One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.

Since you apparently know the URL of this graphic, why don't you use:

InetGet

Downloads a file from the internet using the http or ftp protocol.
InetGet ( "URL", "filename" [, reload [, background]] )
Parameters

URL URL of the file to download. See remarks below.filename Local filename to download to.reload [optional]
0 = (default) Get the file from local cache if available
1 = Forces a reload from the remote sitebackground [optional]
0 = (default) Wait until the download is complete before continuing.
1 = return immediately and download in the background (see remarks).

Return Value

Success:Returns 1.Failure:Returns 0.

Remarks

Internet Explorer 3 or greater must be installed for this function to work.
The URL parameter should be in the form "http://www.somesite.com/path/file.html" - just like an address you would type into your web browser.

To use a username and password when connecting simply prefix the servername with "username:password@", e.g.
"http://myuser:mypassword@www.somesite.com"

The InetGet function works with http:// https:// and ftp:// - to change the transfer type when using ftp see the FtpBinaryMode option.

"background" Parameter

By default the function waits until the download has finished before returning. Sometimes with large downloads this is not always wanted. If the background parameter is set to 1 the function returns immediately and the download continues in the background. When in this mode two macros are used to keep track:
@InetGetActive = 1 while downloading, or 0 when finished.
@InetGetBytesRead = the number of bytes downloaded, or -1 if there has been an error.

Note, only one download can be active at once, if you call the function again before a download is complete it will fail.

To abort a download call the function with just the first parameter set to "abort" like so:

InetGet("abort")

Related

FtpBinaryMode (Option), FtpSetProxy, InetGetSize, HttpSetProxy

Example

InetGet("http://www.mozilla.org", "C:\foo.html")
InetGet("http://www.autoitscript.com", "C:\mydownload.htm", 1)
InetGet("ftp://ftp.mozilla.org/pub/mozilla.org/README", "README.txt", 1)

; Advanced example - downloading in the background
InetGet("http://www.nowhere.com/somelargefile.exe", "test.exe", 1, 1)

While @InetGetActive
TrayTip("Downloading", "Bytes = " & @InetGetBytesRead, 10, 16)
Sleep(250)
Wend

MsgBox(0, "Bytes read", @InetGetBytesRead)

Gene

NSearch · December 13, 2005

I was sending the following commands to the browser in order to navigate to the next page, but using the new method (in the background) it will not work. Any ideas.

Send("{TAB 22}") ;tab to correct input location

Send($number1)

sleep(200)

Send("{TAB}") ;tab to second input location

Send($number2)

sleep(200)

Send("{TAB}") ; tab to submit

Send("{ENTER}")

seandisanti · December 13, 2005

I was sending the following commands to the browser in order to navigate to the next page, but using the new method (in the background) it will not work. Any ideas.
Send("{TAB 22}") ;tab to correct input location
Send($number1)
sleep(200)
Send("{TAB}") ;tab to second input location
Send($number2)
sleep(200)
Send("{TAB}") ; tab to submit
Send("{ENTER}")

yes, you could open up the ie.au3 file, and see how to use it correctly to access the elements of the web page more precisely without using tabs etc which may change based on ads on the page etc...

blademonkey · December 13, 2005

I'm don't understand what you're trying to do.

What do you mean when you say:

I do not even want the web page to open.

Are you trying saying that you dont want have the web browser to be visible by the user?

I want the webpage to run completely in the background.

To me background means that the window is visible but just not in focus. Is that what you mean? to have a script automate something on a non-focused window?

This sounds alot like an auto Ad clicker. Which may entirely be possible but to me it's unclear if that is what you're trying to create.

more info needed.

Edited December 13, 2005 by blademonkey

NSearch · December 13, 2005

I am trying to grab pictures off of a website and store them in a folder for further processing. I have to navigate to the page, and enter numbers to get to the point which I grab the picture. I would like to do this all without the user having to bother with it (meaning that the mouse, and browser is tied up). I am currently searching the ie.au3 forum for ways to focus on the form input.....examples would be appreciated. Thanks.

blademonkey · December 13, 2005

I am trying to grab pictures off of a website and store them in a folder for further processing. I have to navigate to the page, and enter numbers to get to the point which I grab the picture. I would like to do this all without the user having to bother with it (meaning that the mouse, and browser is tied up). I am currently searching the ie.au3 forum for ways to focus on the form input.....examples would be appreciated. Thanks.

I'm assuming that the numbers are randomly generated and you have to retype them in. Is this correct?

NSearch · December 13, 2005

I am reading them from a txt file.

seandisanti · December 13, 2005

I am reading them from a txt file.

do these functions sound like they'd do what you want? i think they do...

_IEFormGetCount()
Get the count of the number of forms in the document

_IEFormGetCollection()
Obtain a collection object variable representing the frames in the document

_IEFormGetObjByIndex()
Obtain an object variable reference to a form by 0-based index

_IEFormGetObjByName()
Obtain an object variable reference to a form by name

_IEFormGetNameByIndex()
Obtain the name of a form by its 0-based index

_IEFormElementGetCount()
Obtain a count of the number of form elements within a given form

_IEFormElementGetCollection()
Obtain a collection object variable of all form elements within a given form

_IEFormElementGetObjByIndex()
Obtain a object reference to a form element within a form by 0-based index

_IEFormElementGetObjByName()
Obtain a object reference to a form element within a form by name

_IEFormElementGetTypeByIndex()
Obtain the type of a givien form element within a form by 0-based index
(button, checkbox, fileUpload, hidden, image, password, radio, reset, submit, or text)

_IEFormElementOptionGetCount()
Get count of Options within a Select drop-down form element

_IEFormElementGetValue()
Get the value of a specifid form element

_IEFormElementSetValue()
Set the value of a specified form element

_IEFormSubmit()
Submit a specified form

_IEFormReset()
Reset a specified form

they're all in the ie.au3 file

***edit*** removed part of my response that may have been taken incorrectly.

Edited December 13, 2005 by cameronsdad

NSearch · December 13, 2005

Hopefully someone can help me focus on the first_number, and second_number form objects and input the correct data.

I am getting the following error:

If IsObj($o_object.elements.item ($s_name, $i_index)) Then

If IsObj($o_object^ ERROR

Error: Variable must be of type "Object"

When using the following code:

$o_SearchForm = _IEFormGetObjByName ($oIE, "jsp_name")
$o_Keywords = _IEFormElementGetObjByName ($o_SearchForm, "first_number")
_IEFormElementSetValue ($o_Keywords, "Testing")

Source Code

<table width="460" align="left" cellpadding="0" cellspacing="0" border="0">
  <FORM METHOD="POST" ACTION="/servlet/InvoiceServlet" METHOD="POST">
    <INPUT TYPE="hidden" NAME="jsp_name" VALUE="Invoiced">
    <INPUT TYPE="HIDDEN" NAME="orig_country"
      VALUE='US' >
    <INPUT TYPE="HIDDEN" NAME="language"
      VALUE='english' >
    <INPUT TYPE="hidden" NAME="service_type" 
      VALUE='E' >
    <INPUT TYPE="hidden" NAME="pay_type" 
      VALUE='invoice' >
    <INPUT TYPE="hidden" NAME="request_type" 
      VALUE='NewReq' >

    <tr> 
      <td CLASS="pagesubheader2" align=left valign="top"> 
        <table BORDER=0 ALIGN=left CELLPADDING=0 CELLSPACING=0 WIDTH=461>
          <tr> 
            <td align=right height="23" valign="top" width="36%">
              <b>Service Type:</b> 
            </td>
            <td align=right height="23" valign="top" width="2%">&nbsp;</td>
            <td align=left height="23" valign="top" width="62%">
              <img src="/images/shared/picture.gif" alt="Picture">

            </td>
          </tr>
          <tr> 
            <td width="36%" height="31" valign="top" align="right">
              <b>First Number:</b>
            </td>
            <td width="2%" height="31" valign="middle" align="right">&nbsp;</td>
            <td width="62%" height="31" valign="top" align="left">
              <input type="TEXT" name="first_number" size="15" maxlength="20"
                VALUE='' >
              <BR>
              
              
              
            </td>
          </tr>
          <tr> 
            <td width="36%" align="right" valign="top">
              <b>&nbsp;Second Number:</b>
            </td>
            <td width="2%" align="right">&nbsp;</td>
            <td width="62%" align="left"> 
              <input type=TEXT name="second_number" size="15" maxlength="9"
                VALUE='' >
              (9 digits)
              <BR>
              
              
              
            </td>
          </tr>
          <tr align="center"> 
            <td width="36%">&nbsp;</td>
            <td>&nbsp;</td>
            <td align="left" width="62%"><br>
              <input type="SUBMIT" value="Send Request"><BR><BR>
            </td>
          </tr>
        </table>
      </td>
    </tr>

Navigate a website without a browser.

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members