Sign in to follow this  
Followers 0
NSearch

Navigate a website without a browser.

30 posts in this topic

I am working on a program that navigates to a particular website page. I have to have the web browser open and in the foreground. I was curious if there is anyway to navigate a webpage without bringing up the web browser. I am trying to capture pictures on a website. I am able to capture the pictures, but I would like to be able to do it in the background. Any ideas?

Share this post


Link to post
Share on other sites



I am working on a program that navigates to a particular website page. I have to have the web browser open and in the foreground. I was curious if there is anyway to navigate a webpage without bringing up the web browser. I am trying to capture pictures on a website. I am able to capture the pictures, but I would like to be able to do it in the background. Any ideas?

use dale's IE.au3

you can make your browser session do everything you want it to, even if you use the optional parameter to make the browser window hidden


1100111 00001011101111 00011101101111 00010111100100 00001111110100 00110111110010 00101101111001 0011100i didn't make up this form of encryption, but i like it.credit to the lvl 6 challenge on arcanum.co.nz

Share this post


Link to post
Share on other sites

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

Share this post


Link to post
Share on other sites

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

if you're not using the beta, get the file IE.au3. Open it up and read the documentation. The function that creates the browser window has an optional paramter which denotes whether the window is visible or not. I think it has to be set to 0 for invisible window...

1100111 00001011101111 00011101101111 00010111100100 00001111110100 00110111110010 00101101111001 0011100i didn't make up this form of encryption, but i like it.credit to the lvl 6 challenge on arcanum.co.nz

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

I have searched the thread, but I do not see any examples of what I am looking for. I am sure that I have overlooked it. Does anyone have an example of what I am talking about. Thank you.

You're right, there is no example of this and it makes a good one. This can readily be done with IE.au3 and adding in INetGet()

This creates a browser window (as cameronsdad points out, pass a parameter of 0 to make it invisible and then be certain to call _IEQuit when done), navigates to a webpage and downloads all images to a local folder using the source image name.

#include <IE.au3>

$sImgDir = "c:\foo\"; Please make certain this folder already exists (silent failure if not)
$sWebPage = "http://www.autoitscript.com/forum/index.php?"; webpage with images

$oIE = _IECreate()
_IENavigate($oIE, $sWebPage)
$oIMGs = _IETagNameGetCollection($oIE.document, "img")

; Loop through all IMG tags and save file to local directory using INetGet
For $oIMG in $oIMGs
    $sImgUrl = $oIMG.src
    $sImgFileName = $oIMG.nameProp
    INetGet($sImgUrl,  $sImgDir & $sImgFileName)
Next

You can find more about the attributes of the IMG tag at MSDN

Dale

Edited by DaleHohm

Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

Damn. Dale, you're good. I think I might have to use that.

Share this post


Link to post
Share on other sites

Damn. Dale, you're good. I think I might have to use that.

:"> This has been added as an example in reply 3 of the IE.au3 post

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

This is not exactly what I am looking for. I do not even want the web page to open. I have seen perl scripts which accomplish what I am looking for. I just wonder if it is possible using Autoit. I want the webpage to run completely in the background. I am also navigating through a https. That may cause a problem. One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.

Also, with the code provided I got the following error, and I believe it is caused from trying to run it on a https.

$sImgUrl = $oIMG.src

$sImgUrl = $oIMG.src^ERROR

Error:The requested action with this object failed

Share this post


Link to post
Share on other sites

wont work for firefox browser


Valik Note Added 19 October 2006 - 08:38 AMAdded to warn level I just plain don't like you.

Share this post


Link to post
Share on other sites

This is not exactly what I am looking for. I do not even want the web page to open. I have seen perl scripts which accomplish what I am looking for. I just wonder if it is possible using Autoit. I want the webpage to run completely in the background. I am also navigating through a https. That may cause a problem. One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.

Also, with the code provided I got the following error, and I believe it is caused from trying to run it on a https.

$sImgUrl = $oIMG.src

$sImgUrl = $oIMG.src^ERROR

Error:The requested action with this object failed

No, your error is not caused by https. Once the page is loaded it acts as any other. You do need to be running the beta version of AutoIt (beta 63 or higher) to use IE.au3 Can't tell if that's an issue though since you don't specify. Hopefully you tried the script as written and saw that it worked. If you can reproduce your trouble on a page I can access I'll be happy to take a look.

If you read what I posted, you'll notice the suggestion to use _IECreate(0) instead of _IECreate() so that the browser window never becomes visible (it runs "completely in the background").

Also, I wasn't trying to write a custom script for you that gave you "exactly what you were looking for", but rather to to show you how it could be done. Is it so hard for you to add a conditional statement to the INetGet line to see if the image name is the one you want before you capture it?

Dale

@t0ddie -- how observant. Thanks for sharing...


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

I certainly did not mean for my statements to come across the way that they did. I was not asking for you to write a script for me. I was just trying to carry on the conversation. I apprecite the help that you have given everyone that visits this forum.

Thanks again.

Share this post


Link to post
Share on other sites

One other thing. I am only trying to capture one picture on the page, I know the name of it, and it will always be in the same location on the page. Is this possible.

Since you apparently know the URL of this graphic, why don't you use:

InetGet

Downloads a file from the internet using the http or ftp protocol.

InetGet ( "URL", "filename" [, reload [, background]] )

Parameters

URL URL of the file to download. See remarks below.filename Local filename to download to.reload [optional]

0 = (default) Get the file from local cache if available

1 = Forces a reload from the remote sitebackground [optional]

0 = (default) Wait until the download is complete before continuing.

1 = return immediately and download in the background (see remarks).

Return Value

Success:Returns 1.Failure:Returns 0.

Remarks

Internet Explorer 3 or greater must be installed for this function to work.

The URL parameter should be in the form "http://www.somesite.com/path/file.html" - just like an address you would type into your web browser.

To use a username and password when connecting simply prefix the servername with "username:password@", e.g.

"http://myuser:mypassword@www.somesite.com"

The InetGet function works with http:// https:// and ftp:// - to change the transfer type when using ftp see the FtpBinaryMode option.

"background" Parameter

By default the function waits until the download has finished before returning. Sometimes with large downloads this is not always wanted. If the background parameter is set to 1 the function returns immediately and the download continues in the background. When in this mode two macros are used to keep track:

@InetGetActive = 1 while downloading, or 0 when finished.

@InetGetBytesRead = the number of bytes downloaded, or -1 if there has been an error.

Note, only one download can be active at once, if you call the function again before a download is complete it will fail.

To abort a download call the function with just the first parameter set to "abort" like so:

InetGet("abort")

Related

FtpBinaryMode (Option), FtpSetProxy, InetGetSize, HttpSetProxy

Example

InetGet("http://www.mozilla.org", "C:\foo.html")

InetGet("http://www.autoitscript.com", "C:\mydownload.htm", 1)

InetGet("ftp://ftp.mozilla.org/pub/mozilla.org/README", "README.txt", 1)

; Advanced example - downloading in the background

InetGet("http://www.nowhere.com/somelargefile.exe", "test.exe", 1, 1)

While @InetGetActive

TrayTip("Downloading", "Bytes = " & @InetGetBytesRead, 10, 16)

Sleep(250)

Wend

MsgBox(0, "Bytes read", @InetGetBytesRead)

Gene


[font="Verdana"]Thanks for the response.Gene[/font]Yes, I know the punctuation is not right...

Share this post


Link to post
Share on other sites

I was sending the following commands to the browser in order to navigate to the next page, but using the new method (in the background) it will not work. Any ideas.

Send("{TAB 22}") ;tab to correct input location

Send($number1)

sleep(200)

Send("{TAB}") ;tab to second input location

Send($number2)

sleep(200)

Send("{TAB}") ; tab to submit

Send("{ENTER}")

Share this post


Link to post
Share on other sites

I was sending the following commands to the browser in order to navigate to the next page, but using the new method (in the background) it will not work. Any ideas.

Send("{TAB 22}") ;tab to correct input location

Send($number1)

sleep(200)

Send("{TAB}") ;tab to second input location

Send($number2)

sleep(200)

Send("{TAB}") ; tab to submit

Send("{ENTER}")

yes, you could open up the ie.au3 file, and see how to use it correctly to access the elements of the web page more precisely without using tabs etc which may change based on ads on the page etc...

1100111 00001011101111 00011101101111 00010111100100 00001111110100 00110111110010 00101101111001 0011100i didn't make up this form of encryption, but i like it.credit to the lvl 6 challenge on arcanum.co.nz

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

I'm don't understand what you're trying to do.

What do you mean when you say:

I do not even want the web page to open.

Are you trying saying that you dont want have the web browser to be visible by the user?

I want the webpage to run completely in the background.

To me background means that the window is visible but just not in focus. Is that what you mean? to have a script automate something on a non-focused window?

This sounds alot like an auto Ad clicker. Which may entirely be possible but to me it's unclear if that is what you're trying to create.

more info needed.

Edited by blademonkey

---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

I am trying to grab pictures off of a website and store them in a folder for further processing. I have to navigate to the page, and enter numbers to get to the point which I grab the picture. I would like to do this all without the user having to bother with it (meaning that the mouse, and browser is tied up). I am currently searching the ie.au3 forum for ways to focus on the form input.....examples would be appreciated. Thanks.

Share this post


Link to post
Share on other sites

I am trying to grab pictures off of a website and store them in a folder for further processing. I have to navigate to the page, and enter numbers to get to the point which I grab the picture. I would like to do this all without the user having to bother with it (meaning that the mouse, and browser is tied up). I am currently searching the ie.au3 forum for ways to focus on the form input.....examples would be appreciated. Thanks.

I'm assuming that the numbers are randomly generated and you have to retype them in. Is this correct?

---"Educate the Mind, Make Savage the Body" -Mao Tse Tung

Share this post


Link to post
Share on other sites

I am reading them from a txt file.

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

I am reading them from a txt file.

do these functions sound like they'd do what you want? i think they do...

_IEFormGetCount()

Get the count of the number of forms in the document

_IEFormGetCollection()

Obtain a collection object variable representing the frames in the document

_IEFormGetObjByIndex()

Obtain an object variable reference to a form by 0-based index

_IEFormGetObjByName()

Obtain an object variable reference to a form by name

_IEFormGetNameByIndex()

Obtain the name of a form by its 0-based index

_IEFormElementGetCount()

Obtain a count of the number of form elements within a given form

_IEFormElementGetCollection()

Obtain a collection object variable of all form elements within a given form

_IEFormElementGetObjByIndex()

Obtain a object reference to a form element within a form by 0-based index

_IEFormElementGetObjByName()

Obtain a object reference to a form element within a form by name

_IEFormElementGetTypeByIndex()

Obtain the type of a givien form element within a form by 0-based index

(button, checkbox, fileUpload, hidden, image, password, radio, reset, submit, or text)

_IEFormElementOptionGetCount()

Get count of Options within a Select drop-down form element

_IEFormElementGetValue()

Get the value of a specifid form element

_IEFormElementSetValue()

Set the value of a specified form element

_IEFormSubmit()

Submit a specified form

_IEFormReset()

Reset a specified form

they're all in the ie.au3 file

***edit*** removed part of my response that may have been taken incorrectly.

Edited by cameronsdad

1100111 00001011101111 00011101101111 00010111100100 00001111110100 00110111110010 00101101111001 0011100i didn't make up this form of encryption, but i like it.credit to the lvl 6 challenge on arcanum.co.nz

Share this post


Link to post
Share on other sites

Hopefully someone can help me focus on the first_number, and second_number form objects and input the correct data.

I am getting the following error:

If IsObj($o_object.elements.item ($s_name, $i_index)) Then

If IsObj($o_object^ ERROR

Error: Variable must be of type "Object"

When using the following code:

$o_SearchForm = _IEFormGetObjByName ($oIE, "jsp_name")
$o_Keywords = _IEFormElementGetObjByName ($o_SearchForm, "first_number")
_IEFormElementSetValue ($o_Keywords, "Testing")

Source Code

<table width="460" align="left" cellpadding="0" cellspacing="0" border="0">
  <FORM METHOD="POST" ACTION="/servlet/InvoiceServlet" METHOD="POST">
    <INPUT TYPE="hidden" NAME="jsp_name" VALUE="Invoiced">
    <INPUT TYPE="HIDDEN" NAME="orig_country"
      VALUE='US' >
    <INPUT TYPE="HIDDEN" NAME="language"
      VALUE='english' >
    <INPUT TYPE="hidden" NAME="service_type" 
      VALUE='E' >
    <INPUT TYPE="hidden" NAME="pay_type" 
      VALUE='invoice' >
    <INPUT TYPE="hidden" NAME="request_type" 
      VALUE='NewReq' >

    <tr> 
      <td CLASS="pagesubheader2" align=left valign="top"> 
        <table BORDER=0 ALIGN=left CELLPADDING=0 CELLSPACING=0 WIDTH=461>
          <tr> 
            <td align=right height="23" valign="top" width="36%">
              <b>Service Type:</b> 
            </td>
            <td align=right height="23" valign="top" width="2%">&nbsp;</td>
            <td align=left height="23" valign="top" width="62%">
              <img src="/images/shared/picture.gif" alt="Picture">

            </td>
          </tr>
          <tr> 
            <td width="36%" height="31" valign="top" align="right">
              <b>First Number:</b>
            </td>
            <td width="2%" height="31" valign="middle" align="right">&nbsp;</td>
            <td width="62%" height="31" valign="top" align="left">
              <input type="TEXT" name="first_number" size="15" maxlength="20"
                VALUE='' >
              <BR>
              
              
              
            </td>
          </tr>
          <tr> 
            <td width="36%" align="right" valign="top">
              <b>&nbsp;Second Number:</b>
            </td>
            <td width="2%" align="right">&nbsp;</td>
            <td width="62%" align="left"> 
              <input type=TEXT name="second_number" size="15" maxlength="9"
                VALUE='' >
              (9 digits)
              <BR>
              
              
              
            </td>
          </tr>
          <tr align="center"> 
            <td width="36%">&nbsp;</td>
            <td>&nbsp;</td>
            <td align="left" width="62%"><br>
              <input type="SUBMIT" value="Send Request"><BR><BR>
            </td>
          </tr>
        </table>
      </td>
    </tr>

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0