Jump to content
Sign in to follow this  
Brickoneer

automating the internet without IE

Recommended Posts

Brickoneer

Hey guys,

My entire program is based off automating a website... but it is running a bit slow. So I thought that it would speed up a LOT if I could bypass IE and communicate on the port level. But I don't know where to start! A great portion of my script relies on the IE.au3 tools, so it is a major rewrite. Cutting IE out of the loop would make my script run much faster, wouldn't it?

I looked at http.au3 and got it to access a page... but it returned only the first 10-14 characters of the page. :P Any ideas what's wrong?

Anyway... can someone point me in the direction I need to go? I'm clueless. :">

Many thanks,

Brick

Edited by Brickoneer

Share this post


Link to post
Share on other sites
PsaltyDS

Hey guys,

My entire program is based off automating a website... but it is running a bit slow. So I thought that it would speed up a LOT if I could bypass IE and communicate on the port level.

Not.

But I don't know where to start! A great portion of my script relies on the IE.au3 tools, so it is a major rewrite. Cutting IE out of the loop would make my script run much faster, wouldn't it?

Not.

I looked at http.au3 and got it to access a page... but it returned only the first 10-14 characters of the page. :P Any ideas what's wrong?

Anyway... can someone point me in the direction I need to go? I'm clueless. :">

Many thanks,

Brick

Most web pages these days use forms, frames, various scripting engines, perhaps dynamic .aspx pages, etc. How do intend to reproduce all that outside of a purpose built browser like Firefox or IE? Is the site you work with so basic you can do without the full interpretive functionality of a real browser?

Your only option right now for automation with AutoIt is IE. It's possible, but unlikely (especially since you couldn't even get HTTP.au3 to read a single full page), you could create a better IE UDF than Dale did, but if you can't just stick with IE.au3.

As for slowness, that is more likely a problem with your code than the IE UDF. Exactly what is slow about it? What code performs the part you find 'slow'?

;)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
DaleHohm

Not.

Not.

It's possible you could create a better IE UDF than Dale did

Not! :P

As PsaltyDS said, it all depends on what is perceived to be slow. It is going to take the same amount of time to download bytes regardless. One of the things that takes a browser some time, because it is incredibly complex, it page composition - this means layout control, processing of client-side script, dynamic HTML etc. If your needs do not require any of this and don't require authentication, cookies, certificates, ssl etc that the browser handles for you, then you can communicate at the port level or with HTTP.au3. Part of what has made the HTTP protocol so successful is that it IS very simple - you can communicate with a web server on port 80 with just a few strategic characters to download a file.

All that said, understanding what is perceived to be slow, there may be ways to optimize your use of IE.au3 that will greatly increase your throughput... on the other hand, you can also do it in the background with an invisible IE window and then perhaps you don't care how long it takes... For me I always consider whether an extra 5 hours coding to save say, 5 minutes per run really saves me time...

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
PsaltyDS

... For me I always consider whether an extra 5 hours coding to save say, 5 minutes per run really saves me time...

Dale

Of course, 5 hours of coding to save yourself 5 minutes of run time does get you mad huge Geek Points!

:P


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
Brickoneer

Thanks for the replies guys.

I'm not exactly sure what is causing the slow down, but I'm assuming bypassing the rendering would at least improve preformance.

The pages I would be trying to automate are table-intensive PHP sites, so the most complicated thing I'd have to deal with is a PHP session ID. Also, I've heard that tables are tough on the browsers load time, so cutting out the rendering might be extra-helpful here.

But should I be trying to figure out the http.au3 or go straight down to the TCP functions and do it from there?

Thanks,

brick

[edit] If I did save 5 minutes at runtime, I only need to use the program 70 times to make that 5 hours back.

But of course since I have 0 experience with low-level stuff... it will probably be a couple hundred hours of work. :P

Edited by Brickoneer

Share this post


Link to post
Share on other sites
DaleHohm

I'm not exactly sure what is causing the slow down

If you set out to solve problems without first understanding them, you're doomed to solving the wrong problems. Code is as life.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.