Sign in to follow this  
Followers 0
OverloadUT

HTTP UDF's

81 posts in this topic

#1 ·  Posted (edited)

I am working on a fairly large project and a big part of it is an AutoIt application that uploads a lot of data to a PHP script.

For a while I was using INetGet and submitting all my data as GET variables in the URL. However, the amount of data to upload got too big (over 1000 characters) and the INetGet function started to fail.

I also hated that INetGet ONLY works if the webserver responds with "200 OK" - There is no way to tell the difference between a 404 error and simply not being able to connect to the server in the first place.

So I write this library of UDF's. Basically, it's a fully compliant HTTP/1.1 client. It uses only the TCP functions; no DllCall needed here! I had a blast learning all about the HTTP protocol and how to use it.

Advantages over INetGet:

The data is downloaded right in to variables instead of to files. You can of course then write this data to a file if you wish.

You can read all of the headers supplied by the webserver.

You can get the HTTP Response Code from the webserver.

When it failes, you know exactly what failed instead of having to guess.

; ===================================================================
; HTTP UDF's
; v0.5
;
; By: Greg "Overload" Laabs
; Last Updated: 07-22-06
; Tested with AutoIt Version 3.1.1.131
; Extra requirements: Nothing!
;
; A set of functions that allow you to download webpages and submit
; POST requests.
;
; Main functions:
; _HTTPConnect - Connects to a webserver
; _HTTPGet - Submits a GET request to a webserver
; _HTTPPost - Submits a POST request to a webserver
; _HTTPRead - Reads the response from a webserver
; ===================================================================

I consider this UDF package to be a "beta" and that's why I call it version 0.5. However, I would love people to give it a try and tell me how it works for them. I have done extensive testing and I think I have the parser working perfectly.

I plan on improving this library. It's possible that I might change the way the functions work (I don't think the _HTTPConnect function is necessary; I could move that functionality right in to the Get or Post functions.)

To Do:

  • Add a function that downloads the data right to a file. You might not want to download a 10 meg file in to memory before saving it to a file.
  • Add a method to get the progress of the download. This will probably be in the form of a callback function, if that's possible in AutoIt.
  • Add the ability to automatically follow "Location:" headers. Currently you'd have to do it manually.
  • Other things I can't think of!

Post #25 For Updated functions to work with current AutoIt versions.

Edited by SmOke_N
Redirect to correct post for download
1 person likes this

Share this post


Link to post
Share on other sites



What resource did you find most helpful in learning the http protocol?

Looks nice though and it gives me a few ideas.


Share this post


Link to post
Share on other sites

@OverloadUT

Thanks for sharing, maybe when you are done you can add it as part of the Include package.\

You are the best!

RK


"When the power of love overcomes the love of power, the world will know peace"-Jimi Hendrix

Share this post


Link to post
Share on other sites

Thank you everyone for the kind words. If anyone actually gives it a try, I'd love to hear how it works for them.

What resource did you find most helpful in learning the http protocol?

This website was my main source. Well, that and a LOT of trial and error. :whistle:

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

Thank you everyone for the kind words. If anyone actually gives it a try, I'd love to hear how it works for them.

Edit: Since $socket is optional and $data is more likely to be used, shouldn't it be at the end? Like(_HTTPPost($host, $page, $data = "" , $socket = -1).

I was woundering how can I submit a search on google.

I realy don't understand how to use the _HTTPPost() function.

here is what I did:

#include <HTTP.au3>
#include <IE.au3>
$host = "www.google.com"
$page = "/imghp?hl=en&tab=wi&q="


ConsoleWrite("Example GET Request:"&@CRLF)
$socket = _HTTPConnect($host)
ConsoleWrite("Socket Created: "&$socket&@CRLF)
$get = _HTTPGet($host, $page, $socket)
ConsoleWrite("Bytes sent: "&$get&@CRLF)
$recv = _HTTPRead($socket,1)
If @error Then
    ConsoleWrite("_HTTPRead Error: "&@error&@CRLF)
    ConsoleWrite("_HTTPRead Return Value: "&$recv &@CRLF)
Else
    ConsoleWrite("HTTP Return Code: "&$recv[0]&@CRLF)
    ConsoleWrite("HTTP Return Response: "&$recv[1]&@CRLF)
    ConsoleWrite("Number of headers: "&UBound($recv[3])&@CRLF)
    ConsoleWrite("Size of data downloaded: "&StringLen($recv[4])&" bytes"&@CRLF)
    ConsoleWrite("Page downloaded: "&@CRLF&$recv[4]&@CRLF)
    $O_IE = _IECreate()
    _IEBodyWriteHTML( $O_IE , $recv[4] )
EndIf
_HTTPPost($host, $page, $socket, "What should I put here??????" );I want to search for "Prosche"
_HTTPClose($socket)

RK

Edited by rbhkamal

"When the power of love overcomes the love of power, the world will know peace"-Jimi Hendrix

Share this post


Link to post
Share on other sites

wewt i've been after an HTTPPost command

tahnks i'll play with it a bit.

Share this post


Link to post
Share on other sites

:"> You know what would be REALLY nice --> something to handle cookies.

Althouhg i have 0 knowledge on cookies, they're still necessary for pages that require

logins :whistle:

Cheers!


[quote name='Valik' post='301213' date='Jan 31 2007, 10:36 PM']You seem to have a habit of putting things in the wrong place. I feel sorry for any female you attempt to have sex with.[/quote][font="Lucida Sans Unicode"][/font]

Share this post


Link to post
Share on other sites

Google actually does all of its searches as GET requests, not POST requests. This is so that you can link a page of results and actually have it work.

In that case, to do a google search, you would want to do this:

$host = "www.google.com"
$page = "/imghp"
$vars = "hl=en&tab=wi&q="

$searchterm = "porche"

$url = $page&"?"&_HTTPEncodeString($vars&$searchstring)

$socket = _HTTPConnect($host)
$get = _HTTPGet($host,$url,$socket)
$recv = _HTTPRead($socket,1)
ConsoleWrite("Data received:"&@CRLF$recv[4]&@CRLF)

Any variables after the question mark need to be HTTP Encoded, which is why I put the "vars" in a separate variable from the page.

Share this post


Link to post
Share on other sites

:"> You know what would be REALLY nice --> something to handle cookies.

Althouhg i have 0 knowledge on cookies, they're still necessary for pages that require

logins :whistle:

Cheers!

Well, you can read the cookies sent by the server using these functions, but I currently don't have a way of sending custom headers with a get or post request - that's something I'll need to add. I might create special cookie handling functions as well.

Share this post


Link to post
Share on other sites

I've just begun playing around with your script and have a few suggestions:

1. Change "Dim $recv = TCPRecv($socket,16)" into "Dim $recv = TCPRecv($socket,4096)" to speed things up (especially when receiving larger files)

2. I'm unable to use this at work due to proxy settings (TCPConnect doesn't like the proxy). Is it possible to add such settings?

3. How about adding a referrer to the _HTTPPost and _HTTPGet

4. It would be nice if the _HTTPSetUserAgent would accept an array of values (*)

(*) My useragent is built out of 3 products and 1 comment: "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7". Here's a link to the 2616RFC

Other than that it's excellent and perfectly suits my needs :)

Share this post


Link to post
Share on other sites

This is a great UDF, I was wondering if it can be used with HTTPS.

it only works if I use www. not http. is there a reason for this?

Thanks


[center]AutoIT + Finger Print Reader/Scanner = COOL STUFF -> Check Out Topic![/center][center][font=Arial Black]Check out ConsultingJoe.com[/font][/center][center]My Scripts~~~~~~~~~~~~~~Web Protocol Managing - Simple WiFi Scanner - AutoTunes - Remote PC Control V2 - Audio SpectrascopePie Chart UDF - At&t's TTS - Custom Progress Bar - Windows Media Player Embed[/center]

Share this post


Link to post
Share on other sites

For https you have to add an extra parameter to the _HTTPConnect-function call. Instead of _HTTPConnect("www.yourserver.com"), make it _HTTPConnect("www.yourserver.com",443)

443 is the default port number for ssl :lmao:

Share this post


Link to post
Share on other sites

For https you have to add an extra parameter to the _HTTPConnect-function call. Instead of _HTTPConnect("www.yourserver.com"), make it _HTTPConnect("www.yourserver.com",443)

443 is the default port number for ssl :lmao:

Really, Thanks. I was trying 80.

So that means that instead of https://www.site.com its just www.site.com:443?


[center]AutoIT + Finger Print Reader/Scanner = COOL STUFF -> Check Out Topic![/center][center][font=Arial Black]Check out ConsultingJoe.com[/font][/center][center]My Scripts~~~~~~~~~~~~~~Web Protocol Managing - Simple WiFi Scanner - AutoTunes - Remote PC Control V2 - Audio SpectrascopePie Chart UDF - At&t's TTS - Custom Progress Bar - Windows Media Player Embed[/center]

Share this post


Link to post
Share on other sites

I tried that and it didn't work.

Here is the Script I am using:

#include <HTTP.au3>
$from = "JOE"
$number = "***-***-****"
$msg = "hi"

$host = "wmg.tmomail.net"
$page = "/keys/write.php"
$vars = "trackResponses=No&Send.x=Yes&DOMAIN_NAME=@tmomail.com&min=" & $number & "&require_sender=" & $from & "&text=" & $msg & "count=" & 140-StringLen($msg) & "&msgTermsUse=1"

$data = _HTTPEncodeString($vars)
ConsoleWrite($data & @CRLF)
$socket = _HTTPConnect($host, 443)
$post = _HTTPPost($host, $page, $socket, $data)
ConsoleWrite(@error & @CRLF)
$recv = _HTTPRead($socket,1)
FileWrite( @ScriptDir & "\result.html", $recv[4] )

[center]AutoIT + Finger Print Reader/Scanner = COOL STUFF -> Check Out Topic![/center][center][font=Arial Black]Check out ConsultingJoe.com[/font][/center][center]My Scripts~~~~~~~~~~~~~~Web Protocol Managing - Simple WiFi Scanner - AutoTunes - Remote PC Control V2 - Audio SpectrascopePie Chart UDF - At&t's TTS - Custom Progress Bar - Windows Media Player Embed[/center]

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

I tried that and it didn't work.

I've tried the url in a browser from above code example, but it gives me a 404-not found-message.

So HTTP UDF *does* seem to work, but you are probably using the wrong url.

Edit: the 404-error could be generated by the php-file

Edited by martijn

Share this post


Link to post
Share on other sites

I've tried the url in a browser from above code example, but it gives me a 404-not found-message.

So HTTP UDF *does* seem to work, but you are probably using the wrong url.

Edit: the 404-error could be generated by the php-file

The address should be: https://wmg.tmomail.net/customer_site/jsp/messaging_lo.jsp

So how do I format it?

Thanks


[center]AutoIT + Finger Print Reader/Scanner = COOL STUFF -> Check Out Topic![/center][center][font=Arial Black]Check out ConsultingJoe.com[/font][/center][center]My Scripts~~~~~~~~~~~~~~Web Protocol Managing - Simple WiFi Scanner - AutoTunes - Remote PC Control V2 - Audio SpectrascopePie Chart UDF - At&t's TTS - Custom Progress Bar - Windows Media Player Embed[/center]

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

It's a shame I didn't see that this script does allow to connect to a server with a ssl-portnumber (443), but that it doesn't support ssl-communication. In other words: it's sending a http request over port 443, where in fact it should send a https-request (ssl encrypted) over port 443. That is not implemented. I suggest to use something like Automation. I've illustrated with a little example:

#include <IE.au3>

Dim $oIE, $o_form, $o_from, $o_to, $o_msg, $o_terms, $b_showbrowser = true
Dim $from = "Joe", $to = "3334445555", $msg = "Hi again"

; Create a browser window and navigate to tool
    $url = "https://wmg.tmomail.net/customer_site/jsp/messaging_lo.jsp"
    $oIE = _IECreate ($url, 0, $b_showbrowser)
; get pointers to the form and from, to, message and terms fields
    $o_form = _IEFormGetObjByName ($oIE, "message_form")
    $o_from = _IEFormElementGetObjByName ($o_form, "require_sender")
    $o_to = _IEFormElementGetObjByName ($o_form, "min")
    $o_msg = _IEFormElementGetObjByName ($o_form, "text")

    _IEFormElementCheckBoxSelect($o_form, 0, "", 1, "byIndex")

; Set field values and submit the form
    _IEFormElementSetValue ($o_from, $from)
    _IEFormElementSetValue ($o_to, $to)
    _IEFormElementSetValue ($o_msg, $msg)
    _IEFormSubmit ($o_form)
    _IELoadWait($oIE)
    _IEQuit ($oIE)

Ofcourse, uncomment the IEQuit to see the result of this code ;-)

Works for me :lmao:

Edited by martijn

Share this post


Link to post
Share on other sites

Hey To start of great script!

just when im using the HTTP example, im constatly getting a

_HTTPRead Error: 5

_HTTPRead Return Value: HTTP/1.1 200 OK

Reply. ive tried heaps of diffrent addresses,

any clue whats going on?

it may be a proxy server, but IE is set for it, and one of the address i am using is localhost anyway.

any help would be awsome.

Share this post


Link to post
Share on other sites

Hey To start of great script!

just when im using the HTTP example, im constatly getting a

_HTTPRead Error: 5

_HTTPRead Return Value: HTTP/1.1 200 OK

Reply. ive tried heaps of diffrent addresses,

any clue whats going on?

it may be a proxy server, but IE is set for it, and one of the address i am using is localhost anyway.

any help would be awsome.

It because this script wasn't intended for AutoIT v3.2.2.0, i tried it with an earlier version of AU3 and it works... could someone please update it?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0