WinHTTP functions

[Potarski]

delete psot please

Edited February 23, 2011 by Potarski

[cypher175]

_WinHttpSetOption($hRequest, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_REDIRECTS)

Does this Code Enable or Disable Redirects..?

I was trying to get a cookie from a website and could only get it by using that code.

but when I tried to longin to a website using that same code in my script, I couldn't login..?

someone explain me whats going on with both of these Response Headers: HTTP/1.1 200 OK & HTTP/1.1 302 Found...?

[cypher175]

If using this WinHTTP Function within multiple .exe's simultaneously, will it flood the DLL call or something..??

Cause I have 200 .exe's all running simultaneously performing some database gathering on a website using my Win7 X64 i7 980X CPU with 12GB Ram PC and I'm noticing some stalling/lagging ever few seconds..

Not sure if its the website lagging or the multiple calls to use the WinHTTP Function from all the multiple processes..?

Im also using multiple fast proxies so I can overcome the max connections per IP to the server restriction..

What all can I check for to see what might be causing this Lag..?

If I use only 4 processes of my script, I notice no lag at all..?

so what do guize..?

[trancexx]

Every process runs in its own memory space. No dll flood is possible.

As for redirections follow the link.

[ProgAndy]

_WinHttpSetOption($hRequest, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_REDIRECTS)

Does this Code Enable or Disable Redirects..?

This disables redirection.

I was trying to get a cookie from a website and could only get it by using that code.

but when I tried to longin to a website using that same code in my script, I couldn't login..?

someone explain me whats going on with both of these Response Headers: HTTP/1.1 200 OK & HTTP/1.1 302 Found...?

302 Found means: The current URL does not exists, but the document was found at another location. The browser should look there. Since you have disabled redirects, you have to do this manually. Read the loaction header, which should contain the new url.

200 OK means, that everything worked and you got the document you requested.

[cypher175]

is there a way to clear the cookies that get cached using these winhttp functions at all..?

[republican]

I don't know why I receive ERROR_INTERNET_INCORRECT_HANDLE_STATE when try to get responds from 'webmail.stu.edu.cn' in either windows 7 x86 or x64(caused by _WinHttpQueryDataAvailable()), here is the simple code.

#include "WinHttp.au3"

Global $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.3; .NET CLR 2.0.50727; ffco7) Gecko/2008092417 Firefox/3.0.3")

Global $hConnect = _WinHttpConnect($hOpen, "webmail.stu.edu.cn")

$Var = _WinHttpSimpleRequest($hConnect)
If @error Then MsgBox(64,@error,$Var)
; Close handles
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

Thanks

[ProgAndy]

The error is raised by _WinHttp_ReceiveResponse, but I don't know the cause. Probably the server does not answer to our request, but I cannot understand why this would happen.

[republican]

The error is raised by _WinHttp_ReceiveResponse, but I don't know the cause. Probably the server does not answer to our request, but I cannot understand why this would happen.

Yet, this error is very strange. I try to use 'xmlhttp' but also gets an error.

I think is a bug of WinHttp.dll. What's your opinion?

Now my way to solve this problem is to use WinInet. It works well in windows 7.

[trancexx]

It's not a bug. WinHTTP refuses to process invalid responses by the server. This is valid reaction. If some other API allows it, use that.

What's invalid?

HTTP/1.1 200 OKDate: Fri, 11 Mar 2011 17:57:45 GMTServer: Apache/2.2.6 (Unix) mod_ssl/2.2.6 đˇ'    PHP/5.2.4X-Powered-By: PHP/5.2.4Set-Cookie: USER=deleted; expires=Thu, 11-Mar-2010 17:57:44 GMTTransfer-Encoding: chunkedContent-Type: text/htmlX-Pad: avoid browser bug

...That Server line.

edit:

#include <WinAPI.au3>
#include "WinHttp.au3"

Global $hOpen = _WinHttpOpen()
Global $hConnect = _WinHttpConnect($hOpen, "webmail.stu.edu.cn")
Global $hRequest = _WinHttpOpenRequest($hConnect)
_WinHttpSendRequest($hRequest)
_WinHttpReceiveResponse($hRequest)
If _WinAPI_GetLastError() = $ERROR_WINHTTP_INVALID_SERVER_RESPONSE Then MsgBox(48 + 262144, "Error 1", "ERROR_WINHTTP_INVALID_SERVER_RESPONSE")

; Get full header
Global $sHeader = _WinHttpQueryHeaders($hRequest)
; As the result of the previous error
If _WinAPI_GetLastError() = $ERROR_WINHTTP_INCORRECT_HANDLE_STATE Then MsgBox(48 + 262144, "Error 2", "ERROR_WINHTTP_INCORRECT_HANDLE_STATE")

_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

ConsoleWrite($sHeader & @CRLF)

Edited March 11, 2011 by trancexx

[4ggr35510n]

Thank you so much trancexx and ProgAndy for this terrific UDF and great examples helping get started with it!

I was just wondering is it possible to "converse" somehow HTML document into an XML* on which it would be possible to operate with javascript to obtain some data from the page?

* I guess HTML is already an extended version of XML... So it's not the smartest question ever, but I hope you'll get my point

And wouldn't it be a game not worth a candle? I guess every .getElementsBy* operations can be successfully substituted with RegExp's functions, as in _WinHttpSimpleFormFill...

Thank you in advance!

Edited March 13, 2011 by 4ggr35510n

[ProgAndy]

@4ggr35510n: It depends on the format whether you can parse it as XML. If it is valid XHTML you can do it, but many websites are written in "normal" wich is not XML conform. And the websites in XHTML are often not valid, so a strict parser won't accept it.

Maybe someone is willing to give libxml2 a try, but currently I don't feel like creating a UDFs from scratch.

[4ggr35510n]

I don't feel like needing UDF :]

But it would be very helpful if you could just point me a bit in a right direction

I can handle with all autoit DllCall/DllStruct functions.

I believe most functions I need can be found here:

http://xmlsoft.org/html/libxml-HTMLparser.html

But I have some language and knowledge issues to understanding the proper way to do it.

Should I just use htmlCreateMemoryParserCtxt() to create ParserCtxt from "char" Structure (received as in _WinHTTPReadData()) and then htmlCtxtReadDoc() to "parse an XML in-memory document and build a tree"?

I know I don't know a lot about that

I'll really appreciate you help!

Hoping for your help,

4gr

[chipDE]

Hi,

can someone tell me how I use WinHTTP at a proxy with authentication?

I have thie script:

#include <winhttp.au3>

$sProxyServer = "PROXY"
$hw_open = _WinHttpOpen("AutoIt v3 WinHTTP with Proxy ", $WINHTTP_ACCESS_TYPE_NAMED_PROXY, $sProxyServer)
_WinHttpSetOption($hw_open, $WINHTTP_OPTION_PROXY_PASSWORD, "PASSWORD")
_WinHttpSetOption($hw_open, $WINHTTP_OPTION_PROXY_USERNAME, "USERNAME")

$hw_connect = _WinHttpConnect($hw_open, "www.wieistmeineip.de")
$h_openRequest = _WinHttpOpenRequest($hw_connect)
_WinHttpSendRequest($h_openRequest)
_WinHttpReceiveResponse($h_openRequest)
MsgBox(0, '', _WinHttpQueryHeaders($h_openRequest))
If _WinHttpQueryDataAvailable($h_openRequest) Then
    Local $data=""
    Do
        $data&=_WinHttpReadData($h_openRequest)
    Until @error
    MsgBox(0, '', $data)
EndIf

_WinHttpCloseHandle($h_openRequest)
_WinHttpCloseHandle($hw_connect)
_WinHttpCloseHandle($hw_open)

But the MsgBox only deliver these errors#

HTTP/1.0 407 Proxy Authentication Required

Connection: close

Date: Mon, 14 Mar 2011 10:51:15 GMT

Via: 1.0 proxy.********.de:8888 (squid/2.7.STABLE9)

Content-Length: 1331

Content-Type: text/html

Proxy-Authenticate: Basic realm="Squid proxy-caching web server"

X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0

X-Cache: MISS from proxy.*********.de

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">

<TITLE>ERROR: Cache Access Denied</TITLE>

<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>

</HEAD>

<BODY>

<H1>ERROR</H1>

<H2>Cache Access Denied</H2>

<HR noshade size="1px">

<P>

While trying to retrieve the URL:

<A HREF="http://www.wieistmeineip.de/">http://www.wieistmeineip.de/</A>

<P>

The following error was encountered:

<UL>

<LI>

<STRONG>

Cache Access Denied.

</STRONG>

</UL>

</P>

<P>Sorry, you are not currently allowed to request:

<PRE> http://www.wieistmeineip.de/</PRE>

from this cache until you have authenticated yourself.

</P>

<P>

You need to use Netscape version 2.0 or greater, or Microsoft Internet

Explorer 3.0, or an HTTP/1.1 compliant browser for this to work. Please

contact the <A HREF="mailto:root">cache administrator</a> if you have

difficulties authenticating yourself or

<A HREF="http://proxy.*******.de/cgi-bin/chpasswd.cgi">change</a> your default password.

</P>

<BR clear="all">

<HR noshade size="1px">

<ADDRESS>

Generated Mon, 14 Mar 2011 10:52:35 GMT by proxy.*******.de (squid/2.7.STABLE9)

</ADDRESS>

</BODY></HTML>

[ProgAndy]

The passwords must be set for each request handle and cannot be stored globally. This means, you cannot use _WinHttpSimpleRequest.

expandcollapse popup

#include<WinHTTP.au3>
#include<WinApi.au3>
Global Const $tagWINHTTP_PROXY_INFO = "DWORD  dwAccessType;ptr lpszProxy;ptr lpszProxyBypass;"

Func _WinHttpProxyInfoCreate($dwAccessType, $sProxy, $sProxyBypass)
    Local $tWINHTTP_PROXY_INFO[2] = [DllStructCreate($tagWINHTTP_PROXY_INFO), DllStructCreate('wchar proxychars[' & StringLen($sProxy)+1 & ']; wchar proxybypasschars[' & StringLen($sProxyBypass)+1 & ']')]
    DllStructSetData($tWINHTTP_PROXY_INFO[0], "dwAccessType", $dwAccessType)
    If StringLen($sProxy) Then DllStructSetData($tWINHTTP_PROXY_INFO[0], "lpszProxy", DllStructGetPtr($tWINHTTP_PROXY_INFO[1], 'proxychars'))
    If StringLen($sProxyByPass) Then DllStructSetData($tWINHTTP_PROXY_INFO[0], "lpszProxyBypass", DllStructGetPtr($tWINHTTP_PROXY_INFO[1], 'proxybypasschars'))
    DllStructSetData($tWINHTTP_PROXY_INFO[1], "proxychars", $sProxy)
    DllStructSetData($tWINHTTP_PROXY_INFO[1], "proxybypasschars", $sProxyBypass)
    Return $tWINHTTP_PROXY_INFO
EndFunc

$hInternet = _WinHttpOpen()

$hConnect = _WinHttpConnect($hInternet, "api.hostip.info")
$standard = _WinHttpSimpleRequest($hConnect, "GET", "/get_html.php")
_WinHttpCloseHandle($hConnect)


$tProxyInfo = _WinHttpProxyInfoCreate($WINHTTP_ACCESS_TYPE_NAMED_PROXY, "127.0.0.1:8123", "localhost")
_WinHttpSetOption($hInternet, $WINHTTP_OPTION_PROXY, $tProxyInfo[0])

$hConnect = _WinHttpConnect($hInternet, "api.hostip.info")

_WinHttpSetOption($hConnect, $WINHTTP_OPTION_PROXY_USERNAME, "test")
_WinHttpSetOption($hConnect, $WINHTTP_OPTION_PROXY_PASSWORD, "pass")

$mit_proxy = ""
$hRequest = _WinHttpOpenRequest($hConnect, "GET", "/get_html.php")
If $hRequest Then
    ; Set password here
    _WinHttpSetCredentials($hRequest, $WINHTTP_AUTH_TARGET_PROXY, $WINHTTP_AUTH_SCHEME_BASIC, "test", "pass")
    If _WinHttpSendRequest($hRequest) Then
        _WinHttpReceiveResponse($hRequest)
        $mit_proxy = _WinHttpSimpleReadData($hRequest)
    EndIf
    _WinHttpCloseHandle($hRequest)
EndIf

_WinHttpCloseHandle($hConnect)

PS: I tested with polipo and this configuration file:

dnsNameServer = "8.8.8.8"
authCredentials = "test:pass"

[guinness]

Is this going to be Fixed in the next release? >>

I was having a problem with Obfuscator and then remembered there was a discussion sometime go about it. I took your advice and edited the UDF and now it works

[trancexx]

There's nothing broken to be fixed.

[guinness]

OK fair point then.

[ProgAndy]

OK fair point then.

Yep. When we make other changes, we maybe could split this long string into parts and add some information to the helpfile that #Obfuscator_Ignore_Variables has to be added to the mainscript, but that's it since #Obfuscator_Ignore_Variables is only recognized in the mainscript. I wish options would be recognized in all UDFs and that there was something like #Obfuscator_Ignore_Parameters_Of_Function=_FunctionName

[NicePerson]

How to upload a file as asked in this thread :