WinHttpRequest Woes

[SoulA]

Trying to gather data from a web page and my initial plan was to use WinHttpRequest. The problem is when I go through this page, login and send the post data the page has a weird redirect that it does. So it will have a "logging in page" which redirects you to where you actually want to go. So this is the response text I get instead of the actual page I want. Is there anyway I can get the page after that page? I thought WinHTTP will follow redirects unless told otherwise.

Here is the code I am using. Should I just browse to the page and then browse to it again to avoid getting the "logging in" response? I can somehow save the cookie or something to avoid having to login again.

$sURL = "URL" $oHTTP = ObjCreate("WinHttp.WinHttpRequest.5.1") 
$oHTTP.Open("POST", $sURL, false) 
$oHTTP.SetCredentials("user", "pass", 0) $sPostData("post data") 
$oHTTP.Send($sPostData) 
FileWrite($sFilePath, $oHTTP.ResponseText)

[SoulA]

Still looking for help on this. Where are you Dale and Manadar.

[jvanegmond]

Thank you for your confidence in me. I've never used any WinHttp library however. I always found it easier to write my own webclient using the TCP functions.

It is possible that the screen you get after login is a screen that uses a HTML or Javascript redirect. I think if WinHttp redirects at all it will only be with 30x HTTP replies.

[SoulA]

Thank you for your confidence in me. I've never used any WinHttp library however. I always found it easier to write my own webclient using the TCP functions.

It is possible that the screen you get after login is a screen that uses a HTML or Javascript redirect. I think if WinHttp redirects at all it will only be with 30x HTTP replies.

Yeah that is the issue. It does use an HTML redirect. This is a small task and writing my own webclient with the TCP functions seems like overkill at this point unless you already have some decent code you'd like to share that could help me get on the right track.

[jvanegmond]

Ok, in that case you need to get the HTML redirect tag from the response text and parse it. Then navigate to the new URL.

A HTTP client does not handle HTML or Javascript redirects. I think that's a responsibility of the DOM, which a HTTP client does not have. Writing your own method to do a HTTP request using TCP functions won't help in this case either. Doing the request yourself is not magically going to create a DOM. 8)

But, DOM is more Dale's area of expertise. : )

[SoulA]

Ok, in that case you need to get the HTML redirect tag from the response text and parse it. Then navigate to the new URL.

A HTTP client does not handle HTML or Javascript redirects. I think that's a responsibility of the DOM, which a HTTP client does not have. Writing your own method to do a HTTP request using TCP functions won't help in this case either. Doing the request yourself is not magically going to create a DOM. 8)

But, DOM is more Dale's area of expertise. : )

It is a weird setup because it just presents you with a "logging in" page that redirects you to the URL you were going to in the first place. Then if I try and navigate there again using the same code I get the same thing. It's like I need to setup the cookie or something.

[jvanegmond]

You can see what requests your script makes if you attach a packet sniffer like Wireshark. (Or for quick scans I actually like WPE Pro 0.9x...)

If you have problems understanding the HTTP then post back here.

[trancexx]

It is a weird setup because it just presents you with a "logging in" page that redirects you to the URL you were going to in the first place. Then if I try and navigate there again using the same code I get the same thing. It's like I need to setup the cookie or something.

Disable redirection.

[DaleHohm]

You can see what requests your script makes if you attach a packet sniffer like Wireshark. (Or for quick scans I actually like WPE Pro 0.9x...)

If you have problems understanding the HTTP then post back here.

HTTP Header examination is built into DebugBar - nice if you already have it. Fiddler also works well here.

I must admit that I am confused by the wording of your question... are you trying to prevent the redirect? are you trying to trap the redirect? are you trying to follow the redirect?

Dale

[SoulA]

HTTP Header examination is built into DebugBar - nice if you already have it. Fiddler also works well here.

I must admit that I am confused by the wording of your question... are you trying to prevent the redirect? are you trying to trap the redirect? are you trying to follow the redirect?

Dale

I will look at the results of debug bar and I have fiddler and I'll give some info that it returns. To be more clear I want to follow the redirect. The redirect is an HTML redirect so I guess WinHTTPRequest doesn't follow it. The response text that I get is the "logging in" text instead of the page that you would get redirected to after seeing that.

Using the code that I posted previously I am only get the "logging in" text. I'm guessing that it is because I'm adding the set credentials every time instead of somehow telling the page that I'm already authenticated with a cookie or something of that nature? How would I tell the page that I'm already authed for this site and to take me straight to the page I want to see without taking me to the "logging in" page?

Is that all making sense?

[DaleHohm]

The HTTP headers should show you the type of authentication being used and will show you any cookie data being sent.

Dale