Sign in to follow this  
Followers 0
SoulA

WinHttpRequest Woes

11 posts in this topic

Trying to gather data from a web page and my initial plan was to use WinHttpRequest. The problem is when I go through this page, login and send the post data the page has a weird redirect that it does. So it will have a "logging in page" which redirects you to where you actually want to go. So this is the response text I get instead of the actual page I want. Is there anyway I can get the page after that page? I thought WinHTTP will follow redirects unless told otherwise.

Here is the code I am using. Should I just browse to the page and then browse to it again to avoid getting the "logging in" response? I can somehow save the cookie or something to avoid having to login again.

$sURL = "URL" $oHTTP = ObjCreate("WinHttp.WinHttpRequest.5.1") 
$oHTTP.Open("POST", $sURL, false) 
$oHTTP.SetCredentials("user", "pass", 0) $sPostData("post data") 
$oHTTP.Send($sPostData) 
FileWrite($sFilePath, $oHTTP.ResponseText)

Share this post


Link to post
Share on other sites



Still looking for help on this. Where are you Dale and Manadar.

Share this post


Link to post
Share on other sites

:D Thank you for your confidence in me. I've never used any WinHttp library however. I always found it easier to write my own webclient using the TCP functions.

It is possible that the screen you get after login is a screen that uses a HTML or Javascript redirect. I think if WinHttp redirects at all it will only be with 30x HTTP replies.

Share this post


Link to post
Share on other sites

:D Thank you for your confidence in me. I've never used any WinHttp library however. I always found it easier to write my own webclient using the TCP functions.

It is possible that the screen you get after login is a screen that uses a HTML or Javascript redirect. I think if WinHttp redirects at all it will only be with 30x HTTP replies.

Yeah that is the issue. It does use an HTML redirect. This is a small task and writing my own webclient with the TCP functions seems like overkill at this point unless you already have some decent code you'd like to share that could help me get on the right track.

Share this post


Link to post
Share on other sites

Ok, in that case you need to get the HTML redirect tag from the response text and parse it. Then navigate to the new URL.

A HTTP client does not handle HTML or Javascript redirects. I think that's a responsibility of the DOM, which a HTTP client does not have. Writing your own method to do a HTTP request using TCP functions won't help in this case either. Doing the request yourself is not magically going to create a DOM. 8)

But, DOM is more Dale's area of expertise. : )

Share this post


Link to post
Share on other sites

Ok, in that case you need to get the HTML redirect tag from the response text and parse it. Then navigate to the new URL.

A HTTP client does not handle HTML or Javascript redirects. I think that's a responsibility of the DOM, which a HTTP client does not have. Writing your own method to do a HTTP request using TCP functions won't help in this case either. Doing the request yourself is not magically going to create a DOM. 8)

But, DOM is more Dale's area of expertise. : )

It is a weird setup because it just presents you with a "logging in" page that redirects you to the URL you were going to in the first place. Then if I try and navigate there again using the same code I get the same thing. It's like I need to setup the cookie or something.

Share this post


Link to post
Share on other sites

You can see what requests your script makes if you attach a packet sniffer like Wireshark. (Or for quick scans I actually like WPE Pro 0.9x...)

If you have problems understanding the HTTP then post back here.

Share this post


Link to post
Share on other sites

It is a weird setup because it just presents you with a "logging in" page that redirects you to the URL you were going to in the first place. Then if I try and navigate there again using the same code I get the same thing. It's like I need to setup the cookie or something.

Disable redirection.

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

You can see what requests your script makes if you attach a packet sniffer like Wireshark. (Or for quick scans I actually like WPE Pro 0.9x...)

If you have problems understanding the HTTP then post back here.

HTTP Header examination is built into DebugBar - nice if you already have it. Fiddler also works well here.

I must admit that I am confused by the wording of your question... are you trying to prevent the redirect? are you trying to trap the redirect? are you trying to follow the redirect?

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

HTTP Header examination is built into DebugBar - nice if you already have it. Fiddler also works well here.

I must admit that I am confused by the wording of your question... are you trying to prevent the redirect? are you trying to trap the redirect? are you trying to follow the redirect?

Dale

I will look at the results of debug bar and I have fiddler and I'll give some info that it returns. To be more clear I want to follow the redirect. The redirect is an HTML redirect so I guess WinHTTPRequest doesn't follow it. The response text that I get is the "logging in" text instead of the page that you would get redirected to after seeing that.

Using the code that I posted previously I am only get the "logging in" text. I'm guessing that it is because I'm adding the set credentials every time instead of somehow telling the page that I'm already authenticated with a cookie or something of that nature? How would I tell the page that I'm already authed for this site and to take me straight to the page I want to see without taking me to the "logging in" page?

Is that all making sense?

Share this post


Link to post
Share on other sites

The HTTP headers should show you the type of authentication being used and will show you any cookie data being sent.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0