Jump to content
trancexx

WinHTTP functions

Recommended Posts

Posted (edited)

Hello everyone!

Please excuse my probably dumb question and in case this is more suited for a seperate thread please let me know. I've been trying to solve this problem for over a week now and I'm at a point where I'm getting really frustrated. So any help, links, hints are appreciated!

It started with this small script that should help me populate our ecommerce database. It scrapes data from the manufacturers website, then looks for this product in the online shops of our distributors. It's necessary to be logged in to see details like product prices and stock - information we also maintain in our database. 

So far my script is working with Internet Explorer. However, using IE has downsides: It's pretty slow and - worse - sometimes I don't stay logged in for whatever reason. This also happens when browsing manually.

So I thought it might be a good idea to use WinHttp-UDF instead, because it's more straightforward and lightweight. However, I'm having a hard time trying to figure out how I can request a product page after logging in with _WinHttpSimpleFormFill(). 

Here is what I managed to find out on my own:

#include <WinHttp.au3>
#include <String.au3>
#include <Array.au3>

Global Const $g_sZitecUser = "myUsername", $g_sZitecPass = "myPassword", $g_sZitecDomain = "www.zitec-shop.com/de/rillenkugellager-63/p-G1112003141"

Call("LogIn", $g_sZitecUser, $g_sZitecPass)

Func LogIn($sUser, $sPass)
  Local $sFileHTM = @ScriptDir & "\Form.htm"
  ; Open session
  Local $hOpen = _WinHttpOpen()
  ; Connect
  Local $hConnect = _WinHttpConnect($hOpen, "https://www.zitec-shop.com")
  ; Fill in form
  Local $sRead = _WinHttpSimpleFormFill($hConnect, "/de/login", "loginForm", _
        "name:j_username", $g_sZitecUser, _
        "name:j_password", $g_sZitecPass, _
        "type:submit", 1)
  ; This returns the frontpage with me logged in. Great! There is also a CSRF-Token within the response. So I put it in a variable
  If $sRead Then
    ; Get CSRF-Token
    If StringRegExp($sRead, "CSRFToken = '" & "((?:\w+?\-+?)*(?:\w+))" & "';?") Then
      Local $aCSRF[] = StringRegExp($sRead, "CSRFToken = '" & "((?:\w+?\-+?)*(?:\w+))" & "';?", 1)
    Else
      Msgbox(0, "Fehlermeldung", "Es konnte kein CSRF-Token gefunden werden.")
      ; Close handles
      _WinHttpCloseHandle($hConnect)
      _WinHttpCloseHandle($hOpen)
      Exit
    EndIf
  Else
    Msgbox(0, "Fehlermeldung", "Fehlercode: " & @error)
    ; Close handles
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
    Exit
  EndIf

  ; Now I want to request a product page, but even though I receive HTML-Code that implies I am still logged in (username is visible), prices and stock information are still hidden.
  ; How do you request another page and send credentials & token correctly?
  ; My attempt at a request
  Local $sURL = "/de/rillenkugellager-63/p-G1112003141"
  Local $sRequest = _WinHttpSimpleSSLRequest($hConnect, Default, $sURL, Default, "Authorization: Token " & $aCSRF[0]) ; "POST" gives me a 405 error

  If @error Then
    Msgbox(0, "Fehlermeldung", "Fehlercode: " & @error)
    ; Close handles
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
    Exit
  Else
    Msgbox(0, "Die Website hat geantwortet", "Die Website hat geantwortet!")
    $hFileHTM = FileOpen($sFileHTM, 2)
    FileWrite($hFileHTM, $sRequest)
    FileClose($hFileHTM)
    ShellExecuteWait($sFileHTM)
  EndIf

  ; Close handles
  _WinHttpCloseHandle($hConnect)
  _WinHttpCloseHandle($hOpen)
EndFunc ;==> LogIn

I'm quite a newbie. What I don't understand is: How do I figure out how to phrase the content of a header in a post? Like, is it always "username=user&password=pass&token=token" or something else?

Thanks for reading,

Flieder

[Edit]

After staying away from this problem for a few days, I found out that everything is working as intended. I don't have to extract and add the CSRF-Token to my request. The cookie returned with _WinHttpSimpleFormFill() includes every information needed and is stored in the cache as long as the Internethandle stays open.

My problem seems to stem from the getPrice.json! I have no idea how to solve this yet, but I wanted to keep everyone updated who stumbles across this post and faces similiar issues.

Kind regards,

Flieder

And before I forget: Thank you, trancexx and ProgAndy, for creating this UDF. Your work leads to people like me, with little to no network understanding, getting interested in subjects they never would have thought about on their own :)

Edited by Flieder

Share this post


Link to post
Share on other sites
Posted (edited)

Hello everybody!

I’m getting stuck with deflate response from a Http request.

I have used _WinHttpSetOption for enabling but no success. Here is my function:

Func HTTP_PUT ($IP, $URI, ByRef $iStatusCode, $sPostData, $type=-1, $QueryData=True, $verb="PUT")

Local $hOpen = _WinHttpOpen()
_WinHttpSetTimeouts($hOpen, 10000)
Local $hConnect = _WinHttpConnect($hOpen, "http://"&$IP)
Local $hRequest = _WinHttpOpenRequest($hConnect, $verb, $URI)
Local $user,$password
Local $sChunk, $sOut

    Switch $type
        Case 0; setup.html credentials
            $user=$SystemCredentials[0]
            $password=$SystemCredentials[1]
        Case 1 ;Client credentials
            $user=$SystemCredentials[2]
            $password=$SystemCredentials[3]
    EndSwitch

                _WinHttpSetOption($hRequest, $WINHTTP_OPTION_DECOMPRESSION, $WINHTTP_DECOMPRESSION_FLAG_DEFLATE)

                ; Send the request
                _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                _WinHttpWriteData($hRequest, $sPostData)

                ; Wait for the response
                _WinHttpReceiveResponse($hRequest)

                Sleep(100)

                If $QueryData==True Then

                    ; Check if there is any data available and read if yes

                    If _WinHttpQueryDataAvailable($hRequest) Then
                        While 1
                            $sChunk = _WinHttpReadData($hRequest)
                            If @error Then ExitLoop
                            $sOut &= $sChunk
                        WEnd
                    EndIf

                EndIf

                ; Query status code
                $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                ; Check status code
                If $iStatusCode == $HTTP_STATUS_DENIED Or $iStatusCode == $HTTP_STATUS_PROXY_AUTH_REQ Then
                    $sOut=""
                    $sChunk=""
                    ; Query Authorization scheme
                    Local $iSupportedSchemes, $iFirstScheme, $iAuthTarget


                    If _WinHttpQueryAuthSchemes($hRequest, $iSupportedSchemes, $iFirstScheme, $iAuthTarget) Then
                        ; Set passed credentials

                        _WinHttpSetCredentials($hRequest, $iAuthTarget, $iFirstScheme, $user, $password)


                        ; Send request again now

                           _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                            _WinHttpWriteData($hRequest, $sPostData)



                        ; And wait for the response again
                        _WinHttpReceiveResponse($hRequest)

                        Sleep (100)
                        $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                    EndIf

                    If $QueryData==True Then


                    ; Check if there is any data available and read if yes

                        If _WinHttpQueryDataAvailable($hRequest) Then
                            While 1
                                $sChunk = _WinHttpReadData($hRequest)
                                If @error Then ExitLoop
                                $sOut &= $sChunk
                            WEnd
                        EndIf


                    EndIf

                EndIf


    ; Close handles
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

Return $sOut

EndFunc

 

Thanks in advance.

Edited by kaximbeider

Share this post


Link to post
Share on other sites
On 8/12/2019 at 4:52 PM, kaximbeider said:

Hello everybody!

I’m getting stuck with deflate response from a Http request.

I have used _WinHttpSetOption for enabling but no success. Here is my function:

Func HTTP_PUT ($IP, $URI, ByRef $iStatusCode, $sPostData, $type=-1, $QueryData=True, $verb="PUT")

Local $hOpen = _WinHttpOpen()
_WinHttpSetTimeouts($hOpen, 10000)
Local $hConnect = _WinHttpConnect($hOpen, "http://"&$IP)
Local $hRequest = _WinHttpOpenRequest($hConnect, $verb, $URI)
Local $user,$password
Local $sChunk, $sOut

    Switch $type
        Case 0; setup.html credentials
            $user=$SystemCredentials[0]
            $password=$SystemCredentials[1]
        Case 1 ;Client credentials
            $user=$SystemCredentials[2]
            $password=$SystemCredentials[3]
    EndSwitch

                _WinHttpSetOption($hRequest, $WINHTTP_OPTION_DECOMPRESSION, $WINHTTP_DECOMPRESSION_FLAG_DEFLATE)

                ; Send the request
                _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                _WinHttpWriteData($hRequest, $sPostData)

                ; Wait for the response
                _WinHttpReceiveResponse($hRequest)

                Sleep(100)

                If $QueryData==True Then

                    ; Check if there is any data available and read if yes

                    If _WinHttpQueryDataAvailable($hRequest) Then
                        While 1
                            $sChunk = _WinHttpReadData($hRequest)
                            If @error Then ExitLoop
                            $sOut &= $sChunk
                        WEnd
                    EndIf

                EndIf

                ; Query status code
                $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                ; Check status code
                If $iStatusCode == $HTTP_STATUS_DENIED Or $iStatusCode == $HTTP_STATUS_PROXY_AUTH_REQ Then
                    $sOut=""
                    $sChunk=""
                    ; Query Authorization scheme
                    Local $iSupportedSchemes, $iFirstScheme, $iAuthTarget


                    If _WinHttpQueryAuthSchemes($hRequest, $iSupportedSchemes, $iFirstScheme, $iAuthTarget) Then
                        ; Set passed credentials

                        _WinHttpSetCredentials($hRequest, $iAuthTarget, $iFirstScheme, $user, $password)


                        ; Send request again now

                           _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                            _WinHttpWriteData($hRequest, $sPostData)



                        ; And wait for the response again
                        _WinHttpReceiveResponse($hRequest)

                        Sleep (100)
                        $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                    EndIf

                    If $QueryData==True Then


                    ; Check if there is any data available and read if yes

                        If _WinHttpQueryDataAvailable($hRequest) Then
                            While 1
                                $sChunk = _WinHttpReadData($hRequest)
                                If @error Then ExitLoop
                                $sOut &= $sChunk
                            WEnd
                        EndIf


                    EndIf

                EndIf


    ; Close handles
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

Return $sOut

EndFunc

 

Thanks in advance.

I believe that the data served by the server is compressed by deflate inside a zlib stream (RFC1951 + RFC1950), and WinHttp probably works with only former. When you find server that delivers raw deflate (RFC1951) this can be confirmed.

Sorry if I'm no help.

Share this post


Link to post
Share on other sites

Yes, that appears to be the issue.  So, only raw deflate works. For example bing.com will deliver data in raw deflate form if $WINHTTP_DECOMPRESSION_FLAG_DEFLATE is used and that will be processed correctly. However processing postman-echo.com/deflate will give error.

Share this post


Link to post
Share on other sites

Brilliant! Thanks for the response.

I have made some testing with curl and it was working. I was able to read the response. But as I understood I cannot do anything else with winhttp? 

Do I need an extra UDF for using zlib? Or do you have something that I could test?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...