_INetGetSource not working

[Seminko]

Hey fellas,

I have a script that grabs data from hearthpwn.com.

It was working fine for half a year or so but now _INetGetSource started giving me an error. So I googled and came across WinHttp.au3 and this script:

expandcollapse popup

#include <Constants.au3>
#include <WinHttp.au3>

; Designed by guinness 2013.
Global Const $HTTP_GOOD_STATUSCODE = '|' & $HTTP_STATUS_OK & '|' & $HTTP_STATUS_MOVED & '|' & $HTTP_STATUS_REDIRECT & '|' & _
        $HTTP_STATUS_REDIRECT_METHOD & '|' & $HTTP_STATUS_NOT_MODIFIED & '|' & $HTTP_STATUS_REDIRECT_KEEP_VERB & '|' & _
        $HTTP_STATUS_BAD_REQUEST & '|' & $HTTP_STATUS_DENIED & '|' & $HTTP_STATUS_FORBIDDEN & '|' & _
        $HTTP_STATUS_BAD_METHOD & '|'

Example()

Func Example()
    Local $iPort = 80, _
            $sDomain = 'www.hearthpwn.com'

    ; Open the handles.
    Local Const $hOpen = _WinHttpOpen()
    Local Const $hConnect = _WinHttpConnect($hOpen, $sDomain, $iPort)
    Local Const $hRequest = _WinHttpOpenRequest($hConnect, 'HEAD')


    ; Send a request.
    _WinHttpSendRequest($hRequest)

    ; Wait for a response.
    _WinHttpReceiveResponse($hRequest)

    ; Retrieve the http status and check if matches the OK status code.
    Local $sRequestCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)

    ; Check if the status code is in the $HTTP_GOOD_STATUSCODE string.
    Local $fIsUp = StringInStr($HTTP_GOOD_STATUSCODE, '|' & $sRequestCode & '|') > 0

    ; Display value of $fIsUp.
    MsgBox($MB_SYSTEMMODAL, '', 'The return was: ' & $fIsUp & @CRLF)

    ; Close the handles.
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
EndFunc   ;==>Example

When I run this I get 'The return was: False' so I guess something is with the site itself? Is there any way around it?

Here is an example script of the _INetGetSource:

#include <INet.au3>

$Source = _INetGetSource("http://www.hearthpwn.com")
If @error Then
    MsgBox(1, "Error", "Error")
Else
    MsgBox(1, "", $Source)
EndIf

Thanks,

Jan

[rootx]

expandcollapse popup

#include <Constants.au3>
#include <WinHttp.au3>

; Designed by guinness 2013.
Global Const $HTTP_GOOD_STATUSCODE = '|' & $HTTP_STATUS_OK & '|' & $HTTP_STATUS_MOVED & '|' & $HTTP_STATUS_REDIRECT & '|' & _
        $HTTP_STATUS_REDIRECT_METHOD & '|' & $HTTP_STATUS_NOT_MODIFIED & '|' & $HTTP_STATUS_REDIRECT_KEEP_VERB & '|' & _
        $HTTP_STATUS_BAD_REQUEST & '|' & $HTTP_STATUS_DENIED & '|' & $HTTP_STATUS_FORBIDDEN & '|' & _
        $HTTP_STATUS_BAD_METHOD & '|'

Example()

Func Example()
    Local $iPort = 80, _
            $sDomain = 'www.hearthpwn.com'

    ; Open the handles.
    Local Const $hOpen = _WinHttpOpen()
    Local Const $hConnect = _WinHttpConnect($hOpen, $sDomain, $iPort)
    Local Const $hRequest = _WinHttpOpenRequest($hConnect)


    ; Send a request.
    _WinHttpSendRequest($hRequest)

    ; Wait for a response.
    _WinHttpReceiveResponse($hRequest)

    ; Retrieve the http status and check if matches the OK status code.
    Local $sRequestCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)

    ; Check if the status code is in the $HTTP_GOOD_STATUSCODE string.
    Local $fIsUp = StringInStr($HTTP_GOOD_STATUSCODE, '|' & $sRequestCode & '|') > 0

    ; Display value of $fIsUp.
    MsgBox($MB_SYSTEMMODAL, '', 'The return was: ' & $fIsUp & @CRLF)

    ; Close the handles.
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
EndFunc   ;==>Example

Return True

[Seminko]

16 minutes ago, rootx said:

Return True

Not on my end... just checked.

Ideas?

 

EDIT: the script you send is actually updated a bit and it does return true in fact. However, below still doesn't work. No idea why. The site is up and runs smoothly.

_INetGetSource("http://www.hearthpwn.com")

Edited November 12, 2016 by Seminko

[Seminko]

Ok, tried to see what error I get with @extended and apparently as per THIS thread the error means:

Quote

12039 ERROR_INTERNET_HTTP_TO_HTTPS_ON_REDIR
The application is moving from a non-SSL to an SSL connection because of a redirect.

Ok so logically I tried httpS://www.hearthpwn.com and now what we get is the same error the other way around:

Quote

12040 ERROR_INTERNET_HTTPS_TO_HTTP_ON_REDIR
The application is moving from an SSL to an non-SSL connection because of a redirect.

 

Edited November 13, 2016 by Seminko

[Seminko]

Really? Nobody has such error before?

[Seminko]

Guys? My script is dead without getting this sorted.

Noone's seen this happen ever?

[Melba23]

Seminko,

The site you are using looks a lot like a game to me - how about persuading me that it is not?

M23

[Seminko]

Hi Melba,

I see you're not coming to rescue me. Not yet at least 

I hope I've checked the most up to date rules found here --> https://www.autoitscript.com/forum/topic/169097-forum-rules/

The rules state:

Quote

1. Do not ask for help with AutoIt scripts, post links to, or start discussion topics on the following subjects:

• Launching, automation or script interaction with games or game servers, regardless of the game.

My script does not launch, automate or in any way interact with a game or game servers.

The site I linked is related to a game, yes. However, my script only interacts with the site itself, it copies card information into an excel file instead of me spending hours doing it myself.

So PLEASE, guide me!

Very much obliged

 

[Melba23]

Seminko,

OK, I am convinced - but alas I cannot offer any help.

M23

[Seminko]

Bumping again. There has to be a solution.

[AndyG]

#include <Inet.au3>
$a=_INetGetSource("http://www.hearthpwn.com")
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $a = ' & $a & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

shows me the sourcecode...what i am doing wrong?

Edited November 20, 2016 by AndyG

[mikell]

Hmm this works for me...

Msgbox(0,"", BinarytoString(InetRead("http://www.hearthpwn.com")))

 

[Seminko]

6 hours ago, mikell said:

Hmm this works for me...

Msgbox(0,"", BinarytoString(InetRead("http://www.hearthpwn.com")))

 

Well, it doesn't return an error, yes. But what does it return for you? It returns 0 for me. I use _InetGetSource to get the source code.

 

6 hours ago, AndyG said:

#include <Inet.au3>
$a=_INetGetSource("http://www.hearthpwn.com")
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $a = ' & $a & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

shows me the sourcecode...what i am doing wrong?

Does it? All I get is:

@@ Debug(58) : $a = 
>Error code: 13

-------------

Mikell, could you try with _INetGetSource? If it works for you both than the issue has to be on my end. Maybe some internet option?

[mikell]

My previous code - as does the one from AndyG I presume - returns the whole source code of the page
_INetGetSource is nothing but a clean  BinarytoString(InetRead(...))

You could try to set a non-AutoIt user agent using  HttpSetUserAgent()

[Seminko]

55 minutes ago, mikell said:

My previous code - as does the one from AndyG I presume - returns the whole source code of the page
_INetGetSource is nothing but a clean  BinarytoString(InetRead(...))

You could try to set a non-AutoIt user agent using  HttpSetUserAgent()

I tried below code but the result is still the same:

#include <Inet.au3>
HttpSetUserAgent("MyUserAgent")
$a=_INetGetSource("http://www.hearthpwn.com")
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $a = ' & $a & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

It's strange that I'm apparently the first one reporting such issue.

[mikell]

What about this ?

$oHTTP = ObjCreate("Microsoft.XMLHTTP")
$oHTTP.Open("GET", "http://www.hearthpwn.com", 0)
$oHTTP.Send()
$sData = $oHTTP.ResponseText
Msgbox(0,"", $sData)

 

[Seminko]

8 hours ago, mikell said:

What about this ?

$oHTTP = ObjCreate("Microsoft.XMLHTTP")
$oHTTP.Open("GET", "http://www.hearthpwn.com", 0)
$oHTTP.Send()
$sData = $oHTTP.ResponseText
Msgbox(0,"", $sData)

 

The Msgbox is empty

[j0kky]

Let's play at lower level:

expandcollapse popup

TCPStartup()
If @error Then _ReportError(@error, @extended, @ScriptLineNumber - 1)
Local $sIPAddress = TCPNameToIP("www.hearthpwn.com")
If @error Then _ReportError(@error, @extended, @ScriptLineNumber - 1)
Local $iSocket = TCPConnect($sIPAddress, 80)
If @error Then _ReportError(@error, @extended, @ScriptLineNumber - 1)
TCPSend($iSocket, "GET /?cookieTest=1 HTTP/1.1" & @CRLF & _
    "Host: www.hearthpwn.com" & @CRLF & _
    "Connection: close" & @CRLF & @CRLF)
If @error Then _ReportError(@error, @extended, @ScriptLineNumber - 1)
Local $sData = "", $hTimer = TimerInit()
While 1
    $sRecv = _TCPRecv($iSocket, 2048)
    If $sRecv Then $sData &= $sRecv
    If @error Or @extended Then
        If @error Then _ReportError(@error, @extended, @ScriptLineNumber - 1)
        ExitLoop
    EndIf
    If $sRecv Then $hTimer = TimerInit()
    If TimerDiff($hTimer) > 3000 Then ExitLoop
    Sleep(10)
WEnd
ConsoleWrite("Response received: " & $sData & @CRLF)
TCPCloseSocket($iSocket)
TCPShutdown()

Func _ReportError($iError, $iExtended, $iLine)
    ConsoleWrite("Error: " & $iError & @CRLF & _
        "Extended: " & $iExtended & @CRLF & _
        "Line: " & $iLine & @CRLF)
    TCPShutdown()
    Exit
EndFunc

Func _TCPRecv($iMainsocket, $iMaxLen, $iFlag = 0)
    If IsArray($iMainsocket) And (UBound($iMainsocket, 0) = 1) And (UBound($iMainsocket) > 0) Then $iMainsocket = $iMainsocket[0]
    If $iFlag = Default Then $iFlag = 0
    $iMainsocket = Number($iMainsocket)
    $iMaxLen = Number($iMaxLen)
    $iFlag = Number($iFlag)
    If $iMainsocket < 0 Or _
            $iMaxLen < 1 Or _
            Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(-4, 0, -1) ; invalid parameter

    Local $hWs2 = DllOpen("Ws2_32.dll")
    If @error Then Return SetError(-2, 0, -1) ;missing DLL
    Local $bError = 0, $nCode = 0, $nExtended = 0

    If Not $bError Then
        $aRet = DllCall($hWs2, "int", "ioctlsocket", "uint", $iMainsocket, "long", 0x8004667e, "ulong*", 1) ;FIONBIO
        If @error Then
            $bError = -1
        ElseIf $aRet[0] <> 0 Then ;SOCKET_ERROR
            $bError = 1
        EndIf
    EndIf

    Local $tBuf = DllStructCreate("char[" & $iMaxLen & "]")
    $aRet = DllCall($hWs2, "int", "recv", "uint", $iMainsocket, "ptr", DllStructGetPtr($tBuf), "int", $iMaxLen, "int", 0)
    If @error Then
        $bError = -1
    ElseIf ($aRet[0] = -1) Or ($aRet[0] = 4294967295) Then ;SOCKET_ERROR
        $bError = 1
        $aRet = DllCall($hWs2, "int", "WSAGetLastError")
        If @error Then
            $bError = -1
        ElseIf $aRet[0] = 0 Or $aRet[0] = 10035 Then ;WSAEWOULDBLOCK
            $nCode = -10 ;internal function value, it means no error
        EndIf
    ElseIf $aRet[0] = 0 Then
        $bError = 1
        $nCode = -10
        $nExtended = 1 ;connection closed
    Else
        Local $sResult = DllStructGetData($tBuf, 1) ;data
        If BitAND($iFlag, 2) = 2 Then ;EOT
            If StringRight($sResult, 1) = Chr(3) Then
                $sResult = StringTrimRight($sResult, 1)
                $nExtended = 2 ;End of Text reached
            EndIf
        EndIf
        If BitAND($iFlag, 1) = 1 Then $sResult = Binary($sResult)
    EndIf

    If $bError < 0 Then
        $nCode = -1 ;internal error
        $nReturn = "" ;failure
    ElseIf $bError > 0 Then
        If Not $nCode Then
            $aRet = DllCall($hWs2, "int", "WSAGetLastError")
            If @error Then
                $nCode = -1
            Else
                $nCode = $aRet[0]
            EndIf
            If $nCode = 0 Then $nCode = -3 ;undefined error
        EndIf
        If $nCode = -10 Then $nCode = 0
        $nReturn = ""
    Else
        $nReturn = $sResult
    EndIf
    DllClose($hWs2)
    Return SetError($nCode, $nExtended, $nReturn)
EndFunc   ;==>_TCPRecv

Post here the console output.

Edited November 22, 2016 by j0kky

[Seminko]

Strange strange, your code, j0kky, actually returned the source. Here is the console:

Response received: HTTP/1.1 200 OK
Date: Tue, 22 Nov 2016 16:39:34 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=d865e594f7af57763539bb79728c06fb01479832774; expires=Wed, 22-Nov-17 16:39:34 GMT; path=/; domain=.hearthpwn.com; HttpOnly
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Vary: Accept-Encoding
X-AspNetMvc-Version: 1.1
X-AspNet-Version: 4.0.30319
X-Cobalt-Cache: Cached
X-UA-Compatible: IE=edge,chrome=1
X-Frame-Options: SAMEORIGIN
Server: cloudflare-nginx
CF-RAY: 305dd17796573e32-PRG

e673

.
.
.

site code

So I tried _INetGetSource again, just to make sure it has not started working all of the sudden. I has not.

[j0kky]

Try to execute AndyG and Mikell codes replacing:

"http://www.hearthpwn.com"

with:

"http://www.hearthpwn.com/?cookieTest=1"

 

Edited November 22, 2016 by j0kky