InetRead does not work on certain pages: why?

[Strydr]

I've been playing around with arrays and hit a stumper (for me).  I ended up  in TutsTeach's 7th Tutorial about arrays and it seemed to explain what I needed just fine.  However, when attempting to use it on a different website, it doesn't work.  I can use a combination of _INetGetSource and ObjCreate("WinHttp.WinHttpRequest.5.1" to get the file (not really sure how this works, the code is from someone else's work)  and get information from it, but not InetRead.  I'm guessing that it has something to do with how the website in question is structured but I have no idea how to get around it, if I can.

The working website from TutsTeach's Tutorial is:

https://www.youtube.com/playlist?list=PLNpExbvcyUkOJvgxtCPcKsuMTk9XwoWum

and the one I'm trying to get is:

https://steamdb.info/app/488040/info/

 

Anybody know why?  Or, better yet, explain a work around?  The only other post that I could find directly addressing this issue was never answered...

Thanks for the help!

[JLogan3o13]

How about posting the code you are working with, rather than having someone sit through 15 minutes of video. Help us help you

[Strydr]

OMG!  I didn't  realize that I was doing that to you guys!  My sincere apologies!!

Quote

#include <String.au3>

$Source = BinaryToString(InetRead('https://www.youtube.com/playlist?list=PLNpExbvcyUkOJvgxtCPcKsuMTk9XwoWum', 1))
;$Source = BinaryToString(InetRead('https://steamdb.info/app/488040/info/', 1))
$FirstChunks = _StringBetween($Source, 'content="', '">')

For $a In $FirstChunks
    ConsoleWrite($a & @CRLF)
Next

The script is  TutsTeach's example.  If you REM out the first "$Source" line and un-REM the second one, it only changes the website.  But it stops working.  I've played around with the code enough to establish that the error is occurring at the "InetRead" part. I think.

I hope that helps you help me, and, again, my apologies to anyone who was sifting thru that video to figure out what my problem is!!!!

Edited October 2, 2018 by Strydr

[Strydr]

Ok, this is what I've come up with.

First, try not to use any INet commands in general.  (Edit: as Trancexx points out, you can use the HttpSetUserAgent to set Autoit to NOT use IE.  Tested just fine in my program!  Thank you, Trancexx!)   Apparently, what's happening is that INet commands use IE explorer security settings .  It seems that your OS will arbitrarily decide that the security certificate for that site is invalid. So, in effect, what will happen, is that any program that you have that uses it, will randomly stop working on that page.  No scraping, no copying, no working with that page at all.  Once it stops, it won't restart, your program is done without compromising the IE security settings of the user's computer.  So a program that works, may stop working.  A program that works on one computer may not work on another.  Not a very comforting situation, hence my recommendation against using INet commands in general.

Instead, make an object out of the site (example scripts to follow) and add a heading that tells Autoit NOT to use IE, but to emulate Mozilla.

Quote

Local $WinHttpReq = ObjCreate("WinHttp.WinHttpRequest.5.1")
        If Not @error Then
            $WinHttpReq.Open("GET", "user url", false)

            $WinHttpReq.SetRequestHeader("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 4.0.20506)")         ;this line MUST be added to prevent Autoit from using IE

            $WinHttpReq.Send()
            Local $Data = $WinHttpReq.ResponseText  ;$Data is now a string that can be worked with

 

Another way is to use the http protocols included with the <WinHttp.au3> but I suspect that this actually takes longer due to opening and closing winsocks(?).

This example is courtesy of Danp2:

Quote

#include <String.au3>
#include <Array.au3>
#include <WinHttp.au3> ; https://www.autoitscript.com/forum/topic/84133-winhttp-functions/

Local $sResponseText, $iResult = 0
Local $sURL = "https://steamdb.info/app/264710/info/"
Local $Array_32bit_Icons

Local $aURL = _WinHttpCrackUrl($sURL)
Local $hOpen = _WinHttpOpen()

; Get connection handle
Local $hConnect = _WinHttpConnect($hOpen, $aURL[2], $aURL[3])

If @error Then
    $iResult = 1
Else
    $sResponseText = _WinHttpSimpleSSLRequest($hConnect, "GET", $aURL[6])

    If @error Then
        $iResult = 2
    Else
        $Array_32bit_Icons = _StringBetween($sResponseText, 'avatar" src=', '" alt=')
        _ArrayDisplay($Array_32bit_Icons)
    EndIf
EndIf

_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

ConsoleWrite("$iResult=" & $iResult & @CRLF)
ConsoleWrite("$sResponseText=" & $sResponseText & @CRLF)

 

This also works, but you need to go to the Github page to download and install the latest WinHttp.au3 file, it does NOT come with Autoit.

All three of these methods work.  My guessing says that the fastest would be the Inet command, then the object creation method, and the slowest would be the http method, but I wouldn't put money on it.  When I'm a little further in my project, I will test as I've got enough entries in my main program that any speed differences should show up.

Hope this helps someone!

Edited October 10, 2018 by Strydr

[trancexx]

There's nothing wrong using Inet…() functions for such simple tasks. That site probably just checks user agent string. Just do something like this and see if it works:

;...
HttpSetUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134")
$Source = BinaryToString(InetRead('https://steamdb.info/app/488040/info/', 1))
;...

 

[Strydr]

!!    I didn't know that I could do that!  I will test and edit my post accordingly, thank you!!!  Sounds as if anybody attempting to use it just needs to know how to use the 'set user agent' properly.

Do you have any idea which method is fastest?  My ignorant guess would be that the InetRead command would be fastest because I'm not taking the extra step to make an object?

Thank you, again, for your help!!!

-I'm already getting old, I'm gonna be a freakin' fossil by the time I have any idea wth I'm doing!  So much to learn, so little time...

Edited October 8, 2018 by Strydr