InetGet help - hit and miss downloads

thegoodguy · February 26, 2022

9 hours ago, TheXman said:

Let's see if the problematic URL works with WinHTTP. If the script below fails, at least it should provide us with a little more information.

Can you try running the script and posting your console results?

Test script

#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w 5 -w 6 -d

#include <Constants.au3>

test()

Func test()
    Local $oComErr = Null

    ;Set up local COM error handler
    $oComErr = ObjEvent("AutoIt.Error", com_error_handler)
    #forceref $oComErr

    With ObjCreate("winhttp.winhttprequest.5.1")
        ;Set up & send request
        .Open("GET", "https://www.britannica.com/dictionary/bum")
        .Send()

        ;Display response
        ConsoleWrite("HTTP Response Code: " & .status & " (" & .StatusText & ")" & @CRLF & @CRLF)
        ConsoleWrite("HTTP Response Headers:" & @CRLF)
        ConsoleWrite(.GetAllResponseHeaders())
        ConsoleWrite("HTTP Response (First 500 bytes):" & @CRLF)
        ConsoleWrite(StringLeft(.ResponseText, 500) & @CRLF) ;First 500 bytes of response
    EndWith

EndFunc

Func com_error_handler($oError)
    With $oError
        ConsoleWrite(@CRLF & "COM ERROR DETECTED!" & @CRLF)
        ConsoleWrite("  Error ScriptLine....... " & .scriptline & @CRLF)
        ConsoleWrite("  Error Number........... " & "0x" & Hex(.number) & " (" & .number & ")" & @CRLF)
        ConsoleWrite("  Error WinDescription... " & StringStripWS(.windescription, $STR_STRIPTRAILING) & @CRLF)
        ConsoleWrite("  Error RetCode.......... " & "0x" & Hex(Number(.retcode)) & " (" & Number(.retcode) & ")" & @CRLF)
        ConsoleWrite("  Error Description...... " & StringStripWS(.description   , $STR_STRIPTRAILING) & @CRLF)
    EndWith
    MsgBox($MB_ICONERROR + $MB_TOPMOST, "ERROR", "A COM error has occurred.  See console for details.")
    Exit
EndFunc

My output:

HTTP Response Code: 200 (OK)

HTTP Response Headers:
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Date: Sat, 26 Feb 2022 00:52:01 GMT
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Server: cloudflare
Set-Cookie: PHPSESSID=6ntp204l8k8qrn4odft5i5027l; path=/
Vary: Accept-Encoding
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=u9huK2Mw0pEiZx8JPIFGaG%2Fxh4IgcVEkSIIq5Fn1pucLfgd4A3lc8v0719dgaGDFoMWxpWoX9cdFLZcsQvT9euv9wDKQ9bMnhQFRkS1nv77VWbiY5QWha4YvrSNQxDTWoe0xvQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
CF-RAY: 6e353034bddc93ce-DFW
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP Response (First 500 bytes):
<!DOCTYPE html>
<html lang="en_US">
    <head>
        <!--SEO-related tags: title, keywords, description-->
                    <title>Bum Definition &amp; Meaning | Britannica Dictionary</title>
                            <meta name="description" content="BUM meaning: 1 : a person who has no place to live and no job and who asks people for money; 2 : a person who is lazy or who does something badly">
        
        <!--Static meta tags -->
        <meta charset="utf-8">
        <me

Err, much less output:

Quote

COM ERROR DETECTED!
Error ScriptLine....... 17
Error Number........... 0x80020009 (-2147352567)
Error WinDescription... Exception occurred.
Error RetCode.......... 0x80072F7D (-2147012739)
Error Description...... An error occurred in the secure channel support

This was the output when I tried with the BBC URL:

Quote

HTTP Response Code: 200 (OK)

HTTP Response Headers:
Cache-Control: public, stale-if-error=90, stale-while-revalidate=30, max-age=30
Connection: keep-alive
Date: Sat, 26 Feb 2022 10:29:51 GMT
Keep-Alive: timeout=120
Via: 1.1 BBC-GTM, 1.1 Belfrage
Content-Length: 2290326
Content-Type: text/html;charset=utf-8
Server: BBC-GTM
Vary: accept-encoding
belfrage-cache-status: HIT
bid: bruce
brequestid: 350d32bc5974452a80d66b47cc0ed6ca
bsig: 832e42282e34e9767bb086598e65bbbe
moz-cache-status: HIT
req-svc-chain: GTM,BELFRAGE,MOZART
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-mrid: n1
x-msig: ff5581a11599b0c65ea21c59f8a28e89
x-xss-protection: 1; mode=block
x-lb-nocache: true
nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":0.05}
X-BBC-Edge-Cache-Status: MISS
X-BBC-Origin-Response-Status: 200
report-to: {"group":"default","max_age":2592000,"endpoints":[{"url":"https://europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net/report-endpoint","priority":1}],"include_subdomains":true}
Strict-Transport-Security: max-age=2592000

HTTP Response (First 500 bytes):
<!DOCTYPE html>
<html lang="en-gb" class="b-pw-1280 b-reith-sans-font no-touch" id="responsive-news">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=1">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <meta name="google-site-verification" content="Tk6bx1127nACXoqt94L4-D-Of1fdr5gxrZ7u2Vtj9YI">
    <link href="//static.bbc.co.uk" rel="preconnect" crossorigin>
    <link href="//m.files.bbci.co.uk" rel="p
+>10:29:58 AutoIT3.exe ended.rc:0
>Exit code: 0    Time: 2.542

thegoodguy · February 26, 2022

This may be a stupid question, but why can my browser load both URLs? Presumably both Autoit and the browser access the same security/encryption protocols.

ad777 · February 26, 2022

@thegoodguy

the output you have was Error = 13.

this how you fix your TLS just to make sure you dont get error:

check your TLS is checked from Internet Properties.

Start -> Control Panel -> Network and internet -> Internet Options -> advanced

and make sure TLS 1.0 and TLS 1.1 and TLS 1.2 is checked in Internet Properties.

Edited February 26, 2022 by ad777

Musashi · February 26, 2022

39 minutes ago, thegoodguy said:

This may be a stupid question, but why can my browser load both URLs? Presumably both Autoit and the browser access the same security/encryption protocols.

"There are no stupid questions". I have run at least across a few, but yours is not one of them .

As far as I understand it, and this topic, as I said, is not really my area of expertise, the AutoIt functions like InetRead, InetGet, etc. can behave differently than your browser.

27 minutes ago, ad777 said:

check your TLS is checked from Internet Properties.

Start -> Control Panel -> Network and internet -> Internet Options -> advanced

and make sure TLS 1.0 and TLS 1.1 and TLS 1.2 is checked in Internet Properties.

This seems like it could be a good advice. When I uncheck the boxes for TLS 1.1 and TLS 1.2, I also get error 13.

thegoodguy · February 26, 2022

1 hour ago, ad777 said:

@thegoodguy

the output you have was Error = 13.

this how you fix your TLS just to make sure you dont get error:

check your TLS is checked from Internet Properties.

Start -> Control Panel -> Network and internet -> Internet Options -> advanced

and make sure TLS 1.0 and TLS 1.1 and TLS 1.2 is checked in Internet Properties.

Yes! this worked. Both URLs successful.

Heap big thanks! :)

jugador · February 27, 2022

@TheXman
If run the above code then encountered TLS error (@Error=> An error occurred in the secure channel support)

but using WinHTTP UDF no TLS error if add

Local $h_Open = _WinHttpOpen()
_WinHttpSetOption($h_Open, _
    $WINHTTP_OPTION_SECURE_PROTOCOLS, _
    BitOR($WINHTTP_FLAG_SECURE_PROTOCOL_TLS1_1, $WINHTTP_FLAG_SECURE_PROTOCOL_TLS1_2))

try the same with 'winhttp.winhttprequest.5.1'

Local Const $SecureProtocol_SSL2   = 0x0008
Local Const $SecureProtocol_SSL3   = 0x0020
Local Const $SecureProtocol_TLS1   = 0x0080
Local Const $SecureProtocol_TLS1_1 = 0x0200
Local Const $SecureProtocol_TLS1_2 = 0x0800

With ObjCreate("winhttp.winhttprequest.5.1")
        ;Set up & send request
        .option(9) = BitOR($SecureProtocol_TLS1_1, $SecureProtocol_TLS1_2)    ;@Error=>  The parameter is incorrect.
        ......
EndWith

but this not work getting @Error=> The parameter is incorrect.

If use default value of this option is 0x0028 which indicates $SecureProtocol_SSL2 or SSL $SecureProtocol_SSL3 then it's ok (no parameter incorrect error)

.option(9) = 0x0028

but will get TLS error (An error occurred in the secure channel support) at the time of

.Send()

So how to add $SecureProtocol_TLS1_1 & $SecureProtocol_TLS1_2 option while using "winhttp.winhttprequest.5.1" :sweating:

TheXman · February 27, 2022

9 hours ago, jugador said:

So how to add $SecureProtocol_TLS1_1 & $SecureProtocol_TLS1_2 option while using "winhttp.winhttprequest.5.1"

@jugador

The first answer, in the link below, gives a very good explanation of the issue and how to resolve it.

https://stackoverflow.com/questions/31886775/winhttp-errors-on-option-9-win2008-classic-asp

TL;DR

If you are using an OS (like Windows 7) that doesn't support those particular TLS options in a COM application, then do not use that option in your code, instead, set the appropriate values in the registry and it will work. Microsoft published an "Easy Fix" for Windows 7, which is referenced in the link provided in the topic below, that creates and sets the necessary registry entries to allow TLS1.1 & TLS1.2 to be used in COM implementations of the WinHTTP request object. Whether you do the registry entries manually or run the "Easy Fix", it's a one-time modification that tells COM WinHTTP request apps which secure protocols can be used.

The topic reply below, posted in 2018, has a link to the Microsoft article and registry entries.

Edited February 27, 2022 by TheXman

jugador · February 27, 2022

"https://www.britannica.com/dictionary/bum" site run without any TSL @error if use Msxml2.XMLHTTP.6.0

@TheXman any disadvantage or risk if use "Msxml2.XMLHTTP.6.0" over "winhttp.winhttprequest.5.1"

TheXman · February 27, 2022

@jugador

This topic was originally about InetGet and is marked as solved.

The only reason I brought up WinHTTP was to help me identify where the problem may have been. If you have questions about WinHTTP, XMLHTTP, ServerXMLHTTP, cURL, or any other HTTP request methods, I would suggest that you start another topic.

As far as XMLHTTP is concerned, if it works for you, then I don't see any disadvantage or risk. The same would be true for any of the other methods.

Edited February 27, 2022 by TheXman

Sign In

InetGet help - hit and miss downloads

Recommended Posts

thegoodguy

Link to comment

Share on other sites

thegoodguy

Link to comment

Share on other sites

ad777

Link to comment

Share on other sites

Musashi

Link to comment

Share on other sites

thegoodguy

Link to comment

Share on other sites

jugador

Link to comment

Share on other sites

TheXman

Link to comment

Share on other sites

jugador

Link to comment

Share on other sites

TheXman

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta