InetGet help - hit and miss downloads

thegoodguy · February 25, 2022

Hi,

I'm trying to download a number of word definitions using InetGet. I'm having trouble. The InetGet help entry says "The URL parameter should be in the form "http://www.somesite.com/path/file.html" - just like an address you would type into your web browser."

InetGet("https://www.bbc.co.uk/news/live/world-europe-60517447", "C:\Users\xxx\Desktop\Downloaded\test1.html",19)
InetGet("https://www.britannica.com/dictionary/bum", "C:\Users\xxx\Desktop\Downloaded\test2.html",19)
Exit

Why does the first one work, but the second one doesn't? :/

Thanks in advance.

Musashi · February 25, 2022

30 minutes ago, thegoodguy said:

Why does the first one work, but the second one doesn't? 😕

The second works for me (Win7) . First few lines of the code :

<!DOCTYPE html>
<html lang="en_US">
    <head>
        <!--SEO-related tags: title, keywords, description-->
                    <title>Bum Definition &amp; Meaning | Britannica Dictionary</title>
                            <meta name="description" content="BUM meaning: 1 : a person who has no place to live and no job and who asks people for money; 2 : a person who is lazy or who does something badly">
        
        <!--Static meta tags -->
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        
[...]

ad777 · February 25, 2022

@thegoodguy

try _INetGetSource

Musashi · February 25, 2022

@thegoodguy

Or :

Local $bData = InetRead("https://www.britannica.com/dictionary/bum")
Local $iError = @error, $iBytesRead = @extended
Local $sData = BinaryToString($bData)
ConsoleWrite("Error = " & $iError & @CRLF & "Bytes downloaded = " & $iBytesRead & @CRLF & $sData)

What does your SciTE Output shows ?

thegoodguy · February 25, 2022

1 hour ago, ad777 said:

@thegoodguy

try _INetGetSource

I get the same resut as with InNetGet :(

Musashi · February 25, 2022

Have you tried my example ?

To save the code, use :

Local $bData = InetRead("https://www.britannica.com/dictionary/bum")
Local $iError = @error, $iBytesRead = @extended
Local $sData = BinaryToString($bData)

; just for test purposes :
; ConsoleWrite("Error = " & $iError & @CRLF & "Bytes downloaded = " & $iBytesRead & @CRLF & $sData)

; save the code to file :
Local $hFile = FileOpen(@ScriptDir & "\test002.html", 18)
FileWrite($hFile, $sData)
FileClose($hFile)

(you can change the destination directory and filename to your wishes).

Edited February 25, 2022 by Musashi
typo

thegoodguy · February 25, 2022

1 hour ago, Musashi said:

@thegoodguy

Or :

Local $bData = InetRead("https://www.britannica.com/dictionary/bum")
Local $iError = @error, $iBytesRead = @extended
Local $sData = BinaryToString($bData)
ConsoleWrite("Error = " & $iError & @CRLF & "Bytes downloaded = " & $iBytesRead & @CRLF & $sData)

What does your SciTE Output shows ?

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\xxx\Desktop\testinetget.au3" /UserParams
+>20:26:25 Starting AutoIt3Wrapper v.2.1.0.8    Environment(Language:0409 Keyboard:00000809 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64)
>Running AU3Check (3.3.14.2) from:C:\Program Files (x86)\AutoIt3
+>20:26:25 AU3Check ended.rc:0
>Running:(3.3.14.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\xxx\Desktop\testinetget.au3"
Error = 13
Bytes downloaded = 12029
+>20:26:25 AutoIT3.exe ended.rc:0
>Exit code: 0    Time: 1.528

Musashi · February 25, 2022

22 minutes ago, thegoodguy said:

Error = 13
Bytes downloaded = 12029
+>20:26:25 AutoIT3.exe ended.rc:0
>Exit code: 0 Time: 1.528

Hmm, it works for me without problems

Error = 0
Bytes downloaded = 112843

Could be a permission-, time-out- or firewall issue :think: . Do you work in a network environment ?

Danp2 · February 25, 2022

I seem to recall an issue with different supported TLS versions on different OS. What version of Windows are you folks using?

Musashi · February 25, 2022

21 minutes ago, Danp2 said:

What version of Windows are you folks using?

Win 7 Pro 64 Bit , Servicepack 1 ==> apparently the version also used by thegoodguy see :

1 hour ago, thegoodguy said:

+>20:26:25 Starting AutoIt3Wrapper v.2.1.0.8 Environment(Language:0409 Keyboard:00000809 OS:WIN_7/Service Pack 1 CPU:X64 OS:X64)

Edited February 25, 2022 by Musashi

ad777 · February 25, 2022

32 minutes ago, Danp2 said:

I seem to recall an issue with different supported TLS versions on different OS. What version of Windows are you folks using?

Win 7 Ultimate x32 bit

ad777 · February 25, 2022

@thegoodguy

the error code comming from either your wireless is off or timeout.

Try Troubleshooting your Wifi.

Musashi · February 25, 2022

12 minutes ago, ad777 said:

the error code comming from either your wireless is off or timeout.

@Danp2 could be right with his assumption regarding TLS, see e.g. reading-data-from-web-https-on-windows-7

@TheXman 's post (and the rest of the thread) might also shed some light into the dark, see problem-with-winhttp

Edited February 25, 2022 by Musashi

thegoodguy · February 25, 2022

2 hours ago, Musashi said:

Hmm, it works for me without problems

Error = 0
Bytes downloaded = 112843

Could be a permission-, time-out- or firewall issue . Do you work in a network environment ?

I do not work in a network environment. Just my PC and a wired connection to my router via powerline.

I'm not really a nitty-gritty computer user, but all of the suggested issues that you guys have given above, wouldn't they also impact the working URL (https://www.bbc.co.uk/news/live/world-europe-60517447) ?

Musashi · February 26, 2022

12 minutes ago, thegoodguy said:

but all of the suggested issues that you guys have given above, wouldn't they also impact the working URL (https://www.bbc.co.uk/news/live/world-europe-60517447) ?

No !

If the problem is TLS related, and I'm pretty sure it is, then one server may accept an older TLS version from your PC, but another may not. For details, see Transport_Layer_Security .

thegoodguy · February 26, 2022

31 minutes ago, Musashi said:

No !

If the problem is TLS related, and I'm pretty sure it is, then one server may accept an older TLS version from your PC, but another may not. For details, see Transport_Layer_Security .

Ok. Please firgive me if I'm approaching this incorrectly.

Loading both URLs in a browser, I can get technical info on the security/encryption. This is the info of both of the test URLs. They appear to be using the same level of security TLS 1.2, do they not?

image.png.9875bf5fd5135cc3237abc1c7287c74b.png

TheXman · February 26, 2022

Let's see if the problematic URL works with WinHTTP. If the script below fails, at least it should provide us with a little more information.

Can you try running the script and posting your console results?

Test script

#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w 5 -w 6 -d

#include <Constants.au3>

test()

Func test()
    Local $oComErr = Null

    ;Set up local COM error handler
    $oComErr = ObjEvent("AutoIt.Error", com_error_handler)
    #forceref $oComErr

    With ObjCreate("winhttp.winhttprequest.5.1")
        ;Set up & send request
        .Open("GET", "https://www.britannica.com/dictionary/bum")
        .Send()

        ;Display response
        ConsoleWrite("HTTP Response Code: " & .status & " (" & .StatusText & ")" & @CRLF & @CRLF)
        ConsoleWrite("HTTP Response Headers:" & @CRLF)
        ConsoleWrite(.GetAllResponseHeaders())
        ConsoleWrite("HTTP Response (First 500 bytes):" & @CRLF)
        ConsoleWrite(StringLeft(.ResponseText, 500) & @CRLF) ;First 500 bytes of response
    EndWith

EndFunc

Func com_error_handler($oError)
    With $oError
        ConsoleWrite(@CRLF & "COM ERROR DETECTED!" & @CRLF)
        ConsoleWrite("  Error ScriptLine....... " & .scriptline & @CRLF)
        ConsoleWrite("  Error Number........... " & "0x" & Hex(.number) & " (" & .number & ")" & @CRLF)
        ConsoleWrite("  Error WinDescription... " & StringStripWS(.windescription, $STR_STRIPTRAILING) & @CRLF)
        ConsoleWrite("  Error RetCode.......... " & "0x" & Hex(Number(.retcode)) & " (" & Number(.retcode) & ")" & @CRLF)
        ConsoleWrite("  Error Description...... " & StringStripWS(.description   , $STR_STRIPTRAILING) & @CRLF)
    EndWith
    MsgBox($MB_ICONERROR + $MB_TOPMOST, "ERROR", "A COM error has occurred.  See console for details.")
    Exit
EndFunc

My output (same on Win7 x64 & Win10 x64):

HTTP Response Code: 200 (OK)

HTTP Response Headers:
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Date: Sat, 26 Feb 2022 00:52:01 GMT
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Server: cloudflare
Set-Cookie: PHPSESSID=6ntp204l8k8qrn4odft5i5027l; path=/
Vary: Accept-Encoding
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=u9huK2Mw0pEiZx8JPIFGaG%2Fxh4IgcVEkSIIq5Fn1pucLfgd4A3lc8v0719dgaGDFoMWxpWoX9cdFLZcsQvT9euv9wDKQ9bMnhQFRkS1nv77VWbiY5QWha4YvrSNQxDTWoe0xvQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
CF-RAY: 6e353034bddc93ce-DFW
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP Response (First 500 bytes):
<!DOCTYPE html>
<html lang="en_US">
    <head>
        <!--SEO-related tags: title, keywords, description-->
                    <title>Bum Definition &amp; Meaning | Britannica Dictionary</title>
                            <meta name="description" content="BUM meaning: 1 : a person who has no place to live and no job and who asks people for money; 2 : a person who is lazy or who does something badly">
        
        <!--Static meta tags -->
        <meta charset="utf-8">
        <me

Edited February 27, 2022 by TheXman

Musashi · February 26, 2022

11 minutes ago, thegoodguy said:

They appear to be using the same level of security TLS 1.2, do they not?

That is correct, but in some circumstances the Servers have a different tolerance towards older versions.

@TheXman 's script will certainly bring new insights .

Musashi · February 26, 2022

More recent versions of TLS can be installed subsequently on your PC, but please be very careful before you fiddle around in the registry. Unless you are certain of what you are doing, it is better to ask again. There are members in the forum who are more familiar with this topic than I am ).
This article may be of interest : update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-winhttp-in-windows

An alternative approach could be the use of the command line tool cURL . The latest binary builds for Microsoft Windows can be found here : https://curl.se/windows/ (since Windows 10, cURL is even shipped by Microsoft)

EDIT : Here's a quick and dirty script to figure out your current settings :

Local $sHKLM = @OSArch = "x64" ? "HKLM64" : "HKLM"
Local $vResult, $DefaultSecureProtocols=0, $DisabledByDefaultTLS1_1=0, $DisabledByDefaultTLS1_2=0

$vResult = RegRead($sHKLM & "\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp", "DefaultSecureProtocols")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DefaultSecureProtocols'" & @error)
Else
    ; Info : on my system the value is 0x00000A00 (2560)
    ; => TLS 1.1 (0x00000200) + TLS 1.2 (0x00000800)
    $DefaultSecureProtocols = "0x" & Hex($vResult, 8)
EndIf

$vResult = RegRead($sHKLM & "\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client", "DisabledByDefault")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DisabledByDefaultTLS1_1'" & @error)
Else
    ; Info : on my system the value is 0x00000000 (0)
    $DisabledByDefaultTLS1_1 = "0x" & Hex($vResult, 8)
EndIf

$vResult = RegRead($sHKLM & "\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client", "DisabledByDefault")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DisabledByDefaultTLS1_2'" & @error)
Else
    ; Info : on my system the value is 0x00000000 (0)
    $DisabledByDefaultTLS1_2 = "0x" & Hex($vResult, 8)
EndIf

ConsoleWrite("=======================================" & @CRLF)
ConsoleWrite("==> Results : " & @CRLF)
ConsoleWrite("DefaultSecureProtocols  = " & $DefaultSecureProtocols & @CRLF)
ConsoleWrite("DisabledByDefaultTLS1_1 = " & $DisabledByDefaultTLS1_1 & @CRLF)
ConsoleWrite("DisabledByDefaultTLS1_2 = " & $DisabledByDefaultTLS1_2 & @CRLF)
ConsoleWrite("=======================================" & @CRLF)

Edited February 26, 2022 by Musashi

thegoodguy · February 26, 2022

1 hour ago, Musashi said:

More recent versions of TLS can be installed subsequently on your PC, but please be very careful before you fiddle around in the registry. Unless you are certain of what you are doing, it is better to ask again. There are members in the forum who are more familiar with this topic than I am ).
This article may be of interest : update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-winhttp-in-windows

An alternative approach could be the use of the command line tool cURL . The latest binary builds for Microsoft Windows can be found here : https://curl.se/windows/ (since Windows 10, cURL is even shipped by Microsoft)

EDIT : Here's a quick and dirty script to figure out your current settings :

Local $sHKLM = @OSArch = "x64" ? "HKLM64" : "HKLM"
Local $vResult, $DefaultSecureProtocols=0, $DisabledByDefaultTLS1_1=0, $DisabledByDefaultTLS1_2=0

$vResult = RegRead($sHKLM & "\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp", "DefaultSecureProtocols")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DefaultSecureProtocols'" & @error)
Else
    ; Info : on my system the value is 0x00000A00 (2560)
    ; => TLS 1.1 (0x00000200) + TLS 1.2 (0x00000800)
    $DefaultSecureProtocols = "0x" & Hex($vResult, 8)
EndIf

$vResult = RegRead($sHKLM & "\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client", "DisabledByDefault")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DisabledByDefaultTLS1_1'" & @error)
Else
    ; Info : on my system the value is 0x00000000 (0)
    $DisabledByDefaultTLS1_1 = "0x" & Hex($vResult, 8)
EndIf

$vResult = RegRead($sHKLM & "\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client", "DisabledByDefault")
If @error Then
    MsgBox(4096, "Error", "Error reading 'DisabledByDefaultTLS1_2'" & @error)
Else
    ; Info : on my system the value is 0x00000000 (0)
    $DisabledByDefaultTLS1_2 = "0x" & Hex($vResult, 8)
EndIf

ConsoleWrite("=======================================" & @CRLF)
ConsoleWrite("==> Results : " & @CRLF)
ConsoleWrite("DefaultSecureProtocols  = " & $DefaultSecureProtocols & @CRLF)
ConsoleWrite("DisabledByDefaultTLS1_1 = " & $DisabledByDefaultTLS1_1 & @CRLF)
ConsoleWrite("DisabledByDefaultTLS1_2 = " & $DisabledByDefaultTLS1_2 & @CRLF)
ConsoleWrite("=======================================" & @CRLF)

First off; you guys are amazing. knowledgable and quick to respond. Thank you.

This is the result from your "dirty" script. All 3 came up with "error reading" pop ups. It doesn't sound like that's the same thing as having a value of zero.

=======================================
==> Results :
DefaultSecureProtocols = 0
DisabledByDefaultTLS1_1 = 0
DisabledByDefaultTLS1_2 = 0
=======================================

InetGet help - hit and miss downloads

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members