Jump to content
Sign in to follow this  
jasty

21s / 42s delays in _INetGetSource

Recommended Posts

So I'm using _INetGetSource to read text from URLs but some Windows 10 users are getting very long 21s or 42s long delays when trying to read from some URLs.

Both of these methods had the same result for some URLs but not for others.

Func ReadUrl0($url)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("GET", $url, 0)
    $oHTTP.Send()
    return $oHTTP.Responsetext
EndFunc

Func ReadUrl1($url)
    return _INetGetSource($url)
EndFunc

The call actually does succeed after this long delay.  I tried changing my User Agent before the calls with this command but it didn't help

        HttpSetUserAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko")

From what I'm reading these delays are indicative of certain TCP timeouts but I'm unclear how to fix things.   

These delays do not appear when viewing URLs in the browser only with AutoIt.

Is there a method for reading text from a URL that won't have these long delays?

Here's an example URL that's problematic.  http://pastebin.com/raw/2u1HiHnv

Is it maybe firewall related?  Has anyone encountered this?

Edited by jasty

Share this post


Link to post
Share on other sites

Hello jasty,

Unfortunately I did not encounter the same issues regarding speed. My results returned in well under 1s:

--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
79F2BF5FEF5B07B4
8F1FB0BAFC8D472A
65C306EC291D54D8
8DEB106EFB2655C4
19A8E089141BDD36
C22BCE01153FB712
AEC387630C3BCE8A
D127D3A59D0BE7CD
AAC0579B9746238D
527EAA905D993B48
15E9236C7CC2B3EF
042FE9B11712E9E7
78BA2D39E4F2E248
D9551BF895F09AAC
7BEED1752EE1BFD6
E0FC340F10E3F69E
49DF5572F6299EE0
D2A7EE9625378DD7
9B9927749DC0CEBF
A3D175C31206EAE5
6F49E68ECE90BEB9
B6A908924F536521
> Time ReadUrl0: 0.023s

79F2BF5FEF5B07B4
8F1FB0BAFC8D472A
65C306EC291D54D8
8DEB106EFB2655C4
19A8E089141BDD36
C22BCE01153FB712
AEC387630C3BCE8A
D127D3A59D0BE7CD
AAC0579B9746238D
527EAA905D993B48
15E9236C7CC2B3EF
042FE9B11712E9E7
78BA2D39E4F2E248
D9551BF895F09AAC
7BEED1752EE1BFD6
E0FC340F10E3F69E
49DF5572F6299EE0
D2A7EE9625378DD7
9B9927749DC0CEBF
A3D175C31206EAE5
6F49E68ECE90BEB9
B6A908924F536521
> Time ReadUrl1: 0.04s

<html><head></head><body><pre>79F2BF5FEF5B07B4
8F1FB0BAFC8D472A
65C306EC291D54D8
8DEB106EFB2655C4
19A8E089141BDD36
C22BCE01153FB712
AEC387630C3BCE8A
D127D3A59D0BE7CD
AAC0579B9746238D
527EAA905D993B48
15E9236C7CC2B3EF
042FE9B11712E9E7
78BA2D39E4F2E248
D9551BF895F09AAC
7BEED1752EE1BFD6
E0FC340F10E3F69E
49DF5572F6299EE0
D2A7EE9625378DD7
9B9927749DC0CEBF
A3D175C31206EAE5
6F49E68ECE90BEB9
B6A908924F536521</pre></body></html>
> Time ReadUrl2: 3.218s

+>11:04:00 AutoIt3.exe ended.rc:0
+>11:04:00 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 4.235

Using this code:

#include <Inet.au3>
#include <IE.au3>

$url = 'http://pastebin.com/raw/2u1HiHnv'

$hTimer = TimerInit()
ConsoleWrite(ReadUrl0($url) & @CRLF)
ConsoleWrite("> Time ReadUrl0: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF)

$hTimer = TimerInit()
ConsoleWrite(ReadUrl1($url) & @CRLF)
ConsoleWrite("> Time ReadUrl1: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF)

$hTimer = TimerInit()
ConsoleWrite(ReadUrl2($url) & @CRLF)
ConsoleWrite("> Time ReadUrl2: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF)

Func ReadUrl0($url)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("GET", $url, 0)
    $oHTTP.Send()
    Return $oHTTP.Responsetext
EndFunc   ;==>ReadUrl0

Func ReadUrl1($url)
    Return _INetGetSource($url)
EndFunc   ;==>ReadUrl1

Func ReadUrl2($url)
    $oIE = _IECreate($url, 0, 0, 1)
    $sReturn = _IEDocReadHTML($oIE)
    _IEQuit($oIE)
    Return $sReturn
EndFunc   ;==>ReadUrl2

With the added _IE* functions it did take a little longer, but that may be more from creating and quitting IE than anything (especially since that was the first time IE has been opened in quite a while, if that has any bearing on the time. Subsequent runs had the _IE* time around 0.7s - 0.9s).

Edited by mistersquirrle

RAWR! I'm hungry :( Feed the panda squirrle

Share this post


Link to post
Share on other sites

I've done more investigating of this issue and have traced it to some kind of issue with IPv6 connections.   

A server can specify in its DNS record that it accepts both ipv4 and ipv6 connections and windows is configured in this case to prefer ipv6 over ipv4.

For whatever reason on the ISP or router level the ipv6 connections fail and the fall back to ipv4 happens after a 21s TCP timeout (stupid long).   

Web browsers commonly get around this by attempting both ipv4 and ipv6 connections and using whichever connects first but Autoit is just using the default windows behavior.

This is a common enough problem among my clients that I need to fix it.

I think these are my 3 options:

1) Disable ipv6 in my connection options somehow.

2) Specify a preference for ipv4 over ipv6 somehow.

3) Emulate what browsers do and try both connections asynchronously and discarding the slow one.

Does anyone know how to accomplish any of these options?

EDIT:  CURL library seems to have an option for this!  This is probably the solution.

https://curl.haxx.se/libcurl/c/CURLOPT_IPRESOLVE.html

https://www.autoitscript.com/forum/topic/173067-curl-udf-autoit-binary-code-version-of-libcurl-with-ssl-support/

Edited by jasty

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...