jasty Posted September 12, 2019 Share Posted September 12, 2019 (edited) So I'm using _INetGetSource to read text from URLs but some Windows 10 users are getting very long 21s or 42s long delays when trying to read from some URLs. Both of these methods had the same result for some URLs but not for others. Func ReadUrl0($url) Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", $url, 0) $oHTTP.Send() return $oHTTP.Responsetext EndFunc Func ReadUrl1($url) return _INetGetSource($url) EndFunc The call actually does succeed after this long delay. I tried changing my User Agent before the calls with this command but it didn't help HttpSetUserAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko") From what I'm reading these delays are indicative of certain TCP timeouts but I'm unclear how to fix things. These delays do not appear when viewing URLs in the browser only with AutoIt. Is there a method for reading text from a URL that won't have these long delays? Here's an example URL that's problematic. http://pastebin.com/raw/2u1HiHnv Is it maybe firewall related? Has anyone encountered this? Edited September 12, 2019 by jasty Link to comment Share on other sites More sharing options...
mistersquirrle Posted September 16, 2019 Share Posted September 16, 2019 (edited) Hello jasty, Unfortunately I did not encounter the same issues regarding speed. My results returned in well under 1s: expandcollapse popup--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop 79F2BF5FEF5B07B4 8F1FB0BAFC8D472A 65C306EC291D54D8 8DEB106EFB2655C4 19A8E089141BDD36 C22BCE01153FB712 AEC387630C3BCE8A D127D3A59D0BE7CD AAC0579B9746238D 527EAA905D993B48 15E9236C7CC2B3EF 042FE9B11712E9E7 78BA2D39E4F2E248 D9551BF895F09AAC 7BEED1752EE1BFD6 E0FC340F10E3F69E 49DF5572F6299EE0 D2A7EE9625378DD7 9B9927749DC0CEBF A3D175C31206EAE5 6F49E68ECE90BEB9 B6A908924F536521 > Time ReadUrl0: 0.023s 79F2BF5FEF5B07B4 8F1FB0BAFC8D472A 65C306EC291D54D8 8DEB106EFB2655C4 19A8E089141BDD36 C22BCE01153FB712 AEC387630C3BCE8A D127D3A59D0BE7CD AAC0579B9746238D 527EAA905D993B48 15E9236C7CC2B3EF 042FE9B11712E9E7 78BA2D39E4F2E248 D9551BF895F09AAC 7BEED1752EE1BFD6 E0FC340F10E3F69E 49DF5572F6299EE0 D2A7EE9625378DD7 9B9927749DC0CEBF A3D175C31206EAE5 6F49E68ECE90BEB9 B6A908924F536521 > Time ReadUrl1: 0.04s <html><head></head><body><pre>79F2BF5FEF5B07B4 8F1FB0BAFC8D472A 65C306EC291D54D8 8DEB106EFB2655C4 19A8E089141BDD36 C22BCE01153FB712 AEC387630C3BCE8A D127D3A59D0BE7CD AAC0579B9746238D 527EAA905D993B48 15E9236C7CC2B3EF 042FE9B11712E9E7 78BA2D39E4F2E248 D9551BF895F09AAC 7BEED1752EE1BFD6 E0FC340F10E3F69E 49DF5572F6299EE0 D2A7EE9625378DD7 9B9927749DC0CEBF A3D175C31206EAE5 6F49E68ECE90BEB9 B6A908924F536521</pre></body></html> > Time ReadUrl2: 3.218s +>11:04:00 AutoIt3.exe ended.rc:0 +>11:04:00 AutoIt3Wrapper Finished. >Exit code: 0 Time: 4.235 Using this code: #include <Inet.au3> #include <IE.au3> $url = 'http://pastebin.com/raw/2u1HiHnv' $hTimer = TimerInit() ConsoleWrite(ReadUrl0($url) & @CRLF) ConsoleWrite("> Time ReadUrl0: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF) $hTimer = TimerInit() ConsoleWrite(ReadUrl1($url) & @CRLF) ConsoleWrite("> Time ReadUrl1: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF) $hTimer = TimerInit() ConsoleWrite(ReadUrl2($url) & @CRLF) ConsoleWrite("> Time ReadUrl2: " & Round(TimerDiff($hTimer) / 1000, 3) & 's' & @CRLF) Func ReadUrl0($url) Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", $url, 0) $oHTTP.Send() Return $oHTTP.Responsetext EndFunc ;==>ReadUrl0 Func ReadUrl1($url) Return _INetGetSource($url) EndFunc ;==>ReadUrl1 Func ReadUrl2($url) $oIE = _IECreate($url, 0, 0, 1) $sReturn = _IEDocReadHTML($oIE) _IEQuit($oIE) Return $sReturn EndFunc ;==>ReadUrl2 With the added _IE* functions it did take a little longer, but that may be more from creating and quitting IE than anything (especially since that was the first time IE has been opened in quite a while, if that has any bearing on the time. Subsequent runs had the _IE* time around 0.7s - 0.9s). Edited September 16, 2019 by mistersquirrle We ought not to misbehave, but we should look as though we could. Link to comment Share on other sites More sharing options...
jasty Posted September 26, 2019 Author Share Posted September 26, 2019 (edited) I've done more investigating of this issue and have traced it to some kind of issue with IPv6 connections. A server can specify in its DNS record that it accepts both ipv4 and ipv6 connections and windows is configured in this case to prefer ipv6 over ipv4. For whatever reason on the ISP or router level the ipv6 connections fail and the fall back to ipv4 happens after a 21s TCP timeout (stupid long). Web browsers commonly get around this by attempting both ipv4 and ipv6 connections and using whichever connects first but Autoit is just using the default windows behavior. This is a common enough problem among my clients that I need to fix it. I think these are my 3 options: 1) Disable ipv6 in my connection options somehow. 2) Specify a preference for ipv4 over ipv6 somehow. 3) Emulate what browsers do and try both connections asynchronously and discarding the slow one. Does anyone know how to accomplish any of these options? EDIT: CURL library seems to have an option for this! This is probably the solution. https://curl.haxx.se/libcurl/c/CURLOPT_IPRESOLVE.html https://www.autoitscript.com/forum/topic/173067-curl-udf-autoit-binary-code-version-of-libcurl-with-ssl-support/ Edited September 26, 2019 by jasty Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now