The fastest way to get data?

[youtuber]

Hi

Friends,What is the fastest way to get your data from the internet?

or do you have any other ideas faster?

expandcollapse popup

#include <Inet.au3>
#include <String.au3>
#include <Array.au3>
$url = "https://www.autoitscript.com/forum/"
_aINetGetSource($url)
ConsoleWrite("" & @CRLF)
_aInetRead($url)
ConsoleWrite("" & @CRLF)
_HTTPGet($url)
ConsoleWrite("" & @CRLF)
_MicrosoftXMLHTTP($url)

Func _aINetGetSource($url)
    ConsoleWrite("INetGetSource start : " & @SEC & ":" & @MSEC & @CRLF)
    $source = _INetGetSource($url)
    $StringBetween = _StringBetween($source, '<title>', '</title>')
    If IsArray($StringBetween) Then
        ConsoleWrite("INetGetSource Finish : " & @SEC & ":" & @MSEC & @CRLF)
        ConsoleWrite($StringBetween[0] & @CRLF)
    EndIf
EndFunc   ;==>_aINetGetSource

Func _aInetRead($url)
    ConsoleWrite("InetRead start : " & @SEC & ":" & @MSEC & @CRLF)
    $source = BinaryToString(InetRead($url, 1))
    $StringBetween = _StringBetween($source, '<title>', '</title>')
    If IsArray($StringBetween) Then
        ConsoleWrite("InetRead Finish : " & @SEC & ":" & @MSEC & @CRLF)
        ConsoleWrite($StringBetween[0] & @CRLF)
    EndIf
EndFunc   ;==>_aInetRead


Func _HTTPGet($url)
    ConsoleWrite("HTTPGet start : " & @SEC & ":" & @MSEC & @CRLF)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("GET", $url, False)
    If (@error) Then Return SetError(1, 0, 0)
    $oHTTP.Send()
    If (@error) Then Return SetError(2, 0, 0)
    $sReceived = $oHTTP.ResponseText
    $iStatus = $oHTTP.Status
    If $iStatus = 200 Then
        $StringBetween = _StringBetween($sReceived, '<title>', '</title>')
        If IsArray($StringBetween) Then
            ConsoleWrite("HTTPGet Finish : " & @SEC & ":" & @MSEC & @CRLF)
            ConsoleWrite($StringBetween[0] & @CRLF)
        Else
            Return SetError(3, 0, 0)
        EndIf
    EndIf
EndFunc   ;==>_HTTPGet


Func _MicrosoftXMLHTTP($url)
    ConsoleWrite("MicrosoftXMLHTTP : " & @SEC & ":" & @MSEC & @CRLF)
    $oHTTP = ObjCreate("Microsoft.XMLHTTP")
    $oHTTP.Open("GET", $url, 0)
    $oHTTP.Send()
    If @error Then Return SetError(1, 0, 0)
    $sReceived = $oHTTP.ResponseText
    $StringBetween = _StringBetween($sReceived, '<title>', '</title>')
    If IsArray($StringBetween) Then
        ConsoleWrite("MicrosoftXMLHTTP Finish : " & @SEC & ":" & @MSEC & @CRLF)
        ConsoleWrite($StringBetween[0] & @CRLF)
    EndIf
EndFunc   ;==>_MicrosoftXMLHTTP

 

[Ascer]

Hola

All this functions are bad writen by AutoIt creators.

They freeze your app until don't finish work. Personally use COM object Shell.Explorer.2 for this action.  

[youtuber]

@Ascer how can you give an example?

[Ascer]

Let me write...

[mLipok]

  On 2/13/2018 at 2:54 PM, youtuber said:

 

Func _HTTPGet($url)
    ConsoleWrite("HTTPGet start : " & @SEC & ":" & @MSEC & @CRLF)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("GET", $url, False)
    If (@error) Then Return SetError(1, 0, 0)
    $oHTTP.Send()
    If (@error) Then Return SetError(2, 0, 0)
    $sReceived = $oHTTP.ResponseText
    $iStatus = $oHTTP.Status
    If $iStatus = 200 Then
        $StringBetween = _StringBetween($sReceived, '<title>', '</title>')
        If IsArray($StringBetween) Then
            ConsoleWrite("HTTPGet Finish : " & @SEC & ":" & @MSEC & @CRLF)
            ConsoleWrite($StringBetween[0] & @CRLF)
        Else
            Return SetError(3, 0, 0)
        EndIf
    EndIf
EndFunc   ;==>_HTTPGet


Func _MicrosoftXMLHTTP($url)
    ConsoleWrite("MicrosoftXMLHTTP : " & @SEC & ":" & @MSEC & @CRLF)
    $oHTTP = ObjCreate("Microsoft.XMLHTTP")
    $oHTTP.Open("GET", $url, 0)
    $oHTTP.Send()
    If @error Then Return SetError(1, 0, 0)
    $sReceived = $oHTTP.ResponseText
    $StringBetween = _StringBetween($sReceived, '<title>', '</title>')
    If IsArray($StringBetween) Then
        ConsoleWrite("MicrosoftXMLHTTP Finish : " & @SEC & ":" & @MSEC & @CRLF)
        ConsoleWrite($StringBetween[0] & @CRLF)
    EndIf
EndFunc   ;==>_MicrosoftXMLHTTP

 

Expand  

 

  On 2/13/2018 at 3:18 PM, Ascer said:

All this functions are bad writen by AutoIt creators.

They freeze your app until don't finish work. Personally use COM object Shell.Explorer.2 for this action.  

Expand  

Which one are bad written ?
Which one you are proposing to use ?

 

[Ascer]

@youtuber

It's an example of Shell using

expandcollapse popup

#include <Timers.au3>

; Start error catching.
Local $oMyError = ObjEvent("AutoIt.Error","MyErrFunc") ; COM error handle

; Create a gui for _Timer usage.
Local $hGUI = GUICreate("")

; Website we want to take a source.
Local $sWebsite = "https://www.autoitscript.com/forum/"

; Set time for actions delay.
Local $fTime = 0

; Create an object.
Local $oShell = ObjCreate("Shell.Explorer.2")

; Exit if $oShell isn't object.
If Not IsObj($oShell) Then
    ConsoleWrite("+++ Error durining creating Shell object." & @CRLF)
    Exit
EndIf

; Create an graphic interface. Need for works.
GUICtrlCreateObj($oShell, -1, -1)

; Set execute timer each 200ms
_Timer_SetTimer($hGUI, 200, 'Example')

; Create infinite loop.
For $i = 0 To 100000000000000
    Sleep(10)
    ConsoleWrite("Look loop still working durining reading object.." & $i & @CRLF)
Next

Func Example($1, $2, $3, $4)

    ; Execute action
    shellGetSource($oShell)

EndFunc

Func shellGetSource(ByRef $oObj)

    ; Get current time delay to avoid spam
    Local $fDelay = TimerDiff($fTime)

    ; When delay lower than 2s stop actions.
    If $fDelay < 2000 Then
        Return False
    EndIf

    ; Navigate to web
    Local $bNavigate = shellNavigate($oObj, $sWebsite)

    ; Set time to make a smooth loop.
    $fTime = TimerInit()

    ; Page is not ready break function
    If Not $bNavigate Then
        Return False
    EndIf

    ; Page is ready lets read a body. HERE IS OUR WEBSITE IN HTML
    Local $sHtmlBody = $oObj.document.body.innerHTML

    ; Make sure that all is ok
    If StringLen($sHtmlBody) > 100 Then

        ; All is ok
        ConsoleWrite("Succesfully read a web. " & $sWebsite & @CRLF)

    Else

        ; Error rised!
        ConsoleWrite("+++ Error durining get website source. " & $sWebsite & @CRLF)

    EndIf

    ConsoleWrite($sHtmlBody)

    ; Exit program
    Exit

EndFunc

Func shellReadyState(ByRef $oObj, $sUrl)

    ; Check web for current url.
    If $oObj.locationURL = $sUrl Then

        ; Check if site is ready.
        If $oObj.readyState = 4 Then

            ; Second check for some webs.
            If String($oObj.document.readyState) = "complete" Then

                ; Web already ready
                Return True

            EndIf

        EndIf

    EndIf

    ; Web dont ready yet.
    Return False

EndFunc

Func shellNavigate(ByRef $oObj, $sUrl)

    ; Check if page is alredy loaded
    If shellReadyState($oObj, $sUrl) Then

        ; Page ready!
        Return True

    EndIf

    ; Navigate to source website.
    $oObj.Navigate($sUrl)

    ; Page not ready!
    Return False

EndFunc

Func MyErrFunc()
    Local $HexNumber = hex($oMyError.number,8)
    ConsoleWrite("We intercepted a COM Error !"       & @CRLF  & @CRLF & _
             "err.description is: "    & @TAB & $oMyError.description    & @CRLF & _
             "err.windescription:"     & @TAB & $oMyError.windescription & @CRLF & _
             "err.number is: "         & @TAB & $HexNumber              & @CRLF & _
             "err.lastdllerror is: "   & @TAB & $oMyError.lastdllerror   & @CRLF & _
             "err.scriptline is: "     & @TAB & $oMyError.scriptline     & @CRLF & _
             "err.source is: "         & @TAB & $oMyError.source         & @CRLF & _
             "err.helpfile is: "       & @TAB & $oMyError.helpfile       & @CRLF & _
             "err.helpcontext is: "    & @TAB & $oMyError.helpcontext _
            )
    SetError(1)
Endfunc

@mLipok 

When i try to call a website to download a file or a lot of data via winhttp or InetRead my app got freeze until this action has end.

[mLipok]

  On 2/13/2018 at 2:54 PM, youtuber said:

Hi

Friends,What is the fastest way to get your data from the internet?

or do you have any other ideas faster?

Expand  

In such cases the proper step by step is:

1. Buy faster PC, best choises is: PC with i9 processor and SSD M.2 for example Samsung 960 Pro EVO
   but it will be enought to have i5 >3GHz with any SSD faster than 300MB/s Write/Read and faster than 40000 IOPS
2. Call to your ISP to connect you to FiberOptic internet for example 100Mbit/100Mbit with ping less than 5 ms
3. Optimalize your script

 

ps.
Recently I change my computer to post lease  USFF PC with Intel i7 + 8GB RAM + SSD and this machine is about 4 times faster then my previous computer, and this new post lease PC is about 3 times cheaper then my old computer.
Currently I have ISP with ping about 11 to our local (Polish services) but to autoitscript.com I have ping about 40 ms  which is poor.

 

 

 

 

[youtuber]

@Ascer thank you, but the example looks very complicated.
It seems like a difficult example for me.

[mLipok]

  On 2/13/2018 at 4:32 PM, Ascer said:

@mLipok 

When i try to call a website to download a file or a lot of data via winhttp or InetRead my app got freeze until this action has end.

Expand  

I think this is releated how this (mentioned by you) COM objects and WinAPI function works.

You are using normal IE object, so this is different way, different behavior.

btw.
Why you are using your own function instead using IE.au3 UDF ?

 

 

[mLipok]

  On 2/13/2018 at 4:32 PM, Ascer said:

When i try to call a website to download a file or a lot of data via winhttp or InetRead my app got freeze until this action has end.

Expand  

I cooperate with a middle company (about 100 staff) which is developing for about 1500 clients, specyfic big CRM like program, with many connection to GOV web services.
This CRM is designed mostly in C++ and have few modules in C#

Of course most of features from this CRM works good and there is no problem with Freeze  effect.

But most of functions which are transferring data or generating reports, unfortunately freeze main APP.

For example when you try to import about 100 MB data in several thousand of PDF files , (of course ASAP) then main APP is not interested in problems like refresh interface .

but lets back to AutoIt.

Even using IE object you will find the same problem .
For example when  you click in file upload button which opens FileOpenDialog.

Question:
Did you try to do some checking for example with WinHTTP object and VBScript ?

 

[Ascer]

@mLipok

IE udf doesn't work for me. 3 year ago i totally removed Internet Explorer with my PC, speed increased about 50%.

All my API projects are based on WinHttp.dll. Everything working ok when i send example an emails via Google API but there are a websites (Polish) that have sick connection. In this fact i use this:

Local $iResolveTimeout = 5000 ; ms. Time to find a host
Local $iConnectTimeout = 10000 ; ms. Time to connest to the host.
Local $iSendTimeout = 30000 ; ms. Time to send a data (depent on: speed pc, speed ur upload, host download speed, size of file) 
Local $iReceiveTimeout = 30000 ; ms. Time to receive a data from host.

$oHTTP.SetTimeouts(iResolveTimeout, iConnectTimeout, $iSendTimeout, $iReceiveTimeout)

 When host failed i make a next connection after some time until don't receive needed data.

I don't know VBscript programming language but COM objects look very similar to Autoit so i try to import some code and check for results.

[mLipok]

  On 2/14/2018 at 6:57 AM, Ascer said:

3 year ago i totally removed Internet Explorer with my PC

Expand  

But in your example (given to @youtuber) you are still using IE components  >>   C:\Windows\SysWOW64\ieframe.dll
You can check this by using:

ObjName($oObj,4)

For this reason I ask you why you are not using standard IE.au3 UDF , as all this things which you are show in your example can be replaced by IE.au3 UDF

 

btw.
as to your comment:

  On 2/13/2018 at 4:32 PM, Ascer said:

@mLipok 
When i try to call a website to download a file or a lot of data via winhttp or InetRead my app got freeze until this action has end.

Expand  

about winhttp:
@trancexx in WinHttp.au3 in description to: _WinHttpOpen wrote:

  Quote

; Remarks .......: <b>You are strongly discouraged to use WinHTTP in asynchronous mode with AutoIt. AutoIt's callback implementation can't handle reentrancy properly.</b>
;                  +For asynchronous mode set [[$iFlag]] to [[$WINHTTP_FLAG_ASYNC]]. In that case [[$WINHTTP_OPTION_CONTEXT_VALUE]] for the handle will inernally be set to [[$WINHTTP_FLAG_ASYNC]] also.

Expand  

about: InetRead
When you need to download files you should use InetGet():

  Quote

By default the function waits until the download has finished before returning. If the background parameter is set to $INET_DOWNLOADBACKGROUND (1) the function returns immediately and the download continues in the background. The function InetGetInfo() can be used to check the status of the download. It takes the handle returned from InetGet().

Expand