Jump to content
Sign in to follow this  
FaridAgl

UrlDownloadEx() - Download with maximum speed

Recommended Posts

FaridAgl

If you ever tried to download some remote file from your script using InetGet(), InetRead() or other download functions, you probably noticed that the download speed was never as good as when you try to download the same file using a download manager.

 

But that's over, you won't lack the high speed download capability of download managers in your AutoIt scripts, not anymore.

 

Enjoy:

#include-once
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

#include <WinHttp.au3> ;http://www.autoitscript.com/forum/topic/84133-winhttp-functions/
#include <FileConstants.au3>

; #FUNCTION# ====================================================================================================================
; Name ..........: UrlDownloadEx
; Description ...: Download a single file by splitting it in segments and using several simultaneous connections to download these segments from a single server.
; Syntax ........: UrlDownloadEx($sUrl [, $sFileName = Default [, $iNumberOfConnections = Default [, $CallbackFunction = Default [, $vParam = 0]]]])
; Parameters ....: $sUrl                 - Url of the file to download.
;                  $sFileName            - [optional] Local filename to download to. Default is "".
;                  $iNumberOfConnections - [optional] Number of simultaneous connections. Default is 8.
;                  $CallbackFunction     - [optional] An application-defined callback function to call when progress is made. Default is Null.
;                  $vParam               - [optional] An application-defined value to be passed to the callback function. Default is 0.
; Return values .: Success - Returns a binary string if file name is not supplied.
;                          - Returns 1 if file name is supplied
;                          - @extended receives the number of received bytes.
;                  Failure - Returns 0 and sets @error:
;                  |1  - Invalid number of connections
;                  |2  - Invalid callback function
;                  |3  - Invalid url
;                  |4  - _WinHttpOpen failed
;                  |5  - _WinHttpConnect failed
;                  |6  - _WinHttpOpenRequest failed
;                  |7  - _WinHttpSendRequest failed
;                  |8  - _WinHttpReceiveResponse failed
;                  |9  - _WinHttpQueryHeaders failed
;                  |10 - _WinHttpOpenRequest failed, while trying to create multiple request
;                  |11 - _WinHttpQueryHeaders failed, while trying to prepare multiple request
;                  |12 - Not enough memory to allocate buffer
;                  |13 - _WinHttpQueryDataAvailable failed
;                  |14 - Download aborted, callback function returned False
;                  |15 - FileOpen failed
;                  |16 - FileWrite failed
; Author ........: FaridAgl
; Remarks .......: The Url parameter should be in the form "http://www.somesite.com/path/file.html", just like an address you would type into your web browser.
;                  + Please use a high number of connections with caution. Some servers may have specific limitations for a number of connections from one client, when the number of connections is high, the servers can put your computer to a black list.
;                  + For some connection types or for low performance computers or routers, the download speed may decrease when you increase the number of connections.
;                  + The callback function is called with the following parameters: $iReceivedBytes, $iTotalReceivedBytes, $iDownloadSize, $vParam
;                  + To continue downloading, the callback function must return True; to stop downloading, it must return False.
; Example .......: Yes
; ===============================================================================================================================
Func UrlDownloadEx($sUrl, $sFileName = Default, $iNumberOfConnections = Default, $CallbackFunction = Default, $vParam = 0)
    If ($sFileName = Default Or $sFileName = -1) Then $sFileName = ""

    If ($iNumberOfConnections = Default Or $iNumberOfConnections = -1) Then
        $iNumberOfConnections = 8
    ElseIf (IsInt($iNumberOfConnections) = 0 Or $iNumberOfConnections < 1) Then
        Return SetError(1, 0, 0)
    EndIf

    If ($CallbackFunction = Default Or $CallbackFunction = -1) Then
        $CallbackFunction = Null
    Else
        If (IsFunc($CallbackFunction) = 0) Then Return SetError(2, 0, 0)
    EndIf

    Local $avUrlComponents = _WinHttpCrackUrl($sUrl, $ICU_DECODE)
    If ($avUrlComponents = 0) Then Return SetError(3, 0, 0)

    Local Const $SERVER_NAME = $avUrlComponents[2]
    Local Const $OBJECT_NAME = $avUrlComponents[6]
    Local Const $SERVER_PORT = $avUrlComponents[3]
    Local Const $FLAGS = ($avUrlComponents[1] = $INTERNET_SCHEME_HTTPS) ? ($WINHTTP_FLAG_SECURE) : (0)

    Local $hSession = _WinHttpOpen("Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))", $WINHTTP_ACCESS_TYPE_DEFAULT_PROXY, $WINHTTP_NO_PROXY_NAME, $WINHTTP_NO_PROXY_BYPASS, 0)
    If ($hSession = 0) Then Return SetError(4, 0, 0)

    Local $hConnect = _WinHttpConnect($hSession, $SERVER_NAME, $SERVER_PORT)
    If ($hConnect = 0) Then
        _WinHttpCloseHandle($hSession)
        Return SetError(5, 0, 0)
    EndIf

    Local $hRequest = _WinHttpOpenRequest($hConnect, "GET", $OBJECT_NAME, 0, $WINHTTP_NO_REFERER, $WINHTTP_DEFAULT_ACCEPT_TYPES, $FLAGS)
    If ($hRequest = 0) Then
        _WinHttpCloseHandle($hConnect)
        _WinHttpCloseHandle($hSession)
        Return SetError(6, 0, 0)
    EndIf

    If (_WinHttpSendRequest($hRequest, "Range: bytes=0-0", $WINHTTP_NO_REQUEST_DATA, 0, 0) = 0) Then
        _WinHttpCloseHandle($hRequest)
        _WinHttpCloseHandle($hConnect)
        _WinHttpCloseHandle($hSession)
        Return SetError(7, 0, 0)
    EndIf

    If (_WinHttpReceiveResponse($hRequest) = 0) Then
        _WinHttpCloseHandle($hRequest)
        _WinHttpCloseHandle($hConnect)
        _WinHttpCloseHandle($hSession)
        Return SetError(8, 0, 0)
    EndIf

    Local $sStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE, $WINHTTP_HEADER_NAME_BY_INDEX, $WINHTTP_NO_HEADER_INDEX)
    If (Int($sStatusCode, 1) <> $HTTP_STATUS_PARTIAL_CONTENT) Then
        _WinHttpCloseHandle($hRequest)
        _WinHttpCloseHandle($hConnect)
        _WinHttpCloseHandle($hSession)
        Return SetError(9, 0, 0)
    EndIf

    Local $sContentRange = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_CONTENT_RANGE, $WINHTTP_HEADER_NAME_BY_INDEX, $WINHTTP_NO_HEADER_INDEX)
    _WinHttpCloseHandle($hRequest)

    Local Enum $REQUEST_HANDLE, $RECEIVED_BYTES, $DOWNLOAD_OFFSET, $BYTES_TO_DOWNLOAD
    Local $avConnections[$iNumberOfConnections][4]

    Local $iDownloadSize = Int(StringTrimLeft($sContentRange, StringLen("bytes 0-0/")), 1)
    Local $iDownloadSizePerConnection = Floor($iDownloadSize / $iNumberOfConnections)

    Local $iLastByteToDownload = 0
    For $i = 0 To $iNumberOfConnections - 1
        $avConnections[$i][$REQUEST_HANDLE] = _WinHttpOpenRequest($hConnect, "GET", $OBJECT_NAME, 0, $WINHTTP_NO_REFERER, $WINHTTP_DEFAULT_ACCEPT_TYPES, $FLAGS)
        If ($avConnections[$i][$REQUEST_HANDLE] = 0) Then
            For $j = $i - 1 To 0 Step -1
                _WinHttpCloseHandle($avConnections[$j][$REQUEST_HANDLE])
            Next

            _WinHttpCloseHandle($hConnect)
            _WinHttpCloseHandle($hSession)
            Return SetError(10, 0, 0)
        EndIf

        $avConnections[$i][$RECEIVED_BYTES] = 0
        $avConnections[$i][$DOWNLOAD_OFFSET] = $i * $iDownloadSizePerConnection
        If ($i <> $iNumberOfConnections - 1) Then
            $avConnections[$i][$BYTES_TO_DOWNLOAD] = $iDownloadSizePerConnection
            $iLastByteToDownload = $avConnections[$i][$DOWNLOAD_OFFSET] + $iDownloadSizePerConnection - 1
        Else
            $avConnections[$i][$BYTES_TO_DOWNLOAD] = $iDownloadSizePerConnection + Mod($iDownloadSize, $iNumberOfConnections)
            $iLastByteToDownload = $iDownloadSize - 1
        EndIf

        _WinHttpSendRequest($avConnections[$i][$REQUEST_HANDLE], "Range: bytes=" & $avConnections[$i][$DOWNLOAD_OFFSET] & "-" & $iLastByteToDownload, $WINHTTP_NO_REQUEST_DATA, 0, 0)
        _WinHttpReceiveResponse($avConnections[$i][$REQUEST_HANDLE])

        $sStatusCode = _WinHttpQueryHeaders($avConnections[$i][$REQUEST_HANDLE], $WINHTTP_QUERY_STATUS_CODE, $WINHTTP_HEADER_NAME_BY_INDEX, $WINHTTP_NO_HEADER_INDEX)
        If (Int($sStatusCode, 1) <> $HTTP_STATUS_PARTIAL_CONTENT) Then
            For $j = $i - 1 To 0 Step -1
                _WinHttpCloseHandle($avConnections[$j][$REQUEST_HANDLE])
            Next

            _WinHttpCloseHandle($hConnect)
            _WinHttpCloseHandle($hSession)
            Return SetError(11, 0, 0)
        EndIf
    Next

    Local $tBuffer = DllStructCreate("BYTE[" & $iDownloadSize & "]")
    If (@error) Then Return SetError(12, 0, 0)

    Local $pBuffer = DllStructGetPtr($tBuffer)

    Local $iTotalReceivedBytes = 0
    While (True)
        For $i = 0 To $iNumberOfConnections - 1
            If ($avConnections[$i][$RECEIVED_BYTES] = $avConnections[$i][$BYTES_TO_DOWNLOAD]) Then ContinueLoop

            If (_WinHttpQueryDataAvailable($avConnections[$i][$REQUEST_HANDLE]) = 1) Then
                $iTotalReceivedBytes += @extended

                _WinHttpReadData($avConnections[$i][$REQUEST_HANDLE], 2, @extended, $pBuffer + $avConnections[$i][$DOWNLOAD_OFFSET] + $avConnections[$i][$RECEIVED_BYTES])
                $avConnections[$i][$RECEIVED_BYTES] += @extended

                If ($CallbackFunction <> Null) Then
                    If ($CallbackFunction(@extended, $iTotalReceivedBytes, $iDownloadSize, $vParam) = False) Then
                        For $j = 0 To $iNumberOfConnections - 1
                            If ($avConnections[$j][$REQUEST_HANDLE] > 0) Then _WinHttpCloseHandle($avConnections[$j][$REQUEST_HANDLE])
                        Next

                        _WinHttpCloseHandle($hConnect)
                        _WinHttpCloseHandle($hSession)
                        Return SetError(14, 0, 0)
                    EndIf
                EndIf

                If ($avConnections[$i][$RECEIVED_BYTES] = $avConnections[$i][$BYTES_TO_DOWNLOAD]) Then
                    _WinHttpCloseHandle($avConnections[$i][$REQUEST_HANDLE])
                    $avConnections[$i][$REQUEST_HANDLE] = 0
                EndIf
            Else
                For $j = 0 To $iNumberOfConnections - 1
                    If ($avConnections[$j][$REQUEST_HANDLE] > 0) Then _WinHttpCloseHandle($avConnections[$j][$REQUEST_HANDLE])
                Next

                _WinHttpCloseHandle($hConnect)
                _WinHttpCloseHandle($hSession)
                Return SetError(13, 0, 0)
            EndIf
        Next

        If ($iTotalReceivedBytes = $iDownloadSize) Then
            _WinHttpCloseHandle($hConnect)
            _WinHttpCloseHandle($hSession)
            ExitLoop
        EndIf
    WEnd

    If ($sFileName <> "") Then
        Local $hFile = FileOpen($sFileName, BitOR($FO_OVERWRITE, $FO_CREATEPATH, $FO_BINARY))
        If ($hFile = -1) Then Return SetError(15, 0, 0)

        If (FileWrite($hFile, DllStructGetData($tBuffer, 1)) = 0) Then
            FileClose($hFile)
            Return SetError(16, 0, 0)
        EndIf

        FileClose($hFile)
        Return SetError(0, $iDownloadSize, 1)
    EndIf

    Return SetError(0, $iDownloadSize, DllStructGetData($tBuffer, 1))
EndFunc

Example 1 - Download to a local file and use callback function to monitor and control the download progress:

#include "UrlDownloadEx.au3"

Global $t1 = TimerInit()
Global $vResult = UrlDownloadEx("http://www.autoitscript.com/files/autoit3/autoit-v3-setup.exe", @ScriptDir & "\AutoIt 3.3.12.0.exe", Default, BytesReceived)
ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))

Func BytesReceived($iReceivedBytes, $iTotalReceivedBytes, $iDownloadSize, $vParam)
    If ($iTotalReceivedBytes >= 2 * 1024 * 1024) Then Return False ;Stop downloading

    ConsoleWrite(StringFormat("%u bytes received.\n%u/%u\nParam: %s\n\n", $iReceivedBytes, $iTotalReceivedBytes, $iDownloadSize, $vParam))
    Return True ;Continue downloading
EndFunc

Example 2 - Download to memory and set the number of connections to 2:

#include "UrlDownloadEx.au3"

Global $t1 = TimerInit()
Global $vResult = UrlDownloadEx("http://icdn4.digitaltrends.com/image/microsoft_xp_bliss_desktop_image-650x0.jpg", Default, 2)
ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))

You can also download all of the above scripts at once:

UrlDownloadEx + Examples.zip

Edited by FaridAgl
  • Like 1

Share this post


Link to post
Share on other sites
coffeeturtle

Thank you for sharing this! Very nice job!

Share this post


Link to post
Share on other sites
guinness

Of course I ran it once and was like "Where is the AutoIt exe?" but then I commented out line 8 and voila it worked. I somehow get the feeling I gave you inspiration for the delegate function BytesReceived() concept. -_0

5* from me at least.

Edited by guinness

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
guinness

Could you explain the usage of $vParam to me please. I get it's used in the callback function, but I can't see why it is needed.


UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
FaridAgl

I somehow get the feeling I gave you inspiration for the delegate function BytesReceived() concept. -_0

Well, I was going to admit it ;)

 

Could you explain the usage of $vParam to me please. I get it's used in the callback function, but I can't see why it is needed.

I believe this is somehow the usual way of defining a callback (delegate), an additional parameter that can deliver whatever you may need into your callback function.

Share this post


Link to post
Share on other sites
FaridAgl

Oh, I forgot to say.

it's not possible to abort (cancel) the download by returning False from the callback function, and retrieve the downloaded data, that makes no sense because the file is getting downloaded as multiple segments, and there isn't any reliable data to retrieve as long as the download isn't completed. This is why 1st example doesn't create any file.

Share this post


Link to post
Share on other sites
guinness

Well, I was going to admit it ;)

 

I believe this is somehow the usual way of defining a callback (delegate), an additional parameter that can deliver whatever you may need into your callback function.

Haha. I understand, it's some sort of user parameter.

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
FaridAgl

Haha. I understand, it's some sort of user parameter.

That's so true.

 

Thank you for sharing this! Very nice job!

Thank you for your support.

Also I would like to say thank you to trancexx, because of her great WinHTTP UDF.

Share this post


Link to post
Share on other sites
FaridAgl

Latest version.

What error did you get? Check the @error value.

Share this post


Link to post
Share on other sites
GtaSpider

Hey,

Nice work! The speed difference surely depends on the speed of the server, and how much speed it is able to give per connection. I made a little Example where you can see the speed. The file is hosted on a very fast server, so the difference shouldn't be that huge. But you can try it with other files from slower servers or may servers which are on the other side of the world (from your position ;) )

Thanks for sharing Farid!

#include "UrlDownloadEx.au3"


HotKeySet("{ESC}","_exit")
Global $t1 = TimerInit()
ProgressOn("Exit with {ESC}", "Download 50mb","Start download...")
AdlibRegister("_update",250)



Global $vResult = UrlDownloadEx("http://download.thinkbroadband.com/50MB.zip", @ScriptDir & "\[temp]50MB.zip", 1, BytesReceived);Quite the same speed over the complete download
ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))
AdlibRegister("_update",1000)

If Not BytesReceived(0,0,0,0) Then Exit;if abborted, exit

Global $vResult = UrlDownloadEx("http://download.thinkbroadband.com/50MB.zip", @ScriptDir & "\[temp]50MB2.zip", Default, BytesReceived);speed differ from time to time. you have to but an higher amout of MS in the AdlibRegister (like 1000 instead of 250)
ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))

Func BytesReceived($iReceivedBytes, $iTotalReceivedBytes, $iDownloadSize, $vParam)
    Local Static $hTimer = TimerInit(), $iBytes, $iKbS, $sKbS, $iDownloadSize2, $iTotalReceivedBytes2, $fExit = False
    If $vParam = 1 Then
        ProgressSet($iTotalReceivedBytes2 / $iDownloadSize2 * 100, StringFormat("%.1f%%\r\n%.2f %s",$iTotalReceivedBytes2 / $iDownloadSize2 * 100,$iKbS,$sKbS))
        Local $iTime = 0
        $iKbS = 1000 / TimerDiff($hTimer) * $iBytes
        While $iKbS > 1000
            $iTime += 1
            $iKbS /= 1024
        WEnd
        Switch $iTime
            Case 0
                $sKbS = "Bytes/s"
            Case 1
                $sKbS = "KB/s"
            Case 2
                $sKbS = "MB/s"
            Case 3
                $sKbS = "GB/s"
        EndSwitch

        $iKbS = $iKbS
        $hTimer = TimerInit()
        $iBytes = 0
    ElseIf $vParam = 2 Then
        $fExit = True
    Else
        $iBytes += $iReceivedBytes
        $iDownloadSize2 = $iDownloadSize
        $iTotalReceivedBytes2 = $iTotalReceivedBytes
    EndIf
    Return Not $fExit
EndFunc   ;==>BytesReceived


Func _update()
    BytesReceived(0, 0, 0, 1)
EndFunc   ;==>speed

Func _exit()
    BytesReceived(0, 0, 0, 2)
EndFunc

Greetz,

Spider 

Example 3.au3

Edited by GtaSpider

www.AutoIt.de - Moderator of the German AutoIt Froum

Share this post


Link to post
Share on other sites
FaridAgl

Thank you, a very nice example.

You are absolutely true, it depends on the servers and how the server is configured. Usually servers have a download speed limit per each connection, so downloading a file with multiple connection can boost the download speed.

However, if your Internet connection's download bandwidth is low, or the server has no limit on the download speed, the difference shouldn't be that much.

Also some servers won't allow multiple connections for downloading a single file, or they don't even support the "Range" http header field.

But, most of the times, you would enjoy using this method of downloading.

Share this post


Link to post
Share on other sites
mesale0077

>"C:\Users\hppc\Desktop\autoit-v3.3.11.6\install\SciTE4AutoIt3(8)\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\hppc\Desktop\UrlDownloadEx + Examples(1)\Example 1.au3" /UserParams    
+>15:37:05 Starting AutoIt3Wrapper v.2.2.0.3 SciTE v.3.4.1.0   Keyboard:0001041F  OS:WIN_8/  CPU:X64 OS:X64    Environment(Language:041F)
+>         SciTEDir => C:\Users\hppc\Desktop\autoit-v3.3.11.6\install\SciTE4AutoIt3(8)   UserDir => C:\Users\hppc\Desktop\autoit-v3.3.11.6\install\SciTE4AutoIt3(8)\AutoIt3Wrapper
>Running AU3Check (3.3.11.6)  from:C:\Users\hppc\Desktop\autoit-v3.3.11.6\install  input:C:\Users\hppc\Desktop\UrlDownloadEx + Examples(1)\Example 1.au3
+>15:37:05 AU3Check ended.rc:0
>Running:(3.3.11.6):C:\Users\hppc\Desktop\autoit-v3.3.11.6\install\autoit3.exe "C:\Users\hppc\Desktop\UrlDownloadEx + Examples(1)\Example 1.au3"
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

this code worked

but not 50mb

try it

#include "UrlDownloadEx.au3"


HotKeySet("{ESC}","_exit")
Global $t1 = TimerInit()
ProgressOn("Exit with {ESC}", "Download 50mb","Start download...")
AdlibRegister("_update",250)



Global $vResult = UrlDownloadEx("http://www.autoitscript.com/files/autoit3/autoit-v3-setup.exe", @ScriptDir & "\AutoIt 3.3.12.0.exe", 1, BytesReceived);Quite the same speed over the complete download
ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))
AdlibRegister("_update",1000)

If Not BytesReceived(0,0,0,0) Then Exit;if abborted, exit

;Global $vResult = UrlDownloadEx("http://download.thinkbroadband.com/50MB.zip", @ScriptDir & "\[temp]50MB2.zip", Default, BytesReceived);speed differ from time to time. you have to but an higher amout of MS in the AdlibRegister (like 1000 instead of 250)
;ConsoleWrite(StringFormat("Result:\t%s\nSize:\t%u\nError:\t%u\nTimer:\t%u\n", $vResult, @extended, @error, TimerDiff($t1)))

Func BytesReceived($iReceivedBytes, $iTotalReceivedBytes, $iDownloadSize, $vParam)
    Local Static $hTimer = TimerInit(), $iBytes, $iKbS, $sKbS, $iDownloadSize2, $iTotalReceivedBytes2, $fExit = False
    If $vParam = 1 Then
        ProgressSet($iTotalReceivedBytes2 / $iDownloadSize2 * 100, StringFormat("%.1f%%\r\n%.2f %s",$iTotalReceivedBytes2 / $iDownloadSize2 * 100,$iKbS,$sKbS))
        Local $iTime = 0
        $iKbS = 1000 / TimerDiff($hTimer) * $iBytes
        While $iKbS > 1000
            $iTime += 1
            $iKbS /= 1024
        WEnd
        Switch $iTime
            Case 0
                $sKbS = "Bytes/s"
            Case 1
                $sKbS = "KB/s"
            Case 2
                $sKbS = "MB/s"
            Case 3
                $sKbS = "GB/s"
        EndSwitch

        $iKbS = $iKbS
        $hTimer = TimerInit()
        $iBytes = 0
    ElseIf $vParam = 2 Then
        $fExit = True
    Else
        $iBytes += $iReceivedBytes
        $iDownloadSize2 = $iDownloadSize
        $iTotalReceivedBytes2 = $iTotalReceivedBytes
    EndIf
    Return Not $fExit
EndFunc   ;==>BytesReceived


Func _update()
    BytesReceived(0, 0, 0, 1)
EndFunc   ;==>speed

Func _exit()
    BytesReceived(0, 0, 0, 2)
EndFunc
 added udf but dont work ?

but example 3 worked

dowload

Edited by mesale0077

Share this post


Link to post
Share on other sites
FaridAgl

Error:  14

It means the callback function returned False, so the download got aborted.

Check your code and make sure you are returning True from the callback function all the times.

Remember, it's necessary for the callback function to return True, if you don't return any value from the callback function, it will be considered as False and the download progress will be aborted.

Share this post


Link to post
Share on other sites
t0nZ

Fantastic work!

I tested It in a dirty download benchmark with temporary links-> .

But Combofix.exe don't get downloaded...

Share this post


Link to post
Share on other sites
FaridAgl

Is this the link?

http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe

If yes, I can't even download it in the browser, because:

404: Page Not Found

Share this post


Link to post
Share on other sites
t0nZ

The static link is only 

http://www.bleepingcomputer.com/download/combofix/dl/12/

my example search the temporary link (using the static link and the filename.exe as parameters) and it tries to download.

the temporary link is like

http://download.bleepingcomputer.com/dl/3a314486dc4cdc746d35b26ea1bb0837/53b10c72/windows/security/anti-virus/c/combofix/ComboFix.exe

Your UrlDownloadEx() is OK with the other two downloads, I think it's a property of the combofix hosting server, maybe it doesn't permit downloading in pieces ?

Share this post


Link to post
Share on other sites
FaridAgl

Yes, that's possible and can be the reason.

Check the @error value and tell me what is that.

Share this post


Link to post
Share on other sites
t0nZ

Tried to download Combofix:

|8  - _WinHttpReceiveResponse failed

Share this post


Link to post
Share on other sites
FaridAgl

I've tested it with this temporary link:

http://download.bleepingcomputer.com/dl/ccf409088ff492398a2318ab3b9dc11e/53b199e7/windows/security/anti-virus/c/combofix/ComboFix.exe

And it just worked.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×