Multi segment file downloading

[qsek]

The reason i wrote this was because i didnt found any download manager in the whole web! where you can customize the starting position of the segments.

Most programs will just start one segment at 0%, the second at 50%, third at %25 ,75%, 12,5% and so on.

There is very little information about multi part/segement downloading in the forums so i gathered some informations and knocked this script together.

No messing with dlls or IE functions, just plain TCPSend and TCPRecv (i like it pure . You can adjust the header to your needs.

The initial segment size is 2 MB and the segments will start downloading from the beginning of the file.

This is ideal for viewing movies or avi files while downloading.

You can adjust the segment size while downloading.

If you want more then 4 segements you'll have to figure out yourself how to do it

The GUI is still basic. I was too lazy to implement the Progress bar.

You can see the completion percent in the consolewrites as soon as a segment finishes.

expandcollapse popup

#include <Array.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>


;Create a GUI with a progress indicator for download status
; #### Unfinished ####
Opt("GUIOnEventMode", 1)

$Form1 = GUICreate("Multi Segment downloader", 294, 56, 192, 124)
$Progress1 = GUICtrlCreateProgress(8, 8, 275, 20)
$Label1 = GUICtrlCreateLabel("Chunksize (MB):", 8, 32)
$edit = GUICtrlCreateInput("2",90,32,50,20)
$Updown1 = GUICtrlCreateUpdown($edit)
GUISetState(@SW_SHOW)



TCPStartup()

$sFile = @ScriptDir & '\file.avi' ; File to wrtie the download data to

$download= "http://speedtest.qsc.de/500MB.qsc" ; Internetfile to download

$refer = $download ; $refer = complete http path

$download = StringReplace($download,"http://","",1)

$string = StringSplit($download, "/",2)

;$string[0] = speedtest.qsc.de ; needed for 'Host:' in http referr

$PFileURL = ""
For $i = 1 To UBound($string)-1
    $PFileURL &= "/"&$string[$i]
next

;$PFileURL = /folder/misc/500MB.qsc ; needed for file path in http referr

TCPStartup()


$ip = TCPNameToIP($string[0])

ConsoleWrite($string[0]&" -> IP: "&$ip & @CRLF)


$size = 10000000 ; preset the size to 10MB, just in case we have an problem getting it from http responses

;~ $size = InetGetSize("http://"&$download) ; <-- some servers dont allow this

ConsoleWrite("Size of file: "&$size&" bytes"& @CRLF)


dim $chunkarr[4][7] ; [4] = maximum nr of simultanious chunks
;[0] = socket
;[1] = chunkstart
;[2] = chunkend
;[3] = chunksize
;[4] = data
;[5] = downloding
;[6] found the 0d0a0d0a after header?
ConsoleWrite(@CRLF& @CRLF & @CRLF)

$chunksize = GUICtrlRead($edit)*1024*1024 ; initial size of each segment

ConsoleWrite("Chunksize: "&$chunksize&" bytes"& @CRLF)

$nextchunkPos = 0 ; Starting at Byte

FileDelete ( $sFile)

$hFile = _WinAPI_CreateFile($sFile, 3,4,2)

While 1
    $chunksize = GUICtrlRead($edit)*1024*1024
    Sleep(50)
    $chunksactive = 0
    For $i = 0 To UBound($chunkarr)-1 ; go through each chunk
        if $chunkarr[$i][5] = 0 and $nextchunkPos < $size then ; Unused ChunkSlot --> Request new chunk
            
            $chunkarr[$i][1] = $nextchunkPos    ; start
            $chunkarr[$i][2] = $nextchunkPos+$chunksize-1   ; end
            
            $chunkarr[$i][4] = Binary("")       ; data
            $chunkarr[$i][5] = 1        ; downloading
            $chunkarr[$i][6] = 0        ; found the 0d0a0d0a after header
            
            if $chunkarr[$i][2]+$chunksize >= $size then $chunkarr[$i][2] = $size-1; if last chunk set end to last byte

            $chunkarr[$i][3] = $chunkarr[$i][2]-$chunkarr[$i][1]+1  ; size
            
            RequestChunk($i)

            $nextchunkPos = $chunkarr[$i][2]+1

        endif
        
        if $chunkarr[$i][5] = 1 then ; current chunk is marked downloading
            $chunksactive += 1
            RecvChunk($i)
        endif
        
        
        
    next
    if $chunksactive = 0 then ExitLoop
WEnd

ConsoleWrite( @CRLF&" #####    DOWNLOAD FINISHED    #####" & @CRLF)
_WinAPI_CloseHandle($hFile)

Exit


func RequestChunk($chunknr)
    
    ; Prepare Header
    $h  = "GET "&$PFileURL&" HTTP/1.1" & @CRLF
    $h &= "Referer: "&$refer & @CRLF
    $h &= "Range: bytes=" & $chunkarr[$chunknr][1] & "-" & $chunkarr[$chunknr][2] & @CRLF ; <----  ### important part for multi segment downloading ###
    $h &= "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8) Gecko/20100722 Ant.com Toolbar 2.0.1 Firefox/3.6.8" & @CRLF
    $h &= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" & @CRLF
    $h &= "Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3" & @CRLF
    $h &= "Accept-Encoding: gzip,deflate" & @CRLF
    $h &= "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" & @CRLF
    $h &= "Keep-Alive: 115" & @CRLF
    $h &= "Connection: keep-alive" & @CRLF
    $h &= "Host: " & $string[0] & @CRLF
    $h &= "Pragma: no-cache" & @CRLF
    $h &= @CRLF
    
;~  ConsoleWrite("+############### HEADER ################"& @CRLF)
;~  ConsoleWrite($h & @CRLF)

;~  ConsoleWrite("+-------------------- HEADER END ------------------"& @CRLF)
    
    $chunkarr[$chunknr][0] = TCPConnect($ip, 80)
    $err=@error
    $bs = TCPSend( $chunkarr[$chunknr][0], $h)
    ConsoleWrite( "-Chunk "&$chunknr&" request: Range: bytes= " & $chunkarr[$chunknr][1] & "-" & $chunkarr[$chunknr][2] &" / "&$chunkarr[$i][3]&" ("&$bs&" bytes sent)"&@CRLF)
    
    If  $chunkarr[$chunknr][0] = -1 Or $err Then
        ConsoleWrite("  TCPConnect error" & @CRLF)
        Exit
    EndIf

endfunc


func RecvChunk($chunknr)

    $data = TCPRecv( $chunkarr[$chunknr][0], 800,1)
    
    If @error <> 0 Then ; this chunk closed its connection
        ConsoleWrite(@CRLF&"Closed Connection (Chunk "&$chunknr&" finished)" & @CRLF)

        if BinaryLen($chunkarr[$chunknr][4]) < $chunkarr[$chunknr][3] then ; downloaded bytes are not equal requested size
            $cs = TCPCloseSocket ( $chunkarr[$chunknr][0] )
            ConsoleWrite("!Chunk mismatch: "&$chunknr&" finished ("&$cs&") with "&BinaryLen($chunkarr[$chunknr][4])&" bytes ("&floor(($nextchunkPos/$size)*100)&"%)"&@CRLF)
            
            ConsoleWrite(BinaryToString($chunkarr[$chunknr][4]) & @CRLF)
            ConsoleWrite("!resetting chunk("&$chunknr&") data and request again... " & @CRLF)
            $chunkarr[$chunknr][4] = Binary("")     ; reset data

            RequestChunk($chunknr)
            
        else; Chunk finished
            $chunkarr[$chunknr][5] = 0 
            $tBuffer = DLLStructCreate("byte[" & BinaryLen ($chunkarr[$chunknr][4]) & "]")

            DLLStructSetData($tBuffer, 1, $chunkarr[$chunknr][4])
            local $nBytes

            $fp = _WinAPI_SetFilePointer($hFile, $chunkarr[$chunknr][1])
    ;~      ConsoleWrite("File Pointer to "&$chunkarr[$chunknr][1]&" --> Result: "&$fp & @CRLF)

            $wf = _WinAPI_WriteFile($hFile,DLLStructGetPtr($tBuffer), $chunkarr[$i][3], $nBytes)
    ;~      ConsoleWrite("Write File "&$chunkarr[$i][3]&" bytes: "&$wf & @CRLF)

            $tBuffer = 0
            ;~ ConsoleWrite('3) ' & FileRead($sFile) & @LF)
            $cs = TCPCloseSocket ( $chunkarr[$chunknr][0] )
            ConsoleWrite("+Chunk "&$chunknr&" finished ("&$cs&") with "&BinaryLen($chunkarr[$chunknr][4])&" bytes ("&floor(($nextchunkPos/$size)*100)&"%)"&@CRLF)
        
        endif
        return 0
    endif

    ; This chunk is still receiving data..
    if not ($data = "") then
        $timer = TimerInit()
        do
            if $chunkarr[$chunknr][6] = 0 then ; if we didnt found the 0d0a0d0a position yet
                $data2 = binary("")
                For $i = 0 To BinaryLen($data)
                    ; search the first bytes of the incoming packet for the 0d0a0d0a string
                    ; $data2 will be used as fill up variable and usually contains the whole header
                    ; cant use stringinstr or Regex functions since any 0x00 character in the file will mess both up seriously!
                    $bit = BinaryMid($data,$i,1)
                    $data2 &= $bit

                    if StringRight($data2,8) = binary("0D0A0D0A") then ; found header, stored in $data2
                        $marr = StringRegExp(BinaryToString($data2),"Content-Range: bytes \d+-\d+/(\d+)",3) ; get exact size of file from header
                        
                        $size = $marr[0]

                        $data2 = binary("")
                        $chunkarr[$chunknr][6] = 1 ; found the 0d0a0d0a position
                        ConsoleWrite("Content Range Size: [" &$marr[0]&"], 0D0A0D0A pos: ["&$i&"]"&@CRLF)
                    endif
                Next
                $data = $data2; reset $data, from now on the file starts
            endif

            $chunkarr[$chunknr][4] &= $data ; save the data of this chunk to global array
            $data = TCPRecv( $chunkarr[$chunknr][0], 4048,1)
            
        until $data == "" or TimerDiff($timer)  > 10000 ; 10 second timeout
        
    endif
    
    return 1
endfunc

[KaFu]

Looks nice, gos straight to my snippet folder .

[ripdad]

qsek, nice script

I've been tweaking on it a little - hope you don't mind.

Would like to see it come to full fruition - hope others will join in.

Among other things - I've added 1 progress bar.

I tried to add 4 of them, but it was too cpu intensive.

So, the one progress bar only samples the 1st segment in real time.

(it is not an exact percentage - but a representation of overall progress of all 4 segments combined)

- Edit: ( old script deleted - See new post below )

Edited October 25, 2010 by ripdad

[wakillon]

Good work both !

This look promising !

[ripdad]

Update:

- Rearranged and cleaned code

- Changed algo

- Added InputBox

- Added Time

Edit - ( old script deleted )

- Some improvements & added a few goodies

Off to a new project .. hopefully someone else will pick up!

MultiSegment_Downloader.au3

Edited October 27, 2010 by ripdad

[ripdad]

Last Update