TCPRecv Data Corruption

[Biatu]

Hello Team,

I keep getting bad data when downloading a file when using TCP in a follow manner:

$sData&=TCPRecv($hSocket,$iChunk,1)

I get this output:

0x370x7ABC0xAF271C000x044C4BE01D144
5490x07000000002700000000000000865D
230x67E2A175E0275D0026968E700017F7E
C05BBEAF4FF94012F44EF7CE6F5EC47270x
5FBFBB7AA32D7E41D591B95C14FE2A7E6B4

And when I use

$sData&=StringToBinary(TCPRecv($hSocket,$iChunk,1))

I get:

0x377AC2BCC2AF271C00044C4BC3A01D14454907000000002700000000000000E280A05D

Its Supposed to be this:

0x377ABCAF271C00044C4BE01D14454907000000002700000000000000865D2367E2A175

Looks like edianness, but how would I correct this?

Edit:
The correct file size is 116MB, and the TCPRecv gets 116MB, but I get a 182MB file.

Edited September 18, 2015 by Biatu

[jchd]

Looks like the input is UTF8 encoded. Try this:

$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))

But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file?

[Biatu]

Looks like the input is UTF8 encoded. Try this:

$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))

But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file?

If your refering to:

0x370x7ABC0xAF271C000x044C4BE01D144
5490x07000000002700000000000000865D
230x67E2A175E0275D0026968E700017F7E
C05BBEAF4FF94012F44EF7CE6F5EC47270x
5FBFBB7AA32D7E41D591B95C14FE2A7E6B4

This was populated straight from $var&=TCPRecv, and the reason that the length of each binary string within this differs is because I use a varying chunk size.

I called BinaryToSring with each call of TCPRecv for the other output, just like your example, but im not writing each call to a file, its going straight into

a variable.

The file is a binary 7z archive.

[Biatu]

Should have posted this sooner...
 

expandcollapse popup

#Include <Constants.au3>
#Include <Array.au3>
#include <WinAPIProc.au3>

GetData("http://download.drp.su/driverpacks/DP_Misc_15092.7z",True,False);"http://speedtest.tele2.net/1GB.zip"

Func GetData($sURL,$iReturnBinary=False,$iOffload=True)
    Local $aURL,$aHeader,$sDomain="",$sIP="",$sHeader="",$iSize=-1,$iRecieve=0,$iRecieved=0,$iPort=0,$iHeader=-1
    Local $sOffloadFile,$sData,$sRecieve,$aElement,$iChunk=(1024*1024*1),$iOffloadRecieve=0
    $aURL=StringRegExp($sURL,"(?i)(.*)://([^/]+)(.*)",1);Split URL
    If UBound($aURL,1)<2 Then Exit 1
    $sDomain=$aURL[1]
    $sURL=$aURL[2]
    Switch $aURL[0];Get Port from Protocol
        Case "http"
            $iPort=80
        Case else
            Exit 12
    EndSwitch
    TCPStartup()
    Opt("TCPTimeout",2000)
    _ArrayDisplay($aURL)
    $sIP=TCPNameToIP($sDomain);Resolve Host
    If @Error Then Exit 2
    $hSocket=TCPConnect($sIP,$iPort);Connect to Host
    If @error Then Exit 3
    
    ;Construct Header
    $sHeader="GET /"&$sURL&" HTTP/1.1"&@CRLF
    $sHeader&="Host: "&$sDomain&@CRLF
    $sHeader&="Connection: keep-alive"&@CRLF&@CRLF&@CRLF
    TCPSend($hSocket,$sHeader);Send Request
    If @Error Then Exit 4
    
    Do;Recieve Header
        $sData&=BinaryToString(TCPRecv($hSocket,1,0))
        If @Error Then ExitLoop
        $iRecieved=BinaryLen($sData)
        If $iRecieved=0 Then ContinueLoop
        If BinaryMid($sData,1+$iRecieved-4,4)=@CRLF&@CRLF Then
            ExitLoop
        EndIf
    Until False
    $aHeader=StringSplit($sData,@CRLF,1);Split Header
    _ArrayDisplay($aHeader)
    If @Error Then Exit 3
    _ArrayColInsert($aHeader,1); Format Header
    For $iIndex=1 to $aHeader[0][0]
        $aElement=StringSplit($aHeader[$iIndex][0],": ",3)
        If @Error Then ContinueLoop
        $aHeader[$iIndex][0]=$aElement[0]
        $aHeader[$iIndex][1]=$aElement[1]
    Next
    $iHeader=_ArraySearch($aHeader,"Content-Length");Get FileSize
    If Not @Error Then $iSize=$aHeader[$iHeader][1]
    $sData=""
    $iRecieved=0
    
    Do;Get Data
        $hTimer=TimerInit()
        $sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1))
        $iTime=TimerDiff($hTimer);Get time of recieve (Previously used to calculate iChunk)
        If BinaryLen($sRecieve)=0 Then
            ContinueLoop
        EndIf
        $sData&=$sRecieve
        $iRecieve=BinaryLen($sRecieve);Get Current Bytes
        $iRecieved+=BinaryLen($sRecieve); Populate Total Bytes
        ToolTip(Round(BinaryLen($sData)/1024/1024,3)&"|"&Round($iRecieved/1024/1024,3)&"|"&StringFormat("%04f",$iTime));Debug
        If $iChunk>$iRecieve Then;If Request is bigger than recieved, truncate.
            $iChunk=$iRecieve
        Else;Get Moar Data!
            $iChunk*=2
        EndIf
        If $iRecieved>=$iOffloadRecieve+(1024*1024*1) Then; If 1MB Reached, Offload to File and clear WS (Disabled/Pending)
            ConsoleWrite("Offload:"&$iRecieved&","&$iOffloadRecieve+(1024*1024*1)&@CRLF)
            $iOffloadRecieve+=$iRecieve+(1024*1024*1)
            ;$sData=""
            _WinAPI_EmptyWorkingSet()
        EndIf
        If $iSize<>-1 Then
            If $iRecieved=$iSize Then; Request Satisfied.
                FileWrite("Test.7z",$sData)
                ExitLoop
            EndIf
        EndIf
    Until False
EndFunc

 

[jchd]

Why you are converting binary (.7z) to a string is a mystery. Don't do that.

Also, FIleOpen the output file in binary mode before FileWrite.

Your script would be simpler if you used @trancexx HTTP or even IE UDF.

Edited September 19, 2015 by jchd

[Biatu]

Why you are converting binary (.7z) to a string is a mystery. Don't do that.

Also, FIleOpen the output file in binary mode before FileWrite.

Your script would be simpler if you used @trancexx HTTP or even IE UDF.

$sRecieve&=TCPRecv($hSocket,$iChunk,1)
Return: 0x370x7ABC0xAF271C000x044C4BE01D144...
Even though I TCPRecv binary, I end up with a string of binary strings.

I'm not intending to write a file with the data, that is for debugging. The file is to be stored in memory temporarily.

The file I am downloading is 116MB, and the function recieves 116MB, however the data that is written is 182MB. And that is where my issue comes from.

I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect.

[jchd]

This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string.

[Biatu]

This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string.

If you change 
$sRecieve&=BinaryToString(TCPRecv($hSocket,$iChunk,1))
to
$sRecieve&=TCPRecv($hSocket,$iChunk,1)

and run that script, you will see the difference in the output file with a hex editor.

[jchd]

Did you change the script to write the file in binary mode?

[guinness]

I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect.

Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points!

[Biatu]

Yes, and no change.

[Biatu]

Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points!

Good point, i will try.

[Biatu]

I apologize for the confusion, I fixed it.

;What did not work...

$sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1))
$sData&=$sRecieve

;What does...

$sRecieve=TCPRecv($hSocket,$iChunk,1)
$sData&=BinaryToString($sRecieve)

 

[jchd]

???

[Biatu]

???

I know....but trust me, the function works fine now.

Edited September 19, 2015 by Biatu

[Biatu]

Never mind, it was the FileWrite func. Even though when jchd previously advised to FileOpen as binary...it did not work. But now, for some reason it does.