Biatu

TCPRecv Data Corruption

16 posts in this topic

#1 ·  Posted (edited)

Hello Team,

I keep getting bad data when downloading a file when using TCP in a follow manner:

$sData&=TCPRecv($hSocket,$iChunk,1)

I get this output:

0x370x7ABC0xAF271C000x044C4BE01D144
5490x07000000002700000000000000865D
230x67E2A175E0275D0026968E700017F7E
C05BBEAF4FF94012F44EF7CE6F5EC47270x
5FBFBB7AA32D7E41D591B95C14FE2A7E6B4

And when I use

$sData&=StringToBinary(TCPRecv($hSocket,$iChunk,1))

I get:

0x377AC2BCC2AF271C00044C4BC3A01D14454907000000002700000000000000E280A05D

Its Supposed to be this:

0x377ABCAF271C00044C4BE01D14454907000000002700000000000000865D2367E2A175

Looks like edianness, but how would I correct this?

Edit:
The correct file size is 116MB, and the TCPRecv gets 116MB, but I get a 182MB file.

Edited by Biatu

What is what? What is what.

Share this post


Link to post
Share on other sites



Looks like the input is UTF8 encoded. Try this:

$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))

But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Looks like the input is UTF8 encoded. Try this:

$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))

But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file?

If your refering to:

0x370x7ABC0xAF271C000x044C4BE01D144
5490x07000000002700000000000000865D
230x67E2A175E0275D0026968E700017F7E
C05BBEAF4FF94012F44EF7CE6F5EC47270x
5FBFBB7AA32D7E41D591B95C14FE2A7E6B4

This was populated straight from $var&=TCPRecv, and the reason that the length of each binary string within this differs is because I use a varying chunk size.

I called BinaryToSring with each call of TCPRecv for the other output, just like your example, but im not writing each call to a file, its going straight into

a variable.

The file is a binary 7z archive.


What is what? What is what.

Share this post


Link to post
Share on other sites

Should have posted this sooner...
 

#Include <Constants.au3>
#Include <Array.au3>
#include <WinAPIProc.au3>

GetData("http://download.drp.su/driverpacks/DP_Misc_15092.7z",True,False);"http://speedtest.tele2.net/1GB.zip"

Func GetData($sURL,$iReturnBinary=False,$iOffload=True)
    Local $aURL,$aHeader,$sDomain="",$sIP="",$sHeader="",$iSize=-1,$iRecieve=0,$iRecieved=0,$iPort=0,$iHeader=-1
    Local $sOffloadFile,$sData,$sRecieve,$aElement,$iChunk=(1024*1024*1),$iOffloadRecieve=0
    $aURL=StringRegExp($sURL,"(?i)(.*)://([^/]+)(.*)",1);Split URL
    If UBound($aURL,1)<2 Then Exit 1
    $sDomain=$aURL[1]
    $sURL=$aURL[2]
    Switch $aURL[0];Get Port from Protocol
        Case "http"
            $iPort=80
        Case else
            Exit 12
    EndSwitch
    TCPStartup()
    Opt("TCPTimeout",2000)
    _ArrayDisplay($aURL)
    $sIP=TCPNameToIP($sDomain);Resolve Host
    If @Error Then Exit 2
    $hSocket=TCPConnect($sIP,$iPort);Connect to Host
    If @error Then Exit 3
    
    ;Construct Header
    $sHeader="GET /"&$sURL&" HTTP/1.1"&@CRLF
    $sHeader&="Host: "&$sDomain&@CRLF
    $sHeader&="Connection: keep-alive"&@CRLF&@CRLF&@CRLF
    TCPSend($hSocket,$sHeader);Send Request
    If @Error Then Exit 4
    
    Do;Recieve Header
        $sData&=BinaryToString(TCPRecv($hSocket,1,0))
        If @Error Then ExitLoop
        $iRecieved=BinaryLen($sData)
        If $iRecieved=0 Then ContinueLoop
        If BinaryMid($sData,1+$iRecieved-4,4)=@CRLF&@CRLF Then
            ExitLoop
        EndIf
    Until False
    $aHeader=StringSplit($sData,@CRLF,1);Split Header
    _ArrayDisplay($aHeader)
    If @Error Then Exit 3
    _ArrayColInsert($aHeader,1); Format Header
    For $iIndex=1 to $aHeader[0][0]
        $aElement=StringSplit($aHeader[$iIndex][0],": ",3)
        If @Error Then ContinueLoop
        $aHeader[$iIndex][0]=$aElement[0]
        $aHeader[$iIndex][1]=$aElement[1]
    Next
    $iHeader=_ArraySearch($aHeader,"Content-Length");Get FileSize
    If Not @Error Then $iSize=$aHeader[$iHeader][1]
    $sData=""
    $iRecieved=0
    
    Do;Get Data
        $hTimer=TimerInit()
        $sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1))
        $iTime=TimerDiff($hTimer);Get time of recieve (Previously used to calculate iChunk)
        If BinaryLen($sRecieve)=0 Then
            ContinueLoop
        EndIf
        $sData&=$sRecieve
        $iRecieve=BinaryLen($sRecieve);Get Current Bytes
        $iRecieved+=BinaryLen($sRecieve); Populate Total Bytes
        ToolTip(Round(BinaryLen($sData)/1024/1024,3)&"|"&Round($iRecieved/1024/1024,3)&"|"&StringFormat("%04f",$iTime));Debug
        If $iChunk>$iRecieve Then;If Request is bigger than recieved, truncate.
            $iChunk=$iRecieve
        Else;Get Moar Data!
            $iChunk*=2
        EndIf
        If $iRecieved>=$iOffloadRecieve+(1024*1024*1) Then; If 1MB Reached, Offload to File and clear WS (Disabled/Pending)
            ConsoleWrite("Offload:"&$iRecieved&","&$iOffloadRecieve+(1024*1024*1)&@CRLF)
            $iOffloadRecieve+=$iRecieve+(1024*1024*1)
            ;$sData=""
            _WinAPI_EmptyWorkingSet()
        EndIf
        If $iSize<>-1 Then
            If $iRecieved=$iSize Then; Request Satisfied.
                FileWrite("Test.7z",$sData)
                ExitLoop
            EndIf
        EndIf
    Until False
EndFunc

 


What is what? What is what.

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

Why you are converting binary (.7z) to a string is a mystery. Don't do that.

Also, FIleOpen the output file in binary mode before FileWrite.

Your script would be simpler if you used @trancexx HTTP or even IE UDF.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Why you are converting binary (.7z) to a string is a mystery. Don't do that.

Also, FIleOpen the output file in binary mode before FileWrite.

Your script would be simpler if you used @trancexx HTTP or even IE UDF.

$sRecieve&=TCPRecv($hSocket,$iChunk,1)
Return: 0x370x7ABC0xAF271C000x044C4BE01D144...
Even though I TCPRecv binary, I end up with a string of binary strings.


I'm not intending to write a file with the data, that is for debugging. The file is to be stored in memory temporarily.

The file I am downloading is 116MB, and the function recieves 116MB, however the data that is written is 182MB. And that is where my issue comes from.

I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect.


What is what? What is what.

Share this post


Link to post
Share on other sites

This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string.

If you change 
$sRecieve&=BinaryToString(TCPRecv($hSocket,$iChunk,1))
to
$sRecieve&=TCPRecv($hSocket,$iChunk,1)

and run that script, you will see the difference in the output file with a hex editor.


What is what? What is what.

Share this post


Link to post
Share on other sites

Did you change the script to write the file in binary mode?


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect.

Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points!


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

Yes, and no change.


What is what? What is what.

Share this post


Link to post
Share on other sites

Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points!

Good point, i will try.


What is what? What is what.

Share this post


Link to post
Share on other sites

I apologize for the confusion, I fixed it.

;What did not work...

$sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1))
$sData&=$sRecieve

;What does...

$sRecieve=TCPRecv($hSocket,$iChunk,1)
$sData&=BinaryToString($sRecieve)

 


What is what? What is what.

Share this post


Link to post
Share on other sites

???


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

???

I know....but trust me, the function works fine now.

Edited by Biatu

What is what? What is what.

Share this post


Link to post
Share on other sites

Never mind, it was the FileWrite func. Even though when jchd previously advised to FileOpen as binary...it did not work. But now, for some reason it does.
 


What is what? What is what.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now