Biatu Posted September 18, 2015 Share Posted September 18, 2015 (edited) Hello Team,I keep getting bad data when downloading a file when using TCP in a follow manner:$sData&=TCPRecv($hSocket,$iChunk,1)I get this output:0x370x7ABC0xAF271C000x044C4BE01D144 5490x07000000002700000000000000865D 230x67E2A175E0275D0026968E700017F7E C05BBEAF4FF94012F44EF7CE6F5EC47270x 5FBFBB7AA32D7E41D591B95C14FE2A7E6B4And when I use$sData&=StringToBinary(TCPRecv($hSocket,$iChunk,1))I get:0x377AC2BCC2AF271C00044C4BC3A01D14454907000000002700000000000000E280A05DIts Supposed to be this:0x377ABCAF271C00044C4BE01D14454907000000002700000000000000865D2367E2A175Looks like edianness, but how would I correct this?Edit:The correct file size is 116MB, and the TCPRecv gets 116MB, but I get a 182MB file. Edited September 18, 2015 by Biatu What is what? What is what. Link to comment Share on other sites More sharing options...
jchd Posted September 18, 2015 Share Posted September 18, 2015 Looks like the input is UTF8 encoded. Try this:$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file? This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Looks like the input is UTF8 encoded. Try this:$sData = BinaryToString(TCPRecv($hSocket, $iChunk, $SB_UTF8))But don't use StringToBinary and only apply BinaryToString on the whole lot. Also I wonder whether this is actually useful as your snippet includes strings of binary zeroes - unusual in real text. Are you certain this is a text file?If your refering to:0x370x7ABC0xAF271C000x044C4BE01D144 5490x07000000002700000000000000865D 230x67E2A175E0275D0026968E700017F7E C05BBEAF4FF94012F44EF7CE6F5EC47270x 5FBFBB7AA32D7E41D591B95C14FE2A7E6B4This was populated straight from $var&=TCPRecv, and the reason that the length of each binary string within this differs is because I use a varying chunk size.I called BinaryToSring with each call of TCPRecv for the other output, just like your example, but im not writing each call to a file, its going straight intoa variable.The file is a binary 7z archive. What is what? What is what. Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Should have posted this sooner... expandcollapse popup#Include <Constants.au3> #Include <Array.au3> #include <WinAPIProc.au3> GetData("http://download.drp.su/driverpacks/DP_Misc_15092.7z",True,False);"http://speedtest.tele2.net/1GB.zip" Func GetData($sURL,$iReturnBinary=False,$iOffload=True) Local $aURL,$aHeader,$sDomain="",$sIP="",$sHeader="",$iSize=-1,$iRecieve=0,$iRecieved=0,$iPort=0,$iHeader=-1 Local $sOffloadFile,$sData,$sRecieve,$aElement,$iChunk=(1024*1024*1),$iOffloadRecieve=0 $aURL=StringRegExp($sURL,"(?i)(.*)://([^/]+)(.*)",1);Split URL If UBound($aURL,1)<2 Then Exit 1 $sDomain=$aURL[1] $sURL=$aURL[2] Switch $aURL[0];Get Port from Protocol Case "http" $iPort=80 Case else Exit 12 EndSwitch TCPStartup() Opt("TCPTimeout",2000) _ArrayDisplay($aURL) $sIP=TCPNameToIP($sDomain);Resolve Host If @Error Then Exit 2 $hSocket=TCPConnect($sIP,$iPort);Connect to Host If @error Then Exit 3 ;Construct Header $sHeader="GET /"&$sURL&" HTTP/1.1"&@CRLF $sHeader&="Host: "&$sDomain&@CRLF $sHeader&="Connection: keep-alive"&@CRLF&@CRLF&@CRLF TCPSend($hSocket,$sHeader);Send Request If @Error Then Exit 4 Do;Recieve Header $sData&=BinaryToString(TCPRecv($hSocket,1,0)) If @Error Then ExitLoop $iRecieved=BinaryLen($sData) If $iRecieved=0 Then ContinueLoop If BinaryMid($sData,1+$iRecieved-4,4)=@CRLF&@CRLF Then ExitLoop EndIf Until False $aHeader=StringSplit($sData,@CRLF,1);Split Header _ArrayDisplay($aHeader) If @Error Then Exit 3 _ArrayColInsert($aHeader,1); Format Header For $iIndex=1 to $aHeader[0][0] $aElement=StringSplit($aHeader[$iIndex][0],": ",3) If @Error Then ContinueLoop $aHeader[$iIndex][0]=$aElement[0] $aHeader[$iIndex][1]=$aElement[1] Next $iHeader=_ArraySearch($aHeader,"Content-Length");Get FileSize If Not @Error Then $iSize=$aHeader[$iHeader][1] $sData="" $iRecieved=0 Do;Get Data $hTimer=TimerInit() $sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1)) $iTime=TimerDiff($hTimer);Get time of recieve (Previously used to calculate iChunk) If BinaryLen($sRecieve)=0 Then ContinueLoop EndIf $sData&=$sRecieve $iRecieve=BinaryLen($sRecieve);Get Current Bytes $iRecieved+=BinaryLen($sRecieve); Populate Total Bytes ToolTip(Round(BinaryLen($sData)/1024/1024,3)&"|"&Round($iRecieved/1024/1024,3)&"|"&StringFormat("%04f",$iTime));Debug If $iChunk>$iRecieve Then;If Request is bigger than recieved, truncate. $iChunk=$iRecieve Else;Get Moar Data! $iChunk*=2 EndIf If $iRecieved>=$iOffloadRecieve+(1024*1024*1) Then; If 1MB Reached, Offload to File and clear WS (Disabled/Pending) ConsoleWrite("Offload:"&$iRecieved&","&$iOffloadRecieve+(1024*1024*1)&@CRLF) $iOffloadRecieve+=$iRecieve+(1024*1024*1) ;$sData="" _WinAPI_EmptyWorkingSet() EndIf If $iSize<>-1 Then If $iRecieved=$iSize Then; Request Satisfied. FileWrite("Test.7z",$sData) ExitLoop EndIf EndIf Until False EndFunc What is what? What is what. Link to comment Share on other sites More sharing options...
jchd Posted September 19, 2015 Share Posted September 19, 2015 (edited) Why you are converting binary (.7z) to a string is a mystery. Don't do that.Also, FIleOpen the output file in binary mode before FileWrite.Your script would be simpler if you used @trancexx HTTP or even IE UDF. Edited September 19, 2015 by jchd This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Why you are converting binary (.7z) to a string is a mystery. Don't do that.Also, FIleOpen the output file in binary mode before FileWrite.Your script would be simpler if you used @trancexx HTTP or even IE UDF.$sRecieve&=TCPRecv($hSocket,$iChunk,1)Return: 0x370x7ABC0xAF271C000x044C4BE01D144...Even though I TCPRecv binary, I end up with a string of binary strings.I'm not intending to write a file with the data, that is for debugging. The file is to be stored in memory temporarily.The file I am downloading is 116MB, and the function recieves 116MB, however the data that is written is 182MB. And that is where my issue comes from.I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect. What is what? What is what. Link to comment Share on other sites More sharing options...
jchd Posted September 19, 2015 Share Posted September 19, 2015 This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 This doesn't "return 0x..." This is what you see when you display it because doing so forces conversion to a string. Write recieved data to a binary file and/or perform whatever you want on the binary variable but don't convert it to a string.If you change $sRecieve&=BinaryToString(TCPRecv($hSocket,$iChunk,1))to$sRecieve&=TCPRecv($hSocket,$iChunk,1)and run that script, you will see the difference in the output file with a hex editor. What is what? What is what. Link to comment Share on other sites More sharing options...
jchd Posted September 19, 2015 Share Posted September 19, 2015 Did you change the script to write the file in binary mode? This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
guinness Posted September 19, 2015 Share Posted September 19, 2015 I know of the simpler UDFs out there, but my goal is write my own functions. With all do respect.Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points! UDF List: _AdapterConnections() • _AlwaysRun() • _AppMon() • _AppMonEx() • _ArrayFilter/_ArrayReduce • _BinaryBin() • _CheckMsgBox() • _CmdLineRaw() • _ContextMenu() • _ConvertLHWebColor()/_ConvertSHWebColor() • _DesktopDimensions() • _DisplayPassword() • _DotNet_Load()/_DotNet_Unload() • _Fibonacci() • _FileCompare() • _FileCompareContents() • _FileNameByHandle() • _FilePrefix/SRE() • _FindInFile() • _GetBackgroundColor()/_SetBackgroundColor() • _GetConrolID() • _GetCtrlClass() • _GetDirectoryFormat() • _GetDriveMediaType() • _GetFilename()/_GetFilenameExt() • _GetHardwareID() • _GetIP() • _GetIP_Country() • _GetOSLanguage() • _GetSavedSource() • _GetStringSize() • _GetSystemPaths() • _GetURLImage() • _GIFImage() • _GoogleWeather() • _GUICtrlCreateGroup() • _GUICtrlListBox_CreateArray() • _GUICtrlListView_CreateArray() • _GUICtrlListView_SaveCSV() • _GUICtrlListView_SaveHTML() • _GUICtrlListView_SaveTxt() • _GUICtrlListView_SaveXML() • _GUICtrlMenu_Recent() • _GUICtrlMenu_SetItemImage() • _GUICtrlTreeView_CreateArray() • _GUIDisable() • _GUIImageList_SetIconFromHandle() • _GUIRegisterMsg() • _GUISetIcon() • _Icon_Clear()/_Icon_Set() • _IdleTime() • _InetGet() • _InetGetGUI() • _InetGetProgress() • _IPDetails() • _IsFileOlder() • _IsGUID() • _IsHex() • _IsPalindrome() • _IsRegKey() • _IsStringRegExp() • _IsSystemDrive() • _IsUPX() • _IsValidType() • _IsWebColor() • _Language() • _Log() • _MicrosoftInternetConnectivity() • _MSDNDataType() • _PathFull/GetRelative/Split() • _PathSplitEx() • _PrintFromArray() • _ProgressSetMarquee() • _ReDim() • _RockPaperScissors()/_RockPaperScissorsLizardSpock() • _ScrollingCredits • _SelfDelete() • _SelfRename() • _SelfUpdate() • _SendTo() • _ShellAll() • _ShellFile() • _ShellFolder() • _SingletonHWID() • _SingletonPID() • _Startup() • _StringCompact() • _StringIsValid() • _StringRegExpMetaCharacters() • _StringReplaceWholeWord() • _StringStripChars() • _Temperature() • _TrialPeriod() • _UKToUSDate()/_USToUKDate() • _WinAPI_Create_CTL_CODE() • _WinAPI_CreateGUID() • _WMIDateStringToDate()/_DateToWMIDateString() • Au3 script parsing • AutoIt Search • AutoIt3 Portable • AutoIt3WrapperToPragma • AutoItWinGetTitle()/AutoItWinSetTitle() • Coding • DirToHTML5 • FileInstallr • FileReadLastChars() • GeoIP database • GUI - Only Close Button • GUI Examples • GUICtrlDeleteImage() • GUICtrlGetBkColor() • GUICtrlGetStyle() • GUIEvents • GUIGetBkColor() • Int_Parse() & Int_TryParse() • IsISBN() • LockFile() • Mapping CtrlIDs • OOP in AutoIt • ParseHeadersToSciTE() • PasswordValid • PasteBin • Posts Per Day • PreExpand • Protect Globals • Queue() • Resource Update • ResourcesEx • SciTE Jump • Settings INI • SHELLHOOK • Shunting-Yard • Signature Creator • Stack() • Stopwatch() • StringAddLF()/StringStripLF() • StringEOLToCRLF() • VSCROLL • WM_COPYDATA • More Examples... Updated: 22/04/2018 Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Yes, and no change. What is what? What is what. Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Study those UDFs then instead of guessing. Honestly, speaking from experience, a good programmer knows when to use the tools around them effectively and not to try and re-invent the wheel. It won't earn you any bonus points!Good point, i will try. What is what? What is what. Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 I apologize for the confusion, I fixed it.;What did not work... $sRecieve=BinaryToString(TCPRecv($hSocket,$iChunk,1)) $sData&=$sRecieve ;What does... $sRecieve=TCPRecv($hSocket,$iChunk,1) $sData&=BinaryToString($sRecieve) What is what? What is what. Link to comment Share on other sites More sharing options...
jchd Posted September 19, 2015 Share Posted September 19, 2015 ??? This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt) Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 (edited) ???I know....but trust me, the function works fine now. Edited September 19, 2015 by Biatu What is what? What is what. Link to comment Share on other sites More sharing options...
Biatu Posted September 19, 2015 Author Share Posted September 19, 2015 Never mind, it was the FileWrite func. Even though when jchd previously advised to FileOpen as binary...it did not work. But now, for some reason it does. What is what? What is what. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now