TCP and binary data

[Chance]

How does binary concatenation work in autoit?

I'm trying to transfer dynamically sized binary data through TCP, but certain things work and certain things don't.

My problem is that I'd like that the binary data never touch the hard drive and stay in a variable.

here's an example with comments showing the problems.

$Time = TimerInit ()
While TimerDiff($Time) < 10000
    $Data = TcpRecv ($Client[$ID][0], $INT_BUFFERSIZE)
    If @error Then ExitLoop
    If $Data Then
        $DidItWork = 1
        $File &= $Data
        FileWrite(@ScriptDir & "\test3.zip", $Data); < --- this always works, writes file correctly, probably because FileWrite Fails the second time around
        ;ExitLoop
    EndIf
WEnd
If $DidItWork Then
    MsgBox(16, "", "Success!")
    FileWrite(@ScriptDir & "\test2.zip", $File); < --- this always fails or writes file as hex aka 0x[0-9A-Z-a-z]; Note: $File is always 2x the size of $Data
EndIf

the data is sent to my server script (autoit) using this function~

; taken from mankos server script or an altered version, this works nicely as it walks through the binary string

Func _SendBinary()
    Local $a
    $binary = Binary(FileRead(".\data.zip"))
    While BinaryLen($binary)
         $a = TCPSend($Server, $binary)
         $binary = BinaryMid($binary, $a + 1, BinaryLen($binary) - $a)
     WEnd
EndFunc

[guinness]

Oh goody, I've been working on a TCP script and waiting for someone to ask a question or post an example (the latter being better.)

Anyway, look at this >> it clearly demonstrates how to do this. Once working you may want to add some better error checking.

[Chance]

Thanks guinness, I managed to figure out the problem when I noticed how they were using the binary variables.

initializing them like $Data = Binary('') seems to work, and also initially sending the server script the size of the data being sent makes it easier to handle.

[guinness]

Most excellent DeputyDerp.

Edit: You don't happen to have any code to show how you did it do you? Just in case someone pops by in the future wondering. Thanks.

Edited January 16, 2013 by guinness

[Chance]

Of course, my bad.

TcpSend ($Client[$ID][0], "begin"); just send a command to tell the client to send us something

                While Not $Size
                    $Size = Int(TcpRecv ($Client[$ID][0], $INT_BUFFERSIZE)); the client will initially send the size of what will be sent in plain text
                WEnd

                Sleep(100); give the client time to do things

                $Time = TimerInit ()
                $File = Binary(''); initialize the file variable~

                While (TimerDiff($Time) < 60000) Or ($Size > BinaryLen($File)); once $File is the same size or bigger, escape the loop
                    $Data = TcpRecv ($Client[$ID][0], $INT_BUFFERSIZE)
                    If @error Then
                        $DidItWork = 0
                        ExitLoop
                    EndIf
                    If $Data Then
                        $DidItWork = 1
                        $File &= $Data; I've tested this with files up to 7MB and it works, files upwards of 20+ MB seems to still cause this loop to write the file twice to the buffer, will look into that later
                    EndIf
                WEnd
                If $DidItWork then _Function_That_Does_Something($File)

The function on the client side that sends the binary has stayed the same like in my first post.

It was always working but it was the reciever in my server that had the problem of either fetching 1/10 of the file or copying it twice to the buffer.

It's actually still happening with files bigger than 20 MB, I tried it with a zip files about 22MB and ended up with a 44MB buffer.

Writing it to a file and checking what was wrong, I see a lot of binary mixed in with autoit hex strings "0x[0-9A-Z-a-z]".

It seems that the receiver randomly receives some AutoIt binary and can't convert it to actual binary.

[guinness]

As discussed in the thread posted today. Add a Sleep(20) in the loop and/or send it in smaller bytes.

[kylomas]

DeputyDerp,

I have not played with the TCP stuff yet, however, this

It seems that the receiver randomly receives some AutoIt binary and can't convert it to actual binary.

is rather hard to beleive.

Would this not work?

$Data = TcpRecv ($Client[$ID][0], $INT_BUFFERSIZE,1) ; third parm forces binary format for received data

kylomas

edit: additional data

In your TCPRecv loop you are not checking for what kind of error might occur. Is this not neccessary?

Edited January 17, 2013 by kylomas

[kylomas]

@guinness - Have you seen this?

kylomas

edit: Nevermind, I'm sure that you've seen all the TCP examples

Edited January 17, 2013 by kylomas

[Chance]

I thought I had figured this out, I was transfering 100mb files successfully, then I added more error checking and it seems that messageboxes that are behind areas that should not be called are being executed while randome parts of the function are being called.

for example

;///

;tcpstuff

If @Error then

MsgBox(16,"error", "error")

return

endif

;///

If $DidItWork then

;create a GUI

endif

Oddly, when downloading, the error messagebox gets executed and so does the gui creation code. Sometimes it just crashes AutoIt.

I had to switch to another PC because I kicked my computer and again, oddly enough it didnt fix the problem and all I got was a harddrive failure or something and it says it can't detect the hard drive anymore (something about boot order failure), luckily I usually save all my stuff to a network drive.

I'll try and get back on this when I'm less mad.

[guinness]

@guinness - Have you seen this?

kylomas

edit: Nevermind, I'm sure that you've seen all the TCP examples

In this instance I have.

[guinness]

These work for me when transmitting an .au3 file.

Server:

expandcollapse popup

#include <FileConstants.au3>

#region Server
TCPStartup()
OnAutoItExitRegister('Cleanup')

Server()

Func Server()
    Local Const $TCP_RECEIVELEN = 4096
    Local $iListen = TCPListen('127.0.0.1', 8075)
    Local $iAccept = 0, $sTCPRecv = ''

    While 1
        $iAccept = TCPAccept($iListen)
        If $iAccept >= 0 Then
            While 1
                $sTCPRecv = TCPRecv($iAccept, $TCP_RECEIVELEN)
                If $sTCPRecv <> '' Then
                    If FileReceive($sTCPRecv, @ScriptDir & '\RecFile\', $iAccept) Then
                        ExitLoop 2 ; Exit the first loop.
                    EndIf
                    ConsoleWrite('Date from client: ' & $sTCPRecv & @CRLF)
                EndIf
            WEnd
        EndIf
    WEnd
EndFunc   ;==>Server

Func Cleanup()
    TCPShutdown()
EndFunc   ;==>Cleanup
#endregion Server

; By guinness, based on: http://www.autoitscript.com/forum/topic/46341-the-first-file-over-tcp-function/#entry511126
Func FileReceive($sSentData, $sFilePath, $iSocket)
    Local Const $TCP_RECEIVELEN = 4096

    Local $aArray = StringRegExp($sSentData, '(?s)^(\[\FILESEND\])\|([^|]+)\|(\d+)', 3)
    If Not (UBound($aArray) = 3) Then
        Return SetError(1, 0, False)
    EndIf
    If Not ($aArray[0] == '[FILESEND]') Then
        Return SetError(1, 0, False)
    EndIf
    If TCPSend($iSocket, $sSentData) = 0 Then
        Return SetError(2, 0, False)
    EndIf

    $sSentData = ''
    Local $iStringLen = Int($aArray[2]), $sTCPRecv = ''
    $sFilePath = StringRegExpReplace($sFilePath, '\\+$', '') & '\' & $aArray[1]

    While 1
        $sTCPRecv = TCPRecv($iSocket, $TCP_RECEIVELEN)
        If @error Then
            ExitLoop
        EndIf
        $sSentData &= $sTCPRecv
        If StringLen($sSentData) = $iStringLen Then
            ExitLoop
        EndIf
        Sleep(10)
    WEnd

    If Not (StringLen($sSentData) = $iStringLen) Then
        Return SetError(3, 0, False)
    EndIf

    Local $hFileOpen = FileOpen($sFilePath, $FC_CREATEPATH + $FO_OVERWRITE + $FO_BINARY)
    If $hFileOpen = -1 Then
        Return SetError(4, 0, False)
    EndIf
    $sSentData = StringRegExpReplace($sSentData, '\[FILE\](0x[[:xdigit:]]+)\[/FILE\]', '\1')
    FileWrite($hFileOpen, Binary($sSentData))
    FileClose($hFileOpen)
    Return True
EndFunc   ;==>FileReceive

Client:

expandcollapse popup

#include <Constants.au3>

#region Client
TCPStartup()
OnAutoItExitRegister('Cleanup')

Client()

Func Client()
    Local $iCount = 0
    Local $iConnect = TCPConnect('127.0.0.1', 8075)

    Local $sFilePath = FileOpenDialog('Select a file.', @ScriptDir, 'All (*.*)')
    If @error = 0 Then
        If FileSend($sFilePath, $iConnect) Then
            MsgBox($MB_SYSTEMMODAL, '', 'File was successfully sent.')
        Else
            MsgBox($MB_SYSTEMMODAL, '', 'An error occurred when sending the file.')
        EndIf
    EndIf
    Return True

;~  If @error = 0 Then
;~      While 1
;~          If TCPSend($iConnect, 'Send__' & $iCount) = 0 Then ExitLoop
;~          $iCount += 1
;~          Sleep(100)
;~      WEnd
;~  EndIf
EndFunc   ;==>Client

Func Cleanup()
    TCPShutdown()
EndFunc   ;==>Cleanup
#endregion Client

; By guinness, based on: http://www.autoitscript.com/forum/topic/46341-the-first-file-over-tcp-function/#entry511126
Func FileSend($sFilePath, $iSocket)
    Local Const $TCP_RECEIVELEN = 4096

    Local $hFileOpen = FileOpen($sFilePath, $FO_READ + $FO_BINARY)
    If $hFileOpen = -1 Then
        Return SetError(1, 0, False)
    EndIf
    Local $sFileRead = String(FileRead($hFileOpen))
    FileClose($hFileOpen)

    $sFileRead = '[FILE]' & $sFileRead & '[/FILE]'
    Local $sSendData = '[FILESEND]|' & StringTrimLeft($sFilePath, StringInStr($sFilePath, '\', 2, -1)) & '|' & StringLen($sFileRead)
    If TCPSend($iSocket, $sSendData) Then
        Local $hTimer = TimerInit(), $sTCPRecv = ''
        Do
            Sleep(10)
            $sTCPRecv = TCPRecv($iSocket, $TCP_RECEIVELEN)
        Until $sTCPRecv == $sSendData Or TimerDiff($hTimer) >= 500
        If Not ($sTCPRecv == $sSendData) Then
            Return SetError(1, 0, False)
        EndIf
    EndIf

    Local $iReturn = TCPSend($iSocket, StringLeft($sFileRead, $TCP_RECEIVELEN))
    $sFileRead = StringTrimLeft($sFileRead, $iReturn)
    While StringLen($sFileRead) > $TCP_RECEIVELEN
        $iReturn = TCPSend($iSocket, StringLeft($sFileRead, $iReturn))
        If @error Then
            ExitLoop
        EndIf
        $sFileRead = StringTrimLeft($sFileRead, $iReturn)
        Sleep(10)
    WEnd
    If StringLen($sFileRead) Then
        TCPSend($iSocket, $sFileRead)
    EndIf
    Return True
EndFunc   ;==>FileSend

Notice how I posted runnable code and not just snippets, I suggest you do the same from here on in.

Edited January 19, 2013 by guinness

[Chance]

Thanks guiness.

Well, I tried playing around with it until I was able to get a function that would be more stable when downloading small images and files all the way to 250mb files.

This is what I ended up with and I don't wana touch it anymore since it's working perfectly for everything I've tried already.

expandcollapse popup

Func _RecieveData($ID)

    Local $Time = TimerInit()
    Local $File = Binary('')
    Local $Data
    Local $Size = 0
    Local $Downloaded = 0
    Local $DidItWork = 0
    Local $Strikes = 0

    While Not $Size
        $Size = TcpRecv($Client[$ID][0], $INT_BUFFERSIZE, 0)
        If TimerDiff($Time) > 130000 Then
            _Disconnect($ID); there's something wrong with this person, get rid of them
            Return SetError(1, 0, 0)
        EndIf
    WEnd

    $Size = Int($Size)
    If Not $Size Then Return SetError(2, 0, 0)

    For $i = 0 To 100e+10; loop for like 50 years or something
        $Data = TcpRecv ($Client[$ID][0], $INT_BUFFERSIZE, 1)
        ;If @error Then Return SetError(3, 0 , 0); sometimes the client disconnects, so avoid checking error because even though error is set, the client still ends up sending correct information
        If $Data Then
            $Downloaded += BinaryLen($Data)
            $DidItWork = 1
            $File &= $Data
            _GUICtrlStatusBar_SetText($hStatusBar, "Downloading: " & $Downloaded & " of " & $Size & " Bytes", 0)
        Else
            If $DidItWork Then $Time = TimerInit()
            $DidItWork = 0
            $Strikes += 1
        EndIf
        If (Int(($Strikes - 1) / 15) = ($Strikes - 1) / 15) And Not $DidItWork Then; every 15 times, check if  something is wrong
            If TimerDiff($Time) > 60000 Then Return SetError(4, 0, 0)
        EndIf
        If ($Size <= $Downloaded) Then ExitLoop
    Next

    Return SetError(0, 0, $File)
EndFunc

[guinness]

If you want me to test that function, I can't. See previous post.

[trancexx]

Binary concatenation using operator & is not docummented feature. If your code works now and wouldn't tomorrow, you roughly would know why.

[guinness]

My fault, don't know how I let that slip by me. Thanks trancexx.

[guinness]

I updated my example in post #11.

It sent a 9MB file just fine, though you will need to increase $TCP_RECEIVELEN if you want it to be quicker.

Edited January 19, 2013 by guinness