Sign in to follow this  
Followers 0
ParoXsitiC

TCP and HTTP Requests

3 posts in this topic

Using the following UDF:

Func _HTTPConnect($host, $port = 80)
    Dim $ip = TCPNameToIP ( $host )
    Dim $socket = TCPConnect ( $ip, 80 )

    If ($socket == -1) Then
        SetError(1, @error)
        Return -1
    EndIf
    
    $_HTTPLastSocket = $socket
    SetError(0)
    Return $socket
EndFunc

Func _HTTPClose($socket = -1)
    If $socket == -1 Then
        If $_HTTPLastSocket == -1 Then
            SetError(1)
            Return 0
        EndIf
        $socket = $_HTTPLastSocket
    EndIf
    TCPCloseSocket($socket)
    
    SetError(0)
    Return 1
EndFunc

Func _HTTPRead($socket = -1, $flag = 0)
    If $socket == -1 Then
        If $_HTTPLastSocket == -1 Then
            SetError(1)
            Return
        EndIf
        $socket = $_HTTPLastSocket
    EndIf
    
    Dim $timer = TimerStart()
    Dim $performancetimer = TimerStart()
    Dim $downloadtime = 0
    
    Dim $headers[1][2] ; An Array of the headers found
    Dim $numheaders = 0 ; The number of headers found
    Dim $body = "" ; The body of the message
    Dim $HTTPVersion ; The HTTP version of the server (almost always 1.1)
    Dim $HTTPResponseCode ; The HTTP response code like 200, or 404
    Dim $HTTPResponseReason ; The human-readable response reason, like "OK" or "Not Found"
    Dim $bytesreceived = 0 ; The total number of bytes received
    Dim $data = "" ; The entire raw message gets put in here.
    Dim $chunked = 0 ; Set to 1 if we get the "Transfer-Encoding: chunked" header.
    Dim $chunksize = 0 ; The size of the current chunk we are processing.
    Dim $chunkprocessed = 0 ; The amount of data we have processed on the current chunk.
    Dim $contentlength ; The size of the body, if NOT using chunked transfer mode.
    Dim $part = 0 ; Refers to what part of the data we're currently parsing:
    ; 0 - Nothing parsed, so HTTP response should come next
    ; 1 - Currently parsing headers
    ; 2 - Currently waiting for the next chunk size - this is skipped if the transfer-encoding is not chunked
    ; 3 - Currently waiting for or parsing body data
    ; 4 - Currently parsing footers
    While 1
        Sleep(10)
        Dim $recv = TCPRecv($socket,16)
        If @error <> 0 Then
            ;ConsoleWrite("Server closed connection")
            ;@error appears to be -1 after the server closes the connection. A good way to tell that we're finished, because we always send
            ;the "Connection: close" header to the server.
            ; !!! This is no longer used because we can now tell that we're done by checking the content-length header or properly handling
            ; chunked data.
        EndIf
        
        If $recv <> "" Then
            $bytesreceived = $bytesreceived + StringLen($recv)
            $timer = TimerStart()
            $data &= $recv
            ConsoleWrite("Bytes downloaded: "&$bytesreceived&@CRLF)
        EndIf
        
        Dim $split = StringSplit($data,@CRLF,1)
        $data = ""
        Dim $i
        For $i=1 To $split[0]
            If $i=$split[0] Then
                If $part < 2 OR $chunked = 1 Then
                    ; This is tricky. The last line we've received might be truncated, so we only want to process it under special cases.
                    ; Non chunked data doesn't always send a CRLF at the end so there's no way to tell if this is truly the last line without parsing it.
                    ; However, we don't want to parse it if it's only a partial header or something.
                    ; The solution: We will only process this last line if we're at the body section and the transfer-encoding is NOT chunked.
                    $data = $split[$i]
                    ExitLoop
                EndIf
            EndIf
            
            Dim $newpart = $part
            Switch $part
                Case 0 ; Nothing parsed, so HTTP response should come next
                    If $split[$i] <> "" Then
                        Dim $regex = StringRegExp($split[$i],"^HTTP/([0-9.]+) ([0-9]+) ([a-zA-Z0-9 ]+)$",3)
                        If @extended = 0 Then
                            SetError(5)
                            Return $split[$i]
                        Else
                            $HTTPVersion = $regex[0]
                            $HTTPResponseCode = $regex[1]
                            $HTTPResponseReason = $regex[2]
                            If $HTTPResponseCode <> 100 Then
                                $newpart = 1
                            EndIf
                        EndIf
                    EndIf
                Case 1, 4 ; Currently parsing headers or footers
                    ;If the line is blank, then we're done with headers and the body is next
                    If $split[$i] == "" Then
                        If $part = 1 Then
                            If $chunked Then
                                $newpart = 2
                            Else
                                $newpart = 3
                            EndIf
                        ElseIf $part = 4 Then
                            ; If $part is 4 then we're processing footers, so we're all done now.
                            ExitLoop 2
                        EndIf
                    Else ;The line wasn't blank
                        ;Check to see if the line begins with whitespace. If it does, it's actually
                        ;a continuation of the previous header
                        Dim $regex = StringRegExp($split[$i],"^[ \t]+([^ \t].*)$",3)
                        If @extended = 1 Then
                            If $numheaders == 0 Then
                                SetError(6)
                                Return $split[$i]
                            EndIf
                            $headers[$numheaders-1][1] &= $regex[0]
                        Else;The line didn't start with a space
                            Dim $regex = StringRegExp($split[$i],"^([^ :]+):[ \t]*(.*)$",3)
                            If @extended = 1 Then
                                ;This is a new header, so add it to the array
                                $numheaders = $numheaders + 1
                                ReDim $headers[$numheaders][2]
                                $headers[$numheaders-1][0] = $regex[0]
                                $headers[$numheaders-1][1] = $regex[1]
                                
                                ; There are a couple headers we need to know about. We'll process them here.
                                If $regex[0] = "Transfer-Encoding" AND $regex[1] = "chunked" Then
                                    $chunked = 1
                                ElseIf $regex[0] = "Content-Length" Then
                                    $contentlength = Int($regex[1])
                                EndIf
                            Else
                                SetError(6)
                                Return $split[$i]
                            EndIf
                        EndIf
                    EndIf
                Case 2 ; Awaiting chunk size
                    $regex = StringRegExp($split[$i],"^([0-9a-f]+);?.*$",3)
                    If @extended = 0 Then
                        SetError(8)
                        Return $split[$i]
                    EndIf
                    $chunksize = $regex[0]
                    $chunksize = Dec($chunksize)
                    $chunkprocessed = 0
                    
                    If $chunksize == 0 Then
                        $newpart = 4
                    Else
                        $newpart = 3
                    EndIf
                Case 3 ; Awaiting body data
                    $body &= $split[$i]
                    
                    $chunkprocessed = $chunkprocessed + StringLen($split[$i])
                    
                    If $chunked Then
                        If $chunkprocessed >= $chunksize Then
                            $newpart = 2
                        Else
                            $body &= @CRLF
                            $chunkprocessed = $chunkprocessed + 2; We add 2 for the CRLF we stipped off.
                        EndIf
                    Else
                        If $chunkprocessed >= $contentlength Then
                            ExitLoop 2
                        Else
                            If $i < $split[0] Then
                                ; Only add a CRLF if this is not the last line received.
                                $body &= @CRLF
                                $chunkprocessed = $chunkprocessed + 2; We add 2 for the CRLF we stipped off.
                            EndIf
                        EndIf
                    EndIf
                Case Else
                    ; This should never happen
            EndSwitch
            $part = $newpart
        Next
        
        If $bytesreceived == 0 AND TimerDiff($timer) > $_HTTPRecvTimeout Then
            SetError(3)
            Return 0
        ElseIf $bytesreceived > 0 AND TimerDiff($timer) > $_HTTPRecvTimeout Then
            ConsoleWrite($body)
            SetError(4)
            Return $bytesreceived
        EndIf
    WEnd
    $downloadtime = TimerDiff($performancetimer)
    ConsoleWrite("Performance: Download time: "&$downloadtime&@CRLF)
    
    Switch $flag
        Case 0
            SetError(0)
            Return $body
        Case 1
            Dim $return[5]
            $return[0] = $HTTPResponseCode
            $return[1] = $HTTPResponseReason
            $return[2] = $HTTPVersion
            $return[3] = $headers
            $return[4] = $body
            SetError(0)
            Return $return
        Case Else
            SetError(7)
            Return 0
    EndSwitch
EndFunc

Func _HTTPGetCookies($host, $page, $cookies="", $socket = -1)
    Dim $command
    
    If $socket == -1 Then
        If $_HTTPLastSocket == -1 Then
            SetError(1)
            Return
        EndIf
        $socket = $_HTTPLastSocket
    EndIf
    

    $command = "GET "&$page&" HTTP/1.1"&@CRLF
    $command &= "Host: " &$host&@CRLF
    $command &= "Cookie: "&$cookies&@CRLF&@CRLF
    

    Dim $bytessent = TCPSend($socket, $command)
    
    If $bytessent == 0 Then
        SetExtended(@error)
        SetError(2)
        return 0
    EndIf
    
    SetError(0)
    Return $bytessent
EndFunc



Func _HTTPPostCookies($host, $page, $data = "", $cookies="", $socket = -1)
    Dim $command
    
    If $socket == -1 Then
        If $_HTTPLastSocket == -1 Then
            SetError(1)
            Return
        EndIf
        $socket = $_HTTPLastSocket
    EndIf
    
    Dim $datasize = StringLen($data)
    
    $command = "POST "&$page&" HTTP/1.1"&@CRLF
    $command &= "Host: " &$host&@CRLF
    $command &= "User-Agent: "&$_HTTPUserAgent&@CRLF
    $command &= "Connection: close"&@CRLF
    $command &= "Content-Type: application/x-www-form-urlencoded"&@CRLF
    $command &= "Content-Length: "&$datasize&@CRLF
    $command &= "Cookie: "&$cookies&@CRLF&@CRLF
    $command &= $data&@CRLF
    
    Dim $bytessent = TCPSend($socket, $command)
    
    If $bytessent == 0 Then
        SetExtended(@error)
        SetError(2)
        return 0
    EndIf
    
    SetError(0)
    Return $bytessent
EndFunc

I thought that maybe it just needed a delay between the two requests so as a quick fix I tried just using TCPRevc($Socket, 10) between each request, this does improve the sucess ratio but doe's not work 100% of the time.

Is it possible to send GET and POST HTTP Requests this fast to a server and not wait for a response? Also, I am having problems with the opening and closing of sockets. It seems like in theory I would only have to open the socket once, send all my GETs and POSTs and then close the socket, however the script seems to only want to work when I open and close the socket for each request.

The HTTPRead function in the UDF is very slow. I would ideally want to just send requests and not wait for any response, but if this is not possible then I will just use wininet functions to get and post data instead.

Share this post


Link to post
Share on other sites



Apparently I'll want to pipeline all the GET requests and then do the POST requests. I cant find any examples on how to pipeline HTTP requests though.

Share this post


Link to post
Share on other sites

Update: I've tried using WinHttp.WinHttpRequest.5.1 object but I am unable to set cookies between send() methods. So that won't work.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0