Kealper

Fast multi-client TCP server

40 posts in this topic

#21 ·  Posted (edited)

Yes, it probably could if it was implemented correctly, I have used this for quite a few projects myself.

EDIT: Spelling.

Edited by Kealper

Share this post


Link to post
Share on other sites



#22 ·  Posted

Yes, it probably could if it was implemented correctly, I have used this for quotes few projects myself.

Cool thanks for the quick reply I'll test it out and report back if I come up with any problems. :bye:


A true renaissance man

Share this post


Link to post
Share on other sites

#23 ·  Posted

Great work, i was looking for exactly something like this.

Your comments made it a billion times easier to understand whats needs to be done in a script of this nature!

Share this post


Link to post
Share on other sites

#24 ·  Posted (edited)

Great work, i was looking for exactly something like this.

Your comments made it a billion times easier to understand whats needs to be done in a script of this nature!

Thanks, glad to hear it's as useful to other as it is to me!

PS: This bit of code shown below can be ignored/removed, it was some debug code that sneaked in to one of the changes I had made, and it seems I forgot to remove it before posting the updated version! >_<

All this did is just show the total number of clients currently connected every second in the console, nothing direly important or code-breaking if removed.

AdlibRegister("asdf", 1000)
Func asdf()
ConsoleWrite($Clients[0][0] & @CRLF)
EndFunc
Edited by Kealper

Share this post


Link to post
Share on other sites

#25 ·  Posted

Thanks, glad to hear it's as useful to other as it is to me!

PS: This bit of code shown below can be ignored/removed, it was some debug code that sneaked in to one of the changes I had made, and it seems I forgot to remove it before posting the updated version! >_<

All this did is just show the total number of clients currently connected every second in the console, nothing direly important or code-breaking if removed.

AdlibRegister("asdf", 1000)
Func asdf()
ConsoleWrite($Clients[0][0] & @CRLF)
EndFunc

Figured that was the case.. have been using it to debug also :P

Share this post


Link to post
Share on other sites

#26 ·  Posted

I've just made some more changes to it, this time to increase efficiency when large (over 1,000) active clients are connected and one or more clients disconnects. Turns out in this case that it is up to 500x faster, on my computer at least, to build a copy of the original array without the sub-elements you don't want and then overwrite the old array with the new copy than it is to use ReDim to re-size the one array. I would have figured the latter would have been much more efficient, but it seems there are always curve-balls in the game of programming.

I've changed the original Sleep(1) call in the main loop to a custom sleep function (credit goes to monoceres for the basics behind that function) which takes delays in microseconds instead of milliseconds, because it turns out the lower-level function AutoIt's Sleep wraps appears to only be able to go down to ~10ms, so there was a decent gain in response time just by switching that over. I've got it set to 5000us (5ms) sleep delay in the example, but you can safely go down to 1000us (1ms) without much of a resource-usage hike, from my testing that is; Anything lower than 1000us doesn't seem to give much more of a performance increase, so diminishing returns, etc...

I also corrected some more spelling (again... :( ), tweaked the little example echo server code in that base to better demonstrate my choice of why that line dealing with pulling packets from the buffer that many have said was incorrect is actually exactly how I intended it to function. :)


Share this post


Link to post
Share on other sites

#27 ·  Posted

Kealper: Thank you for the Idea. I'm refactored your code a bit:

#cs ----------------------------------------------------------------------------
Fast TCP Server
Author: Petr Krstev
Based on "Fast multi-client TCP server" by Ken Piper
#ce ----------------------------------------------------------------------------
TCPStartup()
Opt("TCPTimeout", 0)

#region ;Safe-to-edit things are below
Global $BindIP = "0.0.0.0" ;Listen on all addresses
Global $BindPort = 8080 ;Listen on port 8080
Global $Timeout = 15000 ;Max idle time is 15 seconds before calling a connection "dead"
Global $PacketSize = 2048 ;Max packet size per-check is 2KB
Global $MaxClients = 50 ;Max simultaneous clients is 50
#endregion ;Stuff you shouldn't touch is below

Global $Listen
Global $Socket[$MaxClients] ; Rows of data
Global $RemoteIP[$MaxClients] ; Rows of data
Global $TimeStamp[$MaxClients] ; Rows of data
Global $Buffer[$MaxClients] ; Rows of data
Global $Stack[$MaxClients+2] ; Stack to hold numbers of the free rows of data
Global $SP = 0
Global $Ws2_32 = DllOpen("Ws2_32.dll") ;Open Ws2_32.dll, it might get used a lot
Global $NTDLL = DllOpen("ntdll.dll") ;Open ntdll.dll, it WILL get used a lot
Global $CleanupTimer = TimerInit() ;This is used to time when things should be cleaned up
Global $NeedExit = False

PreloadStack()

OnAutoItExitRegister("Close") ;Register this function to be called if the server needs to exit

$Listen = TCPListen($BindIP, $BindPort) ;Start listening on the given IP/port
If @error Then Exit 1 ;Exit with return code 1 if something was already bound to that IP and port

While 1
if $NeedExit Then ExitLoop ;Exit demanded from client
USleep(5000, $NTDLL) ;This is needed because TCPTimeout is disabled. Without this it will run one core at ~100%.
;The USleep function takes MICROseconds, not milliseconds, so 1000 = 1ms delay.
;When working with this granularity, you have to take in to account the time it takes to complete USleep().
;1000us (1ms) is about as fast as this should be set. If you need more performance, set this from 5000 to 1000,
;but doing so will make it consume a bit more CPU time to get that extra bit of performance.
Check() ;Check recv buffers...
DoIt() ; ...and do things
If TimerDiff($CleanupTimer) > 1000 Then ;If it has been more than 1000ms since Cleanup() was last called, call it now
$CleanupTimer = TimerInit() ;Reset $CleanupTimer, so it is ready to be called again
Cleanup() ;Clean up the dead connections
EndIf
Local $iSock = TCPAccept($Listen) ;See if anything wants to connect
If $iSock = -1 Then ContinueLoop ;If nothing wants to connect, restart at the top of the loop
If $SP = 0 Then ;If $SP = 0 then the max connection limit has been reached
TCPCloseSocket($iSock) ;It has been reached, close the new connection and continue back at the top of the loop
ContinueLoop
EndIf

Local $FreeSock = Pop() ;Get the next free row in data
$Socket[$FreeSock] = $iSock ;Set the socket ID of the connection
$RemoteIP[$FreeSock] = SocketToIP($iSock, $Ws2_32) ;Set the IP Address the connection is from
$TimeStamp[$FreeSock] = TimerInit() ;Set the timestamp for the last known activity timer
$Buffer[$FreeSock] = "" ;Blank the recv buffer
WEnd

Func Check() ;for incoming data
If $SP >= $MaxClients Then Return ;If there are no clients connected, stop the function right now
For $i = 0 To $MaxClients-1 ;Loop through all rows
If $Socket[$i] > 0 Then ;valid socket in the row
Local $sRecv = TCPRecv($Socket[$i], $PacketSize) ;Read $PacketSize bytes from the current client's TCP buffer
If $sRecv <> "" Then ;If there was data sent from the client
$Buffer[$i] &= $sRecv ;add it to the buffer
$TimeStamp[$i] = TimerInit() ;update the activity timer
EndIf
EndIf
Next
EndFunc

Func DoIt() ;Processing incoming data
For $i = 0 To $MaxClients-1 ;Loop through all rows
If $Buffer[$i] <> "" Then ;Data buffer in the row is not empty
#region ;Example data processing stuff here. This is handling for a simple "echo" server with line handling
Local $sLine = StringSplit($Buffer[$i], @CRLF, 1) ;Split up the data in to an array,...
For $j = 1 To $sLine[0]-1 ; ...so it is easy to loop through each line
If $sLine[$j] = "terminate" Then $NeedExit = True ;Client want terminate the server itself (not just connection)
TCPSend($Socket[$i], "Echoing line: '" & $sLine[$j] & "'" & @CRLF) ;Echo back the line the client sent
Next
$Buffer[$i] = $sLine[$sLine[0]] ;Put last incomplete line back in the buffer
#endregion ;Example
EndIf
Next
EndFunc

Func Cleanup() ;Clean up any disconnected clients to regain resources
If $SP >= $MaxClients Then Return ;If no clients are connected then return
For $i = 0 To $MaxClients-1 ;Loop through all rows
If $Socket[$i] > 0 Then ;valid socket in the row
$Buffer[$i] &= TCPRecv($Socket[$i], $PacketSize) ;Dump any data not-yet-seen in to their recv buffer
If @error Or TimerDiff($TimeStamp[$i]) > $Timeout Then ;Check to see if the connection has been inactive for a while or if there was an error
TCPCloseSocket($Socket[$i]) ;If yes, close the connection
$Socket[$i] = -1 ;Set the socket ID to an invalid socket
Push($i) ;Free this row
EndIf
EndIf
Next
EndFunc

Func Close()
DllClose($Ws2_32) ;Close the open handle to Ws2_32.dll
DllClose($NTDLL) ;Close the open handle to ntdll.dll
For $i = 0 To $MaxClients-1 ;Loop through all rows
If $Socket[$i] > 0 Then TCPCloseSocket($Socket[$i]) ;Force the client's connection closed
Next
TCPCloseSocket($Listen) ;Force the server's binded connection closed
TCPShutdown() ;Shut down networking stuff
; MsgBox(0,"","!",1)
EndFunc

Func SocketToIP($iSock, $hDLL = "Ws2_32.dll") ;A rewrite of that _SocketToIP function that has been floating around for ages
Local $structName = DllStructCreate("short;ushort;uint;char[8]")
Local $sRet = DllCall($hDLL, "int", "getpeername", "int", $iSock, "ptr", DllStructGetPtr($structName), "int*", DllStructGetSize($structName))
If Not @error Then
$sRet = DllCall($hDLL, "str", "inet_ntoa", "int", DllStructGetData($structName, 3))
If Not @error Then Return $sRet[0]
EndIf
Return "0.0.0.0" ;Something went wrong, return an invalid IP
EndFunc

Func USleep($iUsec, $hDLL = "ntdll.dll") ;A rewrite of the _HighPrecisionSleep function made by monoceres (Thanks!)
Local $hStruct = DllStructCreate("int64")
DllStructSetData($hStruct, 1, -1 * ($iUsec * 10))
DllCall($hDLL, "dword", "ZwDelayExecution", "int", 0, "ptr", DllStructGetPtr($hStruct))
EndFunc

Func PreloadStack()
Local $i
For $i = 0 to $MaxClients-1
Push($i)
Next
EndFunc

Func Push($Value)
if $SP < $MaxClients Then
$Stack[$SP] = $Value
$SP += 1
Return 1
Else
Return 0
EndIf
EndFunc

Func Pop()
if $SP > 0 Then
$SP -= 1
Return $Stack[$SP]
Else
Return -1
EndIf
EndFunc

My changes:

A/ using 4 1D flat arrays instead of 2D array. Each 1D array represents one column in DB/table holds same type of data. Same Index = same connection.

B/ using a "Stack" data structure for information about unused rows in the DB/table.

C/ dividing the Func "Check" into 2 Funcs: "Check" and "DoIt". One for Checking up for a new data, second for a processing new data.

D/ The Func "DoIt" have another type of line parsing. More simple with the possibily to invoke server termination from client.

E/ The Func "CleanUp" stay for socket disconnection and Timeout detection and lost the massive changes on the 2D array. Just push the free socket index on the Stack.

F/ The Func "Close" - added closing of listen socket too.

G/ There is 3 new Funcs: Push, Pop, PreloadStack. Push and Pop classical Stack working (LiFo data structure). PreloadStack pushes all the free indexes in the stack.

Krst'a

Share this post


Link to post
Share on other sites

#28 ·  Posted (edited)

Test: create 1000 connections, then disconnect all connections.

Script on the first page: 90 seconds

Krsta's script: 94 seconds

my script: 16 seconds (14 sec without creating/deleting list view items.)

This is my implementation:

This is the code I used to test the time: (i just changed the ports on your scripts to 1337, and max connections to 1000)

TCPStartup()

Global Const $number_of_connections = 1000 ; Total number of connections to perform
Global Const $ip = @IPAddress1 ; IP address
Global Const $port = 1337 ; Port
Global $connection_array[$number_of_connections + 1]
$errors = 0
$timer = TimerInit()

For $i = 1 To $number_of_connections
$connection_array[$i] = TCPConnect($ip, $port)
If @error Then
$errors += 1
EndIf
Next
MsgBox(0, "", $errors & " errors" & @CRLF & $number_of_connections & " connections in " & Round(TimerDIff($timer) / 1000) & " seconds.")
$timer = TimerInit()

For $i = 1 To $number_of_connections
TCPCloseSocket($connection_array[$i])
If @error Then
$errors += 1
EndIf
Next
MsgBox(0, "", $errors & " errors" & @CRLF & $number_of_connections & " disconnects in " & Round(TimerDIff($timer) / 1000) & " seconds.")
$timer = TimerInit()

TCPShutdown()
Edited by caleb41610

Share this post


Link to post
Share on other sites

#29 ·  Posted

caleb41610: Thank you for pointing out performance-related problem. It is always important.

Tests that I did proved similar results. After 6 hours of unsuccessful experiments, I was very desperate.

Finally I found why the results are so different. All that magic is in command "TCPListen" where we (Me and Kealper too) use default value for the third parameter.

I get similar results to yours (without list items stuf), when I change my script (and Kealper too) like this:

$Listen = TCPListen($BindIP, $BindPort, $MaxClients) ;Start listening on the given IP/port

Finally, I made some additional changes to my script (see attachment) and now the result time is about 5s and there is still room to do further optimizations.

Thank you once more and Good luck

Petr

Test: create 1000 connections, then disconnect all connections.

Script on the first page: 90 seconds

Krsta's script: 94 seconds

my script: 16 seconds (14 sec without creating/deleting list view items.)

This is my implementation:

This is the code I used to test the time: (i just changed the ports on your scripts to 1337, and max connections to 1000)

ST.au3

Share this post


Link to post
Share on other sites

#31 ·  Posted (edited)

Hmm.. Seems one of the forum updates or something cleared the followers list, so I wasn't getting notifications of people posting here or I would have responded sooner...

I didn't even catch that I didn't have the third parameter in this on the post or my local copy of that code... I have it in any of the stuff I do, it must have just slipped my mind to add it to the base while I was working on it, sorry about that!

Using caleb41610's test script, on the code in the original post (after correcting that simple mistake!) will do 1000 connections in 10 seconds, where as Krsta's final code will do 1000 connections in 5 seconds... The reason for this is how accepting connections is handled. Mine makes processing client data a priority by always checking if any clients have sent something before checking the new connection queue and getting new clients in, where as Krsta's makes accepting new clients a priority, by only checking current client data when no new connections are queued. If I change mine around a bit and make it so it only calls Check() when $iSock = -1, I see similar performance, down to 1000 clients in 5 seconds (mine has rounded up to 6 seconds a few times, so its average is slightly slower than Krsta's). If I had to guess, mine is still slower because it uses a dynamic array size for the main client array instead of Krsta's fixed array size, but for the things I do with this code the slight performance hit it incurs is worth the benefit of a flexible max client limit. So it seems mine is still being bit in the butt by overhead in ReDim, and since Krsta isn't using ReDim, no butt-biting goes on there. This has shown me what a difference it can make to switch to fixed-size for the clients array if I ever have a need for a client that can accept connections more consistently though.

Also I chose to do the dynamically re-sized 2d array for simplicity's sake, and for more flexibility, as max clients can be adjusted on-the-fly without requiring a restart (by just changing $MaxClients), and it would be much less work while developing off of this (as it was intended) to change a 2d array than it would be to create another 1d array, in my personal opinion.

(Updated first post with the quick-fix I overlooked before)

EDIT: I should also point out another down-side I hadn't thought of while initially writing this... With a fixed-size array, it tears through CPU cycles walking through the entire array each time, even if there isn't many clients connected. So if you've got it set to 10,000 total connections, it will have to walk through those 10,000 elements each time.

With a dynamically-sized array, the number of elements will always only be the number of active clients, so if the limit is 10,000 but there are only 10 clients connected, the array will only be around 10 elements large(11 if you use one as an index). Just thought I'd throw that out there too for a bit more understanding of why I made things the way I did.

Edited by Kealper

Share this post


Link to post
Share on other sites

#32 ·  Posted

ahoy

i have taken the liberty to transcribe a part of your code into a usable function

acts partly like adlibregister but does only check and run on call (so it does not interrupt current processes)

original code:

If TimerDiff($CleanupTimer) > 1000 Then ;If it has been more than 1000ms since Cleanup() was last called, call it now
        $CleanupTimer = TimerInit() ;Reset $CleanupTimer, so it is ready to be called again
        Cleanup() ;Clean up the dead connections
    EndIf

my function:

Func _starttimer($sttimr = 'Cleanup', $sttimrdiff = 1000)
    $sttimrn = $sttimr & 'Timer'
    If IsDeclared($sttimrn) = 0 Then Assign($sttimrn, _Timer_Init(), 2)
    If TimerDiff(Eval($sttimrn)) > $sttimrdiff Then
        Assign($sttimrn, TimerInit())
        Return Execute($sttimr & '()')
    EndIf
EndFunc

you can call it like this in your loop and it will start the given function after X msec

_starttimer('Cleanup', 1000)

best regards,

stefan

Share this post


Link to post
Share on other sites

#33 ·  Posted (edited)

Since I'm using this tcp server script..
Here's some good additional, A packet dynamic encryption :D

Func PacketEnc($EncData,$EncKey)
Local $EncOut1 = Random(111111,999999,1)
Local $EncOut2 = _Crypt_EncryptData($EncOut1&$EncData, $EncKey, $CALG_AES_256)
Return BinaryToString($EncOut2)
EndFunc

Func PacketDec($DecData,$DecKey)
Local $DecOut1 = _Crypt_DecryptData($DecData, $DecKey, $CALG_AES_256)
Local $DecOut2 = BinaryToString($DecOut1)
Return StringTrimLeft($DecOut2, 6)
EndFunc

-edit: removed deprecated _StringEncrypt()

Edited by Jimtags

Share this post


Link to post
Share on other sites

#34 ·  Posted

_StringEncrypt() is deprecated in the latest version of AutoIt.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#35 ·  Posted

_StringEncrypt() is deprecated in the latest version of AutoIt.

 

But still usable.. anyway that's just a example for how to make the packet not static.. It can help someone maybe :D

Share this post


Link to post
Share on other sites

#36 ·  Posted

But still usable.. anyway that's just a example for how to make the packet not static.. It can help someone maybe :D

I should have removed it when it was deprecated, but I was being too nice for those who needed time to upgrade their scripts (don't I regret that decision now). You're welcome to use outdated code in your own scripts, just don't let you bad habits leak onto the Forums in the future please.

To all,

I provided an alternative StringEncrypt() in _Crypt_EncryptData(), though it's not backwards compatible with _StringEncrypt().

1 person likes this

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#37 ·  Posted (edited)

To add horror to what guinness said, old _StringEncrypt() is less reliable and robust than ROT13.

Edit: made clear I was talking about the legacy version.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#38 ·  Posted

To add horror to what guinness said, StringEncrypt() is less reliable and robust than ROT13.

I guess you meant the old version? Yeah I second that, even ROT5 is more secure than that old function.

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#39 ·  Posted

Exactly. Post edited for clarity.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#40 ·  Posted (edited)

Something is broken with newest Autoit version and Win7 64bit.

Running the server script with no modifications, it allows telnet connections but immediately kills the connection when you type anything (less than 1 second).

Disabling the "cleanup" function causes it to reply the text properly.

FOUND THE PROBLEM:
The newest version of AutoIt changes the behavior of TCPRecv:

'?do=embed' frameborder='0' data-embedContent>>

You need to change line 90 from

 
If @error Or TimerDiff($Clients[$i][2]) > $Timeout Then ;Check to see if the connection has been inactive for a while or if there was an error
 
To:
 
 
If (@error And @error <> -1 ) Or TimerDiff($Clients[$i][2]) > $Timeout Then ;Check to see if the connection has been inactive for a while or if there was an error
 
@error = -1 is not actually an error... it just means there is no data received at the moment.  
 
 
Also, another problem I found is that if there is anything in the recv buffer, it will never timeout.  So in your telnet example if you type something but dont push enter, it will never timeout.  This is because when there is anything in the recv buffer, it keeps resetting the timeout timer.  This means a client could be connected forever when they really shouldn't be.  Perhaps instead of just checking to see if the buffer is empty, see if it has changed as well.
 
 
 
 
Edited by JFC14

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • toasterking
      Parsing binary data frames from an asynchronous source
      By toasterking
      I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.
       
      #cs Author: ToasterKing This is an example of a way to parse streaming binary data that follows a strict format with a header and footer. In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD. The _BinaryParse() function accumulates incoming data in a buffer. Once a footer is found, it searches backward for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle, then moves on to looking for the next frame. #ce ; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable. Local $fSomeData $fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame. _BinaryParse($fSomeData) ; Call the function with the received data. It isn't a complete frame, so it is just stored in the buffer until more data is received. $fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame. _BinaryParse($fSomeData) ; The function should be able to parse both frames now. Func _BinaryParse($fNewData) Local Static $fBinaryReceived = Binary("") ; Buffer for received data ConsoleWrite("Hey, the function is called!" & @CRLF) ; Add new data to the buffer. ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt. It must be converted to strings first. ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data. To be converted back to binary properly, only one instance ; of "0x" must exist at the beginning of the string. $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x","")) ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF) Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data If $iLength > 0 Then Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on Local $fByte1,$fByte2 For $i = 1 To $iLength If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF) If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that ConsoleWrite("Header found before the footer!" & @CRLF) $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer) $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer) ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes. Normally, you'd do something more useful with it here. EndIf $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp EndIf Next EndIf EndFunc  
    • topten
      Parental Filter
      By topten
      Hi I have an idea of creating a parental filter (freeware) , which will be protecting children from navigating to adult websites.  I want to ask, what is your opinion, how to do it best way?
      How to catch the traffic from the very beginning?
      For example if a user is starting IE or Firefox, or if he has some alternate BLABLA-browser - how can I catch the url before actual navigation to website, so that my script could first check if the website is ok, and then if it decides that the website is not ok, just block connection to this IP.
       
      Looking forward to your opinions, Great thanx in advance!
    • Enforcer
      TCP port forwarding
      By Enforcer
      Hello everyone. I'm wondering if its possible:
      I want to try create a simple autoit script that will
      1) first he must connect to some IP in internet port 80 
      2) then he must listen for incoming connections from other IP'S  ports 8000 - 8200 [in total over 200 connections]
      3) that script must send all recieved data from that [IP in internet port 80 ] to [other IP'S  ports 8000 - 8200]
      is that very hard to do ? Any suggestions or maybe some similar examples that can help me to do this ?
      all those 200 connections must receive data at the same time at once is it possible to do so ?
       
    • DrAhmed
      Unstable TCP socket connection server keeps disconnecting and connecting all the time
      By DrAhmed
      I am working on a code of  Client-Server Connection for a while (Like Teamviewer) , the problem is that the server keeps disconnecting and connecting all the time , In the client i wrote a code to notify me each time server is online , I am getting around 10 notifications each 1 minute from same server :
      This is the server code :
      TCPStartup() Local $ConnectedSocket Local $My_IP = "127.0.0.1" Local $My_PORT = '5000' Local $UserID = "MyID" ; Connect Connect($My_IP, $My_PORT) While 1 If TCPConnect($My_IP, $My_PORT) <> -1 Then $recv = TCPRecv($ConnectedSocket, 2048) $RecvSpl = StringSplit($recv, "|") Switch $RecvSpl[1] Case "order1" ; Do something MsgBox(64,"Success","You are now connected to the client") AddLogToClinet("My server is Online", $UserID, "Green") EndSwitch Else $Connected = False Connect($IP, $My_PORT) EndIf Sleep(100) WEnd ; Connect To the Client Func Connect($IP, $Port) Local $iConnectAttempts = 0 TCPStartup() Do $ConnectedSocket = TCPConnect($IP, $Port) Sleep(1000) Until ($ConnectedSocket > 0 or $iConnectAttempts > 10) Return SetError(@error, 0, $ConnectedSocket) EndFunc ;==>Connect ; Send To Client Func SockSend($Cmd, $Text) TCPSend($ConnectedSocket, $Cmd & $SockSpl & $Text & $PocketSpl, 4) EndFunc ;==>SockSend ; Add Log To Server Func AddLogToClinet($Data, $UserID, $Color) SockSend("AddLog", $Data & $SockSpl & $UserID & $SockSpl & $Color) EndFunc ;==>AddLogToClinet At first the Connection function was : 
      ; Connect To the Client ( Old ) Func Connect($IP, $Port) TCPStartup() $ConnectedSocket = TCPConnect($IP, $Port) If @error Then Connect($IP, $My_PORT) ; <<<<<<<<<<<<<<< This was the causing recursion error Else $Connected = True EndIf EndFunc ;==>Connect It was working  good , except I had 'recursion level has been exceeded  error (old one)

       I've tried  increasing the delay for current code by :
      Opt("TCPTimeout", 5000) However it did not help yet what do you suggest to me ?
       
    • Andreik
      Image streaming
      By Andreik
      I am working on a desktop remote application so I made a simple server and client to send binary screen captures from client to server. All it's good if I test both scripts on my computer but when the client is in other network many screen shots are corrupted and some of them looks good. Have any idea why?
       
      Client:
      #include <ScreenCapture.au3> #include <Memory.au3> #include <WinAPI.au3> #include <GDIPlus.au3> TCPStartup() $Client = TCPConnect(@IPAddress1,12100) _GDIPlus_Startup() While True Local $hHBitmap = _ScreenCapture_Capture('') Local $hBitmap = _GDIPlus_BitmapCreateFromHBITMAP($hHBitmap) $bData = _GDIPlus_StreamImage2BinaryString($hBitmap) _GDIPlus_BitmapDispose($hBitmap) _WinAPI_DeleteObject($hHBitmap) TCPSend($Client,'~stream:' & BinaryLen($bData)) While BinaryLen($bData) $a = TCPSend($Client, $bData) $bData = BinaryMid($bData, $a+1, BinaryLen($bData)-$a) WEnd Sleep(10) WEnd TCPCloseSocket($Client) TCPShutdown() _GDIPlus_Shutdown() Func _GDIPlus_StreamImage2BinaryString($hBitmap, $sFormat = "JPG", $iQuality = 100) ;UEZ Local $sImgCLSID, $tGUID, $tParams, $tData Switch $sFormat Case "JPG" $sImgCLSID = _GDIPlus_EncodersGetCLSID($sFormat) $tGUID = _WinAPI_GUIDFromString($sImgCLSID) $tData = DllStructCreate("int Quality") DllStructSetData($tData, "Quality", $iQuality) ;quality 0-100 Local $pData = DllStructGetPtr($tData) $tParams = _GDIPlus_ParamInit(1) _GDIPlus_ParamAdd($tParams, $GDIP_EPGQUALITY, 1, $GDIP_EPTLONG, $pData) Case "PNG", "BMP", "GIF", "TIF" $sImgCLSID = _GDIPlus_EncodersGetCLSID($sFormat) $tGUID = _WinAPI_GUIDFromString($sImgCLSID) Case Else Return SetError(1, 0, 0) EndSwitch Local $hStream = _WinAPI_CreateStreamOnHGlobal() ;http://msdn.microsoft.com/en-us/library/ms864401.aspx If @error Then Return SetError(2, 0, 0) _GDIPlus_ImageSaveToStream($hBitmap, $hStream, DllStructGetPtr($tGUID), DllStructGetPtr($tParams)) If @error Then Return SetError(3, 0, 0) Local $hMemory = _WinAPI_GetHGlobalFromStream($hStream) ;http://msdn.microsoft.com/en-us/library/aa911736.aspx If @error Then Return SetError(4, 0, 0) Local $iMemSize = _MemGlobalSize($hMemory) If Not $iMemSize Then Return SetError(5, 0, 0) Local $pMem = _MemGlobalLock($hMemory) $tData = DllStructCreate("byte[" & $iMemSize & "]", $pMem) Local $bData = DllStructGetData($tData, 1) _WinAPI_ReleaseStream($hStream) ;http://msdn.microsoft.com/en-us/library/windows/desktop/ms221473(v=vs.85).aspx _MemGlobalFree($hMemory) Return $bData EndFunc ;==>_GDIPlus_StreamImage2BinaryString  
      Server
      Global $Buffer, $BufferSize Global $Count = 0 TCPStartup() $Server = TCPListen(@IPAddress1,12100) If @error Then MsgBox(0,'',@error) Do $Socket = TCPAccept($Server) Sleep(10) Until $Socket <> -1 While True $Recv = TCPRecv($Socket,10240) If $Recv = -1 Then ExitLoop ElseIf $Recv Then If StringLeft($Recv,7) = '~stream' Then $BufferSize = StringSplit($Recv,':')[2] $Buffer = '0x' Do $Recv = TCPRecv($Socket,10240) If BinaryLen($Recv) <> 0 Then $BufferSize -= BinaryLen($Recv) $Buffer &= StringTrimLeft($Recv,2) EndIf Until $BufferSize = 0 $hFile = FileOpen(@ScriptDir & '\Screen' & $Count & '.jpeg',18) FileWrite($hFile,$Buffer) FileClose($hFile) $Buffer = Null $Count += 1 EndIf If $Count = 100 Then ExitLoop EndIf Sleep(10) WEnd TCPCloseSocket($Socket) TCPCloseSocket($Server) TCPShutdown() Exit