Kealper

Since it looks like some people were unsure of what I was trying to say in my benchmark, I've edited my post about the benchmark with this clarification:

I guess you must have read the reply as soon as I had posted it, I changed them over to regular ol' code tags as soon as I had posted it and seen that it didn't like [autoit] ones EDIT: We're just not going to talk about why I edited this.

The map data type is available in many popular programming languages, and has certain uses that traditional arrays aren't quite as suited-for, especially in an interpreted language such as AutoIt. You can emulate maps using traditional arrays without too much hassle, but in interpreted languages, performance can suffer quite a bit. Comparing the performance of arrays to maps is a bit of an apples-to-oranges comparison, as a traditional array can't do things maps can do. Because of this, I've written a little benchmark that runs on the latest beta version of AutoIt (3.3.15.0 at the time of writing). It uses a 2d array to create a case-sensitive key/value store. If you add data to a key that does not exist, that key is created with the data it was supposed to be assigned. Basically, it's using 2d arrays to create fakes of the new native map type. Native maps can usually handle enormous amounts of key/value pairs and still be very fast at modifying or reading the data in them. AutoIt's new maps are no exception, either! CLARIFICATION: Please understand that this benchmark is NOT comparing the performance of arrays to the performance of maps. The two serve different purposes, and if you put arrays versus maps in a fair benchmark, arrays will be faster. This is true for pretty much any programming language. This benchmark is comparing the methods available for doing a native, dynamically-sized, in-memory key/value store. In the current stable release of AutoIt, the only way to do this without relying on external COM objects (Scripting.Dictionary) is to use a dynamically-sized two-dimensional array. (Or two one-dimensional dynamically-sized arrays, but we'll not get into that as there's not really a good reason to do that over a two-dimensional one.) In the current beta of AutoIt, the map type allows for exactly that, without all of the boilerplate code needed to implement them, as well as a huge speed increase to them since the heavy lifting is being done internally in the interpreter as opposed to being done in AutoIt. One of the key features of this data type is that the amount of things it can store is not a fixed size. This means the more stuff you want to add, the bigger it will grow. The more things you delete from it, the smaller it shrinks. If you need to iterate over a map, it won't have to waste time running through, for example, 9,975 blank entries just to read the 25 populated entries like you would with a 10,000-element fixed-size array. For those saying that this benchmark is not fair to arrays and that there's a more efficient way to do it so they end up doing much better in the benchmark: I know, this isn't meant to be "arrays vs. maps", it's meant to be "fake maps vs. native maps". There's also some optimization that can be done with my FakeMap code, because ReDim is actually pretty slow, but adding that would make the example code harder to read while the end result would still be the same. The benchmark creates a specified number of elements randomly, and sets their values to a random number as well. It then creates one last element and reads that last element's data and prints it into the console. It also prints out the times that each step took, in milliseconds. So, here is the result of these basic benchmarks: 1000+1 elements: Fake map population time (1000 random elements): 1251.122ms Fake map new element 1001 time: 2.022ms Read data: Hello, World! Fake map read element 1001 time: 1.12ms Real map population time (1000 random elements): 2.489ms Real map new element 1001 time: 0.01ms Read data: Hello, World! Real map read element 1001 time: 0.01msSo from here, you can see that with only 1000 elements, the fake map took over 1.2 seconds to create 1000 new elements, 2ms to add the 1001st element, and then another 1ms to read that 1001st element. The native map only took about 2.5ms to create the first 1000 elements, and 0.1ms to create and read the 1001st. Wow! Here's that same benchmark, but instead of 1,000, we'll bump it up to 10,000! Fake map population time (10000 random elements): 123555.641ms Fake map new element 10001 time: 21.328ms Read data: Hello, World! Fake map read element 10001 time: 11.637ms Real map population time (10000 random elements): 27.36ms Real map new element 10001 time: 0.014ms Read data: Hello, World! Real map read element 10001 time: 0.014msTwo minutes for 10,000 new elements in the fake map? Yikes! As you can also see, although the fake map gets exponentially slower with each order of magnitude we go up, the native maps stay fairly consistent still, even at 10,000. At this scale, the amount of time calls to Random() take starts to show. But what if we go further? Let's say... 1,000,000 elements. For obvious reasons, I won't run the fake map code with this many elements, otherwise we'd be here until this time next year waiting for the results... But here is what happens when we have a 1-million-element map, and add one more on to it, then read it back: Real map population time (1000000 random elements): 24357.752ms Real map new element 1000001 time: 0.063ms Read data: Hello, World! Real map read element 1000001 time: 0.029msAs we can see, the two million calls made to Random() certainly did a number on that run time, weighing in at just over 24 seconds. We can safely assume that most of that 24 seconds was spent just generating random numbers because... Adding another element on top of that, then reading it back, is still incredibly fast, given how much data there is, at around 60 microseconds (that's 1/1000th of one millisecond!) to add a new value, and around 30 microseconds to read it back. Even more wow! The code used for this back-of-the-napkin basic benchmark: Global $FakeMap = MapCreate() Global $Map[] Global $t Global $TestSize = 1000 #Region Fake map benchmarking $t = TimerInit() For $i = 1 To $TestSize MapSet($FakeMap, Random(1, 1000000, 1), Random(1, 1000000, 1)) Next ConsoleWrite("Fake map population time (" & $TestSize & " random elements): " & Round(TimerDiff($t), 3) & "ms" & @CRLF) $t = TimerInit() MapSet($FakeMap, "asdf", "Hello, World!") ConsoleWrite("Fake map new element " & ($TestSize + 1) & " time: " & Round(TimerDiff($t), 3) & "ms" & @CRLF) $t = TimerInit() ConsoleWrite("Read data: " & MapRead($FakeMap, "asdf") & @CRLF) ConsoleWrite("Fake map read element " & ($TestSize + 1) & " time: " & Round(TimerDiff($t), 3) & "ms" & @CRLF) #EndRegion #Region Real/native map benchmarking $t = TimerInit() For $i = 1 To $TestSize $Map[Random(1, 1000000, 1)] = Random(1, 1000000, 1) Next ConsoleWrite("Real map population time (" & $TestSize & " random elements): " & Round(TimerDiff($t), 3) & "ms" & @CRLF) $t = TimerInit() $Map["asdf"] = "Hello, World!" ConsoleWrite("Real map new element " & ($TestSize + 1) & " time: " & Round(TimerDiff($t), 3) & "ms" & @CRLF) $t = TimerInit() ConsoleWrite("Read data: " & $Map["asdf"] & @CRLF) ConsoleWrite("Real map read element " & ($TestSize + 1) & " time: " & Round(TimerDiff($t), 3) & "ms" & @CRLF) #EndRegion #Region Wrapper functions to emulate map support for this test Func MapCreate() Local $aMap[1][2] $aMap[0][0] = 0 Return $aMap EndFunc Func MapRead($aMap, $sKey) For $i = 1 To UBound($aMap, 1) - 1 If $aMap[$i][0] == $sKey Then Return $aMap[$i][1] Next Return SetError(1, 0, "") EndFunc Func MapSet(ByRef $aMap, $sKey, $vVal) For $i = 1 To UBound($aMap, 1) - 1 If $aMap[$i][0] == $sKey Then $aMap[$i][1] = $vVal Return EndIf Next Local $iSize = UBound($aMap, 1) ReDim $aMap[$iSize + 1][2] $aMap[$iSize][0] = $sKey $aMap[$iSize][1] = $vVal $aMap[0][0] = $iSize EndFunc #EndRegion

Hmm.. Seems one of the forum updates or something cleared the followers list, so I wasn't getting notifications of people posting here or I would have responded sooner... I didn't even catch that I didn't have the third parameter in this on the post or my local copy of that code... I have it in any of the stuff I do, it must have just slipped my mind to add it to the base while I was working on it, sorry about that! Using caleb41610's test script, on the code in the original post (after correcting that simple mistake!) will do 1000 connections in 10 seconds, where as Krsta's final code will do 1000 connections in 5 seconds... The reason for this is how accepting connections is handled. Mine makes processing client data a priority by always checking if any clients have sent something before checking the new connection queue and getting new clients in, where as Krsta's makes accepting new clients a priority, by only checking current client data when no new connections are queued. If I change mine around a bit and make it so it only calls Check() when $iSock = -1, I see similar performance, down to 1000 clients in 5 seconds (mine has rounded up to 6 seconds a few times, so its average is slightly slower than Krsta's). If I had to guess, mine is still slower because it uses a dynamic array size for the main client array instead of Krsta's fixed array size, but for the things I do with this code the slight performance hit it incurs is worth the benefit of a flexible max client limit. So it seems mine is still being bit in the butt by overhead in ReDim, and since Krsta isn't using ReDim, no butt-biting goes on there. This has shown me what a difference it can make to switch to fixed-size for the clients array if I ever have a need for a client that can accept connections more consistently though. Also I chose to do the dynamically re-sized 2d array for simplicity's sake, and for more flexibility, as max clients can be adjusted on-the-fly without requiring a restart (by just changing $MaxClients), and it would be much less work while developing off of this (as it was intended) to change a 2d array than it would be to create another 1d array, in my personal opinion. (Updated first post with the quick-fix I overlooked before) EDIT: I should also point out another down-side I hadn't thought of while initially writing this... With a fixed-size array, it tears through CPU cycles walking through the entire array each time, even if there isn't many clients connected. So if you've got it set to 10,000 total connections, it will have to walk through those 10,000 elements each time. With a dynamically-sized array, the number of elements will always only be the number of active clients, so if the limit is 10,000 but there are only 10 clients connected, the array will only be around 10 elements large(11 if you use one as an index). Just thought I'd throw that out there too for a bit more understanding of why I made things the way I did.

I've just made some more changes to it, this time to increase efficiency when large (over 1,000) active clients are connected and one or more clients disconnects. Turns out in this case that it is up to 500x faster, on my computer at least, to build a copy of the original array without the sub-elements you don't want and then overwrite the old array with the new copy than it is to use ReDim to re-size the one array. I would have figured the latter would have been much more efficient, but it seems there are always curve-balls in the game of programming. I've changed the original Sleep(1) call in the main loop to a custom sleep function (credit goes to monoceres for the basics behind that function) which takes delays in microseconds instead of milliseconds, because it turns out the lower-level function AutoIt's Sleep wraps appears to only be able to go down to ~10ms, so there was a decent gain in response time just by switching that over. I've got it set to 5000us (5ms) sleep delay in the example, but you can safely go down to 1000us (1ms) without much of a resource-usage hike, from my testing that is; Anything lower than 1000us doesn't seem to give much more of a performance increase, so diminishing returns, etc... I also corrected some more spelling (again... ), tweaked the little example echo server code in that base to better demonstrate my choice of why that line dealing with pulling packets from the buffer that many have said was incorrect is actually exactly how I intended it to function.

Thanks, glad to hear it's as useful to other as it is to me! PS: This bit of code shown below can be ignored/removed, it was some debug code that sneaked in to one of the changes I had made, and it seems I forgot to remove it before posting the updated version! >_< All this did is just show the total number of clients currently connected every second in the console, nothing direly important or code-breaking if removed. AdlibRegister("asdf", 1000) Func asdf() ConsoleWrite($Clients[0][0] & @CRLF) EndFunc

Yes, it probably could if it was implemented correctly, I have used this for quite a few projects myself. EDIT: Spelling.

If that is all your code, try putting TCPStartup() as the first line, that is needed to initialize sockets in AutoIt... Below is something I just whipped up that should work with the default example in the original post, although I have not actually tested this, it will at least give you an idea on what's what. TCPStartup() HotKeySet("{F5}", "SendData") Global $Sock = TCPConnect("127.0.0.1", 8080) While 1 Local $sRecv = TCPRecv($Sock, 8192) If $sRecv = "" Then ContinueLoop MsgBox(4096, "Server Reply", $sRecv) WEnd Func SendData() Local $sData = InputBox("Send To Server", "Enter the string of data you wish to send to the server...") If $sData = "" Then Return TCPSend($Sock, $sData & @CRLF) EndFunc After it runs, press F5 to bring up an input box, enter something simple in to it like "Hello" or such and press enter... you should instantly see a message box pop up saying the text you just sent to the server (which it echoed back).

Would you mind posting your client code so I could give it a stern looking-at, to see if I could spot the problem? If you'd rather not post it here, sending it in a PM to me would work as well... If telnetting to it works fine and the client doesn't, it sounds like the problem is with the client.

I've stated my reason for doing the buffer like I did, and that reason is that this is only the boiler-plate code with a very basic example. If that implementation is not what you are looking for, then making it work for your specific use-case is entirely up to you Although, I would not mind helping you with that if you need it.

Ok, finally got around to fixing that... $sRecv = StringLeft($Clients[$i][3], StringInStr($Clients[$i][3], @CRLF, 0, -1)) This is intended, it grabs all complete packets from the buffer and leaves any incomplete packets in the buffer, so if the client is still sending a large amount of data, it lets that packet get received completely while it is off checking other things. Actually parsing the packets is left up to the person implementing this in to their own project, but making a parser is fairly trivial, and can be done by using StringSplit($sRecv, @CRLF, 1) to create an array, where each element is a complete packet... From there you just have to loop through the array and process the packets as-necessary. This ensures that only a string of complete, ready-to-be-parsed packets are given to your own code, because crashes are bad The StringTrimLeft thing that removed the already-read packets from the buffer was an error which was taken care of. Finally the error of it skipping over buffered data until new data was received is also fixed, seems I just forgot to have it check the buffer and instead had it checking the data right out of TCPRecv. The code in the original post has been updated with these fixes, and thanks a lot for pointing out those bugs, they were pretty serious ones and I'm surprised I missed them!

Thanks for pointing those out, admittedly, the simple echo server stuff was added after-the-fact at like 4-5 in the morning so a friend could see some basic example for using it. I'm a bit embarrassed that I didn't catch those before posting this originally... As for the issue of not reading the entire buffer, I hadn't even noticed that at the time I wrote the example buffer code, I'll update the original post with the fixes later today, I'm not at a place where I can test the code edits currently.

Glad to be of assistance!

This? Me likey a lot I have to admit, that Console.au3 is something I fully intend on staring at the source to... My console reading function stopped working when AutoIt dropped the raw file read mode, since I was getting console text by reading the device file "con"... Thanks for pointing me in the direction of this!

Thanks for thanking! I don't post many code snippets on here, but it is nice to see that the ones I do post are appreciated