Polling InetGet background handlers too often can freeze the whole script?

[micechal]

Hi. Recently I've been working on a simple little script that'd request given URL given number of times, but instead of doing it one after another (sequentially), I wanted to do given number of requests at a time (simultaneous).

Here's what I managed to achieve already:

expandcollapse popup

#include <Array.au3>

$url = "http://www.darmowylicznik.pl/licznik.php?id=109634"
Local $count = 100 ;how many times to download given URL?
Local $threadsCount = 4 ;how many simultaneous tasks?

Local $handlers[$threadsCount] ;prepare an array to keep InetGet handlers
Local $availableIndexes[1] ;declare an array to keep track of unoccupied $handlers indexes

For $i = 0 To $threadsCount - 1 ;initially populate the unoccupied indexes list with how much "threads" were declared
   _ArrayAdd($availableIndexes, $i)
Next

$timer = TimerInit() ;for benchmarking various $threadsCount values

For $i = 1 To $count
   If UBound($availableIndexes) - 1 == 0 Then
      ConsoleWrite("No unoccupied indexes left. Waiting for at least one of them to finish its job..." & @CRLF)
   EndIf
   While UBound($availableIndexes) - 1 == 0
      For $k = 0 To $threadsCount - 1
         If InetGetInfo($handlers[$k], 2) Then
            InetClose($handlers[$k])
            _ArrayAdd($availableIndexes, $k)
            ConsoleWrite("InetGet handler at index [" & $k & "] has finished. Handle closed. Added " & $k & " as an available index." & @CRLF)
         EndIf
         Sleep(50)
      Next
   WEnd
   ConsoleWrite("Preparing to create new InetGet task. Available indexes: " & UBound($availableIndexes) - 1 & @CRLF)
   ;_ArrayDisplay($availableIndexes) ;for troubleshooting
   Local $currIndex = $availableIndexes[1]
   $handlers[$currIndex] = InetGet($url, "NUL", 1, 1)
   _ArrayDelete($availableIndexes, 1)
   ConsoleWrite("Created new background InetGet task on [" & $currIndex & "] array index. Available indexes left: " & UBound($availableIndexes) - 1 & @CRLF)
Next

Local $mainTime = Round(TimerDiff($timer))
ConsoleWrite("Main loop finished in " & $mainTime & " ms." & @CRLF)

From what I've been able to test, it works well most of the time when the Sleep() at line 27. waits for at least 50 ms. Anything lower than that often makes the whole script to get stuck at

 

No unoccupied indexes left. Waiting for at least one of them to finish its job...

state and will remain so forever.

Unless there is some nasty bug / misconception I wasn't able to spot, then it would mean that polling InetGet handlers too often prevents them from downloading any data?

Regards

[JohnOne]

From where is the line...

If InetGetInfo($handlers[$k], 2) Then

Getting info?

I  do not see any calls to Inet* anything before that.

[micechal]

This line is only run when there are no unoccupied indexes available:

While UBound($availableIndexes) - 1 == 0

so InetGet() had already been called at least 4 times (depends on the thread count).

[iamtheky]

I added an array display to this because once you fill $handles, you check them in an odd fashion.

1234

5234

5634

5674

5678

9678

...

such that you would have already checked 3 of the 4 items in the array in the loop prior.  I dont understand the gain from that.

it does this because

$availableindexes follows this routine, from which you only delete the item in index 1.

"" 0 1 2 3

"" 1 2 3

"" 23

"" 3

so, i would assume that nothing is happening 'simultaneously' it is following a very predictable pattern that happens sequentially.   You are requesting data and storing it, then accessing it 4 at a time (3 of which you accessed on the previous loop without acting on the return).

expandcollapse popup

#include <Array.au3>

$url = "http://www.darmowylicznik.pl/licznik.php?id=109634"
Local $count = 100 ;how many times to download given URL?
Local $threadsCount = 4 ;how many simultaneous tasks?

Local $handlers[$threadsCount] ;prepare an array to keep InetGet handlers
Local $availableIndexes[1] ;declare an array to keep track of unoccupied $handlers indexes

For $i = 0 To $threadsCount - 1 ;initially populate the unoccupied indexes list with how much "threads" were declared
   _ArrayAdd($availableIndexes, $i)
Next

$timer = TimerInit() ;for benchmarking various $threadsCount values

For $i = 1 To $count
   If UBound($availableIndexes) - 1 == 0 Then
      ConsoleWrite("No unoccupied indexes left. Waiting for at least one of them to finish its job..." & @CRLF)
   EndIf
   While UBound($availableIndexes) - 1 == 0
      For $k = 0 To $threadsCount - 1
         If InetGetInfo($handlers[$k], 2) Then
            InetClose($handlers[$k])
            _ArrayAdd($availableIndexes, $k)
            ConsoleWrite("InetGet handler at index [" & $k & "] has finished. Handle closed. Added " & $k & " as an available index." & @CRLF)
         EndIf
         Sleep(50)
      Next
   WEnd
   ConsoleWrite("Preparing to create new InetGet task. Available indexes: " & UBound($availableIndexes) - 1 & @CRLF)
   ;_ArrayDisplay($availableIndexes) ;for troubleshooting
   Local $currIndex = $availableIndexes[1]
   $handlers[$currIndex] = InetGet($url, "NUL", 1, 1)
   _ArrayDisplay($availableIndexes, "available")
   _ArrayDelete($availableIndexes, 1)
   ConsoleWrite("Created new background InetGet task on [" & $currIndex & "] array index. Available indexes left: " & UBound($availableIndexes) - 1 & @CRLF)
   _ArrayDisplay($handlers, "handlers")
Next

Local $mainTime = Round(TimerDiff($timer))
ConsoleWrite("Main loop finished in " & $mainTime & " ms." & @CRLF)

Edited February 3, 2015 by boththose