Sign in to follow this  
Followers 0
micechal

Polling InetGet background handlers too often can freeze the whole script?

4 posts in this topic

Hi. Recently I've been working on a simple little script that'd request given URL given number of times, but instead of doing it one after another (sequentially), I wanted to do given number of requests at a time (simultaneous).

Here's what I managed to achieve already:

#include <Array.au3>

$url = "http://www.darmowylicznik.pl/licznik.php?id=109634"
Local $count = 100 ;how many times to download given URL?
Local $threadsCount = 4 ;how many simultaneous tasks?

Local $handlers[$threadsCount] ;prepare an array to keep InetGet handlers
Local $availableIndexes[1] ;declare an array to keep track of unoccupied $handlers indexes

For $i = 0 To $threadsCount - 1 ;initially populate the unoccupied indexes list with how much "threads" were declared
   _ArrayAdd($availableIndexes, $i)
Next

$timer = TimerInit() ;for benchmarking various $threadsCount values

For $i = 1 To $count
   If UBound($availableIndexes) - 1 == 0 Then
      ConsoleWrite("No unoccupied indexes left. Waiting for at least one of them to finish its job..." & @CRLF)
   EndIf
   While UBound($availableIndexes) - 1 == 0
      For $k = 0 To $threadsCount - 1
         If InetGetInfo($handlers[$k], 2) Then
            InetClose($handlers[$k])
            _ArrayAdd($availableIndexes, $k)
            ConsoleWrite("InetGet handler at index [" & $k & "] has finished. Handle closed. Added " & $k & " as an available index." & @CRLF)
         EndIf
         Sleep(50)
      Next
   WEnd
   ConsoleWrite("Preparing to create new InetGet task. Available indexes: " & UBound($availableIndexes) - 1 & @CRLF)
   ;_ArrayDisplay($availableIndexes) ;for troubleshooting
   Local $currIndex = $availableIndexes[1]
   $handlers[$currIndex] = InetGet($url, "NUL", 1, 1)
   _ArrayDelete($availableIndexes, 1)
   ConsoleWrite("Created new background InetGet task on [" & $currIndex & "] array index. Available indexes left: " & UBound($availableIndexes) - 1 & @CRLF)
Next

Local $mainTime = Round(TimerDiff($timer))
ConsoleWrite("Main loop finished in " & $mainTime & " ms." & @CRLF)

From what I've been able to test, it works well most of the time when the Sleep() at line 27. waits for at least 50 ms. Anything lower than that often makes the whole script to get stuck at

 

No unoccupied indexes left. Waiting for at least one of them to finish its job...

state and will remain so forever.

Unless there is some nasty bug / misconception I wasn't able to spot, then it would mean that polling InetGet handlers too often prevents them from downloading any data?

Regards

Share this post


Link to post
Share on other sites



This line is only run when there are no unoccupied indexes available:

While UBound($availableIndexes) - 1 == 0
so InetGet() had already been called at least 4 times (depends on the thread count).

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

I added an array display to this because once you fill $handles, you check them in an odd fashion.

1234

5234

5634

5674

5678

9678

...

such that you would have already checked 3 of the 4 items in the array in the loop prior.  I dont understand the gain from that.

it does this because

$availableindexes follows this routine, from which you only delete the item in index 1.

"" 0 1 2 3

"" 1 2 3

"" 23

"" 3

so, i would assume that nothing is happening 'simultaneously' it is following a very predictable pattern that happens sequentially.   You are requesting data and storing it, then accessing it 4 at a time (3 of which you accessed on the previous loop without acting on the return).

#include <Array.au3>

$url = "http://www.darmowylicznik.pl/licznik.php?id=109634"
Local $count = 100 ;how many times to download given URL?
Local $threadsCount = 4 ;how many simultaneous tasks?

Local $handlers[$threadsCount] ;prepare an array to keep InetGet handlers
Local $availableIndexes[1] ;declare an array to keep track of unoccupied $handlers indexes

For $i = 0 To $threadsCount - 1 ;initially populate the unoccupied indexes list with how much "threads" were declared
   _ArrayAdd($availableIndexes, $i)
Next

$timer = TimerInit() ;for benchmarking various $threadsCount values

For $i = 1 To $count
   If UBound($availableIndexes) - 1 == 0 Then
      ConsoleWrite("No unoccupied indexes left. Waiting for at least one of them to finish its job..." & @CRLF)
   EndIf
   While UBound($availableIndexes) - 1 == 0
      For $k = 0 To $threadsCount - 1
         If InetGetInfo($handlers[$k], 2) Then
            InetClose($handlers[$k])
            _ArrayAdd($availableIndexes, $k)
            ConsoleWrite("InetGet handler at index [" & $k & "] has finished. Handle closed. Added " & $k & " as an available index." & @CRLF)
         EndIf
         Sleep(50)
      Next
   WEnd
   ConsoleWrite("Preparing to create new InetGet task. Available indexes: " & UBound($availableIndexes) - 1 & @CRLF)
   ;_ArrayDisplay($availableIndexes) ;for troubleshooting
   Local $currIndex = $availableIndexes[1]
   $handlers[$currIndex] = InetGet($url, "NUL", 1, 1)
   _ArrayDisplay($availableIndexes, "available")
   _ArrayDelete($availableIndexes, 1)
   ConsoleWrite("Created new background InetGet task on [" & $currIndex & "] array index. Available indexes left: " & UBound($availableIndexes) - 1 & @CRLF)
   _ArrayDisplay($handlers, "handlers")
Next

Local $mainTime = Round(TimerDiff($timer))
ConsoleWrite("Main loop finished in " & $mainTime & " ms." & @CRLF)
Edited by boththose

,-. .--. ________ .-. .-. ,---. ,-. .-. .-. .-.
|(| / /\ \ |\ /| |__ __||| | | || .-' | |/ / \ \_/ )/
(_) / /__\ \ |(\ / | )| | | `-' | | `-. | | / __ \ (_)
| | | __ | (_)\/ | (_) | | .-. | | .-' | | \ |__| ) (
| | | | |)| | \ / | | | | | |)| | `--. | |) \ | |
`-' |_| (_) | |\/| | `-' /( (_)/( __.' |((_)-' /(_|
'-' '-' (__) (__) (_) (__)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0