Checking To See If A Website Is Dead? How Can I Do This? Thanks!

neogia · April 23, 2006

Ok, I tried like 15 from here, but none of them worked. Don't I just have to set the Internet Explorer Properites >> Connections >> LAN Settings >> Proxy Settings and Port Number, and then try to access a page? Do I need a certain type of internet? I've got cable internet from Charter. I just have never used a proxy before except at work. If you could enlighten me, I'd be grateful.

neogia · April 23, 2006

Hah, nevermind, I just did a test with the full list, ran it with no throttling (10 at a time, etc). And apparently Windows does limit the number of processes, mine stopped at 339, lol. Or maybe that was just how many proxies were in the list. It only took about a minute to test all of them, too. And you were right, about half of them work, half don't.

CrewXp · April 23, 2006

lol, so I should be able to run about 20 at a time then. How would I do this (throttling the number that's used in the test each time)

CrewXp · April 23, 2006

hey neogia, I sent you a pm with a question. Thanks

neogia · April 23, 2006

Here's a version that throttles, and works. I haven't had time to look at yours to see why it doesn't yet, but I will, trust me. Work sucks. The throttle is set to 100 running at a time. To change that, just change... you guessed it, $throttle to whatever you want. 20 takes forever. If you set it to $aLines[0], it will just run all of them at once, which turns out to be the fastest, but most resource-intensive setting. I'm sure there's probably a happy medium, and I'll leave that to you to find.

#include <Coroutine.au3>
#include <File.au3>

Dim $aLines
Dim $aWorkingProxies[1]
$aWorkingProxies[0] = 0;Count
$sFilePath = "C:\Scripting\CheckProxy\proxy.txt"
_FileReadToArray($sFilePath, $aLines)
$throttle = 100; Change this to change the throttle (20 takes forever!)
Dim $aRunningProxies[$throttle]
$CheckProxy = _CoCreate('Func _CheckProxy($proxy)|  If $proxy <> "" Then|       HttpSetProxy(2, $proxy)|    EndIf|  $code = InetGet("http://www.google.com", @TempDir & "\deadtest.tmp")|Local $aRetArr[2]|$aRetArr[0] = $code|$aRetArr[1] = $proxy|    Return $aRetArr|EndFunc')
$iCount = $throttle + 1
For $i = 1 To $iCount - 1
    $temp = $aLines[$i]
    $aRunningProxies[$i - 1] = _CoStart($CheckProxy, '$temp')
    ToolTip($i & "/" & $aLines[0])
Next
While $iCount < $aLines[0] Or _AnyAlive() == 1
    _RunNextProxy()
    ToolTip($iCount - 1 & "/" & $aLines[0])
    Sleep(50)
WEnd
ToolTip("")
$GUI = GUICreate("Results", 270, 620)
$edit = GUICtrlCreateEdit("Working Proxies", 10, 10, 250, 600)
$dispString = ""
For $i = 0 To UBound($aWorkingProxies) - 1
    $dispString &= $aWorkingProxies[$i] & @CRLF
Next
GUICtrlSetData($edit, $dispString)
GUISetState()
While 1
    If GUIGetMsg() == -3 Then
        Exit
    EndIf
WEnd
_CoCleanup()

Func _RunNextProxy()
    For $i = 0 To 19
        If _CoStatus($aRunningProxies[$i]) == "returned" Then
            $temp = _CoGetReturn($aRunningProxies[$i])
            If $temp[0] = 1 Then
                $aWorkingProxies[0] += 1
                ReDim $aWorkingProxies[$aWorkingProxies[0] + 1]
                $aWorkingProxies[$aWorkingProxies[0]] = $temp[1]
            EndIf
            If $iCount <= $aLines[0] Then
                $temp = $aLines[$iCount]
                $iCount += 1
                $aRunningProxies[$i] = _CoStart($CheckProxy, '$temp')
            EndIf
        EndIf
    Next
EndFunc

Func _AnyAlive()
    For $i = 0 To UBound($aRunningProxies) - 1
        If _CoStatus($aRunningProxies[$i]) == "running" Then
            Return 1
        EndIf
    Next
    Return 0; No multithreaded functions are still running
EndFunc

CrewXp · April 23, 2006

Woo!! Fixed something in it, and it seems to work fine now.

No need to look at the code I pm'd you. This seem to work fine. I'll just incorporate it now and change it to work.

Also: is there any way to change Coroutine to make it not show a new tray icon for each thread? #NoTrayIcon? Where should I put that. Thanks

Edited April 23, 2006 by CrewXp

CrewXp · April 23, 2006

hehe. Sent you 'what should be' one last pm regarding this. Oh yeah, and the no tray icon thing.

neogia · April 23, 2006

For no tray icon, just add #NoTrayIcon to the function created with $CheckProxy = _CoCreate()..., because that's the "template" that creates an instance for each time you call _CoStart($CheckProxy,...). As for turning the script into a function, it's just a matter of passing in the array of proxies, however you go about doing that is up to you. I've converted it to #NoTrayIcon and encompassed in a wrapper function.

#include <Coroutine.au3>
#include <File.au3>


Func _CheckProxy($prArray)
    Local $aWorkingProxies[1]
    Global $aWorkingProxies[0] = 0;Count
    $throttle = 100; Change this to change the throttle (20 takes forever!)
    Global $aRunningProxies[$throttle]
    $CheckProxy = _CoCreate('Func _CheckProxy($proxy)|#NoTrayIcon|  If $proxy <> "" Then|       HttpSetProxy(2, $proxy)|    EndIf|  $code = InetGet("http://www.google.com", @TempDir & "\deadtest.tmp")|Local $aRetArr[2]|$aRetArr[0] = $code|$aRetArr[1] = $proxy|    Return $aRetArr|EndFunc')
    $iCount = $throttle + 1
    For $i = 1 To $iCount - 1
        $temp = $prArray[$i]
        $aRunningProxies[$i - 1] = _CoStart($CheckProxy, '$temp')
        ToolTip($i & "/" & $prArray[0])
    Next
    While $iCount < $prArray[0] Or _AnyAlive() == 1
        _RunNextProxy($prArray[$iCount])
        $iCount += 1
        ToolTip($iCount - 1 & "/" & $prArray[0])
        Sleep(50)
    WEnd
    ToolTip("")
    $GUI = GUICreate("Results", 270, 620)
    $edit = GUICtrlCreateEdit("Working Proxies", 10, 10, 250, 600)
    $dispString = ""
    For $i = 0 To UBound($aWorkingProxies) - 1
        $dispString &= $aWorkingProxies[$i] & @CRLF
    Next
    GUICtrlSetData($edit, $dispString)
    GUISetState()
    While 1
        If GUIGetMsg() == -3 Then
            Exit
        EndIf
    WEnd
    _CoCleanup()
EndFunc

Func _RunNextProxy($proxy)
    For $i = 0 To 19
        If _CoStatus($aRunningProxies[$i]) == "returned" Then
            $temp = _CoGetReturn($aRunningProxies[$i])
            If $temp[0] = 1 Then
                $aWorkingProxies[0] += 1
                ReDim $aWorkingProxies[$aWorkingProxies[0] + 1]
                $aWorkingProxies[$aWorkingProxies[0]] = $temp[1]
            EndIf
            $aRunningProxies[$i] = _CoStart($CheckProxy, '$proxy')
        EndIf
    Next
EndFunc

Func _AnyAlive()
    For $i = 0 To UBound($aRunningProxies) - 1
        If _CoStatus($aRunningProxies[$i]) == "running" Then
            Return 1
        EndIf
    Next
    Return 0; No multithreaded functions are still running
EndFunc

CrewXp · April 25, 2006

hey neogia.

When you read this, could you check your pm? I'm done some work on the code and have it working so far. The only problems is that sometimes it shows a working proxy as a non-working one (0). And it takes longer than usual to complete. Thanks again.

neogia · April 26, 2006

Yeah, I've been trying to track down the reason that it slowed down, I have noticed that as well. I'm at work right now, so no AutoIt access, but if I had to guess, try changing the first for loop in _RunNextProxy() from Ubound($aRunningProxies) - 1 to something like 20, and see if that increases the speed. I'll do some more testing when I get home.

Edit: Oh, and I uploaded the most recent bug-free version of Coroutine.au3 (v1.0.3) try that also. See my signature.

Edited April 26, 2006 by neogia

CrewXp · April 26, 2006

thx so much again.

CrewXp · April 26, 2006

oh yeah, I tried your most recent version of coroutine and I get

"Error reading the file"

"{Script Path}\1" ;Where script path= the place my script is

for every coroutine proxytest it spawns.

neogia · April 26, 2006

oh yeah, I tried your most recent version of coroutine and I get
"Error reading the file"
"{Script Path}\1" ;Where script path= the place my script is
for every coroutine proxytest it spawns.

Blast! I uploaded the wrong one. Sorry, I can't fix until I get home ~11:15CST

CrewXp · April 26, 2006

yea sure, ok. Also. haha I dont see exactly where to set it to 20. thx

neogia · April 26, 2006

Bah! I can't for the life of me figure out why wrapping it in a function (_CheckProxy()) runs so much slower. With throttle set to 100, this script runs all 658 proxies in about 205 seconds steadily.

#include <Coroutine.au3>
#include <File.au3>

HotKeySet("{ESC}", "_Exit")

Dim $aLines
Dim $aWorkingProxies[1]
Global $numReturned = 0
Global $timer = TimerInit()
$aWorkingProxies[0] = 0;Count
$sFilePath = "C:\Scripting\CheckProxy\proxy.txt"
_FileReadToArray($sFilePath, $aLines)
$throttle = 150; Change this to change the throttle (20 takes forever!)
Dim $aRunningProxies[$throttle]
$CheckProxy = _CoCreate('Func _CheckProxy($proxy)|#NoTrayIcon|  If $proxy <> "" Then|       HttpSetProxy(2, $proxy)|    EndIf|  $code = InetGet("http://www.google.com", @TempDir & "\deadtest.tmp")|Local $aRetArr[2]|$aRetArr[0] = $code|$aRetArr[1] = $proxy|    Return $aRetArr|EndFunc')
$iCount = $throttle + 1
For $i = 1 To $iCount - 1
    $temp = $aLines[$i]
    $aRunningProxies[$i - 1] = _CoStart($CheckProxy, '$temp')
    ToolTip("Started: " & $i & "/" & $aLines[0] & @CR & "Ended: " & $numReturned & "/" & $i & @CR & "Elapsed Time: " & Round(TimerDiff($timer)/1000, 1) & " seconds", 0, 0)
Next
While $iCount < $aLines[0] Or _AnyAlive() == 1
    _RunNextProxy()
    ToolTip("Started: " & $iCount - 1 & "/" & $aLines[0] & @CR & "Ended: " & $numReturned & "/" & $iCount - 1 & @CR & "Elapsed Time: " & Round(TimerDiff($timer)/1000, 1) & " seconds", 0, 0)
    Sleep(50)
WEnd
ToolTip("")
$GUI = GUICreate("Results", 270, 620)
$edit = GUICtrlCreateEdit("Working Proxies", 10, 10, 250, 600)
$dispString = $aWorkingProxies[0] - 1 & " Working Proxies Found:" & @CRLF
For $i = 1 To UBound($aWorkingProxies) - 1
    If $aWorkingProxies[$i] == "" Then
        $dispString &= "<no proxy>" & @CRLF
    Else
        $dispString &= $aWorkingProxies[$i] & @CRLF
    EndIf
Next
$dispString = StringTrimRight($dispString, 2)
GUICtrlSetData($edit, $dispString)
GUISetState()
While 1
    If GUIGetMsg() == -3 Then
        Exit
    EndIf
    Sleep(100)
WEnd
_CoCleanup()

Func _RunNextProxy()
    For $i = 0 To UBound($aRunningProxies) - 1
        If _CoStatus($aRunningProxies[$i]) == "returned" Then
            $numReturned += 1
            $temp = _CoGetReturn($aRunningProxies[$i])
            If $temp[0] = 1 Then
                $aWorkingProxies[0] += 1
                ReDim $aWorkingProxies[$aWorkingProxies[0] + 1]
                $aWorkingProxies[$aWorkingProxies[0]] = $temp[1]
            EndIf
            If $iCount <= $aLines[0] Then
                $temp = $aLines[$iCount]
                $iCount += 1
                $aRunningProxies[$i] = _CoStart($CheckProxy, '$temp')
            EndIf
        EndIf
    Next
EndFunc

Func _AnyAlive()
    For $i = 0 To UBound($aRunningProxies) - 1
        If _CoStatus($aRunningProxies[$i]) == "running" Then
            Return 1
        EndIf
    Next
    Return 0; No multithreaded functions are still running
EndFunc

Func _Exit()
    _CoCleanup()
    Exit
EndFunc

Requires the latest version of Coroutine.au3 (really fixed this time :"> )

CrewXp · April 26, 2006

when you said about 250 seconds, did you mean this script, or the function-wrapped one. Is this the same as the one you wrapped in a function?

thx

neogia · April 26, 2006

The ~205 seconds one is the one I just posted, I can't figure out why wrapping it into a function slows it down. I'm trying it again.

Checking To See If A Website Is Dead? How Can I Do This? Thanks!

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members