Sign in to follow this  
Followers 0
CrewXp

Checking To See If A Website Is Dead? How Can I Do This? Thanks!

37 posts in this topic

I have created a proxy tester. It loads a list of proxies to an array. One by one, sets the http proxy to $ip (the proxy), tries to inetget google (to see if the proxy is dead). But it doesn't work right.

I tried using adlibenable, but the script freezes when trying to inetget a dead site (or in this case a proxy that doesn't work). I have to wait for it to timeout (which is usually like 7 seconds. And when I have a list of 80 proxies, it takes a while)

What's the best way to do this?

Summary: HttpSetProxy(2, $ip) ;Where $ip is the xxx.xxx.xxx.xx:8080 Proxy

;Code to inetget google to see if the proxy works

;If successful, msgbox $ip works!!

;Else goto next proxy

Thanks

Share this post


Link to post
Share on other sites



Isn't there discussion in other threads about problems with InetGet and timing issues and workarounds that have been implemented in the latest beta version that may be relevant to your problem?

Share this post


Link to post
Share on other sites

yea, there is. I'm posting in the hang problem thread, but I still would like a good method to figure out if the site's dead once it's fixed. I'm not sure exactly the best way to do this. Thanks

Share this post


Link to post
Share on other sites

yea, there is. I'm posting in the hang problem thread, but I still would like a good method to figure out if the site's dead once it's fixed. I'm not sure exactly the best way to do this. Thanks

I thought maybe IsValidURL but I keep getting unable to use the URLMON.DLL, I'm sure it's my inexperience in using DLL's and possibly calling it wrong (even got the encountered error pop up once). Reading up on it, it looks like you might be able to use COM though?

[center]Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.[/center]

Share this post


Link to post
Share on other sites

isvalidurl?? Never heard of that before. And I'm kind of trying to not use inetget since it's not working right right now. I'm waiting for db to release it. I tried the usual ping, but it pings using my regular connection, not with the IE proxy settings.

Share this post


Link to post
Share on other sites

why not just ping it?

Share this post


Link to post
Share on other sites

when I ping it, it returns the same amount of time for each proxy I use. I think it doesn't ping it using my proxy in IE settings, it uses the regular connection.

Share this post


Link to post
Share on other sites

I have created a proxy tester. It loads a list of proxies to an array. One by one, sets the http proxy to $ip (the proxy), tries to inetget google (to see if the proxy is dead). But it doesn't work right.

I tried using adlibenable, but the script freezes when trying to inetget a dead site (or in this case a proxy that doesn't work). I have to wait for it to timeout (which is usually like 7 seconds. And when I have a list of 80 proxies, it takes a while)

What's the best way to do this?

Summary: HttpSetProxy(2, $ip) ;Where $ip is the xxx.xxx.xxx.xx:8080 Proxy

;Code to inetget google to see if the proxy works

;If successful, msgbox $ip works!!

;Else goto next proxy

Thanks

Does you proxies answer to ping?

if ping($proxy) then 
   If TestProxy($proxy) Then 
      ;Success
   esle
      ;failure
   endif
endif

Share this post


Link to post
Share on other sites

nope. Here's a working (but slow) test proxy.'

210.114.174.36:50050

and for a non-working one, make it up. Ex:

123.456.123.18:8080

Share this post


Link to post
Share on other sites

Is this the best way to do this??

For $i = 1 To UBound($prArray) - 1
$ip=StringStripWS($prArray[$i], 3)
$len = StringLen($ip)
If $len>6 Then
Local $sProxRegKey = "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings"
RegWrite($sProxRegKey, "ProxyEnable", "REG_DWORD", 1)
        RegWrite($sProxRegKey, "ProxyServer", "REG_SZ", $ip)
    FileDelete(@TempDir & "\deadtest.tmp")
    HttpSetProxy(2, $ip)
    $Inet=InetGet("http://www.google.com", @TempDir & "\deadtest.tmp", 1,1)
While @InetGetActive
Wend

EndIf
Next
Func checkifalive()
If @InetGetBytesRead>0 Then
Msgbox(0,"CrewKit ADLIB","Proxy set to "&$ip)
InetGet("abort")
EndIf
$timeout=$timeout+100
If $timeout>3000 Then
InetGet("abort")
$timeout=0
EndIf
ToolTip($seconds&" Seconds"&@LF&"Proxy: "&$ip&@LF&@InetGetBytesRead&" Bytes Read",0,0)
$seconds=$seconds+10/100
EndFunc

Share this post


Link to post
Share on other sites

even on working ones, sometimes it just skips it for some reason. The ones it does show the msgbox on works though.

Is there any faster way than that? Thx

Share this post


Link to post
Share on other sites

bump, thx

Share this post


Link to post
Share on other sites

Well, I've created a multithreaded version. It doesn't directly fix the problem of decreasing the timeout, but you can run a bunch of them at the same time, and therefore only have to wait for the time out of approximately one bad proxy. You'll have to use the beta version of my Coroutine.au3 UDF, as the one in my signature hasn't been updated yet. Tested and works correctly for no proxy and fake proxies, tried your "slow" one, and it reported 0. Tried it in Internet Explorer and that proxy was dead, so it looks like it worked for that one too.

#include <Array.au3>
#include <Coroutine.au3>

Local $prArray[5]
$prArray[0] = ""
$prArray[1] = "1.2.3.4:8080"
$prArray[2] = ""
$prArray[3] = "210.114.174.36:50050"
$prArray[4] = "1.2.3.4:8080"

$CheckProxy = _CoCreate('Func _CheckProxy($proxy)|  If $proxy <> "" Then|       HttpSetProxy(2, $proxy)|    EndIf|  $code = InetGet("http://www.google.com", @TempDir & "\deadtest.tmp")|   Return $code|EndFunc')
; Assuming $prArray is an array that contains one proxy per element in the form: xxx.xxx.xxx.xxx:xxxx
Local $PIDs[UBound($prArray)]
For $i = 0 To UBound($prArray) - 1
    $nonArrayVar = $prArray[$i];Can't send array elements to _CoStart()
    $PIDs[$i] = _CoStart($CheckProxy, '$nonArrayVar'); ("Multithreaded function to start", "Variables to pass to function")
Next
Local $results[UBound($prArray)]
While _AnyAlive()
    Sleep(100)
WEnd
For $i = 0 To UBound($prArray) - 1
    If _CoStatus($PIDs[$i]) == "returned" Then
        $results[$i] = _CoGetReturn($PIDs[$i])
    EndIf
Next
_ArrayDisplay($results, "")
_CoCleanup()

Func _AnyAlive()
    For $i = 0 To UBound($prArray) - 1
        If _CoStatus($PIDs[$i]) == "running" Then
            Return 1
        EndIf
    Next
    Return 0; No multithreaded functions are still running
EndFunc

Coroutine.au3


[u]My UDFs[/u]Coroutine Multithreading UDF LibraryStringRegExp GuideRandom EncryptorArrayToDisplayString"The Brain, expecting disaster, fails to find the obvious solution." -- neogia

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

hi, thanks. this is a different approach i never thought of, and it seems to work. what's the max it can handle at a time though? 10?

Anyways, in the example its 5 only. I've tried, but can't get anything yet. Is there any way to make this script read my whole txt file (with about 20 proxies, the number varying each time its updated)??

$count=UBound($prArray1)
If $count>5 Then
While $count>0
If $count-5>0 Then
$prArray2 = _ArrayCreate("delete")
_ArrayDelete( $prArray2,1)
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
Doit($prArray2)
Else
$prArray2 = _ArrayCreate("delete")
_ArrayDelete( $prArray2,1)
For $i=1 To $count
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
Next
Doit($prArray2)
EndIf
WEnd
Else
$prArray2 = _ArrayCreate("delete")
_ArrayDelete( $prArray2,1)

For $i=1 To $count
_ArrayAdd($prArray2,$prArray1[$count-1])
$count=$count-1
Next
Doit($prArray2)
EndIf

Func Doit($prArray)
$CheckProxy = _CoCreate('Func _CheckProxy($proxy)|  If $proxy <> "" Then|       HttpSetProxy(2, $proxy)|    EndIf|  $code = InetGet("http://www.google.com", @TempDir & "\deadtest.tmp")|   Return $code|EndFunc')
; Assuming $prArray is an array that contains one proxy per element in the form: xxx.xxx.xxx.xxx:xxxx
Local $PIDs[UBound($prArray)]
For $i = 0 To UBound($prArray) - 1
    $nonArrayVar = $prArray[$i];Can't send array elements to _CoStart()
    $PIDs[$i] = _CoStart($CheckProxy, '$nonArrayVar'); ("Multithreaded function to start", "Variables to pass to function")
Next
Local $results[UBound($prArray)]
While _AnyAlive()
    Sleep(100)
WEnd
For $i = 0 To UBound($prArray) - 1
    If _CoStatus($PIDs[$i]) == "returned" Then
        $results[$i] = _CoGetReturn($PIDs[$i])
    EndIf
Next
_ArrayDisplay($results, "")
_CoCleanup()

EndFunc

Func _AnyAlive()
    For $i = 0 To UBound($prArray) - 1
        If _CoStatus($PIDs[$i]) == "running" Then
            Return 1
        EndIf
    Next
    Return 0; No multithreaded functions are still running
EndFunc

Where Doit($prArray) is the function is your CheckProxy.au3

And $prArray1 is the big array containing every proxy

Or... is there a simpler approach to this. Thanks!

Edited by CrewXp

Share this post


Link to post
Share on other sites

It can handle any number of proxies, it just starts a new instance of CheckProxy() each time you call _CoStart(), so the array you pass to DoIt(), no matter how many proxies it has in it, will be stepped through, calling _CoStart() with each one. Each _CoStart() creates a new process, though, so Windows might limit that number, but I'm not sure. That should only come into play with hundreds of processes, and even then, when one finishes, it will just allow a new one to be run, so nothing you need to worry about. Put as many as you want in the array, and pass it to your DoIt() function, and it should work.

...I've tried, but I can't get anything yet...

What does that mean? could you be a little more specific?

Also, could you give a working proxy so I can check it?


[u]My UDFs[/u]Coroutine Multithreading UDF LibraryStringRegExp GuideRandom EncryptorArrayToDisplayString"The Brain, expecting disaster, fails to find the obvious solution." -- neogia

Share this post


Link to post
Share on other sites

#16 ·  Posted (edited)

like what you posted is AWESOME. A different approach than I did. But overall, I'm trying to read in a list of lots of proxies. I tried modifying the example to read in all proxies from the list (dat file below), then call your program (ten at a time to save memory). If it finds working ones in the ten, it stops and shows them. Otherwise it runs the next ten.

Here is a example.

http://www.crewxp.com/pl3.dat

Is really a txt file with proxies line by line.

Using the program I originally created, I found these working ones: (The problem with mine is it sometimes displays the working ones and sometimes doesnt)

84.96.120.250:80

84.58.218.229:8000

83.16.77.246:80

83.144.107.180:80

83.100.233.49:8080

82.68.24.145:2301

82.60.53.139:80

1/2 of the proxies in the dat file work, 1/2 doesn't. Edited by CrewXp

Share this post


Link to post
Share on other sites

#18 ·  Posted (edited)

lol its ok. I'll wait then.

yea, it works properly. But the example you gave is just for 5. I have a big list of them and can't run all of them at the same time. That's over a hundred.

I wanted to read them all in a array, then search ten at a time for working ones. If none found, it searches the next ten till the end. (Or if the user sets $findnext to 1. Aka: If the one found is too slow.)

Edited by CrewXp

Share this post


Link to post
Share on other sites

Ok, I'm glad it's working for you. You should be able to run 100 at a time, though. But I can see how you would want to save memory by running 10 at a time. Could you post your proxy list so I can try it when I get home?


[u]My UDFs[/u]Coroutine Multithreading UDF LibraryStringRegExp GuideRandom EncryptorArrayToDisplayString"The Brain, expecting disaster, fails to find the obvious solution." -- neogia

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0