Sign in to follow this  
Followers 0
ripdad

_TCP_PassiveProxyFilter_v0.3b

3 posts in this topic

#1 ·  Posted (edited)

Most of this code is from an similar project posted here.

The difference is that this one uses 100% TCP.

It is also just a skeleton of what it could achieve - and not limited like the other one was.

So, what is meant by "Passive"?

Unlike the other code... As the proxy receives data from the web server,
it will immediately forward it to the browser. So, when you download a file,
you will see it in real time from the browser, along with whatever progress
dialog it may have.

I consider this script a good start at what I'd like to see it become eventually --
If and when I find the time to work on it.

Please let me know if any crashes, bugs or quirks. Thanks!

Updated: August 31, 2014 - Various improvements.

Known issues:

- Some TCP functions do not work correctly in 64bit. I have not tried to track that down yet.

- HTTPS is not supported yet.

_TCP_PassiveProxyFilter_v0.3b.au3

Edited by ripdad
1 person likes this

"The mediocre teacher tells. The Good teacher explains. The superior teacher demonstrates. The great teacher inspires." -William Arthur Ward

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Updated 1st post to v0.2b

---

Now that I have some time, let's talk a little about this script.

It is your browser that makes the request to a web server
based on the source code of web documents (ie: html, js, css, etc.).

There are plug-ins that allow "some control" over the content,
but not "total control".

A firewall is good, but they don't handle traffic content.
You still need a firewall - even if it's Windows Firewall.

At first, I tried to find a way to intercept web traffic by various
means, but nothing seemed to work the way I wanted.

In the end, the only way to get "total control" is with a local proxy.

---

Three main points:

1) This is a single socket script; one for the browser and one for the remote server.
There is no point in attempting true multiple socket transfers, because AutoIt
is not multi-threaded. As a result, it will be slower loading a web page --
from your browsers point of view.

2) And so, I consider this program good for "Light to Medium" browsing.

3) I came to the conclusion, that it's best that the browser handle secure
connections. As a result, This Script does not support secure connections.

---

I took the script for a test ride on 'tube to see how it performs with streaming,
and it did well (for me), with no problems.

I downloaded a 56Mb file at about 650kb/s. Not bad for a single socket script.

I have not tested it for sites that hosts java/flash games, etc.

---

Okay, lets talk about some particular things of operation.

Most likely, the first time you run this script, Windows Firewall will pop up and ask
permission to allow this program to get access. This is normal - just click yes.

The GUI appears afterwards and you'll notice that "AutoProxy" is unchecked. This is a
feature that is in Firefox and Internet Explorer that uses "system settings" found in
the registry.

In order to use this script, you must configure your browser to use a proxy.
Most are located under Tools->Options->Network (or something like that).

I don't know about other browsers, but in Firefox, under the "Connection Settings" dialog,
there is a setting called: "Use system proxy settings". It is usually set there by default.
If you leave it there and place a checkmark in "AutoProxy", then you are good to go!

For manual entry, on that same dialog, it's called: "Manual proxy configuration".
The IP and Port are located at the top of this programs title bar.
Type the IP number in "HTTP proxy" and the Port number in "Port".
Click ok and that should do it.

Keep in mind that, if you go the manual entry method, the proxy must be running when
browsing. If the proxy is not running, the browser will show a connection error.

If you go with the "AutoProxy" method, the browser will work as expected when both
are running. Without the proxy running or "AutoProxy" is unchecked, the browser will
work in its normal mode of operation, as if there was no proxy.

Of the two methods, manual entry is the most secure.

As a note, you can set the behavior of "AutoProxy" to default as "checked" near
the top of the script. ie: Local $AutoProxy = 1

In addition, I added code to save the original settings of the registry when
that feature is used. When this program closes, it will restore those settings.

---

Let's move on to the Blacklist and Whitelist features.

The only reason I added a Blacklist to this script, is because even if you have
a site whitelisted, there are some things you don't want from that same site.

The List's are not strict. You can add a partial string and it will detect on that.
ie: ad-serve or ad-serve.com or www.ad-serve.com
Any one of those should work.

Most major sites have resource or image servers in addition to their own servers.
I suppose these days, most are using Cloud servers in one way or another.

Some sites have no resource server; the site server handles all of it. Some sites
have one resource server and others might have as many as a dozen of them.

So, in order to get "The Full Browsing Experience" of a site, you will also
need to add their resource servers. You'll be able to know who they are by
viewing the edit window of the GUI when browsing to a site.

For example, you whitelisted wwww.somesite.com and went to it in your browser.
You might notice in the edit window of the GUI several entries that read
something like this:

Unlisted: www.i1img-cdn.com
Unlisted: www.i7img-cdn.com

It's a safe bet that those are resource servers for the site you whitelisted.
If you can, add the partial string and it will catch them all. ie: img-cdn.com
If you can't, then you'll have to add each one on their own line.

Sometimes, when reading a topic in a forum, the member will post an image
from an outside source. For instance: www.photobucket.com

If you really want to see that image - then you'll have to whitelist that
outside source -- either temporarily or permanently.

---

Known Bugs or Quirks

1) Sometimes, the browser will reset. I haven't been able to track that down yet,
as it doesn't happen often enough.

2) A TCPConnect timeout can last up to 20 seconds while freezing the GUI, if a
site is not responding. I am currently looking for something that will replace
it, in this particular code. So far, all it does is cause more problems if I try
some existing code that deals with timeouts.

---

A few more things...

I wrote this script in UltraEdit32, using AutoIt v3.3.6.0, and then checked
against v3.3.8.1 and v3.3.10.2, on WinXP Pro SP2. So, it should be compatible
across those versions of AutoIt.

I tested this script using FireFox and nothing else. I don't have the time
nor the will to make it compatible for all browsers, if it is indeed needed.

I have not done any extensive testing on Vista+.

TODO: the upload code - to properly handle files larger than 100kb.

 

Edited by ripdad

"The mediocre teacher tells. The Good teacher explains. The superior teacher demonstrates. The great teacher inspires." -William Arthur Ward

Share this post


Link to post
Share on other sites

I had a chance to work on this recently -- so here is an update, such as it is.

1st post updated to v0.3b


"The mediocre teacher tells. The Good teacher explains. The superior teacher demonstrates. The great teacher inspires." -William Arthur Ward

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • Simpel
      By Simpel
      Hi, 
      I tried a function by @kor here: 
      But it does not work through a proxy. So I tried a bit. With some forum help months ago I had a www-solution through a proxy for some other stuff. But trying it with ntp didn't work.
      #include <WinHttp.au3> Global Const $sProxy = "http://prx01.xxx.de:8080" Global Const $sUserName = "UserName", $sPassword = "PassWord" $vOpen = _WinHttpOpen(Default, $WINHTTP_ACCESS_TYPE_NAMED_PROXY, $sProxy) $vConnect = _WinHttpConnect($vOpen, "pool.ntp.org", 123) ; Port 123 $vRequest = _WinHttpSimpleSendRequest_ProxyAuth($vConnect) Global $sSource = _WinHttpSimpleReadData($vRequest) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sSource = ' & $sSource & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console Exit Func _WinHttpSimpleSendRequest_ProxyAuth($hConnect, $sType = Default, $sPath = Default, $sReferrer = Default, $sDta = Default, $sHeader = Default) Local $hRequest = _WinHttpSimpleSendRequest($hConnect, $sType, $sPath, $sReferrer, $sDta, $sHeader) If $hRequest Then Local $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE) If $iStatusCode = $HTTP_STATUS_PROXY_AUTH_REQ Then Local $iSupportedSchemes, $iFirstScheme, $iAuthTarget ; Query Authorization scheme If _WinHttpQueryAuthSchemes($hRequest, $iSupportedSchemes, $iFirstScheme, $iAuthTarget) Then _WinHttpSetCredentials($hRequest, $iAuthTarget, $iFirstScheme, $sUserName, $sPassword) ; Set passed credentials _WinHttpSendRequest($hRequest) ; Set passed credentials _WinHttpReceiveResponse($hRequest) ; And wait for the response again EndIf EndIf EndIf Return $hRequest EndFunc (This snippet only tries to get raw data.) 
      $sSource is "". With no port (instead of 123) I only get the informations I see when I type url into a browser - so it's not the information I seek. (There is not time included in http source code.)
      Some hints? Regards, Conrad
    • ripdad
      By ripdad
      This script is based on algorithm code from EnrMa.
      Updated: January 07, 2017
      Changes are in the script header.
       LocalProxyServer_v0.15b.au3
       
      Known Issues:
      POST is not working consistently in uploading files. AutoIt x64 does not work properly with this script.  
    • RyukShini
      By RyukShini
      #include "WinHttp.au3" #include <array.au3> #include <file.au3> #include <WinAPI.au3> #include <string.au3> #include <WindowsConstants.au3> #include <FileConstants.au3> #include <ListviewConstants.au3> #include <GUIConstantsEx.au3> #include <MsgBoxConstants.au3> ; Learning HTTP Requests with WinHTTP ; Global $idOP, $usr, $list, $idEXIT, $call HotKeySet("{ESC}", "Terminate") Local $hGUI = GUICreate("Learning", 750, 500, 223, 202, $WS_BORDER) Local $idEXIT = GUICtrlCreateButton("Exit", 660, 440, 75, Default) Local $idCon = GUICtrlCreateButton("Start", 10, 10, 725, Default) Local $siteLab = GUICtrlCreateLabel("Select Test", 20, 48 ,Default, 35) GUICtrlSetFont (-1,9, 800); bold Local $IG = GUICtrlCreateCheckbox("test", 75, 45) Local $realm = GUICtrlCreateCheckbox("test1", 75, 65) Local $site3 = GUICtrlCreateCheckbox("test2", 200, 45) Local $site4 = GUICtrlCreateCheckbox("test3", 200, 65) $list = GUICtrlCreateListView( " PROXY | NAME1 | NAME2 | STATUS ", 20, 100, 710, 300, $LVS_NOSORTHEADER+$LVS_SINGLESEL) GUISetState(@SW_SHOW, $hGUI) While 1 Switch GUIGetMsg() Case $GUI_EVENT_CLOSE, $idEXIT Exit Case $IG If _IsChecked($IG) Then $call = "test" EndIf Case $idCon Call($call) EndSwitch WEnd Func test() $url = "www.w3schools.com" $sPage = "/tags/demo_form.asp" Local $proxies Local $names _FileReadToArray("proxies.txt", $proxies) ; read the list of names to array _FileReadToArray("Okay.txt", $names) ; read the list of names to array For $i = 1 To UBound($proxies) - 1 $Read = $names[$i] $Datastring = ('') $newreadamount = _StringBetween($read,$Datastring, ':') $newreadamount[0] = StringReplace($newreadamount[0], ":", "") $name1 = $newreadamount[0] $Datastring2 = (':') $newreadamount2 = _StringBetween($read,$Datastring2, '') $newreadamount2[0] = StringReplace($newreadamount2[0], ":", "") $name2 = $newreadamount2[0] $sAdditionalData = "name1="&$name1&"&name2="&$name2 MsgBox(4096, "Test", $proxies[$i] & " - " & $name1&":"&$name2,1) ; Initialize and get session handle $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0", $WINHTTP_ACCESS_TYPE_NAMED_PROXY, $proxies[$i]) _WinHttpSetTimeouts($hOpen, 15, 15, 15, 15) ; Get connection handle $hConnect = _WinHttpConnect($hOpen, $url) ; Make a request $hRequest = _WinHttpOpenRequest($hConnect, "POST", $sPage) ; Send it. Specify additional data to send too. This is required by the Google API: _WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", $sAdditionalData) ; Wait for the response _WinHttpReceiveResponse($hRequest) ; See what's returned Dim $sReturned If _WinHttpQueryDataAvailable($hRequest) Then ; if there is data Do $sReturned &= _WinHttpReadData($hRequest) Until @error EndIf ; Close handles _WinHttpCloseHandle($hRequest) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) ; See what's returned MsgBox(4096, "Returned", $sReturned, 1) If StringInStr($sReturned,'Input was received as:') Then GUICtrlCreateListViewItem($proxies[$i] & "|"&$name1&"|"&$name2&"|Success", $list) Sleep(500) ContinueLoop ElseIf $sReturned = "" Then FileWrite("nottested.txt",$name1&":"&$name2 & @CRLF) GUICtrlCreateListViewItem($proxies[$i] & "|"&$name1&"|"&$name2&"|Bad Proxy", $list) Sleep(500) ContinueLoop EndIf Next EndFunc Func Terminate() Exit 0 EndFunc Func _IsChecked($idControlID) Return BitAND(GUICtrlRead($idControlID), $GUI_CHECKED) = $GUI_CHECKED EndFunc ;==>_IsChecked This is my code and it runs just fine.
      My problem is that if I receive Bad Proxy I need it to test the request again with the same array input $name1 / $name2 until I receive
      'Input was received as:'
      So basically
       
      ElseIf $sReturned = "" Then FileWrite("nottested.txt",$name1&":"&$name2 & @CRLF) GUICtrlCreateListViewItem($proxies[$i] & "|"&$name1&"|"&$name2&"|Bad Proxy", $list) Sleep(500) TRY AGAIN WITH NEW PROXY AND SAME CREDENTIALS EndIf Is this possible and if so do you have either some example code and/or some helpfile I can read.

      Thanks in advance!
    • RyukShini
      By RyukShini
      I am coding a country specific IP checker with a buddy, he made the website/api and I am coding a client to run through IPs.
      i split my IPs so it doesn't read the port instead of IP:PORT just IP, however after reading through every IP
      I need to add all ports back in their right place again so it becomes IP:PORT again, but this part made my head spin a bit.
      Any help is highly appreciated!
      Thanks in advance.
      #include <ColorConstants.au3> ; Including required files #include <GUIConstantsEx.au3> #include <file.au3> #include <Array.au3> #include <string.au3> #include <StringConstants.au3> Example() Func Example() While 1 Local $ip Local $ips = "18k.txt" ;sets file path Local $strFileContents = FileRead($ips) $Arrayips = StringRegExp($strFileContents,'((?:\d+)(?:\.\d+){3})',3) For $i = 1 To UBound($Arrayips) - 1 ; Creating the object $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", "http://mysite:8080/json/" & $Arrayips[$i], False) ; Post url ; Header data > $oHTTP.SetRequestHeader("Host", "mysite.xyz:8080") $oHTTP.SetRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0") $oHTTP.SetRequestHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") $oHTTP.SetRequestHeader("Accept-Language", "en-US,en;q=0.5") $oHTTP.SetRequestHeader("DNT", "1") $oHTTP.SetRequestHeader("Connection", "keep-alive") $oHTTP.SetRequestHeader("Cache-Control", "max-age=0") ; Header data < ; Performing the Request $oHTTP.Send() $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status If $oStatusCode <> 200 then MsgBox(4096, "Response code", $oStatusCode) ExitLoop EndIf $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists FileWrite($file, $oReceived) $read = FileRead("Received.html") ;read fil if StringInStr($read, '"US",') Then FileWrite("output.txt", $Arrayips[$i] & @CRLF) Else ContinueLoop EndIf ;Sends the post data with the given details Next WEnd EndFunc ;==>Example  
    • Simpel
      By Simpel
      Hi there.
      I'm testing this code:
      #include "WinHttp.au3" ; http://www.w3schools.com/php/demo_form_validation_escapechar.php $sUserName = "SomeUserName" $sEmail = "some.email@something.com" $sDomain = "www.w3schools.com" $sPage = "/php/demo_form_validation_escapechar.php" ; Data to send $sAdditionalData = "name=" & $sUserName & "&email=" & $sEmail ; Initialize and get session handle $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0") ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $hOpen = ' & $hOpen & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ; Get connection handle $hConnect = _WinHttpConnect($hOpen, $sDomain) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $hConnect = ' & $hConnect & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ; Make a request $hRequest = _WinHttpOpenRequest($hConnect, "POST", $sPage) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $hRequest = ' & $hRequest & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console ; Send it. Specify additional data to send too. This is required by the Google API: $hSendRequest = _WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", $sAdditionalData) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $hSendRequest = ' & $hSendRequest & @CRLF & '>Error code: ' & @error & @TAB & "@extended: " & @extended & @CRLF) ;### Debug Console ; Wait for the response $hReceiveResponse = _WinHttpReceiveResponse($hRequest) ; See what's returned Dim $sReturned If _WinHttpQueryDataAvailable($hRequest) Then ; if there is data Do $sReturned &= _WinHttpReadData($hRequest) Until @error EndIf ; Close handles _WinHttpCloseHandle($hRequest) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) ; See what's returned ConsoleWrite("Returned: " & $sReturned & @CRLF) But sending a request fails.
      Console outputs this:
      >Running:(3.3.12.0) ... --> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop @@ Debug(14) : $hOpen = 0x00000000007DF540 >Error code: 0 @@ Debug(18) : $hConnect = 0x00000000008271C0 >Error code: 0 @@ Debug(22) : $hRequest = 0x00000000008B88E0 >Error code: 0 @@ Debug(26) : $hSendRequest = 0 >Error code: 1 @extended: 0 Returned: +>17:00:18 AutoIt3.exe ended.rc:0 +>17:00:18 AutoIt3Wrapper Finished. >Exit code: 0    Time: 0.6003 I'm testing it at work. There, we always have to identify at a proxy to be allowed to go to internet. But requests via inet.au3 will work. Maybe our firewall will block the request?
      Any ideas? I don't have any possibilities to test at home, because of OSX.
       
      Regards, Conrad