NetSession UDF ~ Per-Process Embedded Browser Settings (Proxy/Agent/Etc.)

[supersonic]

drego,

thank you - that's what I was looking for.

But how can I bind to a dedicated (= the current) process? The functions seems to affect other processes as well (e. g. Citrix XenApp Client). Any ideas?

Regards,

-supersonic.

[drego]

drego,

thank you - that's what I was looking for.

But how can I bind to a dedicated (= the current) process? The functions seems to affect other processes as well (e. g. Citrix XenApp Client). Any ideas?

Regards,

-supersonic.

 

I'll look into it now. In my initial tests (Years ago) I was only using it with once process for the purpose of not relying on the global registry settings. After I made the UDF recently I tested by setting a proxy in two different AutoIt processes, Hopefully the options weren't being set globally. I'll perform a test now to see if one changes the other.

Thank you for the report.

Edited January 21, 2015 by drego

[supersonic]

Thank you, I appreciate it.

[drego]

Thank you, I appreciate it.

 

I have just ran a test. Here's what I did.

1. Changed my application so that it only sets the proxy/agent 1 time at the beginning of script execution.
2. Ran the app and then ran a second and third instance of the app, each using a different proxy.
3. Began navigating with all of them at the same time.

Since I set the proxy only once in the app, if your problem were affecting me the proxy of all of the app instances would change to 1 proxy. They did not. Each instance of the app retained it's own proxy & agent settings so as far as running multiple processes, each compiled by AutoIt, they will each have whatever unique settings you apply to them.

I'm not sure about your Citrix XenApp Client but given these tests I don't know how it could be affected. Are you sure it's changing the Internet settings for "Citrix XenApp Client?" Is this a client you developed or by Citrix? Can you show me any proxy server logs to confirm it? Any data at all showing that the proxy settings were somehow applied to a second application? I only ask because it doesn't happen between multiple AutoIt compiled applications so I'm not sure why it would happen to one by Citrix. RIght now I could spawn 100 processes, each with different proxy settings and have them perform multiple embedded browser operations at the same time without interfering with each-other.

Thanks.

Edited January 21, 2015 by drego

[drego]

I just updated the main post about a bug I'm encountering with the new _ClearCookies function. There are no compile errors but upon calling it during execution I'm getting: Error: Error parsing function call.

If anyone can take a look and knows what is causing this it will be a great help to this seemingly simple function and will be much appreciated.

[SmOke_N]

Not sure what makes you think clearing the flash player folder is clearing cookies... care to enlighten?

Edit:

Maybe that's just an addition, I have no idea what that .cpl is doing, but I assume that's the attempt you're making to clear the IE cookies.

Edited January 21, 2015 by SmOke_N

[drego]

Not sure what makes you think clearing the flash player folder is clearing cookies... care to enlighten?

Edit:

Maybe that's just an addition, I have no idea what that .cpl is doing, but I assume that's the attempt you're making to clear the IE cookies.

 

Correct, the first clears IE cookes and the second empties the Flash cache. This function used to work without any problems.

Edit: Obviously it needs to be updated as I see now the directory structure is different. But is there anything else wrong? Also on the topic of clearing the flash cache, many sites use it to track people.

Edited January 21, 2015 by drego

[drego]

Correction, the Flash cache is in the same location. I commented out the flash code and got the same error so the problem is in the IE cookies command:

ShellExecuteWait("RunDll32.exe","InetCpl.cpl,ClearMyTracksByProcess 2")
; Running this from the command prompt work fine.

Edited January 21, 2015 by drego

[drego]

This is really strange. I commented out ALL the code from the _ClearCookies function, recompiled my program, and still get the same error when I call the function and only when I call the function.

[drego]

LoL it was my error in calling the function in 1 place without (). A silly mistake on my part. There is nothing wrong with the function but in the process of troubleshooting someone did make a slight improvement to it and I also added a needed line so I'm now uploading version 0.9c.

[coffeeturtle]

Thanks for the quick update to 0.9c...just reminding that your example still references 0.9b, but you probably are already making the edits.

System Requirements:

• The existance of urlmon.dll on the system usually located in WindowsSystem32

Releases:

NetSession-0.9c.au3

 

[drego]

Yeah, I was making some edits all around the post. Thanks for reminding me.

[SmOke_N]

I think I've finally exhausted my patience with the username/password thing.  urlmon and wininet are not cooperating with me at all.

[drego]

Announcement: I have just released NetSession-0.9d. This release contains a new function that was on the Planned Features list I forgot to add in the last version I posted. As now stated in the main post this will likely be the last update until we add more DLL/WinAPI code, enabling more features from the Planned Features list

[drego]

I think I've finally exhausted my patience with the username/password thing.  urlmon and wininet are not cooperating with me at all.

 

You did get the UDF working for you right? I'm assuming you're taking about further auth testing. Maybe post some examples of what you've tried?

[SmOke_N]

This was the last code I was messing with... but it requires a modification to wininet.au3 library (see attachment).

The issue is, that I'm fairly confident that you cannot globally set the username/password with urlmksetsessionoption, so I went to InternetOpen/Connect/SetOption route.

The commented code and commented out code was just for testing different things so it's not going to run out of the box.

expandcollapse popup

#Region new
Func _proxySetEx($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "", $sUserAgent = "")

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    _WinINet_Startup()
#cs
    Local $tUsername, $tPassword, $iEnum, $iErr
    Local $hIOpen = -1, $hInternet = -1
    Local $sServerName, $sServerPort
    If ($sUser <> Default And StringLen($sUser)) Or _
        ($sPass <> Default And StringLen($sPass)) Then
        $hIOpen = _WinINet_InternetOpen($sUserAgent, $INTERNET_OPEN_TYPE_PROXY, 0, $sProxy, "")
        If @error Then
            Return SetError(1, @error, 0)
        EndIf

        $sServerName = StringRegExpReplace($sProxy, "(.*?)(\:.*?)$", "$1")
        $sServerPort = StringRegExpReplace($sProxy, "(.*?\:)(.*?)$", "$2")
        $hInternet = _WinINet_InternetConnect($hIOpen, $INTERNET_SERVICE_HTTP, $sServerName, $sServerPort, _
            BitOR($INTERNET_DEFAULT_HTTP_PORT, $INTERNET_DEFAULT_HTTPS_PORT, $INTERNET_DEFAULT_SOCKS_PORT), $sUser, $sPass, 0)
        If @error Then
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(2, 0, 0)
        EndIf
    EndIf

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY_USERNAME, $sUser)
        If @error Then
            _WinINet_InternetCloseHandle($hInternet)
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(3, 0, 0)
        EndIf
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY_PASSWORD, $sPass)
        If @error Then
            _WinINet_InternetCloseHandle($hInternet)
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(4, 0, 0)
        EndIf
    EndIf
#ce
    Local $tProxy
;~      $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("wchar proxy[" & StringLen($sProxy) + 2 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
​;~      $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("wchar proxybypass[" & StringLen($sProxyByPass) + 2 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY, $tProxyBuffer)
;~      If Not _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer) Then
;~          _WinINet_InternetCloseHandle($hInternet)
;~          _WinINet_InternetCloseHandle($hIOpen)
;~          Return SetError(4, @error, 0)
;~      EndIf
    EndIf
Return 1
;~  If $hInternet <> -1 Then _WinINet_InternetCloseHandle($hInternet)
;~  If $hIOpen <> -1 Then _WinINet_InternetCloseHandle($hIOpen)

    Return 1
EndFunc
#EndRegion new


Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

And the old code that works (without user/pass), but what you set with urlmksetsessionoption stays after the exe has ran (your code does too).  So my non-user/pass proxy's are still set once the code exits (not ideal but the point was to get it to work first, and worry about other issues later).

expandcollapse popup

Func _proxySet($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "")

    Local Const $INTERNET_OPTION_PROXY = 38
    Local Const $INTERNET_OPTION_PROXY_USERNAME = 43
    Local Const $INTERNET_OPTION_PROXY_PASSWORD = 44
    Local Const $INTERNET_OPEN_TYPE_PROXY = 3

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    Local $tUsername, $tPassword, $iEnum, $iErr

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        $sUser = _WinAPI_WideCharToMultiByte($sUser)
        $tUsername = DllStructCreate("char username[" & StringLen($sUser) + 1 & "]")
        DllStructSetData($tUsername, "username", $sUser)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_USERNAME, $tUsername)
        $iErr = @error
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        $sPass = _WinAPI_WideCharToMultiByte($sPass)
        $tPassword = DllStructCreate("char password[" & StringLen($sPass) + 1 & "]")
        DllStructSetData($tPassword, "password", $sPass)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_PASSWORD, $tPassword)
        $iErr = @error
    EndIf

    Local $tProxy
    $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("char proxy[" & StringLen($sProxy) + 1 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
    $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("char proxybypass[" & StringLen($sProxyByPass) + 1 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer)
        $iErr = @error
    EndIf

    If Not $iEnum Then
        Return SetError($iErr, 0, 0)
    EndIf

    Return $iEnum
EndFunc

Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

Modified WinINet.au3 attachments:

WinINet.zip

Edited January 21, 2015 by SmOke_N

[supersonic]

drego,

currently I can't provide any screenshots from our proxy server regarding the Citrix XenApp Client issue due too many sensitive information. As SmOke_N stated before the settings seems to work "somehow" globally. So, I'm still looking for a real solution to bind a dedicated proxy server to an AutoIt process.

[drego]

drego,

currently I can't provide any screenshots from our proxy server regarding the Citrix XenApp Client issue due too many sensitive information. As SmOke_N stated before the settings seems to work "somehow" globally. So, I'm still looking for a real solution to bind a dedicated proxy server to an AutoIt process.

 

Though the settings might in some way/cases affect other processes that use IE controls the UDF is still a "real solution to bring a dedicated proxy server to an Autoit process."

I don't know what your use case is but right now I have many processes forked doing work for me on the backend, each using their own proxy and agent on their own ie controls separate of the others. If you think you can somehow improve things be my guest. Right now this is the only way to change these settings for embedded IE controls on a per-process basis or per-autoit-compiled-process basis.

Edited January 25, 2015 by drego

[drego]

This was the last code I was messing with... but it requires a modification to wininet.au3 library (see attachment).

The issue is, that I'm fairly confident that you cannot globally set the username/password with urlmksetsessionoption, so I went to InternetOpen/Connect/SetOption route.

The commented code and commented out code was just for testing different things so it's not going to run out of the box.

expandcollapse popup

#Region new
Func _proxySetEx($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "", $sUserAgent = "")

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    _WinINet_Startup()
#cs
    Local $tUsername, $tPassword, $iEnum, $iErr
    Local $hIOpen = -1, $hInternet = -1
    Local $sServerName, $sServerPort
    If ($sUser <> Default And StringLen($sUser)) Or _
        ($sPass <> Default And StringLen($sPass)) Then
        $hIOpen = _WinINet_InternetOpen($sUserAgent, $INTERNET_OPEN_TYPE_PROXY, 0, $sProxy, "")
        If @error Then
            Return SetError(1, @error, 0)
        EndIf

        $sServerName = StringRegExpReplace($sProxy, "(.*?)(\:.*?)$", "$1")
        $sServerPort = StringRegExpReplace($sProxy, "(.*?\:)(.*?)$", "$2")
        $hInternet = _WinINet_InternetConnect($hIOpen, $INTERNET_SERVICE_HTTP, $sServerName, $sServerPort, _
            BitOR($INTERNET_DEFAULT_HTTP_PORT, $INTERNET_DEFAULT_HTTPS_PORT, $INTERNET_DEFAULT_SOCKS_PORT), $sUser, $sPass, 0)
        If @error Then
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(2, 0, 0)
        EndIf
    EndIf

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY_USERNAME, $sUser)
        If @error Then
            _WinINet_InternetCloseHandle($hInternet)
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(3, 0, 0)
        EndIf
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY_PASSWORD, $sPass)
        If @error Then
            _WinINet_InternetCloseHandle($hInternet)
            _WinINet_InternetCloseHandle($hIOpen)
            Return SetError(4, 0, 0)
        EndIf
    EndIf
#ce
    Local $tProxy
;~      $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("wchar proxy[" & StringLen($sProxy) + 2 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
​;~      $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("wchar proxybypass[" & StringLen($sProxyByPass) + 2 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        _WinINet_InternetSetOption(0, $INTERNET_OPTION_PROXY, $tProxyBuffer)
;~      If Not _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer) Then
;~          _WinINet_InternetCloseHandle($hInternet)
;~          _WinINet_InternetCloseHandle($hIOpen)
;~          Return SetError(4, @error, 0)
;~      EndIf
    EndIf
Return 1
;~  If $hInternet <> -1 Then _WinINet_InternetCloseHandle($hInternet)
;~  If $hIOpen <> -1 Then _WinINet_InternetCloseHandle($hIOpen)

    Return 1
EndFunc
#EndRegion new


Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

And the old code that works (without user/pass), but what you set with urlmksetsessionoption stays after the exe has ran (your code does too).  So my non-user/pass proxy's are still set once the code exits (not ideal but the point was to get it to work first, and worry about other issues later).

expandcollapse popup

Func _proxySet($sUser = "", $sPass = "", $sProxy = "", $sProxyByPass = "")

    Local Const $INTERNET_OPTION_PROXY = 38
    Local Const $INTERNET_OPTION_PROXY_USERNAME = 43
    Local Const $INTERNET_OPTION_PROXY_PASSWORD = 44
    Local Const $INTERNET_OPEN_TYPE_PROXY = 3

    Local Const $tagInternetProxyInfo = _
        "dword dwAccessType;ptr lpszProxy;ptr lpszProxyByPass"

    Local $tUsername, $tPassword, $iEnum, $iErr

    ; set username
    If $sUser <> Default And StringLen($sUser) Then
        $sUser = _WinAPI_WideCharToMultiByte($sUser)
        $tUsername = DllStructCreate("char username[" & StringLen($sUser) + 1 & "]")
        DllStructSetData($tUsername, "username", $sUser)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_USERNAME, $tUsername)
        $iErr = @error
    EndIf

    ; set password
    If $sPass <> Default And StringLen($sPass) Then
        $sPass = _WinAPI_WideCharToMultiByte($sPass)
        $tPassword = DllStructCreate("char password[" & StringLen($sPass) + 1 & "]")
        DllStructSetData($tPassword, "password", $sPass)
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY_PASSWORD, $tPassword)
        $iErr = @error
    EndIf

    Local $tProxy
    $sProxy = _WinAPI_WideCharToMultiByte($sProxy)
    $tProxy = DllStructCreate("char proxy[" & StringLen($sProxy) + 1 & "]")
    DllStructSetData($tProxy, "proxy", $sProxy)

    Local $tProxyByPass
    $sProxyByPass = _WinAPI_WideCharToMultiByte($sProxyByPass)
    $tProxyByPass = DllStructCreate("char proxybypass[" & StringLen($sProxyByPass) + 1 & "]")
    DllStructSetData($tProxyByPass, "proxybypass", $sProxyByPass)

    Local $tProxyBuffer = DllStructCreate($tagInternetProxyInfo)
    If (($sProxyByPass <> Default) And StringLen($sProxyByPass)) Or _
        (($sProxy <> Default) And StringLen($sProxy)) Then
        DllStructSetData($tProxyBuffer, "dwAccessType", $INTERNET_OPEN_TYPE_PROXY)
        DllStructSetData($tProxyBuffer, "lpszProxy", DllStructGetPtr($tProxy))
        DllStructSetData($tProxyBuffer, "lpszProxyByPass", DllStructGetPtr($tProxyByPass))
        $iEnum += _WinAPI_UrlMkSetSessionOption($INTERNET_OPTION_PROXY, $tProxyBuffer)
        $iErr = @error
    EndIf

    If Not $iEnum Then
        Return SetError($iErr, 0, 0)
    EndIf

    Return $iEnum
EndFunc

Func _WinAPI_UrlMkSetSessionOption($nOption, $tBuffer)

    Local $aRet = DllCall("urlmon.dll", "long", "UrlMkSetSessionOption", _
        "dword", $nOption, "ptr", DllStructGetPtr($tBuffer), _
        "dword", DllStructGetSize($tBuffer), "dword", 0)
    If Not IsArray($aRet) Or $aRet[0] <> 0 Then
        If IsArray($aRet) Then Return SetError($aRet[0], 0, 0)
        Return SetError(1, 0, 0)
    EndIf

    Return 1
EndFunc

Modified WinINet.au3 attachments:

WinINet.zip

 

Were you able to set the proxy user/pass successfully on a process without passing it in the URL in this way? I've still been passing it in the URL and of course that doesn't work for visiting HTTPS sites.