Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Use WebClient with socks proxy

Is there any way to use a socks proxy with WebClient? Specifically with the DownloadString method that it provides?

I don't want to use any third party stuff like privoxy, freecap whatever and I can't use commercial libraries like those from Chilkat. I tried using stuff from http://www.mentalis.org/ in fact I used their WebRequest implementation but they don't seem to have something similar for WebClient.

like image 307
Para Avatar asked Oct 29 '12 13:10

Para


People also ask

Is it possible to use a SOCKS proxy with httpclient 5?

Not .NET 5, not .NET Core 3.1, or any other version you can think of. 6. The usage for a SOCKS proxy is actually the same as HTTP proxy usage with HttpClient, except that you can now specify the scheme of the address as socks. For example :

What are the supported socks and HTTPS protocols?

Supported protocols are 4 (SOCKS v.4), 5 (SOCKS v.5) and connect (HTTPS proxy). If the protocol is not specified, SOCKS version 5 is used. -x proxy_address [:port] Connect to destination using a proxy at proxy_address and port.

Do proxy servers work with Web Sockets?

Some proxy servers are harmless and work fine with Web Sockets; others will prevent Web Sockets from working correctly, causing the connection to fail. In some cases additional proxy server configuration may be required, and certain proxy servers may need to be upgraded to support Web Sockets.

How to connect to a proxy server using SSH client?

Use ProxyCommand as option for SSH client. -X proxy_protocol Use proxy_protocol when talking to the proxy server. Supported protocols are 4 (SOCKS v.4), 5 (SOCKS v.5) and connect (HTTPS proxy).


Video Answer


2 Answers

SOCKS is not supported directly by the WebRequest/WebResponse classes and by extension, the WebClient class (it relies on WebRequest to do its work).

It really can't, as it works on the transport layer (TCP/IP) and not through a simple redirect to a server that forwards HTTP requests (which is the level that the WebRequest/WebResponse classes work on).

You can create a specialized derivation of WebRequest/WebResponse (that uses ProxySocket to handle the low-level handshaking and then) and then create a specialized WebClient class which overrides the GetWebRequest and GetWebResponse methods.

Once you have that class substituted for your WebClient instances, it should work as normal (you might have to set up the proxy in each case where you use it though).

like image 185
casperOne Avatar answered Oct 16 '22 14:10

casperOne


Here is how I ended up doing it, thank you casperOne for the answer

public class SocksWebClient : WebClient
    {
        public IProxyDetails ProxyDetails { get; set; }
        public string UserAgent { get; set; }

        protected override WebRequest GetWebRequest(Uri address)
        {
            WebRequest result = null;

            if (ProxyDetails != null)
            {
                if (ProxyDetails.ProxyType == ProxyType.Proxy)
                {
                    result = (HttpWebRequest)WebRequest.Create(address);
                    result.Proxy = new WebProxy(ProxyDetails.FullProxyAddress);
                    if (!string.IsNullOrEmpty(UserAgent))
                        ((HttpWebRequest)result).UserAgent = UserAgent;
                }
                else if (ProxyDetails.ProxyType == ProxyType.Socks)
                {
                    result = SocksHttpWebRequest.Create(address);
                    result.Proxy = new WebProxy(ProxyDetails.FullProxyAddress);
                    //TODO: implement user and password

                }
                else if (ProxyDetails.ProxyType == ProxyType.None)
                {
                    result = (HttpWebRequest)WebRequest.Create(address);
                    if (!string.IsNullOrEmpty(UserAgent))
                        ((HttpWebRequest)result).UserAgent = UserAgent;
                }
            }
            else
            {
                result = (HttpWebRequest)WebRequest.Create(address);
                if (!string.IsNullOrEmpty(UserAgent))
                    ((HttpWebRequest)result).UserAgent = UserAgent;
            }


            return result;
        }

    }

The SocksHttpWebRequest class is taken from the blog linked to by @casperOne, the code for which is as follows:

using System;
using System.Collections.Specialized;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
using Org.Mentalis.Network.ProxySocket;

namespace Ditrans
{
    public class SocksHttpWebRequest : WebRequest
    {

        #region Member Variables

        private readonly Uri _requestUri;
        private WebHeaderCollection _requestHeaders;
        private string _method;
        private SocksHttpWebResponse _response;
        private string _requestMessage;
        private byte[] _requestContentBuffer;

        // darn MS for making everything internal (yeah, I'm talking about you, System.net.KnownHttpVerb)
        static readonly StringCollection validHttpVerbs =
            new StringCollection { "GET", "HEAD", "POST", "PUT", "DELETE", "TRACE", "OPTIONS" };

        #endregion

        #region Constructor

        private SocksHttpWebRequest(Uri requestUri)
        {
            _requestUri = requestUri;
        }

        #endregion

        #region WebRequest Members

        public override WebResponse GetResponse()
        {
            if (Proxy == null)
            {
                throw new InvalidOperationException("Proxy property cannot be null.");
            }
            if (String.IsNullOrEmpty(Method))
            {
                throw new InvalidOperationException("Method has not been set.");
            }

            if (RequestSubmitted)
            {
                return _response;
            }
            _response = InternalGetResponse();
            RequestSubmitted = true;
            return _response;
        }

        public override Uri RequestUri
        {
            get { return _requestUri; }
        }

        public override IWebProxy Proxy { get; set; }

        public override WebHeaderCollection Headers
        {
            get
            {
                if (_requestHeaders == null)
                {
                    _requestHeaders = new WebHeaderCollection();
                }
                return _requestHeaders;
            }
            set
            {
                if (RequestSubmitted)
                {
                    throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
                }
                _requestHeaders = value;
            }
        }

        public bool RequestSubmitted { get; private set; }

        public override string Method
        {
            get
            {
                return _method ?? "GET";
            }
            set
            {
                if (validHttpVerbs.Contains(value))
                {
                    _method = value;
                }
                else
                {
                    throw new ArgumentOutOfRangeException("value", string.Format("'{0}' is not a known HTTP verb.", value));
                }
            }
        }

        public override long ContentLength { get; set; }

        public override string ContentType { get; set; }

        public override Stream GetRequestStream()
        {
            if (RequestSubmitted)
            {
                throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
            }

            if (_requestContentBuffer == null)
            {
                _requestContentBuffer = new byte[ContentLength];
            }
            else if (ContentLength == default(long))
            {
                _requestContentBuffer = new byte[int.MaxValue];
            }
            else if (_requestContentBuffer.Length != ContentLength)
            {
                Array.Resize(ref _requestContentBuffer, (int) ContentLength);
            }
            return new MemoryStream(_requestContentBuffer);
        }

        #endregion

        #region Methods

        public static new WebRequest Create(string requestUri)
        {
            return new SocksHttpWebRequest(new Uri(requestUri));
        }

        public static new WebRequest Create(Uri requestUri)
        {
            return new SocksHttpWebRequest(requestUri);
        }

        private string BuildHttpRequestMessage()
        {
            if (RequestSubmitted)
            {
                throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
            }

            var message = new StringBuilder();
            message.AppendFormat("{0} {1} HTTP/1.0\r\nHost: {2}\r\n", Method, RequestUri.PathAndQuery, RequestUri.Host);

            // add the headers
            foreach (var key in Headers.Keys)
            {
                message.AppendFormat("{0}: {1}\r\n", key, Headers[key.ToString()]);
            }

            if (!string.IsNullOrEmpty(ContentType))
            {
                message.AppendFormat("Content-Type: {0}\r\n", ContentType);
            }
            if (ContentLength > 0)
            {
                message.AppendFormat("Content-Length: {0}\r\n", ContentLength);
            }

            // add a blank line to indicate the end of the headers
            message.Append("\r\n");

            // add content
            if(_requestContentBuffer != null && _requestContentBuffer.Length > 0)
            {
                using (var stream = new MemoryStream(_requestContentBuffer, false))
                {
                    using (var reader = new StreamReader(stream))
                    {
                        message.Append(reader.ReadToEnd());
                    }
                }
            }

            return message.ToString();
        }

        private SocksHttpWebResponse InternalGetResponse()
        {
            var response = new StringBuilder();
            using (var _socksConnection =
                new ProxySocket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                var proxyUri = Proxy.GetProxy(RequestUri);
                var ipAddress = GetProxyIpAddress(proxyUri);
                _socksConnection.ProxyEndPoint = new IPEndPoint(ipAddress, proxyUri.Port);
                _socksConnection.ProxyType = ProxyTypes.Socks5;

                // open connection
                _socksConnection.Connect(RequestUri.Host, 80);
                // send an HTTP request
                _socksConnection.Send(Encoding.ASCII.GetBytes(RequestMessage));
                // read the HTTP reply
                var buffer = new byte[1024];

                var bytesReceived = _socksConnection.Receive(buffer);
                while (bytesReceived > 0)
                {
                    response.Append(Encoding.ASCII.GetString(buffer, 0, bytesReceived));
                    bytesReceived = _socksConnection.Receive(buffer);
                }
            }
            return new SocksHttpWebResponse(response.ToString());
        }

        private static IPAddress GetProxyIpAddress(Uri proxyUri)
        {
            IPAddress ipAddress;
            if (!IPAddress.TryParse(proxyUri.Host, out ipAddress))
            {
                try
                {
                    return Dns.GetHostEntry(proxyUri.Host).AddressList[0];
                }
                catch (Exception e)
                {
                    throw new InvalidOperationException(
                        string.Format("Unable to resolve proxy hostname '{0}' to a valid IP address.", proxyUri.Host), e);
                }
            }
            return ipAddress;
        }

        #endregion

        #region Properties

        public string RequestMessage
        {
            get
            {
                if (string.IsNullOrEmpty(_requestMessage))
                {
                    _requestMessage = BuildHttpRequestMessage();
                }
                return _requestMessage;
            }
        }

        #endregion

    }
}

Note that, as @casperOne pointed out, this makes use of a (free) third party library called ProxySocket.

like image 14
2 revs, 2 users 75% Avatar answered Oct 16 '22 15:10

2 revs, 2 users 75%