HTTP 407 proxy authentication error when calling a web service

Question

I'm working on a .NET app that calls 3rd party web services over the internet. The services do not use SOAP, so we manually construct an XML request document, send it to the service via HTTP, and retrieve an XML response.

Our code is a Windows service that is run in the context of a normal Windows domain account, and sits behind a proxy server (Microsoft ISA Server) configured to require NTLM authentication. The account running our service has permission to access the internet through the proxy server.

The code looks like this:

// Create the request object.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";

// Configure for authenticating proxy server requiring Windows domain credentials.
request.Proxy = New WebProxy(proxyAddress) { UseDefaultCredentials = true };

// Set other required headers.
request.Accept = acceptableMimeType;
request.Headers.Add(HttpRequestHeader.AcceptCharset, acceptableCharset);
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "none");
request.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-gb");
request.Headers.Add(HttpRequestHeader.CacheControl, "no-store");
request.Headers.Add(HttpRequestHeader.ContentEncoding, "none");
request.Headers.Add(HttpRequestHeader.ContentLanguage, "en-gb");
request.ContentType = requestMimeType;
request.ContentLength = requestBytes.Length;

// Make the method call.
using(Stream stream = request.GetRequestStream()) {
    stream.Write(requestBytes, 0, requestBytes.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

// Extract the data from the response without relying on the HTTP Content-Length header
// (we cannot trust all providers to set it correctly).
const int bufferSize = 1024 * 64;
List<byte> responseBytes = new List<byte>();
using(Stream stream = new BufferedStream(response.GetResponseStream(), bufferSize)) {
    int value;
    while((value = stream.ReadByte()) != -1) {
        responseBytes.Add((byte) value);
    }
}

This works fine if the proxy server is turned off, or the URL has been whitelisted as not requiring authentication, but as soon as authentication is active, it always fails with an HTTP 407 error.

I put the above code in a test harness, and tried every method I could think of for configuring the request.Proxy property, without success.

I then noticed that all the 3rd party web services that we have to call are HTTPS. When I tried accessing them as HTTP instead, the proxy authentication started working. Is there some extra hoop I have to jump through to get proxy authentication and HTTPS to play nicely?

PS: The same problems occur with the open source SmoothWall proxy server, so I can't just write it off as a bug in ISA Server.

PPS: I'm aware that you can configure proxy settings in app.config, but (a) doing it in code shouldn't make any difference, and (b) the application design requires that we read the proxy settings from a database at runtime.

Answer 1

Have you tried setting the proxy in the app.config ?

To disable the proxy, in the App.config file add the following configuration

<system.net>
  <defaultProxy enabled="false" useDefaultCredentials="false">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>

To enable the proxy and to use the default proxy settings(specified in IE) add this configuration in your App.config

<system.net>
  <defaultProxy enabled="true" useDefaultCredentials="true">
    <proxy/>
    <bypasslist/>
    <module/>
  </defaultProxy>
</system.net>