Credentials not being passed on WCF call in production server

Question

The scenario

We have a web application that makes a call to a web service (hosted in IIS the same machine) using the following code: -

using (HttpContext.Current.Request.LogonUserIdentity.Impersonate())
{
    var client = new Services.ReportFormatter(endpointName);  // endpoint name is configured in config
                                                       // Services.ReportFormatter is the generated client code using svcutil
    var response = client.DoWork();
}

The intention of the code is that the WCF call is executed using the credentials of the web app user (I hope that is blatantly obvious!) and the code works like a charm on Dev and QA machines. Needless to say, it fails in production!

The problem

The .Net exception being generated is

The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate,NTLM,Basic realm="Services"'. InnerException:The remote server returned an error: (401) Unauthorized.

Interestingly, the IIS logs suggest that no user credentials are being passed on the WCF request

Fields: 
cs-method cs-uri-stem cs-uri-query              s-port cs-username c-ip sc-status sc-substatus sc-win32-status 
GET       /MyWebApp/MyPage.aspx                   - 443 chrisf x.x.x.x
POST      /MyServicesApp/ReportFormatter.svc/ntlm - 80  -            x.x.x.x 401 2 2148074254
POST      /MyServicesApp/ReportFormatter.svc/ntlm - 80  -            x.x.x.x 401 1 0
POST      /MyServicesApp/ReportFormatter.svc/ntlm - 80  -            x.x.x.x 401 1 2148074252

Note that the request to the web app has my user name logged - but the request to the service does not. Comparing these logs to those from my dev machine, I see my name on both requests - so I think this is the cause of the failure.

Here is the WCF config - although the fact that it works in Dev + QA makes me suspect that this is not at fault. Any ideas/help would be greatly appreciated.

Web app (i.e. the WCF client) config.

<bindings>
  <basicHttpBinding>

    <binding name="ReportFormatter" closeTimeout="00:01:00" openTimeout="00:01:00"
        receiveTimeout="00:10:00" sendTimeout="00:05:00" allowCookies="false"
        bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
        maxBufferSize="6553600" maxBufferPoolSize="524288" maxReceivedMessageSize="6553600"
        messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
        useDefaultWebProxy="true" >
      <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
      <security mode="TransportCredentialOnly">
        <transport clientCredentialType="Windows" />
        <message clientCredentialType="UserName" />
      </security>
    </binding>
  </basicHttpBinding>
</bindings>

<client>
  <endpoint binding="basicHttpBinding" bindingConfiguration="ReportFormatter" contract="Services.ReportFormatter"
    name="ReportFormatter_Endpoint" address="http://localhost/MyServicesApp/ReportFormatter.svc/ntlm"/>
</client>

The web service application's config file contains this

    <bindings>
        <basicHttpBinding>
            <binding name="BasicHttpWindowsBindingWinCred" maxReceivedMessageSize="20000000">
                <readerQuotas maxStringContentLength="20000000" maxArrayLength="1000000" maxBytesPerRead="20000000" />
                <security mode="TransportCredentialOnly">
                    <!-- clientCredentialType Windows requires IIS to be configured to support Windows authentication-->
                    <transport clientCredentialType="Windows" />
                    <message clientCredentialType="UserName" />
                </security>
            </binding>
        </basicHttpBinding>
    </bindings>## Heading ##
    <services>
      <service behaviorConfiguration="IIS.Service1Behavior" name="Services.FormatReportImpl">
          <endpoint address="ntlm" binding="basicHttpBinding" bindingConfiguration="BasicHttpWindowsBindingWinCred" name="FormatReportBindingWindowsAuthentication" contract="Services.ReportFormatter" bindingNamespace="http://www.donkey.com/Reporting" />
      </service>
    </services>

OS Details

Production : Win 2003 32bit SP2 (IIS 6.0)

QA : Win 2003 32bit SP2 (IIS 6.0)

Dev: Various! My own system is Win7 64 (IIS 7.5) - we also have developers on Vista.

---

The cause of the problem and the solution

As usual, the devil is in the detail. Why oh why is it always in the detail! Gory details included in case they help some one else with similar problems.

The cause

In order to keep the question as simple as possible, I omitted to mention that the web app and the web service were not hosted in the default IIS web site. Instead, they are hosted in a separate web site (the reason being that we host several apps. + versions of the same app. on the same server) and we use the host header information to identify the web site.

Thus, the production site (and web service) is only accessible via a URL like "app.companyName.com". Thus I lied when I said that the client web.config had address="http://localhost/MyServicesApp/ReportFormatter.svc/ntlm"

it actually had

address="http://app.companyName.com/MyServicesApp/ReportFormatter.svc/ntlm"

I feel a bit stupid now because I knew that we could not pass a NTLM authentication token across a machine boundary - and as far as the WCF client is concerned "app.companyName.com" is a remote machine; it has no idea that DNS maps that address back to the same machine - well it might not have an idea (keep reading).

The excuse

In order to excuse my stupidity here is an excuse: The reason that I lied in the question was that I had already checked that the above type of setup worked on the QA server. I checked that the following worked in the client's WCF config address worked on the QA machine.

address="http://qaServerName.domainName.local/MyServicesApp/ReportFormatter.svc/ntlm"

I thought logically, that address would have to go to the domain DNS to be resolved and so logically would go through the same processing as "app.companyName.com". I even setup a DNS alias of "dnsAliasForTheQAServer" (on our local domain) and then tested that the following worked

address="http://dnsAliasForTheQAServer.domainName.local/MyServicesApp/ReportFormatter.svc/ntlm"

and it did! Now you must admit that if the machine thinks it is called "qaServerName" then you would think that "dnsAliasForTheQAServer.domainName.local" would be treated like a remote address. So I reasoned that this could not be the problem causing the failure in the production server.

It was only when desperation led us to actually make a new DNS record ("qaServerNameDnsRecord2") that we managed to reproduce the error on the QA machine! So I conclude that some very clever name resolution is being done on the WCF client and/or DNS that resolves the following host names

• "qaServerName"
• "qaServerName.domainName.local"
• "dnsAliasForTheQAServer.domainName.local"

to qaServerName - and thus logically "Localhost" (since both client and server are on the same machine) but the name resolution is not as clever when the host name is

• "qaServerNameDnsRecord2"
• "qaServerNameDnsRecord2.domainName.local"

I am sure that if I knew a thing about DNS or how windows resolves names then this would not be a surprise. But I have learnt that in future I will not be as quick to

The solution

Armed with the knowledge that the problem was that the address was resolving to an address that looked like it was remote, the solution was simple - just make the address look local (if the web service was hosted in the default web site that would be super easy - just put address="http://localhost/..." as the endpoint address!).

The first thing I tried was to edit the hosts file on the QA server and add an entry that mapped 'app.companyName.com' to 127.0.0.1, but that did not work.

So the next (rather inelegant) thing I did was add an extra identity for the web site with a host header value of 127.0.0.1 and then change the config to read address="http://127.0.0.1/..." (I suppose I could have used localhost instead of 127.0.0.1). And that sorted everything out.

Answer 1

I suppose in production WCF and webserver are running on different machines and in DEV and QA they are not?

You need delegation of user identities via Kerberos authentication to work for impersonating the WCF. Reason: The WCF does not know the user credentials and therefore cannot properly impersonate. In the Kerberos scenario, an authentication token will be used instead of the true credentials.

If you need a starting point to get into the Kerberos topic, google for "Kerberos IIS delegation 2003 windows authentication" or similar.