Sporadic exceptions calling a web service that is load balanced

Question

I have a web service that I am running on three load balanced web servers and I am getting sporadic errors. Now, I admit that the load balanced part may be a bit of a red herring, but when I test with only 1 web server I cannot reproduce the error. If I test with all three web servers I can get the error (but it is not 100% of the time, more like 50%). All testing is done through the load balancer, we just tell the load balancer how many servers we want to farm.

The code is simple single request code. That is, there is no state. A request is made and a response is returned. The web service code is c# .NET 4 running on IIS 7.5. The client code is both a web site and a desktop app.

I get one of two exceptions:

System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> System.ServiceModel.FaultException: The security context token is expired or is not valid. The message was not processed.

Or I get:

System.ServiceModel.Security.SecurityNegotiationException: Secure channel cannot be opened because security negotiation with the remote endpoint has failed. This may be due to absent or incorrectly specified EndpointIdentity in the EndpointAddress used to create the channel. Please verify the EndpointIdentity specified or implied by the EndpointAddress correctly identifies the remote endpoint. ---> System.ServiceModel.FaultException: The request for security token has invalid or malformed elements.

As you can see from the following snips from my .config files, I am not using security as this is strictly an internal web service. (names have been changed to protect the innocent--namely me).

Server Side:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Service Side web.config -->
...
  <system.serviceModel>
    <serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
    <services>
      <service behaviorConfiguration="InternalUseOnly.InternalUseOnlyServiceBehavior" name="InternalUseOnly.InternalUseOnlyService">
        <endpoint address="" bindingNamespace="http://somecompany.com/webservices" binding="wsHttpBinding" contract="InternalUseOnly.IInternalUseOnlyService">
          <identity>
            <dns value="localhost" />
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="InternalUseOnly.InternalUseOnlyServiceBehavior">
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
...
</configuration>

Client side

<?xml version="1.0" encoding="UTF-8"?>
<!-- Client Side web.config -->
<configuration>
...
    <system.serviceModel>
      <bindings>
        <wsHttpBinding>
          <binding name="WSHttpBinding_IInternalUseOnlyService" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
            <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
            <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
            <security mode="Message">
              <transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
              <message clientCredentialType="Windows" negotiateServiceCredential="true" algorithmSuite="Default" />
            </security>
          </binding>
        </wsHttpBinding>
      </bindings>
      <client>
        <endpoint address="http://intranet.somecompany.com/InternalUseOnly/InternalUseOnlyService.svc" binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_IInternalUseOnlyService" contract="InternalUseOnlyService.IInternalUseOnlyService" name="WSHttpBinding_IInternalUseOnlyService">
          <identity>
            <dns value="localhost" />
          </identity>
        </endpoint>
      </client>
    </system.serviceModel>
...
</configuration>

Thoughts anyone?

---

Additional information: After reviewing the answers below I have tried two things, both without success.

The most obvious change (which I did not notice at first) was to change one of properties on the client to allow cookies <system.serviceModel><bindings><wsHttpBinding><binding name="blah, blah, blah" ... other properties... allowCookies="true" /> It defaults to false. Further, our load balancer uses cookies to keep affinity. But, it did not make a difference (no clue why yet).

Next, I tried various security options in the client side app.config file. This included both <security mode="None" /> and a more elaborate:

<security mode="None">
    <transport clientCredentialType="None" proxyCredentialType="None" />
    <message clientCredentialType="None" establishSecurityContext="false" negotiateServiceCredential="false"/>
</security>

although the settings in the last one was just a guess on my part. I did not make any server side changes to the app.config as I don't know what to change and, sadly, I can only test with production as we only have 1 dev web server, not three.

Answer 1

I am going to go out on a limb here and guess that the security involved is the Message security specified on the client side:

<security mode="Message">
    <transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
    <message clientCredentialType="Windows" negotiateServiceCredential="true" algorithmSuite="Default" />
</security>

If you are creating a client and connecting, the negotiated windows credential token may be cached. If you don't have sticky sessions enabled, the token might be passed back to the wrong server and will fail. My guess is that its always on the second call?