WCF maxes CPU when waiting on _TransparantProxyStub_CrossContext function during call

Question

I'm getting heavy CPU usage when making calls to Cisco's AXL SOAP API using WCF. I start by creating a service model clientbase using generated classes from wsdl. I'm using basichttpbinding and transfermode as buffered. When executing a call, the CPU maxes out, and a CPU profile shows that 96% of CPU time is at _TransparentProxyStub_CrossContext@0 from clr.dll that is called after calls such as base.Channel.getPhone(request);. More correctly, the call maxes out the CPU core that the process is running on.

Here's a snip of the client creation from the wsdl generate

[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.ServiceModel", "4.0.0.0")]
public partial class AXLPortClient : System.ServiceModel.ClientBase<AxlNetClient.AXLPort>, AxlNetClient.AXLPort
{

    public AXLPortClient()
    {
    }

    public AXLPortClient(string endpointConfigurationName) : 
            base(endpointConfigurationName)
    {
    }

    ...

This is how I create the client:

public class AxlClientFactory : IAxlClientFactory
{
    private const string AxlEndpointUrlFormat = "https://{0}:8443/axl/";

    public AXLPortClient CreateClient(IUcClientSettings settings)
    {
        ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, errors) => true;
        ServicePointManager.Expect100Continue = false;                      

        var basicHttpBinding = new BasicHttpBinding(BasicHttpSecurityMode.Transport);
        basicHttpBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;

        basicHttpBinding.MaxReceivedMessageSize = 20000000;
        basicHttpBinding.MaxBufferSize = 20000000;
        basicHttpBinding.MaxBufferPoolSize = 20000000;

        basicHttpBinding.ReaderQuotas.MaxDepth = 32;
        basicHttpBinding.ReaderQuotas.MaxArrayLength = 20000000;
        basicHttpBinding.ReaderQuotas.MaxStringContentLength = 20000000;

        basicHttpBinding.TransferMode = TransferMode.Buffered;
        //basicHttpBinding.UseDefaultWebProxy = false;

        var axlEndpointUrl = string.Format(AxlEndpointUrlFormat, settings.Server);
        var endpointAddress = new EndpointAddress(axlEndpointUrl);
        var axlClient = new AXLPortClient(basicHttpBinding, endpointAddress);
        axlClient.ClientCredentials.UserName.UserName = settings.User;
        axlClient.ClientCredentials.UserName.Password = settings.Password;
        return axlClient;
    }
}

The generated wsdl code for the AXL API is very large. Both initial and subsequent calls have the CPU issue, although subsequent calls are faster. Is there anything else I can do to debug this issue? Is there a way to reduce this high CPU usage?

Update

A bit more info with the bounty:

I've created the C# classes like so:

svcutil AXLAPI.wsdl AXLEnums.xsd AXLSoap.xsd /t:code /l:C# /o:Client.cs /n:*,AxlNetClient

You have to download the wsdl for Cisco's AXL api from a call manager system. I'm using the 10.5 version of the API. I believe the a major slowdown is related to XML processing. The WSDL for the api is huge with the resulting classes making a 538406 lines of code!

Update 2

I've turned on WCF tracing with all levels. The largest time difference is in the process action activity between "A message was written" and "Sent a message over a channel" in which nearly a full minute passes between these two actions. Other activities (construct channel, open clientbase and close clientbase) all execute relatively fast.

Update 3

I've made two changes to the generated client classes. First, I removed the ServiceKnownTypeAttribute from all the operation contracts. Second, I removed the XmlIncludeAtribute from some of the serializable classes. These two changes reduced the file size of the generated client by more than 50% and had a small impact on test times (a reduction of about 10s on a 70s test result).

I also noticed that I have roughly 900 operation contracts for a single service interface and endpoint. This is due to the wsdl for the AXL API grouping all operations under a single namespace. I'm thinking about breaking this up, but that would mean creating multiple clientbases that would each implement a reduced interface and end up breaking everything that implements this wcf library.

Update 4

It looks like the number of operations is the central problem. I was able to separate out operations and interface definitions by verb (e.g. gets, adds, etc) into their own clientbase and interface (a very slow process using sublime text and regex as resharper and codemaid couldn't handle the large file that's still 250K+ lines). A test of the "Get" client with about 150 operations defined resulted in a 10 second execution for getPhone compared to a previous 60 second result. This is still a lot slower than it should be as simply crafting this operation in fiddler results in a 2 second execution. The solution will probably be reducing the operation count even more by trying to separate operations further. However, this adds a new problem of breaking all systems that used this library as a single client.

Answer 1

I've finally nailed down this problem. The root cause does appear to be the number of operations. After splitting up the generated client from 900+ operations to 12 each (following guidance from this question) I was able to reduce the processor time spent on generating requests to nearly zero.

This is the final process for optimizing the generated service client from Cisco's AXL wsdl:

Generate client code using wsdl like so:

svcutil AXLAPI.wsdl AXLEnums.xsd AXLSoap.xsd /t:code /l:C# /o:Client.cs /n:*,AxlNetClient

Process the generated client file to break up into sub clients:

I created this script to process the generated code. This script does the following:

1. Remove ServiceKnownType, FaultContract, and XmlInclude attributes.

These are useful for xml processing, but the generated classes appear to be incorrect from what I understand. The serviceknowntype for example, is identical for all operations even though many of the knowntypes are unique for each operaiton. This reduces the total size of the generated file from 500K+ lines to 250K+ with a minor performance increase in client instantiation time.

2. Separate out the operation contracts form the interface and methods from the clientbase that implement the interface.

3. Create subclients each with 12 operations and their respective implementation.

These subclients have three main parts. The first part is a partial class of the original clientbase client. I want this solution to be backwards compatible so I've got methods here that reference the subclient so that calls to the old super-client still work by calling the new subclient. A static get accessor will initiate the subclient if any of it's implemented operations are referenced. There is also events added for when close or abort is called so that subclients can still run these operations.

The second and third parts of the subclient is the interface and subclient class that implements the 12 operations.

I then removed the interface and client methods from the original generated client. I replaced the client constructors for the original client to simply store the binding and endpoint data for subclients to use when needed. Close and abort calls were recreated as event invokers that each subclient would subscribe to when instantiated.

Lastly, I've moved authentication to a custom endpoint behavior similar to what's described here. Using the IClientMessageInspector to send the authentication header immediately saves in one round trip call to the server where WCF likes to send an anonymous request first before authenticating. This gives me roughly a 2 sec increase depending on the server.

Overall, I've got a performance increase from 70s to 2.5s.