Why does my application wait for synchronization so much?

Question

I am working on a performance test which involves several clients bombarding the server with 150 requests each, as fast as they can.

The server is constructed from 3 WCF services, one is opened to outside with httpbinding, it talks to 2 other services via net.pipe (IPC). one of the services is in charge of DB connections (SQL server 2008 R2).

This DB connection service uses the following connection string enhancments:

Min Pool Size=20; Max Pool Size=1000; Connection Timeout=20;

and is WCF throttled(like all other WCF services).

I noticed that when I activate 1 client it may take 3 seconds, but when I activate 3 clients it may take 8-9 or more.

I checked with the SQL server profiler to see how many concurrent processes are used, and I saw that only about 8 processes are being used.

So I realized that somewhere in the server the requests get queued instead of concurrently processed.

In order to get to the bottom of it I've used a performance profiler (ANTS to be exact) which showed me that about 70% of the time was wasted on "Waiting for synchronization

When I open the call graph I find two things that look strange but I'm not sure what they mean:

1. System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke is being used in the top of the tree, is that ok for concurrent processing?
2. All synchronized problems involve some kind of SQL Server activity, like ExecuteNonQuery, ExecuteReader and so on (When I get to the bottom of the call tree)

I noticed that the DB connection service uses a DAL project(some legacy code, unfortunately) which is totally static.

After reading this I'm not sure if DAL's code is problematic or not, here is a sample of a stored procedure call.

    public static int PerformStoredProcedure(string storedP,string ext,out string msg)
    {
        msg = "";
        SqlCommand command = GetSqlCommand(storedP,ext);
        command.Connection.Open();
        int result = (int)PerformStoredProcedure(command,out msg);
        command.Connection.Close();
        return result;
    }

This Method is usualy called from the DB connection service:

    public static int PerformStoredProcedureWithParams(string storedP,string ext,out string msg, params object[] pars)
    {
        msg = "";
        SqlCommand command = GetSqlCommand(storedP,ext);
        UpdateCommandParams(command, pars);
        command.Connection.Open();
        int result = (int)PerformStoredProcedure(command,out msg);
        command.Connection.Close();
        return result;
    }

So, is there anything wrong here?

Or maybe I should look somewhere else?

Edit:

After Brijesh's comment I realized I hadn't change the default InstanceContextMode and ConcurrencyMode of the WCF services... Kind of beginner's mistake I guess.

I'm still not certain if I should use PerSession/Multiple or PerCall/ Single. As I see it each service should handle each request as on object, regardless of the client.

What should I use?

2nd EDIT:

After using PerCall and PerSession/Multiple, I noticed that there is still no change (at least in the DB service). what I see is that the main entry point service may open up alot of threads, but only a few (still around 8-10 threads) are opened at the DB connection service.

is there any other reason why this can happen? I ruled out DAL being a problem because not enough requests go in the DB service so I figure its something in the service or something in the clients...

3rd EDIT:

Here are the config files:

Manager's config wcf service section:

<services>
  <service behaviorConfiguration="ServicesBehavior" name="Verifone.GenericPP.GPPManagerService.GPPManagerServiceImpl">
    <host>
      <baseAddresses>
        <add baseAddress="http://localhost:9090/GPPManagerService/"/>
      </baseAddresses>
    </host>
    <endpoint contract="Verifone.GenericPP.GPPManagerService.IGPPManagerService"  binding="basicHttpBinding" address="GPPManagerService"></endpoint>
  </service>
</services>
<behaviors>
  <serviceBehaviors>
    <behavior name="ServicesBehavior">
      <!--amith 13-05-2012-->
      <serviceThrottling
        maxConcurrentCalls="1000"
        maxConcurrentSessions="1000"
        maxConcurrentInstances="1000"
      />
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<bindings>
  <basicHttpBinding>
    <binding name="basicHttpBinding" maxBufferSize="10000000" maxReceivedMessageSize="10000000">
      <readerQuotas maxStringContentLength="10000000" maxArrayLength="10000000"/>
      <security mode="None">
        <transport clientCredentialType="None"/>
      </security>
    </binding>

Manager's clients:

      <endpoint name="endpoint1" contract="IDBConnectionContract"  bindingConfiguration="basicHttpBinding"  binding="basicHttpBinding" address="http://localhost:9010/DBConnectionService/DBService"></endpoint>
  <endpoint name="endpoint2" contract="IGPPService"  bindingConfiguration="basicHttpBinding"  binding="basicHttpBinding" address="http://localhost:9095/GenericPPService/GenericPPService"></endpoint>

DB Connection Service:

<service behaviorConfiguration="ServicesBehavior" name="Verifone.DBConnectionService.DBConnectionContracImpl">
    <host>
      <baseAddresses>
        <add baseAddress="http://localhost:9010/DBConnectionService/"/>
        <add baseAddress="net.pipe://localhost/DBConnectionService/"/>
      </baseAddresses>
    </host>
    <endpoint contract="Verifone.DBConnectionService.IDBConnectionContract"  binding="basicHttpBinding" address="DBService"></endpoint>

    <endpoint contract="Verifone.DBConnectionService.IDBConnectionContract"  binding="netNamedPipeBinding"  bindingConfiguration="NetNamedPipeBinding_Configuration"  address="" name="pipeEndpoint"/>
  </service>

The business logic service's client is pretty much the same like the Manager's.

All services are self hosted and I have a DBConnectionProxy class at Manager's and Business code which they activate like this:

 DBConnectionContractClient _dbConnectionContractClient = null;
        try
        {
            objDBConnectionContractClient = new DBConnectionContractClient();
            objDBConnectionContractClient.ExecuteStoredProcedure(input, out result);
        }

Answer 1

PerCall You may consider this instancing mode in these circumstances.

• If your service is stateless

• If your service has light-weight initialization code (or none at
  all).

• If your service is single threaded.

Some good tutorials. See the third link on tuning.

WCF Instancing, Concurrency, and Throttling – Part 1

WCF Instancing, Concurrency, and Throttling – Part 2

WCF Instancing, Concurrency, and Throttling – Part 3