Service Fabric: Calls between services are delayed?

Question

We are working on a service fabric application made up of several different services, and a key part of the way our application works is that these services need to call each other in high volumes.

We have had no problems until recently when we increased the load on our application and found it massively slowed down. After much investigation and timing various things we found that the issue seems to be that when we are making a lot of calls to one type of service (of which we have several instances), the calls seemed to be some delay between us calling the service, and the service actually beginning to process the request.

We are calling between services as described by Microsoft here

To be clearer: ServiceA gets a reference to ServiceB, then calls ServiceB.GetResult(), we log the time that this method is called in ServiceA, and the first thing we do in GetResult() is log the time that the processing begins. When there is no load there is only a few ms, once we increased the load we found there to be a 4-5 second delay between these times.

Is this some kind of limit in service fabric? We have multiple instances of ServiceB and the resource usage on the cluster is essentially nothing, CPU hovers around 10% and memory use is about 1/4 on all nodes, but the throughput of the service is very low becuase it is waiting here.

Why does it wait? Is there some kind of defined limit on how different calls a service can handle at a time? Have we done something wrong with our communication?

Thank you.

Answer 1

The MaxConcurrentCalls setting seemed to be what I needed.

When connecting to service:

            FabricTransportSettings transportSettings = new FabricTransportSettings
            {
                MaxConcurrentCalls = 32
            };

            ServiceProxyFactory serviceProxyFactory = new ServiceProxyFactory(
                (c) => new FabricTransportServiceRemotingClientFactory(transportSettings));

            service = serviceProxyFactory.CreateServiceProxy<T>(serviceUri);

Creating service listeners:

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
    {
        FabricTransportListenerSettings listenerSettings = new FabricTransportListenerSettings
        {
            MaxConcurrentCalls = 32
        };
        return new[]
        {
            new ServiceInstanceListener(
            (context) => new FabricTransportServiceRemotingListener(context,this,listenerSettings))
        };
    }