How to determine concurrent HTTP request bottleneck in .NET?

Question

I'm trying to make as many HTTP requests to a URL as possible, as quickly as possible.

I'm using this code to allow me to throttle the maximum Degrees of Parallelism, so I don't overflow memory by spawning lots and lots of Tasks at once.

    public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
    {
        return Task.WhenAll(
            from partition in Partitioner.Create(source).GetPartitions(dop)
            select Task.Run(async delegate {
                using (partition)
                    while (partition.MoveNext())
                        await body(partition.Current);
            }));
    }

This seems to be working fine.

body() essentially boils down to:

async Task Body()
{
    var r = WebRequest.Create("// the url");
    await r.GetResponseAsync();
}

However, I seem to have a bottleneck somewhere. If I try and do 2500 iterations, with varying values for dop I get these results:

DOP: 50
Total Time: 00:00:14.4801781
Average (ms): 246.6088
StDev: 84.1327983759009

DOP: 75
Total Time: 00:00:09.8089530
Average (ms): 265.758
StDev: 110.22912244956

DOP: 100
Total Time: 00:00:11.9899793
Average (ms): 344.9168
StDev: 173.281468939295

DOP: 200
Total Time: 00:00:09.1512825
Average (ms): 627.0492
StDev: 572.616238312676

DOP: 500
Total Time: 00:00:09.3556978
Average (ms): 1361.5328
StDev: 1798.70589239157

DOP: 750
Total Time: 00:00:12.6076035
Average (ms): 2009.058
Normal Total: 5022646
StDev: 2348.20874093199


DOP: 1000
Total Time: 00:00:11.4721195
Average (ms): 2453.782
StDev: 2481.56238190299

DOP: 2000
Total: 00:00:11.6039888
Average (ms): 4100.5536
StDev: 2459.36983911063

Which seem to suggest that dop=50 is less than the bottleneck. When you get above dop~=100 however, you'll notice the Average time each request takes (thats the average of how long Func<T, Task> body takes to run 2500 times) increases almost linearly with DOP (theres a bit of noise in these results admittedly, but theyre repeatable with small error).

This suggests that there is a "queue" inside the work body is doing, right?

I'm already setting

ServicePointManager.DefaultConnectionLimit = int.MaxValue;

and if i do

servicePoint = ServicePointManager.FindServicePoint("// the url", null);

and monitor

servicePoint.CurrentConnections 

on each execution of body, its always equal to dop (except for the inital ramp up and tail off).

I've tried this from various networks, so its unlikely to be hardware based, and it shouldn't be the remote server as its designed for heavy inbound loads (not that the numbers i'm talking about are even heavy)

How can I better profile what I'm doing?

Answer 1

The total time to perform all that work levels of between 9 and 11 seconds. That makes sense because when increasing the DOP (exponentially) you'll eventually saturate the backend resource or the network or something else.

I bet if you had posted lower DOP benchmark numbers we would see higher total times.

When you double the number of concurrent requests at this point the average completion time doubles.

Look at the throughput measured in items per second or the total time taken. That's the interesting metric. Per-item latency is not.