Enterprise Server WCF Stress/Harness Testing

Question

I am writing in order to ask a question on WCF performance.

1. Background

We have a client-server system that runs on .NET 3.5. The server is a C# service and the client is a silverlight app.

I have written a stress tester that is a winforms app and works like that:

• It spawns worker processes
• The worker processes fire different stress loads [eg for (i=0;i<100;i++) Send(payload)]
• The service is running as [ServiceBehavior(InstanceContextMode =InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Single)].
• All the timeouts (operation, send receive,..) have been set to maximum
• serviceThrottling: maxConcurrentSessions="200"
• message tracing is enabled and prints everything

2. The Issue

After some thresholds(dependent on the stress test loads and the environment under test) my worker processes (normal .NET 3.5 C# console apps) begin to throw exceptions. I was getting timeoutExceptions initially but after a lot of read & tweak sessions I got to a point where I get only two exceptions (only under very high load)

• The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
• An error occurred while receiving the HTTP response to . This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.

Alongside with that I can see substantial message loss. The server never fails(which is good). I think the server is simply rejecting messages it cannot handle.

3. Actions Taken

I have read a lot of articles and tweaked a lot of parameters(including throttling, timeouts, etc). For example I have tried all of the sugggestions here: http://www.codeproject.com/KB/WCF/WCFThrottling.aspx

I do get performance improvement but depending on what environment I am running the server I will hit this upper limit and start having messages rejected.

No errors at all are reported in the trace logs

4. Ideally...

...I would want to have a server that gets slower as load increases, but does not miss messages. (ie its ok if a client takes 10 secs to get a reply if 500 others are using the service)

Is this possible under WCF over Http and a single-threaded, single context servive? Am I missing a setting here or am I hitting an upper bound and getting the expected WCF behaviour?

NB: I have tried using ConcurrencyMode.Multiple. I do get an improvement (ie more load is needed to hit the upper bound) but I do hit the same exceptions.

Answer 1

Your best option would be to use the netMsmqBinding. Here is an overview of how it works if you're not familiar. This binding will allow your service to queue the excess load using MSMQ, so that messages will not be lost. This should solve your problem.

Answer 2

I guess I 'found' my answer!

I will explain here what I did in case someone else bumps into this one.

First of all you can check the Calls Oustanding counter, which is something like a T*CP backlog*.

In addition in perfmon can see how many calls are being handled concurrently, and how many are queued.

These two indicators allowed me to validate that indeed I was hitting the roof. Therefore I needed code that would send/rcv some ACKs upon messaging. Luckily MS have already done this for us! Its called reliable WCF messaging.

http://msdn.microsoft.com/en-us/library/ms730123.aspx http://msdn.microsoft.com/en-us/library/aa480191.aspx

By implementing a WsHttpBinding the server could cope under loads where it previously failed. In addition the server managed to cope with loads twice as much as before.

Clearly there is a performance overhead but it is indeed reliable.

Great, so I guess that now that I found the solution I can tell to the users that all of that fantastic stuff is out there, but they ain't gonna get nothing of this because their client runs on silverlight 3 which does not have reliable WCF implementations...yey!