Load Balancing MS Dynamics CRM 2011 Sandbox Processing Service

Question

Seems like I'm hitting a wall with this one, and not having much documentation on the matter isn't helping either.

I run large custom import jobs to CRM regularly, and all of them involve calling the organization service from a custom client to push records into CRM. The records go through custom sandboxed plugins.

The imports take quite some time and I'm trying to improve the import speed by setting up the Sandbox Processing Service on multiple servers.

Here's what I tried so far:

1. Add another server with the Sandbox Processing Service.

                             / Sandbox Processing Service (crmsb1)
   Client > Front End (crm) <
                             \ Sandbox Processing Service (crmsb2)
   

   Adding another server with the Sandbox Processing Service didn't help. The Front End server didn't distribute the load onto both servers, it continued to use the original server. Only when disabling the Sandbox Processing Service on the first Back End server did the second server start processing plugin requests. (Bringing back the first online didn't help.)

2. Two load balanced servers with all roles.

                           / Full Server (crm1)
   Client > IIS ARR (crm) <
                           \ Full Server (crm2)
   

   I follwoed this guide (and lots of others) to the letter and got the Web Application working through the reverse proxy, but failed executing requests through the Organization Service. After eliminating countless WCF errors by continuously changing yet another setting, I've hit a wall with this error The security context token is expired or is not valid. The message was not processed. leaving me to assume that load balancing will only work for the Web App.

3. Call the Organization Service manually on multiple servers.

           / Full Server (crm1)
   Client <
           \ Full Server (crm2)
   

   I was successful in setting up the import client to split the import job into multiple chunks and calling all the Organization Service endpoints in a round robin manner for each of the chunks simultaneously. This significantly sped up the import process, as I hoped, and gives me the capability to scale even further, if needed.

I read anything I could find on the matter, but I'm still confused exactly how load balancing works in CRM 2011 and what configurations are possible.

These two guides used load balancing extensively, but they do not describe in detail how they set it up.

This is all setup in a test environment before I go live. I used domain accounts for all the services, setup the correct SPNs, and didn't use SSL. I tried with & without Client Affinity in ARR (it works with cookies, so I'll assume that the WCF service isn't locked to one backend).

My questions are as follows:

1. Is it possible to setup the Sandbox Processing Service itself (1 above) on multiple servers in an active/active setup (load balanced), or is it only supported as active/passive (HA)?

2. If the only way to use multiple Sandbox Processing Service instances, is by setting up multiple servers with the Front End and Back End roles (2 above), is there any way to use a load balancer, like NLB or ARR, for the Organization Service, or is a load balancer only supported for the Web App, and I have to use the 3rd method above to load balance the Sandbox Processing Service?

Answer 1

This is in response to your comment on my article (http://bit.ly/QOEvLF) I will cross post the reply there as well:

The Sandbox services are not able to sit behind a load balancer - or the way they're designed doesn't lend itself well to that configuration. I believe the CRM Implementation guide contains information on this, but the Sandbox services will open a network Socket and listen on a specific port for traffic. When the sandbox service successfully comes online it calls back to CRM to register itself as "available" (so services like async and front end know that it's available). The roles that require isolation (sandboxing) will then round robin call into the sandbox services directly over that TCP channel on demand. For more information around this see this: http://bit.ly/CRMSetupFAQ. Also be advised that you may need to setup a custom SPN for the sandbox service as well.

Let me know if you have further questions on this - thanks!

Sean