pattern for .NET parallelism beyond a single computer

Question

I suspect that I will soon exhaust the speed improving possibilities of threading on multiple cores in a single computer.

What does this .NET desktop programmer need to learn to move a parallel-feasible problem onto multiple computers? My preference is to minimize the total lifecycle programming effort so it would be preferred if there were minimal changes between on-premises deployment and off-premises deployment.

With respect to programmer man-hours, is linux, LAMP or some other stack way better than C#.NET on Windows for such an application?

Edit: Some additional information from my own comments below. The compute-intensive part of the problem can be made arbitrarily large so overheads to distribute/recombine are not a worry because overhead will be only a small percentage of the time you have to wait for a result. This is a one man development team. Just a suggestion and I don't know if it is any good or not: how about WCF and XML as means to distribute the problem in a completely on-premises Azure-ignorant way and trust that it will (someday) work on Azure without changes and without the benefits of being Azure aware. This is just an unresearched idea and I'm hoping somebody has a better one even if it is not a Windows solution.

Another edit: Digipede has an offering for performance improvements and a paper on the distinction between a cluster and a grid.

http://www.digipede.net/downloads/Digipede_CCS_Whitepaper.pdf

Since my problem is more grid-like than cluster and I want to do it cheaply, I'll just try the WCF approach.

Answer 1

The main thing to watch out for when moving from multi threaded to distributed computing is the increased overhead to spool up jobs on remote machines compared to spooling up another thread on the current machine. The granularity of the work items needs to be large enough to justify significantly slower communication between nodes - messaging between threads on the same computer is many orders of magnitude faster than messaging between different computers over the network.

Sharing resources is more difficult across machines. Sharing objects in memory is simple in multiple threads in the same process, but takes some engineering to achieve similar across machines. Locks basically don't exist across machines. Look to using a message queue service/server to coordinate work between multiple machines, return results to aggregator, etc.

You mention "on premises vs off premises". If you are considering off-premises computing resources, be sure to search around for cloud computing or elastic computing service providers. Oddly enough, these are not used in the same breath as parallel programming as often as you'd think. Cloud computing offers you the option to scale your parallelism up to hundreds or thousands of compute nodes that you pay for only while you're actually using them. When your computation is done, or the live source for your data to analyze goes home at the end of the day, you can "lights out" your cloud nodes and stop the billing clock until you start them up again.

Amazon, Google, and Microsoft are three big providers of cloud service (among others), and each has very different characteristics, strengths and weaknesses. I work on Azure stuff at Microsoft. Azure's built-in message queues are pretty slick for running producer/consumer workflows at scale.

Whether you use LAMP or .NET as your platform is really less about performance questions and more about the tools and skill sets you have within your development team. Deliberately selecting a target platform that is a mismatch with your dev team's skill set is a great way to add a lot of time and retraining costs to your project schedule.

C#/.NET works very well for coding parallel systems compared to C++ or scripting in other environments. Consider language features, debugging tools, and prebuilt libraries and services available to you when evaluating which platform is best suited to your skill set and desired system design.