Sending C# Workers over different Virtual Machines

Question

My question is very simple, but the answer might not be.

I have thousands of workers performing the same task, and I would like them to be executed in a parallel fashion on many remote virtual machines (Cloud or Network)

Here is the idea :

class ThreadManager
{
    public void main()
    {
      for (int i = 0; i<300; i++)
      {
        myWorker wk;
        if (i < 100)
            wk = new myWorker(IP_Computer_1);
        else if (i < 200)
            wk = new myWorker(IP_Computer_2);
        else 
            wk = new myWorker(IP_Computer_3);

        wk.RunWorkerAsync();
      }
    }

    internal class myWorker :: BackgroundWorker
    {     
       public string IP_Computer;
       {...}//constructor

       protected override void OnDoWork(DoWorkEventArgs e)
       {              
          WriteToDatabaseTable("BAZINGA !  Greetings from computer " + Dns.GetHostName());
          //SQL server DB is hosted on a publicly accessible domain
          base.OnDoWork(e);
       }
    }
}

Of course, this is pseudo code, you get the idea : Assigning threads/workers to different computers/VM's over the network (not only multi-cores)

What basic/simplest options do I have ? I am open to ANY light and efficent solution (I do not want to go into complex scheduling/app workflow reengineering, etc... let's keep it as plain as it is, please)

NB : This is NOT an innocent question. I am aware of all the fuss about grid/cloud computing, HPC , Amazon web services, Azure etc... I spent a lot of time reading around and trying stuff, and my opinion is that there are a lot of (money) business going out there. Sometimes it's good to go back to basics and ask simple questions to see if we really need an overly sophisticated/complicated/invasive/expensive solution to solve basic issues (we are not a Fortune 500 with a giant network, but just a small research company with specific/atomic computing needs)

Answer 1

The simplest way to do this is to model your "DoWork" as a method on a WCF service. Use NetTCP bindings, and load balance across multiple machines. This is appropriate if the machines are stateless clones, and all state is maintained on the database. In that case you shouldn't care about which machine services your request, and you shouldn't need to futz with IP addresses at the application level.

That provides no communication guarantees, no reliability or failover. For example, if the servicing machine "dies" during a request, the request would not be restarted on another machine. This may be just fine for your needs.

If you want guaranteed delivery (aka fire-and-forget) and transaction durability, consider using the MSMQ Transport for your WCF Service. In this case the service machine dequeues transactionally, and the transaction commits only if the database update has been performed.

Answer 2

How about NGrid an open source Grid computing. If you want to do it all by yourself you need quite a bit of coding to get fault tolerance baked into your custom solution. The most difficult is keeping failures local without impacting your whole system.