How to implement MapReduce in C# using PLINQ?

Question

How to implement MapReduce in C# using PLINQ?

Suppose, you have 7-8 WebServices to collect data and on each receiving (async manner) you have to put that data into some tables of a database, in my case it is SQL Server 2008. For instance the data you are getting from each Web Service is:

 <employees>
  <employee>
   <name>Ramiz</name>
  </employee>
  <employee>
   <name>Aamir</name>
  </employee>
  <employee>
   <name>Zubair</name>
  </employee>
</employees>

And, on each receiving of response this data goes into a table name - Employee:

Employee
===
EmployeeID (PK)
EmployeeName

Once the data goes into table, it has to return as json to the client which is ASP.NET (MVC 3) application is making this call using client-side JavaScript (ajax).

Suppose, a WebServiceEmployee1 has returned with data and other 6 are in queue (still trying to get the data). Then, it should goes register the resultset into table instead of waiting other 6 and return data of inserted employee to client in json. And, keep it connected and doing while others do the same way.

Please see, in my toolbelt I have ASP.NET MVC 3 (Razor), SQL SERVER 2008 R2, jQuery.

Thanks.

Answer 1

Without clear understanding on the processing you want to perform, I suggest reading though the following MSDN documentation: http://bit.ly/Ir7Nvk It describes map/reduce with PLINQ with examples like:

public IDMultisetItemList PotentialFriendsPLinq(SubscriberID id, 
                                           int maxCandidates)
{
  var candidates = 
    subscribers[id].Friends.AsParallel()                    
    .SelectMany(friend => subscribers[friend].Friends)
    .Where(foaf => foaf != id && 
           !(subscribers[id].Friends.Contains(foaf)))
    .GroupBy(foaf => foaf)                               
    .Select(foafGroup => new IDMultisetItem(foafGroup.Key,
                                       foafGroup.Count()));
  return Multiset.MostNumerous(candidates, maxCandidates);
}

The "map" being Friends.AsParallel, SelectMany, and Where and the "reduce" phase is the GroupBy and Select