What is the best way to query data in Azure Service Fabric?

Question

What is the best way to query data in Azure Service Fabric? Is there something on top of the Reliable Dictionary? For example: map / reduce?

Example: A customer object is stored in a reliable collection. The key is the CustomerID. And now I like to find all customers with surname that starts with "A" which are coming from "Sydney" and have ordered something within the last month.

What is the best way to implement this query functionally within Azure Service Fabric? How would the performance look like? Lets assume there are several hundred thousand of customers in the list.

Answer 1

Reliable Collections are IEnumerable (or support creating enumerators), similar to single-machine .NET collections, which means you can query them with LINQ. For example:

IReliableDictionary<int, Customer> myDictionary =
   await this.StateManager.GetOrAddAsync<IReliableDictionary<int, Customer>>("mydictionary");

return from item in myDictionary
       where item.Value.Name.StartsWith("A")
       orderby item.Key
       select new ResultView(...);

Performance depends on what kind of query you're doing, but one of the major benefits of Reliable Collections is that they are local to the code doing the querying, meaning you're reading directly from memory rather than making a network request to external storage.

With a very large data set you'll end up partitioning the service that contains the dictionary (a built-in feature in Service Fabric). In that case, you will have the same query in each partition but then you'll need to aggregate the results. Typically in this case it's good to set up another dictionary that can act as an index for common queries.