C# Web API - Store And Persist Data inside In-Memory Cache

Question

I am writing a REST API which needs to provide integration services with my organization's ActiveDirectory, specifically to query user and group data and then provide an endpoint in the API for an autocomplete field query.

My organization's ActiveDirectory is very large and it has about 130K user and group objects combined.

Querying all of these objects and storing them in our current backing store (MongoDB) takes approximately 40 minutes.

We decided to check if there is an option to skip the usage of Mongo and store all of the queried AD objects in the Web API memory.

Looking at other questions in SO I realized a Singleton wouldn't work because the data stored inside it will be lost every time the IIS Application Pool is reset, and then the API can't provide data for about 40 minutes which can't happen.

I also looked at this question which refers to the namespace System.Runtime.Caching. But MemoryCache provided by the namespace will also lose all of it's data upon reset of the IIS Application Pool.

My question is - is there any other solution to store the data from AD inside the Web API memory. We currently want to avoid using a persistent store (either relational or document DB) to hold the information, but if no viable solution appears we might stick to Mongo (unless a better store is offered).

Answer 1

In-Memory sounds like a bad plan from a scalability stand point. If you want to load balance your API, you could potentially have multiple copies of this data In-Memory, but they could all be slightly different leading to different results when one instance of your API handles requests or another instance.

In-Memory, probably infers a list, or better still a dictionary. You could consider Redis server which supports multiple nodes, and its stores the data in memory. Its similar to the dictionary in that you store key value pairs as <string, string>. All the instances of your API could point to the same Redis cluster so you would gain consistency, scale and performance.

You could also consider Service Fabric which has special Collections that allow you to share state across stateful and stateless services. Like Redis the data is serialised and stored across the Service Fabric cluster. Its very fault tolerant, has inbuilt HA and DR.

Nothing would beat a Singleton Dictionary in terms of performance though so it depends what you need now and in future.