Mongoose (or similar ODM) in-memory record registry?

Question

The specification

I have a MongoDB that holds a collection of records, let's call them operations for simplicity's sake. Some of these operations are running, and the ones that are running each contain a series of events that arrive in real time.

I am emitting these events in real-time via socket.io, as well as providing an API endpoint that aims to provide an up-to-date list of events.

The current situation

Seeing as events arrive in rapid succession (up to thousands per second), it seems suboptimal to .save() the record (in this instance I am using Mongoose as the object mapper) upon every incoming event. The current situation is I am throttling the .save() call to only get executed every 2 seconds. Because of this, the on-demand list is always anywhere between 0 and 2 seconds behind on the real-time stream whenever the operation is ongoing.

The proposed optimisation

I am considering implementing an in-memory "registry" of sorts that holds references to all running operations (hitting the memory limit is hardly a concern seeing as there will be no more than 10 concurrently running operations in the foreseeable future).

Whenever a request arrives, the "registry" would first be searched for the record, and if found, the latest version will be served from there. If not, it'll actually query the DB.

---

tldr: gap between real-time and on-demand events because of throttled model.save() calls, proposed optimisation is to use an in-memory store for a specific subset of records.

The question

Is this an effective optimisation or am I missing the point of Mongoose, and perhaps, overlooking other, more viable / relevant solutions?

Answer 1

Redis I would suggest to take a look at Redis if you have not considered it already, You can use Redis to save records after they are saved in MongoDB and retrieve from Redis if available, if not query MongoDB and save to redis again.

Or may be use Cassandra for both purpose ?