Real-time web apps with mongodb and nodejs

Question

I've been thinking about how to make a real-time web application using nodejs/socket.io/mongodb. The idea is pretty similar to google docs, where objects edited on a page are emitted and rerendered on all client browsers.

What is the best way to go about doing this? From what I've read I can think of 3 ways:

1) Using mongodb oplogs

Add a 'listener' to mongodb collections, rerender parts of page whenever changes are made to collection (cons: slow?)

2) Using local json

Retrieve mongodb data into json file, use fs to edit, save to mongodb and delete json when done (cons: cumbersome to have extra layer between database and actual app)

3) Using purely socket.io

Rerender without storing, save only after all changes have been made (cons: files probably not rendered correctly in all browsers)

Is there a better way to achieve this? (How does google docs work anyway?) Would really appreciate any help anyone can offer!

Answer 1

We built a real-time app last year, basically a tool for authors to work on the same page where they could add/remove/edit elements (text, images, videos, etc.)

What we used were:

• Node.js, with Hapi.js framework (express based)
• Socket.io
• No MongoDB but the awesome RethinkDB instead, which is real-time by default and basically uses listeners to tell you whenever something has changed. (mongoDB sucks in our opinion, we used it in the past and it feels like "never again", but that's our opinion)
• React/Redux in order to update the DOM only for elements that have changed, Angular with its both-ways wouldn't have worked well in our opinion, since several users may modify the same page at the same time and therefore re-rendering all the elements would cause to lose the focus.

And honestly, it's pretty awesome how fast it is.

Answer 2

This is easy to solve without much of complication and saving documents to databases. You should only save document locations. Node has some very awesome features built for this kind of applications. I recommend you to look into these topics:

• EventEmitters

• Streams

Node filesystem has classes that you can use to build this for documents:

• fs.FSWatcher

• fs.ReadStream

• fs.WriteStream

You can use socket.io to hook up these events to your client application.

Answer 3

I would go with option 1 & 3 but with slight difference. 1. The first option to tail mongoDB opLog is a good one but the overhead becomes very big on the DB where your app will be making millions of transactions. The meteorJS library is already doing this and you can explore them as they are mature and stable to use than writing our own services.

3. Option 3 to use socket.io. You can actually use socket.io for publishng changes as well as writing to database if you are using rethinkDB which supports native changefeeds. Native changefeeds meaning, everytime there is a write to the table/collection that you want to watch realtime, it gives a callback with the old and new data where you can use socket.io to publish to all clients.
4. Another way to do it which is a more robust solution is using rabbitMQ like how Paul had mentioned above.

Answer 4

If I were to do this, I'd probably use a blend. Redis or rabbitmq to manage the socket.io connection list to get the publish and subscribe behavior as quick as possible, with a timer job that periodically flushes the writes of the document to the mongodb for longer term persistence, though arguably you could leave all docs in Redis if you wanted.

Answer 5

"Building a collaborative document editing application" is actually a chapter in the book "Mastering Node.js". They use:

• ShareDB is a realtime database backend based on Operational Transformation (OT) of JSON documents. It is the realtime backend for the DerbyJS web application framework. https://github.com/share/sharedb

• Waves are hosted, structured documents that allow seamless and low latency concurrent modifications. To provide this live experience, Google Wave uses the Operational Transformation (OT) framework of concurrency control. https://svn.apache.org/repos/asf/incubator/wave/whitepapers/operational-transform/operational-transform.html

• Quill is a modern WYSIWYG editor built for compatibility and extensibility. https://github.com/quilljs/quill

• WebSocket and websocket-json-stream

• GitHub repo of the complete source code: https://github.com/PacktPublishing/Mastering-Node.js-Second-Edition/tree/master/Chapter06/sharedb

Also, MongoDB recently published a white paper about data streaming with Apache Kafka to achieve real-time capability: https://webassets.mongodb.com/kafka_and_mongodb.pdf