How to configure Meteor Oplog Tailing on a Sharded Mongo DB

Question

As we're developing a greedy real-time monitoring application on Meteor, we reached the limit of our single MongoDB instance.

We migrated the DB to a sharded cluster with 2 shards for now, but we might expand up to 6 shards. (I have 2 BladeE chassis with 28 servers)

How do we configure Meteor Oplog Tailing on a mongo db cluster with sharding enabled?

Thanks,

Answer 1

Now there is a good news :) You can use sharded MongoDB database with Meteor easily with a bit of tweaking

Although Meteor core development team hasn't yet add Oplog Tailing support for sharding to their RoadMap, the workaround is so simple. You just add these 2 Meteor packages: cultofcoders:redis-oplog and disable-oplog, add another Redis server, tweak your code a bit and you are good to go.

The reason why Oplog Tailing doesn't work with sharded MongoDB database is just because core development team hasn't yet planed to support it. In fact, it's now possible to add support for sharded database. If you add a bunch of new records and read the Oplogs with tailable cursors from all shards, you will notice that MognoDB balancer will move data, lets say, from shard01 to shard02 where record id 0001 got removed from shard01 and added to shard02. This situation seems to be confusing for Meteor as it doesn't know whether records are actually removed/added by users or by MongoDB balancer. However there is a way to know whether users or MongoDB balancer removed/added the data because we can distinguish by fromMigrate flag — read more about this at MongoDB official site blog — so what we can do for now is to wait for update from core development team or work around with few tricks.

And the most promising workaround I've found so far is Meteor package called cultofcoders:redis-oplog. It's opensource and available on Github (Please check up their repository for full documentation. It's very-easy-to-use). The idea behind this package is to use another Redis server as pub and sub system — it doesn't store any data. it's just for pub and sub — instead of Meteor's which heavily rely on Oplog. This way, we don't have to worry about Oplog for sharded database that Meteor hasn't yet supported. Redis is mature and has been being used in production by many big companies. It's fully compatible with Meteor and you don't have to change the way you use Meteor. However you have to tweak your code a bit when updating collection's data in order to publish changes to Redis, and then the package cultofcoders:redis-oplog will handle the rest.