How to write scaleable code in Meteor

Question

I've found some articles about Meteor and its scaling capabilities, but non of them fully explains how to do it. Most of them promotes Oplog Tailing, but that's not something I as a programmer needs to think about when I write my code, right?

When I look at modulus.io's pricing modal, they offer me a varies number of servos my app will use, and claim that "Each Servo runs a single instance of your application." Running multiple instances is the way to scale a Meteor application, right?

But what problems come with this? Race conditions with the database can occur, right? How can I handle those? Do I need to design my application so race condition never occurs, or can I handle them when they occur?

Before thinking about scaling, I used to use setInterval to run background tasks (for example, to send me an email each hour with some data). However, I don't want each instance of my app to do that, but just one. What's the solution to this? Is it possible with Meteor?

Answer 1

To have a scalable Meteor app there are several things you need to seriously consider. Here are only the design considerations, less so on the proxying techniques or load balancing methods.

1) Avoid writing to or reading from the file system. Meteor's design decision of having an Assets.getText/getBinary only for files bundled in production, but no intuitive way to read or write files for this reason. If you have multiple dynamos you can get a request that can have data only on one server but not the other. Use S3 or GridFS to store any uploaded files off your drones.

2) If you have a Meteor.setInterval in your code, it will fire in concurrency with other 'drones', so make sure you instead code it out to work separately to the app itself via a cron job or making sure only a single drone of your app does this.

3) If you have myCollection.find().observe.. anywhere in your code on the server side that isn't to do with a client instance, avoid it. This may create race conditions since all of them will get the information at the same time. Consider creating a separate app or making sure only a single drone can run these or they will run multiple times.

4) Avoid the use of variables that store user state. E.g you have a global variable like UsersOnline = [] that you alter using a Meteor method or publish method. Consider storing the states on mongo or on redis so that each drone can access each other's states.

5) Make sure that each of the drones uses the oplog instead of the polling interval system. Otherwise there will be a delay in the other drone of your app when data is changed by another drone.

Be careful with queries that you use to ensure that oplog tailing is supported for them. Not all query types are supported, particularly where observers are involved. You can use the facts package to determine this.

6) Consider only using your drones for the websocket/ddp server connection. That is use WebAppInternals.setBundledJsCssPrefix('https://xxx.cloudfront.net'); (if you use cloudfront, any other cdn works fine) as a CDN in front of your app. This ensures that your app loads fast but each drone isn't serving up static assets again and again. It also decreases each of their load

7) Be careful with any packages you add and make sure they also follow the above.

On the cron jobs/observers

You can either use a separate Meteor app, or make sure a single instance of your Meteor app does this. If you have access to an api that can determine which drones are online, you can make the one with the lowest port do the task. Be careful that you don't make one do a task on a constant port since that single drone may be offline, as a consideration of redundancy.