Background tasks not connected with any client in Meteor

Question

I would like to run some client-independent regular tasks in the background of a Meteor app (like scraping some pages). So they should not be inside any client thread, but once they finish, I would like to update all clients with information. What is the best way to achieve this?

Answer 1

Run them on your server side code. If by regular you mean timed tasks every day or something:

You could use a cron job with Tom Coleman's cron package : https://github.com/tmeasday/meteor-cron.

You'll need to install the meteorite package manager first : npm install meteorite -g and then install the cron package in your project dir mrt add cron-tick

Server js

var MyCron = new Cron();

// this job will happen every day (60 seconds * 60 * 24)
MyCron.addJob(60*60*24, function() {
    //Scrape your stuff

    //Update your collections
});

As soon as you run your update/insert/edit they will be pushed to all clients.

Answer 2

To do this in a way that allows arbitrary external processes update the Meteor clients, use the DDP protocol that's associated with Meteor. Your server processes can write to the DDP channel, and when they do your clients will update. Have a look at this post for an example and a use case, which may be similar to yours:

Using node ddp-client to insert into a meteor collection from Node

The protocol is fairly straight forward, and the post shows an example of a node.js process writing to a Mongo collection that updates the clients in real time.

Answer 3

You could try calling a Meteor.setInterval on the server (perhaps in Meteor.startup). That should work, though it might not be as flexible as the cron solution.