Background tasks in Meteor

Question

I'm wondering, is there is way to implement background taks, maybe with workers pool. Can you show me direction, i'm thinking about writing package for this?

Answer 1

2019 update

Before thinking about writing a package for anything, first look if there are existing packages that do what you need. In the Meteor world, this means looking on Atmosphere for "job/queue/task/worker management/scheduling" packages, then on npm for the same search terms. You also need to define your requirements more precisely:

• do you want persistence, or would an in-memory solution work?
• do you want to be able to distribute jobs to different machines?

Meteor-specific

• job-collection - reliable (I used it in 2014 in production at a startup), but currently in maintenance mode. Lets you schedule persistent jobs to be run anywhere (servers, clients).
• SteveJobs - actively maintained by Max Savin, the author of several powerful Meteor tools
• littledata:synced-cron - "A simple cron system for Meteor. It supports syncronizing jobs between multiple processes."

Abandoned packages:

• artwells:queue - priorities, scheduling, logging, re-queuing. Queue backed by MongoDB. Last code commit: 2015-Oct.
• super basic cron packages: easycron. Last update: Dec 2015.
• differential:workers - Spawn headless worker meteor processes to work on async jobs. Last code commit: Jan 2015
• cron (since 2015)
• PowerQueue - abandoned since 2014. Queue async tasks, throttle resource usage, retry failed. Supports sub queues. No scheduling. No tests, but nifty demo. Not suitable for running for a long while due to using recursive calls.

Npm packages

Meteor has been able to use npm packages directly for several years now, so this question amounts to finding job/worker/queue management packages on NPM. If you don't care about persistence:

• Async "provides around 70 functions that include the usual 'functional' suspects (map, reduce, filter, each...) as well as some common patterns for asynchronous control flow (parallel, series, waterfall...)"
• d3-queue - minimalistic, written by D3 author Mike Bostock

If you do want persistence, since Meteor uses MongoDB already, it may be advantageous to use a job scheduling package with persistence to MongoDb. The most powerful and popular seems to be Agenda, but unfortunately it hasn't been maintained in months, and it has a significant backlog of issues.

If you're willing to add a dependency backed by redis to your project, there are more choices:

• bull - the most full-featured job queue solution for Node, backed by Redis
• bee - simple, fast, robust. Does not suffer from a memory leak that Bull exhibits
• Kue - priority job queue for Node

Like MongoDB, Redis can also provide high-availability (via Redis Sentinel), and if you want to distribute jobs among multiple worker machines, you can point them all at the same Redis server.