Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to communicate Web and Worker dynos with Node.js on Heroku?

Web Dynos can handle HTTP Requests

and while Web Dynos handles them Worker Dynos can handle jobs from it.

But I don't know how to make Web Dynos and Worker Dynos to communicate each other.

For example, I want to receive a HTTP request by Web Dynos

, send it to Worker Dynos

, process the job and send back result to Web Dynos

, show results on Web.

Is this possible in Node.js? (With RabbitMQ or Kue or etc)?

I could not find an example in Heroku Documentation

Or Should I implement all codes in Web Dynos and scaling Web Dynos only?

like image 943
jwchang Avatar asked Jul 11 '12 09:07

jwchang


People also ask

How do dynos work heroku?

The containers used at Heroku are called “dynos.” Dynos are isolated, virtualized Linux containers that are designed to execute code based on a user-specified command. Your app can scale to any specified number of dynos based on its resource demands.

Do worker dynos sleep heroku?

In addition to the web dyno sleeping, the worker dyno (if present) will also sleep. Free web dynos do not consume free dyno hours while sleeping. If a sleeping web dyno receives web traffic, it will become active again after a short delay (assuming your account has free dyno hours available).

What is web and worker in heroku?

Web: Web dynos are dynos of the “web” process type that is defined in your Procfile. Only web dynos receive HTTP traffic from the routers. Worker: Worker dynos can be of any process type declared in your Procfile, other than “web”. Worker dynos are typically used for background jobs, queueing systems, and timed jobs.


1 Answers

As the high-level article on background jobs and queuing suggests, your web dynos will need to communicate with your worker dynos via an intermediate mechanism (often a queue).

To accomplish what it sounds like you're hoping to do follow this general approach:

  • Web request is received by the web dyno
  • Web dyno adds a job to the queue
  • Worker dyno receives job off the queue
  • Worker dyno executes job, writing incremental progress to a shared component
  • Browser-side polling requests status of job from the web dyno
    • Web dyno queries shared component for progress of background job and sends state back to browser
  • Worker dyno completes execution of the job and marks it as complete in shared component
  • Browser-side polling requests status of job from the web dyno
    • Web dyno queries shared component for progress of background job and sends completed state back to browser

As far as actual implementation goes I'm not too familiar with the best libraries in Node.js, but the components that glue this process together are available on Heroku as add-ons.

Queue: AMQP is a well-supported queue protocol and the CloudAMQP add-on can serve as the message queue between your web and worker dynos.

Shared state: You can use one of the Postgres add-ons to share the state of an job being processed or something more performant such as Memcache or Redis.

So, to summarize, you must use an intermediate add-on component to communicate between dynos on Heroku. While this approach involves a little more engineering, the result is a properly-decoupled and scalable architecture.

like image 173
Ryan Daigle Avatar answered Oct 01 '22 04:10

Ryan Daigle